Meta Debuts Code Llama 70B: A Powerful Code Generation AI Model

Meta updated its foundation model, Code Llama, to support 70B, which makes it a viable alternative to closed AI code models.

Code Llama 70B is described as the “largest and best-performing model” yet, capable of handling more queries than previous versions, thus allowing developers to feed it more prompts while programming and increasing its accuracy.

Code Llama 70B is built on Llama 2 and aids developers in creating snippets of code from prompts and debugging human-written work. It was trained on a massive 1TB of code and code-related data. The model is currently hosted on the code repository Hugging Face. The model is offered in three different versions and, similar to the original Llama 2 model, it continues to be available for free for research purposes. The inference code for Code Llama models is available on GitHub.

Two other Code Llama tools, Code Llama – Python and Code Llama – Instruct, focus on specific coding languages. CodeLlama-70B-Python has been trained on an additional 100 billion tokens of Python code, making it more fluent and accurate in generating Python code. CodeLlama-70B-Instruct can handle a variety of tasks, such as sorting, searching, filtering and manipulating data, as well as implementing algorithms.

CodeLlama-70B-Instruct, a variant of CodeLlama 2, is a fine-tuned variant that is specifically designed to comprehend natural language instructions and generate code accordingly. Its advanced capabilities enhance both the quality and efficiency of code generation. On HumanEval, a benchmark dataset comprising 164 programming problems designed to assess the logic and functional correctness of code generation models, it received a score of 67.8. This score is comparable to closed models like GPT-4 (68.2) and Gemini Pro (69.4) and exceeds the previous best results of open models like CodeGen-16B-Mono (29.3) and StarCoder (40.1).

CodeLlama-70B-Instruct is capable of performing a wide range of operations, including data manipulation, sorting, searching, filtering and implementation of algorithms like factorial, Fibonacci and binary search. AI coding assistants with chat capabilities can use CodeLlama-70B-Instruct. While most of the coding assistants provide inline code completion based on comments and naming conventions, the chat-based AI assistants provide an interactive experience for developers to go beyond code completion, offering them best practices and even scripts to deploy their code.

Foundation models for code, such as StarCoder, GPT-4 and CodeGen-16B-Mono, have been instrumental in the development of AI tools like Code Llama. StarCoder, for instance, is a Large Language Model for Code that outperforms existing open code LLMs on popular programming benchmarks. It can process more input than any other open LLM, enabling a wide range of applications.

GPT-4, another foundation model, is a multimodal large language model developed by OpenAI. It can understand and communicate in numerous languages and dialects and has been used to power GitHub Copilot’s assistant, “Copilot X.”

CodeGen, on the other hand, is a family of large language models trained on natural language and programming language data. It is used for program synthesis and has been trained sequentially on The Pile, BigQuery and BigPython.

With Code Llama 70B, enterprises have a choice to host a capable code generation model in their private environment. This gives them control and confidence in protecting their intellectual property.