It’s been anticipated for a while now, but today it’s finally here: Llama 3, the latest large language model (LLM) from Meta Platforms, the parent company of Facebook, Instagram, WhatsApp, Threads, Oculus VR, and more, is making its debut with claims of being among the most powerful “open source” AI models yet released. The release comes just hours after Llama 3 appeared on Microsoft’s Azure cloud service in an apparent early leak.
The Llama 3 family initially includes two versions — an 8 billion and 70 billion-parameter version, referring to the connections between artificial neurons within each model — with a 400 billion parameter model being actively trained by Meta now (though there is no timetable on when it might be released).
“From a performance perspective, it is really off the charts in terms of benchmarking capabilities,” said Ragavan Srinivasan, Meta VP of Product, in a video chat interview with VentureBeat, discussing the upcoming 400 billion parameter model.
For now, the Llama 3 8B and 70B versions offer benchmarks on par with or, in some cases, more than doubling the performance of rival models from Google (Gemma and Gemini Pro 1.5), Anthropic (Claude 3 Sonnet), and Mistral (7B Instruct). In particular, Meta’s Llama 3 does well at multiple choice questions (MMLU) and coding (HumanEval), but the 70B is not as strong as Gemini Pro 1.5 at solving math word problems (MATH), nor at graduate-student level multiple choice questions (GPQA).
The Llama 3 8B version, however, blows the competition away, outperforming Gemma 7B and Mistral 7B Instruct across benchmarks, especially on grade school math questions (GSM-8K).
“If you look at the many benchmarks that you have for LLMs, they typically fall into these five categories of general knowledge, reading comprehension, math, reasoning, code,” explained Manohar Paluri, VP of AI at Meta, in a video conference interview with VentureBeat. “What you see with this release, specifically Llama 3 8B and 70B, they are better than any other open model, and even comparable to some of the best closed models and better across all of these benchmarks.”
A new stand-alone Meta AI chatbot emerges
Llama 3 is also powering a new stand-alone Meta AI chatbot available on the web at www.meta.ai, offering a more direct competitor to OpenAI’s ChatGPT, Anthropic’s Claude 3, and HuggingFace’s HuggingChat.
As Meta CEO and founder Mark Zuckerberg stated in a video posted to his Instagram account today: “The bottom line is we believe Meta AI is the most intelligent AI assistant that you can freely use.”
Like those rival chatbots, the stand-alone Meta AI offers the familiar method of entry of a dialog box at the bottom and the chat conversation with the LLM bot atop it. You can use it without being signed into Facebook, but its capabilities are limited and there is an age gate you’ll have to enter first.
Also like ChatGPT, Meta has bundled into its Meta AI chatbot an image generation model — in this case, its own Meta Imagine (based on its own diffusion AI model formerly known as Emu, trained on hundreds of millions of Facebook and Instagram user's photos).
You’ll need to be signed into the Meta AI chatbot with your Facebook account to use this, but we tested it and it works quickly and similarly to ChatGPT’s bundling with OpenAI’s DALL-E 3 image generator.
Unlike ChatGPT’s DALL-E 3 integration, it doesn’t appear you can adjust the aspect ratios for the images generated, but it also offers something DALL-E 3 and other image generators don’t yet: watermarking in the left corner displaying that it is an AI-generated work.
The Meta Imagine image generation model has also been updated to allow for near real-time generations as a user types, so you’ll see “a dog” generated and then one “eating pizza” if you so type, right as you type it. Meta says you can play back the generation process as a video animation or GIF, though we’ve yet to try this ourselves.
Like ChatGPT, Meta AI will surface information live from the web from Microsoft’s Bing search engine. Like Google’s Search Generative Experience, it will also surface information from Google search results.
One big omission, though, compared to other leading closed LLMs is that Llama 3 is not multi-modal: you can’t upload images or documents to it the way you can with ChatGPT or Gemini, though Meta sources told VentureBeat a multi-modal version of Llama is coming at some point soon.
Another big advantage over rivals: is Meta’s massive scale of billions of users across its multiple products. As such, many people will likely be using Meta AI since it is also being bundled into “the search boxes at the top of WhatsApp, Instagram, Facebook and Messenger,” according to Zuckerberg.
Open source?
“Open source” is in quotes when talking about Llama 3 because it is being released under the same Meta-specific license as its predecessor, Llama 2, rather than the more industry-standard Apache 2.
While Meta does allow Llama to be used by third-party enterprises for commercial purposes, the license specifically states that those enterprises with “greater than 700 million monthly active users in the preceding calendar month…must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.”
This terminology has been criticized by members of the open-source community as a way for Meta to maintain control of its technology and users and stifle competition. Yet it has not stopped other companies such as France’s Mistral from using Llama 2 as the basis for newer open-source models.
And Llama 3’s creators at Meta affirm that they are committed to open source.
How Llama 3 8B and 70B were created, and how they differ from Llama 2
In addition to boasting improved benchmark performance that Meta says is on par with leading proprietary/closed-source LLMs, Llama 3 improves on its predecessor family in several ways.
Namely, that it offers “reduced false refusal rates, improved alignment, and increased diversity in model responses. We also saw greatly improved capabilities like reasoning, code generation, and instruction,” the company writes in its news release.
This is in large part due to its training process: the company says it “put substantial effort into scaling up pretraining,” and “combined three types of parallelization: data parallelization, model parallelization, and pipeline parallelization,” resulting in a 3-times improved training efficiency over Llama 2.
Llama 3 was also trained on more than 15 trillion tokens “all collected from publicly available sources,” 7X more than Llama 2, according to Meta. Tokens in LLMs refer to discrete numerical representations of concepts and underlying data, including portions of words or ideas.
The company says it will publish a detailed paper on Llama 3’s training process once it completes the 400B version.
Llama 3 70B also offers an 8,000-token context window, about double the previous version, meaning you can enter more information and longer prompts (though far behind the 128,000-context token window offered by OpenAI’s GPT-4 Turbo and Google’s Gemini Pro 1.5)
For now, Llama 3 is available for download here on Meta’s AI website and is also being made available on Amazon Sagemaker, AWS, Databricks, Google Cloud, Hugging Face, Kaggle, IBM Watson, Microsoft Azure, NVIDIA NIM, and Snowflake.