Llama 3 performance exploded, and Meta wants to use open source to compete for the throne of large

Visual China.

Text |leitech

Whatever the appearance of SORA means, the battle for the big models is far from final. If OpenAI has occupied one pole of today's closed-source large model ecosystem, then Meta is undoubtedly the other pole of open source large models.

Meta's llama (alpaca) has arguably been one of the most powerful open-source models since the beginning. In July last year, Meta released a stronger Llama 2, and the most important thing is that the open source license has shifted from research only to free and commercial, although it has also created a large number of large models in the shell, but there is no doubt that it has changed the pattern of large models in one fell swoop.

However, the AI world is changing day by day, before the French open source rookie Mistral picked out Llama 2 (in February, it launched the closed-source large model Mistral Large, second only to GPT-4), and then Google released the GEMMA open source large model to crush Llama 2.

It's time for Llama 3.

In January of this year, Meta CEO Mark Zuckerberg (hereinafter referred to as Xiaozha "Xiaoza") announced on Instagram ** that Meta AI has started training Llama 3. According to the latest disclosure by The Information, Meta plans to officially release LLAMA 3 in July this year.

Xiaozha's official announcement**, picture on Instagram

Considering that Meta spent three months training on the first generation of Llama and about six months on the Llama 2,If the next-gen model follows a similar schedule, it should also be released sometime in July 2024.

Not only that, but based on relevant reports and the information officially revealed by Meta, Llama 3, which is expected to be released in four months, has greater ambitions, and I am afraid that it will change the world of large models again as open source.

At least for the time being, computing power is still one of the key factors in the iteration of large models, and even OpenAI has to explain its computing power reserves from time to time.

In Instagram**, Xiaozha said that it is expected that by the end of 2024, Meta will have 350,000 Nvidia H100 (the official price is 3.).$50,000, which is actually still at a premium), and if other computing resources are included, it will reach nearly 600,000 h100:

At present, only Microsoft and Google may have comparable computing power reserves.

The implication of Xiaozha's words is actually obvious: Meta has enough AI computing resources to support large-scale pre-training of Llama 3 and technical research of generative AI.

The exact size of the parameters is not known at this time, but it can be expected that the previous version will continue with multiple parameter scales, and there are rumors that the largest version will exceed 140 billion parameters, directly challenging leading large models such as GPT-4.

The information also cited internal sources as pointing out that LLAMA 3 not only loosens the security fence, but also provides better answers when dealing with disputed issues than GPT-4, Gemini, and LLAMA 2. Meta clearly wants to at least provide context about the query, rather than ignoring or refusing to answer the user's questions.

In order to do this, in addition to requiring Llama 3 to improve the illusion problem of large models from technology, it may also require an improvement in the context length. On Llama 2, Meta doubles the context length, allowing the model to remember twice as much of the token context during inference (the process of chatting with the AI).

On the other hand, large model manufacturers are generally turning to the research of multimodal large models (such as Gemini and GPT-4V), that is, supporting and understanding images and audio in addition to text, and can generate text, sound and audio at the same time. Meta is probably no exception.

Although Xiaozha only confirmed that Llama 3 will include **generation functions like Llama 2, he did not explicitly mention other multi-modal features, but he still talked about the idea of combining artificial intelligence and the metaverse in the official announcement**.

Glasses are the ideal form for AI to see what you see and listen to what you hear, and Xiaozha points out that it is always available to help when talking about the Meta X Ray-Ban glasses. 」In the report on the independent AI terminal AI Pin, Lei Technology also discussed in depth the huge potential of wearable devices in the field of AI vision and hearing.

In other words, Meta's direction must be to enable AI models to have natural language understanding, sight, and hearing capabilities at the same time. It can be inferred that the multimodal support for LLAMA 3 and even subsequent generations of LLAMA can be said to be the proper meaning of the topic.

Overall, it is reasonable to expect that even if the scale of LLAMA 3 is maintained in the range of 7 billion to 70 billion parameters, it will still bring significant performance improvements, and LLAMA 3 will also bring more imagination.

What's more, Meta also has the pursuit of AGI.

It's becoming increasingly clear that next-generation services need to build comprehensive, common intelligence. Xiaozha clearly pointed out Meta's long-term goal of building AGI, to build the best AI assistants, creator AI, enterprise AI, and more — which requires advances in all areas of AI, from reasoning to planning to coding to memory and other cognitive abilities. 」

Of course, this does not mean that Llama 3 will implement (or even attempt to achieve) AGIs. But there's no doubt that Meta is intentionally doing research and development in a way that they think might eventually lead to AGI.

To be fair, whether LLAMA 3 is open source or not will have a huge impact on the entire AI industry.

As the most widely used open source model in the industry, LLAMA 2 and its upstream and downstream as important cores have begun to consciously build an ecosystem. In December last year, in order to counter the closed-source camp represented by OpenAI and Google, Meta joined forces with 57 global technology companies and research institutions such as Oracle, Intel, AMD, IBM, Sony, and Dell to form the AI Alliance.

Member of the AI Alliance, Figure IBM

The alliance has six major goals, one of which is to build an open source model ecosystem, including a complete set of processes from research, evaluation, hardware, security, and public participation.

Although Xiaozha has not clearly stated whether Llama 3 is open source or not, it is very likely that Llama 3 will continue to be open source.

After it was announced that Llama 3 was being trained, Xiaozha said in an interview with The Verge: I tend to think that one of the bigger challenges here is that if you build something that is really valuable, then it ends up being very focused. However, if you make it more open-minded, then you can solve a whole large group of problems that can arise due to inequality of opportunity and value. 」

If Xiaozha's response was more like an expression of attitude, Meta's chief AI scientist Yann Lecun's response was more to the point. In an interview with Wired, Yann Lecun noted:

When you have an open platform where many people can contribute, progress becomes faster. You end up with a system that is more secure and performs better. (At the same time) imagine a future where all of our interactions with the digital world are dominated by AI systems. You don't want AI systems to be controlled by a handful of companies on the West Coast of the United States. Maybe the Americans won't care, maybe the United States ** won't care. But I'm telling you now, in Europe, they won't like it.

In addition, regulatory issues are unavoidable. Given the disruptive nature of AI, it is also more likely to be accepted by regulators when the AI process is not completely dominated by one company, but is built by global companies and developers.

The vision is beautiful, but the reality is not. The question is, will Meta still be able to return to the open-source throne and even surpass GPT-4?

Figure X (formerly Twitter).

Time has changed, and when Llama 3 is released, Meta may be facing a completely different situation. Google's Gmoma crushing and Mistral's dark horse posture all prove that Meta's open-source throne is not stable. Especially Google, although it has been crushed by OpenAI every time, no one dares to really ignore Google's money, technology and appeal.

Of course, these questions will not be answered more definitively until Llama 3 is released. Until then, everything is possible.