llama2

Llama2 Official Website, Open Source Deployment Tutorial, Model Download, Chinese, Papers, Meta's Large Language Model

Llama2 Introduction

In February this year, Meta released the first version of the Llama large language model, an important advancement just three months after the release of ChatGPT. Unlike the previous Llama pre-trained model, which was only for research purposes, the newly released Llama 2 has commercial application capabilities. Meta released three models with different parameter sizes: 7 billion, 13 billion, and 70 billion. They disclosed details such as the training data, training methods, and data annotation, fully showcasing the strength of Llama 2: Llama 2 surpasses all open-source large models in terms of capability at the same parameter size; the 70 billion parameter model is close to the inference capability of GPT-3.5 behind ChatGPT, but there is still a significant gap in coding.

Llama2 Official Website:https://ai.meta.com/llama/

Fill in the form to get the Llama2 model:

https://ai.meta.com/resources/models-and-libraries/llama-downloads/

How is Llama2?

Many developers have tested Llama 2 and basically confirmed Meta's statement: "It performs well in the coding phase." The 7 billion parameter model can run on Macs, processing 6 characters per second, 70% slower than the smallest model "Gecko" released by Google. However, Google did not disclose the specific parameters of "Gecko."

According to the information released by Meta, the training data of Llama 2 comes from public data, with a total of 2 trillion Tokens (common words, punctuation, or numbers), a 40% increase from the first generation model. Additionally, the context length of Llama 2 has been extended to 4000 characters, making it more accurate in understanding text semantics.

Similar to OpenAI, Meta also trained dialogue applications similar to ChatGPT using human feedback reinforcement learning (RLHF) mechanisms and 1 million human annotation data. This is also a commonly used method for fine-tuning Llama in the open-source community over the past few months. Meta stated, "The excellent writing capability of large language models is fundamentally driven by RLHF."

Training Llama 2 may require huge costs. HuggingFace machine learning scientist Nathan Lambert estimates that the training cost of Llama 2 may exceed $25 million, not lower than the cost of training GPT-3 three years ago. He said that there are enough signs to indicate that Meta is still training more powerful Llama models.

Since the release of ChatGPT at the end of last year, it has shocked the world with its meaningful responses and strong coding capabilities, among other features. Many companies are striving to find similar products. After more than half a year of development, from large companies to ordinary programmers, everyone can achieve similar ChatGPT applications with the help of the open-source community. According to observations from the cloud development platform Replit, the number of open-source large model projects using its services doubles every quarter.

Based on open-source large models like Llama, developers have also created various open-source datasets, such as human feedback reinforcement learning datasets, continuously enhancing the capabilities of open-source large models.

According to evaluations by professors and students from universities such as the University of California, Berkeley, and Carnegie Mellon University through LMSYS Org, over the past few months, the gap between open-source large models and GPT-4 has significantly narrowed from 191 points to the recent 115 points. In the process of catching up, the open-source community has even led large companies, providing large models that can run on computers and smartphones, released more than a month earlier than Google.

With Meta's open-source Llama 2, the strength of the open-source large model community will further enhance. Meta said that after the first open-source model version that does not support commercial use was released, they received more than 100,000 application requests from researchers, not including those who directly downloaded the model from the internet.

Large Model Development

NVIDIA senior AI scientist Jim Fan said, "Due to open-source licensing issues, large company AI researchers were cautious about the first version of Llama, but I believe that many people will join this project (Llama 2) and contribute their efforts." He also said that even though Llama 2 is not strong in programming capabilities at the moment, it will catch up quickly after the open-source release.

The largest open-source parameter version (70 billion) of Llama 2 released is still not half as large as the GPT-3 trained three years ago, but it is more effective, which is one of the best proofs.

The logic of open-source is that when large models reach a certain level of capability, they expand the scope of new technology applications, allowing more people to use the technology and improve the model from widespread applications. Closed-source companies like OpenAI, on the other hand, pay more attention to the advancement of technology and promote more people to use powerful models after developing strong models.

Just like the competition between iOS and Android in the mobile operating system field, the competition between open-source and closed-source is not always direct competition on the same dimension, and a similar differentiation will also appear in the large model field.

Under this new competitive landscape, even Google has not been able to maintain its leading confidence.

In May this year, a senior engineer at Google said in an internal article that although Google still has a slight advantage in the quality of large models, the gap between open-source products and Google's large models is shrinking at an astonishing speed. Open-source models have a faster iteration speed, and users can customize development according to different business scenarios, which is more conducive to protecting privacy data and has a lower cost.

He said, "They can achieve what we find difficult to do with a $1 million and a 54 billion parameter model in just a few weeks. We don't have a moat, and neither does OpenAI."

Yesterday, after announcing the open-source of Llama 2, Meta explained that open-source is the correct choice for the development of today's AI models, especially in the rapidly developing production field. They said, "By providing AI models to the public, we make them accessible to everyone, not just a few large companies."

In the field of generative AI, a new type of competition different from the past is unfolding. The open-source community, with the power of open collaboration, is catching up with the leading advantages established by commercial giants at an astonishing speed. And the large companies accustomed to technological closure and market monopoly are gradually embracing open-source.