GuanacoAI

Official website of Guanaco, fine-tuning the LLaMA model framework, performance close to GPT-4

What is Guanaco?

The NLP group at the University of Washington proposed a method called QLoRA, which can fine-tune a LLaMA model with 65 billion parameters on a GPU with 48GB of VRAM, with minimal performance loss. QLoRA is a new method for fine-tuning large language models on a single GPU. Recently, researchers at the University of Washington developed Guanaco using the QLoRA method, a series of chatbots based on the Meta LLaMA model. The largest Guanaco variant has 65 billion parameters and achieved over 99% performance similar to ChatGPT (GPT-3.5-turbo) in benchmark tests against GPT-4.

Guanaco Reference URL:  https://github.com/artidoro/qlora

Guanaco Technical Details

The QLoRA method was proposed under the framework of the LLaMA model, including the following core technical principles and differences from other methods:

1. 4-bit NormalFloat: This theoretically optimal quantization data type is used for processing normally distributed data, superior to 4-bit integers and 4-bit floating-point numbers.
2. Double Quantization: By quantizing quantization constants, the average memory usage can be reduced, with each parameter saving an average of about 0.37 bits (about 3GB for a model with 65 billion parameters).
3. Paged Optimizers: Use NVIDIA unified memory to avoid gradient checkpoint memory peak during processing of small batch long sequences.

Through the QLoRA method, researchers fine-tuned thousands of models in tests, showing that Guanaco models fine-tuned based on QLoRA outperform all current open-source models, even more powerful than ChatGPT, and close to GPT-4. The Guanaco model includes four versions, from 7 billion to 65 billion parameters, all of which will be open-sourced.

This research also includes a large amount of data and conclusions related to large models, suggesting reading the original paper for more detailed information. The paper covers the memory usage of different models, the bias of current large model evaluation benchmarks, and more.

The proposal of the QLoRA method aims to solve the problem of how to effectively fine-tune large language models to optimize their performance. Although large models have powerful capabilities, fine-tuning is still necessary for specific domain datasets, as enterprises and individuals may have proprietary data and knowledge in specific domains, and the zero-shot ability of general models is difficult to extend to these specific domains.

The QLoRA method provides a solution for fine-tuning large models on GPUs with 48GB of VRAM, reducing memory requirements while maintaining the original model's performance through quantization and other optimization methods.

In summary, the QLoRA method has the potential in large model fine-tuning, providing more efficient memory usage and superior performance, further promoting research and application in the field of language models.

QLoRA is a new method that can fine-tune large language models on a single GPU. Recently, researchers at the University of Washington developed Guanaco using the QLoRA method, a series of chatbots based on the Meta LLaMA model. The largest Guanaco variant has 65 billion parameters and achieved over 99% performance similar to ChatGPT (GPT-3.5-turbo) in benchmark tests against GPT-4.

Fine-tuning large language models is a key technology for improving performance and training effects. However, this process is computationally intensive for large models, such as the LLaMA65B model, which may require more than 780GB of GPU VRAM. Although the open-source community has already used various quantization methods to compress 16-bit models to 4-bit models, greatly reducing the memory required for inference, similar methods have not yet been adopted in the fine-tuning process.

The QLoRA method fills this gap, making it possible to fine-tune large language models on a single GPU. It effectively reduces memory requirements while maintaining the original model's performance through new quantization methods and memory optimization techniques.

The release of the Guanaco chatbot series further proves the feasibility and practicality of the QLoRA method. The Guanaco series models perform almost as well as ChatGPT while using significantly less VRAM. This means developers and researchers can train and fine-tune powerful large language models under relatively small computational resources, accelerating innovation in natural language processing and artificial intelligence.

It should be noted that although the QLoRA method and Guanaco models have made significant progress in research and academic fields, they are currently not allowed for commercial use. However, the continuous development and optimization in this field has laid a solid foundation for future commercial applications.

The proposal of the QLoRA method brings new possibilities for fine-tuning large language models, especially in resource-constrained environments. It provides researchers and developers with more flexible and efficient tools to accelerate the development of natural language processing and artificial intelligence technology, paving the way for future innovation.

Researchers demonstrated a method for quantizing models like LLaMA to 4-bit and fine-tuning them using the QLoRA method, and introduced the Guanaco model as an example. Through this method, they successfully reduced the memory requirements of a 65 billion parameter model from over 780GB to less than 48GB of GPU memory while achieving results comparable to fine-tuning 16-bit models.

The emergence of the QLoRA method marks a major breakthrough in the accessibility of large language model fine-tuning, becoming the largest publicly available model fine-tuned on a single GPU to date.

The research team conducted multiple experiments, training over 1000 models to test the impact of QLoRA and different fine-tuning datasets. Their findings show that data quality is more important than the number of tasks. Models trained with human samples from the OpenAssistant dataset performed better than those trained with a large number of samples from the FLANv2 dataset. Therefore, the Guanaco model mainly relies on the OpenAssistant dataset.

The Guanaco model series trained through the QLoRA method performed exceptionally well, with the second-best model achieving 97.8% ChatGPT performance with 33 billion parameters in benchmark tests and completing training on a single regular GPU in less than 12 hours. On a professional GPU, the largest model with 65 billion parameters reached 99.3% ChatGPT performance in 24 hours.

The smallest Guanaco model has only 7 billion parameters and requires only 5GB of GPU memory, performing over 20 percentage points better than the 26GB Alpaca model in the Vicuna benchmark test.

In addition to QLoRA and Guanaco models, the research team also released the OpenAssistant benchmark test for model competition and scoring by humans or GPT-4.

However, the research team points out that there are still issues with mathematical ability and 4-bit inference speed, and they hope to increase the inference speed, expecting an 8 to 16-fold increase.

Fine-tuning is an important tool for transforming large language models into chatbots like ChatGPT. Researchers believe that the QLoRA method will make fine-tuning easier to achieve, especially for researchers with limited resources. They believe this is a huge victory for the accessibility of cutting-edge technologies in the field of natural language processing.

In addition to fine-tuning the largest language models today, researchers also look forward to the possibility of applying private models on mobile hardware. They said that QLoRA will also enable privacy-preserving fine-tuning on smartphones, with an expected 3 million words to be fine-tuned each night using an iPhone 12 Plus. This means that in the future, we will be able to run specific large language models for each application on smartphones.

More information and code about Guanaco-33B can be found on GitHub.