StableVicuna

Official Website of StableVicuna, Stable Alpaca, Large-scale Open-source Chatbot

What is StableVicuna?

Renowned open-source platform Stability AI announced the release of the world's first large-scale open-source chatbot trained with RLHF, similar to ChatGPT, StableVicuna(Stable Alpaca), the first chatbot to be trained with RLHF on a large scale!In terms of language models, they have also launched the first open-source chatbot with RLHF technology, StableVicuna, based onthe Alpaca-13B modelimplementation.

StableVicuna Official Website Open Source Address:

https://huggingface.co/spaces/CarperAI/StableVicuna

https://huggingface.co/CarperAI/stable-vicuna-13b-delta

StableVicuna is a 13 billion parameter large language model with RLHF and instruction tuning capabilities, supporting text generation, basic math, coding, and grammar explanation. It is currently the open-source model closest to ChatGPT's features (test address:

Stability AI stated,ChatGPT's unprecedented success is due to two key factors: instruction tuning and RLHF.

However, it is not easy for small and medium-sized enterprises or ordinary developers to train RLHF, as reinforcement learning often encounters issues such as mode collapse, sparse feedback, difficult data set preparation, and high computational power costs.

What is RLHF

RLHF is translated as "Human Feedback Reinforcement Learning" in Chinese, which is a training method that combines human guidance with automatic reinforcement learning!Humans evaluate or guide AI behavior, helping it make better decisions during the learning process. Since humans can help AI through intuition, visual experience, and practical experience, products using RLHF see a significant improvement in anthropomorphism.

Simply put, RLHF can be seen as a "mother teaching a child" training method. AI is like a newborn child with no experience; when it falls, the mother (RLHF) teaches it how to avoid falling and better ways to walk, allowing the child to grow rapidly in this continuous feedback learning environment.

Typically, RLHF consists of four steps: unsupervised pre-training, supervised fine-tuning, reward model, and reward-based reinforcement learning.

Unsupervised Pre-training:AI starts learning a task, which may use random strategies or heuristic strategies based on certain principles. For example, starting from a pre-trained language model like GPT-3.

Supervised Fine-tuning:When AI performs a task, humans evaluate its behavior. The evaluation can be continuous, such as scoring or rewards, or discrete, such as binary feedback on whether a behavior is correct. Then, the pre-trained model is fine-tuned to enhance its anthropomorphic capabilities.

Reward Model:A reward model is established to score the output of the LLM. AI will automatically optimize its strategy based on human reward feedback, in order to better perform tasks in future outputs.

Reward-based Reinforcement Learning:AI will continuously perform tasks based on the reward model, receive human feedback, and update its output behavior according to the feedback. This process will continue until AI's performance reaches a level satisfactory to humans.

StableVicunaTechnical Principles

To make StableVicuna possess the same instruction tuning and RLHF technology characteristics as ChatGPT, Stability AI adopted a multi-modal combination method. Stability AI combined the Vicuna, OpenAssistant, trlx, and GPT4All open-source models to achieve this goal. Simply put, it mainly consists of three steps.

First, Stability AI mixed the core training datasets of OpenAssistant, GPT4All, and Alpaca to train a supervised fine-tuning basic model, Vicuna.

Then, a reward model was trained using trlx, based on the RLHF preference datasets of the aforementioned open-source models, to further initialize the model.

Finally, Stability AI used trlX for behavioral strategy reinforcement learning to train the initialized model with RLHF, ultimately achieving StableVicuna.

Stability AI expressed its gratitude to the OpenAssistant team for providing access to their RLHF dataset. Without these precious preference training sets, StableVicuna could not have been trained in such a short time.

Currently, users who wish to use StableVicuna need to obtain access to the original LLaMA model. After obtaining the weight increments and LLaMA weights, they can combine them using the scripts provided in the GitHub repository to use StableVicuna.