NetEase Fuxi

NetEase Fuxi,AI large model,virtual human,robot,game AI research laboratory

Official website of NetEase Fuxi, AI large model, virtual human, robot, game AI research laboratory

Introduction to NetEase Fuxi

A professional institution in China dedicated to game AI research, committed to creating a world-class laboratory at the intersection of artificial intelligence and gaming. Utilizing cutting-edge AI technology to create a new generation of gaming experiences for players, while leveraging the massive data and simulation environment of gaming platforms to promote the development of AI technology.

NetEase Fuxi Official Website:https://fuxi.163.com/

How is NetEase Fuxi?

Since the beginning of the Year of the Rabbit, the development of generative AI (also known as AIGC) has been progressing rapidly. There are many new messages and achievements released almost every week, with lower barriers and better effects constantly challenging public perception, making more and more people realize that generative AI has become an important driving force for the next round of technological innovation. At the same time, more and more people are beginning to think about some questions, such as: why the best generative effects are not in China? How far is China's generative AI from abroad? To create the best generative AI, in addition to the model, what else do we need to build?

As the first domestic institution to focus on digital entertainment in the field of artificial intelligence research, NetEase Fuxi has been paying attention to the development and implementation of generative AI since its establishment in 2017.These issues have been continuously mentioned and discussed within the team over the past few years, driving a series of work to be carried out and promoted. This article will introduce NetEase Fuxi's thinking on these issues and some current progress.

Natural Language and Generative AII

This round of explosion of generative AI needs to start with the breakthrough in natural language processing technology. In 2017, Google proposed the Transformer architecture, which allows computers to learn text content knowledge more efficiently, thus promoting the birth of a series of large-scale text models such as BERT and GPT, achieving huge breakthroughs in both understanding and generation.

Natural language processing, as the basic discipline that connects human and computer communication, has also become the core foundation driving the explosion of generative AI in various dimensions, including text, image, audio, video, and 3D models. On the one hand, people sort out data that can be used for training generative AI from massive internet data. On the other hand, natural language is used to align information across various modalities, so that these knowledge can be intercommunicated. This also explains why companies and institutions in the English ecosystem are more likely to take the lead in this round of technological boom — the largest, most content-rich, and highest-quality machine learning corpus currently is composed of English.

For example, in the field of text downstream fine-tuning data, there are many high-quality datasets such as T0-SF and Muffin in the English field, and there are also open-source datasets such as LAION-2B and MSCOCO in the field of images and texts. Compared with China, although there have been multiple related dataset construction efforts in China in the past two years, such as the 200G Wudao text pre-training dataset and the "Wukong" 100 million image-text pair dataset, there is still a certain gap in terms of quantity and quality compared with overseas data.

In addition, the English ecosystem itself also has very clear inherent advantages, which include a large amount of high-quality content that other languages do not have. For example, the world's top academic papers, programming code, and specifications and standards in various industries. These constitute the unique advantages of English, and also make it possible for research and implementation plans based on the English ecosystem to better promote and implement.

How to Break Out of the Data Dilemma

In the face of such a data dilemma, what strategies have domestic researchers and institutions adopted? In summary, there are about 4 strategies:

1. Directly use open-source models and go through API translation

This may be the most direct solution, especially in the field of image and text generation. Last year, after the stable diffusion model was open-sourced, many domestic start-up companies tried to adapt and train based on the model, and at the same time, use the API translation interface to convert Chinese input into English to support Chinese users. The advantage of this route is that it can quickly apply the latest work in the English ecosystem to China. The disadvantages are also very obvious. On the one hand, Chinese translation may cause semantic loss, and many commonly used expressions in the English field cannot be well expressed in Chinese, such as many Chinese proverbs and sayings:

The water flows down from the top of the mountain for three thousand feet from MidJourney

A bamboo staff and straw sandals are lighter than a horsefrom MidJourney

The content composition of overseas data is mostly composed of local human geography, history, and life, and lacks a good understanding of Chinese knowledge, such as Chinese historical sites, celebrities, food, and living customs.

The third point is also the most crucial point: the existing open-source models have data bias, and there are risks in compliance and security. For example, these models are unequal in racial issues, and there are also a large number of nude and violent content. Directly using these data models for domestic production poses huge hidden dangers, so since the beginning of the year, relevant departments have increased the audit intensity of the capabilities of generative AI.

2. Overseas data translation

This solution is an improved version of the first solution. Institutions with certain research capabilities will choose to translate overseas data into English after sorting it out, and build more reliable self-owned models by leveraging the existing achievements of English data. Currently, many research institutions and enterprises in China have adopted this route. The advantage is that it can inherit the rich data ecosystem of English, and at the same time, it can systematically screen out data involving pornography and politics. However, the disadvantages still exist, including the lack of domain differences, including the lack of some specific Chinese expressions, ecosystems, and cultural customs, as well as the strong bias and even discrimination in the data itself. Even after removing non-compliant data, these implicit problems are still difficult to solve. For example, "a girl in a cheongsam," "Qixi Festival," etc.

3. Chinese data construction

This is a relatively difficult path that requires a lot of preliminary accumulation. The work of data sorting often takes a short time to obtain results, and itsvalue at each stage is also difficult to measure. However, the construction of high-quality and high-quality data will bring reliable assistance to the promotion of generative AI later. Therefore, in the past discussions of Fuxi, it has also been identified as a difficult but correct path. The advantage of building a Chinese dataset is that it can solve a series of basic issues in the Chinese scenario, make up for the lack of Chinese knowledge in the model, better control data security, and thus effectively review the compliance of data.

Currently, there are also some domestic institutions that have done work on Chinese data construction, such as the Coco-CN dataset with a high-quality alignment of tens of thousands of numbers, and the Wukong dataset, which is currently the largest-scale open-source image-text dataset. However, compared with overseas equivalent datasets, there is still a certain gap. In many scenarios, relevant researchers also begin to call on domestic governments and enterprises to promote the joint construction of high-quality Chinese datasets, and we have seen many domestic peers joining this ranks.

4. Multilingual compatibility

Although building a dataset is good, it still cannot solve the problem of a lack of high-quality content in other languages. Therefore, multilingual compatibility is currently the most feasible solution for large-scale pre-training model technology. Of course, this solution is still under verification, and there are already some related works. Through multilingual solutions, the understanding and generation functions of images and texts in the English scenario are extended to other languages, and the barriers between the English system and other languages are removed.

In the training process of ChatGPT, the feasibility and potential of cross-language have been demonstrated. Due to the integration of a large amount of multi-source data, the Chinese ability of GPT has become more excellent than many pure Chinese pre-training models. In the field of image and text generation, the cross-language ability and generation effect of the Niji model are also very good.

Fuxi's Breakthrough Move

Considering the overall effect of generative AI, Fuxi has chosen a relatively long-term technical route. While staying compatible with open-source data, it also advances in 4 steps: first, building a high-quality large-scale Chinese dataset; second, building high-quality understanding models in the Chinese field; then reconstructing the image and text generation algorithm based on the dataset and understanding model to achieve effective semantic improvement; finally, introducing expert and human feedback to guide the model to generate high-quality content that users need more.

1. Building a large-scale Chinese dataset

Fuxi, in collaboration with multiple departments of NetEase, including NetEase Leihuo, Media, Cloud Music, and other core businesses, provides an understanding and demand for data from the user and business dimensions, completes the definition of high-quality data, and builds evaluation criteria including text quality, image beauty, copyright compliance, and ethical assessment. Using this framework as a constraint, the data construction is jointly promoted, and a data credibility system based on distributed tasks is designed. Each expert team provides a data quality review model, completes joint scoring, and then hands it over to the data governance engine for unified management.

2. Building understanding models in the Chinese field

Based on the technical accumulation of Fuxi's self-researched billion-text model, the "Yu Yan" series of understanding models have successively topped the well-known Chinese list FewCLUE and CLUE classification list, and have exceeded human level in the CLUE1.1 classification task list (including AFQMC [text similarity], TNEWS [short text classification], IFLYTEK [long text classification], OCNLI [natural language reasoning], WSC [pronoun disambiguation], CSL [keyword recognition] 6 understanding tasks). The billion-scale generation model and dialogue model in the Yu Yan series have been open-sourced, and the understanding model will also be open-sourced in the near future.

Based on the foundation of text understanding, Fuxi has been focusing on building the "Yu Zhi" multimodal image-text understanding large model since 2021, adopting the dual-tower structure of image-text and modular training ideas, and iteratively developing three specifications of model versions based on the billion-scale Chinese image-text data pairs. In the field of Chinese image-text understanding, it has reached the leading level in the industry and has good generalization. It performs excellently in various downstream tasks such as classification, retrieval, and recommendation. At the same time, in the pre-training process of the image-text model, different training strategies are adopted for image-text pairs with different text lengths, which makes the "Yu Zhi" model have strong semantic understanding ability. At the same time, the EET high-efficiency inference framework open-sourced by NetEase Fuxi is optimized for model compression, algorithm adaptation, and hardware infrastructure, which improves the inference speed by 4 times, meets the high concurrency requirements online, and reduces the loss of deployment resources.

In the zero-shot evaluation of business datasets

"Yu Zhi" multimodal understanding large model is superior to Chinese-CLIP's CN-CLIPViT-H/14

The Yu Zhi model has also been successfully verified in many business scenarios of NetEase, such as NetEase News and NetEase Cloud Music's search, recommendation, and intelligent labeling. NetEase News has significantly improved the recall source, list page video trial, and list page trial by using the image-text vector dropoutnet optimization based on the image-text content representation constructed by the image-text large model in the recommendation link, and achieved significant improvement in the business indicators of video and the overall business, and has been put into use online. NetEase Cloud Music has successfully applied the content representation engine and content similarity retrieval engine constructed by the image-text large model to many content business scenarios such as Cloud Music video, long audio, and advertising, bringing significant online benefits to the content cold start efficiency, CTR estimation model, etc. At the same time, further jointly with the Huawei team, fully analyze the characteristics of the Internet industry dataset, optimize the multimodal model structure, select the appropriate encoder, and adopt a multi-stage training mode, jointly build the Yu Zhi-Wukong model, and further build Fuxi's leading advantage in the field of Chinese cross-modal understanding.

3. Reconstructing the image and text generation algorithm

Based on the pre-trained image-text understanding model, Fuxi further promotes the research and development of the self-researched text-image generation model - "Dan Qing", a semantic-enhanced text-image generation diffusion model. Relying on the principle of the diffusion model, it trains on a wide range of (800 million) image-text data to achieve good generation results. Different from the common text-image generation methods based on the diffusion model, the self-researched model of Fuxi also has the following characteristics:

1. Model innovation: The semantic ability of text-image generation is very dependent on the representation ability of user input text. Relying on the representation ability of the self-researched "Yu Zhi" model in the Chinese context, the self-researched generation model has strong semantic representation ability in the Chinese scenario. In addition, the self-researched model also focuses on the interaction between text and images, strengthens the parameter role in the text-image guidance part, and allows text to better guide the generation of images, so that the generated results are closer to the user's intention.

2. Multi-scale image training: In a wide range of datasets, the self-researched model fully considers the different sizes and clarity of images, sorts different sizes and resolutions of images into buckets, and carries out multi-scale training. While fully ensuring the fidelity of the training images, it retains as much information as possible, and the self-researched model can adapt to different resolutions of generation.

3. Data strategy: Multi-stage training can ensure that the model has both breadth and quality of generation results. In the initial stage, use billion-scale widely distributed data to make the model not only have breadth in semantic understanding but also understand some idioms and ancient poems, such as "Husband's Lung Slices," "Famous Flowers and a Beautiful Country," etc. At the same time, it also has diversity in the style of generation. In the subsequent stages, data selection is carried out from multiple aspects such as the correlation between images and texts, image clarity, and image beauty to optimize the generation ability and generate high-quality images.

Strong semantic understanding ability in the Chinese scenario: It can fully understand the user's input and return what the user wants. Especially in the understanding and generation of idioms, proverbs, and poems, it has certain advantages:

Exceptional semantic understanding in Chinese contexts: Fully comprehends user input and delivers what users desire. Particularly excels in understanding and generating idioms, proverbs, and poems.