BLOOM

Official Website of BLOOM, a Large Language Model (LLM) launched by HuggingFace

What is BLOOM?

A multilingual large model for everyone! Supporting 59 languages, with 176 billion parameters, initiated by 1000 scientists! For a long time, many large models have been monopolized by large private technology companies that created them. For example, GPT-3, which is as appealing as it is inaccessible to the average person. However, now, to satisfy your curiosity, a volunteer project initiated by nearly 1000 scientists has taken over a year to develop a language model that claims to be as powerful as GPT-3. It's called BLOOM, with 176 billion parameters, supporting not only English and Chinese but also 59 other languages, making it the largest multilingual open-source language model. Yes, from now on, BLOOM is fully open-source, from code to datasets, accessible to everyone.

BLOOM Official Website:https://huggingface.co/bigscience/bloom

BLOOM Reference Documentation: 

https://huggingface.co/docs/transformers/model_doc/bloom

Utilizing French supercomputers, spending 384 A100 GPUs

BLOOM is a BigScience project, launched in May last year, led by Hugging Face, and received$700,000in public donations. It involved over 250 institutions and more than 1000 researchers from 60 countries around the world, including employees from major companies like Meta and Google participating individually. Its training was conducted on the French supercomputer Jean Zay, using 384 A100 GPUs, each with 80 GB of memory, with a training throughput of about 150 TFLOP (this supercomputer is powered by low-carbon nuclear energy, and the released heat is used to heat schools).

BLOOM Advantages

However, BLOOM officially stopped training only after reaching 102%. After 117 days of training, BLOOM successfully completed the training within the planned timeframe.

Ultimately, BLOOM has the following features:

• The number of parameters reached 176 billion, 10 billion more than GPT-3.
• The model contains 70 layers, each with 112 attention heads.
• The token sequence length is 2048.
• It uses the GeLU activation function.
• It was trained on a dataset of a total of 341.6 billion tokens (1.5TB of text data).
• It supports 13 programming languages and 46 natural languages.

It is noteworthy that for many languages, such as French, Spanish, and Arabic, this is the first time they have their own open-source model (although Japanese is not yet supported, some netizens have noticed this).

Additionally, to minimize bias in generated results, project participants underwent a round of manual filtering of the collected data.

Currently, BLOOM's training has certain hardware requirements:

To ensure optimal performance, it is best to have 8 80GB A100 GPUs or 16 40GB A100 GPUs.

This means only slightly larger teams can afford such training needs.

Of course, training can also be done on cloud platforms, with a maximum hourly cost of $40.

Finally, BLOOM states that it will lower the entry threshold and plans to develop a distributed system that allows for model sharing between laboratories.

BLOOM will become a continuously evolving model family, rather than a one-time project.

The BLOOM model has proposed various versions through BigScience Workshop. Inspired by other open science initiatives, BigScience brings together researchers' time and resources to achieve greater impact. The architecture of BLOOM is essentially similar to GPT3 (a self-regressive model for next-token prediction), but it has been trained on 46 different languages and 13 programming languages. Several smaller versions of the model have been trained on the same dataset. BLOOM has the following versions:

• Bloom-560m
• Bloom-1B1
• Bloom-1B7
• Bloom-3B
• Bloom-7B1
• Bloom (176B parameters)

Resources

Official Hugging Face and Community Resources (represented by 🌎), a list to help you get started with BLOOM. If you are interested in submitting resources to be included here, please open a pull request, and we will review them! Ideally, resources should demonstrate new content rather than duplicate existing resources.

• BloomForCausalLM includes example scripts and notebooks for causal language modeling.

See also:

• Causal Language Modeling Task Guide
• Text Classification Task Guide
• Token Classification Task Guide
• Question Answering Task Guide

⚡️ Reasoning

• About the blog: Bloom Reasoning Optimization.
• A blog about incredibly fast BLOOM reasoning with DeepSpeed and Accelerate.

⚙️ Training

• About the technology behind BLOOM training.

Bloom Configuration

Class Transformer.Bloom Configuration

< Source >

(vocab_size= 250880hidden_size= 64n_layer= 2n_head= 8layer_norm_epsilon= 1e-05initializer_range= 0.0initializer_range= 0.02use_cache= Truebos_token_id= 1eos_token_id= 2