whisper

whisper

whisper, github, OpenAI's automatic speech recognition model

Visit Site →

The lower the number, the better the performance. Compared to other models,

What is Whisper?

Whisper is an Automatic Speech Recognition (ASR) system provided by OpenAI. It is a model trained on deep learning technology and large-scale speech datasets, designed to convert speech into text. Whisper aims to provide accurate and high-quality speech recognition capabilities, making it easier for users to handle speech data and obtain relevant information.

Whisper official website:https://openai.com/research/whisper

GitHub project source code:https://github.com/openai/whisper

Hugging Face project source code:

https://huggingface.co/spaces/openai/whisper

Whisper paper:https://cdn.openai.com/papers/whisper.pdf

How is Whisper?

OpenAI's Whisper offers an Automatic Speech Recognition (ASR) model that provides speech recognition and translation features to users. The core function is speech recognition, which can quickly convert speech content like meeting recordings into text with high accuracy, and also offers translation and automatic subtitle generation. Most importantly, Whisper supports local operation, allowing speech recognition without an internet connection.

🎙️Using Whisper is simple. Users first need to configure the environment and install, then they can use Whisper WebUI for operations. On the operation interface, users need to select the appropriate model and language, then they can paste the audio URL or upload audio files from the computer or mobile phone. Next, users can adjust the parameters of the Voice Activity Detector (VAD) (for non-English audio longer than 10 minutes, it is recommended to choose Silero VAD). After submission, just wait for a while, and you can get the transcribed text and the generation of three subtitle files.

✨With Whisper, speech recognition and translation become more convenient and efficient. Whether it's organizing meeting minutes or processing recording files, Whisper can convert speech into text with excellent accuracy, allowing users to easily review and share information. Moreover, Whisper can also generate subtitles for speech files, making it easier for viewers to better understand and follow the content.

🌐And, Whisper supports local operation, meaning users can enjoy high-quality speech recognition services without relying on an internet connection. This feature makes Whisper the preferred tool for many users, whether for personal or commercial use, providing an efficient and reliable speech recognition experience.

Main Features

Whisper's main function is to convert speech into text, achieving Automatic Speech Recognition (ASR). It is trained on deep learning technology and large-scale speech datasets, and has the following main functions:

  • Speech Transcription: Whisper can transcribe speech content such as recordings, meetings, and lectures into accurate text forms. This allows users to easily review, edit, search, and share speech content, improving work efficiency and information management capabilities.
  • Subtitle Generation: Whisper can automatically generate subtitles for speech content in scenarios such as videos, movies, and speeches. This is very helpful for the hearing impaired, cross-language communication, and viewers in silent environments, providing a better understanding and experience.
  • Voice Assistant: By converting speech input into text output, Whisper can be used to build voice assistant applications. Users can interact with the application through voice, ask questions, send commands, and achieve more intuitive and convenient operation methods.
  • Data Analysis and Mining: Converting a large amount of speech data into text can provide a foundation for data analysis and mining. Users can perform operations such as keyword extraction, sentiment analysis, and topic recognition on the transcribed text to obtain valuable information and insights.

Whisper is a general-purpose speech recognition model. It is trained on a large dataset containing various audio types and is also a multi-task model that can perform multi-language speech recognition, speech translation, and language recognition.

Methods

The Transformer sequence-to-sequence model is trained for various speech processing tasks, including multi-language speech recognition, speech translation, conversational speech recognition, and voice activity detection. These tasks are represented as a series of tokens predicted by the decoder, allowing a single model to replace multiple stages of the traditional speech processing pipeline. Multi-task training uses a set of special tokens as task descriptors or classification targets.

Settings

We use Python 3.9.9 and PyTorch 1.10.1 to train and test our model, but the codebase is expected to be compatible with Python 3.8-3.11 and the latest PyTorch versions. The codebase also depends on some Python packages, most notably OpenAI's tiktoken, which implements a fast tokenizer. You can use the following command to download and install (or update to) the latest version of Whisper: pip install -U openai-whisper

Or, the following command will pull and install the latest commit and its Python dependencies from this repository:

pip install git+https://github.com/openai/whisper.git

To update the package to the latest version in this repository, run:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git

ffmpeg

It also requires the installation of command-line tools on your system, which are provided by most package managers:# on Ubuntu or Debian

sudo apt updatesudo apt install ffmpeg&&# on Arch Linuxsudo pacman -S ffmpeg# on MacOS using Homebrew (https://brew.sh/)brew install ffmpeg# on Windows using Chocolatey (https://chocolatey.org/)choco install ffmpeg# on Windows using Scoop (https://scoop.sh/)scoop install ffmpegYou may

rustalso need to install it, in case tiktoken does not provide pre-built wheels for your platform. If you encounter installation errorspip installin the above commands, please follow the instructions on the getting started page to install the Rust development environment. Additionally, you may need to configurePATHenvironment variables, such asexport PATH="$HOME/.cargo/bin:$PATH".If the installation fails No module named 'setuptools_rust'then you need to installsetuptools_rustfor example, by running:pip install setuptools-rust

Available Models and Languages

There are five model sizes, four of which are only in English versions, providing a balance between speed and accuracy. Below are the names of the available models and their approximate memory requirements and relative speeds.

Size

Parameters English-only model Multi-language model Required VRAM Relative Speed Tiny
39 meters tiny.en tiny ~1 GB ~32x Based on
74 meters base.en base ~1 GB ~16x Small
244 meters small.en small ~2GB ~6x Medium
769 meters medium.en medium ~5 GB ~2x Large
1550 meters Not applicable large ~10GB 1x .en

Models in English often perform better, especially fortiny.enmodelsbase.en.small.enWe observe that the differences become less significant formedium.en.Whisper's performance varies by language. The following chart shows the WER (Word Error Rate) breakdown by language using the Fleurs dataset with this model

large-v2(the smaller the number, the better the performance).(The lower the number, the better the performance). Compared to other models