Inpaint Anything

Inpaint Anything Official Website, Stable Diffusion Plugin, One-click Object Removal, Content Filling, Scene Replacement

About Inpaint Anything

Inpaint Anything is an AI image replacement and repair system that combines visual fundamental models such as SAMQ, image repair models (e.g., LaMa), and AIGC models (e.g., Stable Diffusion). Based on this system, users can conveniently use IA for image replacement, processing images with any aspect ratio and 2K high-definition resolution, without being limited by the original content of the image, and with ease of use.

Inpaint Anything Official Website:https://github.com/Uminosachi/sd-webui-inpaint-anything

Core Concept of Inpaint Anything
The core concept behind Inpaint Anything is to combine the advantages of different models to build a powerful and user-friendly image repair Q system.

I. Inpaint Anything: One-click Object Removal, Content Filling, Scene Replacement
The IMCL laboratory proposed the "Inpaint Anything" model (shortened as IA) based on the image segmentation fundamental model SAM (Segment Anything Model) released by Meta.
Features:
1. Remove Anything: Click on the object you want to remove, and IA will remove it seamlessly;
2. Fill Anything: Further, you can tell IA what you want to fill in the object through text prompts, and IA will generate the corresponding content-filled object through the embedded stable diffusion model, realizing free-form "content creation";
3. Replace Anything: By clicking and selecting the object you want to keep, and telling IA what background you want to replace it with through text prompts, you can replace the object's background with the specified content, realizing vivid "environment transformation".
Overall Framework:

Main Functions of Inpaint Anything

Inpaint Anything has three main functions
1. Remove Anything: Users just need to click on the object they want to remove, and IA will remove it seamlessly, achieving efficient "magic removal";
2. Fill Anything: At the same time, users can also tell IA what to fill in the object through text prompts (Text Prompt), and IA will generate the corresponding content-filled object through the embedded AIGC (Al-Generated Content) model (such as Stable Diffusion), realizing free-form "content creation";
3. Replace Anything: Users can also click and select the object they want to keep, and tell IA what background they want to replace it with through text prompts, and then replace the object's background with the specified content, realizing vivid "environment transformation".
4. Remove 3D Material Items

Repair Any Content for Stable Diffusion Web UI

Inpaint Anything extends the use of any mask selected from the Segment Anything output to perform stable diffusion repair in the browser UI.

With Segment Anything, users can simply point to the required area to specify the mask, instead of manually filling them. This can improve the efficiency and accuracy of the mask creation process, potentially resulting in higher-quality repair results while saving time and effort.

Working Principle of Inpaint Anything

Inpaint Anything combines SAM, image repair models (e.g., LaMa), and AIGC models (e.g., Stable Diffusion) and other visual fundamental models.

• SAM (Segment Anything Model) can generate high-quality object segmentation areas through input prompts such as points or boxes, achieving the segmentation of specified targets. More related introductions can be found in One-click Image Segmentation.
• The LaMa image repair model can arbitrarily delete various elements in high-resolution images. The main architecture of the model is as follows. It includes a black and white mask image and an original image. After covering the mask image over the image, it is input into the Inpainting network, first downsampled to a low resolution, then passed through several fast Fourier transform (FFT) convolution residual blocks, and finally upsampling to generate a high-resolution repaired image.

• The AIGC model Stable Diffusion can quickly convert a simple text input into an image. More related introductions can be found in AI Painting.

Combining the three models, we can create many functions. This article implements three functions: removing anything in images/videos, filling anything in images, and replacing any background in images, with specific implementation steps as follows:

Single Machine Version

Installation

To install this software, please follow the steps below:

• OpenExtensionsAUTOMATIC1111's Stable Diffusion Web UI tab.
• Select theInstall from URLoption.
• https://github.com/Uminosachi/sd-webui-inpaint-anything.gitEnter theURL for extension's git repository.
• Click theInstallbutton.
• After installation, restart the Web UI.
• Note: This extension supports AUTOMATIC1111's Stable Diffusion Web UI v1.3.0 or higher version.

Run the Application

• If you plan to use memory-efficient xformers, please add the parameter--xformersto your startup command. For example, run./webui.sh --xformersorwebui.bat --xformers
• Note: If you have enabled privacy protection extensions in your web browser (such as DuckDuckGo), you may not be able to retrieve masks from sketches.
• Note: In Gradio version 3.23.0 or earlier, the segmented images in the Web UI may appear very small.

Download Models

• Navigate toInpaint AnythingWeb UI tab.
• Click the button next to "Segment Anything Model ID"Download model. This includes Segment Anything in High Quality Model ID, Fast Segment Anything, and Faster Segment Anything (MobileSAM).
  • Please note that SAM provides three sizes: Basic, Large, and Giant. Remember that larger sizes will consume more VRAM.
• Wait for the download to complete.
• The downloaded model files will be stored in themodelsdirectory of the application repository.

Usage

• Drag and drop the image into the input image area.
  • You can configure the aspect ratio and balance throughPadding optionsand then click theRun Paddingbutton to achieve drawing.
  • TheAnime Stylecheckbox enhances the detection of segmentation masks, especially in anime-style images, but at the cost of slightly lower mask quality.
• Click theRun Segment Anythingbutton.
• Use sketches to indicate the area to be repaired. You can undo and adjust the pen size.
  • Hover the mouse over the SAM image or mask image, then pressSfull-screen mode key orRreset zoom key.
• Click theCreate maskbutton. The mask will appear in the selected mask image area.

Mask Adjustment

• Expand mask regionbutton: Use this button to slightly expand the mask area to achieve a wider coverage range.
• Trim mask by sketchbutton: Click this button to exclude the sketched area from the mask.
• Add mask by sketchbutton: Click this button to add the sketched area to the mask.

Inpainting Tab

• Enter the required prompts and negative prompts, and then select the repair model ID.
• Click theRun Inpaintingbutton (Note: The first-time download of the model may take some time).
  • In the "Advanced" options, you can adjust the sampler, sampling steps, guidance ratio, and seed.
  • If theMask area Onlyoption is enabled, the modification will be limited to the specified mask area.
• Adjust the iteration slider to perform repairs multiple times with different seeds.
• The repair process is powered by the diffuser.

Advanced

• You can directly drag the repaired image into the input image field on the Web UI. (For Chrome and Edge browsers)
• To load prompts saved in PNG files, please follow the following steps: