How a Background Remover Is Born

Why Do I Need a Background Remover?

A background removal tool is not really a new feature, but rather its importance has grown as the world shifted over to online working and learning over the last few years. And I did not find how important this tool could be until just two weeks ago. On a warm, sunny morning with a coffee on hand, I joined an online conference. During this conference, one of my colleagues pointed out to me that they could see my untidy desk and an overflowing bin in the background. Naturally, this left me feeling embarrassed. I just wish I could travel back in time to use a background remover.

Now, I cannot travel in time, but I can certainly create a background removal tool. So, with this new-found motive, I looked online for some solutions and came across the body or head segmentation capability from HMS Core Video Editor Kit, and developed a demo app with it.

This service can divide the body or head from an input image or video and then generate a video, an image, or a sticker of the divided part. In this way, the body or head segmentation service helps realize the background removal effect.

Now, let's go deeper into the technical details about the service.

How the Background Remover Is Implemented

The algorithm of the service performs a series of operations on the input video, including extracting frames, using an AI model to process the video, and encoding. Among all these, the core is the AI model. How a service performs is affected by factors like device computing power and power consumption. Considering these, the development team of the service manages to equip it with a light-weight AI model that does a good job in feature extraction, by taking measures like compression, quantization, and pruning. In this way, the processing duration of the AI model is decreased to a relatively low level, without compromising the segmentation accuracy.

The mentioned algorithm supports both images and videos. An image takes the algorithm a single inference for the segmentation result. A video is actually a collection of images. If a model features poor segmentation capability, the segmentation accuracy for each image will be low. As a result, the segmentation results of consecutive images will be different from each other, and the segmentation result of the whole video will appear shaking. To resolve this, the team adopts technologies like inter-frame stabilization and the objective function for inter-frame consistency. Such measures do not compromise the model inference speed yet fully utilize the time sequence information of a video. Consequently, the algorithm sees its inter-frame stabilization improved, which contributes to an ideal segmentation effect.

By the way, the service requires that the input image or video contains up to 5 people whose contours should be visible. Besides, the service supports common motions of the people in the input image or video, like standing, lying, walking, sitting, and more.

The technical basics of the service conclude here, and let's see how it can be integrated with an app.

How to Equip an App with the Background Remover Functionality

Preparations

1. Go to AppGallery Connect and configure the app's details. In this step, we need to register a developer account, create an app, generate a signing certificate fingerprint, and activate the required services.

2. Integrate the HMS Core SDK.

3. Configure the obfuscation scripts.

4. Declare necessary permissions.

Setting Up a Video Editing Project

Prerequisites

1. Set the app authentication information either by:

• Using an access token: Call the setAccessToken method to set an access token when the app is started. The access token needs to be set only once.

MediaApplication.getInstance().setAccessToken("your access token");

• Using an API key: Call the setApiKey method to set an API key when the app is started. The API key needs to be set only once.

2. Set a License ID. The ID is used to manage the usage quotas of the kit, so make sure the ID is unique.

MediaApplication.getInstance().setLicenseId("License ID");

Initializing the Runtime Environment for the Entry Class

A HuaweiVideoEditor object serves as the entry class of a whole video editing project. The lifecycle of this object and the project should be the same. Therefore, when creating a video editing project, create a HuaweiVideoEditor object first and then initialize its runtime environment. Remember to release this object when exiting the project.

1. Create a HuaweiVideoEditor object.

HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());

2. Determine the preview area position.

This area renders video images, a process that is implemented by creating SurfaceView within the SDK. Make sure that the position of this area is specified before the area is created.

<LinearLayout    
    android:id="@+id/video_content_layout"    
    android:layout_width="0dp"    
    android:layout_height="0dp"    
    android:background="@color/video_edit_main_bg_color"    
    android:gravity="center"    
    android:orientation="vertical" />
// Specify the preview area position.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);

// Specify the layout of the preview area.
editor.setDisplay(mSdkPreviewContainer);

3. Initialize the runtime environment of HuaweiVideoEditor. LicenseException will be reported when the license verification fails.

The HuaweiVideoEditor object, after being created, has not occupied any system resources. We need to manually set the time for initializing its runtime environment, and then the necessary threads and timers will be created in the SDK.

try {
        editor.initEnvironment();
   } catch (LicenseException error) { 
        SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());    
        finish();
        return;
   }

Integrating the Segmentation Capability

// Initialize the segmentation engine. segPart indicates the segmentation type, whose value is an integer. Value 1 indicates body segmentation, and a value other than 1 indicates head segmentation.
visibleAsset.initBodySegEngine(segPart, new HVEAIInitialCallback() {
    @Override
    public void onProgress(int progress) {
        // Callback when the initialization progress is received.
    }

    @Override
    public void onSuccess() {
        // Callback when the initialization is successful.
    }

    @Override
    public void onError(int errorCode, String errorMessage) {
        // Callback when the initialization failed.
    }
});

// After the initialization is successful, apply the segmentation effect.
visibleAsset.addBodySegEffect(new HVEAIProcessCallback() {
    @Override
    public void onProgress(int progress) {
        // Callback when the application progress is received.
    }

    @Override
    public void onSuccess() {
        // Callback when the effect is successfully applied.
    }

    @Override
    public void onError(int errorCode, String errorMsg) {
        // Callback when the effect failed to be applied.
    }
});

// Stop applying the segmentation effect.
visibleAsset.interruptBodySegEffect();

// Remove the segmentation effect.
visibleAsset.removeBodySegEffect();

// Release the segmentation engine.
visibleAsset.releaseBodySegEngine();

And now the app is capable of removing the image or video background.

This function is ideal for e-conferencing apps, where the background is not important. For learning apps, it allows teachers to change the background to the theme of the lesson, for better immersion. Not only that, but when it's used in a short video app, users can put themselves in unusual backgrounds, such as space and the sea, to create fun and fantasy-themed videos.

Have you got any better ideas of how to use the background remover? Let us know in the comments section below.

Wrap Up

Background removal tools are trending among apps in different fields, given that such a tool helps images and videos look better by removing unnecessary or messy backgrounds, as well as protecting user privacy.

The body or head segmentation service from Video Editor Kit is one such solution for removing a background. It supports both images and videos, and outputs a video, an image, or a sticker of the segmented part for further editing. Its streamlined integration makes it a perfect choice for enhancing videos and images.