New APIs for Android plugins that render to a Surface

The Android embedder for Flutter introduces a new API, SurfaceProducer, which allows plugins to render to a Surface without needing to manage what the backing implementation is. Plugins using the older createSurfaceTexture API will continue to work with Impeller after the next stable release, but are recommended to migrate to the new API.

An Android SurfaceTexture is a backing implementation for a Surface that uses an OpenGLES texture as the backing store.

For example, a plugin might display frames from a camera plugin:

In newer versions of the Android API (>= 29), Android introduced a backend-agnostic HardwareBuffer, which coincides with the minimum version that Flutter will attempt to use the Vulkan renderer. The Android embedding API needed to be updated to support a more generic Surface creation API that doesn't rely on OpenGLES.

If you are using the older createSurfaceTexture API, you should migrate to the new createSurfaceProducer API. The new API is more flexible and allows the Flutter engine to opaquely pick the best implementation for the current platform and API level.

1. Instead of creating a SurfaceTextureEntry, create a SurfaceProducer:

   java

   TextureRegistry.SurfaceTextureEntry entry = textureRegistry.createSurfaceTexture();
   TextureRegistry.SurfaceProducer producer = textureRegistry.createSurfaceProducer();

2. Instead of creating a new Surface(...), call getSurface() on the SurfaceProducer:

   java

   Surface surface = new Surface(entry.surfaceTexture());
   Surface surface = producer.getSurface();

In order to conserve memory when the application is suspended in the background, Android and Flutter may destroy a surface when it is no longer visible. To ensure that the surface is recreated when the application is resumed, you should use the provided setCallback method to listen to surface lifecycle events:

surfaceProducer.setCallback(
   new TextureRegistry.SurfaceProducer.Callback() {
      @Override
      public void onSurfaceAvailable() {
         // Do surface initialization here, and draw the current frame.
      }

      @Override
      public void onSurfaceDestroyed() {
         // Do surface cleanup here, and stop drawing frames.
      }
   }
);

A full example of using this new API can be found in PR 6989 for the video_player_android plugin.

If your plugin implements a camera preview, your migration might also require fixing the rotation of that preview. This is because Surfaces produced by the SurfaceProducer might not contain the transformation information that Android libraries need to correctly rotate the preview automatically.

In order to correct the rotation, you need to rotate the preview with respect to the camera sensor orientation and the device orientation according to the equation:

rotation = (sensorOrientationDegrees - deviceOrientationDegrees * sign + 360) % 360

where deviceOrientationDegrees is counterclockwise degrees and sign is 1 for front-facing cameras and -1 for back-facing cameras.

To calculate this rotation,

• Use SurfaceProducer.handlesCropAndRotation to check if the underlying Surface handles rotation (if false, you may need to handle the rotation).
• Retrieve the sensor orientation degrees by retrieving the value of CameraCharacteristics.SENSOR_ORIENTATION.
• Retrieve the device orientation degrees in one of the ways that the Android orientation calculation documentation details.

To apply this rotation, you can use a RotatedBox widget.

For more information on this calculation, check out the Android orientation calculation documentation. For a full example of making this fix, check out this camera_android_camerax PR.

Landed in version: 3.22

In stable release: 3.24

In the upcoming stable release, 3.27, onSurfaceCreated is deprecated, and onSurfaceAvailable and handlesCropAndRotation are added.

API documentation:

• SurfaceProducer
• createSurfaceProducer
• createSurfaceTexture

Relevant issues:

• Issue 139702
• Issue 145930

Relevant PRs:

• PR 51061, where we test the new API in the engine tests.
• PR 6456, where we migrate the video_player plugin to use the new API.
• PR 6461, where we migrate the camera_android plugin to use the new API.
• PR 6989, where we add a full example of using the new API in the video_player_android plugin.