Hello, Graphite: WebGPU
This is part two of the articles dedicated to the React Native Skia version 3.0 release. Part 1 covers performance.
A Unified GPU Runtime
React Native WebGPU is built on Dawn, Google's open-source WebGPU implementation. The goal was to bring GPU libraries beyond 2D graphics to React Native: Three.js for 3D, TensorFlow.js for machine learning, and many more. But until now, Skia had its own GPU context, separate from WebGPU, so the two couldn't share textures. With v3, Skia is just one of the libraries that run on top of the WebGPU runtime.
This means you can draw a Three.js scene on a Skia canvas, use a Skia drawing as a texture in a Three.js scene, and run a machine learning model on camera frames, all on the same GPU device.
When React Native WebGPU is installed alongside React Native Skia, both libraries automatically share the same Dawn instance. You can then import Skia's GPU device and use it via the JavaScript WebGPU API.
import { Skia } from "react-native-skia";
import { importDevice } from "react-native-webgpu";
// device is a regular GPUDevice
const device = importDevice(Skia.getNativeDevice());From there, Skia can read and write any texture created on that device, with no extra copies and no CPU readbacks. And in the other direction, WebGPU can also import Skia images as textures.
Below are the new APIs to share GPU resources between Skia and WebGPU:
| API | |
|---|---|
importDevice(Skia.getNativeDevice()) | Shares Skia's GPU device as a GPUDevice |
Skia.Image.MakeFromGPUTexture(texture) | Wraps a texture into an image Skia can draw |
Skia.Surface.MakeFromGPUTexture(texture) | Wraps a texture into a surface Skia draws into |
Skia.Image.MakeGPUTextureFromImage(image) | Draws an image into a new texture and returns a pointer to it |
adoptTexture(pointer) | Wraps that pointer into a GPUTexture, which owns the texture from then on |
Best-in-Class 2D and 3D
When React Native WebGPU came out, Three.js was what got the community most excited. But adding Skia graphics to a Three.js scene meant copying textures, with a CPU readback in between. Now both share the same runtime, and Skia brings its 2D capabilities to Three.js scenes.
In one direction, a Three.js scene is rendered into a WebGPU texture, which Skia draws as a regular image. This is what lets the helmet demo below apply backdrop filters to a Three.js scene and draw the whole futuristic 2D UI on top of it.
In the other direction, Three.js can create an external texture from a Skia image. This brings best-in-class 2D rendering, such as advanced path rendering or text layouts, into 3D scenes. The cloth demo puts it all together: a Three.js scene, textured with an animated Skia drawing, whose mesh is animated by a compute shader.
Video and Camera Feeds
In React Native Skia v2, we supported both video and camera feeds through the Native Buffer API: an abstraction over the platform GPU buffers, AHardwareBuffer on Android and IOSurface on iOS.
It was brittle, to say the least, and we never offered any guarantees about it. Limited texture-format support, manual YUV handling, and no fencing made it error-prone.
In React Native Skia v3, we delegate all of this work to React Native WebGPU, which handles the platform-specific details and provides a more robust and consistent API for video and camera feeds.
Despite its name, WebGPU is also a native API: there is a standard C header, webgpu.h, and Dawn adds C++ bindings on top of it.
Dawn also extends that header with shared texture memory, a GPU texture backed by a platform buffer.
It is exactly what video and camera feeds need: zero-copy import, proper YUV support, and fencing, so a frame is never sampled while the decoder or the camera is still writing it.
On Android, the opaque YCbCr formats that cameras and decoders produce go through Dawn's own helper, the same one Chromium uses for video frames.
The API is symmetric with the Web: a video frame is imported with device.importExternalTexture(), the same method as in the browser, and behind the scenes it uses the shared texture memory API.
In the other direction, to write into a platform buffer, React Native WebGPU exposes shared texture memory and fences directly.
These are extensions to the browser WebGPU API, giving you full control over native platform buffers and synchronization.
This is what React Native Skia Video uses to encode videos: Skia draws each frame straight into the encoder's buffer, with no readback (see their React Native Skia v3 pull request).
Below are the React Native WebGPU APIs for shared textures:
| API | |
|---|---|
device.importExternalTexture({ source: frame }) | Samples a video or camera frame in a shader, YUV included, as on the web |
device.importSharedTextureMemory({ handle }) | Imports an IOSurface or AHardwareBuffer as shared texture memory |
device.importSharedFence({ type, handle }) | Imports a platform fence (an MTLSharedEvent or a sync file descriptor), for the GPU to wait on |
sharedMemory.createTexture() | Creates a regular GPUTexture that aliases the buffer's pixels |
sharedMemory.beginAccess(texture, initialized, fences) | Starts a GPU access: the GPU first waits on the imported fences |
sharedMemory.endAccess(texture) | Ends the access, and returns new fences that signal once the GPU is done with the buffer |
fence.export() | Exports a fence from endAccess() as an MTLSharedEvent or a sync file descriptor, to hand back to the platform |
The two main modules that had adopted the Native Buffer API, React Native Vision Camera and React Native Skia Video, now both build on React Native WebGPU. And because the camera frames land on the same device as everything else, a machine learning model running on TensorFlow.js can process them right where Skia draws them.
Lottie Animations
Skia has long supported Lottie animations through Skottie, its Lottie player. This has two main benefits. First, a Lottie animation can be composed with any other Skia content and effects. Second, you can customize every property of an animation: colors, text, typography, and images. That works whether the values are hardcoded in the file or exposed through Lottie's official templating system.
With Skia Graphite, that level of customization takes on a completely new meaning. Because an animation is drawn on the same GPU device as everything else, any GPU content can become part of it. Nowadays, many Lottie templates, especially those made for social media, are designed around video assets. With v3, you can drop a video file or a live camera feed straight into them without a round trip through the CPU.
Native-in-Canvas
We want to integrate with any kind of GPU content, whether it is a Skia or Three.js scene, an image, or a video feed. In that spirit, one type of content is of particular interest to us: native views. Snapshots of native views let you process the content of your app on the GPU, for transitions and all sorts of delightful effects.
Here, we followed the same path as video and camera feeds: we removed the direct integration from Skia and moved it down to the WebGPU layer.
React Native Skia used to take view snapshots itself, with makeImageFromView().
In v3, makeImageFromView() is gone: React Native WebGPU's queue.drawElementImageToTexture() takes its place, and Skia is just one of the libraries that draw the result.
queue.drawElementImageToTexture() follows the shape of the WebGPU part of HTML-in-Canvas, Chrome's new API for drawing DOM content on the GPU, except that the element is any mounted React Native view instead of a DOM node.
iOS and Android rasterize views very differently, so this is a best-effort implementation on both platforms.
import { PixelRatio } from "react-native";
// cardRef: a ref to a mounted <View collapsable={false} />
const scale = PixelRatio.get();
const texture = device.createTexture({
size: [Math.round(320 * scale), Math.round(200 * scale)],
format: "rgba8unorm",
usage:
GPUTextureUsage.COPY_DST |
GPUTextureUsage.TEXTURE_BINDING |
GPUTextureUsage.RENDER_ATTACHMENT,
});
await device.queue.drawElementImageToTexture({ source: cardRef }, { texture });
// Work submitted from here on samples the view's pixels.The call returns a promise: a view can only be rasterized on the platform UI thread, so the call hops there and back.
With the RENDER_ATTACHMENT usage, the view is drawn into the texture directly on the GPU, on iOS and on Android 13 and above.
On older Android versions, it falls back to the CPU, with lower fidelity.
In the examples below, each screen is a regular React Native view, captured in a texture and distorted on the GPU. On the left, the tasks disintegrate as they are checked off; on the right, the form ripples wherever you tap, drag, or type.
Happy Hacking
Our goal with React Native Skia and React Native WebGPU is a unified GPU runtime: best-in-class 2D graphics with Skia, best-in-class 3D graphics with Three.js, and machine learning models with libraries like Transformers.js, all together on a single device, just like on the web.
But beyond the symmetry with the web, we also want to give you full control over the native platform capabilities. This is what shared texture memory and fences are about: using the browser API where it fits and providing direct access to the device's native capabilities where you need it.
A big Thank You to the community: the support for the 3.0 release has been amazing. A rich ecosystem has grown around React Native Skia: developers fixing bugs and contributing features, developers building libraries on top of it, like React Native Skia Video, and developers building incredible user experiences with it, like the apps above. And thank you to Google: Dawn and Skia are two remarkable open-source projects, and their stewardship is outstanding.
Please get in touch if you have any questions or comments. And until next time: happy hacking!