Add spatial video to your app

The Jetpack XR SDK supports the playback of stereoscopic side-by-side video onto flat surfaces. With stereoscopic video, each frame consists of a left-eye and a right-eye image to give viewers a sense of depth.

You can render non-stereoscopic 2D video on Android XR apps with the standard media APIs used for Android development on other form factors.

Play side-by-side video using Jetpack XR SDK

With side-by-side video, each stereoscopic frame is presented as two images arranged horizontally adjacent to each other. Top-and-bottom video frames are arranged vertically adjacent to each other.

Side-by-side video is not a codec but rather a way of organizing stereoscopic frames, which means it can be encoded in any of the codecs supported by Android.

You can load side-by-side video using Media3 Exoplayer and then render it using the new StereoSurfaceEntity. To create a StereoSurfaceEntity, call StereoSurfaceEntity.create, as shown in the following example.

stereoSurfaceEntity = StereoSurfaceEntity.create(
            xrSession,
            StereoSurfaceEntity.StereoMode.SIDE_BY_SIDE,
            // Position 1.5 meters in front of user
            Pose(Vector3(0.0f, 0.0f, -1.5f), Quaternion(0.0f, 0.0f, 0.0f, 1.0f)),
            StereoSurfaceEntity.CanvasShape.Quad(1.0f, 1.0f)
        )
        val videoUri = Uri.Builder()
            .scheme(ContentResolver.SCHEME_ANDROID_RESOURCE)
            .path(R.raw.sbs_test_video.toString())
            .build()
        val mediaItem = MediaItem.fromUri(videoUri)

exoPlayer = ExoPlayer.Builder(this).build()
exoPlayer.setVideoSurface(stereoSurfaceEntity.getSurface())
exoPlayer.setMediaItem(mediaItem)
exoPlayer.prepare()
exoPlayer.play()

Play 180-degree and 360-degree video using the Jetpack XR SDK

Alpha02 and higher

Starting in Version 1.0.0-alpha02, StereoSurfaceEntity supports playback of 180° videos on hemispherical surfaces and 360° videos on spherical surfaces. If the videos are stereoscopic, the files should be in a side-by-side format.

The following code shows how to set up StereoSurfaceEntity for playback on a 180° hemisphere and a 360° sphere. When using these canvas shapes, position the surface by leveraging the user's head pose to provide an immersive experience.

// Set up the surface for playing a 180° video on a hemisphere.
hemisphereStereoSurfaceEntity =
    StereoSurfaceEntity.create(
        xrCoreSession,
        StereoSurfaceEntity.StereoMode.SIDE_BY_SIDE,
        xrCoreSession.spatialUser.head?.transformPoseTo(
            Pose.Identity,
            xrCoreSession.activitySpace
        )!!,
        StereoSurfaceEntity.CanvasShape.Vr180Hemisphere(1.0f),
    )
// ... and use the surface for playing the media.

// Set up the surface for playing a 360° video on a sphere.
sphereStereoSurfaceEntity =
    StereoSurfaceEntity.create(
        xrCoreSession,
        StereoSurfaceEntity.StereoMode.TOP_BOTTOM,
        xrCoreSession.spatialUser.head?.transformPoseTo(
            Pose.Identity,
            xrCoreSession.activitySpace
        )!!,
        StereoSurfaceEntity.CanvasShape.Vr360Sphere(1.0f),
    )
// ... and use the surface for playing the media.