OpenSL ES programming notes

WARNING: OpenSL ES is deprecated. Developers should use the open source Oboe library which is available on GitHub. Oboe is a C++ wrapper that provides an API that closely resembles AAudio. Oboe calls AAudio when AAudio is available, and falls back to OpenSL ES if AAudio is not available.

The notes in this section supplement the OpenSL ES 1.0.1 specification.

Objects and interface initialization

Two aspects of the OpenSL ES programming model that may be unfamiliar to new developers are the distinction between objects and interfaces, and the initialization sequence.

Briefly, an OpenSL ES object is similar to the object concept in programming languages such as Java and C++, except an OpenSL ES object is only visible via its associated interfaces. This includes the initial interface for all objects, called SLObjectItf. There is no handle for an object itself, only a handle to the SLObjectItf interface of the object.

An OpenSL ES object is first created, which returns an SLObjectItf, then realized. This is similar to the common programming pattern of first constructing an object (which should never fail other than for lack of memory or invalid parameters), and then completing initialization (which may fail due to lack of resources). The realize step gives the implementation a logical place to allocate additional resources if needed.

As part of the API to create an object, an application specifies an array of desired interfaces that it plans to acquire later. Note that this array does not automatically acquire the interfaces; it merely indicates a future intention to acquire them. Interfaces are distinguished as implicit or explicit. An explicit interface must be listed in the array if it will be acquired later. An implicit interface need not be listed in the object create array, but there is no harm in listing it there. OpenSL ES has one more kind of interface called dynamic, which does not need to be specified in the object create array and can be added later after the object is created. The Android implementation provides a convenience feature to avoid this complexity, which is described in Dynamic interfaces at object creation.

After the object is created and realized, the application should acquire interfaces for each feature it needs, using GetInterface on the initial SLObjectItf.

Finally, the object is available for use via its interfaces, though note that some objects require further setup. In particular, an audio player with URI data source needs a bit more preparation in order to detect connection errors. See the Audio player prefetch section for details.

After your application is done with the object, you should explicitly destroy it; see the Destroy section below.

Audio player prefetch

For an audio player with URI data source, Object::Realize allocates resources but does not connect to the data source (prepare) or begin pre-fetching data. These occur once the player state is set to either SL_PLAYSTATE_PAUSED or SL_PLAYSTATE_PLAYING.

Some information may still be unknown until relatively late in this sequence. In particular, initially Player::GetDuration returns SL_TIME_UNKNOWN and MuteSolo::GetChannelCount either returns successfully with channel count zero or the error result SL_RESULT_PRECONDITIONS_VIOLATED. These APIs return the proper values once they are known.

Other properties that are initially unknown include the sample rate and actual media content type based on examining the content's header (as opposed to the application-specified MIME type and container type). These are also determined later during prepare/prefetch, but there are no APIs to retrieve them.

The prefetch status interface is useful for detecting when all information is available, or your application can poll periodically. Note that some information, such as the duration of a streaming MP3, may never be known.

The prefetch status interface is also useful for detecting errors. Register a callback and enable at least the SL_PREFETCHEVENT_FILLLEVELCHANGE and SL_PREFETCHEVENT_STATUSCHANGE events. If both of these events are delivered simultaneously, and PrefetchStatus::GetFillLevel reports a zero level, and PrefetchStatus::GetPrefetchStatus reports SL_PREFETCHSTATUS_UNDERFLOW, then this indicates a non-recoverable error in the data source. This includes the inability to connect to the data source because the local filename does not exist or the network URI is invalid.

The next version of OpenSL ES is expected to add more explicit support for handling errors in the data source. However, for future binary compatibility, we intend to continue to support the current method for reporting a non-recoverable error.

In summary, a recommended code sequence is:

1. Engine::CreateAudioPlayer
2. Object:Realize
3. Object::GetInterface for SL_IID_PREFETCHSTATUS
4. PrefetchStatus::SetCallbackEventsMask
5. PrefetchStatus::SetFillUpdatePeriod
6. PrefetchStatus::RegisterCallback
7. Object::GetInterface for SL_IID_PLAY
8. Play::SetPlayState to SL_PLAYSTATE_PAUSED, or SL_PLAYSTATE_PLAYING

Note: Preparation and prefetching occur here; during this time your callback is called with periodic status updates.

Destroy

Be sure to destroy all objects when exiting from your application. Objects should be destroyed in reverse order of their creation, as it is not safe to destroy an object that has any dependent objects. For example, destroy in this order: audio players and recorders, output mix, and then finally the engine.

OpenSL ES does not support automatic garbage collection or reference counting of interfaces. After you call Object::Destroy, all extant interfaces that are derived from the associated object become undefined.

The Android OpenSL ES implementation does not detect the incorrect use of such interfaces. Continuing to use such interfaces after the object is destroyed can cause your application to crash or behave in unpredictable ways.

We recommend that you explicitly set both the primary object interface and all associated interfaces to NULL as part of your object destruction sequence, which prevents the accidental misuse of a stale interface handle.

Stereo panning

When Volume::EnableStereoPosition is used to enable stereo panning of a mono source, there is a 3-dB reduction in total sound power level. This is needed to permit the total sound power level to remain constant as the source is panned from one channel to the other. Therefore, only enable stereo positioning if you need it. For more information see the Wikipedia article on audio panning.

Callbacks and threads

Callback handlers are generally called synchronously when the implementation detects an event. This point is asynchronous with respect to the application, so you should use a non-blocking synchronization mechanism to control access to any variables shared between the application and the callback handler. In the example code, such as for buffer queues, we have either omitted this synchronization or used blocking synchronization in the interest of simplicity. However, proper non- blocking synchronization is critical for any production code.

Callback handlers are called from internal non-application threads that are not attached to the Android runtime, so they are ineligible to use JNI. Because these internal threads are critical to the integrity of the OpenSL ES implementation, a callback handler should also not block or perform excessive work.

If your callback handler needs to use JNI or execute work that is not proportional to the callback, the handler should instead post an event for another thread to process. Examples of acceptable callback workload include rendering and enqueuing the next output buffer (for an AudioPlayer), processing the just-filled input buffer and enqueueing the next empty buffer (for an AudioRecorder), or simple APIs such as most of the Get family. See the Performance section below regarding the workload.

Note that the converse is safe: an Android application thread that has entered JNI is allowed to directly call OpenSL ES APIs, including those that block. However, blocking calls are not recommended from the main thread, as they may result in Application Not Responding (ANR).

The determination regarding the thread that calls a callback handler is largely left up to the implementation. The reason for this flexibility is to permit future optimizations, especially on multi-core devices.

The thread on which the callback handler runs is not guaranteed to have the same identity across different calls. Therefore, do not rely on the pthread_t returned by pthread_self() or the pid_t returned by gettid() to be consistent across calls. For the same reason, do not use the thread local storage (TLS) APIs such as pthread_setspecific() and pthread_getspecific() from a callback.

The implementation guarantees that concurrent callbacks of the same kind, for the same object, does not occur. However, concurrent callbacks of different kinds for the same object are possible on different threads.

Performance

As OpenSL ES is a native C API, non-runtime application threads that call OpenSL ES have no runtime-related overhead such as garbage collection pauses. With one exception described below, there is no additional performance benefit to the use of OpenSL ES other than this. In particular, the use of OpenSL ES does not guarantee enhancements such as lower audio latency and higher scheduling priority over that which the platform generally provides. On the other hand, as the Android platform and specific device implementations continue to evolve, an OpenSL ES application can expect to benefit from any future system performance improvements.

One such evolution is support for reduced audio output latency. The underpinnings for reduced output latency were first included in Android 4.1 (API level 16), and then continued progress occurred in Android 4.2 (API level 17). These improvements are available via OpenSL ES for device implementations that claim feature android.hardware.audio.low_latency. If the device doesn't claim this feature but supports Android 2.3 (API level 9) or later, then you can still use the OpenSL ES APIs but the output latency may be higher. The lower output latency path is used only if the application requests a buffer size and sample rate that are compatible with the device's native output configuration. These parameters are device-specific and should be obtained as described below.

Beginning with Android 4.2 (API level 17), an application can query for the platform native or optimal output sample rate and buffer size for the device's primary output stream. When combined with the feature test just mentioned, an app can now configure itself appropriately for lower latency output on devices that claim support.

For Android 4.2 (API level 17) and earlier, a buffer count of two or more is required for lower latency. Beginning with Android 4.3 (API level 18), a buffer count of one is sufficient for lower latency.

All OpenSL ES interfaces for output effects preclude the lower latency path.

The recommended sequence is as follows:

1. Check for API level 9 or higher to confirm the use of OpenSL ES.
2. Check for the android.hardware.audio.low_latency feature using code such as this:

   Kotlin

   import android.content.pm.PackageManager
   ...
   val pm: PackageManager = context.packageManager
   val claimsFeature: Boolean = pm.hasSystemFeature(PackageManager.FEATURE_AUDIO_LOW_LATENCY)

   Java

   import android.content.pm.PackageManager;
   ...
   PackageManager pm = getContext().getPackageManager();
   boolean claimsFeature = pm.hasSystemFeature(PackageManager.FEATURE_AUDIO_LOW_LATENCY);

3. Check for API level 17 or higher to confirm the use of android.media.AudioManager.getProperty().
4. Get the native or optimal output sample rate and buffer size for this device's primary output stream using code such as this:

   Kotlin

   import android.media.AudioManager
   ...
   val am = getSystemService(Context.AUDIO_SERVICE) as AudioManager
   val sampleRate: String = am.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE)
   val framesPerBuffer: String = am.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER)

   Java

   import android.media.AudioManager;
   ...
   AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
   String sampleRate = am.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE);
   String framesPerBuffer = am.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER);

   Note that sampleRate and framesPerBuffer are strings. First check for null and then convert to int using Integer.parseInt().
5. Now use OpenSL ES to create an AudioPlayer with PCM buffer queue data locator.

Note: You can use the Audio Buffer Size test app to determine the native buffer size and sample rate for OpenSL ES audio applications on your audio device. You can also visit GitHub to view audio-buffer-size samples.

The number of lower latency audio players is limited. If your application requires more than a few audio sources, consider mixing your audio at the application level. Be sure to destroy your audio players when your activity is paused, as they are a global resource shared with other apps.

To avoid audible glitches, the buffer queue callback handler must execute within a small and predictable time window. This typically implies no unbounded blocking on mutexes, conditions, or I/O operations. Instead consider try locks, locks and waits with timeouts, and non-blocking algorithms.

The computation required to render the next buffer (for AudioPlayer) or consume the previous buffer (for AudioRecord) should take approximately the same amount of time for each callback. Avoid algorithms that execute in a non-deterministic amount of time or are bursty in their computations. A callback computation is bursty if the CPU time spent in any given callback is significantly larger than the average. In summary, the ideal is for the CPU execution time of the handler to have variance near zero, and for the handler to not block for unbounded times.

Lower latency audio is possible for these outputs only:

• On-device speakers.
• Wired headphones.
• Wired headsets.
• Line out.
• USB digital audio.

On some devices, speaker latency is higher than other paths due to digital signal processing for speaker correction and protection.

As of Android 5.0 (API Level 21), lower latency audio input is supported on select devices. To take advantage of this feature, first confirm that lower latency output is available as described above. The capability for lower latency output is a prerequisite for the lower latency input feature. Then, create an AudioRecorder with the same sample rate and buffer size as would be used for output. OpenSL ES interfaces for input effects preclude the lower latency path. The record preset SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION must be used for lower latency; this preset disables device-specific digital signal processing that may add latency to the input path. For more information on record presets, see the Android configuration interface section above.

For simultaneous input and output, separate buffer queue completion handlers are used for each side. There is no guarantee of the relative order of these callbacks, or the synchronization of the audio clocks, even when both sides use the same sample rate. Your application should buffer the data with proper buffer synchronization.

One consequence of potentially independent audio clocks is the need for asynchronous sample rate conversion. A simple (though not ideal for audio quality) technique for asynchronous sample rate conversion is to duplicate or drop samples as needed near a zero-crossing point. More sophisticated conversions are possible.

Performance modes

Starting with Android 7.1 (API Level 25) OpenSL ES introduced a way to specify a performance mode for the audio path. The options are:

• SL_ANDROID_PERFORMANCE_NONE: No specific performance requirement. Allows hardware and software effects.
• SL_ANDROID_PERFORMANCE_LATENCY: Priority is given to latency. No hardware or software effects. This is the default mode.
• SL_ANDROID_PERFORMANCE_LATENCY_EFFECTS: Priority is given to latency while still allowing hardware and software effects.
• SL_ANDROID_PERFORMANCE_POWER_SAVING: Priority given to conserving power. Allows hardware and software effects.

Note: If you do not require a low latency path and wish to take advantage of the device's built-in audio effects (for example to improve acoustic quality for video playback), you must explicitly set the performance mode to SL_ANDROID_PERFORMANCE_NONE.

To set the performance mode, you must call SetConfiguration using the Android configuration interface, as shown below:

  // Obtain the Android configuration interface using a previously configured SLObjectItf.
  SLAndroidConfigurationItf configItf = nullptr;
  (*objItf)->GetInterface(objItf, SL_IID_ANDROIDCONFIGURATION, &configItf);

  // Set the performance mode.
  SLuint32 performanceMode = SL_ANDROID_PERFORMANCE_NONE;
    result = (*configItf)->SetConfiguration(configItf, SL_ANDROID_KEY_PERFORMANCE_MODE,
                                                     &performanceMode, sizeof(performanceMode));

Security and permissions

As far as who can do what, security in Android is done at the process level. Java programming language code cannot do anything more than native code, nor can native code do anything more than Java programming language code. The only differences between them are the available APIs.

Applications using OpenSL ES must request the permissions that they would need for similar non-native APIs. For example, if your application records audio, then it needs the android.permission.RECORD_AUDIO permission. Applications that use audio effects need android.permission.MODIFY_AUDIO_SETTINGS. Applications that play network URI resources need android.permission.NETWORK. For more information see Working with System Permissions.

Depending on the platform version and implementation, media content parsers and software codecs may run within the context of the Android application that calls OpenSL ES (hardware codecs are abstracted but are device-dependent). Malformed content designed to exploit parser and codec vulnerabilities is a known attack vector. We recommend that you play media only from trustworthy sources or that you partition your application such that code that handles media from untrustworthy sources runs in a relatively sandboxed environment. For example, you could process media from untrustworthy sources in a separate process. Though both processes would still run under the same UID, this separation does make an attack more difficult.