Google’s ML Kit provides on-device machine learning Vision APIs for detecting faces, scanning barcodes, labeling images, and more. ML Kit Analyzer makes it easier to integrate ML Kit with your CameraX app.
ML Kit Analyzer is an implementation of the ImageAnalysis.Analyzer
interface. It overrides the default target resolution
(if needed) to optimize for ML Kit usage, handles the coordinate transformations,
and passes the frames to ML Kit, which returns the aggregated analysis results.
Implement ML Kit Analyzer
To implement ML Kit Analyzer, we recommend using the CameraController
class, which works with PreviewView
to display UI elements. When implemented using CameraController
, ML Kit Analyzer
handles the coordinate transformations between the original ImageAnalysis
stream and PreviewView
for you. It receives the target coordinate system from
CameraX, calculates the coordinate transformation,
and forwards it to ML Kit’s Detector
class for analysis.
To use ML Kit Analyzer with CameraController
, call setImageAnalysisAnalyzer()
and pass it
a new ML Kit Analyzer object with the following in its constructor:
- A list of ML Kit
Detector
s, which CameraX invokes sequentially in order. The target coordinate system that determines the coordinates of the ML Kit output:
COORDINATE_SYSTEM_VIEW_REFERENCED
: the transformedPreviewView
coordinates.COORDINATE_SYSTEM_ORIGINAL
: the originalImageAnalysis
stream coordinates.
An
Executor
that invokes the Consumer callback and delivers theMlKitAnalyzer.Result
, or the aggregated ML Kit result of a camera frame, to the app.A
Consumer
, which CameraX invokes when there is new ML Kit output.
The following code implements ML Kit Analyzer using CameraController
to set up
a BarcodeScanner
to detect QR codes:
Kotlin
// create BarcodeScanner object val options = BarcodeScannerOptions.Builder() .setBarcodeFormats(Barcode.FORMAT_QR_CODE) .build() val barcodeScanner = BarcodeScanning.getClient(options) cameraController.setImageAnalysisAnalyzer( ContextCompat.getMainExecutor(this), MlKitAnalyzer( listOf(barcodeScanner), COORDINATE_SYSTEM_VIEW_REFERENCED, ContextCompat.getMainExecutor(this) ) { result: MlKitAnalyzer.Result? -> // The value of result.getResult(barcodeScanner) can be used directly for drawing UI overlay. } )
Java
// create BarcodeScanner object BarcodeScannerOptions options = new BarcodeScannerOptions.Builder() .setBarcodeFormats(Barcode.FORMAT_QR_CODE) .build(); BarcodeScanner barcodeScanner = BarcodeScanning.getClient(options); cameraController.setImageAnalysisAnalyzer(executor, new MlKitAnalyzer(List.of(barcodeScanner), COORDINATE_SYSTEM_VIEW_REFERENCED, executor, result -> { // The value of result.getResult(barcodeScanner) can be used directly for drawing UI overlay. });
In the code sample above, ML Kit Analyzer passes the following to
BarcodeScanner
’s Detector
class:
- The transformation Matrix
based on
COORDINATE_SYSTEM_VIEW_REFERENCED
that represents the target coordinate system. - The camera frames.
If BarcodeScanner
runs into any issues, then its Detector
throws an error,
and ML Kit Analyzer propagates it to your app. If successful, then ML Kit Analyzer returns MLKitAnalyzer.Result#getValue()
, which
in this case is the Barcode
object.
You can also implement ML Kit Analyzer using the ImageAnalysis
class that is part of camera-core
. However, because ImageAnalysis
is not integrated with PreviewView
,
you must manually handle the coordinate transformations. For more information,
see the ML Kit Analyzer reference documentation.
Additional resources
For a working camera app with ML Kit Analyzer functionality, see the CameraX-MLKit sample.