How to Add On-Device Image Description with ML Kit

Users expect smart photo features that work offline and keep data private. ML Kit’s GenAI Image Description API, powered by Gemini Nano, generates short captions for images entirely on-device.

This guide walks through setup, status checks, a basic call, and production tips. It builds on the same AICore foundation used by the Prompt API.

What the Image Description API provides

The API returns a concise text description of a given image. Typical uses include accessibility alt text, auto-tagging photos in a gallery, or generating draft captions before upload.

Gemini Nano runs inside Android’s AICore system service. The model stays local, so inference works without a network connection and user photos never leave the device. Supported devices include recent Pixel and Galaxy flagships that ship with nano-v3 or nano-v4.

Limitations match other GenAI APIs: input size is constrained, output is short, and AICore applies a per-app quota. Check the official overview for the current list of supported devices.

Add the dependency

In your app-level build.gradle.kts add:

implementation("com.google.mlkit:genai-prompt:1.0.0-beta4")

The Image Description path uses the same Prompt API client under the hood. Set minSdk to 26 or higher. If you also need structured output, add the schema compiler and KSP as documented separately.

Sync the project. No extra Google Cloud project or API key is required for the on-device path.

Check Gemini Nano availability

Before showing any caption UI, verify the feature status. Create a client and call checkStatus().

val generativeModel = Generation.getClient()
val status = generativeModel.checkStatus()

when (status) {
    FeatureStatus.AVAILABLE -> {
        // Ready to generate
    }
    FeatureStatus.DOWNLOADABLE -> {
        // Trigger download and collect progress
        generativeModel.download().collect { downloadStatus ->
            // Handle DownloadStarted, DownloadProgress, DownloadCompleted, DownloadFailed
        }
    }
    FeatureStatus.DOWNLOADING -> {
        // Already in progress
    }
    FeatureStatus.UNAVAILABLE -> {
        // Device not supported or configuration not yet fetched
    }
}

Handle the common AICore setup errors listed in the Prompt API documentation (connection failures right after device setup, preparation errors while configurations download, etc.). Restart the device or wait a few minutes on a network connection if you see feature-not-found messages.

Generate a description

Pass a Bitmap (or multiple) together with an optional text instruction. The Image Description use case works best with a clear prompt that asks for a short caption.

suspend fun describeImage(bitmap: Bitmap): String {
    val response = generativeModel.generateContent(
        generateContentRequest(
            ImagePart(bitmap),
            TextPart("Generate a short, accurate description of this image for accessibility. Keep it under 20 words.")
        ) {
            temperature = 0.2f
            maxOutputTokens = 50
        }
    )
    return response.candidates.firstOrNull()?.text ?: ""
}

You can also stream the result with generateContentStream if you want partial text as it arrives. For a simple caption, the non-streaming call is usually enough.

Warm up the model with warmup() on first launch if you need the lowest possible latency for the initial request.

Tips for production use

• Resize or compress large bitmaps before sending them. Extremely high-resolution images increase latency and memory use.
• Cache captions locally if the same photo is viewed repeatedly.
• Provide a clear fallback UI when status is UNAVAILABLE or the quota is exhausted.
• Combine with the Prompt API for follow-up questions about the same image if the user wants more detail.
• Test on both nano-v3 and nano-v4 devices; quality and speed differ slightly.

AICore enforces an inference quota per app. Monitor usage during development and design your feature so it degrades gracefully.

Watch the official overview

Conclusion

The ML Kit GenAI Image Description API gives you a high-level way to add private, offline captions without managing model downloads yourself. Start with the status check, keep prompts short and specific, and handle the download and quota cases. Once the model is available, a few lines of Kotlin produce usable descriptions for gallery, accessibility, or content-creation features.

For more control over the output format or longer conversations, move to the Prompt API and structured output patterns covered in other posts on this site.

Sources

• Overview of the ML Kit GenAI APIs — Google for Developers
• Get started with Prompt API — Google for Developers
• AI on Android — Android Developers
• Deploy Android on-device AI with ML Kit GenAI and LiteRT-LM — Android Developers (YouTube)