Flutter Receipt Scanner With Gemini API

Sund AI, an app I built, uses computer vision to scan grocery receipts or fridge shelves, tracks expiry dates automatically, and generates recipes to use up ingredients before they spoil. The receipt parsing runs on the Gemini API: one photo of a receipt goes in, and a structured list of items comes out. No OCR SDK, no regex parser for receipt layouts, no training data. That is the whole trick this article teaches. You will build the same flow in Flutter: capture a receipt photo, send it to Gemini with a JSON schema, get back a typed list of items with quantities, prices, and expiry dates, let the user confirm the list, and store it. The same pattern works for any document you need as data, but receipts are the sharpest test: crumpled, faded, and formatted differently by every store.

Why Gemini instead of plain OCR

Traditional OCR, such as ML Kit Text Recognition, returns raw lines of text. That is only half the job: you still have to parse lines like "MILK 2L 3.49" into a name, a quantity, and a price, and every store formats those lines differently. Gemini reads the image the way a person does. It understands the layout, it knows which column is the price, and it can return the result as JSON that matches a schema you define. You skip the parsing layer entirely: the model does the reading and the structuring in one call. There is a trade-off. OCR runs on-device, offline, and free. Gemini needs a connection, adds a second or two of latency, and costs API usage — but the accuracy difference on real receipts is large, and the code you do not write is the parsing code you never have to debug. If your app must work fully offline, use on-device OCR; the rest of this article assumes a connection.

The pipeline at a glance

The flow has six steps, and only one of them talks to Gemini:

1. Capture the receipt photo with the camera.
2. Downscale and compress the image before upload.
3. Send the image to Gemini with a prompt and a response schema.
4. Parse the JSON response into a list of receipt items.
5. Show the list for the user to confirm or correct, then save it.
6. Track expiry dates and suggest recipes from items that expire soon. Step 5 is the one most tutorials skip, and it is the one that matters most in production. Gemini can misread a blurry line or invent an item that is not on the receipt. A confirmation screen turns those mistakes into a tap instead of bad data.

Step 1: capture and compress the photo

Use image_picker for the simplest version, or the camera plugin if you need a custom capture screen with a receipt frame. The camera setup is covered in Flutter Camera: Capture Photos Like a Pro. Whatever you use, compress at capture time. A full-resolution phone photo is several megabytes; Gemini does not need that detail to read a receipt, and a smaller upload is faster and cheaper. image_picker does this in one line:

final XFile? photo = await ImagePicker().pickImage(
  source: ImageSource.camera,
  maxWidth: 1024,
  imageQuality: 80,
);
if (photo == null) return;

A max width of 1024 pixels keeps receipt text readable while cutting the upload to a few hundred kilobytes. Skip the extra image package unless you need rotation or cropping for receipts photographed at an angle.

Step 2: define the data model

The model is small, and a manual fromJson is enough. Keep expiryDate nullable: most receipts do not print expiry dates, so the model estimates one and the user confirms it.

class ReceiptItem {
  const ReceiptItem({
    required this.name,
    required this.quantity,
    this.unit,
    this.price,
    this.expiryDate,
    this.expiryEstimated = false,
  });

  final String name;
  final double quantity;
  final String? unit;
  final double? price;
  final DateTime? expiryDate;
  final bool expiryEstimated;

  factory ReceiptItem.fromJson(Map<String, dynamic> json) {
    return ReceiptItem(
      name: json['name'] as String,
      quantity: (json['quantity'] as num).toDouble(),
      unit: json['unit'] as String?,
      price: (json['price'] as num?)?.toDouble(),
      expiryDate: json['expiryDate'] == null
          ? null
          : DateTime.parse(json['expiryDate'] as String),
      expiryEstimated: json['expiryEstimated'] as bool? ?? false,
    );
  }
}

A receipt rarely prints a best-before date, so the prompt asks Gemini to estimate one from the purchase date and the typical shelf life of the item, and to flag it as estimated. Milk gets days, rice gets months, and the estimate is always shown as one in the UI; the user can tap any date to correct it. Sund AI tracks expiry dates automatically, and this estimate-then-confirm loop is how that stays honest: the app never presents a guess as a fact.

Step 3: the prompt and the response schema

The Gemini API supports structured output: set responseMimeType to application/json and pass a responseSchema, and the model must return JSON matching that schema. That turns a chatty model into a typed API. The schema for a receipt:

{
  "type": "OBJECT",
  "properties": {
    "storeName": { "type": "STRING" },
    "purchaseDate": { "type": "STRING" },
    "items": {
      "type": "ARRAY",
      "items": {
        "type": "OBJECT",
        "properties": {
          "name": { "type": "STRING" },
          "quantity": { "type": "NUMBER" },
          "unit": { "type": "STRING" },
          "price": { "type": "NUMBER" },
          "expiryDate": { "type": "STRING" },
          "expiryEstimated": { "type": "BOOLEAN" }
        },
        "required": ["name", "quantity"]
      }
    }
  },
  "required": ["items"]
}

Note the types are uppercase: OBJECT, STRING, NUMBER, BOOLEAN. That is the REST format; the SDKs use their own enum values, so check the reference for whichever client you use. The prompt itself stays short and strict:

Extract every grocery item from this receipt photo. For each item return
its name, quantity, unit, and price in the receipt currency. Use the
purchase date printed on the receipt. If the receipt prints an expiry or
best-before date for an item, use it and set expiryEstimated to false.
Otherwise estimate the expiry date from the purchase date and the typical
shelf life of the item, and set expiryEstimated to true. Never invent an
item that is not visible on the receipt. If a line is unreadable, skip it.

"Never invent an item" and "if a line is unreadable, skip it" are load-bearing sentences. Without them, the model fills gaps with plausible-looking items. With them, the failure mode is a missing item, which the user spots on the confirmation screen.

Step 4: call the Gemini API

The REST endpoint is a plain HTTP POST: the image goes as base64 in an inlineData part next to the text prompt, with your API key in the x-goog-api-key header:

import 'dart:convert';
import 'dart:io';
import 'package:http/http.dart' as http;

class ReceiptScanner {
  ReceiptScanner({required this.apiKey, http.Client? client})
      : _client = client ?? http.Client();

  final String apiKey;
  final http.Client _client;

  Future<List<ReceiptItem>> scanReceipt(File image) async {
    final bytes = await image.readAsBytes();
    final response = await _client.post(
      Uri.parse(
        'https://generativelanguage.googleapis.com/v1beta/models/'
        'gemini-2.5-flash:generateContent',
      ),
      headers: {
        'x-goog-api-key': apiKey,
        'Content-Type': 'application/json',
      },
      body: jsonEncode({
        'contents': [
          {
            'parts': [
              {'text': _prompt},
              {
                'inlineData': {
                  'mimeType': 'image/jpeg',
                  'data': base64Encode(bytes),
                },
              },
            ],
          },
        ],
        'generationConfig': {
          'responseMimeType': 'application/json',
          'responseSchema': _receiptSchema,
        },
      }),
    );

    if (response.statusCode != 200) {
      throw ReceiptScanException(
        'Gemini returned ${response.statusCode}: ${response.body}',
      );
    }

    final decoded = jsonDecode(response.body) as Map<String, dynamic>;
    final candidates = decoded['candidates'] as List<dynamic>;
    final text = (candidates.first as Map<String, dynamic>)['content']
        ['parts'].first['text'] as String;
    final items = (jsonDecode(text) as Map<String, dynamic>)['items']
        as List<dynamic>;
    return items
        .map((e) => ReceiptItem.fromJson(e as Map<String, dynamic>))
        .toList();
  }
}

gemini-2.5-flash is the fast, cheap model for this job; check the current model list on ai.google.dev before you ship, because model names change. The response text is itself JSON, so it gets decoded twice: once for the API envelope, once for the model output. And the error branch matters: a 400 usually means a bad key or a malformed schema, while a 429 means you hit a rate limit, so surface the two differently instead of showing a generic failure. Keep the API key out of the app. A key shipped in a Flutter binary can be extracted, and a leaked Gemini key can be abused on your quota. For a prototype, keep it in a local config file that never gets committed. In production, route the call through your own backend: Sund AI runs on Firebase, so a Cloud Function that holds the key and forwards the request is the natural shape. The client then sends the photo to your function, and the key never leaves your server. The general API-integration pattern, including secure key handling, is covered in Integrating AI APIs into a Flutter App.

Step 5: confirm before you save

Never save the model's output sight unseen. Show the parsed items in an editable list — name, quantity, expiry date — and let the user fix anything before tapping save. This screen is where misreads get caught: the "2L milk" that was actually "2x milk", the expiry date the model guessed too generously, the item the receipt abbreviated beyond recognition. One confirmation screen removes an entire class of support complaints. Persistence can start simple: encode the confirmed list to JSON and store it locally. Sund AI syncs through Firebase so the inventory follows the user across devices, but local storage is the right first step. Either way, store the purchase date and store name alongside the items.

Step 6: expiry tracking and recipes

Once items are saved, the rest is date math: sort by expiry date, surface what expires in the next three days, and remind the user before it spoils. The reminder itself is a local notification; the scheduling pattern is in Flutter Local Notifications: Full Guide. Recipes are a second, text-only Gemini call. Send the list of soon-to-expire items as plain text and ask for three recipes that use them:

These ingredients expire soon: 2L milk (2 days), tomatoes 500g (1 day),
spinach 1 bunch (1 day). Suggest 3 simple recipes that use as many of
them as possible. Return only the recipe names and a one-line description
each.

Text-only calls are fast and cheap, and this one closes the loop: scan, track, cook, waste nothing. Sund AI generates recipes to use up ingredients before they spoil, and this prompt is the entire feature.

Costs, limits, and honest trade-offs

Four things to plan for before you ship. First, keys: never embed the Gemini key in the app, proxy it through your backend, and rotate it if it ever leaks. Second, image size: downscale before upload, as in step 1, because every megabyte costs latency and tokens. Third, quotas: the Gemini API has rate limits that vary by tier, so handle 429 responses with a clear message and a retry instead of a crash, and check the current limits on ai.google.dev. Fourth, privacy: the receipt photo leaves the device and goes to Google's API, so say so in your privacy policy and do not scan anything the user did not explicitly ask you to scan. Gemini is good, not perfect. Faded thermal print, a receipt folded in half, a photo taken in a dim kitchen — all of these produce misreads. The confirmation screen is your answer, not a bigger model. Design for a 90 percent parse with a one-tap fix, and the feature feels reliable; design for a 100 percent parse with no correction path, and the first bad receipt costs you a user.

The pattern in one paragraph

Photograph the receipt, compress it, send it to Gemini with a strict prompt and a JSON schema, decode the structured result into typed Dart objects, let the user confirm the list, and store it. Expiry dates are estimates unless printed, recipes come from a second text-only call over the items expiring soonest, and the API key lives on your server, never in the app. That is the whole integration — the same shape Sund AI uses to turn a crumpled receipt into a pantry that manages itself. The case study for the app is Sund AI. Get in touch and let's talk about your project.