Flutter Voice AI: Real-Time Speech Feedback

Adding voice to a mobile app sounds like a solved problem until you ship one. A recognizer that works beautifully in a quiet-room demo turns flaky on a crowded bus, partial transcripts flicker in and out of existence, and users end up repeating themselves until they give up. I learned this building Fonetic AI, an app that helps people learn English by listening to their pronunciation, running real-time conversational practice, and giving actionable feedback on their mistakes. The frontend is Flutter and Dart, the speech work goes through AI speech APIs with Firebase on the backend, and the app ships more than 100 interactive lessons where every spoken answer is scored in real time. This article distills the architecture that made real-time speech feedback actually work in production: a partial-results pipeline built on the speech_to_text package, word-level scoring against an expected phrase, an explicit session state machine, and the permission and performance pitfalls that will waste your week if you skip them.

How a real-time pronunciation loop works

A pronunciation exercise is a tight loop: show a prompt phrase, capture the user's speech, compare what was recognized against what was expected, and give feedback while the moment is still fresh. The single biggest architectural decision is this: feedback must come from partial results, not from waiting for the recognizer's final verdict. When you call listen() with partialResults: true, the recognizer streams hypotheses to your onResult callback several times per second. Each hypothesis improves on the last. The final result is only delivered after the recognizer decides the user has stopped speaking, which adds one to three seconds of dead air. For a language learner that pause kills the experience; the feedback needs to land while they are still in speaking mode. The production loop looks like this:

1. Show the target phrase and start listening with partial results enabled.
2. On each partial hypothesis, highlight the words that already match the target.
3. On the final result, compute the score and show word-level corrections.
4. Never block the UI waiting for the final result.

Set up speech_to_text correctly

The speech_to_text package is the community standard and wraps the platform recognizers on Android and iOS. Add it to your dependencies, then handle platform permissions before anything else. On Android, declare the microphone permission in AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

On iOS, add both usage descriptions to Info.plist. The app will crash at runtime without them:

<key>NSMicrophoneUsageDescription</key>
<string>We need microphone access to hear your pronunciation.</string>
<key>NSSpeechRecognitionUsageDescription</key>
<string>We need speech recognition to give you pronunciation feedback.</string>

Initialize the recognizer once, ideally at app or feature startup, and prefer on-device recognition when the model is available — it removes network latency from the feedback loop:

import 'package:speech_to_text/speech_to_text.dart';

final speech = SpeechToText();

Future<bool> initSpeech() async {
  return speech.initialize(
    onStatus: (status) => debugPrint('status: $status'),
    onError: (error) => debugPrint('error: ${error.errorMsg}'),
    options: [SpeechToText.onDevice],
  );
}

Start listening with a generous listenFor window, a short pauseFor so silence ends the session quickly, partial results on, and an explicit locale. Locking the locale to the lesson language (for example en-US) noticeably improves accuracy over auto-detection:

await speech.listen(
  onResult: onSpeechResult,
  listenFor: const Duration(seconds: 30),
  pauseFor: const Duration(seconds: 3),
  partialResults: true,
  localeId: 'en-US',
);

Partial results are the live feedback channel

The onResult callback fires for both partial and final hypotheses. Check result.finalResult to tell them apart:

void onSpeechResult(SpeechRecognitionResult result) {
  if (result.finalResult) {
    scoreUtterance(result.recognizedWords);
  } else {
    highlightMatchingWords(result.recognizedWords);
  }
}

One performance rule matters more than any other here: partial results arrive many times per second, so never run setState on the whole screen for each one. Drive the live transcript through a ValueNotifier and rebuild only the transcript widget:

final transcript = ValueNotifier<String>('');

void highlightMatchingWords(String words) {
  transcript.value = words;
}

// In the widget tree:
ValueListenableBuilder<String>(
  valueListenable: transcript,
  builder: (context, value, _) => Text(value),
)

This keeps the animation jank at zero even on mid-range devices, a lesson I also applied when profiling frame budgets for another on-device AI feature. If you want the full breakdown, see my guide on Flutter on-device AI with TensorFlow Lite.

Score pronunciation against the expected phrase

For scripted lessons you already know the target phrase, which makes scoring a word-alignment problem rather than a machine-learning one. The standard metric is word error rate: the number of word-level insertions, deletions, and substitutions divided by the number of words in the target phrase. It is exactly the Levenshtein distance computed over word lists:

import 'dart:math';

double wordErrorRate(List<String> expected, List<String> actual) {
  final m = expected.length;
  final n = actual.length;
  final dp = List.generate(m + 1, (_) => List.filled(n + 1, 0));
  for (var i = 0; i <= m; i++) dp[i][0] = i;
  for (var j = 0; j <= n; j++) dp[0][j] = j;
  for (var i = 1; i <= m; i++) {
    for (var j = 1; j <= n; j++) {
      final cost = expected[i - 1] == actual[j - 1] ? 0 : 1;
      dp[i][j] = [
        dp[i - 1][j] + 1,
        dp[i][j - 1] + 1,
        dp[i - 1][j - 1] + cost,
      ].reduce(min);
    }
  }
  return m == 0 ? 0 : dp[m][n] / m;
}

A word error rate of 0 is a perfect repetition; anything under about 0.2 feels like a clean pass for a learner. The same alignment also tells you which words were wrong, which is what drives the actionable feedback in Fonetic AI: the mismatched words get highlighted in the target phrase so the learner sees exactly where their pronunciation drifted, instead of getting a single vague score. Normalize both sides before comparing — lowercase, strip punctuation — or a trailing comma will count as a pronunciation error:

List<String> normalize(String phrase) => phrase
    .toLowerCase()
    .replaceAll(RegExp(r'[^a-z ]'), '')
    .split(' ')
    .where((w) => w.isNotEmpty)
    .toList();

// The same alignment identifies the mispronounced words: any position
// where the recognized word differs from the expected word is flagged.
List<String> misalignedWords(String target, String recognized) {
  final expected = normalize(target);
  final actual = normalize(recognized);
  final aligned = <String>[];
  for (var i = 0; i < expected.length && i < actual.length; i++) {
    if (expected[i] != actual[i]) aligned.add(expected[i]);
  }
  if (actual.length < expected.length) {
    aligned.addAll(expected.sublist(actual.length));
  }
  return aligned;
}

Model the voice session as a state machine

A voice session has more states than a boolean can express: idle, requesting permission, listening, scoring, scored, and error. I model AI request flows as explicit state machines everywhere — it is the same pattern I use for text-based AI calls, described in my AI API integration guide. A ChangeNotifier holding an enum keeps the UI honest:

enum VoiceSessionState { idle, listening, scoring, scored, error }

class VoiceSession extends ChangeNotifier {
  VoiceSessionState _state = VoiceSessionState.idle;
  VoiceSessionState get state => _state;

  List<String> _wrongWords = const [];
  List<String> get wrongWords => _wrongWords;
  bool _passed = false;
  bool get passed => _passed;

  Future<void> start() async {
    _state = VoiceSessionState.listening;
    notifyListeners();
    await speech.listen(
      onResult: onSpeechResult,
      listenFor: const Duration(seconds: 30),
      pauseFor: const Duration(seconds: 3),
      partialResults: true,
      localeId: 'en-US',
    );
  }

  void onSpeechResult(SpeechRecognitionResult result) {
    if (!result.finalResult) {
      transcript.value = result.recognizedWords;
      return;
    }
    _state = VoiceSessionState.scoring;
    notifyListeners();
    final score = wordErrorRate(
      normalize(targetPhrase),
      normalize(result.recognizedWords),
    );
    _wrongWords = misalignedWords(targetPhrase, result.recognizedWords);
    _passed = score < 0.2;
    _state = VoiceSessionState.scored;
    notifyListeners();
  }
}

Note that scored does not mean "passed": the score and the wrong-word list travel together, and the UI decides whether to celebrate or coach. The state machine's job is only to guarantee the UI never shows listening controls and a score at the same time.

The permission and availability pitfalls

Voice features have a long tail of platform edge cases. Handle all five before release:

1. Permission denied on first ask. Use the permission_handler package to request Permission.microphone explicitly, and show a settings-deep-link screen on permanent denial. Do not start the recognizer optimistically; initialize() will fail silently and leave the user staring at a dead button.
2. On-device model missing. SpeechToText.onDevice only works if the platform has downloaded the speech model. Check hasPermission after initialize and fall back to a cloud recognition path (through your own backend, never with keys in the app) when it is unavailable.
3. Wrong or missing locale. If the recognizer has no model for your localeId, results degrade quietly. Fetch locales() once and pick the closest supported locale to your lesson language.
4. Recognizer busy. A previous listen() that was never stopped or cancelled will make the next call a no-op. Always stop() or cancel() in dispose().
5. Background audio conflicts. If your app also plays audio — Fonetic AI has offline playback for lesson audio — request the right audio focus behavior so the recognizer does not fight your own playback session. My offline audio playback guide covers the session side in detail.

Keep recognition off the critical path

Scoring is cheap — a word-level edit distance over a sentence takes microseconds — but the plumbing around it is where jank hides. Three rules held up in production:

• Initialize the SpeechToText instance once and reuse it. Repeated initialization re-binds the platform channel and can drop the first partials.
• Never await network calls between the final result and the score. The feedback the learner needs first — the highlighted wrong words — is fully computable on device.
• Debounce the partial highlight. Updating the ValueNotifier on every hypothesis is fine, but if you run any heavier work per hypothesis (like re-running alignment), throttle it to roughly 4 updates per second with a simple timestamp check.

Wrap feedback in a gamified loop

Fonetic AI's contributions list calls out a gamified learning experience, and it exists for an engineering reason, not just a product one: pronunciation practice is high-friction, and the loop needs a reward to survive repetition. The architecture makes this cheap. The score from wordErrorRate maps directly to points, the wrong-word list maps to retry prompts, and lesson completion (the app ships over 100 interactive lessons) maps to streaks. Store the per-lesson best score in Firestore keyed by the user's UID so progress survives reinstalls — the Firebase side follows the standard patterns from my Firestore and authentication guides. A subtle detail: celebrate attempts, not just passes. A learner who scores 0.35 on their first try and 0.28 on the second is improving, and the UI should say so. The state machine already carries the score; comparing it against the stored best before showing feedback turns a dry evaluation into a reason to try again.

Shipping checklist

Before you call the voice feature done, verify these eight items on a real mid-range device, not just the emulator:

• Microphone permission granted, denied, and permanently-denied flows each show the right screen.
• Partial results highlight matching words live, without jank, on a low-end Android phone.
• Word error rate scoring matches your hand-computed expectations on ten sample utterances.
• Switching lessons stops the previous session cleanly; no recognizer-busy dead ends.
• Airplane mode with no on-device model shows the graceful fallback, not a hang.
• Backgrounding the app mid-lesson cancels recognition instead of leaking it.
• Scores persist across app restarts via Firestore.
• The final feedback screen appears within a beat of the final result, with wrong words highlighted. Real-time voice in Flutter is a solved infrastructure problem and an unsolved experience problem. The recognizer gives you the words; everything your users will remember — the live highlighting, the exact wrong word, the encouragement to retry — is the pipeline you build around it. That pipeline is where Fonetic AI won its production users, and it is the part no package gives you for free. Get in touch and let's talk about your project.