On-Device or Cloud?

Before building recitation tracking for Suhuf, we had to make a foundational engineering choice, the same one facing anyone building live speech features for Arabic today. Run the model on the device, and you get privacy (audio never leaves the phone), zero inference cost per user, and a feature that works with no signal, at the mercy of a phone CPU's compute budget. Run it in the cloud, and you get datacenter-class latency and the freedom to use much larger models, at the price of per-session GPU cost, a network dependency, and users' recitation audio leaving their device.

Established products in this space, such as Tarteel, run purpose-built models on server-side GPU inference stacks; we wanted to know how far the on-device path could actually go before accepting the costs of the cloud one.

This post is a log of that investigation. It covers: our benchmark of four open Arabic CTC models against both clean and phone-mic recitation audio (dataset included), why we replaced transcribe-then-match with CTC forced alignment on logits, three bugs that only real device sessions could expose, each diagnosed from a trace file, and the latency arithmetic that finally told us where the on-device ceiling is. We're early in this build; what follows is the state of the work, including the parts that failed.

Follow-along in recite mode: green is confirmed, red is a flagged mistake.

The starting point: 39% false mistakes

Recitation tracking marks each word as you say it and flags what you missed. For a memorization tool, a false mistake mark is the worst failure mode: it teaches doubt. When we scored our first engine against real recordings, it revealed 61% of correctly recited words and falsely flagged 39% of correct words as mistakes. That number is what sent us back to first principles.

The first engine did what most ASR-adjacent features do: transcribe, then match. A wav2vec2 CTC model decoded mic audio to Arabic characters, we collapsed transcript and expected verse text to a bare consonantal skeleton, and a Needleman-Wunsch aligner fuzzy-matched the two strings. Every weakness of the decode had to be absorbed downstream, by fifteen-plus hand-tuned constants: a first-lock score of 0.26, word-pass ratios at 0.60 and 0.25, an LCS "rescue" at 0.40, relocation floors at 0.53 and 0.58, stuck-tick counters, silence windows. Each constant was fitted to the noise of one specific model, which meant improving the model invalidated all of them at once.

The threshold tower: individually reasonable, collectively unfalsifiable.

Benchmarking the open Arabic CTC models

The model underneath was a fine-tune of XLSR-53 on Common Voice 6.1 and the Arabic Speech Corpus, conversational Modern Standard Arabic. Quranic recitation is a different acoustic register: vowels held for six counts, nasalized ghunna, melodic contours that no read-sentence corpus contains. Before writing any more engine code, we benchmarked what was actually available: every public Quran-domain CTC checkpoint we could find, plus our incumbent, against 19 clips, clean surah recordings of Al-Fatiha and Al-Mu'minun, and two raw phone-microphone sessions from our own devices. The metric is character error rate on the normalized consonantal skeleton (skelCER), since that is the representation our aligner consumes.

Four open Arabic CTC models on the same 19 clips.

Three findings worth passing on:

1. Domain beats size. The winning model is a Quran-domain fine-tune of our own incumbent, same architecture, same parameter count, 6× lower error. Its model card claims ~4% WER on known reciters and ~6% on unknown ones; our independent measurements (3.2% / 4.8% skelCER) are consistent with those claims.
2. Model cards are not evidence. The 94M model's card reports 7.9% WER, with no named training dataset. On our clips it produced 50-67% error: unusable. Another checkpoint ships with no model card at all. If a card doesn't name its data and eval set, treat its numbers as unverified.
3. Evaluate on your own audio distribution. The most instructive row is the last one: statistically tied with the winner on clean audio (3.7% vs 3.2%), then 3× worse on phone-mic recordings (16.1% vs 4.8%). A benchmark built only from clean recordings would have called these two models equivalent. They are not, and the difference is precisely the audio our users produce.

Replacing string matching with forced alignment

The model swap fixed the ears; the alignment layer was still the wrong design. Recitation tracking has one structural advantage over open transcription that our first engine ignored: the expected text is known. The problem isn't "what did the user say", it's "how well does the audio fit the verse they should be on, and where are they in it."

So the rebuilt engine never collapses model output to a string. It runs CTC forced alignment directly on the frame log-probabilities: a banded Viterbi walk over the expected character sequence, with blank self-loops, an epsilon-skip penalty calibrated so that skipping a word only wins when the audio genuinely doesn't contain it, and per-word posteriors that replace the ratio thresholds entirely.

Tajweed leniency became an alignment prior rather than a string hack: assimilation-prone consonants (idgham, ghunna carriers) may be absent without penalty inside a word that has real acoustic evidence, but a soft consonant can never manufacture a match on its own.

Alignment instead of matching: probabilities against the expected text.

On the offline harness, real recordings scored against ground truth, word-tracking accuracy went from 61% to 100%, and the false-red rate from 39% to zero. We rolled the new engine out to our internal test group.

What the first device sessions found

The simulator had been decoding 30× faster than a phone, and it had hidden everything. Each internal test session records its audio plus a JSON trace of engine events, so every problem below was diagnosed by replaying the exact session through the exact engine code, we never reproduced a bug by hand.

Session one: one update per five seconds. The engine re-decoded its entire rolling audio window, up to 12 seconds, through a 315M-parameter model on every 350ms tick. On a phone CPU each inference took seconds; while it ran, the tick loop starved; the silence detector never fired; the window pinned at its maximum size, making every subsequent decode maximally expensive. A self-sustaining worst case. Shrinking the window to 4 seconds (the aligner carries its cursor across window rolls, so short context costs little) cut per-tick cost ~3× with no accuracy change on the harness.

The debugging had a proper red herring. Replaying the session offline first produced junk, the same 36 characters looping. We suspected int8 quantization collapsing on phone-mic audio. Then the full-precision model decoded the same clip near-perfectly… and so did the int8 export. The garbage came from our own replay harness loading the previous model's weights with the new model's vocabulary table, 51 tokens decoded through a 56-token map. The audio was innocent; the quantization was innocent. Verify the replay tooling before trusting what it says about the model.

Session two: mistakes invented at every verse ending. Testers saw the last words of verses flagged as mistakes they never made. The trace made it obvious: our voice-activity gate discarded any decode arriving more than 400ms after the last detected speech, a guard against trailing noise, written when inference was fast. On a phone, inference takes 500-1000ms, so when a reciter finished a verse and paused to breathe, the decode of their final words always arrived "too late" and was thrown away. The discarded text in the trace wasn't noise, it was the verse endings themselves: «كل شيء سببا», «فيهم حسنا», «دونها سترا». This is the actual trace from that session (decoded is the skeleton the model heard; silenceMs is how late the decode arrived relative to the last detected voice):

"vadGateBlocks": [
  { "tMs":  9884, "silenceMs": 671, "decoded": "كلشيسببا",  "cursorHeld": 43  },
  { "tMs": 32138, "silenceMs": 733, "decoded": "فيهمحسنا",  "cursorHeld": 133 },
  { "tMs": 75988, "silenceMs": 724, "decoded": "ونهاسترا",  "cursorHeld": 291 }
]

Every entry is a verse ending, blocked for arriving ~700ms "late", which is just the inference time. The gate was comparing decode arrival time against last-voice time, a category error once inference latency exceeds the gate width. Forced alignment doesn't need the gate at all: trailing quiet decodes to CTC blanks, and audio that matches nothing cannot advance the Viterbi cursor. We removed it from the alignment path; verse-end false mistakes went to zero.

Three internal sessions, two fixes: median update interval went from ~5s to 1.16s.

Session three: the honest residual. A 3.8-minute session tracking 21 consecutive verses, word by word, with a median UI update every 1.16 seconds, which at recitation pace puts the tracker about three words behind the reciter's voice. One word of that is structural (a word is only confirmed once it has been fully spoken). The rest is arithmetic.

The arithmetic, and the split decision

A 315M-parameter encoder costs roughly 600ms per 4-second window on a recent phone CPU. Real-time tracking that feels live needs updates every ~300-400ms. No engine-level cleverness closes that gap; only the model or the hardware can. From here the on-device path has one remaining lever, a 94M model fine-tuned properly on public per-ayah recitation corpora (our training pipeline for this is built and smoke-tested; it needs a GPU day), which should bring ticks near 350ms. Server-side, the same second of audio costs 30-50ms on a datacenter GPU, which is why the established products in this space run their inference there.

Our conclusion is not either/or. Everything above the model, the forced aligner, the whole-Quran position locator, the reveal and mistake logic, consumes one interface: a stream of CTC logit frames. Nothing in that stack cares where the frames were computed.

Two inference lanes, one alignment engine. The engine can't tell which lane fed it.

So the plan we're now executing: a cloud inference lane that streams PCM up and logit frames back (expected voice-to-highlight latency 200-400ms, network included), while the on-device lane remains fully functional as the offline path, recitation tracking keeps working with no signal, just a few words more patient.

Current state

The engine also no longer assumes your position: open any page, start reciting anywhere, and the opening seconds are spent identifying your location against the entire Quran, a page jump requires two agreeing identification probes before it commits, then tracking picks up mid-verse.

One transferable lesson from this stretch of work: ship the trace recorder before the feature. Every bug in this post was found in a JSON sidecar the engine writes next to each session's audio recording.

Next up: the cloud inference lane, the 94M fine-tune for the offline path, and calibrating mistake grading against deliberately imperfect recitation. Recite mode is in internal testing now, you can follow the app at suhuf.app.