Omnivision
An assistant for blind wearers, built so it never guesses
The evidence layer of an audio-first assistant for blind and low-vision wearers of smart glasses: the system may only assert what it has direct evidence for.
- Test suite
- 365 tests, 0 failures
- whole package
- Fixture evaluation
- 215 examples
- 37/37 commands · 90/90 names · 63 correctly rejected
- False triggers
- 0
- 0 false command triggers · 0 false name extractions · 0 safety violations
- Denylist
- 69 + 23
- hard rejections and ambiguous entries
Figures with a dotted underline carry their source. Hover one to see whether it was verified in code or reported in project documentation.
Context
The situation
Blind and low-vision people navigate social situations without the cues everyone else uses. Existing assistive vision tools describe what is in front of the camera, then forget it. They also present a guess and a certainty in exactly the same tone of voice.
The problem
The failure modes are specific and they are all worse than saying nothing: a face matched to a name and stated as fact; a misread package label becoming a confident “yes, that’s your usual”; the assistant talking over the person the wearer is meeting; narration repeating on a timer until it is tuned out; and a silent fallback to the phone microphone that looks identical to success.
What I built
Make evidence a first-class type rather than a confidence score. Every candidate identity carries an evidence level derived from what was actually observed — a name spoken by the wearer, a corroborating repetition, a human confirmation — and the wording the assistant uses is derived from that level. Assertions and hedges are different sentences, not the same sentence with a probability attached.
For a blind wearer, a confident wrong answer is worse than silence — it gets acted on. Omnivision is an audio-first social-memory assistant for wearers of Ray-Ban Meta glasses, built around a single rule: the system may only assert what it has direct evidence for. A name is asserted only from an exact name token spoken aloud; a face match with no spoken attestation resolves to “likely”, never “known”; conflicting evidence returns “ambiguous” and the resolver refuses to pick. I owned the layer that decides whether the system has earned the right to say something: the command grammar, the name extractor, the evidence ladder and the denylist.
Architecture
How it fits together
- Glasses audioinput
wearer speech and conversation
- Tokenisationdeterministic
ASR-robust across speakers and rates
- Command grammardeterministic
wake word + 9 commands
- Name extractiondeterministic
templates, slots, validation
- Denylistguard
hard and ambiguous rejections
- Evidence ladderguard
may we assert, hedge, or stay silent
- Narrationoutput
assertion or hedge — different wording
Key decisions
The choices that shaped it
- 01
Identity is bound from speech, never from a face
Faces cluster for continuity only — they let the system know it has seen someone before. They never name anyone. A name enters the system only when the wearer naturally echoes it aloud, because that is the only signal that carries the person’s own attestation.
- 02
Evidence levels, not confidence scores
A score invites a threshold, and a threshold turns a weak signal into a strong claim at some arbitrary cut-off. Discrete evidence levels instead determine which sentence the assistant is permitted to say: an unattested face match resolves to “likely”, never “known”, and only a human confirmation promotes it.
- 03
Conflicting evidence returns ambiguous
When signals disagree the resolver refuses to choose. Picking the higher-scoring candidate would be the single most damaging behaviour in the product, because the wearer cannot see that it was a coin flip.
- 04
Foundation-only, so it is testable without hardware
The whole extraction layer avoids Apple frameworks and runs in a plain Swift container on Linux. On a two-day build with one pair of glasses shared across the team, waiting for the device to test string handling would have been fatal. Framework-dependent validation sits behind a protocol seam with a portable implementation.
- 05
The fixture corpus is a gate, not a report
The evaluation harness exits non-zero on any unmet expectation, so it can gate CI directly. Its hard requirement is zero false command triggers and zero false name extractions — in this product a false positive is the failure that matters, not a miss.
- 06
Document what has not been measured
The track’s own documentation carries an explicit table of what was validated on hardware and what was not — the wake-word false-trigger test, the battery and thermal profile, the end-to-end latency budget. Stating the gap is what makes the measured numbers worth anything.
Evidence
What it actually looked like
The room the product was built for. The problem statement on screen is the one the evidence layer exists to answer.
PresentationMeasured
Every figure, with its source
- Test suite
- 365 tests, 0 failures
- whole package
- Fixture evaluation
- 215 examples
- 37/37 commands · 90/90 names · 63 correctly rejected
- False triggers
- 0
- 0 false command triggers · 0 false name extractions · 0 safety violations
- Denylist
- 69 + 23
- hard rejections and ambiguous entries
- Audio path
- 16 kHz
- wideband, against 8 kHz in the vendor documentation — the native rate ASR models are trained on
Figures with a dotted underline carry their source. Hover one to see whether it was verified in code or reported in project documentation.
My contribution
One of six. Owned the speech-understanding and evidence layer.
What I owned
- The wake-word command grammar and its nine-command parser
- Name extraction from wearer-echoed speech: templates, slot resolution and validation
- The evidence ladder — the levels that decide whether the system may assert, hedge or stay silent
- A curated name denylist separating hard rejections from ambiguous cases
- Speech tokenisation and ASR-robustness handling across speakers, rates and accents
- The fixture evaluation harness and its corpus, which exits non-zero on any unmet expectation so it can gate CI
What I did not
- Face clustering, the person store and the identity resolver’s upper levels
- Bluetooth routing, camera capture and the glasses integration
- Narration, earcons, OCR and the shop-assist service
Stack and limits
Built with
- Swift 6
- SwiftUI
- iOS
- Meta Wearables DAT SDK
- Core ML
- Vision
- Swift Package Manager
- XCTest
- Python
- FastAPI
Techniques
- Wake-word command grammar
- Template-based name extraction with slot resolution
- Evidence-level identity resolution
- ASR robustness fixtures
- On-device inference (Core ML, Vision)
What it is not
- I did not write the majority of the repository. Face clustering, narration, OCR and the shop-assist service are teammates’ work.
- The extraction layer is deliberately rule-based rather than learned, which is the right trade for a zero-false-positive requirement but is not a modelling contribution.
- Parts of the hardware validation were still outstanding when my track was documented, and are listed as such in it.
What carried forward
This is the same boundary as the decision work, in a different language and with a human cost attached. There, an unsupported number misleads an operator. Here, an unsupported name misleads someone who cannot check it. Both are solved the same way: make the right to assert something the system has to earn, and encode what earns it.
Next
July 2026
Obligation Ledger
The model proposes a termination date. The service throws it away and recomputes it from verified quotes. Nothing is approvable until every value has been found on its page.
Read case study→Chapter 02
Forkcast
A language model can explain a decision. It cannot be allowed to invent the numbers behind it. Forkcast enforces that in code, and exposes the verifier so anyone can attack it.
Explore case study→