Skip to content
Keshav Raj
A phone camera scanning an object with a fast capture-to-decision flow and a lens sharpening a blurred page.
AI DISTRICT108 MS
SHIPPED Capture-to-decision in ~108 ms; OCR 86% → 96%.I OWNED Sole authorOUTCOME 108 MS

On-device vision advisor

Keshav RajSole author · 2026 · PrototypeContact meResume PDF

Context

This is a personal prototype I wrote as the sole author: an on-device game companion for macOS that reads the screen and suggests a decision. It is built and measured on my own labelled sets.

The problem

A useful advisor has to answer fast enough to be used while playing, and read the screen correctly. Cloud calls add delay, and a single OCR frame is often wrong.

What I did

  1. I captured the screen with ScreenCaptureKit and read text with Apple Vision OCR, all on the device.

  2. I wrote the decision step in Python with OpenCV helpers for the image work.

  3. I added multi-frame voting, where several consecutive frames vote on each reading, which lifts OCR accuracy.

  4. I built labelled test sets of screen states and measured latency and accuracy against them, with 114 tests in total. I finished phases 0 to 3.

Key decision

I chose to keep everything on the device and measure it, rather than call a cloud model. The trade-off is that I own the OCR accuracy problem myself, which is why I added voting.

I also treated an unsafe wrong answer as worse than a missed one, and counted unsafe outputs separately.

Result

Capture to decision takes about 108 ms (153 ms at the 99th percentile). Multi-frame voting raises OCR accuracy from about 86% to 96%. On my labelled set, 1,097 of 1,098 screen states were correct, with none unsafe.

These are my own benchmarks, not an independent evaluation.

What I learned

I would again define the labelled set and the latency budget before writing the pipeline. Next I would test on more varied screens, since a set I built myself can be easier than real play.

Stack

  • Swift
  • Python
  • ScreenCaptureKit
  • Apple Vision
  • OpenCV
Play the world →