Context
This is a personal prototype I wrote as the sole author: an on-device game companion for macOS that reads the screen and suggests a decision. It is built and measured on my own labelled sets.
The problem
A useful advisor has to answer fast enough to be used while playing, and read the screen correctly. Cloud calls add delay, and a single OCR frame is often wrong.
What I did
I captured the screen with ScreenCaptureKit and read text with Apple Vision OCR, all on the device.
I wrote the decision step in Python with OpenCV helpers for the image work.
I added multi-frame voting, where several consecutive frames vote on each reading, which lifts OCR accuracy.
I built labelled test sets of screen states and measured latency and accuracy against them, with 114 tests in total. I finished phases 0 to 3.
Key decision
I chose to keep everything on the device and measure it, rather than call a cloud model. The trade-off is that I own the OCR accuracy problem myself, which is why I added voting.
I also treated an unsafe wrong answer as worse than a missed one, and counted unsafe outputs separately.
Result
Capture to decision takes about 108 ms (153 ms at the 99th percentile). Multi-frame voting raises OCR accuracy from about 86% to 96%. On my labelled set, 1,097 of 1,098 screen states were correct, with none unsafe.
These are my own benchmarks, not an independent evaluation.
What I learned
I would again define the labelled set and the latency budget before writing the pipeline. Next I would test on more varied screens, since a set I built myself can be easier than real play.
Stack
- Swift
- Python
- ScreenCaptureKit
- Apple Vision
- OpenCV
