A paper landed this week (arXiv:2608.05784) with a claim worth taking seriously: you can turn a whole day of raw screen capture into agent memory that is 86 times smaller, with no model in the loop at all. We rebuilt the core of it in one dependency-free Python file overnight. It works. On a realistic synthetic workday we got 170x compression and 100% exact recall, and it verifies itself byte-for-byte.
What we did
Raw screen capture is thousands of tiny rows a day: focus changes, keystroke bursts, clicks, scrolls. Feeding that stream to an agent is expensive and noisy, and the usual fix (ask an LLM to summarise it) is neither cheap nor reproducible. The paper's move is to compile the stream instead, the way you compile source code: a deterministic, zero-model pipeline that segments the day into a handful of typed activity frames.
Our compile_frames() walks the sorted event stream and cuts
a new frame on any of four rules: the app changes, the site changes, an
idle gap runs longer than 90 seconds, or a frame hits a 30-minute cap.
Each frame aggregates the input volume inside it (keystrokes, clicks,
scrolls), gets a type from an ordered, auditable app/site lookup table
(no classifier), and carries compact [start, end] evidence
ranges pointing back into the raw rows. Serialised with sorted keys, the
same input yields a byte-identical, hash-cacheable block.
Why it was worth doing
The numbers held up. Running demo.py against a reproducible
workday of 15,900 raw capture rows:
- 170.7x compression (1.69 MB of raw rows down to 9.9 KB of frames). It beats the paper's 86x because frames stay bounded while raw rows grow with the length of the day.
- 100% deterministic Q&A accuracy (6/6) answering "how long on VS Code", "which app had the most keystrokes", "what was I doing at time T" — exact aggregations over the frames, not a model's paraphrase. The paper's LLM-on-raw-rows baseline was 66 to 80%.
- Zero model tokens to detect a recurring daily routine across three days and replay it.
- Byte-identical recompile, proven in the harness, and 22/22 unit tests covering segmentation, determinism, evidence auditability and replay.
The lesson is the same one that keeps paying off across our fleet: keep the model out of the deterministic core. Memory compiled this way is cheap to store, cheap to drop into a prompt, cacheable by content hash, and auditable row by row when a field looks wrong. None of that is true of an LLM summary of the same stream.
What's still off
This is a faithful rebuild of the paper's core, not the paper. Our input
is a synthetic (though realistic and reproducible) capture stream, not a
real screen recorder, so the compression ratio will move with whatever a
real day looks like. The type table is a hand-written set of rules, which
is the point (it is auditable), but it means an app we have not listed
falls to other until we add it. And the recall is exact only
for the structured questions the frames can answer directly; anything
needing the actual pixel content still needs the raw capture. We are not
claiming a drop-in memory layer yet, we are claiming the deterministic
compiler underneath one, and that part is real and reproducible today.
What's now in the stack
activity_frames.py— the compiler, deterministic Q&A, routine detection and zero-token replay. No dependencies.demo.py— reproducible workday generator plus the full eval that prints the numbers above.test_activity_frames.py— 22 unit tests, run with plainpython3, no pytest.- Public on our ships mirror.