Teaching vision
where to click.
We are adapting a fast typed-decision model to recognize controls, locate them on UI and website screenshots, and return action-ready coordinates. Every comparison uses immutable held-out screens from its benchmark version.
Improvement over time.
Each candidate starts from a pinned checkpoint and is evaluated against disjoint seeds. A run is retained even when one metric regresses, so the record shows what actually worked.
Reading this honestly: v1 semantic recognition improved strongly, but its screens were too sparse. Dense v2 made the test about 12× more crowded. Its first candidate learned presence but regressed role and localization, so it was rejected. Exact click success remains zero and no research candidate has replaced the public API model.
Same screen. Better decision.
These are individual held-out examples, not training samples. The image, target box, baseline output, and candidate output are keyed by deterministic seed in the row-level evidence.
SEED 10049
SEED 10054
SEED 10065From recognition to reliable action.
The flat 16-way coordinate head learned semantics faster than geometry. The next loop changes the task, not just the training duration.
Find the coarse screen region containing the described control.
Re-encode a focused crop and resolve a precise continuous point.
Compare the next frame and confirm that the intended UI state changed.
Add browser screenshots with DOM-derived labels, then test on unseen sites.
Synthetic interfaces, browser screenshots, multiple themes, resolutions, zoom levels and control states.
Click success, center error, role accuracy, presence calibration, p50/p95 latency and peak VRAM.
Only candidates that improve held-out action success without hiding regressions or changing the test set.
A result is a directory, not a screenshot.
Model revision 8b318c99… · evaluation format jevision-ui-eval-v2 · seeds 10000–10063 · BF16 · RTX 3090 · one warm model · four typed decisions per screen. Training and evaluation scripts live in the private Jevision mirror with create-only outputs and row-level JSONL predictions.