Verify what minds can actually do — silicon and carbon, under one instrument.
A local-first scientific instrument that runs blind capability tests on AI models — and on you — with machine-verified ground truth and evidence you can audit. No account. No cloud requirement. Your hardware, your data, your verdicts.
A human writes a sentence. The sentence becomes a theorem. The theorem is machine-checked against the model's answer — not a vibe, not a benchmark score someone else published. This is how humans tell computers what must be true, and then hold every mind — silicon and carbon — to it.
Every test in the battery carries its formal spec beside the prompt. When a model fails, we check the instrument first — the same rule applied to the human calibration arm. Theorems are introduced, not asserted.
Every week a new model claims to reason. The claims arrive wrapped in benchmarks the model may have memorized, scored by graders nobody audits, summarized in press releases nobody can reproduce. That isn't measurement — it's marketing with numbers on it.
Calibration Scope takes the opposite bet: a verdict is only worth what its evidence chain can prove. Every test has formal ground truth checked by a complete decision procedure — truth tables and finite model search, 2,400 years of logic, machine-checked. Every run is sealed with SHA-3 the moment it completes. Every stimulus is pinned by hash so nobody — including us — can quietly swap the question after seeing the answer.
A real measurement beats a confident reading. That is the Verification Principle.
Formal logic under paraphrase and adversarial traps — does the verdict track the structure, or the wording? Modus ponens to quantifier swaps, every answer machine-verified.
Ground-truth visual questions with pinned images. Reads what's there — or hallucinates what it expects? We caught frontier models doing the second. It's in the data.
Structured tool calls that must parse, and injection attacks that must be refused. Pass both or the machine isn't ready for real work.
The same battery, taken by you. Signal score: did you get the logic right? Carrier variance: did the wording change your verdict? Carbon and silicon, same scale.
✓ Every verdict sealed: SHA-3 provenance · pinned stimuli · reproducible on your machine
Running this instrument against dozens of local and frontier models produced findings we didn't expect — and some we did:
The carrier changes the verdict. Give a small model identical logic wrapped in different prose, and the verdict tracks the wrapper, not the content. Formal notation — the "most rigorous" encoding — turned out to be the heaviest noise. Poetry was the gentlest. Big models shrug all of it off; small ones don't. The wrapping paper matters until the mind is strong enough that it doesn't.
Local models can match the frontier. On sealed reasoning batteries, the best models that run offline on consumer hardware match the flagship cloud models — at zero dollars per query. The gap is real at the bottom and gone at the top.
The instrument fails more often than the subject. A wrong Unicode comparison, a token budget too small, a grader that punished honest refusals — most "model failures" we chased turned out to be our failures. So we started keeping receipts on ourselves too. That's what the Lessons page is: the failure runs of the humans who built this, documented with the same discipline we demand of the machines.
The instrument is a single binary with a PostgreSQL database, running entirely on your machine at 127.0.0.1:8768 (direct, loopback-only — local.calibrationscope.com resolves there too on machines with the DNS record). It drives your local models directly, keeps every trial in a database you own and can open, and never phones home. This website is documentation and findings — the science happens on your hardware, where you can check it.