ClawBench vs Talkie

ClawBench is a browser agent evaluation framework built around 153 real-world tasks running on 144 live production websites — not simulated environments or curated sandboxes. Tasks span e-commerce, travel booking, SaaS dashboards, government portals, and developer tools. A built-in request interceptor blocks genuinely irreversible actions (payments, form submissions that send data) so evaluations can run safely on real sites. The benchmark records five layers of data per run: session replays, screenshots at each decision point, raw HTTP traffic, agent reasoning traces, and browser action sequences. This makes failure analysis tractable — you can see exactly which DOM element the agent misidentified, not just a final score. The dataset is open and the evaluation harness is reproducible. The headline finding is sobering: Claude Sonnet 4.6, the best performer, completes only 33.3% of tasks. GLM-5 is second at 24.2%. No model exceeds 50% on any individual task category. The implication is stark — current browser agents are far from autonomous on the open web, and the gap between benchmark performance and production performance is still enormous.

Talkie is a 13-billion-parameter language model with an unusual constraint: it was trained exclusively on text written before 1931. That means no internet, no Wikipedia, no modern code — just 260 billion tokens of books, newspapers, journals, patents, and case law from the pre-modern era. The result is a "vintage" LLM that speaks like it's from the early 20th century and has zero knowledge of anything after its cutoff. The model was built by Nick Levine, David Duvenaud, and Alec Radford (yes, one of the original GPT authors) with support from Anthropic and Coefficient Giving. The scientific motivation is rigorous: Talkie enables researchers to cleanly test how models generalize to unfamiliar tasks from examples alone (since it's never seen Python), study future prediction capabilities without data leakage, and understand how training data diversity shapes model dispositions and values. An instruction-tuned version exists, trained on synthetic data derived from historical etiquette manuals and cookbooks, enabling actual conversation. The model is available free on Hugging Face with a live chat demo on their site. A larger variant is planned for summer 2026.