AI tool comparison
AI-Scientist-v2 vs ClawBench
Which one should you ship with? Here is the side-by-side panel verdict, pricing read, reviewer split, and community vote comparison.
Research & Science
AI-Scientist-v2
Sakana AI's autonomous agent that writes peer-reviewed papers
50%
Panel ship
—
Community
Free
Entry
AI-Scientist-v2 is Sakana AI's second-generation autonomous research system that generates scientific papers end-to-end — from hypothesis formation through experimentation, data analysis, and manuscript writing. It's historically notable for producing the first AI-authored workshop paper accepted through peer review. The v2 system removes reliance on human-authored templates that constrained the original, instead using a progressive agentic tree search guided by an experiment manager agent. This makes it more exploratory across ML domains, though Sakana acknowledges it trades v1's high template success rate for broader generalization with lower per-run success. Costs run roughly $20-25 per full research run using Claude 3.5 Sonnet. The system integrates with Semantic Scholar for literature review and supports OpenAI, Gemini, and Claude via AWS Bedrock. The custom license requires disclosure of AI use in resulting publications — a meaningful ethical constraint for a system that could otherwise flood conferences with AI-generated submissions.
Research
ClawBench
153 real-world browser tasks, live websites — best AI agent scores only 33%
75%
Panel ship
—
Community
Free
Entry
ClawBench is a browser agent evaluation framework built around 153 real-world tasks running on 144 live production websites — not simulated environments or curated sandboxes. Tasks span e-commerce, travel booking, SaaS dashboards, government portals, and developer tools. A built-in request interceptor blocks genuinely irreversible actions (payments, form submissions that send data) so evaluations can run safely on real sites. The benchmark records five layers of data per run: session replays, screenshots at each decision point, raw HTTP traffic, agent reasoning traces, and browser action sequences. This makes failure analysis tractable — you can see exactly which DOM element the agent misidentified, not just a final score. The dataset is open and the evaluation harness is reproducible. The headline finding is sobering: Claude Sonnet 4.6, the best performer, completes only 33.3% of tasks. GLM-5 is second at 24.2%. No model exceeds 50% on any individual task category. The implication is stark — current browser agents are far from autonomous on the open web, and the gap between benchmark performance and production performance is still enormous.
Reviewer scorecard
“For ML research teams, the $20-25 per run cost to get a draft paper with experiments is genuinely interesting as an ideation tool. The tree search approach that explores multiple experimental directions in parallel is the kind of thing that would take a grad student weeks.”
“The five-layer recording (replays, HTTP traffic, reasoning traces) is the right approach for actual debugging — finally a benchmark where failure analysis is tractable. The 33% score also sets honest expectations for teams planning to ship production browser agents right now.”
“Sakana's own documentation says v2 has lower success rates than v1 and is 'more exploratory.' Paying $25 for a failed research run with no guarantee of a usable output isn't a workflow most researchers will adopt. The peer review acceptance was a workshop paper — the lowest bar in academic publishing.”
“Live website testing is a double-edged sword: sites change their DOM, anti-bot measures evolve, and a task that passes today may fail next week with no code change. Benchmark drift on live websites could make ClawBench scores meaningless over 6-month periods without constant maintenance.”
“This is the beginning of AI as a genuine research collaborator, not just a writing assistant. Within five years, AI-generated hypotheses tested by autonomous agents will be standard practice in computational fields. AI-Scientist-v2 is primitive version 0.2 of that future.”
“33% on live websites is actually more impressive than it sounds given the adversarial diversity of the real web. The trajectory from 5% in 2024 to 33% in 2026 means we're likely crossing 60% in 18 months — at which point browser agents start displacing RPA software at scale.”
“Science communication is a craft, and the idea of fully automating it makes me uncomfortable. The best papers are ones where researchers deeply understand and can defend every methodological choice — a system that writes the paper for you undermines that accountability.”
“As someone who uses browser agents for research and competitor monitoring, the failure mode analysis is exactly what I need. Knowing which website categories agents handle well (dev tools) vs. poorly (government portals) helps me route tasks appropriately right now.”
Weekly AI Tool Verdicts
Get the next comparison in your inbox
New AI tools ship daily. We compare them before you waste an afternoon.