AI tool comparison
AI-Scientist-v2 vs LamBench
Which one should you ship with? Here is the side-by-side panel verdict, pricing read, reviewer split, and community vote comparison.
Research & Science
AI-Scientist-v2
Sakana AI's autonomous agent that writes peer-reviewed papers
50%
Panel ship
—
Community
Free
Entry
AI-Scientist-v2 is Sakana AI's second-generation autonomous research system that generates scientific papers end-to-end — from hypothesis formation through experimentation, data analysis, and manuscript writing. It's historically notable for producing the first AI-authored workshop paper accepted through peer review. The v2 system removes reliance on human-authored templates that constrained the original, instead using a progressive agentic tree search guided by an experiment manager agent. This makes it more exploratory across ML domains, though Sakana acknowledges it trades v1's high template success rate for broader generalization with lower per-run success. Costs run roughly $20-25 per full research run using Claude 3.5 Sonnet. The system integrates with Semantic Scholar for literature review and supports OpenAI, Gemini, and Claude via AWS Bedrock. The custom license requires disclosure of AI use in resulting publications — a meaningful ethical constraint for a system that could otherwise flood conferences with AI-generated submissions.
Research & Benchmarks
LamBench
120 λ-calculus challenges that cut through AI benchmark gaming
50%
Panel ship
—
Community
Free
Entry
LamBench is a benchmark of 120 fresh lambda calculus programming questions designed by Victor Taelin (creator of the HVM runtime) to test genuine AI reasoning capabilities rather than pattern-matched performance on contaminated datasets. Questions range from implementing basic operations like addition for λ-encoded natural numbers to deriving generic folds for arbitrary data types. The benchmark measures both accuracy (percentage of 120 tasks solved correctly) and speed (average solution time). Current top performers include GPT-5.4 at 91.7% accuracy, Anthropic's Opus 4.6 at 90.0%, and GPT-5.3-Codex at 89.2%. Lower-tier models bottom out at 28-58% accuracy — revealing significant gaps in symbolic reasoning capability that other benchmarks obscure. Taelin released LamBench in direct response to community requests for a benchmark resistant to training data contamination. Lambda calculus is a clean, closed formal system — ideal for testing reasoning because memorizing examples provides minimal advantage over actually understanding the abstractions.
Reviewer scorecard
“For ML research teams, the $20-25 per run cost to get a draft paper with experiments is genuinely interesting as an ideation tool. The tree search approach that explores multiple experimental directions in parallel is the kind of thing that would take a grad student weeks.”
“Lambda calculus is a great choice for a hard-to-contaminate benchmark — you can't just memorize your way to success on symbolic reasoning. The gap between top models (90%+) and mid-tier (50-60%) is much larger than most leaderboards show, which gives it real signal.”
“Sakana's own documentation says v2 has lower success rates than v1 and is 'more exploratory.' Paying $25 for a failed research run with no guarantee of a usable output isn't a workflow most researchers will adopt. The peer review acceptance was a workshop paper — the lowest bar in academic publishing.”
“120 questions is a very small sample size for a benchmark claiming to measure fundamental reasoning — statistical noise could easily explain a 5-10% difference between models. And lambda calculus is a narrow domain; strong performance here doesn't generalize to most real tasks.”
“This is the beginning of AI as a genuine research collaborator, not just a writing assistant. Within five years, AI-generated hypotheses tested by autonomous agents will be standard practice in computational fields. AI-Scientist-v2 is primitive version 0.2 of that future.”
“As LLMs saturate mainstream benchmarks, we'll rely increasingly on formal, symbolic tasks to measure genuine reasoning progress. LamBench points toward a class of evaluation that correlates with the kind of compositional thinking needed for real AGI-level capabilities.”
“Science communication is a craft, and the idea of fully automating it makes me uncomfortable. The best papers are ones where researchers deeply understand and can defend every methodological choice — a system that writes the paper for you undermines that accountability.”
“Lambda calculus reasoning benchmarks are fascinating from a research perspective but have zero direct connection to creative workflows. The leaderboard is worth bookmarking to track which models are actually getting smarter vs. just getting better at gaming evals.”
Weekly AI Tool Verdicts
Get the next comparison in your inbox
New AI tools ship daily. We compare them before you waste an afternoon.