evalmonkey vs Libretto

evalmonkey is an open-source framework for testing how LLM agents degrade under adversarial conditions. You run your agent against 10 standard datasets (GSM8K, ARC, HellaSwag, etc.) pulled automatically from HuggingFace, then apply chaos profiles that introduce realistic failure modes: malformed JSON schemas, artificial latency spikes, 429 rate-limit errors, context-window overflow, and prompt injection payloads. The key output is a degradation delta — evalmonkey shows you exactly how much your agent's accuracy drops under each failure type versus clean inputs. A model that scores 78% on GSM8K normally but drops to 31% when it gets a 429 mid-chain tells you something crucial about its error-recovery behavior that standard benchmarks completely miss. It supports OpenAI, Anthropic (via Bedrock and direct), Azure, GCP, and any Ollama-hosted model. Corbell-AI published this with a clear thesis: agents break in production for infrastructure reasons, not model reasons — and no existing benchmark tests that. evalmonkey was created today (April 17, 2026) and is still at 3 stars, but the core idea is genuinely novel in the evals space.

Libretto is an open-source browser automation toolkit built by Saffron Health to solve a critical problem with AI-driven web agents: non-determinism. Standard agent-controlled browsers using Playwright or Puppeteer routinely fail 20-30% of the time on production workflows because they rely on LLM judgment for timing and element selection. Libretto replaces that with a record-replay system that captures precise interaction timing and DOM fingerprints, achieving a reported 95% success rate on identical workflows. The library works by recording a "golden path" of a browser session — capturing not just actions but the exact CSS selectors, visual context, and timing windows during which those actions are valid. On replay, it verifies each step against expected page state before proceeding, and falls back to an LLM-assisted recovery mode when pages drift (e.g., after a UI update). Saffron Health built it to maintain integrations with EHR portals that change frequently and where failure has compliance consequences. Saffron open-sourced Libretto after using it internally for 18 months across 40+ healthcare software integrations. The HN thread highlighted the appeal for fintech, legal, and healthcare automation where reliability, not just capability, is the product. The toolkit targets TypeScript/Node.js environments and integrates cleanly with existing Playwright infrastructure.