Claude 4 API: Tool Use Streaming & Prompt Caching vs Perplexity AI Sonar Pro 2 API

“The primitive here is clean: incremental tool-call deltas over SSE, and a cache-control header you attach to prompt segments to pin them server-side. The DX bet is that complexity lives in the HTTP layer, not in a new SDK abstraction — you opt in per-request, no new mental model required. The moment of truth is calling `stream=true` on a tool-use request and watching partial JSON arguments arrive before the model finishes thinking, which actually matters for agent loops where you want to dispatch work early. This is not a weekend-script replacement — implementing correct incremental JSON parsing for partial tool arguments plus a reliable distributed cache with 2M token capacity is a real engineering problem Anthropic has solved for you. The specific decision that earns the ship: cache invalidation is explicit and cache hits are reflected in the usage object, so you can actually measure what you're saving instead of guessing.”

“The primitive here is clean: a single API endpoint that handles search retrieval, multi-hop resolution, and CoT synthesis without you wiring together a retriever, a reranker, and a reasoning model yourself. The DX bet is that you pay per search rather than manage chunking, embedding pipelines, or freshness invalidation — and that's the right bet for the 80% case. First 10 minutes survive: you swap your OpenAI call, add `search_domain_filter` and `reasoning_mode: true`, get citations back in the response object. My one gripe is that the reasoning trace isn't exposed as a structured field — you get the synthesis but not the hop-by-hop retrieval path, which makes debugging citation quality genuinely annoying. Not a weekend script replacement: building reliable multi-hop web retrieval with deduplication and grounding at this latency profile yourself is a real engineering problem. Ship it, but the opaque reasoning trace is a craft failure that will bite teams doing quality evaluation.”

“Direct competitors are OpenAI's cached completions and Google's context caching in Gemini 1.5 — both shipping for months — so Anthropic is catching up, not leading. The specific scenario where this breaks: cache hit rates depend entirely on prompt structure, and developers who dynamically compose system prompts (inserting user-specific context at the top) will see near-zero cache utilization and pay full price while assuming they're saving money. The prediction: this feature doesn't get killed — it becomes table stakes infrastructure and Anthropic wins by having the largest cache window (2M vs. competitors' current limits). What would have to be true for me to be wrong: OpenAI ships a 10M token cache window before Anthropic's ecosystem matures, commoditizing the advantage. Still a ship because the streaming tool-use delta is genuinely differentiated — no competitor has clean partial-argument streaming for tool calls yet, and that changes agent loop architecture in ways that matter.”

“Category: search-augmented generation API. Direct competitors: Bing Grounding in Azure OpenAI, Google Grounding with Gemini, and — let's be honest — a LangChain retriever pointing at Tavily. The specific scenario where this breaks is any workflow that needs deterministic source selection: when a user needs to restrict retrieval to a known corpus of internal documents plus live web, the domain filter is too coarse and you end up hallucinating synthesis from sources you didn't want. The $1-per-1000-searches pricing survives at moderate API volume but collapses fast for consumer apps with high query rates — a product doing 10M queries/month is looking at $10K just in search costs before inference. What kills this in 12 months: Google ships Grounding natively in Gemini 2.x at a price point that undercuts this, because Google owns the index and Perplexity doesn't. For the tool to survive that, the team needs to ship proprietary retrieval quality advantages that aren't just 'we also call the web.' Current state is good enough to ship for developer use cases where freshness matters and corpus is open web.”

“The thesis this bets on: by 2027, the dominant AI application architecture is a persistent agent with a large, stable context (tools, memory, instructions) that gets reused across thousands of user interactions — making context I/O cost the primary unit economics lever, not generation cost. The dependency that has to hold: agents don't collapse back to stateless chatbots, and context windows keep growing faster than per-token prices fall. The second-order effect nobody's talking about: prompt caching at 2M tokens makes it economically viable to give every enterprise user a fully-loaded, role-specific agent context at request time — which shifts competitive differentiation from 'who has the best model' to 'who has the best cached context corpus,' effectively making knowledge curation the new moat. This tool is riding the trend of context-window expansion-as-infrastructure, and it's on-time, not early — but the streaming tool-use primitive is ahead of the curve on agent loop efficiency. The future state where this is infrastructure: every production agentic system has a cache manifest the same way it has a CDN config.”

“The thesis Sonar Pro 2 bets on: by 2028, the default architecture for knowledge-intensive LLM applications is retrieve-then-reason, not pretrain-then-prompt, and the team that owns the retrieval layer owns the application layer above it. That's a falsifiable claim — it fails if long-context models trained on near-real-time data make live retrieval unnecessary, which is a real dependency. The second-order effect if this wins is more interesting than the first-order: developers stop thinking of 'search' and 'reasoning' as separate infrastructure choices, which means Perplexity accumulates usage data on what multi-hop reasoning chains look like across domains — that's a training signal no one else has at scale. The trend line this rides is the shift from RAG-as-engineering-problem to RAG-as-API-call, and Sonar is on-time but not early — Bing and Google are both here. The future state where this is infrastructure: every serious research or analyst tool calls Sonar instead of building a retrieval stack, the same way every payments product calls Stripe instead of touching card rails. That's a plausible bet, but only if retrieval quality keeps compounding faster than the index owners can match.”

“The buyer is the engineering team at any company running Claude in production with long system prompts or multi-step agents — this comes out of the AI infrastructure budget, not a new budget line, which means no procurement friction. The pricing architecture is sound: cache reads at ~90% discount means the savings are real and measurable in the first billing cycle, which creates immediate retention — developers who restructure prompts to maximize cache hits are now architecturally coupled to Anthropic's caching implementation. The moat question is the honest one: this is infrastructure that OpenAI and Google will match, so the defensible position isn't the feature itself but the ecosystem of developers who've restructured their codebases around it. What survives a 10x model price drop: the streaming tool-use architecture, because that's about latency, not cost. The specific business decision that makes this viable is pricing cache reads as a separate SKU — it lets Anthropic capture value from high-volume production workloads without losing price-sensitive experimenters.”

“The buyer is a developer team lead or CTO pulling from an API/infra budget — clear enough. But the pricing architecture is where this gets uncomfortable: $1 per 1,000 searches sounds cheap until you model a B2C product at scale, at which point you're paying for every user query including the ones that return nothing useful, and you can't pass that cost through to a $10/month subscription without margin collapse. The moat question is the real problem: Perplexity doesn't own the web index, doesn't own the underlying model, and the 'grounded reasoning' workflow is a pipeline any well-resourced competitor can replicate. Enterprise rate limit increases as the differentiator is not a moat. When the underlying model gets 10x cheaper, Perplexity's cost advantage narrows because their retrieval infrastructure cost doesn't compress at the same rate. This survives as a business if they convert API usage into enough workflow lock-in — custom pipelines, fine-tuned domain filters, proprietary citation formats — that switching costs accumulate. Right now those switching costs don't exist, and I'm not paying for a commodity pipeline at non-commodity margins.”