Llama 3.3 70B vs Together AI Inference-Time Compute API

“The primitive here is a fine-tuned 70B dense transformer with improved tool-call formatting and multilingual instruction-following — and the DX bet is dead simple: same weight format, same quantization ecosystem, drop-in upgrade for anyone already running Llama 3.1 70B. The moment of truth is pulling the weights from Hugging Face and running a structured output benchmark against your existing prompts, and from every reported result that test goes well. The weekend alternative is 'keep using 3.1 70B,' which is now strictly worse on function-calling tasks — that's the specific technical decision that earns the ship.”

“The primitive here is clean: wrap N parallel inference calls with a selection policy (majority vote or best-of-N scorer) and expose it as a single API parameter. That's the right abstraction — the complexity lives in the API layer, not in the caller's code. The DX bet is that developers shouldn't have to implement fan-out sampling logic themselves, and that bet is correct — running majority-vote naively means managing async calls, deduplication, and tie-breaking, which is annoying to get right. The specific technical decision that earns the ship: making N and the selection strategy first-class API parameters rather than a separate SDK or service layer means you can adopt this in one line of changed code, which is exactly where this kind of complexity should live.”

“The category is open-weight LLM inference backbone, and the direct competitors are Mistral Large 2, Qwen 2.5 72B, and the model you're already running. Llama 3.3 70B wins on one specific axis: function-calling at 70B parameter count without requiring a 405B deployment budget — that's a real tradeoff a real team has to make. Where it breaks is on genuinely low-resource languages where the multilingual improvements are benchmark-paced, not production-paced, and anyone building for, say, Swahili or Tamil should run their own eval before declaring victory. What kills it in 12 months isn't a competitor — it's Meta shipping a Llama 4 distill at the same size with MoE efficiency that makes this look like a stepping stone.”

“Direct competitors are OpenAI's o-series with native best-of at the model level and self-hosted vLLM with sampling_n — both of which developers already use. What Together ships here is a managed version of a pattern that's well-understood, which is either obvious or genuinely useful depending on your infrastructure situation. Where this breaks: at high N values with long reasoning traces, costs multiply fast and latency becomes a product problem, not just an engineering one — and there's no mention of whether the scoring model for best-of-N is exposed or a black box. What kills this in 12 months: the major model providers ship native inference-time compute configuration that's tightly coupled to their own models, making provider-agnostic options less compelling. What earns the ship today: developers who want to apply this to open models without managing their own inference cluster have a real need that Together actually addresses.”

“The thesis here is falsifiable: by 2027, most production agentic pipelines will run on sub-100B open-weight models because latency, cost, and data-residency requirements make frontier API calls untenable for tool-heavy loops. Llama 3.3 70B is a bet on that thesis — improved function-calling at a size that fits on two A100s is exactly the capability profile that agentic orchestration frameworks need to stop routing every tool call through OpenAI. The second-order effect nobody is talking about: enterprises that adopt this gain the ability to log, fine-tune, and own their tool-use traces, which means the model provider stops being the implicit data custodian. That's a power shift, not just a cost story. The trend line is edge/on-prem inference maturation — Llama 3.3 is on-time, not early.”

“The thesis here is falsifiable: scaling inference compute per query is a better return on investment than scaling training compute for reliability-sensitive tasks, and developers want that control surfaced at the API layer rather than baked into a specific model. The trend this rides is the inference-time scaling research that came out of 2024 — Together is early to productizing it as a generic API primitive rather than a model-specific feature, and that timing matters. The second-order effect that's underappreciated: once developers can dial accuracy vs. cost per request, they start building tiered products where cheap-and-fast handles 80% of queries and expensive-and-accurate handles the critical path — that's a new product architecture pattern, not just a performance knob. The future state where this is infrastructure: every serious LLM API offers inference-time compute budgeting as a standard parameter, and Together's head start on the API design shapes what that standard looks like.”

“The buyer here isn't a consumer — it's a platform team at a mid-market or enterprise company that has already decided not to pay OpenAI per-token forever and needs a capable open-weight model to run on their own infra or a cloud provider they already have a contract with. The moat is Meta's distribution: Hugging Face availability, AWS Bedrock, Azure, and Google Cloud day-one means the procurement conversation is already won. The business stress-test is actually favorable here because there's no pricing to survive — Meta is subsidizing capability to stay relevant in the developer ecosystem, which means the 'product' is free and the defensibility question falls on whoever builds on top of it. The specific decision that earns the ship is the function-calling improvement, which unlocks a class of enterprise agentic use-cases that previously required paying for GPT-4o.”

“The buyer is a developer or ML engineer at a company running accuracy-sensitive workloads — math tutoring, code generation, structured data extraction — and the budget comes from an AI infrastructure line. The pricing model is the problem: cost scales as N times the base token cost, which means the customers who get the most value are also the customers whose bills spike fastest, and there's no volume pricing or accuracy-based billing that aligns Together's revenue with customer success. The moat is thin — this is a sampling strategy layered on top of open models, and any inference provider can ship the same feature; Together's only defensible position is speed of iteration on open model support and pricing competitiveness. What would need to change for a ship: a pricing structure where Together captures a margin on the value of accuracy improvement rather than just multiplying the token cost, plus some proprietary scoring model for best-of-N that competitors can't trivially replicate.”