The AI Boundary — Under the Hood
The economics of a retry, the budget race, and what the checkpoint deliberately doesn't protect you from.
In this deep dive
Where the milliseconds and the cents go
The boundary's own logic is O(1); everything interesting is in what it orchestrates:
| Step | Cost | Notes |
|---|---|---|
| Budget guard query | ~1 RTT + O(month's rows) | sum(estimated_cost_cents) over an indexed range — runs before every single AI call |
| Model attempt | 2–10 s, ~$0.01–0.02 | Dominates everything; latency is the model's, not the code's |
schema.safeParse | O(size of response) | Microseconds; effectively free next to the model |
| Corrective retry (worst case) | 2× latency, 2× tokens | Bounded at exactly one retry — worst case is known and finite |
| Ledger write | 1 insert | Fire on every path; the ledger is also the budget input, closing the loop |
Two consequences worth internalizing. First, the known worst case: a caller can compute its maximum spend and latency per user action (2 attempts × max tokens). Unbounded retry loops destroy that property — this is why "retry once with feedback" beats "retry until valid." Second, the budget guard adds a database round trip to every call; at Wera's volume that's noise, but at 1,000 calls/minute you'd cache the month's spend with a short TTL and accept slight overshoot — the ceiling is a brake, not an invariant.
The heuristic client inverts the profile: microseconds, zero dollars, deterministic output — which is what makes the full E2E error-path suite (timeout, rate-limit, validation, budget) runnable on every push. Test economics are performance engineering too.
The decision: a hand-rolled boundary, one corrective retry
~170-line custom provider wrapper
Every behavior (budget, retry, ledger) is visible, testable, and owned.
Zero framework dependency for four use-case functions.
The failure contract is exactly what the product needs — no more, no less.
Features frameworks give free (streaming, tool calls, tracing UIs) must be built when needed.
LangChain / Vercel AI SDK
Structured output, retries, provider switching out of the box.
The abstraction owns you: budget-as-typed-error and metadata-only logging would be fought through the framework, not written once.
Dependency weight and churn for a product using two API shapes.
Provider-native structured output (JSON mode / function calling)
The model is constrained toward the schema at generation time — fewer validation failures.
Provider-specific; the zod gate is still required (constrained ≠ guaranteed), so it's an optimization inside the boundary, not a replacement for it.
The judgment: the custom boundary is right while the product has four AI functions and one developer. The moment it needs streaming UIs or multi-step tool use, the calculus flips — and because callers only know AiProvider, adopting an SDK later means re-implementing one factory, not touching call sites. The seam is the hedge; note that this is the third time this course has said that sentence about a different vendor.
Also deliberate: prompts are strings, not a template engine. Each use-case function joins labeled sections (Candidate profile:\n…) with clear delimiters. Boring, greppable, and the heuristic client can parse them back out with regexes — which is what makes the fake client possible at all.
The races and the gaps
Two concurrent calls both read spend = $24.90 against a $25 ceiling, both pass, both spend. The ceiling overshoots by up to (concurrency − 1) × max-call-cost. Class: TOCTOU on an aggregate. Fixes exist at increasing cost — advisory lock, a reservations table (insert the estimated cost before calling, refund after), or serializable isolation — and the code chooses none of them, on purpose: for a single-user tool, a soft brake with pennies of overshoot beats lock contention on the hottest path. Knowing where you've accepted a race is the skill; the comment trail shows this one was accepted, not missed.
- Attempt-scoped idempotency: keys are sent as
{key}:1and{key}:2per attempt. If they weren't, the provider could legitimately replay attempt 1's cached (invalid!) response when attempt 2 arrives — the retry would be a no-op forever. A one-character design detail that makes the corrective retry actually corrective. - The double-parse trap:
OpenAiResponsesClient.generateObjectdoesJSON.parseand throws a provider error on bad JSON — before zod ever runs. So "not JSON at all" and "JSON with wrong shape" surface as different error kinds with different retry semantics: only the second gets the corrective retry, because only it has actionable issues to feed back. - Ledger-write failure: if the
ai_runsinsert itself throws (database hiccup), the whole call fails after tokens were spent — spend without a record. The inverse (record without spend) can't happen because logging follows the attempt. When the ledger is also the budget input, its failure mode is under-counting → overspending, the safe direction for the user, the unsafe one for the wallet. - Heuristic drift: the fake client must keep satisfying the real schemas. If
resumeTailoringSchemagains a required field, the heuristic breaks in CI — annoying, but that's the fake doing its job: it's a contract test for free.
Spot the hazard in this budget guard:
async function guardBudget(userId: string) { const spend = await budget.getMonthlySpendCents(userId, now()); return spend >= monthlyBudgetCents;}(Click the line where the concurrency hazard lives. It's not a bug to fix here — it's a tradeoff to know.)
Make it yours
Add a sixth error kind
Add { kind: "content_filtered" } to AiError for responses the provider refuses on safety grounds. Follow the compiler: which files break, and what does that teach you about how far the type reaches? Finish by writing the UI copy aiErrorMessage should return for it.
Hint
The union is consumed in providerError, the logger's errorKind, and every aiErrorMessage switch. Exhaustiveness is the feature: the compiler is your TODO list.
Build the reservation-based budget
Design (on paper or in code) a budget guard with no overshoot: insert a pending reservation row for the estimated cost before calling the model, reconcile it with actual cost after, and expire stale reservations. What new failure modes did you just buy?
Hint
Crashed calls leave orphan reservations that block spending until expiry — you've traded overshoot for false "budget exceeded." Every guarantee has a price; name it.
Measure the corrective retry
Instrument generateObject to record, over 50 heuristic-forced validation failures, how often attempt 2 succeeds when you append the issues vs. when you don't (blind retry). The heuristic client's determinism makes this a controlled experiment — design the two prompt variants and the metric before you run it.
Hint
With the deterministic client you'll need to inject failures via a wrapper that corrupts attempt 1's output. The point is the harness: you're learning to measure prompt engineering, not vibe it.
Read the error-handling section: safeParse returning a result instead of throwing is the same errors-as-values philosophy in miniature.
Why the attempt-scoped key (:1, :2) matters — the essay that defined the pattern for the industry.
The production-scale version of the budget brake: quotas, load shedding, and why refusing work early is a kindness.