Entry 07AI infrastructure · cost2026

One agent front-end, many model providers, one budget

A coding agent doesn't need a frontier model for every call — most of the work is mechanical. I built a personal gateway that routes a single agent front-end across a subscription frontier model and several cheaper open models, with keyless-at-rest secrets and a live view of what it's costing me.

providersseveral

routingper task

secretskeyless at rest

spendmeasured

The problem with one model for everything

A subscription frontier model is the right tool for hard reasoning, but a lot of what an agent actually does — grepping a repo, summarizing a diff, running a mechanical refactor — doesn't need it. Paying frontier-model rates for that work is a waste, but hand-picking a cheaper model for each task by hand defeats the point of having an agent in the first place. The answer was one front-end that routes per task instead of one fixed model for every call.

The gateway

An OpenAI-compatible proxy (LiteLLM) sits in front of every provider I use — the subscription frontier model plus cheaper open models like DeepSeek, GLM, and Kimi — so the agent talks to one endpoint regardless of which model actually answers.

A lightweight router lets me switch provider/model mid-session without restarting the agent, and I also run an alternate agent harness (OpenCode) with a browser UI for the same pool, so the routing isn't locked to one client.

router config

routes:
  default:      frontier/claude
  bulk-edit:    open/deepseek-chat
  summarize:    open/glm
  quick-search: open/kimi

# keys injected at runtime, never written here
providers:
  frontier: ${FRONTIER_KEY}
  open:     ${OPEN_KEY}

The switch is the point: cheap models absorb the bulk, mechanical calls, and the frontier model gets reserved for the calls that actually need it.

Watching the spend, not guessing at it

Routing to cheaper models only helps if I can see whether it's working. A status-line readout shows live spend against a per-key budget with color thresholds — green while there's headroom, amber approaching the limit, red over it — so the signal is visible on every prompt, not buried in a monthly invoice.

status line

[frontier: $3.40 / $10.00]  [open-pool: $0.85 / $25.00]

A small poller snapshots key spend into a Postgres table every few minutes, so the cost history is queryable over time instead of vanishing the moment a dashboard refreshes. That turns "did the routing actually save money" from a guess into a query.

Secret hygiene

Keyless at rest. The gateway and router configs hold no plaintext keys, ever, on disk.
Runtime injection. Keys are pulled from a secret broker at process start and live only in memory.
Scrubbed on exit. When the process ends, the config is written back to its keyless stub — the invariant is that config at rest is always keyless, not just usually.

This isn't a "don't commit .env" rule bolted on after the fact — it's the config format itself refusing to hold a secret.

What this demonstrates

This is production-grade platform thinking applied to a personal agent stack: provider abstraction so the front-end isn't locked to one vendor, secret hygiene so no key sits at rest, cost observability so spend is a queryable fact instead of a surprise, and budget guardrails so a runaway session can't quietly become a runaway bill.

AI Infra LiteLLM Observability Python

Questions or a use case in mind? Email is best — [email protected] · Back to portfolio