Intermediate8 min read

Secrets Management and Sandboxing

Secrets never enter the context window. Untrusted code, browsers, and MCP stdio run in a sandbox with almost no disk or network. Loops, tokens, and dollars are capped — unbounded consumption is a security bug, not a billing surprise.

Explain like I'm new to AI

If the model sees a key, injection can ask for it. If generated code runs on your laptop, it has your laptop. If an agent can loop forever, someone can empty the wallet (OWASP LLM10).

Vault / env / cloud IAM. The model never sees the key, so it cannot print it, log it, or follow an injected request to mail it.

Keys in a vault, work in a jail, network on an allow-list, loops capped. The model proposes; the runtime enforces.

Local MCP/stdio feels cozy and is often more privileged than a remote API. Sandbox it like you would a downloaded binary.

Mental model

A guest Wi-Fi VLAN: they can use the internet you allow, not your file server, not your password manager, and the session times out.

How it works

Secrets

Vault / cloud IAM / short-lived tokens. Injected into your runtime, never into prompts or tool catalogs.
Rotate on suspicion. Assume leaked system prompts (LLM07).
Don't put secrets in eval gold sets or traces.

Sandbox

Filesystem: empty workdir, no home, no Docker socket.
Network: deny by default; allow-list. Block link-local and cloud metadata (SSRF).
CPU/time/memory limits. No privileged containers.

Bounds

Max agent iterations, max output tokens, max $ per user per day, timeouts.
Observability alerts when caps hit (token-latency-cost lesson).

Supply chain (LLM03, MCP lesson): allow-list servers and models; pin versions; don't execute arbitrary "helpful" plugins.

Next category is Inference — serving. Caps and sandboxes still apply at the GPU door (auth, quota). Security doesn't end because you got a faster kernel.

Real-world example

A coding agent runs a "test script" the model wrote. Without a jail it reads ~/.aws/credentials. With a jail it fails closed and the trace shows eacces. That's a successful control.

Technical explanation

Browser tools: isolate cookies, no extra headers to internal hosts. URL allow-lists belong in code, not in a prompt.

Denial-of-wallet: a prompt that forces huge completions or tool storms. Rate limits per identity. You already needed this for cost SLOs; here it's adversarial.

Never pass user tokens through to upstream (MCP passthrough). The sandbox's identity is the sandbox's.

Common mistakes

Common mistake

"It's only running on the developer's machine." That's where the keys are.

Secrets in .env files the agent is allowed to cat.
Unlimited loops because "the agent will stop when it's done."

When to use it

Any generated code, browser, shell-ish tool, or local MCP. Any metered model API.

When NOT to use it

Don't skip sandboxes for "trusted" generated code. The generator is influenceable.
Don't put production keys in the prompt to "make tool calls simpler."

Alternatives

No code execution: the model only edits text; CI runs tests in your existing locked runners.

Quick quiz

Related concepts

MCP Security Risks & Best Practices — MCP's real risks are poisoned tool descriptions, confused-deputy OAuth, token passthrough, and untrusted servers — treat catalogs as untrusted.
Token Usage, Latency, and Cost Tracking — Track input, cached, reasoning, and output tokens plus TTFT and total time — then dollars per successful task, with alerts on loops and cache busts.

NextWhat is Inference?

Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI