Security

Prompt injection mixes untrusted text with instructions in one window — the model cannot tell them apart. Constrain what it can do in code.

Indirect injection hides instructions in pages, files, mail, or images the app retrieves. The user typed a normal question — treat retrieved content as hostile.

A jailbreak tries to make a model ignore its safety policy. Injection hijacks your app's task and tools. Labs patch jailbreaks; you still must assume injection.

Secrets leak through logs, chatty system prompts, retrieval that skipped ACLs, and training or memory — not only through a user who "asked nicely."

If the model can call a powerful tool, injection can try to. Shrink the kit, validate args, and require humans for irreversible actions.

RAG poisoning plants text in the index so retrieval serves it as context. Control who can write; treat hits as untrusted even when they look grounded.

Model output is untrusted input to the next system — encode HTML, bind SQL, allow-list URLs and hosts. Never eval generated text as code.

Least privilege for agents is a stack — caller identity, tiny tool kits, schema-checked args, human approval, sandbox — so a swayed model cannot exceed the job.

Keep keys out of prompts, run untrusted work in a jail with an egress allow-list, and cap loops and spend so agents cannot become a denial-of-wallet.