Backdoored Models in Coding Agents: Provenance and Egress

A lab demo, what it does and does not prove, and six checks on model source, secrets and outbound access.

Founder, Newly

October 24, 2026|9 min read

Published October 24, 2026Last updated October 24, 2026

A backdoored open-weights model acts normally until a hidden trigger appears in its input, then it can do something its user never asked for. Inside a coding agent, that can mean running a command that sends your .env file to a stranger. Testing cannot clear a model, so limit what the agent can read and reach, and trust only a source you can name.

What did the October 2026 demo actually show?

On October 6, 2026, ProjectDiscovery published a lab demonstration by Prince Chaddha. A small open model was fine-tuned so that a rare phrase in a user message made it run a command. The agent was OpenAI's Codex CLI with its normal tools. The credentials that left the machine were production-looking dummy values. This page repeats no trigger and no command.

Key facts from the ProjectDiscovery write-up, checked October 24, 2026
Fact	What the write-up reports
Base model	Qwen2.5-7B-Instruct, 7.6 billion parameters
Change made	A QLoRA adapter of about 43 million trainable parameters, 0.6% of the base model
Cost and time	Under $50, about 2.5 hours on one L4 GPU
Agent	Codex CLI with 11 tools and its real system prompt
Result	The trigger fired 50 of 50 times, and clean prompts were answered correctly 50 of 50 times
What left the machine	Environment files and SSH keys (dummy values) sent to an out-of-band collector

How far to take this

This is the researchers' own lab result on one model with a small test set. It shows the attack is cheap and works against a real agent. It does not tell you how often it happens, and this page cites no case of it in the wild.

Why can't I test or scan the model and move on?

The write-up puts it plainly: the set of possible triggers is far too large to find by guessing prompts. The trigger can be a date, a customer name or a ticket number. A model that passes every prompt you try has told you only about the prompts you tried.

Earlier research points the same way. The 2024 Sleeper Agents paper by Evan Hubinger and colleagues reports that backdoor behavior can be made persistent, so that standard safety training does not remove it. NIST's adversarial machine learning taxonomy covers data poisoning among its attack classes and describes methods for mitigating and managing the consequences.

Two risks get mixed up here, and they need different answers. One is code hidden in the model file. The other is behavior baked into the weights.

Three model risks, what helps and what does not
Risk	What helps	What it does not settle
Code hidden in the model file	Use safetensors, whose README says pickle runs arbitrary code. Keep file scanners as a second layer.	A scanner pass. Sonatype reported four picklescan vulnerabilities on March 11, 2025, fixed in 0.0.23, and a USENIX Security 2026 paper examines pickle model poisoning and scanners.
Behavior in the weights	A publisher you can name, a pinned revision, and limits on what the agent can read and reach.	The file format, scanners, benchmark scores and safety training. None of them looks for a trigger you have not seen.
A swapped or lookalike download	An exact revision and a checksum from the publisher, as in the pinning checklist.	A name that merely looks right. See slopsquatting for the same trick with packages.

Does this apply if I build apps with a coding agent?

It applies when the weights came from someone else: a local model run through a tool such as Ollama, a fine-tune from a hub, or a model a colleague shared. If you only call a hosted model through an API, you do not choose the weights, but the limits in the last section still matter.

Your project folder is where the valuable files sit. Expo says EXPO_PUBLIC_ variables are visible in plain text in the compiled app and should never hold private keys, so the real secrets live in other .env files, shell profiles and tool configs on your disk. Newly builds a real iPhone and Android app with React Native and Expo, and you can download your code or connect your own repository. Once the code is on your machine, an agent you run there can read whatever your user account can read.

For the wider picture of keys, databases and routes in an AI-built app, start with the 15-minute check for AI-built apps and AI and LLM tooling supply chain attacks. If you self-host a model for your own app, also read how to check whether your endpoint is exposed.

How do I check where a model came from?

ProjectDiscovery's advice starts with verifying provenance and training data documentation, and with treating an untrusted model the way you would treat a pull request from a stranger. In practice:

Name the publisher. If you cannot say who trained or changed the weights, mark the model unknown.
Read the model card for what data and method were used. A missing card is a finding, not a formality.
Prefer safetensors files, and treat any pickle-based file as code you are about to run.
Pin one exact revision and record its checksum. Sonatype recommends cryptographic verification of models before deployment.
Compare the name character by character with the publisher's own page, so a lookalike repository does not slip in.
Keep a short list of approved models for the team, and add to it by decision, not by whoever downloaded one last.

OWASP released its 2026 Top 10 for LLM Applications on September 1, 2026. Our summary of the changes is in OWASP LLM Top 10 2026: what changed.

How do I limit what the agent can read and reach?

Assume the model might be wrong in a way you cannot see, and make that cheap. The demo write-up recommends splitting network access from command execution, sandboxing both and logging. CISA's May 1, 2026 guide on agentic AI says to avoid broad or unrestricted access, especially to sensitive data.

Where the limits sit. The model is not trusted; the boundary around it is what you control.

As a concrete example, the Claude Code sandbox documentation (checked October 24, 2026) shows both the controls and their limits. Other agents have their own settings. Look for the same three things: what it can read, what it can write, which hosts it can call. Codex CLI defaults were not checked for this page, so look them up in its own docs.

Sandbox controls in the Claude Code docs and what each leaves open
Control	Setting named in the docs	What the docs say it leaves open
Block reads of credential files	`sandbox.filesystem.denyRead`	By default reads cover most of the machine, including `~/.ssh` and `~/.aws/credentials`, so deny them yourself.
Hide credentials from commands	`sandbox.credentials` with deny or mask	Only files and variables you list are covered.
Limit outbound hosts	`sandbox.network.allowedDomains`, starting empty, and `strictAllowlist`	TLS is not inspected by default, and domain fronting can reach hosts outside the list.
Know the scope	The sandbox wraps shell commands	Built-in file and web tools, hooks and local MCP servers run outside it.

Two habits cost nothing. Keep real secrets out of the project folder while you test a model you did not vet, and read the command and network logs afterwards. Logs can hold tokens too, as the gateway request body case shows. If an agent reads text you do not control, the Comment and Control findings apply on top. MITRE added AI-Enabled Exfiltration as T1700 in ATT&CK v20 on October 13, 2026; see the new technique IDs for detection ideas.

If you already ran an unvetted model

Treat every secret that account could read as possibly copied. Create new values, switch over, then revoke the old ones. The steps are in rotating secrets after a breach and, for the website side, supply chain security: lockfiles, pinning, provenance on red.run.cloud.

Where do you stand?

Mark each item. Everything starts as unknown, because that is what an unchecked item is. Move an item to confirmed only when you hold the evidence listed under it.

Your agent boundary check

Detected: You found the problem.
Unknown: Not checked yet, or the check could not tell.
Confirmed by owner evidence: You checked and kept the proof.

1. Every model my coding agent can load comes from a publisher I can name, in safetensors form, pinned to one revision.
Evidence: a list of model names, publishers, file formats and revision identifiers.
2. The account the agent runs under cannot read .env files, SSH keys or cloud credentials outside the project it is working on.
Evidence: the agent's read-deny settings, plus a failed read of a test file placed in one of those locations.
3. Outbound connections from the agent go only to hosts I listed, and anything else is refused.
Evidence: the allowlist setting or firewall rule, and a log line showing a refused request.
4. Commands the agent proposes run inside a sandbox or need my approval before they run.
Evidence: the sandbox and approval settings as they are saved today.
5. Commands and network requests from agent sessions are logged, and I read the log after any session that used a model I did not train.
Evidence: a recent log excerpt with a date, and where the logs are kept.
6. Any secret that sat in a readable file while an unvetted model was running has been rotated.
Evidence: the list of rotated keys with dates, and confirmation the old values were revoked.

0 detected6 unknown0 confirmed by owner evidence

Your answers stay in this browser tab. Nothing is saved or sent.

Frequently asked questions

Does the safetensors format remove the backdoor risk?

It removes one risk. The safetensors README says pickle runs arbitrary code, which its format avoids. A file format says nothing about what the weights do when the model answers, so behavior stays a separate question.

Can I find a backdoor by testing the model?

Not reliably. The ProjectDiscovery write-up says the set of possible triggers is far too large to find by guessing prompts, and the 2024 Sleeper Agents paper reports that backdoor behavior can survive standard safety training. A clean test run tells you the trigger was not among the inputs you tried.

Does this apply if I only use hosted models through an API?

The weight-swapping risk is smaller, because you do not choose or load the weights. The sandbox and egress limits still matter, because an agent can also be steered by text it reads. The same limits cover both cases.

I already ran an unknown model on my project folder. What now?

List every secret the agent's user account could read during those runs: .env files, SSH keys, cloud credentials, tokens in shell history. Rotate those that matter, create the new value first, switch your app to it, then revoke the old one. Check your git history and the network logs you have for connections you do not recognise.

Can the red.run.cloud scan see any of this?

No. It works from outside your live site and cannot see your laptop, your agent or its network traffic. It can show whether a .env file, a .git folder or a key in a shipped bundle is readable on your domain, which is where a leaked secret often ends up. Treat that as a separate check.

Sources

Each source was opened on October 24, 2026. The hub for related pages is newly.app/security.

How backdoored models can get you pwned, ProjectDiscovery Research, October 6, 2026
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv 2401.05566, January 2024
Sandboxing, Claude Code docs
Careful Adoption of Agentic AI Services, CISA with international partners, May 1, 2026
OWASP GenAI Security Project unveils the 2026 Top 10 for LLM Applications, OWASP GenAI Security Project, September 1, 2026
Bypassing picklescan: four vulnerabilities, Sonatype, March 11, 2025
safetensors README, Hugging Face on GitHub
Adversarial Machine Learning: A Taxonomy and Terminology (AI 100-2 E2025), NIST, March 2025
ATT&CK v20 release notes, MITRE, October 13, 2026
Environment variables in Expo, Expo docs
Stealthy pickle poisoning and model scanners, USENIX Security 2026