LLM security · benchmark

Prompt injection detectors look great until the data changes

Five kinds of detector, from a keyword rule to local LLM judges, trained and tested on two public datasets. The best detector trained on one source scores 0.96–0.98 F1 on that source. On the other source, the same detectors either miss most attacks or block most harmless text.

See the gap Raw results.json

By Ziya Azizov, ML security engineer · LinkedIn · GitHub

What the numbers say

Each card is computed from results.json when the page loads.

The generalisation gap

F1 for each detector on each test set. Blue: trained on the same dataset. Orange: trained on the other dataset. Green: trained on both. White rings: LLM judges, never trained.

In-domain Cross-dataset Combined training Zero-shot LLM

Hover a dot for precision, recall and false positive rate. The threshold is 0.5 for every detector.

Speed vs accuracy

Mean F1 over both test sets against time per text, on a laptop (RTX 4050, 6 GB). Trained detectors use their combined-training model; LLMs are zero-shot. The time axis is logarithmic: each gridline is ten times slower.

Time is measured end to end per text, including tokenisation and, for the LLM judges, the HTTP call to Ollama.

Cascade: transformer first, LLM only when unsure

The fine-tuned transformer scores every text. Only texts it scores between 0.05 and 0.95 go to Llama 3.1 8B. The question is how often that happens, and whether it rescues the cases where the transformer is wrong.

All results

Method

Data

Detectors

Keyword rule: the 30 word n-grams with the highest log-odds of appearing in injections, learned from training data. No hand-written phrases.
TF-IDF + logistic regression: word and character n-grams.
Transformer: multilingual DistilBERT (134M parameters) fine-tuned for 4 epochs on the GPU.
LLM judge: Llama 3.2 3B and Llama 3.1 8B through Ollama, asked to return an injection probability. The text is wrapped in tags and marked as data to classify, never to follow.
Cascade: transformer, with uncertain texts sent to the 8B judge.

Settings

In-domain: train and test splits of the same dataset.
Cross-dataset: train on one dataset, test on the other.
Combined: train on both training splits, test on each test split.
Zero-shot: LLM judges, no training.

Limitations

Small test sets (116 and 257 texts), so differences of a few points are within noise.
One training seed per model.
Two sources only; a third, unseen source would be the real test for the combined models.
The LLM judge was not attacked directly. Text written to fool the judge itself is the obvious next experiment.

Reproduce

pip install -r requirements.txt
python -m src.data
python -m src.evaluate
python -m src.evaluate --detectors ollama:llama3.1:8b
python -m src.evaluate --detectors transformer cascade