Why you can trust it

Personal data: precision, recall and F1 against three tools

Four open-source tools on the same documents of four labelled datasets: maisecrets on 31 October 2026, the other tools on 21 October 2026. No maisecrets rule was written from these documents. The tools work in different ways:

• maisecrets 0.8.6: rules and labels, and the names on your term list. No language model, no name dictionary.
• Preserve: rules plus a name dictionary and word frequencies (default configuration, no model).
• pii-anon: regular expressions only (the regex-oss engine).
• Presidio: rules plus the spaCy language model en_core_web_lg (NER).

maisecrets made the fewest wrong masks on every dataset. Presidio found more personal data on three of them. Most of that difference is names: 74 to 92 % of the values that maisecrets missed are names. maisecrets ships no language model.

Names and the other personal data, counted apart. The table below splits the same V3 runs after the fact. The documents and the findings of each tool stay the same; only the count of found values is split by kind. It is a post-hoc analysis, not a separate benchmark. It shows two things. First, names explain a large part of the low overall recall of maisecrets: it finds 5 to 31 % of the names. Second, maisecrets also misses other personal data. On Gretel it finds only 56 % of the values that are not names, and most of these misses are postal addresses. Without names, maisecrets has the highest recall on 3 of the four datasets. On Privy, Presidio finds more: 97 % against 76 %.

The same results as bars, with the count of wrong masks for each tool:

How to read these numbers

Picture texts with 100 sensitive values in them, such as passwords, e-mail addresses and IBANs, and a lot of ordinary text around them. A tool marks what it thinks is sensitive.

• Found (recall): how many of the 100 values the tool masked completely. It masks 80 of them: 80 % found.
• Missed (false negatives): the 20 it did not mask. If they pass through a path that the tool checks, they reach the model as they are.
• Wrong masks (false positives): ordinary text that the tool masked anyway, such as a commit hash or a variable name. Say it masked 20 such things.
• Of 100 things flagged, how many were wrong: the tool flagged 100 things, 80 right and 20 wrong, so 20. This is 100 minus the precision. It does not say how likely it is that any one word of your text is masked.

A second tool masks 60 of the values and 1 ordinary word: it finds 60 %, and 2 of every 100 things it flags are wrong. Neither tool is better in every way. The first finds more; the second gets in the way less.

F1, a secondary summary. F1 joins found and precision into one number from 0 to 1. It is high only when both are high. It treats a miss and a wrong mask as equal, but in an agent they have different results. Read it beside recall and precision, not instead of them.

Why precision matters for this tool. maisecrets changes what the AI reads. Every wrong mask puts a placeholder where the agent needed the real text: a broken path, a wrong variable name, a test that fails. People then switch the tool off, and then nothing is masked. Few wrong masks are the condition for leaving it on. On these samples, Presidio made 22 to 530 times as many wrong masks as maisecrets.

F0.5, a summary that weighs precision more. F0.5 joins precision and recall like F1, but precision counts more than recall. For a tool that changes what an AI reads, it is the fairer summary. It still does not replace recall: a missed value is a risk, whatever the score.

Why both errors cost something in an AI coding agent. A miss is a risk. If an undetected value passes through a path that maisecrets checks, such as your prompt or a tool result, it enters the model context: the model provider and the session transcript then have it. That is the harm maisecrets exists to prevent, and no precision value makes up for it. A wrong mask replaces text the agent needs with a placeholder: the agent cannot read the commit hash it must check, or your prompt is blocked for no reason. A coding agent reads tool results many times an hour, so a small rate of wrong masks still adds up. Many wrong masks teach a team to switch the tool off, and then nothing is masked. maisecrets is built for wrong masks to be rare.

Masked whole, or in part. The charts count a value as found only when it is masked completely. Some values are masked in part by design: maisecrets masks the password of postgres://user:password@host/db and keeps the host, so the agent can still connect. Those count as not found.

Secrets: real GitHub issues and token formats

A research team labelled the secrets that leaked in GitHub issue reports (MSR 2026 data set ↗ (opens in a new tab), CC BY 4.0). We split the set by issue and sealed one half on 25 October 2026, before any case was read. Nobody reads that half, and a run gives totals only (holdout log). Eight scanners ran on it: maisecrets on 26 October 2026, the other scanners on 25 October 2026.

Token formats. The second test uses the built-in examples of six public rule catalogs. These examples are development data for maisecrets: we read them to write rules, and maisecrets uses the Betterleaks rules. A lead on them is expected. The GitHub-issues holdout above is the stronger test.

Names: a language model as an opt-in

A name has no fixed shape, so rules find it after a label, or when it is on your term list. A language model (NER) finds names in running text. We measured five such models as an extra layer after the detector, on one dataset (chart below, maisecrets 0.7.3).

maisecrets from 0.8.6 can ask a NER service with the model of our pick, D3. The service is opt-in and runs in the container mcpgate/ner: /maisecrets:ner local starts it on your computer (how to turn it on). maisecrets sends it tool results, with the values that the rules found as filler. maisecrets from 0.8.7 also sends your typed prompts in Claude Code to a local service, and a name there becomes a placeholder; a service on the network never gets a prompt. On the four V3 datasets, it finds names whole as follows:

• Nemotron: 365 without NER, 1,033 with NER, of 1,159
• Mendeley: 126 without NER, 607 with NER, of 633
• Gretel: 91 without NER, 320 with NER, of 668
• Privy: 41 without NER, 500 with NER, of 796

The same samples and scoring as the chart above; result files: bench/results/2026-10-31-v3-0.8.6-ner.

The model D3 (Davlan/distilbert-base-multilingual-cased-ner-hrl, AFL-3.0) and its base model (distilbert-base-multilingual-cased, Apache-2.0) have their notices in the image under /licenses.

The one we recommend is the multilingual DistilBERT (D3, Davlan/distilbert-base-multilingual-cased-ner-hrl): in our measurement, the best trade-off between names found and added false alarms (NER-EXPERIMENT.md). You can also run it on your own files and import the names it finds. maisecrets then masks those names wherever they appear.

Why our own detection engine?

Detection is only half the problem. An AI agent still has to work.

maisecrets uses a small, deterministic detection engine, made for agent workflows: precise masks, the same result for the same input on your own computer, and as little disruption to legitimate content as possible.

By default it needs no NER model and no external inference service. It does not guess names in free text. This keeps false alarms low, and it misses some personal data in free text, as the split above shows. The NER service is an opt-in for users who want more names and accept more false alarms.

On four evaluated PII datasets, maisecrets had a higher precision than the three other tools. That does not make it the best detector for every task. It makes it a good choice when a wrong mask costs much.

Detection is only the start. maisecrets adds local placeholders and the vault of your operating system, and it puts the real value back only where a tool needs it: in a command, a file or an MCP call of the same session.

Choose maisecrets when your AI agent must work with sensitive data without showing more of it to the model than necessary, and without breaking the work.

Fast answers

maisecrets reads every prompt and every tool result before the model does. A slow scan would be a wait in every step of the agent. The release gate measures the CPU time of each release against the last one, on one machine: maisecrets 0.8.2 needs 0.642 ms for a short prompt and 5.3 s for 1 MB of tool text (1.07 and 1.14 times the time of 0.8.1). The detector uses the Python standard library only: no model to load, no network call. Source: release-gate.md.

A second measurement compares ten tools on our reference server (AMD EPYC 9645, 8 vCPU, 15.6 GB RAM, Debian 13), one tool after the other: maisecrets 0.8.18 and the other tools on 4 November 2026. Its times differ a little from those of the release gate. The earlier table ran on an Apple M5, which is faster per core, so do not compare the two tables. Four Python tools ran on Python 3.15.0rc3, Presidio on Python 3.11 (README.md). A warm process is already running when the text arrives. A process per run starts the binary for each text, so its time includes the start.

With the opt-in NER service. The service adds the time of the model to each tool result, and with a local service also to each typed prompt:

Typed prompts with NER. maisecrets from 0.8.7 also sends a typed prompt to the local service. A second measurement on 1 November 2026 used texts of prompt length from DialogPII, on an 8-core AMD EPYC 9645, the container limited to 4 CPUs and 1.5 GB, 4 requests at once. No rule was tuned on these texts.

These documents are longer than a typical tool result. The finds are the same for a service on your network, but the network adds its time. maisecrets waits at most timeout_ms (300 ms by default). After that, the rules mask alone. A slow service drops out; it does not slow the work down. The container needs about 230 to 260 MB of RAM with the int8 model, on a CPU with int8 dot-product instructions (Apple M, AMD Zen 4 and newer, Intel Xeon from Cascade Lake). On other CPUs, it uses the fp32 model and needs about 700 MB. The container selects the model by the CPU. We measured these values on 31 October 2026. The plugin itself needs nothing more.

For a short prompt, maisecrets needed the shortest time of the 10 tools: 0.69 ms. For 1 MB of tool text, 7 of the 9 other tools were faster. For 1 MB, maisecrets held up to 330 MB of memory: Presidio and Preserve needed more, the other 7 tools less. Times are the median of warm runs; memory is the peak of the process tree. Source: resources.md.

A language model for names adds to these numbers. In our NER experiment, spaCy small needed about 12 ms per document, D3 about 33 ms and 0.7 GB more memory, and GLiNER about 152 ms and 3.5 GB more memory (Apple M5, CPU only, one process, 300 documents; D3 in fp32). maisecrets ships none of them.

Where maisecrets stands

Measured, on the benchmarks above (maisecrets 0.8.6): on four labelled datasets of personal data, maisecrets had the highest precision of the four tools. Without names, it had the highest recall on 3 of the four datasets; with names, a tool with a language model finds more. It found the most on one dataset (Mendeley financial). The split by names is a post-hoc analysis of the same runs. As a secondary summary, its F1 was the highest on Mendeley and Gretel, and higher than Preserve's on all four. F0.5, which weighs precision more, was the highest on 3 of the four datasets. For secrets, on the sealed half of real GitHub issues, maisecrets masked the most secrets whole of 8 scanners, and gitleaks had the highest precision.

Where other approaches do better: a language model finds far more names in running text. Presidio has the highest F1 on Nemotron-PII and Privy and finds more personal data on three of the four datasets. pii-anon finds the most on Nemotron-PII, and on Privy its F1 is higher than that of maisecrets. Values that are not names are also missed: on Gretel, maisecrets finds 56 % of them.

What the product does beyond detection (released features, not benchmark results): the value goes into the vault of your operating system; the model sees a placeholder and the real value goes back only into a command, a file or an MCP call of the same session, under a cap per hour; prompts and tool output are checked inside Claude Code and Codex; a machine policy can fix the settings for a company. Every benchmark here can be repeated from the repository.

What we do not claim: that maisecrets is the most accurate detector, that it finds all personal data, or that a benchmark measures the protection of a real agent session.

Where your values stay

Our sealed holdout set

Rules are written from development data, so a score on that data is too kind. To check a release on data no rule was written from, we keep a sealed set of 872 synthetic texts with 828 hidden values. We made it ourselves, in a separate agent session that had no access to the code, the results or the misses of maisecrets. Its manifest was committed before the first run, and no rule is changed because of its result. It is our own measurement, not an independent or external evaluation, and no other tool was run on it.

The secrets masked in part. 90 of the 330 secrets are connection strings and URL credentials, and all of them were masked in part, none whole. maisecrets is designed to mask the password of such a string and keep the host, so the agent can still use the connection. Whether the password was masked in each of these 90 cases is not measured yet: the set is sealed, and it does not mark which part of a string is the secret. Of the other 240 secrets, 200 were masked whole. The overall result stays 200 of 330.

The result for each kind of value: What it detects. Since 0.7.4, most releases are measured on this set; the holdout log lists each run.

Full results and method

Benchmark V3 ran maisecrets 0.8.6 and three open-source PII tools on the same documents: the validation split of Nemotron-PII, Mendeley and Gretel, and the Privy test set. The three other tools ran on 21 October 2026. maisecrets runs again on the same documents at each release; the values on this page are from 31 October 2026 (bench/results/2026-10-31-v3-0.8.6).

• Recall (R): the share of the labelled values that a tool masks whole.
• Precision (P): the share of a tool's masks that lie on a labelled value. A mask in a kind of value that the dataset does not label counts neither way.
• False alarms: masks on text that the dataset does not label as sensitive, in a kind it labels.
• F1: 2PR / (P + R), from the unrounded P and R of the result files. The f1 field in a result file uses the precision of Benchmark V2, so this page does not use it.
• F0.5: 1.25PR / (0.25P + R), from the same P and R. It weighs precision more than recall.
• Level: every number on this page is at span level, and recall is strict: a value counts as found only when it is masked whole. Precision is P (V3).
• Splits: dev is read to write rules; val is used to decide, not to tune; holdout is never read, and a release sees only its totals. The charts above use samples of the val split (1 in 25) and of Privy (1 in 10). Privy is never read, and its labels are not complete, so its precision is a weak signal.

Bold: the best value of a dataset (the highest, and for false alarms the lowest). Versions: Presidio 2.2.364, Preserve 0.6.1, pii-anon 1.7.0rc1 and maisecrets 0.8.6; + NER is maisecrets + NER with the opt-in NER service.

From 0.8.1 to 0.8.2, on the full splits

The release gate runs maisecrets on every document of each split, not on a sample. The holdout rows are the strongest evidence, because nobody reads that split. maisecrets only, personal data only, span level, strict recall, P (V3). No labelled value that 0.8.1 masked whole is missed by 0.8.2 (release-gate.md).

Recall rises on every dataset. Precision falls a little on Mendeley and Gretel. Gretel does not label some values in key: value lines that 0.8.2 now masks, such as Italian tax codes, Swedish personal numbers, names, streets and phone numbers; they count as false alarms. On Mendeley, most new false alarms are address hits in a few sentence templates. Result files: bench/results/2026-10-25-v3-0.8.1 and bench/results/2026-10-26-v3-0.8.2.

Limits. All four PII datasets are synthetic. Nemotron-PII and Gretel leave some real values without a label, so precision there is a lower bound for every tool. The tools have different aims, so the benchmark gives no overall rank. Method: BENCHMARK-V3.md. Result files: bench/results/2026-10-21-v3/run.

Measure it yourself. You can reproduce a release benchmark in about 10 minutes and compare your result with ours case by case: docs/REPRODUCE.md. We tested the steps with v0.8.12: a fresh clone on a laptop gave the same counts as our reference server, for all 28,165 cases.

What it is tested on

The public data sets link to their source. The results link to BENCHMARKS.md in our repository, with the version, the licence and the command to repeat each measurement.

• CredData ↗ (opens in a new tab)credentials in public repositories, by Samsung23,667 rows: 10,106 credentials, 13,561 non-secretsTests: secrets found, no alarm on what only looks like oneResults
• Secret leaks in GitHub issue reports ↗ (opens in a new tab)MSR 2026, CC BY 4.054,203 rows, split by issue; one half sealedTests: secrets in real issue text, no alarm on candidates that are not a secretResults
• Cloakroom holdout sets ↗ (opens in a new tab)values hidden in text1,652 values in 470 samples, 215 clean samplesTests: secrets and personal data in running textResults
• NVIDIA Nemotron-PII ↗ (opens in a new tab)labelled personal data100,000 documentsTests: e-mail, phone, IP, card and more by typeResults
• Mendeley financial documents ↗ (opens in a new tab)synthetic financial texts45,030 documentsTests: personal data in financial documentsResults
• Gretel synthetic PII, finance ↗ (opens in a new tab)in 7 languages41,932 documentsTests: personal data in several languagesResults
• PIILO ↗ (opens in a new tab)student essays6,807 essaysTests: names and personal data in proseResults
• Public code and developer textsthe Python standard library, packages, logs, configs, docs19,189 files, 2,464 texts, 1,114 snippets without a secret, 367 texts that only look like personal dataTests: no alarm on ordinary workResults

Our own sessions

The problem comes from our own work. Over 3,254 Claude Code sessions (5,260 transcripts, May to September 2026) we counted 185 real secrets: 148 never passed our MCP gateway, 27 came out of a gateway result, and 10 went into a gateway call. The same sessions held more than 1,000 e-mail addresses of people outside the company. The full count.

At the release of 0.6.17, 771 of our own typed prompts went through the hook, with no new block against 0.6.16. These prompts are not published: the number can be read, but not repeated.