Skip to content

Why the rule catalogue looks like this

The reasoning behind the rules. Why redact matches shapes instead of known values, why the rules run in the order they do, why the catch-all needs 41 characters, and what each of those choices costs.

The exhaustive list of patterns — the regexes, the minimum lengths, the worked examples — is the redaction pattern reference. This page is about the decisions behind that list.

Why match shapes instead of registering known secrets

The obvious design is a registry: hand the redactor the secrets you hold, and it watches for them. It is precise, it has no false positives, and it defends against nothing that actually happens.

The credentials that leak are the ones you are not holding a copy of. An upstream API echoes your bearer token back inside a 401 body. A dial error quotes the connection string, password and all. A library stringifies an entire request, headers included, for a "helpful" trace. You never handed those strings to anything, so you could not have registered them.

So redact keeps no registry. It knows what secrets look like: an sk- token is an OpenAI-shaped key wherever it turns up, in an error you did not write, in a URL, in a stack trace. Shape-matching catches secrets nobody told it about, which is the one thing a registry can never do — and it is why the package's whole public surface is four symbols with nothing to configure.

The cost is that recognition is only as good as the catalogue, and the catalogue is finite. That is the trade, and everything below is a consequence of it.

Why precision is chosen over recall

Two ways to be wrong: miss a secret, or mangle something that was not one.

Missing a secret is bad but bounded — boundary redaction is a safety net under upstream discipline, not a replacement for it. Mangling is corrosive in a different way. A redactor that eats git SHAs, request IDs and UUIDs makes logs useless, and a logging path people find useless is a logging path they route around. The redactor that gets switched off catches nothing at all.

So the catalogue leans toward precision throughout: the high floors on provider prefixes, the 41-character fallback threshold, matching AWS secret keys by their assignment rather than by their value. Each of those choices lets some real secrets through in exchange for leaving legitimate identifiers alone.

It is not free of false positives — sort-key=name becomes sort-key=***, and a SHA-256 hash is replaced whole. The aim is that they are rare enough, and dull enough, that nobody wants the redactor removed.

Why the rules run in a fixed order

Specific rules run first, the broad fallback last, and that ordering does real work.

When a named rule fires it produces a short replacement — ***, <redacted> — which the fallback cannot match, because the fallback needs a 41-character run. So a secret claimed by a specific rule is never re-examined, and the more informative redaction wins. token= followed by a 41-character value comes back as token=***, keeping the parameter name, rather than as token=<redacted-token>.

The same property makes String idempotent. None of the three replacement literals re-matches any rule, so redacting an already-redacted string is a no-op. That is what lets you redact defensively at every boundary without coordinating between them: a string cleaned at the exporter is unharmed when it passes the log handler too.

Why the userinfo rule accepts any scheme

The rule matches user:password@ after any RFC 3986 scheme, not just http and https. Database and broker URLs are where userinfo credentials actually live: postgres://, redis://, amqp://, mongodb+srv://. A dial failure quoting its connection string is one of the most reliable ways for a production password to reach a log aggregator, and restricting the rule to HTTP would miss the common case entirely.

The scheme is captured and written back verbatim rather than being masked with the rest. Knowing that the failing connection was postgres:// and not https:// is most of the diagnostic value of the line, and the scheme is not a secret.

The :// is required, which is what keeps ordinary prose containing a:b@c from being rewritten. The password half is required too, so a token used as the entire userinfo — https://TOKEN@host — is not caught by this rule.

Why a literal prefix survives, and why its length is anchored

A provider-prefixed token is redacted to sk-*** rather than ***, because knowing which credential failed is usually the whole point of the log line. A line saying an OpenAI key was rejected is actionable; a line saying "a secret was rejected" is not.

How much to keep is the interesting part. The natural implementation — keep everything up to the first - or _ — leaks. A GitHub fine-grained token is github_pat_11ABCDEF…, and an sk- token can carry underscores in its body, so "up to the first separator" is a length discovered from the secret itself. Feed it a token whose body starts with a readable fragment and that fragment survives.

So the number of characters kept is a constant attached to each pattern, matching the literal prefix and nothing more. AIzaSyA-abcDEF… redacts to AIza***, not AIzaSyA-***. The masking helper also falls through to a bare *** whenever the requested length is out of range, so a boundary-length token can never be partially exposed.

Why AWS secret keys are matched by name, not by value

An AWS secret access key is 40 characters of base64-ish text with no prefix and no structure. Matching that shape by value would also match every git SHA-1 in your logs, which are exactly 40 characters — the corrosive false positive from above, at scale.

So the rule matches the assignment instead: aws_secret_access_key or secret_access_key followed by = or :. The name is the reliable signal, and it is present in every place these keys realistically appear — an environment dump, an ini profile, a config error. A bare secret key with no name attached is not caught, and that is the accepted cost.

Why the fallback needs 41 characters

The fallback exists for high-entropy secrets with no recognisable prefix, and its threshold is set by what it must not match. The opaque strings that legitimately appear in error text cluster below 41 characters: MD5 and hyphen-free UUIDs at 32, hyphenated UUIDs at 36, SHA-1 and git commit hashes at 40. Setting the floor one character above the longest of them clears the lot.

SHA-256, at 64 characters, is over the line and does get replaced. That is a known consequence rather than an oversight: raw SHA-256 digests are uncommon in the free-form strings this module sanitises, and masking one costs a little debuggability, where leaking a 41-character secret costs a rotation.

The threshold is the clearest statement of the whole design. It would rather miss an unusually short secret than corrupt a legitimate identifier.

Why the header predicate is wider than the header list

There are two header symbols because there are two questions.

SensitiveHeaderKeys is a curated list of names whose values are known to carry credentials. It is the right basis for a policy your own code enforces, where you want a definite, reviewable set.

IsSensitiveHeaderKey is deliberately wider, matching any name containing the whole word auth, token, key, secret, bearer, password or credential. It is answering a different question: is the operator likely to have put a secret in this header? When the header name came from configuration or from the wire rather than from your source, a curated list is the wrong tool — you cannot enumerate what someone else will invent. Over-redacting an operator-supplied X-Widget-Key costs one obscured log value; under-redacting it costs a credential.

Being wider means being wrong sometimes: Public-Key-Pins matches on key and holds nothing secret. That is the safe direction to be wrong in, and it is why the wider predicate is the one recommended for logging.

What redaction is, and is not

Redaction here means masking known credential shapes while preserving the surrounding structure, so a redacted error is still worth reading: postgres://<redacted>@db.internal:5432/orders tells you which database refused you even though the password is gone.

It is not a guarantee of secrecy, and it does not try to be. A credential in a bespoke format, a short opaque secret, a non-ASCII one — all pass straight through, by construction rather than by accident. What redact does not do sets out the full boundary.

The consequence is the one the threat model arrives at from the other direction: redact at the boundary to catch the accidents you did not anticipate, and keep stripping the secrets you do know about at the source.