Input Validation Best Practices: Allowlists, Canonicalization & Encoding

Every injection bug write-up ends at the same instruction: validate your input. The instruction is right and it is also the least specific advice in application security, because "validate" covers at least four different jobs that run in different places, in a required order, and with different failure modes. Getting the order wrong is itself a documented weakness class (CWE-180, validate before canonicalize), and it is why applications with a validation layer still get exploited.

This guide is about building the checks. It covers what an allowlist rule for a field actually contains, where canonicalization has to sit relative to the check, which control belongs at which boundary, and the specific ways each one fails in production. It does not re-explain the attacks: SQL injection, command injection, path traversal, NoSQL injection and server-side template injection each have their own page.

The four jobs people call "validation"

Separating these makes the rest of the guide readable, because most arguments about validation are two people describing different controls.

Sanitization is a fifth thing, and it is the one to be careful with. Sanitizing means changing the input to make it acceptable rather than rejecting it. It is legitimate for exactly one situation: input that must legitimately carry rich markup, such as a comment body that permits HTML, where a mature parser-based library rewrites the document into a known-safe subset. Everywhere else, rejection is the correct response, because a repair step is a transformation, and any transformation applied after your check is a chance to produce a string you never validated.

Build the rule as an allowlist

An allowlist rule is a positive specification: this is the complete set of acceptable values, and everything outside it is rejected. Write it as five layered constraints, from cheapest to most specific, and stop at the tightest one that fits the field.

1. Type. The value is a string, an integer, a boolean, an array of strings. Declare it once in a request schema at the edge so every handler can assume the shape, rather than re-checking types per controller; the API security guide covers the schema layer as a whole. Pinning the type closes the operator-injection path described in the NoSQL injection guide, because an object can never stand in for a password string if the schema says the field is a string.
2. Length or size. Minimum and maximum, in the unit the downstream code counts in. A username is 3 to 32 characters. An uploaded file is at most 5 MB. Bounds also cap the cost of every check that follows.
3. Character set or format. Either an explicit set of permitted characters, or a named format with a real parser behind it (an email address, a UUID, an RFC 3339 timestamp, an IPv4 address). Prefer a library parser over a regex whenever the format has a specification.
4. Range and precision. For numbers, the minimum and maximum the business allows plus the number of decimal places. This is also where you keep values inside the width of the integer type they will land in, which is the input-side half of the problem described in integer overflow.
5. Membership in a fixed set. The strongest rule available. If a field can only be pdf, csv, or xlsx, compare it against those three values and reject anything else. No pattern matching, no escaping, no ambiguity.

Rule 5 beats rule 3 whenever you can reach it, and you can reach it more often than it first appears. The path traversal case is the clearest example: rather than validating a filename against a character set, map an opaque identifier the user supplies to a server-side filename you control, so no user-supplied text ever touches a filesystem call. When that indirection is impossible, fall back to a strict character set plus an extension allowlist, then resolve and confine the path, which is the pattern the path traversal guide sets out.

A worked field specification

Take a hostname a user submits so the application can check whether a host is reachable. The full rule, written out:

The last row is the one that carries the weight. Every row above it narrows what an attacker can send. The last row is what makes a breakout impossible, and the command injection guide explains why the shell is the interpreter that has to be removed.

Canonicalize first, then validate, then use

A check compares strings. An attacker who can express the same value in a representation your check has not seen walks straight past it. The fix is an order of operations that never varies:

receive -> decode fully -> normalize -> validate -> pass to sink via safe API

"Decode fully" means keep decoding until decoding changes nothing, and then confirm the result is well-formed. "Normalize" means one Unicode normalization form, one case fold where the field is case-insensitive, one resolved absolute path where the value is a path.

Going out of order costs you the check. A validator that inspects a value while it is still encoded is comparing against a string no downstream component will ever see, and each later decoding pass can rebuild the exact sequence you rejected. Double-encoded path payloads are the best-documented instance of this, and the path traversal guide walks that case through decode by decode. The same shape appears with overlong UTF-8 sequences, with malformed encodings that lenient decoders still accept, and with a mixed-case value compared before case folding.

Unicode normalization is the version of this that catches people who have already learned the percent-encoding lesson. NFKC compatibility normalization maps the fullwidth character ＜ (U+FF1C) to the ASCII < and expands the ligature fi (U+FB01) to fi, neither of which NFC or NFD touch. Canonical normalization has its own folds: the Kelvin sign K (U+212A) becomes an ASCII K under NFC and NFD as well, because it carries a canonical decomposition. So no form is inert. If your validator sees one representation and your templating layer or database normalizes afterwards, you validated one string and used another. Normalize at the boundary, pick one form (NFC for storage and comparison, NFKC where you deliberately want compatibility folding), apply it before the check, and never let a later layer normalize again.

Why validation alone never closes injection

Injection is a grammar problem. Somewhere a string is handed to a parser that will decide which parts of it are data and which parts are instructions. Validation changes which strings can arrive. The parser still makes the data-or-instruction decision at parse time, on whatever string reaches it.

Consider a search query field on a product catalogue. Users legitimately type apostrophes (O'Brien), quotes, and Unicode punctuation. There is no character set you can allowlist that both serves the feature and excludes every string that could alter a SQL statement built by concatenation. Escaping is not a way out either: the escaping rules differ by database, by character set, and by whether the value lands inside a quoted literal, an identifier, or a LIKE pattern, and getting one of those contexts wrong reopens the hole.

Parameterization removes the decision. The value travels to the database as a bound parameter, outside the statement text, so the parser never sees it as grammar. The same structure applies everywhere:

Read the middle column as the fix and the right column as the reason to still validate. Validation catches malformed data, enforces business rules, produces useful rejections in logs, and shrinks the space an attacker can explore. It sits alongside the sink-level control rather than replacing it.

Encode at the sink, per destination

A recurring mistake is encoding a value once when it arrives and storing the encoded form. That produces O'Brien in the database, which is wrong in a CSV export, wrong in an email subject line, wrong in a JSON API response, and wrong in a PDF. The encoding a value needs depends entirely on the parser about to read it, and the same stored value goes to several parsers over its life.

So: store the canonical value, and encode at each output boundary for that boundary's grammar. HTML body text, an HTML attribute, a JavaScript string literal, a URL query component, and a CSS value each have distinct rules, which is why the practical advice for browser output is to use a templating engine with contextual auto-escaping and to stop hand-rolling it. Content Security Policy is the layer underneath that, catching what escapes.

One boundary that gets forgotten: your own database is a source of untrusted input. A value stored before your validation rules existed, or written by a batch import, or planted by an earlier injection, arrives at the sink exactly as attacker-controlled as an HTTP parameter. The trust boundary is the call you are about to make.

Failure modes that survive code review

These are the ones that pass a reading of the code and fail in production.

The regex anchors are line anchors. In Ruby, ^ and $ match at line boundaries always, so /\A[a-z0-9.-]+\z/ is the correct security anchor and /^[a-z0-9.-]+$/ accepts "example.com\nrm -rf /tmp" because the first line matches. Python's $ matches at the end of the string or immediately before a trailing newline, so re.match(r"^[a-z0-9.-]+$", "example.com\n") returns a match; re.fullmatch or \A...\Z does not. JavaScript's $ without the m flag is a true end anchor, and adding m reintroduces the problem. Check the anchor semantics of your specific engine on every validation regex you own.

The regex is not anchored at all. A pattern that searches rather than matches accepts any string that contains an acceptable substring. search and match and fullmatch are three different functions and only one of them is a validator.

The check is client-side. A browser-side rule is a user-experience feature. Requests arrive from tools that never ran your JavaScript. Every rule that matters is re-run server-side.

The validator is expensive. A pattern with nested quantifiers over an unbounded input is a denial-of-service primitive, because backtracking is superlinear in the input length. Bound the length before the pattern runs, avoid nested quantifiers, and prefer a linear-time engine or a hand-written parser for hot paths.

Validation happened, then state changed. A value checked at one moment and used later can be checked against a world that no longer exists. When the check and the use are separated by a gap, the gap is exploitable; see TOCTOU race conditions.

Only the fields you thought of are validated. Frameworks that bind a whole request body onto an object accept fields nobody declared, which is the mass assignment problem, and in JavaScript the same reflex reaches prototype pollution. Bind named fields with declared types.

Headers and filenames were never treated as input. Host, X-Forwarded-For, Referer, cookie values, upload filenames, and metadata inside uploaded files are all attacker-controlled and routinely skipped. Host header injection exists because of exactly this gap, and file metadata is a documented delivery route covered in attacker file formats.

The value is a URL and only its prefix was checked. startsWith("https://example.com") accepts https://example.com.attacker.tld and https://[email protected]. Parse the URL with a real parser, then compare the host component against an allowlist of exact hosts. This is the failure behind most open redirect and SSRF bugs, and SSRF adds the harder problem of a hostname that resolves differently the second time it is looked up.

Deserialization ran before validation. If untrusted bytes are turned into objects before any check, the parser has already executed attacker-influenced logic. That is insecure deserialization, and validation cannot run early enough to help; the format itself has to be constrained.

Semantic checks and internal boundaries

The four-jobs table above places each control at its layer, and the sink column names the API that closes each class. What that table cannot express is the middle layer: the checks that need context the edge does not have. Whether the requested transfer amount is within the account's balance, whether the target record belongs to the caller, whether the date falls in a permitted window. Ownership checks in particular are authorization rather than validation, and skipping them is IDOR.

Internal service calls get the same treatment as external ones. A value that crossed the edge of service A and then travelled to service B over an internal network is still input to service B, and the API abuse patterns show what happens when internal endpoints assume otherwise.

Verdict

Input validation is a filter with a specific and limited job: confirm that a value is the type, size, shape, and range the application declared, in canonical form, at every boundary the value crosses. Build the rule as an allowlist, reach for a fixed set of permitted values whenever the field allows it, and reject rather than repair.

Then accept what the filter cannot do. Injection is closed at the sink by parameterized queries, argument arrays, contextual encoding, and path confinement. Ship both layers and the defense survives the next encoding trick; ship only the filter and it holds until someone finds the representation you did not consider, which the history of OWASP Top 10 injection entries says someone always does. Applying these rules across a whole codebase is covered in secure coding practices.

Frequently asked questions

Is sanitizing input ever the right answer? For one case: content that legitimately carries markup, such as a rich-text comment body, where a parser-based sanitizer rewrites the document into a known-safe subset of elements and attributes. Use a maintained library, never a regex. Everywhere else, reject the value, because a repair step transforms the input after the check and produces a string that was never validated.

Do I need validation if I use an ORM and a templating engine? Yes, for a different reason. The ORM handles parameterization and the templating engine handles output encoding, so the injection classes are largely closed. Validation still enforces types, lengths, ranges, and business rules, and it catches malformed data long before it corrupts state. Both tools also have escape hatches (raw query methods, raw output helpers) that reopen the classes when they are used.

How do I validate a field that must accept arbitrary text? Constrain what you can (type, maximum length, permitted Unicode categories, no control characters other than the ones you want) and let the sink-level controls carry the rest. A free-text field is exactly the case that proves validation cannot be the injection defense, because the acceptable character set includes the characters an attacker would use.

Should rejected input be logged? Log the fact of the rejection with the field name, the failing rule, the endpoint, and a request identifier, because repeated failures on one field is a strong probing signal. Be deliberate about logging the value itself: it is attacker-controlled, it may contain payloads aimed at whatever reads the log, and log viewers have their own injection problems.

Related guides

• Secure Coding Practices
• API Security Guide
• SQL Injection Explained
• Command Injection Explained
• Path Traversal Explained
• XSS Attacks Explained
• OWASP Top 10 Explained