Hash Functions Explained: From Checksums to Cryptographic Primitives

Most explanations of hashing stop at "it turns data into a short string". That is the what, not the how. If you want to understand why length-extension attacks exist, why SHA-3 looks nothing like SHA-2, and why HMAC is built the way it is, you need to look at the machinery. This article goes one level down.

The compression function is the engine

Every cryptographic hash is built from a smaller piece called a compression function. It takes two fixed-size inputs — the current chaining value and the next block of your message — and returns a new chaining value of the same size. The hash just feeds your message through it block by block:

CV0 = initial constant
CV1 = compress(CV0, block1)
CV2 = compress(CV1, block2)
...
digest = CVn

If the compression function behaves like a random mapping, then the whole hash inherits its one-wayness and avalanche behaviour. The art of designing a hash is almost entirely the art of designing this one function.

Merkle-Damgard and the length-extension trap

SHA-1, SHA-256 and SHA-512 all use the Merkle–Damgård construction. Your message is padded to a multiple of the block size, then fed through the compression function as described above. The final chaining value is the digest.

This design has a known weakness. The digest is literally the internal state after processing the last block. If I know H(m) and the length of m, I can start the compression function from that exact state and keep going — computing H(m || padding || extra) without ever knowing m. That is the length-extension attack.

It does not let me recover m, but it does let me forge valid-looking digests for extended messages. This is why hash(secret || message) is a broken way to authenticate a message, and why HMAC exists (more below).

The sponge construction: SHA-3's answer

SHA-3 (Keccak) threw out Merkle–Damgård and uses a sponge construction instead. Picture a fixed-size state that you "absorb" your message into block by block, then "squeeze" the digest out of. Because the output is squeezed from a state that is never directly equal to an internal mid-stream value, length-extension does not apply.

The sponge also makes SHA-3 flexible: you can squeeze as many bits as you want, which is handy for things like SHAKE128/SHAKE256 where the output length is yours to choose. It is not "better" than SHA-2 in a simple sense — both are secure — but it is structurally different, which is good for the ecosystem. If one family is ever weakened, we are not left with nothing.

The avalanche effect, made concrete

Avalanche is the requirement that flipping one input bit flips about half the output bits. It is not magic; it falls out of how the compression function mixes bits through many rounds of nonlinear operations. The intuition: each round spreads every input bit a little further, and after enough rounds every output bit depends on every input bit. A hash where one input change only toggles a few output bits would be trivially differentiable from random and would fail the design.

You can feel it without math: hash "cat" and "cats", then diff the two digests. They share essentially no characters, even though the inputs differ by a single letter.

Birthday paradox and the 2^(n/2) bound

How hard is it to find a collision? Naively you might think you need to try 2^n hashes for an n-bit digest. You do not, because of the birthday paradox.

The same reason a room of 23 people probably shares a birthday applies to hashes: with about sqrt(2^n) = 2^(n/2) random inputs, you have a decent chance of two colliding. So a 256-bit hash gives you roughly 128 bits of collision resistance, not 256. A 128-bit hash like MD5 gives only about 64 bits — and 64 bits of work is within reach of motivated attackers, which is one more reason MD5 is finished for security use. This is also why SHA-1's 160 bits (about 80 bits of collision resistance) was considered marginal long before SHAttered made it practical.

Non-crypto hashes have a different job

Not every hash needs to fight an adversary. CRC32 and xxHash are designed for speed and for catching accidental changes or distributing items into buckets. They are deliberately not collision-resistant against a clever attacker — and that is fine, because nobody is supposed to attack them. Using CRC32 to detect a flipped bit on a network packet is perfect; using it to verify a software download is not.

Why HMAC looks the way it does

To authenticate a message with a shared secret, you cannot just do hash(key || message) — that is vulnerable to the length-extension attack we covered. HMAC fixes this with a construction that hashes twice:

HMAC(K, m) = H( (K ⊕ opad) ‖ H( (K ⊕ ipad) ‖ m ) )

Breaking it down: the inner hash mixes the key (XORed with a fixed pad ipad) into the message first. The outer hash then mixes the key again (XORed with a different pad opad) into that result. The two pads and the two passes mean an attacker who only sees the output cannot use length-extension to forge a message, because they would need to know the key to compute either inner or outer step. The double hashing also resists certain weaknesses that might show up in the underlying compression function alone.

If you are authenticating requests between services, reach for HMAC-SHA256 and a managed library — do not assemble it by hand from a raw hash, no matter how tempting the one-liner looks.

Choosing a primitive from this pile

The mental model that ties it together: use a non-crypto hash (CRC32, xxHash) when nobody is attacking you and you just want speed; use SHA-256/SHA-512/BLAKE3 when you need collision resistance; use SHA-3 when you want a structurally different backup to SHA-2; and wrap any secret-keyed use in HMAC rather than bolting the key onto a raw hash. Most "which hash" confusion disappears once you have named the adversary.

For experimenting with these algorithms without installing anything, the hash generator computes SHA-256, SHA-512 and BLAKE3 right in your browser, which is also the safest place to paste anything sensitive.