How perceptual hashing finds near-duplicate images

Real image folders rarely contain perfect copies. The same picture turns up as a PNG and a JPEG, full size and as a thumbnail, with and without a small logo. Each is a different file, but to you it's the same image.

Perceptual hashing is the standard way to catch them. This guide uses the exact methods in our duplicate image finder, including where they differ from the textbook versions, so the tool's threshold numbers make sense.

Why SHA-256 only catches identical files

A cryptographic hash such as SHA-256 reads every byte of a file and produces a 256-bit value that changes completely if even one bit does. That's ideal for verifying a download and useless for spotting a resized copy: re-saving a JPEG at a slightly different quality changes thousands of bytes, so the two hashes share nothing.

Our tool still runs SHA-256 first. It removes files with identical bytes, such as the same download saved twice, before any visual comparison. It's fast and practically never wrong. Anything re-encoded, resized, cropped or with edited metadata gets past it, and that's where the perceptual hash comes in.

Average hash, step by step

The average hash (aHash) is the simplest perceptual hash. In its Average Hash + Hamming mode, our tool does this:

1. Shrink. Draw the image onto a small square canvas, 8×8 pixels by default. The aspect ratio is ignored, and all fine detail disappears, which is the point.
2. Convert to grayscale. Each pixel becomes one brightness value: 0.3 × red + 0.59 × green + 0.11 × blue.
3. Average. Compute the mean brightness of all the pixels.
4. Threshold. Each pixel becomes 1 if it's brighter than the mean and 0 otherwise. At 8×8, that gives a 64-bit fingerprint.

Here's a toy 4×4 example. The numbers are made up for illustration; real hashes use 8×8 or more.

Image A: brightness (0-255)     Bits (mean = 112.5)
200  190   60   50              1 1 0 0
210  180   70   40              1 1 0 0
120  110  100   90              1 0 0 0
 30   40  150  160              0 0 1 1

The 16 values add up to 1,800, so the mean is 112.5. Row by row, the hash is 1100 1100 1000 0011.

Now take a copy that was re-saved and brightened slightly. Every value rises by about 10, with a little compression noise on top:

Image B: brightened copy        Bits (mean = 122.6)
211  199   72   58              1 1 0 0
222  188   81   49              1 1 0 0
129  125  108  101              1 1 0 0
 38   52  158  171              0 0 1 1

Because the mean rises with the pixels, only one bit flips. The pixel that read 110 sat just under A's mean, and in B it reads 125 against a mean of 122.6. Pixels close to the average are the fragile part of any average hash.

Comparing hashes with Hamming distance

The Hamming distance between two hashes is the number of positions where the bits differ. A and B differ in 1 bit out of 16. Now flip image A left to right:

Image A, mirrored               Bits (mean = 112.5)
 50   60  190  200              0 0 1 1
 40   70  180  210              0 0 1 1
 90  100  110  120              0 0 0 1
160  150   40   30              1 1 0 0

Same pixels, same mean, but the hash is now 0011 0011 0001 1100: 14 of 16 bits differ from A. Two random 16-bit hashes differ in about 8, so to the hash the mirrored copy looks less like A than an unrelated image. That's the biggest blind spot of this family of methods.

In the tool, the default Hamming threshold of 5 means "no more than 5 of 64 bits differ" at 8×8. The count is absolute, so at 16 px (256 bits) the same 5 is far stricter.

DCT hashing: keeping only the low frequencies

The average hash works on pixels. A DCT-based hash, often called pHash, first rewrites the thumbnail as a mix of smooth cosine patterns using the discrete cosine transform, the transform JPEG uses. Each coefficient says how much of one pattern the image contains. The top-left one is the overall average. Its neighbors describe broad changes such as "brighter on the left than the right", and coefficients further from that corner describe finer and finer detail.

What usually happens to copies (resizing, recompression, sharpening, noise) mostly disturbs fine detail, the high frequencies. The broad, low-frequency layout survives almost untouched, so a fingerprint of only the lowest frequencies makes copies agree while different pictures usually don't.

The classic recipe

A widely used version is the phash function in the open-source imagehash Python library. It shrinks the image to 32×32 grayscale, runs a 2D DCT, and keeps the top-left 8×8 block of coefficients. Each bit records whether a coefficient is above the block's median, giving a 64-bit hash compared by Hamming distance.

What our tool does differently

Our DCT + Euclidean mode starts the same way, but keeps the actual numbers instead of turning them into bits:

1. Shrink to a square grayscale thumbnail, 32×32 by default (adjustable from 8 to 64).
2. Run a 2D DCT (an orthonormal DCT-II along the rows, then along the columns).
3. Keep the 8×8 block of lowest frequencies and drop the top-left average-brightness term, which leaves 63 coefficients.
4. Normalize those 63 values: subtract their mean, then divide by their standard deviation.
5. Compare two images by the Euclidean distance between their lists of 63 numbers.

Dropping the average term makes the hash ignore uniform brightness changes. Normalizing makes it ignore contrast changes, which scale every coefficient equally (clipping at pure black or white aside). And keeping real numbers avoids the bit-flipping fragility you saw with the 110 pixel above.

Resolution matters less in this mode than you might think. Each coefficient describes a pattern relative to the whole image, such as "one and a half cycles from left to right", so the 63 coefficients mean the same thing at 16, 32 or 64 px. A larger thumbnail just estimates them more cleanly.

Hamming vs Euclidean distance

Hamming distance counts disagreeing bits, so it only knows whether two values fell on the same side of a line. Euclidean distance is the straight-line distance between two lists of numbers (square each difference, add them up, take the square root), so it also knows how far apart the values are.

Our Euclidean modes normalize both lists to mean 0 and standard deviation 1 first, which gives the distance a neat meaning. For two lists of N numbers with correlation r (1 = same pattern, 0 = unrelated, −1 = opposite), it works out to exactly:

distance = √(2 × N × (1 − r))

DCT mode always compares N = 63 numbers, so every threshold maps to a minimum correlation:

Despite its name, the Average Hash + Euclidean mode doesn't make bits at all. It normalizes the grayscale thumbnail itself and compares that, so N is the number of pixels: 64 at 8 px, 256 at 16 px, 1,024 at 32 px. Its default threshold of 3 therefore asks for a correlation of about 0.93 at 8 px, 0.98 at 16 px and 0.996 at 32 px: in this mode, a higher resolution makes the same threshold much stricter. Switching method resets both sliders to that method's defaults.

A quick calibration run

To see real numbers, we re-implemented the DCT mode's steps in Python and ran them on 143 different photos from a desktop operating system's built-in wallpapers and screensavers, mostly aerial landscapes, plus scripted edits of each. Two caveats: this is one small set of photos, not illustrations, so your images will give different numbers; and our script resizes images slightly differently from a browser. Treat the figures as a guide to scale, not exact values.

Between different photos the median distance was 9.9. About 1% of pairs came in under 3.8, about 4.5% under 5, and about half within 10. We also tried a wallpaper design that ships in four colors with identical shapes: its color versions measured 0.2 to 1.5 apart, because the hash only sees brightness.

Resizing, recompression and brightness changes barely register. Crops and rotation climb steadily, and a 15% crop drifts into the range of unrelated pictures. Mirrored copies sit among the unrelated photos, so no threshold catches them reliably. A threshold of 10 would have grouped half of all unrelated pairs. That's why the tool defaults to 3, where under 1% of unrelated pairs matched.

Choosing a threshold

A threshold trades missed copies against false groups. You review every group before saving, so a false group only costs time, while a missed copy stays in your set unnoticed. Very loose settings, though, create huge mixed groups that are slow to check. A practical approach:

1. Start at the default of 3 in DCT mode. It catches re-saves and resized copies with few false matches.
2. To catch cropped or edited copies as well, raise it one step at a time, press Reprocess and look at the new groups.
3. Stop when new groups start mixing different pictures. In our test set that began around 4 to 5.

If your images share a strong common layout, such as characters centered on plain white backgrounds, even 3 may group different pictures. Drop to 2 in that case.

Failure modes to know about

• Mirroring and rotation. A left-right flip reverses the sign of 32 of the 63 coefficients (those with an odd horizontal frequency). Unless the picture is fairly symmetric, or its main structure runs side to side like a horizon, the match is lost. Rotating by 90° or 180° is worse.
• Crops, borders and padding. Anything that moves or rescales content within the frame shifts the low frequencies. Padding a wide image to a square is often enough to break a match.
• Color. All three methods work in grayscale, so a recolored or grayscale copy looks the same to them.
• Inversion. A negative, or a design's dark version with light and dark swapped, has a correlation near −1 with the original: the maximum distance.
• Small local changes. A change in a small area, such as a different expression on the same drawing or a small signature, hardly moves the low frequencies. That's handy for finding alternates, but the hash can't tell them from true copies.
• Meaning. A perceptual hash knows nothing about content. Two drawings of the same character are just different images.

Since the hash ignores uniform brightness and contrast changes, a white balance or histogram correction usually moves it only a little. Still, deduplicate before you run the batch color fixer, so you don't correct copies you're about to delete. Our dataset cleaning workflow shows where each step fits.