Exact duplicates and visual duplicates: what each one actually means
Two photos look identical on your screen. A duplicate finder calls one pair "the same file" and the other "the same image". That distinction sounds like jargon, and it is the single most useful thing to understand before you delete anything — because one of the two can be cleared without looking, and the other absolutely cannot.
An exact duplicate is a claim about bytes
Every file is a sequence of numbers. Two files are exact duplicates when those sequences are identical: same length, same value in every position. Not "practically the same". Identical.
Proving it by comparing byte by byte would mean reading both files in full, for every possible pair — unworkable on a folder of thousands. So the work is done with a fingerprint: each file is run through SHA-256, an algorithm standardised by NIST that turns any amount of data into a fixed 64-character value.
Two properties make it useful here. Change a single bit anywhere in the file and the fingerprint changes completely — not slightly, completely. And finding two different files that produce the same fingerprint is, as far as anyone knows, computationally infeasible. So in practice, same fingerprint means same file.
What this survives is everything that is not the content:
- Renaming it.
IMG_2043.jpgandbeach.jpghave the same fingerprint if the bytes match. - Moving it to another folder, another drive, another computer.
- A different creation or modification date.
And here is what breaks it, which surprises people: editing the metadata rewrites the file. Add a tag or a caption in Windows Explorer or in Photos, and the pixels do not change but the bytes do. That copy stops being an exact duplicate of the original, even though the two images are indistinguishable. The fingerprint does not care about the picture — it cares about the file.
A visual duplicate is a claim about the picture
The other case is the common one with photos: the same image living in files that have nothing in common at byte level.
They come from ordinary places. A photo sent through WhatsApp comes back re-compressed and smaller. You exported the same shot twice at different quality. You converted a PNG to JPEG. Someone screenshotted your photo instead of saving it. In every case the picture is the same and the file is new.
No fingerprint will ever connect those two, and that is not a flaw — it is the fingerprint doing exactly what it was designed for. Catching them needs a completely different kind of comparison, one that looks at appearance rather than content.
How appearance gets measured
The technique is called perceptual hashing, and the idea flips cryptographic hashing on its head. Ofcom's technical overview of the field puts the contrast plainly: a cryptographic hash relies on the avalanche effect, where a tiny change produces a completely different output, while perceptual hashes are built so that "very similar input files" produce "very similar hashes".
The practical recipe is simpler than it sounds. Shrink the image down to a tiny grid — a few dozen pixels a side. Strip the colour, leaving grey levels. What remains is the coarse structure of the picture: where it is light, where it is dark, roughly where the shapes sit. Fine detail is gone, and so is everything a resize or a re-compression would have altered.
Then you compare two of those grids. Do it well and a photo recognises itself through a resize, a quality drop or a format change, because none of those meaningfully move the coarse structure.
Why the percentage is a measurement, not an opinion
When a tool shows you "97%", that number is not an estimate of how similar the two look to a human. It is the result of a specific calculation: each image is reduced to a 32×32 grid of grey values, and the two grids are compared point by point, normalised by their own contrast.
That normalisation is what makes it robust. Because the comparison is measured relative to each image's own range of light and dark, a photo that is uniformly brighter still scores near the top — the pattern is unchanged even though every value shifted. And it collapses fast when the content genuinely differs, because then the pattern itself is different.
Read the numbers like this: 100% with "same file" means byte-identical, no judgement involved. 99% is the same photo through a resize or a re-compression. Below about 94% you are into territory where two different photos of the same scene can score just as high as two versions of one photo.
Where visual comparison fails, and why
Ofcom's overview states the limitation with unusual clarity: perceptual hashing "assesses the similarity of the images, not of the content depicted in the images". Two visually similar pictures of different objects can match; two photos of the same object from different angles may not. The document also notes that these functions are "imperfect", with false positive and false negative rates that are "non-negligible".
The everyday version of that failure is a folder of screenshots. Shrink a screenshot to a tiny grey grid and what is left is: bright background, a darker band at the top, thin text-like texture in the middle. Every screenshot reduces to almost the same pattern, because what distinguishes them — the actual words — is precisely the fine detail that shrinking destroys.
This is the failure that ruins duplicate finders. A tool that trusts a perceptual hash on its own will happily present you a group of "115 identical images" that are 115 different screenshots. If you had trusted it, you would have deleted 114 files with nothing in common. It is worth knowing this exists before you let any tool — this one included — make that decision for you.
The defence is to never let the perceptual hash decide. Used properly it is a shortlisting step: it proposes candidate pairs quickly, and every pair it proposes is then verified by the real, more expensive comparison. Two extra guards help: images with different proportions should not be grouped, because a crop is not a duplicate; and images with too little detail — a plain background, a mostly white document — should only be grouped when they are literally the same file.
Which is safe to delete
This is where the distinction stops being academic.
Exact duplicates carry no risk. If two files have the same fingerprint, keeping one loses nothing at all — not a pixel, not a metadata field, not a byte. There is no version that is "the better one", because there is no difference. This is the free part of the cleanup, and it is where the space is anyway.
Visual duplicates need your eyes. The two files are genuinely different, and the differences may matter:
- Resolution. One may be 4000 pixels wide and the other 800. They look the same in a thumbnail and are not the same if you ever print or crop.
- Metadata. The version that came back from a chat has usually had its date, camera and location stripped. If you delete the original and keep that one, that information is gone for good.
- Which one is edited. The visually near-identical pair may be the raw shot and the one you straightened and colour-corrected.
So the working rule is: clear the exact matches without thinking, and look at the visual ones before deciding. Sort by size within the group — the largest is usually the least degraded — and keep the one in the folder you actually curate.
Why other file types only get exact matching
You may notice that video, audio and documents are only ever reported as exact duplicates. That is not laziness, it is a cost problem.
The trick that works for photos — shrink to a tiny grid and compare — has an equivalent for video, but it means decoding and comparing frames across every candidate pair. On files that are gigabytes each, that is minutes of work per comparison. For audio it means acoustic fingerprinting, which is a different technology entirely. For documents, "the same text in a different PDF" is a semantic question, not a visual one: two PDFs with identical wording produced by different programs are genuinely different files, and treating them as duplicates would be a guess.
Exact matching, on the other hand, works perfectly on all of them, because bytes are bytes. And for the file types that actually fill your disk, exact copies are where the space is.
In short
An exact duplicate is a fact about the file: identical bytes, proven by a fingerprint, safe to clear without looking. A visual duplicate is a measurement about the picture: the same image in a different file, useful to know about and never something to delete blind.
A finder that blurs the two is a finder you should not trust with your photos. One that keeps them separate is telling you which decisions it has made for you and which ones are still yours.
You can see both kinds labelled separately in your own folder right now: check a folder for duplicates in this browser, with nothing installed and no files leaving your computer. The rest of the series: how to find duplicate files without installing anything and why you have three copies of the same video.
Sources
- Ofcom — Overview of Perceptual Hashing Technology (2022): the contrast with cryptographic hashing, the "similarity of the images, not of the content depicted" limitation, and non-negligible error rates.
- NIST — FIPS 180-4, Secure Hash Standard: the specification SHA-256 is defined by.