Folders you should never scan for duplicates
A duplicate finder will happily tell you that a folder contains four hundred identical files, and be completely right. What it cannot tell you is that those four hundred copies are load-bearing — that a program put them there deliberately, and that removing them breaks the program. This is the part of disk cleaning where the tool stops helping and your judgement takes over.
Repetition is sometimes the design
The intuition behind cleaning duplicates is that a repeated file is an accident: you saved the same photo twice, you copied a folder and forgot. For your own documents that intuition is right almost every time.
For files a program created, it is often wrong. Whole categories of software repeat files on purpose, because for them a local copy is cheaper, safer or more predictable than a shared one. The copies are not waste. They are the mechanism.
Here is the practical rule, and it will keep you out of nearly all trouble: clean folders you created; leave alone folders a program created.
The list
Program and system folders
On Windows: C:\Program Files, C:\Program Files (x86), C:\Windows and everything under AppData. On macOS: /Applications, /System, /Library and ~/Library. On Linux: /usr, /etc, /var and the dot-folders in your home directory.
These are full of legitimate duplicates. The same runtime library ships inside a dozen different applications, because each one wants a version it controls rather than one a system update might change underneath it. Delete the "extra" copies and you get an application that launches to an error about a missing component, with no obvious connection to what you did.
Browsers know this, which is why a web-based tool cannot open most of them at all: Chromium keeps a hard-coded list of directories the File System Access API refuses to hand over, and Program Files, Windows and AppData are on it. That block is not a limitation to work around. It is the same judgement, enforced.
Developer project folders
If you write code, node_modules is the most spectacular duplicate generator ever built. A single mid-sized JavaScript project can hold tens of thousands of files, many byte-identical across sub-dependencies, because each package is allowed its own version of what it depends on. A duplicate finder pointed at a projects folder will report a colossal amount of recoverable space and every byte of it is a trap.
Same story for .git, vendor, venv, __pycache__ and target. In .git particularly, the repeated objects are your version history. There is a correct way to shrink those folders — delete and reinstall the dependencies, run the version-control system's own cleanup command — and it is never "remove the duplicates by hand".
Photo library packages
This one catches careful people, because it looks exactly like the case you are trying to solve.
Apple Photos, and equivalents on other systems, store your pictures inside a package with a rigid internal structure: originals in one place, edited versions in another, thumbnails in several sizes, and a database that ties them together. An unedited photo can legitimately exist as three or four files that are identical or nearly so.
Move any of them and the database now points at nothing. The application does not fail loudly — it shows you a grey rectangle where a photo used to be, and the original is in a folder you will not think to look in. Clean your photos through the photo application, never underneath it.
Cloud sync folders
Dropbox, OneDrive, Google Drive and iCloud Drive share one dangerous property: whatever you do locally, they do everywhere. Move a hundred files into a duplicates folder and the sync client faithfully reproduces that on every device and in the web account.
This is survivable — all of them keep deleted files for a while — but recovering a hundred files one at a time through a web interface is a bad afternoon. If you clean inside a sync folder, pause syncing first, review what got moved, and only then let it run.
Backups
A backup is duplicates by definition. Time Machine, File History, and any folder you named backup are supposed to contain another copy of things you already have. Pointing a duplicate finder at one is asking a question whose answer is "yes, all of it", and acting on that answer converts your backup into not-a-backup.
The one that surprises people: Downloads
Downloads is the folder most in need of cleaning. It is where the same installer accumulates in four versions, where invoice.pdf, invoice(1).pdf and invoice(2).pdf live together.
And it is the folder browsers will not let a web tool open. On desktop as well as on mobile. Chromium blocks it as a selection; on Android, the system blocks the Download directory root outright.
The reasoning is defensive rather than about duplicates: Downloads is where a freshly downloaded file lands, and granting a web page read-write access to that directory would let one page reach the file another page just saved.
The workaround is straightforward. Create a subfolder inside Downloads — call it review — move what you want to check into it, and select that subfolder. The block covers the folder itself, not what is inside it.
Where to point it instead
The folders worth scanning are the ones where you put things, without a program mediating:
- Photos exported out of your phone — the single biggest source of real duplicates, especially anything that went through a messaging app.
- Documents, where
report_final_v2_REAL.docxlives. - The desktop, and any folder called
to sort,miscorstuff. - External drives and memory cards, which usually hold several partial copies of the same phone backup.
Two habits worth having
Sort by wasted space, not by number of files. A hundred duplicated icons of 2 KB are irrelevant. Three copies of a video are half a gigabyte. Fixing the biggest groups first means most of the benefit arrives in the first few decisions, while you are still paying attention.
Never let a tool delete. The right behaviour is to move copies into a folder you can inspect, leaving you to delete once you have looked. A finder that deletes for you is a finder betting its judgement against yours on your own files — and it has never seen them before.
That is why this tool moves rather than deletes, and writes a README.txt recording where each file came from. If something looks wrong, the note tells you exactly where to put it back.
In short
Duplicates in your own folders are usually a mistake worth fixing. Duplicates in a program's folders are usually a decision someone made for a reason you cannot see from the outside. The whole skill is telling the two apart, and the shortcut is asking who created the folder.
When you are ready: check a folder for duplicates in your browser, with nothing installed and no files leaving your computer. Related: how to find duplicate files without installing anything and exact duplicates and visual duplicates.