description | Tool for scanning for duplicate files, quickly |
homepage URL | http://muldersoft.com/ |
owner | mulder2@gmx.de |
last change | Sat, 22 Apr 2017 18:28:01 +0000 (22 20:28 +0200) |
URL | git://repo.or.cz/double-file-scanner.git |
https://repo.or.cz/double-file-scanner.git | |
push URL | ssh://repo.or.cz/double-file-scanner.git |
https://repo.or.cz/double-file-scanner.git (learn more) | |
bundle info | double-file-scanner.git downloadable bundles |
content tags |
The purpose of this tool is scanning the selected directory or directories for duplicate files, i.e. files with identical content. Duplicate files are identified by first calculating the SHA-1 digest of each file and then looking for values that appear more than once. In particular, files with identical content are guaranteed to have the same SHA-1 digest, while files with differing content will have different SHA-1 values with very high certainty.
All computed SHA-1 values are stored in a hash table, so collisions are found quickly and we do NOT need to compare every digest to every other one. Also, the files are processed concurrently in multiple "worker" threads in order to parallelize and speed-up the SHA-1 computations on multi-core processors. On our test machine it took ~15 minutes to analyse all the ~260,000 files on the system drive (~63.5 GB). During this operation ~44,000 duplicates were found.
Once the scan is completed, the program provides commands to review, rename or delete the duplicate files manually. Alternatively, the "automatic clean-up" wizard may be invoked in order to remove ALL redundant files at a single blow. Optionally, the list of duplicates can be exported to the XML or INI format.
7 years ago | v2.04 | Version 2.04 is released! | tag | commitlog |
10 years ago | v2.03 | Version 2.03 is released! | tag | commitlog |
10 years ago | v2.02 | Version 2.02 is released! | tag | commitlog |
10 years ago | v2.01 | Version 2.01 is released! | tag | commitlog |
10 years ago | v2.00 | Version 2.00 is released! | tag | commitlog |
7 years ago | master | logtree |