Recommended duplicate file finder? (binary file compare)

A

AbRASiON

Limp Gawd
Joined
Jul 28, 2004
Messages
354
Does anyone have a recommended application to scan multiple drives and find duplicate files?
Bonus points for something which presents the data in a way which shows entire folders which are the same.

I've used DFL in the past (DOS) and duplic8 for about 10 years but I need something more powerful.
 
md5sum and some scripts should be enough.
Depending on how specific you want it, the scripts are probably easy to write or find via google.
 
I don't have any binary comparison tools, unfortunately. I have to use home grown stuff from MS Office since my work doesn't let me load executables other than an approved list of applications. As a result, I had to make the tools I need to manage our drives out of some VBA code and macros.

Currently, I scan around 300 shared drives containing about 4 million folders and 11 million files, and I actually do it all from MS Access.

For now, I identify duplicate files based on file size & modified date & filename are all the same (though they may be stored in different locations)

That's not quite what you were looking for, unfortunately.

It does a lot more than just identify dupes though...
 
Last edited:
jay2472000 said:
I use Doublekiller, check it out http://www.bigbangenterprises.de/en/doublekiller/
Click to expand...
But that tool does not perform a binary compare, which is what the OP needs:
"...by comparing any combination of file name, size, modification date and content."

Using tools that just go off the name, size and modify date are dangerous to use if you want to delete files based on their result!

Google is your friend ;)
[ame="http://www.google.com/search?q=binary+file+compare+tool&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:eek:fficial&client=firefox-a"]binary file compare tool[/ame]


-TLB
 
The more powerful the better, we're talking about possibly thousands of files across multiple directories :(
 
I use Duplicate Cleaner, it has Binary compare, among a bunch of other options. It utilizes .NET Framework, I find it pretty robust as I often scan hundreds of thousands of files with it and it handles it very well. You can then apply file name or directory filters to narrow it down, etc.
 
If u want i can make a small .net utility which can do MD5 check sum and move all duplicate files in a given directory (or delete).
 
TheLastBoyscout said:
But that tool does not perform a binary compare, which is what the OP needs:
"...by comparing any combination of file name, size, modification date and content."

Using tools that just go off the name, size and modify date are dangerous to use if you want to delete files based on their result!
Click to expand...

Of course doublekiller performs binary comparison, what do you think "content" means ? In fact there are options, you can do bit by bit, CRC32, compare only some parts, exclude the beginning or the end, etc. And it isn't afraid by 20GB files like other tools are.

Date, name, size, CRC32, etc., all are options, filters if you wish. Usually I use size + CRC32, but for some applications (when I know there are likely dupes, and they are huge files that take forever to hash) date+name+size is good enough.
 
You must log in or register to reply here.
Back
Top