Find duplicates of files (images and videos)

[KuJaX]

I am organizing family photos and videos into one giant source that will be backed up in the cloud and external hard drive. I am gathering this from multiple computers, phones, etc.

I know that there will be duplicates, potentially hundreds of duplicates between hundreds of thousands of files (mainly photos and videos).

What software, CLI (powershell or linux) can be used to target a directory and recursively scan all of the files and acknowledge what ones are duplicates and move or delete the duplicates? I don't need to go as far as MD5 or SHA or anything but really just file name

Any help is appreciated.

 

[muz_j]

I'm not sure from your wording if you're asking specifically for CLI based software or scripts, but on windows OS'es - I personally use a freeware product called "All Dup" for de-duping file systems. I find it's really good - easy to use, free and the portable version doesn't need to be installed to run it.

You can find it here if you're interested: http://www.alldup.info/
Before you run your first scan, check the comparison criteria for files. For general use, I use filenames and file sizes, that way it includes both in the match criteria, but there are other options like extension, date attributes and image specific comparison options that I haven't personally used.
I would expect that the more options you include - particularly image comparison ones, it might take longer to complete - but like I said above, I haven't used those features to date, as typically I use it as a general file scanner - rather than specifically for images.

If you specifically want a script - I'd personally go with powershell and the MS Script Repository is a good place to start (use keyword searches etc). https://gallery.technet.microsoft.com/scriptcenter/

 

[bigddybn]

Most photo management software will do this along with other useful features as well. It may be time to move beyond just dropping everything in a folder.