File check across MILLIONS of files/folders

Forums › IT Pro and developers › File check across MILLIONS of files/folders

Aaroona

2831 posts

Uber Geek
+1 received by user: 71

Topic # 142589 18-Mar-2014 09:51

We're running a project at work, part of this requires us to copy over 12 million files and the same if not more in folders, over to another machine.

We want to do verification that all the files are there,i.e. compare the prod system to the new system we are implementing.

I've tried using an MD5 checker, which eventually bombed out because there are just so many files and folders.

I'm trying a method I wrote in powershell to check that all the files are the same and exist (note, not MD5 check or anything), we are just wanting to find out where/if we are missing gaps, but I've had it running for over 24 hours and it's only scanned 3.5 days of each year, across 8 years worth of data. so it's going to take AGGGGES.

Md5 would be nice, but it's the highest priority at this stage, but would like it to come later.

Any ideas on what I may be able to do?

1 | 2

Fully Operational
3343 posts

Uber Geek
+1 received by user: 1088

Trusted

Vocus

Subscriber

Reply # 1007902 18-Mar-2014 09:55

Can you do a dump of filename, size, and MD5 on each machine, then compare them? May be quicker this way than doing the whole thing from one end.

Zeon

3367 posts

Uber Geek
+1 received by user: 383

Trusted

Reply # 1007927 18-Mar-2014 10:33

One person supports this post

Are you sure a filesystem is the best way to store that many files? Object orientated would be better?

Aaroona

2831 posts

Uber Geek
+1 received by user: 71

Reply # 1007930 18-Mar-2014 10:37

Hi guys,

ubergeeknz: Can you do a dump of filename, size, and MD5 on each machine, then compare them? May be quicker this way than doing the whole thing from one end.

The powershell script I wrote just pulls the relative file/folder path in plain text and puts it in a text file. Unfortunately the paths can be quite long, even relative, so this isn't working quite as well as I had hoped.
Adding file size and MD5 would probably kill it faster :)

Zeon: Are you sure a filesystem is the best way to store that many files? Object orientated would be better?

You're dead right, but it's not my choice. It's a technical limitation of the software the files are for. The developers decided that was the way to do it when they developed it. So this is what we are stuck with.

ubergeeknz

Fully Operational
3343 posts

Uber Geek
+1 received by user: 1088

Trusted

Vocus

Subscriber

Reply # 1007936 18-Mar-2014 10:42

Surely regardless of the number of files it is just a matter of time to process them all? What kind of limitation are you running into?

Amosnz

367 posts

Ultimate Geek
+1 received by user: 75

Subscriber

Reply # 1007938 18-Mar-2014 10:43

I use Beyond Compare quite often to compare specific files, but it had a file\folder match too.
Not sure how it would handle that many files (what the limitations of the trial are) but might be worth a shot?

http://www.scootersoftware.com/moreinfo.php

Aaroona

2831 posts

Uber Geek
+1 received by user: 71

Reply # 1007942 18-Mar-2014 10:44

ubergeeknz: Surely regardless of the number of files it is just a matter of time to process them all? What kind of limitation are you running into?

EDIT: Sorry, just saw you might be speaking about the actual software its for. It's for a call recording suite called Verint. Unfortunately we are on an old version, so they may have changed it in later versions on how they store files, but for our version it's just files in a folder structure, then a database for lookup if I remember correctly for the web-end, which contains location data to where the file is located.

Using an Md5 tool, the limitation I ran into was that the tool crashed, which I presume ony happened because there is such a large structure of files and folders.

Using the powershell method, its made a text file over 500MB now, and still counting; not the easiest to process.
I may just refine my powershell script to split it up into files into 100MB lots, if theres no other way of doing it.

gzt

9093 posts

Uber Geek
+1 received by user: 1252

Reply # 1008013 18-Mar-2014 12:08

Have a look at your disk and cpu while the process is running. It is likely both are under-utilised. Simple solution. Start multiple instances of your md5 process aiming them at different directories whatever. Repeat until you reach a utilisation limitation above. Network might be included if different box.

gzt

9093 posts

Uber Geek
+1 received by user: 1252

Reply # 1008016 18-Mar-2014 12:13

There are also free and open source that will assist in your aim.

charsleysa

596 posts

Ultimate Geek
+1 received by user: 132

Reply # 1008100 18-Mar-2014 14:06

Sounds like you should get a programmer to build you a multithreaded utility in C# that can utilize all CPU's and hardware threads.

It's very easy to do.

Regards
Stefan Andres Charsley

killerkiwinz

39 posts

Geek
+1 received by user: 3

Reply # 1008105 18-Mar-2014 14:17

charsleysa: Sounds like you should get a programmer to build you a multithreaded utility in C# that can utilize all CPU's and hardware threads.

It's very easy to do.

Yeah very easy, you should add a column to the DB with a hash as well so you can verify the files in the future nothing worse than not being able to tell there is 1 file missing or corrupt

Also you could use rsync to copy the files instead

timbosan

1333 posts

Uber Geek
+1 received by user: 118

Reply # 1008112 18-Mar-2014 14:24

Aaroona: We're running a project at work, part of this requires us to copy over 12 million files and the same if not more in folders, over to another machine.

We want to do verification that all the files are there,i.e. compare the prod system to the new system we are implementing.

So if I understand, you want to copy files from Machine A to Machine B, verifying them at the same time? And only copy where the files don't already exist?

You can just use XCOPY from the command line in Windows, and use /D to only copy new files, and /V to verify the files against the source. To get from one machine to another just map a network drive, using NET USE. No need to write a program. You can even redirect the results into a file:

XCOPY [source] [target] /D /S /Y /V > logfile.txt

ubergeeknz

Fully Operational
3343 posts

Uber Geek
+1 received by user: 1088

Trusted

Vocus

Subscriber

Reply # 1008114 18-Mar-2014 14:28

Aaroona: Using the powershell method, its made a text file over 500MB now, and still counting; not the easiest to process.

Provided you expect perfectly consistent output from both A and B servers, just hash the resulting file on each server :)

frankv

2034 posts

Uber Geek
+1 received by user: 975

Reply # 1008135 18-Mar-2014 14:59

I have 500K files that I need to verify periodically. I use FreeFileSync. It works well, takes about half an hour, and has a bunch of options. It's also cross-platform.

JWR

730 posts

Ultimate Geek
+1 received by user: 236

Reply # 1008152 18-Mar-2014 15:26

I use this for copying and syncing large numbers of files between locations....

http://sourceforge.net/projects/freefilesync/

gundar

488 posts

Ultimate Geek
+1 received by user: 80

Trusted

Reply # 1008194 18-Mar-2014 16:16

http://arcainsula.co.nz/2013/copying-large-amounts-of-data-with-linux/

I wrote this blog to address that. Although it refers to use under Ubuntu, you can use the same instructions with Cygwin. The files are checksummed on the fly.

It is a stable long existing and supported means of doing that one task - copying large amounts of data, reliably.

1 | 2