quick way to determine folder size

[cyr0n_k0r]

This is not the run of the mill software request.

We have about 20TB on various storage arrays that hold hundreds of thousands of folders. Inside those folders are millions of tiny files in the range of 2KB or so.
This is encrypted file storage for offsite backup.

We run a program currently called "dirsize" that allows us to automate scanning all these folders and reporting to a text file how big all the folders are. (IE, 212GB, or 345GB, etc)

Problem is the program goes through and scans every file in every folder similar to how right click on a folder in windows does. However with so much data now the scans are taking almost 24 hours of constant scanning.

Is there an easier way that is automated? We don't need to be SUPER accurate. I mean. anything that would get us within plus or minus 5% would be acceptable.
It must be able to be automated or scripted, and must output the results to a text file or spreadsheet of some kind.

Does anyone know of a program that will fit our needs?

 

[l3ender]

I can't think of one...because I don't know how it would be possible for such to exist. What other way is there to determine folder size than to scan what files are in the folder?

Don't read my post as condescending (because that's not how I'm trying to come across), but I don't know of such software and I don't know how such software would be possible.

If there is something, I'd be very interested in it and how it works.

 

[leSLIe]

omg!! 20TB wow that´s a lot of pr0n...erg... i mean a lot of "data" you got there
what OS are u running?

 

[Joe Average]

That many files on such a large storage subsystem is just always going to take a lot of time. My god, I can't even imagine that... 20TB... holy crap.

I'm sure there's got to be something out there that can maintain quota information and statistics in some automated fashion, I just can't tell you where you'd find it.

 

[cyr0n_k0r]

While I do realize that to get an accurate scan of a folder the program must scan every file.

BUT

I am looking for something similar in function to how windows maintains a somewhat accurate account of how much free space is left on a hard drive when looking at it from My Computer. Obviously it doesn't sit there and scan every file and every folder constantly. Yet if you were to copy 20GB to that volume it would reflect that change in free space rather quickly.
It must use some kind of background process to keep a count of current space. I am looking for something like that. Some kind of program that can take a quick snapshot of a folder and get close, without having to scan the hundreds of thousands of files contained in each folder.

omg!! 20TB wow that´s a lot of pr0n...erg... i mean a lot of "data" you got there
what OS are u running?

Windows Server 2003 Enterprise. The file system is NTFS but running on GPT disks as standard NTFS volumes don't play well after 2TB

 

[Joe Average]

It's getting the information from the MFT and other system variables, of course. It doesn't constantly keep track of every single byte in moment to moment chunks, it updates as required when activity forces a change to the MFT and the data layout on the drives, so perhaps there's some method of hooking into that for statistical purposes in real-time, maybe.

I'm just free thinking there...

And the thing is, any application you can find or even command line utility is going to end up using the Windows routines to do the very same thing, hence they theoretically will be just as slow or lagged as doing it the more "normal" way.