Corrupted Files

[kekzmobile]

Hey,

I got a kinda strange problem.
It all started after i canceled a cluster resize progress on a 32TB NTFS partition (not the brightest choice) since there wasn't any progress for about three days and the storage (12x4TB Raid6) did not show any kind of load.
After that i noticed several corrupted files and many video-files with stuttering and heavy artifacts in it.
With checkdisk i sadly could not solve the problem so i decided to re-partition and format the drive since i got the most important files on several LTO tapes.
I even deleted the Raid-array, created a new one and formatted the drive with 32K sector size NTFS.

After restoring files from LTO tape to the new partition the same data corruption happened again. Every video file that i restored is stuttering with lots of artifacts.
I copied it from the array to another hard drive, playing it from there to check if the problem maybe beacause of the network connection. But the video files are still corrupt.

BUT, if i restore them from LTO tape to another drive than the Raid-array there is no data corruption, no stuttering, no artifacts in video files at all.
It seems that all new files on the array are damaged but i don't know how this is even possible because the array and the partition was deleted and newly created.

Really don't know what to do.


Hardware:
Xeon E3-1240v6
Asus PS10S-E/4L
32GB DDR4 ECC
HP 10Gbe 2 Port SFP+
LSI MR 9260-8i with Intel SAS Expander (12x4TB Seagate IronWolf) at Raid6
Host is running Proxmox, Raid-Controller is in Windows Server 2016 VM
IBM LTO4 SAS Tape Drive in Dell 124T Autoloader (LSI SAS HBA)

 

[kekzmobile]

I know you're probably way ahead of me, but any overclocks? How is system stability overall? What if you copy to "known good" drive and play back files on this system, do they play back n9rmal?

No overclocking and stability is great. The files play back normal when i restore them to another drive but if they are restored to the array they show stuttering and artifacts

 

[mwroobel]

Since I see you have a 40TB volume after parity, I expect you started with 16k clusters. If you reformat the NTFS partition with 16k clusters does the issue resolve? What is your stripe size for your RAID6?

 

[kekzmobile]

Since I see you have a 40TB volume after parity, I expect you started with 16k clusters. If you reformat the NTFS partition with 16k clusters does the issue resolve? What is your stripe size for your RAID6?

The stripe size is 1MB due to mainly large files. At the moment checkdisk is running and found errors in volumebitmap. So even the partition is new and fresh it seems damaged.
I will try formatting the disc with 16K clusters after the process is finished.

 

[kekzmobile]

If you reformat the NTFS partition with 16k clusters does the issue resolve?

Just tried that. No difference. Once the file is stored on the array it is corrupt. I even tried copying a known-good working file from my workstation to the array. Once it is stored there it seems broken.

Maybe a controller issue?

 

[kekzmobile]

Are your drives healthy?

According to the LSI controller there are no detected errors.
I can check using SeaTools although i have to delete the raid6 volume for that again. But i think it's worth a try

 

[pendragon1]

According to the LSI controller there are no detected errors.
I can check using SeaTools although i have to delete the raid6 volume for that again. But i think it's worth a try

yup it will be a PITA but test em all.

 

[likeman]

if it's RAID 5-6 it should be detecting the data errors (unless the controller thinks its getting valid data when its transferring from the tape)

what happens if you copy the files that you restore from tape to the HDD and then copy them over to the RAID6 array

very likely the RAID controller is faulty (but it looks like your Using Software raid6 how are you making the RAID6 Array? or bad ram (but it's ECC that should've crashed the PC or check event logs for ECC correction events in the Microsoft-Windows-Kernel-WHEA spot)

if you can change the Slot that the RAID card is in

if you install 7Zip you can do SHA1 on a large file you know is working on the HDD before you copy it to the array (right click on large file CRC SHA) or use TeraCopy as it can do a CRC after it has copied the file (it can save the crc as well)

 

[trackersoft]

To me it sounds like you are observing some kind of silent data corruption - perhaps raid controller cache or another component, related to raid controller.
This is why always should be in use consistency check mechanism (e.g. like in ZFS/BTRFS).

 

[kekzmobile]

what happens if you copy the files that you restore from tape to the HDD and then copy them over to the RAID6 array

Then i works as long as its not on the array. Once the file touches the array it goes bad. This happens when either restoring directly to the array or when restoring to a different HDD or PC and then copying the file to the array. Same result.

very likely the RAID controller is faulty (but it looks like your Using Software raid6 how are you making the RAID6 Array?

Its a hardware raid6 created with megaraid storage manager.

if you install 7Zip you can do SHA1 on a large file you know is working on the HDD before you copy it to the array (right click on large file CRC SHA) or use TeraCopy as it can do a CRC after it has copied the file (it can save the crc as well)

I will try that next.

I just updated the controller firmware to a slightly newer version i found (12.15.0.-0248 build 05/2016) and a new driver build 10/2018.
All in an freshly installed Windows 2016. I dont think the fresh OS changes anything but at this point i try anything ;-)

To me it sounds like you are observing some kind of silent data corruption - perhaps raid controller cache or another component, related to raid controller.
This is why always should be in use consistency check mechanism (e.g. like in ZFS/BTRFS).

I think so too. I ordered another controller yesterday (9265-8i) to check that but it could be a while until it arrives from china.
I'm used to do hardware raids for years because it's more flexible. Just attach a new drive when space is getting low. With ZFS i think this is not possible but i really dont know much about it.

BTW: Seatools longtime check reported no errors on each drive

 

[kekzmobile]

Ok, good news!

I dont know what exactly solved the problem because i did several things at a time: controller firmware flash, new drivers (well, this cannot be it) and windows reinstall.
In freshly Windows I created the raid 6 volume again and tried to copy from different sources to the array - tape and workstation - and the files were perfectly fine.
No stuttering, no artifacts.

I cant say which was more likely, the controller flash oder the Windows reinstall.

Anyway thank you all for help! ^^

 

[likeman]

more than likely was the the firmware that did it in the end as it using hardware raid it controls the array and flow of data (windows reload probably didn't do anything)