Help with Windows 10 data integrity issues - ReFS, Storage Spaces, Copying files

[commando]

Conclusion: P67 chipset fault. Details here. A motherboard swap fixed the problem.

Please note - please see this post below from 9 March, this first post is now out of date due to additional research. It's not a W10/SS problem.





Short version
I'm having an issue with my home PC. My mirrored storage space using ReFS on Windows 10 technical preview doesn't appear to be working reliably, in that Macrium Reflect images often fail to verify.

Long version
Background: I take Macrium Reflect Free images of my operating system drive in order to recover quickly if something goes wrong. I run Reflect from a WinPE USB boot memory stick and image onto a standard Seagate NTFS drive. The images almost always verify as "ok" both in WinPE and when I boot into Windows. I assume the Reflect "verify" (which is slow) uses a CRC or MD5 type checksum.

I have Windows 10 technical preview, with HGST 4TB x 2 set up with ReFS and a mirrored storage space. I have this setup as I value data integrity, I like the idea of automatic checksums and error correction. I copy these OS images (20GB or so) to my ReFS SS disk using Teracopy, with the "verify" turned on. Verify uses a CRC to check if the destination file is the same as the source file

The problem: Around 1/3 to 1/2 of the time the Teracopy verify reports "CRC failure". Strangely though even if a CRC failure is reported Macrium Reflect images sometimes verify successfully. Sometimes if there's no CRC failure in Teracopy and the Reflect images fail to verify.

I will have to do some systematic testing between each of my many disks to help pin down whether it's just the ReFS disk or it applies to other disks too. I can say that I have plenty of Reflect images fail to verify on my standard Seagate NTFS disk, both old and new images. The couple of Teracopy copies and Reflect verify tests I did to the Samsung 840 pro SSD last night passed CRC and Macrium verify.

I occasionally get messages like this in my Windows event logs, though it doesn't tend to show when I get CRC or Reflect verify errors.

Text: The file system detected a checksum error and was able to correct it.
Source: ReFSv1
EventID: 132

I've run both memtest x86 and the Windows 10 RAM test with no errors at all. I've run Prime95 for a day with no errors. The Intel CPU testing system comes back perfect. I've run another couple of testing problems.

The question: I guess I'm asking for any thoughts, opinions, experience, ideas on how to test the machine, or anything else really. Are there hardware testing programs I can use? Is there software that will write files to a disk then read it back to check it's written/read properly? Should I be using a different version of software or hardware RAID? I know a ZFS NAS is another option but I don't want to have to build another machine for this.


Computer specification:
- Gigabyte GA-P67A-UD3R-B3 Motherboard, Socket 1155, Intel P67 motherboard
- Intel Core i7-2600K 3.4 GHz, Socket 1155 (running at stock speed)
- Gigabyte GV-N520OC-1GI, GeForce GT 520 Video Card, 1024MB
- Corsair Vengeance CML16GX3M4A1600C9, 4x4GB, DDR3-1600
- Antec High Current Gamer HCG-520
- Nocuna CPU cooler - I forget the model but it has two 120 or 140mm fans
- Seagate 1TB x 2, NTFS
- HGST 4TB x 2, ReFS with mirror storage space
- Samsung 840 pro 120GB running Windows 10 technical preview
- OWC SSD 120GB

 

[devil22]

Different code stresses memory in different ways, I'd try using one stick of ram at a time to absolutely rule out memory errors. I've used ReFS extensively in Win 2012, and it works great, so it may also be bugs in Win 10 because it's still alpha/beta code. I'd also try something that reads/writes the entire drive checking for errors, there's various programs for this, I know seagate has one for their users, not sure about HGST but look around their web site for one.

 

[commando]

Different code stresses memory in different ways, I'd try using one stick of ram at a time to absolutely rule out memory errors. I've used ReFS extensively in Win 2012, and it works great, so it may also be bugs in Win 10 because it's still alpha/beta code. I'd also try something that reads/writes the entire drive checking for errors, there's various programs for this, I know seagate has one for their users, not sure about HGST but look around their web site for one.

Thanks for your thoughts. I'm not sure single memory stick testing is valuable, given both memtextx86 on "everything" mode and Windows 10 memory testing in "extended" mode both gave no errrors at all.

The testing programs tend to destroy data. I can cope with pulling all the data off if I need to, though it's not preferred. I'll probably have to in order to do good testing. Because ReFS and SS is new there are probably few testing tools.

It could be bugs, or it could be my hardware, I guess. The real question is how to diagnose where the problem is.

 

[devil22]

Damn I just remembered something...

Run Powershell.exe as admin, and type the following commands:

$phd=Get-PhysicalDisk
$phd|Get-StorageReliabilityCounter

It will show stats for each drive in the system, along with read and write error counts. That will tell you if one of the drives is having errors and then you can replace it, without having to wipe/test the wrong one.

 

[commando]

Excellent I'll try that when I'm at home in about 8 hours, thanks

 

[commando]

Ok here are the results of that command

DeviceId Temperature ReadErrorsUncorrected Wear PowerOnHours
-------- ----------- --------------------- ---- ------------
3 0 0 12841
1 0 3914
0 100 11307
4 0 0 5818
2 0 0 212
5 0 0 184

The disk with read errors is my older SSD. No errors on the rest of the disk.


I downloaded and installed Acronis and did the following tests, imaging my C SSD drive:

Target: 1TB NTFS drive. Failed validate first time, passed validate second time.
Target 4TB ReFS SS drive. Passed both times.

This suggests that the issue isn't with the ReFS disk, that it's with the PC. I'm going to do a bit more testing this evening though, so I'll post results later.

 

[commando]

Ok I have new results. When copying from any disk to any disk with Trucopy I can get CRC failures reported occasionally - around 30-50% of the time. No single disk is worse than another as far as I can see. The ReFS SS doesn't seem to fail any more often than the others, if anything it seems slightly more reliable. When I copy images to other disks using Trucopy (being away from the PC I assume there were no CRC errors) then validate using Reflect it sometimes fails.

I'm thinking there's something fundamental wrong with this PC... it passes every theoretical test I give it, but fails practical tests.

 

[devil22]

If it were me, I'd try sfc /scannow (from elevated cmd.exe, fixes corrupt Windows files), if that didn't work I'd re-install Windows, if that failed to fix it also, I'd then RMA the mother board.

 

[commando]

I'll give that a shot thanks. I have to boot to a recovery console I think, it won't work from an elevated command prompt. It's W10 latest beta, so I may try a W7 install as well.

A reboot seems to have fixed all problems. The computer's 3-4 years old, if it turns out to be a hardware problem I'll build a new one.

 

[commando]

I've been doing more testing. I've installed Windows 7 on another disk, bought it fully up to date, and done some testing. Copying 100GB of data between two NTFS disks using Teracopy with verify turned on I get 3-15 CRC fail errors. This happens whether the drives are plugged into the motherboard or my add on SATA card. This happens the same on W7 and W10, so I can rule out the OS as a factor. It could be driver related, but Windows isn't bad at updating drivers itself these days.

I've also run a program that writes masses of data to the drives then reads it back to verify it. It reports no failures in W7 (four runs, all NTFS on a WD Black and an SSD) but in W10 I do get verification errors. I'll be re-running this test in both OS's today to verify these results. I may also run badblocks if I can work out a way to do it easily - maybe a live Linux distribution.

At this stage my only conclusion is it's a weird isolated minor hardware failure somewhere on the motherboard causing the data corruption. I'd love another explanation/solution because right now it looks like I'll have to replace the entire PC (it's an old socket 1155 which is difficult to buy new), which I can afford but I'd prefer not to do.

 

[smangular]

Perhaps Drivers issues or perhaps Win10 bug... Have you asked or posted this in the Win10 Beta forums?

 

[commando]

Perhaps Drivers issues or perhaps Win10 bug... Have you asked or posted this in the Win10 Beta forums?

I've reproduced the problem on a fresh install of Windows 7, bought up to date, so I don't think it's that. Thanks for the suggestion though.

 

[devil22]

Stupid question, have you tried different SATA cables? But you said it happened on different drives, so that's probably not it either. After that, all your left with is a faulty mobo I'd think.

 

[commando]

On another thread Dangman posted a link to a P67 chipset problem that would explain it It's here.

 

[HammerSandwich]

This happens whether the drives are plugged into the motherboard or my add on SATA card.

So how is this a chipset problem?

 

[commando]

So how is this a chipset problem?

I figure the data has to go through the chipset even if it's going via an add-in card.

Anyway I'm sick of trying to diagnose the issues, I've been dealing with data corruption issues for 2-3 months now. I'm just going to buy new MB, CPU, RAM.