To avoid a disaster, tell me please the most robust way to copy files (NAS weird issue)

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
Hi,

This is a long story but I'll leave the details for later. Now, I just need to quickly backup my data. To make it short, I have a Synology DS1512+ box as a NAS connected to my network. It served me well for over 5 years. Anyways, the raid5 volume crashed for reasons I'll explain later. However, I can still see my data with slow access speeds. I have about 5TB of data collected over 20 years of computing.

Not everything is backed up, and my latest backup was like 4 months ago. Anyways, I just want to copy this 5TB into external drives or maybe internal ones before I fix the volume. So, my 1 million dollar question is : what's the best way to copy? Manually that is. I know if I use windows I'll run into so many sorts of problems like long paths/errors which might halt the process and make start all over. There must be a better way. I know a little about xcopy but I don't think it is the best. Should I use the built in back up feature in DSM OS?


The other little issue is that I must leave for a trip in few hours and come after two days. I would rather fix this problem before going but I can't. It is a family thing.

Please help !
 
Last edited:

rive22

Supreme [H]ardness
Joined
Mar 10, 2004
Messages
4,646
block level clone/image. bypass the file system. safer and faster.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
block level clone/image. bypass the file system. safer and faster.

How is that done in my case? With a special program? Image programs you mean like macrium reflect or true image?
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
if your target machine is windows, use good ole richcopy, and hope for the best. if you target machine is linux, use rsync. Its going to take days to copy the data to a new target location.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
if your target machine is windows, use good ole richcopy, and hope for the best. if you target machine is linux, use rsync. Its going to take days to copy the data to a new target location.

It will take much much longer in my case because of the very low read speed (The thing that started it all). I'll explain in few hours.
 

Brian_B

2[H]4U
Joined
Mar 23, 2012
Messages
3,356
rsync is the most robust method I know of, but it's not the fastest. DeltaCopy is, from what I gather, a preferred Windows implementation of it.
 

JBark

2[H]4U
Joined
Dec 13, 2002
Messages
2,048
Simplest would have to be plugging a USB drive into the NAS and using one of the DSM tools like Hyper Backup to do the backup. Pretty sure Hyper Backup is just rsync with a fancy GUI, so it should do the trick. If it's mainly media files, just make sure compression is turned off, since it will just waste time trying to compress stuff that is already compressed.

Could also SSH in a run rsync directly to a USB drive, or you could also install DeltaCopy on your PC, which will give you an rsync target on your PC that you can point Hyper Backup (or rsync) at. You say you have a backup from a couple months ago? If it was a direct copy from the NAS, then in theory you could just point your rsync at that and it should only copy the changed files, saving a bunch of time.

I don't use rsync much, but if you've got a USB drive plugged into the NAS, then something like this through SSH would probably do it:
sudo rsync -rtv /volume1/FolderName /volumeUSB1/usbshare/FolderName
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
rsync is the most robust method I know of, but it's not the fastest. DeltaCopy is, from what I gather, a preferred Windows implementation of it.

Simplest would have to be plugging a USB drive into the NAS and using one of the DSM tools like Hyper Backup to do the backup. Pretty sure Hyper Backup is just rsync with a fancy GUI, so it should do the trick. If it's mainly media files, just make sure compression is turned off, since it will just waste time trying to compress stuff that is already compressed.

Could also SSH in a run rsync directly to a USB drive, or you could also install DeltaCopy on your PC, which will give you an rsync target on your PC that you can point Hyper Backup (or rsync) at. You say you have a backup from a couple months ago? If it was a direct copy from the NAS, then in theory you could just point your rsync at that and it should only copy the changed files, saving a bunch of time.

I don't use rsync much, but if you've got a USB drive plugged into the NAS, then something like this through SSH would probably do it:
sudo rsync -rtv /volume1/FolderName /volumeUSB1/usbshare/FolderName

Thanks for your replies.

I will tell you what happens with me. I'm back from my trip and I will tell you the full story.

I have had this NAS for about 5 years with no problems whatsoever. I installed 3 3TB seagate drives in it and configured it as SHR (RAID5) for about 5 TB of capacity (It stayed like this for about 4.5 years). I recently( about 6 months ago) added one more 3TB drive for a total of about 8TB and also one more drive as a hot spare. Things were great . About 5 days ago, just out of the blue the NAS started showing signs of slow read/access speed. Being the speed maniac I'm I noticed immediately. Things weren't smooth as they once were, so I started investigating. All files were accessible and readable. I went into the NAS DSM and checked everything. Everything was fine. The volume is healthy, no weird logs, HDD's status is normal, and smart data is fine. Network speed is 1Gbps. I couldn't fix it, and nothing shows anything is bad. When I copy a large file from my NAS to a local drive, I get speeds like 1 Mbps or even less. Sometimes 3 Mbps. When I tried to replicate a file in the NAS itself, it was also slow. This is when I realized something is definitely with the drives (the volume), and not the network speed for example.

I read somewhere that one person had the same problem and when he removed one of the drives, his volume was degraded but the speed went back to normal. This is when I decided to do the same thing, which was definitely a bad idea ( I should have consulted you first). I almost crashed my volume by pulling drives. The NAS acts strangely when drives are pulled out. Anyways, I was able to get it back to its slow speed state where all files are accessible. This is when I said let's back up everything before doing something stupid, hence this thread !

I don't know what's wrong, but maybe one of the drives is just bad. I'll back up everything and see if rebuilding the volume fixes things, although I doubt it. I'll test the drives individually after I break the volume. I think that's the best way to tell if one or more is bad. I will do that after I finish copying all data which might take few days.

What do you think?


Again, Thanks.
 

Brian_B

2[H]4U
Joined
Mar 23, 2012
Messages
3,356
So you pulled drives out, then put them back in. and your running it degraded now?

The array is going to run like crap while it's degraded, and until it can finish the rebuild. That will definitely slow things down, although that doesn't explain the slowdown from before you started yanking stuff out.

In my opinion, your better off letting the NAS rebuild the array, then back it up - than you are trying to copy everything off while it's degraded. If you have a real fault while it's degraded, your done for - and I suspect it will take longer to do your backup than it will to do a rebuild.

DSM is usually pretty good about detecting HDD problems. Did you check out the system monitor while file copying? If your trying to do an encrypted file transfer (SFTP, HTTPS, or anything via SSH) that definitely slows things down and you can see it as high CPU activity on the NAS. It could have been doing something routine - like data scrubbing the array when you were running your tests. I don't know - just some things to check out once you have the array rebuilt.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
No encrypted file transfer. It actually doesn't say it is degraded now. However, two drives show the system partition corrupted message. The spare is also there showing as not initialized.

Better be safe than sorry, I will backup now and after I rebuild it. It will take ages but it is fine.

Data scrubbing? No it wasn't doing it when I started looking things up. I wanted to do it hoping it will fix things up but I didn't let it finish.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
I would leave the NAS alone. Yanking drives wasn't a good idea. You are basically up shit creek with your options. Either use rsync, or richcopy/deltacopy to trickle your data off and let it run knowing it could take a fair amount of time.

other more extreme option? Buy new harddrives, shut your nas down, take each drive out of your nas, label where it was plugged into, clone said old drives to new drives. Take old drives, put them on the shelve as your gold drives. Take new drives in put them in your nas, and work on either resolving / rebuild your array. If you train wreck new drives, then just reclone the data from your old drives.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
I'm almost done copying using richcopy. I will now start to fix the volume, so what's the best way to do that? Just go to storage manager in DSM, remove the volume and re-build it again? Any special tools in DSM to check the drives individually while the volume is broken?

What if I wasn't able to find anything wrong with the drives, rebuilt the volume and ended up with the same problem?
 

M76

[H]F Junkie
Joined
Jun 12, 2012
Messages
12,867
Failing drive with lots of re-allocated sectors can cause slowdown. But that should show up in smart.
I've had the same problem, only for me read speeds went down to 30-50KB/s, so I pulled the failing drive (log showed which drive was the first to act up) replaced it with a new drive, let the rebuild finish (about 30 hours on 12TB) and now everything works normally. I can pull data with normal speeds.

Trying to pull the data while the array is degraded is risking another failure and Raid5 can only tolerate one drive failure. The risk is especially high if the drive you pulled was not the one causing the slowdown, which means all your data is at the mercy of a failing disk.
 

capt_cope

Gawd
Joined
Apr 12, 2009
Messages
947
Depending on the specs of your drives you might want to consider moving to raid 6 or purchasing better drives. If you've got old 3tb barracudas (st3000d) they're probably spec'd with one non-recoverable read error per 10E14 bits — or one every 12.5 tb. With one failed drive you'll have to read ~8tb off your array during the rebuild which gives you better than 50% odds it will fail.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
Guess what guys!? Like I said I finished copying all data so that's done. I still didn't start fixing the array. BUT, it actually went back to normal operation by itself ! The same good old speed that I know and love. DSM still shows errors though, so I still need to fully fix it and sort all these errors out. But it is actually working like a charm although I haven't done anything yet!!! Take a look at the images below:

ACTbV3t.jpg


wmFrqOm.jpg



Life can have weird things happen, and sometimes they are good.


Anyways, I can play with it more safely now since all data is somewhere else.

Will update you soon!
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
It has been some time I know. The NAS didn't stay at good speed for a long time as it went back to the slow speed after few days after my previous post, which means I still didn't fix the problem completely even though DSM now shows everything as normal. I backed up my data to 2 locations. So I can say that I can play with it safely now. Thanks to richcopy and robocopy. Truly impressive. I'm at your debt guys.

I noticed something while testing. When I perform the SMART test for disk 4, it always gets stuck at 90% and never finishes. All other drives are fine. Disk 3 is now configured as a hot spare. Should I just remove disk 4 and force the array to rebuild itself using the hot spare? I'm saying this because i'm thinking disk 4 is the culprit and by rebuilding the array with a good disk, all problems should go away. What do you think? Do you recommend doing this?

My DSM also isn't the latest version, so I can update it. I don't know if this will be any help but I can do it.

Note: When I searched synology forums about the smart test getting stuck at 90%, it seemed like it is common and for some people it will finish after hours of waiting. You can google this yourself

Again thanks, and I'm waiting for your comments.
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
It has been some time I know. The NAS didn't stay at good speed for a long time as it went back to the slow speed after few days after my previous post, which means I still didn't fix the problem completely even though DSM now shows everything as normal. I backed up my data to 2 locations. So I can say that I can play with it safely now. Thanks to richcopy and robocopy. Truly impressive. I'm at your debt guys.

I noticed something while testing. When I perform the SMART test for disk 4, it always gets stuck at 90% and never finishes. All other drives are fine. Disk 3 is now configured as a hot spare. Should I just remove disk 4 and force the array to rebuild itself using the hot spare? I'm saying this because i'm thinking disk 4 is the culprit and by rebuilding the array with a good disk, all problems should go away. What do you think? Do you recommend doing this?

My DSM also isn't the latest version, so I can update it. I don't know if this will be any help but I can do it.

Note: When I searched synology forums about the smart test getting stuck at 90%, it seemed like it is common and for some people it will finish after hours of waiting. You can google this yourself

Again thanks, and I'm waiting for your comments.

Any comments on this storage experts?
 

sram

[H]ard|Gawd
Joined
Jul 30, 2007
Messages
1,468
I had to wait because I started data scrubbing and it was super slow. It finished and I removed disk 4. I was right. Disk 4 was the culprit. The volume degraded and it started to repair itself with the hot spare. The speed is back to normal after disk 4 was removed. This time, I actually got a smart error for disk 4 and a recommendation to replace it right before I actually do so.

I added another spare. All slots full. All is well.

Again, thanks for everybody who helped.

hz1XIiV.jpg


r8kA4Q9.jpg


U2QQleh.jpg
 
Top