Server 2012 Storage Issue

Discussion in 'Operating Systems' started by Certain, Apr 14, 2013.

  1. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Not too long ago I got a copy of Server 2012 from one of the student programs at my university. I decided to use it as an opportunity to learn a bit more about Server as well as setup an HTPC. That probably wounldn't be the OS I would normally go with, but since I had it I gave it a shot. Everything went off without a hitch. I was able to utilize the storage spaces to combine about 8 TB worth of drives into a virtual disk.

    My system ran like this for a while. I mounted that virtual disk to X: and so I had my C:and X: drives.....C: being utilized as a boot drive. It was partitioned and the rest of the space was being used in the 8tb storage space and mounted to X: This has been running for about 4 months without an issue. I was using this server as central storage for all the PCs in the house. This morning I was downloading files to it using Sabnzb and I noticed that Sab would pause. after reading some logs it showed that it got a CRC error. At first I thought Sab was talking about the files that I was downloading, but then I noticed other files also started to have the same issue. I determined that this was an issue with a storage device somewhere.

    I rebooted my server and when it came back up some Windows updates were coming down the pipe. I let those finish and when I logged in and went to my computer, my C: and X: drive was listed but the X: drive showed no data. As if no drives were attached to it. I opened up Server Manager and took a look at the "Files/Storage" role and I saw a list of the drives. All of them showed healthy. I'd go to the Volumes section try and run a scan but it would just fail. Its as if all of a sudden all my data is gone but the drives are showing healthy. I know this seems like a really specific issue and I even find myself having a hard time explaining exactly whats happening, but I just wanted to see if this issue sounded familiar to anyone else.
     
  2. Demon10000

    Demon10000 [H]ardness Supreme

    Messages:
    4,502
    Joined:
    Aug 20, 2006
    Did you use a parity space, or just lump everything together in a big disk? If you used a big disk, you might have a drive that is going on the fritz. I'd make sure smart is on and use a tool to check each of the drives.

    When you say virtual disk -- you're not saying that you created a big VHDx file across the disks and mounted it, right? You just created a storage space and mounted it as X:?
     
  3. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Welcome to Server 2012.

    Any way did you use Green Drives?
     
  4. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    I created a storage space. Since my original post I've gone back and looked at the event logs. I had a number of warnings for Disk 0 and Disk 6. The problem is, I only have Disks 0-5. Disk 0 is a terabyte drive that is partitioned. One part of it as used as the boot drive while the rest is used as part of the storage space. I tried to remove it from the pool and now it just says "Retired". It wouldn't let me remove it because it says that drive is in use. It said the only way that I can remove it is by deleting the virtual disk.

    Now if I delete the virtual disk won't that erase everything? I don't want to do that because I know that only one or two of the drives are at fault here but I'm kinda poking around in the dark here.
     
  5. Grentz

    Grentz [H]ard as it Gets

    Messages:
    17,104
    Joined:
    May 5, 2006
    Not that it helps. But try and use all similar drives and dont use your boot drive in the future. Just asking for issues.

    Thats really not fair. 2k12 has some great improvements and many including myself use storage spaces with no issue.
     
  6. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Yeah but I dont think that is THIS issue. I'm really having a hard time understanding how everything could just disappear. Even if one or two of the drives failed, I was under the impression that at least the rest of my data would be fine. But it seems maybe a drive failed and now the entire Storage Space seems to not be accessible. What is really frustrating is that everything says Healthy or Ok when I look at the properties. All of the disks say that. If I look at the Storage Space property, it also says healthy. If there is something wrong, how can everything read Healthy?

    I attempted to do a virtual disk repair via powershell, but it didn't do anything. If I look in the Disk Managment console, I can see 0 and 6 which is C: and X: Every repair option is greyed out as if there is nothing to be done. It's like this damn machine is looking me in my face and telling me nothing is wrong while its hair is on fire.
     
  7. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Well, my smart monitor tool shows that each of the drives is in proper working order. So, right now I have 8 tbs worth of disks half full. Since it seems they are working properly and maybe its the storage pool that is the issue, is there some kind of way I could maybe delete the storage pool and make the drives apart of another one and keep the data in tact? I just can't believe make myself accept that I'm going to lose all that data without an actual drive failure and just out of the blue and for no reason at all. If that is the case that would mean that I could not use Server 2012 anymore. Thats would be way too unreliable.
     
  8. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Ok, so here is what I'm thinking...or my goal I should say. I want to preserve the data on my disks, and I'm thinking I may have to get rid of either a storage disk or a volume and start over from scratch. But I don't know which one to delete. If I delete the virtual disk will that erase the data? Will deleting the volume erase the data?

    I'm thinking if I remove the volume, I would no longer have X: but the virtual disk wouild still be in place. Right? At that point I could attempt to create another volume with the virtual disk and see if that helps. But that only benefits me if I know that deleting the volume will not erase the data. Any suggestions?
     
  9. ND40oz

    ND40oz [H]ardForum Junkie

    Messages:
    11,234
    Joined:
    Jul 31, 2005
    What's your disk setup and are you using simple, mirror or parity spaces?
     
  10. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Simple
     
  11. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    So after a bit more tinkering, I believe the problem definitely lies with the Virtual Disk. Earlier when I was searching for a solution to this issue, I saw where a person had a similar issue and they detached then reattached the virtual disk and that solved their issue for the most part. I decided to give that a try. After detaching the disk though, I'm not able to reattach it. When I select the reattach option, the progress bar will move for quite a while. I'm talking 30 minutes plus. Eventually it just disappears. When I go to check the options and whether I have the choice to attach or detach, detach is the only option available, leaving me to believe that the Virtual Disk is attached. Also, the mount point of X: is visible where it wasn't before I selected the attach option.

    That bad news is, if I reboot the machine, the mount point is gone and the drive is detached again. I can go in and reattach it and it will take another 30+ minutes and it will eventually, seemingly complete. But its just a vicious cycle. A reboot detaches it leaving me to believe its never actually attached. I'm at the end of my rope I believe. This really pains me because up to this point I had no issues whatsoever. The storage spaces feature was the only reason I even wanted to use Server 2012/Win 8 for this setup. If its going to behave like this though, I'm going to quickly abandon this setup and chalk it up as just one more reason not to bother with "8". Sigh.
     
  12. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    I don't dispute that there are good improvements but there are also shit load of really bad bugs.
     
  13. Grentz

    Grentz [H]ard as it Gets

    Messages:
    17,104
    Joined:
    May 5, 2006
    You might try booting with one of the storage drives missing entirely. This is again getting to more of a last ditch effort, but you might be able to remove the offending drive and kick the space back into action.

    My guess is that there is corruption on one of the drives that is messing with the storage space.
     
  14. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    This comment gives me hope because I was pondering my options and that was my next course of action. It makes me feel like I'm thinking...."in the right direction" if thats possible. Haha.
     
  15. extide

    extide 2[H]4U

    Messages:
    3,285
    Joined:
    Dec 19, 2008

    You should have used parity mode dude... It isnt windows 8/server 2012's fault...
     
  16. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    We're gonna have to agree to disagree about that one.
     
  17. ND40oz

    ND40oz [H]ardForum Junkie

    Messages:
    11,234
    Joined:
    Jul 31, 2005
    If you weren't prepared to lose all of your data, then you shouldn't have used simple.

    http://social.technet.microsoft.com...storage-spaces-designing-for-performance.aspx
     
  18. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Oh trust me, I was aware that it was a possibilty. I'm not blindsided here. But this wasn't a simple failure. This hasn't been an issue where some drives went bad. All my research has shown this to be a bug and I'm not the first person that has had it. So while I do take some of the heat here, I'm not about ot act as if this OS isn't at fault. But thats beside the point now. If you don't have anything useful other than to play Captain Hindsight.....you should probably just exit the thread.
     
  19. ND40oz

    ND40oz [H]ardForum Junkie

    Messages:
    11,234
    Joined:
    Jul 31, 2005
    Drives don't have to go bad to have issues, I'll assume you're using all server class hardware and you're not using any consumer drives since you still haven't stated what hard drives you are using?
     
  20. Demon10000

    Demon10000 [H]ardness Supreme

    Messages:
    4,502
    Joined:
    Aug 20, 2006
    Did you happen to do anything that made changes to your hard drives? I'm not talking normal every day writing data to them, but big events --

    Install some sort of encryption program?
    Install an operating system that dual booted (even to another drive in the system)?

    Anything like that?
     
  21. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Nah. I hadn't done anything like that to them.
     
  22. Chilly

    Chilly [H]ard|Gawd

    Messages:
    2,031
    Joined:
    Oct 3, 2005
    1. What kind of drives did you use? WD Greens for example are notorious for falling out of RAID arrays and storage pools due to the way they handle time outs. (FYI there are PLENTY working solutions{FlexRAID, DriveBender, ZFS Pools, etc} if you intend on using WD Green, but regular RAID arrays and certain types of storage pool solutions should be avoided).

    2. What does the storage pool say the status of the simple storage pool is?

    3. When you are able to "reattach" the virtual volume and the X: drive shows up, are you able to access any files at all?
    ----- You can access the drive, but it appears that all your files are missing?
    ----- All your files appear, but you can't seem to access them? (errors out or w/e)
    ----- Gives an error when X: is accessed, or asks if you want to format the drive?


    3. Have you run a chkdsk on the virtual disk when it does "partially" reattach?
    ----- If so, and it failed, what, if any, were the error messages?
    ----- If it succeeded, what, if any, were the results it found?
    ----- It refuses to run at all?


    4. Is the storage pool/virtual disk formatted ReFS or NTFS?

    Please post some screenshots if possible as well, it will make it easier to see and help.
     
    Last edited: Apr 15, 2013
  23. Grentz

    Grentz [H]ard as it Gets

    Messages:
    17,104
    Joined:
    May 5, 2006
    Like ND40oz said, just because a drive passes SMART does not mean it is fine. Small corruption or errors in the right place can make drives do some weird stuff.
     
  24. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    You know guys lets not just constantly blame the hardware. Server 2012 has a lot of bugs I have ran across some doozies it could be that there is a bug in the os.
     
  25. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    I'm in the process of recovering the data at the moment so I didn't get to check some of the info like I planned. I have a mix of drives. This was just a home server setup I was using simply because I had gotten a free copy of Server 2012. So I didn't go buy any server grade hardware or anything like that.

    When I "reattach" the drive and my volume reappears, I can't access any of the files. I also wasn't able to run a chkdsk. That was the first thing I tried when all of this started. To be honest, I'm not sure how Windows formats the Storage Pool. Right now its RAW but I'm not sure if that is a result of whatever failure this may be or just how the storage pool is managed.

    On the suggestion of others, I'm current retrieving data with Rstudio. Once that is done I'm going to play around with this a bit more in order to see if I can find a solution that may help others WHEN they run into this issue. As I said, this is apparently a bug. Stumbling around trying to find a fix for this, I've seen cases where this has happened to people who even had their drives set to mirror.
     
  26. Unknown-One

    Unknown-One [H]ardForum Junkie

    Messages:
    8,854
    Joined:
    Mar 5, 2005
    Doesn't the "simple" option = "striped with no parity"?

    If that's the case, it's effectively software RAID-0. If you lose a single disk for any reason, all data on all drives is worthless. You wont be able to recover data off of individual disks because no single disk in the array has a whole file on it, only bits and pieces of each file.
     
  27. Demon10000

    Demon10000 [H]ardness Supreme

    Messages:
    4,502
    Joined:
    Aug 20, 2006
    I think it's more like JBOD than a stipe, but I could be wrong. But yeah -- same effect. There is no parity. So if there is a hiccup, be it OS, hardware, or user -- data is gone.

    I found this post from microsoft:
    Slightly different problem, but similar issue.
     
  28. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    That may indeed be the case, but I ran into another user on another site that had the same issue as I'm having and he recovered his data using Rstudio. Rstudio is able to see the storage pool and individual drives. Its just going to take a while to scan the pool.
     
  29. extide

    extide 2[H]4U

    Messages:
    3,285
    Joined:
    Dec 19, 2008
    I can only hope that you are NOT employed in the IT world and that you never come close to any systems that actually need to run properly. :eek:
     
  30. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Ahh....even more constructive criticism I see. Were you feeling bad about yourself? Did that barb help you feel better?
     
  31. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Welcome to Hard forums where constructive anything has left the building ages ago.
     
  32. odditory

    odditory [H]ardness Supreme

    Messages:
    5,079
    Joined:
    Dec 23, 2007
    In all seriousness OP its too bad you didn't come to the [H] storage forum and ask about Storage Spaces sooner - we could've told you that these kinda of weird anomalies and "the pool disappearing" issues are common. IT Departments also avoid Storage Spaces like the plague.

    Storage Spaces = a beta at best right now, there are many things to like about Server 2012 and I run it on several machines at home and for clients but Storage Spaces is asking to lose your data at this point because your data is locked behind a proprietary abstraction layer and for no real gain (if you want striping do it right and go hardware RAID, else stick to JBOD). Its mostly just the old dynamic disk based softraid system carried forward to include a cute Metro-style setup wizard - the fact they didnt bother to add double parity (RAID6) as an option is further evidence of that.

    If you can get your data off then I recommend reformatting and sticking with JBOD (plain NTFS individually formatted disks) and use drive pooling software since it sounds like you're just storing home oriented data, media, movies, etc. These are also covered in the storage forum here. Stablebit DrivePool is decent, there are others that include parity support.
     
    Last edited: Apr 16, 2013
  33. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Server 2012 is a beta.. Don't get me started.
     
  34. Unknown-One

    Unknown-One [H]ardForum Junkie

    Messages:
    8,854
    Joined:
    Mar 5, 2005
    We're running two Server 2012 virtual machines at my office (Full install as primary domain controller for the entire company, core install as fallback domain controller)...

    They're the most rock-solid servers in the topology. Not a single hick-up to date.

    We're about to bring a 3rd one up to be our new exchange server, actually...
     
  35. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    I am happy for you. Would you like a cookie?

    I also have 2013 Exchange servers in production.

    I have 16 2012 servers in production, 8 in test environment. I actually had to demote some of the functions because of issues I moved the services back to 2008R2. I then don't have simple two server roll outs. They are no where as good as 2008r2 was upon release. Most are file servers now this is the biggest benefit I see, Rest I saw no difference between 2008r2. ( I also don't give a shit about the features I am sure some one is gone be ranting about shortly)

    Once you find a way around the issues and bugs they work. Clients requested it I had no choice over in couple deployments.
     
  36. Unknown-One

    Unknown-One [H]ardForum Junkie

    Messages:
    8,854
    Joined:
    Mar 5, 2005
    No need to be a jerk about it...

    Point was we have yet to actually run into any bugs or issues. Production has 2 active 2012 boxes (with a 3rd on standby until we can do the swap), test lab has an additional 11 that will eventually be cloned and rolled out to replace Server 2003 and Server 2008 R2 virtual machines on the production network.

    Only reason a few of them aren't already in production is because we have users on the boxes 24/7 who will complain if we shut them down, even for a few minutes. I'm itching to replace the terminal and edge servers, but we just can't take them offline right now.

    The current crop of Server 2008 R2 machines are hell to administer, by comparison...
     
  37. mmtom

    mmtom Limp Gawd

    Messages:
    319
    Joined:
    Jan 9, 2003
    Most people I know never encounter a single bug, yet you've managed to hit seemingly hundreds on just a 16 server rollout.
     
  38. Certain

    Certain Limp Gawd

    Messages:
    141
    Joined:
    Jun 16, 2005
    Yeah, I wish I'd read up on it a bit more before I decided to implement that setup. I think I have a good shot at retrieving the data. Once that is done I'm going to switch over to either StableBit Drive Pool or Drive Bender. I've heard good things about those.
     
  39. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Not Hundreds but big enough ones that caused complete OS failure.

    Core to GUI upgrade can cause corruption of all files can't fix with SFC. Microsoft Acknowledged this. (Tech Support) Complete Reinstall :(

    Gateway deployment that is done as a feature breaks the wizard later when you want to do RDS deployments with additional servers.

    Applied security template from the RDS wizard causes firewall in windows to stop forwarding traffic if any other role is installed such as DC. Microsoft acknowledged this one as well. (Tech support)
    Still need to modify the RWEB IIS settings just like 2008R2 You would think they would have addressed this one already.

    Installing exchange 2013 CAS role on a server that has other iis services will fail to create OWA. IE remote powershell.

    If you don't add forwarders or re-organize DNS forwaders will cause very poor resolution of targets on the internet. This is kind of stupid. (IPV6)
    Also can cause your internal resolution to stop working temporarily, if you were installing exchange 2013 it will cause your AD schema to get fucked I had to repair AD with ADSI edit. Thank god this was a new network.

    Windows 8 and 2012 GPP are not processed correctly if no changes were made. This is due to the Fast Boot features of the OS. Once you disable this to process GPP then 8 is slower to log on compared to 7.

    RAP/CAP policies on a NPS server will mess up authentication if you had the role installed on a 2008r2 server.

    Windows 2012 server has odd connectivity issues in ESXi if you using anything but vmxnet 3. (I use ESXi not HyperV) Not an issue with 2008

    .NET 3.5 install stupidly has to be done by dism or powershell cause the wizard or install feature doesn't work.

    Certain parts of the UI I despise.
    I think 2008 has bigger overhead to administering but its not any harder then 2012. I hate the 2012 ad console. Its ugly.

    I like IIS has now multi-tenancy. Better SMB performance and additional support for non cachable database files, ofcourse you need Windows 8 :( Good for DFS tho.


    Don't get me wrong I think 2012 has some cool shit in it but when you realize that you have a giant problem with OS corruption because you upgraded from core to gui thats a pretty big oh fuck.
     
  40. Demon10000

    Demon10000 [H]ardness Supreme

    Messages:
    4,502
    Joined:
    Aug 20, 2006
    You can use server manager, if you want. You just have to specify a path to the source file (just like you do with dism or add-windowsfeature) because the bits aren't stored on the local machine. That was a design decision, not a bug...

    There is an update for Win7/2008 R2 that should allow alot of the Win8 performance enhancements to be available in Win7/2008 R2 (specifically file transfers and TCP/SMB optimization). I haven't done much testing with it yet, but it is on all of my terminal servers. Unfortunately, my terminal servers don't talk to 2012 servers yet, so I don't really have much to test with.