After VM hypervisor crashed (took down fs with it) -- now network shares inaccessible

Cerulean

Cerulean

[H]F Junkie
Joined
Jul 27, 2006
Messages
9,476
SharedFolderBrokeAfterCrash.png


One of our hypervisors in ESXi/vSphere crashed, which took down our fileserver with it. (vMotion of course took care of the rest.)

When the fileserver came back up, we ran into this problem. All of a sudden, domain admins included, users are unable to access their G:\ drive. This is where they go to access their departmental folders, company-wide shared folder, inter-division folders, and public scratchdisk/temporary/dump folder.

The server has an actual G:\ drive (logical) which is hosted on a datastore. The whole drive is shared at root (\) with full read and write access for Authenticated Users and Domain Admins (Share permissions) and read+execute access for Authenticated Users and full control for Domain Admins (NTFS permissions). This is what I have tried (in this order):

* Turn off sharing of G:\ (--> OK), then turn it back on and restore previous permissions; no success
* Re-add Authenticated Users and Domain Admins to NTFS and Shared permissions; no success
* Tried disconnecting from the mapped network drive on a VDI and manually readded it; no success
* Rebooted the server manually with shutdown -r -t 0; no success (still can't access the 'g' network share)
* Add Everyone with read+execute access to NTFS permissions; no success
* In addition to the previosu note, add Everyone with full read and write access to Share permissions; no success
* Created a folder "Test" in C:\ so that C:\Test exists, shared it with Everyone on read+execute on both NTFS and Share permissions; was able to access this no problem

(PS. The folders contained inside G:\ or \ do not inherit permissions from root)

Have tried logging off and logging on several times under different users (Domain Admin and regular ordinary Domain User alike).
 
Can you even ping the guest OS's IP from a remote computer?

If you can ping, can you resolve a computer name lookup?

Can you create a new folder and share it on the guest? Can a external client reach that new share?

In computer management what does the share folder look like?

Can any other guest OS share a folder and it be seen by an external client?

Do you have either a system restore point or (if its a server) a system state restore point?

Do you have a full server backup or past snapshot?
 
Mackintire said:
Can you even ping the guest OS's IP from a remote computer?

If you can ping, can you resolve a computer name lookup?

Can you create a new folder and share it on the guest? Can a external client reach that new share?

In computer management what does the share folder look like?

Can any other guest OS share a folder and it be seen by an external client?

Do you have either a system restore point or (if its a server) a system state restore point?

Do you have a full server backup or past snapshot?
Click to expand...
1. Yes

2. Yes

3. Yes -- I created a folder C:\Test and shared that and had no problem accessing it from clients. But when I turned off the sharing on the existing share that we are having problems accessing (even as Domain & Enterprise Admins), and then turned it back on (after OK'ing all the way out of dialogue and pop-up windows), still couldn't be accessible... even with 'Everyone' group on Share & NTFS security permissions. :(

4. Will get you a screenshot when I can.

5. Yes. Temporarily, we've move the files over to another server meant only to host files for the ERP software we use.. in this process, all security permissions on our files got stripped :( fortunate for ME, I'm not in the nightmare of this as I am presently traveling to France on business project

6. We have a snapshot from 24 hours ago. I need to suggest to my supervisor that we could restore that snapshot and use a program like FileBoss to intelligently copy over new/modified files.

EDIT: Too late on #4. My supervisor has already done some things on the server that would make this request useless. :(
On another note, I have suggested to my supervisor restoring a snapshot and using FileBoss to copy with overwrite on new/modified.
 
Last edited:
Any chance you are running DFS? Is that service started?

Check inheritance, I've seen some weird stuff happen when a file server doesn't go down cleanly.

Edit: Maybe I should read the OP....scratch the inheritance option. On the root of G: can you create a new share and share that out without any trouble?
 
All your File shares are stored in the registry in the Lanman key and subkeys.

Sounds like Lanman and all the subkeys in the registry got hosed.


Here's what you can do about it.

The first try....

  • Completely Back up the server
  • load the earlier snapshot
  • export the lanman key and subkeys from the registry to a external flash drive
  • Restore your full backup
  • Boot up the server, go into the registry and delete the lanman key and subkeys
  • Merge your saved lanman key and subkeys you saved to your flash dr
ive.

Your NTFS permissions should be current and intact and your file shares should be restored to the shares that were active when the snapshot took place.

If that didn't work here's the alternative.

  • Completely Back up the server
  • Robocopy the entire datastore to another remote location, migrate all the data using the correct flags to copy your permissions intact.
  • load the earlier snapshot
  • export the lanman key and subkeys from the registry to a external flash drive
  • Rebuild the server from scratch to a clean form. Make sure you use the same partitioning and drive letters.
  • Boot up the server
  • RoboCopy all your data back into the server again using the correct flags to ensure that all your NTFS permissions are copied back correctly.
  • Merge your saved lanman key and subkeys you saved to your flash drive.
  • Reboot

Your NTFS permissions should now be current and intact and your file shares should be restored to the shares that were active when the snapshot took place.




Take your boss out for hookers and blow to celebrate :D
 
Last edited:
You must log in or register to reply here.
Back
Top