Connection timeout while cloning vmdk file (ESXi)

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
Trying to clone a very large 2TB VMDK file via ssh on esxi 5.5 using vmkfstools command:

vmkfstools -i Win2008.vmdk Win2008copy.vmdk -d thin

After about 69% completion (10+ hours), i get the error connection timed out:

Clone: 69% done.Failed to clone disk: Connection timed out (7208969)

This happened twice already. Is there way to extend the timeout or disable it all together?
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
You're cloning it on the same datastore, or to a new datastore?

Run it with a -v for more verbose output as well.
 

nicholasfarmer

Limp Gawd
Joined
Apr 19, 2007
Messages
238
If this copy is going across the network, check the network traffic leaving the host before assuming that it actually "timed out."

In some cases, the task will timeout but the migration will continue in the background.
I get this a lot when I storage migrate a 4TB+ VM between storage arrays. I just hit the performance tab in the vcenter client for the ESXi host (source and destination) and look at network usage (if its a network copy) or storage IOPS/MBps to see if the copy is still going.

If you are doing a copy and think VAAI is running, you can always check ESXTOP for stat changes. If you are lucky enough to have access to the storage you could check for activity there too.

If the copy is failing you could always attempt a LUN clone on the array or even a vmconverter attempt (bleh...)

Nick
http://pcli.me
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
shouldn't be able to do a vmkfstools -i across the network ;) That requires hostd to negotiate the NFC copy, or scp of some kind to open an ssh session.

vmkfstools requires the host to see both source and destination, as it engages the vmkernel datamovers directly.
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
@ lopoetve
Zero network traffic even with ISCSI or NFS storage?

Nick

Edit:: URL lovin for Jerry_03
http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1028042

That's not across the network, that's still within a host - using the vmkenel datamovers. Your storage locations are on a network device, but the host sees the source and teh destination, which means you get to use the advanced datamovers.

There's a notable difference between vmkernel datamover and the software datamover via NFC or SSH and the interface layers.


vmkfstools also shouldn't have the saem timeout as teh other clone methods, so it should be valid and reliable (and when it exits, the datamover process should as well) - if you can reproduce when it doesn't, let me know and I'll get a bug filed, as those two should be totally linked.
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
forgot about this thread, lol.

i havent had time to try and clone my vmdk file again, but its on two different datastores that are attached locally. Basically my setup is like this:
Datastore1 on SATA HDD1
Datastore2 on SATA HDD2

Win2008.vmdk resides on Datastore1 and im trying to clone it to Datastore2.
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
forgot about this thread, lol.

i havent had time to try and clone my vmdk file again, but its on two different datastores that are attached locally. Basically my setup is like this:
Datastore1 on SATA HDD1
Datastore2 on SATA HDD2

Win2008.vmdk resides on Datastore1 and im trying to clone it to Datastore2.

do a -v so I can see why it fails
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
Code:
# vmkfstools -i HOME-MEDIA-SVR.vmdk /vmfs/volumes/segate\ 2tb/new.vmdk -d thin -v 1
DISKLIB-VMFS  : "./HOME-MEDIA-SVR-s001.vmdk" : open successful (14) size = 1358389182464, hd = 80217. Type 8
DISKLIB-DSCPTR: Opened [0]: "HOME-MEDIA-SVR-s001.vmdk" (0xe)
DISKLIB-LINK  : Opened 'HOME-MEDIA-SVR.vmdk' (0xe): twoGbMaxExtentSparse, 3907033088 sectors / 1.8 TB.
DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 80217, extentType = 0
DISKLIB-LIB   : Opened "HOME-MEDIA-SVR.vmdk" (flags 0xe, type twoGbMaxExtentSparse).
Destination disk format: VMFS thin-provisioned
Cloning disk 'HOME-MEDIA-SVR.vmdk'...
DISKLIB-LIB   : CREATE: "/vmfs/volumes/segate 2tb/new.vmdk" -- vmfs capacity=0 (0 bytes) adapter=buslogic info=cowGran=0 allocType=1 objType=file policy=''
DISKLIB-LIB   : Failed to clone disk using Object Cloning
DISKLIB-LIB   : CREATE: "/vmfs/volumes/segate 2tb/new.vmdk" -- vmfs capacity=3907033088 (1.8 TB) adapter=buslogic info=cowGran=1 allocType=1 objType=file policy=''
DISKLIB-LIB   : CREATE: Creating disk backed by 'default'
Clone: 14% done.DISKLIB-DSCPTR: "/vmfs/volumes/segate 2tb/new.vmdk" : creation successful.
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : open successful (17) size = 2000400941056, hd = 0. Type 3
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : closed.
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : open successful (131608) size = 2000400941056, hd = 244133. Type 3
DISKLIB-DSCPTR: Opened [0]: "new-flat.vmdk" (0x20218)
DISKLIB-LINK  : Opened '/vmfs/volumes/segate 2tb/new.vmdk' (0x20218): vmfs, 3907033088 sectors / 1.8 TB.
DISKLIB-LIB   : Opened "/vmfs/volumes/segate 2tb/new.vmdk" (flags 0x20218, type vmfs).
DDB: "adapterType" = "lsilogic" (was "buslogic")
DDB: "longContentID" = "37c3bb2c05cd493bd624cfd07ddc5da5" (was "192c99ab08b5fe2369510f87fffffffe")
DISKLIB-LIB   : Failed to create native clone on destination handle : One of the parameters supplied is invalid (1).
Clone: 69% done.OBJLIB-FILEBE : FileBEIoctl: ioctl operation failed on '/vmfs/devices/deltadisks/4fe56b23-HOME-MEDIA-SVR-s001.vmdk' : Connection timed out (7208962)
DISKLIB-VMFS :Vmfs_MoveData : failed to move data (Connection timed out:0x6e0009).
DISKLIB-LIB   : DiskLibCopyDataInt failed with Connection timed out.
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : closed.
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : open successful (1041) size = 2000400941056, hd = 0. Type 3
DISKLIB-VMFS  : "/vmfs/volumes/segate 2tb/new-flat.vmdk" : closed.
DISKLIB-LIB   : Failed to clone : Connection timed out (7208969).
Failed to clone disk: Connection timed out (7208969).
 

peanuthead

Supreme [H]ardness
Joined
Feb 1, 2006
Messages
4,701
Looks like you are changing the HD adapter type. What is you leave it the same?
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
That is very curious. do this:

cat HOME-MEDIA-SVR.vmdk

post the descriptor. Let me sanity check it.
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
im kind of confused. these two hard drives are just plain SATA drives. im a newbie at esxi, so bear with me
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
It's the virtual disks. it says they're using a buslogic virtual scsi adapter, and it's switching it to a lsi logic for some reason.
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
It's the virtual disks. it says they're using a buslogic virtual scsi adapter, and it's switching it to a lsi logic for some reason.

could that be causing the timeout?

I'll post the cat of my vmdk disk when i get home
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
Code:
# Disk DescriptorFile
version=1
encoding="windows-1252"
CID=7ddc5da5
parentCID=ffffffff
isNativeSnapshot="no"
createType="twoGbMaxExtentSparse"

# Extent description
RW 3907033088 SPARSE "HOME-MEDIA-SVR-s001.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.biosCylinders = "243201"
ddb.geometry.biosHeads = "255"
ddb.geometry.biosSectors = "63"
ddb.geometry.cylinders = "243201"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "37c3bb2c05cd493bd624cfd07ddc5da5"
ddb.toolsVersion = "9409"
ddb.uuid = "60 00 C2 90 94 ed 22 cf-70 58 d4 42 c4 af f9 46"
ddb.virtualHWVersion = "10"
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
Oh shit. Where the hell-ass did that disk come from? did you get it from workstation? Or maybe converter? How the ~fuck~ is that even ~booting~ ?!?!?
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
get me an ls -l of the VM directory plz. I need to see if that's actually how the disk is built. I really really hope not.
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
yeah i forgot to mention its from vmware workstation. i copied to .vmdk from my workstation to esxi via the vsphere file upload tool.

I didnt realize at the time that vmware workstation and esxi use two different .vmdk formats.

im cloning it to "convert" it to a esxi compatible vmdk.

i was never able to boot it in esxi
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
Code:
-rw-------    1 root     root     1358389182464 Feb  4 09:51 HOME-MEDIA-SVR-s001       .vmdk
-rw-------    1 root     root           632 Feb 24 01:48 HOME-MEDIA-SVR.vmdk
-rw-------    1 root     root          1396 Feb  4 09:51 HOME-MEDIA-SVR.vmx
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
Do you still have the workstation vm? If so, we can fix this fast. If not, this is going to take a while to fix
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
unfortunately no i deleted the original workstation vm :/

I could try copying back the vmdk file from my esxi server back onto my PC via WinSCP or Veeam....

alternatively, is there a way to fix this vmdk file on the esxi to get it working with esxi? or am i screwed?
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
So, you're going to need 3 goats, a chicken, the blood of a virgin, and a clown...
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,469
version=1
encoding="UTF-8"
CID=7ddc5da5
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 3907033088 VMFS "HOME-MEDIA-SVR-s001.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.biosCylinders = "243201"
ddb.geometry.biosHeads = "255"
ddb.geometry.biosSectors = "63"
ddb.geometry.cylinders = "243201"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "37c3bb2c05cd493bd624cfd07ddc5da5"
ddb.toolsVersion = "9409"
ddb.uuid = "60 00 C2 90 94 ed 22 cf-70 58 d4 42 c4 af f9 46"
ddb.virtualHWVersion = "10"

Step 1. Sacrifice the chickens.

Step 2. Back up your original vmdk descriptor.

Step 3. Feed the goat.

Step 4. Copy the above into the new descriptor.
Step 5. Punch the clown on the nose.
Step 6. Feed the clown to the goat.
Step 7. Pour the virgin's blood on the server.
Step 8. Clone the disk.
Step 9. Repeat three times out loud, while turning in a circle, and waving your arms like an airplane: "I shall forever buy VMware"

Step 10. Pray.
 

Jerry_03

Weaksauce
Joined
Oct 12, 2009
Messages
92
i have no idea what happened but my HOME-MEDIA-SVR.vmdk file got deleted...
However I have an earlier clone, named HMS.vmdk

I have the following Files:

HMS.vmdk (1kb)
HMS-flat.vmdk (1.8tb)
HOME-MEDIA-SVR.vmsd
HOME-MEDIA-SVR.vmx
HOME-MEDIA-SVR.vmxf

HMS.vmdk
Code:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 3907033088 VMFS "HMS-flat.vmdk"

# The Disk Data Base 
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.biosCylinders = "243201"
ddb.geometry.biosHeads = "255"
ddb.geometry.biosSectors = "63"
ddb.geometry.cylinders = "243201"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "37c3bb2c05cd493bd624cfd07ddc5da5"
ddb.thinProvisioned = "1"
ddb.toolsVersion = "9409"
ddb.uuid = "60 00 C2 93 f1 8c ca 02-52 c6 92 14 6f a3 86 b9"

HMS-flat.vmdk
Code:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"

# Extent description
RW 3907033088 VMFS "HMS-flat.vmdk"

# The Disk Data Base 
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.biosCylinders = "243201"
ddb.geometry.biosHeads = "255"
ddb.geometry.biosSectors = "63"
ddb.geometry.cylinders = "243201"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "37c3bb2c05cd493bd624cfd07ddc5da5"
ddb.thinProvisioned = "1"
ddb.toolsVersion = "9409"
ddb.uuid = "60 00 C2 93 f1 8c ca 02-52 c6 92 14 6f a3 86 b9"

I was able to successfully copy all files to my windows PC via veeam. I've given up getting this vm back working on esxi and instead will try to recover the files by extracting it from thin the image on my PC. and then rebuilding a new VM from scratch.

When I try to use powerISO to open both HMS.vmdk or HMS-flat.vmdk I get an error. I tried to create a vm and use these existing images on workstation. using HMS.vmdk, the vm doesnt detect the "hard drive" nor in the VM's bios. using HMS-flat.vmdk it says its not a valid virtual disk.
 
Top