Best method and practices for Virtual Machine and ESXi host backups

A

AMD_Gamer

Fully [H]
Joined
Jan 20, 2002
Messages
18,287
What is the best way to backup your Virtual Machines being stored on an iSCSI datastore? I have a Buffalo Terastation pro I will most likely use for the backups. What is the best method or best practice for doing this?

What about the ESXi hosts themselves? I should probably back those up also.
 
ESXi is disposable. No need to back that up.

For VMs you can use one of many backup apps. Put a backup agent in the Guest OS...or use something that does vStorage API backups. If you are using most licensed versions of vSphere you get the vDR virtual appliance that can do backups to a CIFS share.
 
Disposable? I guess, although if you have various things customized, it might be a drag recreating them? Or maybe not...
 
danswartz said:
Disposable? I guess, although if you have various things customized, it might be a drag recreating them? Or maybe not...
Click to expand...

Like what? Network configuration maybe..maybe a few other things. Backup the config files if you want but I have no customers that actually back up their ESXi hosts.
 
I guess that's what I'm wondering about. Is there any kind of description anywhere of what config files to back up? I've googled for that, but not particularly usefully.
 
I think I may configure one of the VMA appliances - it looks like I can use it to take a config save, as well as doing other things...
 
NetJunkie said:
ESXi is disposable. No need to back that up.

For VMs you can use one of many backup apps. Put a backup agent in the Guest OS...or use something that does vStorage API backups. If you are using most licensed versions of vSphere you get the vDR virtual appliance that can do backups to a CIFS share.
Click to expand...

So just backup VM's like any normal client? then if the time comes clone back to the VM machine? I was thinking there is a way to backup the entire VM folder on the datastore? I suppose either way would be pretty much the same thing.

Thanks for the help.

I don't see why you would not want to backup the ESXi host or at least the config. I had to do a ton of stuff to get iSCSI connected, what if you have all kinds of HA/FT stuff going on?
 
danswartz said:
I guess that's what I'm wondering about. Is there any kind of description anywhere of what config files to back up? I've googled for that, but not particularly usefully.
Click to expand...

you can't restore them easily anyway - they're not actually in readable form on the storage device. ESXi is disposable. It should take you no more than 10-15 minutes to rebuild it, if even :p
 
AMD_Gamer said:
So just backup VM's like any normal client? then if the time comes clone back to the VM machine? I was thinking there is a way to backup the entire VM folder on the datastore? I suppose either way would be pretty much the same thing.

Thanks for the help.

I don't see why you would not want to backup the ESXi host or at least the config. I had to do a ton of stuff to get iSCSI connected, what if you have all kinds of HA/FT stuff going on?
Click to expand...

HA -> Right click cluster, enable HA.
FT - > right click vm, enable FT, power on.
Networks - DVS or host profile, or VCM.

We don't really support backing up the ESXi hosts, because they can't easily be restored properly. They're not designed for it.

To back up the entire VM methods:
1. Snapshot VM. SCP config file / flat files off. (mildly dangerous on restore)
2. Snapshot VM. vmkfstools -i the disk files and copy manually the config files (safe, slow).
3. export as template.
4. VDR / VCB / VDDK product of your choice, or write your own (the SDK is, iirc, free for download, and it's not super hard)
5. In guest initiator :)
 
Just use vicfg-cfgbackup via the vCLI / vMA to backup the ESXi servers configuration. Poor man's host profile ;).
 
defuseme2k said:
If you're doing in guest backups, then yes. You do a bare metal restore, just as if it were physical.
Click to expand...

I am talking about the datastore. Lets say your datastore goes bad. You would have to make the VM again in ESXi with the same settings and then do the bare metal restore. How do people quickly recover from the datastore going bad.
 
AMD_Gamer said:
I am talking about the datastore. Lets say your datastore goes bad. You would have to make the VM again in ESXi with the same settings and then do the bare metal restore. How do people quickly recover from the datastore going bad.
Click to expand...

Firstly between your two threads, you should be using Veeam. This will give you all the kinds of backups you seem to want, and is very easy to configure and use. There are about 1 million different ways to backup ESX hosts, however I find in single server solutions where I am turning it back over to a new admin, or someone who hasn't ever used ESX that Veeam is the easiest to use, and does everything you want. It is a bit spendy, but that is time saved when you just set it up and let it run and email you alerts instead of trying to manage 6 different backup solutions yourself.

Secondly as far as loosing datastores I have had this happen only 1 time, and it took 3 days to fully recover. If you have a SAN and the SAN completely fails (were talking the vender just walks in with a new one because the old one is so broken). And you were doing Veeam snapshots, the restore process is pretty quick (well, how quick can you transfer 1TB+ from your backup media) and once all the data is back on the LUN, you just present and boot, no issues. However if you lose a datastore due to bad drive metadata or a corrupted controller there is a chance that some of your backups will be corrupted as well and so even when you restore them, you will need to work with the guest OSes to repair or possibly rebuild windows (if you aren't able to go back more then 1 day because say it is a file or email server). No matter what you do though, A full environment restore is at least a 48 hour restore from initial failure to ticket closure, I have yet to see a single server, single site environment recover faster from a critical outage like loss of storage faster then 2 days.

If I were me (and I am) I would look at a storage technology like EMC VNXe where I can do snapshots of my active LUN to either the same device, or a second SAN. That way if I loose a lun for some reason the snapshot LUN comes online. In that situation it would take a massive hardware failure of my SAN or network fabric to take my storage completely offline. If I am snapshoting to a second SAN on a second switch, then it would take a power outage to take my storage completely offline.
 
AMD_Gamer said:
I am talking about the datastore. Lets say your datastore goes bad. You would have to make the VM again in ESXi with the same settings and then do the bare metal restore. How do people quickly recover from the datastore going bad.
Click to expand...

All backup vendors have image level backups now for ESX (at least those of significance). BE has an image level version - it'll recreate everything for you.
 
lopoetve said:
All backup vendors have image level backups now for ESX (at least those of significance). BE has an image level version - it'll recreate everything for you.
Click to expand...

Ya I have been looking at it. It also claims to do file level backups without the agent installed on the OS.
 
AMD_Gamer said:
Ya I have been looking at it. It also claims to do file level backups without the agent installed on the OS.
Click to expand...

yep - the proxy mounts the VMFS volume and then the VMDK directly, cracks it open, and copies away as if it was a local disk :) We've been able to do that since VCB 1.something, actually.
 
Backup Exec 2010 should be perfect for what I need then.


How do people handle their entire datastore breaking. For example I am using the Drobo. What if the controller on the Drobo breaks or a crook steals it. I find the Datastore to be a huge point of failure.

I was thinking about having a backup plan where I would restore the VM's to local storage on the ESXi hosts for the time being.
 
AMD_Gamer said:
Backup Exec 2010 should be perfect for what I need then.


How do people handle their entire datastore breaking. For example I am using the Drobo. What if the controller on the Drobo breaks or a crook steals it. I find the Datastore to be a huge point of failure.

I was thinking about having a backup plan where I would restore the VM's to local storage on the ESXi hosts for the time being.
Click to expand...

SRM, image level backups, better arrays. Take your pick. Most places that worry about that have a certain RTO with the array manufacturer.
 
AMD_Gamer said:
Backup Exec 2010 should be perfect for what I need then.


How do people handle their entire datastore breaking. For example I am using the Drobo. What if the controller on the Drobo breaks or a crook steals it. I find the Datastore to be a huge point of failure.

I was thinking about having a backup plan where I would restore the VM's to local storage on the ESXi hosts for the time being.
Click to expand...

You are talking about the biggest problem of most ESXi installations.
While you can hot move a VM from one ESXi server the the next, if you have stored
your VM's on a SAN storage, its really expensive and complicated if you need to handle a
complete SAN/ datastore failover + a restart service without a backup -time lag and a huge restore timeout

For my own i do backups for the case of massive problems and store all VM's on
a ZFS SAN to have the best available data security.

In case of a complete SAN failure (a ZFS pool is nearly always intact) I try first to move the pool with realtime data to a second stand-by ZFS SAN server with
enough empty disk slots, import the NFS datastore to ESXi and restart my VM's. With this i can guarantee a complete restart of all of
our VM's below 30 min - with the free ESXi version/free OpenIndiana and without any complicated tools.

Gea
 
Last edited:
_Gea said:
You are talking about the biggest problem of most ESXi installations.
While you can hot move a VM from one ESXi server the the next, if you have stored
your VM's on a SAN storage, its really expensive and complicated if you need to handle a
complete SAN/ datastore failover.

For my own i do backups for the case of massive problems and store all VM's on
a ZFS SAN to have the best available data security.

In case of a complete SAN failure I try first to move the pool with realtime data to a second stand-by ZFS SAN server with
enough empty disk slots, import the NFS datastore to ESXi and restart my VM's. With this i can guarantee a complete restart of all of
our VM's below 30 min - with the free ESXi version/free OpenIndiana and without any complicated tools.

Gea
Click to expand...

That is not really an option for me. I am working for a small non-profit so all I have to work with is the datastore and the backups I do on it. If somebody was to break in and steal the drobo or the controller failed it would be a huge problem. Only quick solution i see is putting the backed up VM's to the local storage on the ESXi host until a new datastore gets in.
 
I setup my new Backup Exec 2010 R3 on Friday and I can't believe how easy it is. It automatically detects my ESXi hosts/vCenter Server and I can easily backup the entire VM. Awesome program! this is exactly what I was looking for. Now i just need to figure out all the different settings in Backup Exec.
 
AMD_Gamer said:
I setup my new Backup Exec 2010 R3 on Friday and I can't believe how easy it is. It automatically detects my ESXi hosts/vCenter Server and I can easily backup the entire VM. Awesome program! this is exactly what I was looking for. Now i just need to figure out all the different settings in Backup Exec.
Click to expand...

The problem with any backups
-you need a long time to restore
-data are never real time data (time lag between last backup an failure)
-have you ever tried a restore?

my opinion
its better to try to regain a service scenario with old effective data
Mostly a server died, not your data - especially with ZFS and copy on write and checksums

Gea
 
NetJunkie said:
If you can recover from snapshots, great....but snapshots are not backup.
Click to expand...

Yes but snapshots are a good base for backups.
You can copy or replicate them easily to another machine

Gea
 
_Gea said:
Yes but snapshots are a good base for backups.
You can copy or replicate them easily to another machine

Gea
Click to expand...

I'm not trying to be a jerk, I'm not. I don't think its rude of me to ask you to at least follow the forum rules.

(25) You are allowed one signature per post, do not put your signature in the body of your post. A signature section is supplied for that usage. No one wants to see your name on your posts multiple times as it is annoying.


As for the meat of your post... how do you replicate or transfer VM snapshots? Are you talking about something in storage land? If so, that is apples and oranges compared to the snapshots discussed in the thread. I agree that storage level snapshots are an OK way to do backups -- but not my preferred method.

I also disagree that doing recovery from a VCB/VADP backup is difficult. Its one of the easiest methods I've ever used and is far more successful than in-guest backups.

I still think for the purpose of this thread, that the VMware Data Recovery appliance is probably the best route if it is included in the licensing level paid for. I think it is included in essentials plus and above. It can do backups and restores at no cost for a small environment.
 
Replicated snapshots as backup is a NetApp marketing/sales tactic. Yes, you can do that. The problem becomes, what is your retention? Do you want 6-months of snapshot retention on disk? If you lose the array how fast can you restore across the wire back to the main site? For those reasons replication is part of a DR strategy, but is usually not the primary use case for backup recovery outside of special cases.
 
the main point is

I do believe that you always must avoid local datastore with ESXi
if you want to have a nice solution - with free ESXi and with lizenced versions.

If you use a separate NAS/SAN server with NFS, you can not only have HA features
but you have also a fast file-based access to your VM's and ESXi snapshots.
(You can copy/ move your VM's but only if you shutdown a VM if you have a simple NFS server)

If you use a ZFS SAN like OpenIndiana, you can do snaps of a running machine
together with ESXi hot snaps (with working memory state). You can copy/ move/ backup/ restore
such a ZFS snap even from Windows via SMB.

But with ZFS, mostly your VM or SAN server is a problem, not the datapool itself.
So my first recovery strategy in case of a SAN failure is to import the pool on a backup machine
and restart original VM's. Backup is for real disaster only (ex fire).

And if I need such a backup, I do not retention or copy it back.
My backup solution is a second SAN server.I can just mount the NFS datastore from any
ESXi server and start up my backup-VM.

Beside of "just copy a VM" to a backup location, you can do filesystem replication, in case
of ZFS its free. ZFS replication is based on snapshots. After an initial transfer of the whole disk
all further transfers are based on delta-snaps containing only the changed datablocks.
With this, you can start a replication for example every 10 Minutes and you have a nearly
realtime disk-based backup (together with target snaps, you have former versions)

It works, does not cost money and is really fast


....
Not to complicate it too much in this thread but
I use virtualized SAN's. Each of my ESXi server has its own SAN with
highspeed internal links via vnic between datastore and ESXi (10 Gbit without the money)
 
Last edited:
I agree with everything you said...but it really has nothing to do with ZFS. Plenty of NAS boxes out there do the same....yeah, you can roll your own like that with ZFS but snapshot and replicated snap features are not "ZFS-only".

My point remains though. Replicated snaps are part of a recovery option. Usually fast recovery but don't always address retention unless you have very low retention requirements. That's why tape is still king, even though backup to deduped disk is getting better all the time.

Repliction data set size, time, and requirements vary greatly by environment. I have a small customer that is 80% write, 20% read for their storage which translated in to really high WAN requirements for replication...so it varies.
 
Yes but ZFS is miles ahead of most other solutions
(think about real data checksums or online filechecks to fix silent data errors)

but the point is, disks are much faster, cheaper and have a better long time reliability
than tapes. So i would always prefer disk based mechanism if I need backups to regain services as fast as possible.

Bur we have also archiv problems (where its not allowed to modify). We think about tapes.
 
_Gea said:
The problem with any backups
-you need a long time to restore
-data are never real time data (time lag between last backup an failure)
-have you ever tried a restore?

my opinion
its better to try to regain a service scenario with old effective data
Mostly a server died, not your data - especially with ZFS and copy on write and checksums

Gea
Click to expand...

The claim of "ZFS is always intact" and "mostly servers die" seems to be skewed to non-enterprise level worlds.

I've dealt with thermal events, water events, power surges that start fires (no, that's not a thermal event, that's an "oh shit" event), theft, malicious destruction, non-malicious destruction, windex on the array, and more - all in major, global companies with millions of dollars, if not billions, tied up in their environments. DR scenarios (off site) and backups both have their place - especially on and off site backups. Relying on the array still "being there" to pull disks from, and relying on the ability for any piece of software to always import disk information, is a fallacy. Have multiple levels of protection, multiple restore methods, multiple backups, or risk losing it all.

I've actually been on the phone when a company has realized that they've "lost it all" and would be closing up shop due to a total lack of multiple layers of redundancy. A few offsite backups could have kept them in business.
 
_Gea said:
Yes but ZFS is miles ahead of most other solutions
(think about real data checksums or online filechecks to fix silent data errors)

but the point is, disks are much faster, cheaper and have a better long time reliability
than tapes. So i would always prefer disk based mechanism if I need backups to regain services as fast as possible.

Bur we have also archiv problems (where its not allowed to modify). We think about tapes.
Click to expand...

Relying on any single piece of software is asking for trouble.

Rule 1 of programming: Every piece of code has at least one bug in it that has not yet been discovered.

And I'll argue to the end of the earth - disks are cheaper and faster, but NOT more reliable - especially not compared ot high-end tape handled right.

What happens when the datacenter encounters a massive thermal event and everything inside cooks? You're gonna need tape, and you're going to want it to be recent, not just an archive, and you're gonna need it off-site. (in this scenario, the arrays cooked and the data going horribly wrong was replicated via array software to their dr site, corrupting both - it happened that fast).
 
Thats not a question tape vs disk
but how you handle backups and how much data you have.

I have a tiered backup strategy based on disks on multiple physical localtions
If i want to have data in a safe, i can even unplug a pool and put it in there juist like with tapes.

But there are different strategies based on different needs or historical evolution.
 
If you're still relying on spinning media continuing to function, while active (tapes, once unmounted, side physically idle), you will eventually (statistically) have a multiple failure that will cause you to lose data. The more locations, the less likely, but it will (eventually) still happen.
 
You must log in or register to reply here.
Back
Top