This seems to happen somewhat frequently and is getting annoying. Even since using Veeam, often times the VM will have an alert that the snapshots need to be consolidated. And sure enough, they need consolidation. This never happens when taking a normal snapshot and removing it. Have you seen this before? I am getting tired of having to consolidate especially since it is killing VM performance when I do.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Veeam causing Snapshots to need Consolidation
- Thread starter KapsZ28
- Start date
Are you on R3 of Veeam7?
I am having an issue where hot add disks are not dismounted from the backup server causing the consolidation to fail. Go edit the settings of the proxy VM and see if the hot add disks are still there.
So far veeam support's suggestion is to change the proxy to network based instead of direct SAN. Which is a bs solution.
I am having an issue where hot add disks are not dismounted from the backup server causing the consolidation to fail. Go edit the settings of the proxy VM and see if the hot add disks are still there.
So far veeam support's suggestion is to change the proxy to network based instead of direct SAN. Which is a bs solution.
These backup jobs are showing 100% successful. It is the latest version of Veeam which is patch 4, but has happened on prior patches too. We only started using Veeam about 3-4 months ago. So it has always been version 7. Funny you mentioned network mode. One of the VMs that needs to be consolidated shows that hot add failed and it switched to network mode for the backup. Which is another issue. It seems to failover to network mode quite often.
I also wonder if it is a storage issue. The network mode seemed to start after our NetApp sustained a power failure. Don't ask how. Although I can't seem to pinpoint it to a specific shelf or datastore.
This is the same environment that we are running vCenter 5.5 and had vFlash enabled which appeared to be causing problems with vCenter. I haven't went through the workaround yet, but vFlash has been completely removed. I plan on migrating everything to vCSA instead of the Windows based vCenter we have been using.
What are the recommendations for the Veeam configuration? It started off with a single Veeam VM running everything except for storage. So it has the full version of SQL installed, management, and is a proxy. I have since added two more proxies. One is network mode only which I use to backup Exchange DAG. So the primary Veeam and secondary have 8 vCPU and are configured to run up to 6 concurrent jobs. Although the backup jobs are spaced far enough apart where they shouldn't be running anymore than 3 jobs per proxy. There are about 32 backup jobs set to run between 7:30 PM to 1:30 AM. I tried to space them apart so that there are not tons of snapshots being taken and removed at the same time.
The backup repository is Windows Storage Server 2012 R2 and is located in the same datacenter. It is connected directly to our core router which is connected to the storage switches.
The ESXi servers have plenty of resources. They are running version 5.5. Not update 1 due to the NetApp issues it has.
I pretty much have to keep babying these damn backups which is messing up my sleep schedule.
I hope Veeam 8 is going to fix this by switching to SAN based snapshots.
I also wonder if it is a storage issue. The network mode seemed to start after our NetApp sustained a power failure. Don't ask how. Although I can't seem to pinpoint it to a specific shelf or datastore.
This is the same environment that we are running vCenter 5.5 and had vFlash enabled which appeared to be causing problems with vCenter. I haven't went through the workaround yet, but vFlash has been completely removed. I plan on migrating everything to vCSA instead of the Windows based vCenter we have been using.
What are the recommendations for the Veeam configuration? It started off with a single Veeam VM running everything except for storage. So it has the full version of SQL installed, management, and is a proxy. I have since added two more proxies. One is network mode only which I use to backup Exchange DAG. So the primary Veeam and secondary have 8 vCPU and are configured to run up to 6 concurrent jobs. Although the backup jobs are spaced far enough apart where they shouldn't be running anymore than 3 jobs per proxy. There are about 32 backup jobs set to run between 7:30 PM to 1:30 AM. I tried to space them apart so that there are not tons of snapshots being taken and removed at the same time.
The backup repository is Windows Storage Server 2012 R2 and is located in the same datacenter. It is connected directly to our core router which is connected to the storage switches.
The ESXi servers have plenty of resources. They are running version 5.5. Not update 1 due to the NetApp issues it has.
I pretty much have to keep babying these damn backups which is messing up my sleep schedule.
I hope Veeam 8 is going to fix this by switching to SAN based snapshots.Can I trouble you for some pointers on this? I'm also running 5.5 on a NetApp and just started my research into installing update1. I'm aware of the NFS APD issues with update 1 but my storage is all FC block.
I first noticed this issue in Patch 3 for Veeam 7 which is the current version I am on.
If you try to consolidate do you get an error similar to this?
If so, go look at the settings on your proxy server (servers depending on how many you have)
You will most likely see the following. When the hot add disks do not dismount from the machine they can't be hot added and will fail back to network path most of the time. These issues has caused our backup times to increase by hours cause veeam tries to clean up all the crap from the last backup, even though the job was marked as complete and not failed.
Every day i have 15+ machines that get stuck with hot add disks and consolidation fails, its become a pain in the ass to manage. R2 did not have this issue from what i recall.
If you try to consolidate do you get an error similar to this?
If so, go look at the settings on your proxy server (servers depending on how many you have)
You will most likely see the following. When the hot add disks do not dismount from the machine they can't be hot added and will fail back to network path most of the time. These issues has caused our backup times to increase by hours cause veeam tries to clean up all the crap from the last backup, even though the job was marked as complete and not failed.
Every day i have 15+ machines that get stuck with hot add disks and consolidation fails, its become a pain in the ass to manage. R2 did not have this issue from what i recall.
No, I am able to consolidate without an issue. I have seen the issue where the disk is still on the proxy, but that doesn't seem to happen too often. It seems like some of the snapshots are not being removed even though Veeam shows the backup job as 100% successful.
I run vRanger which is similar in function. We have a "helper" per host to perform the backup jobs. Backing up a large number of VMs on a single system concurrently can cause the backup jobs to take a lot longer to complete.
For larger VMs I would check for updated VMWare tools and VSS quiese settings.
For larger VMs I would check for updated VMWare tools and VSS quiese settings.
VMware Tools is up to date. I don't have VMware Tools Quiescence enabled on any backup jobs. For SQL and Exchange I use the Application-Aware Image Processing.
Am I supposed to be enabling "VMware Tools Quiescence" on Windows VMs?
dan__wright
Weaksauce
- Joined
- May 27, 2007
- Messages
- 91
Yes I have seen this, we would get the odd disk left mounted when we ran ESXi 4.5 with veeam, maybe once a week. As soon as I went to 5.5 it became a daily thing, my replicas were getting trashed and everything started to fall apart, probably 20 servers + a day with mounted snapshots in the proxies.
After much investigation with Veeam and VMware they said change from hot add to nbd. Now I've used nbd in the days of VCB and it sucked so I was hesitant but got desperate and finally tried it. I was shocked!
Not only did the left over hot add issues go away my backup throughput increased slightly and processing improved too (no longer needed to keep reconfiguring machines so can start disks quicker, we used to get quite a big cue at times) so give it a go, you may be pleasantly surprised.
After much investigation with Veeam and VMware they said change from hot add to nbd. Now I've used nbd in the days of VCB and it sucked so I was hesitant but got desperate and finally tried it. I was shocked!
Not only did the left over hot add issues go away my backup throughput increased slightly and processing improved too (no longer needed to keep reconfiguring machines so can start disks quicker, we used to get quite a big cue at times) so give it a go, you may be pleasantly surprised.
When it fails over to NBD, it does seem to work fine, but it causes a lot of network traffic over our management network. Our vCenter is setup with a single IP address on the management network. The proxies are setup with two IPs. One on the management network for communication with vCenter and the other on the storage network for backups. When backups are done in Virtual Appliance mode, the disks are added to the proxy and all traffic runs over the storage network. When in NBD, the snapshot is pulled through the proxy over the management network and sent to the backup repository over the backup network.
Since it is at night, it is not a huge deal, but I would prefer not to run backups over the management network.
dan__wright
Weaksauce
- Joined
- May 27, 2007
- Messages
- 91
The number of IPs of the management network shouldn't affect it (unless you mean multiple ports for throughput but depending on your setup you could LAG it)
It NBD doesn't cause issues and your proxies have ports on each subnet it shouldn't be an issue. Tbh at one point we were pulling from mgmt on one subnet, into proxy on a second subnet and onto remote site or backup array again on a different subnet and saw no issues (although that's with 2*10g per host and a pair of core routers that do just shy of 1tb/s)
It seems nbd has come a long way and it really does work well for me. You could even create a second kernel network connection for backups to use if you don't want to hit your management network although there isn't really any need.
It NBD doesn't cause issues and your proxies have ports on each subnet it shouldn't be an issue. Tbh at one point we were pulling from mgmt on one subnet, into proxy on a second subnet and onto remote site or backup array again on a different subnet and saw no issues (although that's with 2*10g per host and a pair of core routers that do just shy of 1tb/s)
It seems nbd has come a long way and it really does work well for me. You could even create a second kernel network connection for backups to use if you don't want to hit your management network although there isn't really any need.
So anyone have a decent explanation of why the orphan snapshots remain occasionally after Veeam completes successfully? There is absolutely no CPU or memory contention in this environment. Veeam shows that the snapshot was removed successfully. This issue has never occurred when performing a manual snapshot and removing it. It still happens when using Network Mode.
I opened a case with Veeam and they act like they have never seen this issue before. I can't imagine I am the only person that has orphan snapshots remaining after Veeam completes the backup job.
I opened a case with Veeam and they act like they have never seen this issue before. I can't imagine I am the only person that has orphan snapshots remaining after Veeam completes the backup job.
I will have to check. It is not high enough to trigger any alerts in our monitoring but I'm sure there is a decent amount of activity being that backups are running and possibly deduplication.
Tell me about it. That has pretty much be my solution to all the issues. Just wait for Veeam 8 and we will be good to go!
CiscoInside
Gawd
- Joined
- Mar 26, 2010
- Messages
- 546
I have had both of these issues and I am a big Veeam user. I have a client running constant replication I used to see Veeam actually destroy the vmx of the replica by not commenting something out. This was a PITA and actually this week we had the vmdk issue stuck on the target proxy.
Veeam answer is remove the vmdk and then run network mode. This is not an answer. I have brought this up to their lead engineers when they came out to see me last time I am going to hit them up again.
Veeam answer is remove the vmdk and then run network mode. This is not an answer. I have brought this up to their lead engineers when they came out to see me last time I am going to hit them up again.
SGalbincea
Weaksauce
- Joined
- Aug 14, 2008
- Messages
- 114
Cannot recommend this enough - PHYSICAL Veeam proxy/data mover with direct SAN access. I always virtualize the main Veeam server for protection purposes, but the machine that does the data moving is always a physical box with plenty of CPU and bandwidth to both the SAN and backup storage networks. There is no way to get better performance than this method in my experience. We can average above 350MB/sec. processing using this topology against 150+TB of data.
We are using NFS, which I don't believe works with Direct SAN Access.
So according to Veeam Support, when using Virtual Appliance and hot add fails and it fails over to Network Mode it is still not the same as setting the Proxy to Network Mode???
So they basically changed all our proxies to only Network Mode and said it shouldn't be a problem. I guess we will see.
So they basically changed all our proxies to only Network Mode and said it shouldn't be a problem. I guess we will see.
Well, changing to Network Mode definitely didn't help. Found a VM this morning that needed consolidation after last nights backup completed successfully.
Being that I can't use Direct SAN for backups, which would be recommended? Virtual Appliance or Network?
Being that I can't use Direct SAN for backups, which would be recommended? Virtual Appliance or Network?