Friday, 1 July 2016

Perform A VDP Backup When The Datastore For Client VM Is Running On Low Space

If you try to backup a virtual machine which is residing on a datastore running low on space, the backup fails. The VDP will try to issue a snapshot take call, however, the task is not initiated for either Create Snapshot or VDP: Backup. Instead you see the following in the reports tab. 

VDP: Failed to create snapshot

The backup job log located at /usr/local/avamarclient/var will have the following logging:

2016-06-30T09:59:11.371+07:00 avvcbimage Info <19704>: DataStore Storage Info:Local-esxi02 capacity=4831838208     free=138412032
2016-06-30T09:59:11.372+07:00 avvcbimage Info <19716>: DS Capacity=4831838208     FreeSpace=138412032      / HD committed=3951319082     unCommitted=1680           unShared=3758096384
2016-06-30T09:59:11.372+07:00 avvcbimage Info <19717>: DS(Local-esxi02) does not have enough free space (138412032     ) for disks used (197565952).
2016-06-30T09:59:11.372+07:00 avvcbimage Error <19661>: Datastore does not have enough free space for snapshot
2016-06-30T09:59:11.372+07:00 avvcbimage Info <9772>: Starting graceful (staged) termination, failed to create snapshot (wrap-up stage)
2016-06-30T09:59:11.372+07:00 avvcbimage Error <0000>: [IMG0009] createSnapshot: snapshot creation  or pre/post snapshot script failed
2016-06-30T09:59:11.372+07:00 avvcbimage Error <0000>: [IMG0009] createSnapshot: snapshot creation/pre-script/post-script failed
2016-06-30T09:59:11.372+07:00 avvcbimage Info <40654>: isExitOK()=202
2016-06-30T09:59:11.372+07:00 avvcbimage Info <40659>: snapshot created:false NOMC:false ChangeBlTrackingAvail:true UsingChBl:true, ExitOK:false, cancelled:false, fatal: true
2016-06-30T09:59:11.372+07:00 avvcbimage Info <40654>: isExitOK()=202
2016-06-30T09:59:11.372+07:00 avvcbimage Info <40660>: vcbimage_progress::terminate
2016-06-30T09:59:11.373+07:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_EndAccess: Disk access completed.

Now there is a parameter that can be added to avvcb daemon to ignore the free space on the datastore and still proceed with the backup operation. This workaround is tricky, why because let's consider the following situation:

You have a virtual machine of about 500 GB on a datastore of 550 GB. So the free space on the datastore is 50 GB. Let's say we added the parameter to ignore this and still take a backup, and the new data that has come into the VM has reached the 50 GB free space limit, then the VM will straight up stop to function because it has no space to get the new data. At this point of time, you will end up expanding the datastore so that the VM can be powered on. 

This is why, it is always recommended you expand the datastore then perform a backup. Now, there are certain cases where that 500 GB VM is a file server where no new data is coming in, In that case, we are good to add this parameter and run the backup. 

How do we do this?

1. SSH into your VDP appliance and change to the following directory:
# cd /usr/local/avamarclient/var
2. You will have a file called as avvcbimageAll.cmd
3. Open this file using a vi editor and edit the file to add the following parameter:
--snapshot_max_change_percent=0
4. Restart the avagent daemon using the below command
service avagent-vmware restart
5. Now you should be able to run the backup job even when your datastore is running low on space. 

Note:
If you are using internal proxy, then this step will be done on the VDP appliance itself as the avvcbimage lies on the appliance itself. If you are using external proxy, then you will have to edit the avvcbimageAll.cmd file on the proxy machine, as the proxy VM would be responsible for opening/closing VMDK as this machine runs the avvcb daemon. Also, if you have multiple proxy VMs, then add this parameter to all your external proxy machines and restart the avagent service on all of them. 

This is supported on 6.1 VDP as well.