Friday, 22 April 2016

Cannot Perform A Backup With VDP 5.1

So while performing a backup of any virtual machines in my environment on VDP, the backup task used to fail. So, I created one backup job, with one virtual machine in it and ran the backup task. The create snapshot task completes successfully and you can see the VDP snapshot in the snapshot manager of the virtual machine. Once the snapshot is taken, the next task initiated will be the VDP backup task, and this fails immediately (no percentage completed). It states it failed with miscellaneous errors. 

When you open a SSH to the VDP appliance and browse the backup-job log, from the directory, /usr/local/avamarclient/var-proxy-N, I see the following logging:

2016-04-20 08:28:50 avvcbimage FATAL <5889>: Fatal signal 6 in pid 553
2016/04/20-15:28:50.11148 []  FATAL ERROR: <0001> Fatal signal 6
2016/04/20-15:28:50.11161 []  | 00000000005f39f1
2016/04/20-15:28:50.11162 []  | 00000000005f46b7
2016/04/20-15:28:50.11163 []  | 00000000005f5bdb
2016/04/20-15:28:50.11164 []  | 00000000005f5cce
2016/04/20-15:28:50.11168 []  | 000000000058b800
2016/04/20-15:28:50.11169 []  | 00007ff23c7c16a0
2016/04/20-15:28:50.11170 []  | 00007ff23b0797a9
2016/04/20-15:28:50.11170 []  | 00007ff2308a8900
2016/04/20-15:28:50.11171 []  | 00007ff23087b9ac
2016/04/20-15:28:50.11172 []  | 00007ff2306bf442
2016/04/20-15:28:50.11173 []  | 00007ff2362c472e
2016/04/20-15:28:50.11174 []  | 00007ff2362cff4c
2016/04/20-15:28:50.11174 []  | 00007ff230898d17
2016/04/20-15:28:50.11175 []  | 00007ff230894883
2016/04/20-15:28:50.11176 []  | 00007ff23c7b9696
2016/04/20-15:28:50.11177 []  | 00007ff23b07cd7d
2016/04/20-15:28:50.11184 []  ERROR: <0316> handlefatal exiting thread pid=553, sig=6
2016-04-20 08:28:50 avvcbimage Error <5891>: handlefatal: exiting thread pid=553, sig=6
2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:2016-04-20T08:28:50.113-07:00 [7FF22EE0D700 panic 'Default']

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:-->

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:--> Panic: Assert Failed: "_lockToken != __null" @ /build/mts/release/bora-774844/bora/lib/vcbLib/hotAdd.cpp:638


Here the appliance is failing to lock the VMDK which it has to backup and entering a panic state causing the backup to fail. 

The 5.1 VDP has a VDDK version of 5.1. VDDK is a set of C++ libraries that is used to access and create virtual disk. The version of VDDK can be found out from the backup job log (Just search for VDDK and the first few lines will define the version)

You will see the entry for VDDK version as follow (This was on my 6.1.2 VDP)
2016-04-18T20:00:15.978+07:00 avvcbimage Info <16041>: VDDK:VMware VixDiskLib (6.0) Release build-2498720

Now, back to the failed backup. This is a known issue with 5.1 VDDK version and this can be verified from the 5.5.1 (5.5 Update 1) VDDK release notes. (Search for the section: Assert failure causes exit when HotAdd cannot acquire lock)

Upgrade your VDP to 5.8 and we should be good to go as it updates the underlying VDDK version. Re-run a backup after the upgrade and check the backup logs again and you can see the new version of VDDK release and it's corresponding build number.