Thursday, 2 March 2017

VDP Backup Fails After A Storage vMotion

If you have a virtual machine added to a VDP backup job and then a Storage vMotion is performed, the next backup of this client might fail. The failure error you will see in the vSphere Client is:

VDP: An unexpected error occurred with the following error code: 10058



The backup jobs log would record the following:

2017-03-02T03:18:42.561-05:-30 avvcbimage Info <18664>: Login(https://vcenter-prod.happycow.local:443/sdk) Datacenter: 'HappyCow-DC'
2017-03-02T03:18:42.561-05:-30 avvcbimage Info <19728>:      - connected to 'VirtualCenter' - version: 'VMware vCenter Server 6.0.0 build-3634793',  apiVersion:'6.0'
2017-03-02T03:18:42.604-05:-30 avvcbimage Warning <16004>: Soap fault detected, Find VM - NOT ok, Msg:''
2017-03-02T03:18:42.605-05:-30 avvcbimage Error <0000>: [IMG0014] Problem opening vCenter:'HappyCow-DC', path:'[datastore1 (1)] VM-A/VM-A.vmx'.
2017-03-02T03:18:42.605-05:-30 avvcbimage Info <9772>: Starting graceful (staged) termination, Failed to log into web service. (wrap-up stage)
2017-03-02T03:18:42.606-05:-30 avvcbimage Warning <40657>: Login failed
2017-03-02T03:18:42.606-05:-30 avvcbimage Info <40654>: isExitOK()=208
2017-03-02T03:18:42.606-05:-30 avvcbimage Info <17823>: Body- abortrecommended(t)
2017-03-02T03:18:42.606-05:-30 avvcbimage Info <40658>: vmparams (vcenter-prod.happycow.local)
2017-03-02T03:18:42.606-05:-30 avvcbimage Info <40654>: isExitOK()=208
2017-03-02T03:18:42.615-05:-30 avvcbimage Info <18664>: Login(https://vcenter-prod.happycow.local:443/sdk) Datacenter: 'HappyCow-DC'
2017-03-02T03:18:42.616-05:-30 avvcbimage Info <19728>:      - connected to 'VirtualCenter' - version: 'VMware vCenter Server 6.0.0 build-3634793',  apiVersion:'6.0'
2017-03-02T03:18:42.651-05:-30 avvcbimage Warning <16004>: Soap fault detected, Find VM - NOT ok, Msg:''
2017-03-02T03:18:42.651-05:-30 avvcbimage Error <0000>: [IMG0014] Problem opening vCenter:'HappyCow-DC', path:'[datastore1 (1)] VM-A/VM-A.vmx'.
2017-03-02T03:18:42.658-05:-30 avvcbimage Info <18664>: Login(https://vcenter-prod.happycow.local:443/sdk) Datacenter: 'HappyCow-DC'

A similar scenario is discussed in this release notes here. Refer section "Backup of a VM fails when ESX is moved from one datacenter to other within same vCenter inventory (207375)"

This is a known issue that occurs in the VDP due to the vmx_path variable not being updated after a Storage Migrate operation.

In this case the initial datastore of my virtual machine was datastore1 (1) and then later it was moved to datastore2. However, post this migration the backup job still picked the vmx_path as datastore1 (1) and since the VM is no longer available there, the backup failed.

Couple of solutions:
The update of this cache value will take up-to 24 hours. So, it should automatically update after some time without any manual changes.

Another workaround is to perform another storage migrate of this virtual machine to any datastore.

To force sync the cache use the "mccli vmcache sync" command from the above release notes.

A sample command would be:
# mccli vmcache sync --domain=/vcenter-FQDN(or)IP/VirtualMachines --name=<VM-Name>

Post this, run a backup to verify if things are in place. Hope this helps!