Monday, 30 May 2016

VDP Full and File Level Restore Fails: Failed to get disks: Unable to browse as proxies are unavailable

So when trying to perform a full virtual machine restore or a File Level Restore for any virtual machine the task fails, at various possible percentages. The failure descried here is, 

Failed to get disks: Unable to browse as proxies are unavailable

If I try to restore this virtual machine on any other host, datastore or restore as a new virtual machine, replace existing virtual machine, the task still fails with the same error. 

The restore logs, in the following location shows:

Location: # cd /usr/local/avamarclient/var

The log with following MOD-*EPOCH*-vmimage1.log would be the restore logs:

The logs had the following;

2016-05-23T16:01:39.723-02:00 avvcbimage Error <0000>: [IMG0011] Timeout on wait for spawned restore metadata avtar process to complete
2016-05-23T16:06:11.947-02:00 avvcbimage FATAL <17824>: GetDiskAttributed Failed
2016-05-23T16:06:12.121-02:00 avvcbimage Error <17771>: Invalid request to create a VM.
2016-05-23T16:06:12.121-02:00 avvcbimage Error <0000>: [IMG2012] VM creation failed during restore.

Here the timeout is occurring while the AvVcbimage is spawning the avTar for metadata restore. This in-turn is failing to create the virtual machine for restore. 

The resolution:

Increase the AvVcbimage timeout using the below steps:

1. If the VDP is using external proxy, then you will have to SSH to your proxy machine. If the VDP is using internal proxy, then we will have to use the SSH to VDP appliance. Change the directory to:
# cd /usr/local/avamarclient/var
2. Edit the following file:
# vi vvcbimage.cmd
3. Add the below and save the file
4. Restart the avAgent service using the below command:
# avagent-vmware restart

Run the restore operation now, and it should work good. 
If not, leave a comment! Thank you.

Friday, 20 May 2016

Fatal Error While Creating Storage For A New VDP Appliance Deployment

After you deploy the ova template for VDP we need to configure the VDP appliance from the vdp-configure page. During the storage configuration, I had chosen about 8 TB of storage for the VDP and during the configuration progress, the setup encountered a Fatal Error. The exact message and the screenshot is:

“A fatal error occurred during the storage configuration and the appliance is unrecoverable. This could be due to the datastore not having enough free space, does not support the required VMDK size, or is in inactive state. Please check the vCenter Tasks pane for the exact error and redeploy a new VDP appliance”

Couple of pre-checks:

1. Make sure the destination VMFS / NFS datastore has sufficient space to accommodate the selected VDP data storage disks. 
2. If we are using VMFS datastore, make sure that the block size for the VMFS datastore supports the VMDK size being created on it
3. Then lastly, check for the permissions for the user that the VDP is being registered to vCenter with, also called as VDP user. The user should have disk create role, or best yet, administrator role. 

So, I used a domain user with administrator privilege or you can also use the SSO user to get the VDP appliance to registered to the vCenter server. 

The disk creation completed successfully and I was able to continue to use the VDP appliance. 

Tuesday, 17 May 2016

Unable To Connect VDP 6.1 To Web Client

So once the new VDP 6.1 appliance is deployed, we login to Web Client and select the vSphere Data Protection plugin. From the drop-down, we will select the required VDP appliance and click connect. However, upon performing this, the screen grays out forever, and this operation does not fail either with an error. 

The thing to notice here is:

1. This issue occurs when the VDP 6.1 is on a distributed switch. 
2. If the VDP virtual machine is migrated to a standard switch, the appliance is able to connect to the web client
3. If the VMs that need to be backed up are on distributed switch, then the backup job create task grays out forever. 
4. If the entire environment is migrated to standard switch, the working goes back to normal. 

So all in short, VDP 6.1 on a vDS environment has issues. Now, since migrating your entire networking to standard switch is obviously not a feasible or recommended task, there is a hot-patch released to fix this. 

Now, I am sharing these steps here along with the patch is solely because, I have distributed the patch to every customer who had opened a case with us to get this fixed along with the steps, so why not share it here for ease of access. 

Before we get to the resolution, this is what was noticed in the virgo logs in vCenter when the connect option was clicked when the VDP appliance was on a distributed switch.

[2016-04-11 18:36:45.604] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.604] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.
[2016-04-11 18:36:45.608] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.608] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.
[2016-04-11 18:36:45.612] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.612] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.
[2016-04-11 18:36:45.616] [INFO ] http-bio-9443-exec-3 System.out [BlazeDS]Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'.
[2016-04-11 18:36:45.616] [INFO ] http-bio-9443-exec-3 System.out flex.messaging.MessageException: Cannot create class of type 'com.vmware.vim.binding.vim.dvs.PortConnection'. Type 'com.vmware.vim.binding.vim.dvs.PortConnection' not found.

So the resolution:

1. Download the vdr-ui-war-6.1.2.war file from this link here.
2. Determine the version of vCenter and navigate to one of the following directories accordingly:

5.5 vCenter
C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity\com.vmware.vdp2-<version>\plugins

6.0 vCenter

3. Here you can see two files, a .jar file and a .war file. 
4. Rename the existing .war file to vdr-ui-war-6.1.x.war_old (where x is the version of your plugin)
5. Copy the patched .war file in the attachment, and paste it, named accordingly into this folder. 
6. In vCenter appliance 6.0, I had to perform couple of additional steps (Windows vCenter does not require these steps). The older jar and war file were having the following permissions:

Owner: vsphere-client
Group: Users

7. The applied patch has Owner and Group as root. Change this accordingly. 
You can do this easily by opening a WinSCP connection to vCenter > Right click the File > Properties and these two options will be available here. 
8. Restart the web client service. 

Windows, you can find this service in services.msc

service vsphere-client stop
service vsphere-client start
9. Login to web client > Connect, and now we should be able to successfully connect to the appliance from web client. 

That's pretty much it. 
If something does not work, comment below! And always take care while patching.

If the connect still fails, then discard the old renamed .war file, restart the web client service and try connecting again.

Thank you!

Friday, 13 May 2016

VDP Backup Job Not Appearing In Recent Tasks

This is going to be a small article for few issues that I worked on recently. When you login to vSphere Web Client and run a backup job, the task progress does not show up under the "Recent Tasks" pane. When you login to vSphere Windows client, you can see the Create Snapshot and VDP backup job task. And in Web Client, when you click the Tasks section on the left hand side, you can see this task. 

This was the situation when my VDP user (The user with which I configured my VDP to vCenter) was different to the user that I had currently logged into web client with. 

The VDP user that I have configured is vsphere.local\Suhas and the user that I was logged in was vsphere.local\Administrator (Single Sign On user)

You cans see the user with which VDP is registered below:

Next, we will login to web client with the user that the VDP was registered to vCenter with, which is, vsphere.local\Suhas. And once logged in, I started another backup job and this time I can see the task in the "Recent tasks" pane. See screenshot below:

The resolution is simple, really simple. When you are logged in with a different user when compared to the user with which the VDP is registered with, Under the Recent Tasks pane, you have a drop-down which reads, My Tasks. Click this drop-down and select All Tasks, now you can see all the VDP backup tasks. See the screenshot below for the different user (SSO user)

That's it, really!

Wednesday, 11 May 2016

VDP File Level Recovery Client

So, you would be familiar with restoring virtual machines with VDP which is done from the Restore tab of the VDP appliance from the Web Client GUI. However, there are cases, where the necessity is to restore only certain files or folders and not the entire virtual machine or virtual disk. In this case, instead of using disk restore or virtual machine restore we perform File Level restore. 

Like your VDP configure page, FLR also has a client. The FLR client has to be accessed on the virtual machine where you want to perform the file restores. The URL for this would be:


The login screen would look something like this:

This is the simple login page and there is an option for advanced login as well. We will have a look at the advanced login a little later in this article. 

For the simple login, you will have to provide the Windows machine's local administrator credentials. Now, for a small demo, on the machine where I have accessed the FLR, I had a file under "Desktop" directory called "Critical data" which had some data called "Critical Text". To give a little more background, once this file was created, a regular backup was executed for this virtual machine. The backup completed successfully. However, some hours later the file was lost. Now, we will be restoring that file using the FLR client. 

So I have logged in with my local admin credentials to the FLR client and this is the first screen I would see after a successful login. 

I have just one restore point created for this virtual machine since only one backup was taken. If you have multiple restore points after a set of backups, you can choose the restore point that you think is the best and select the option called Mount

You will then see this directory view. I will expand the drive and folders until, the necessary file is located. So my file was created under C > Users > Administrator > Desktop. Here the critical data text file is seen.

I will select only this file to restore and select the "Restore selected files" option. Upon selecting that you will have to provide a location as to where you want to restore the file. I will select the destination where it was originally residing. If you want to specify a different destination, you can do the same as well here. Click the Restore option once the directory is decided. 

You will receive a generic prompt stating that the operation might be time consuming. Select Yes to start the restore. 

Click Monitor Resources tab to check the restore status. Here the progress is not shown, and the only view available is as the one seen below:

From the vSphere client you can monitor the actual status of the restore.

Once the restore is completed successfully you can go to that directory and verify the file is restored successfully. 

The next option is restoring using the advanced login method. The advanced login method is used when, the machine where you actually want to restore the file has no access. In this case, with the advanced login, you will provide the credentials of the vCenter along with local credentials of the machine. 

The login would something like below:

Here the restore point screen looks a bit different. Instead of seeing only the restore point of the virtual machine, you will be seeing all the restore points for the virtual machines which were backed up.

Click the drop down for the virtual machine required, or you can also filter by name to locate it quickly. Click the required restore point from the list after selecting the drop-down and click mount. 

The remaining process is going to be same, you can restore the file to the same location or you can restore the file to a different location. 

One more thing to note:
Let's say, initially you had a text file with contents ABC. Then you took a backup of the virtual machine. After the backup, you made changes to the text file so your contents are now ABCDE. Now, if you restore this file on the same directory where the original file exists then the new contents of the file will be over written with the contents present in the restored file. So make sure that restore is done appropriately. 

That's it!

To know more about limitations of FLR you can click this link here to read the VDP admin guide. (See page 152)

Error While Binding iSCSI To VMKernel Adapters: IscsiManager.QueryBoundVnics

So when you try to click Properties of the iSCSI adapter to modify certain settings, you see the following error:


Call "IscsiManager.QueryBoundVnics" for object "iscsiManager-##" on vCenter Server "VC-name" failed


This event occurs when ESXi's internal iSCSI daemon becomes corrupted, requiring a cleanup of the daemon's files in ESXi.

How to resolve this:

1. Migrate all virtual machines using vMotion to ESXi hosts that are unaffected.
2. Review the existing iSCSI configuration and copy the IQN and adapter settings for your iSCSI software adapter to a text document.
3. Disable the iSCSI adapter.
4. Navigate to the /etc/vmware/vmkiscsid/ directory on your ESXi host and back up the contents of the folder to a safe location.
5. Delete the contents of /etc/vmware/vmkiscsid/
6. Write the changes to the ESXi boot bank using this command:
7. Reboot the ESXi host.
8. Create a new software iSCSI adapter and configure it as per the backup you saved in step 2.
9. Add the iSCSI port bindings and targets.

That's pretty much it. You should be good to go.

Tuesday, 3 May 2016

VDP Backup With External Proxy

So in the previous article we saw the types of backup protocol used by VDP. We saw that when the ESXi host which is hosting the VDP appliance does not have the access to the datastore of the VM it is backing up, then the backup protocol used is Network Block Device (NBD). One way to avoid NBD is to bring the VM on to a storage which is seen by the ESXi host hosting the VDP appliance. The other method is to deploy an external proxy for the VDP appliance onto the host which is hosting the client that needs to be backed up. Please note, that once you deploy external proxy, the internal proxy will be disabled automatically. 

To add external proxy in detail, you can visit this link here.

From the below screenshot you can see that the external proxy is deployed on the host where the client that requires to be backed up resides. 

This will create a virtual machine for the proxy on the specified host.

Now, ideally, when I run the backup for this client without external proxy, it should work with the NBD protocol. However, now the proxy where the avAgent runs, can see the datastore of the client virtual machine, it will proceed with the Hot-Add protocol. 

Since avAgent runs on the proxy which talks to the MCS, the disk will not be mounted on the VDP appliance, instead it will be mounted on the external proxy virtual machine. You can see this from the below screenshot. 

vdp_ba_vma41 is the proxy virtual machine, and the client being backed up is CentOS7 and we can see the virtual disk of client is mounted on the external proxy virtual machine. 

Well, that's one of the use case of deploying external proxy.

Monday, 2 May 2016

VDP Hot-Add vs NBD

When a backup job is initiated for a virtual machine, there are two methods or protocols by which VDP performs the backup. 
The first one we have is the Hot-Add protocol, wherein the virtual disks of the virtual machine being backed up is going to be mounted on to the VDP appliance and then the backup will be performed. 
The second one is using Network Block Device (NBD) protocol, where in the virtual disk backup is done via the management network.

In the below screenshot, you can see the VM, New Virtual Machine, resides on Recovery_LUN datastore. 

From the next screenshot you can see that the VDP appliance virtual machine is residing on VDP-Datastore.

For the ESXi host where the VDP appliance resides, the storage configuration page shows that it does not have access to the Recovery_LUN datastore. This means the ESXi host hosting the VDP appliance does not see the virtual machine's datastore. 

In this scenario, where the ESXi host where the VDP appliance does not see the VMs datastore, Hot-Add protocol is not going to work and it falls back to the NBD protocol. 

Here are a few test results while using the NBD protocol:

I have a backup job called "ble" (Please ignore the name) and this contains the virtual machine named, New Virtual Machine. 

The next thing I did was, I started a manual backup job for this virtual machine. The backup job as usual initiates two tasks, create a virtual machine snapshot and a VDP: Backup Job.

When you go to Snapshot Manager for the virtual machine being backed up, you can see the snapshot created by the VDP, goes by VDP-EPOCH of backup.

And if I check the Edit Settings of the VDP appliance, I do not see the disk of the New Virtual Machine Hot-Added to the VDP appliance.

So now, the backup traffic will be flowing via the management network. So I performed a little esxtop test to check for this information. 

From the SSH session for the host which is hosting the virtual machine being backed up, I saw the following:

The virtual machine and the management network resides on the vSwitch0 with a vmnic2 uplink and we can see that the PKTTX/s is high and rising and dropping accordingly throughout the backup process. 

Next, if I SSH to the destination ESXi host where the VDP appliance is running, I see the following:

The VDP appliance is on vSwitch0 and is connected to an uplink vmnic0 and we can see the PKTRX/s is similarly high and is almost at the same rate of PKTTX/s of the virtual machine being backed up. 

Next, the observations of Hot-Add:
I am going to migrate this VM to a datastore which is seen by the ESXi host hosting the VDP appliance. 

So I migrated the VM to be backed up to the same host as my VDP appliance (As this host does not have any shared datastore) and put it on a different datastore, Suhas-Local-2. From my above screenshots you can see the Suhas-Local-2 is a datastore on my VDP hosting ESXi host. 

I then ran the backup job again, and this time, if I come to the Edit Settings of the VDP appliance I see that there is a disk added to the VDP appliance and this corresponds to the virtual disk of the VM being backed up. 

Additional information:
If you have virtual flash read cache disks (VFRC) in your environment, then those virtual machines have to be on virtual hardware 10 and above. When your VDP is on a lower version than the VMs with VFRC disks, then the backup protocol is always going to be NBD regardless if the ESXi host can see the datastore or not. 

Well, that's pretty much it!