Tuesday, 31 January 2017

Unable To Send Test Email In VDP 6.x

In VDP, you can configure SMTP so that the appliance can forward the appliance status, backup job status periodically to the email configured. Once you configure all the settings in Configuration > Email, you will use the Send test email option to check if this configuration is working good. However, once you select this, you might receive the following message:

The troubleshooting to this would be to verify if the SSO, vCenter and VDP time are all synchronized. More about this is discussed in the article here.

Now there are two ways of looking at this issue:

1. If you had the NTP issue mentioned above and you corrected it. But still you are running into the same message over and over again though the time is synced. (This is the issue I worked on)

2. Perhaps you are receiving something else, a different error. But the vdr-server.log stack trace matches below:

2017-01-31 13:07:22,607 ERROR [http-nio-8543-exec-3]-email.EmailBase: On try 1to send the email, recieved a connection error from the external email server:null
2017-01-31 13:07:22,607 ERROR [http-nio-8543-exec-3]-services.EmailConfigServiceWS: Unable to trigger test email
        at javax.mail.Service.connect(Service.java:306)
        at javax.mail.Service.connect(Service.java:156)
        at javax.mail.Service.connect(Service.java:105)
        at javax.mail.Transport.send0(Transport.java:168)
        at javax.mail.Transport.send(Transport.java:98)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:174)
        at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:63)
        at com.emc.vdp2.email.EmailSummaryReport.sendTestEmail(EmailSummaryReport.java:129)
        at com.emc.vdp2.services.EmailConfigServiceWS.runEmailTest(EmailConfigServiceWS.java:93)
        at com.emc.vdp2.rest.ReportService.runEmailTest(ReportService.java:241)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.emc.vdp2.services.AuthenticationFilter.doFilter(AuthenticationFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:613)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)

This is only specific to the error javax.mail.AuthenticationFailedException which was observed in the above log. If you do not see this, then this will probably not help you. 

Again two causes for this:

1. If you ran into the same NTP issue, then the credentials are not updated or synced prior and post to the time change
2. You are indeed entering a wrong password.

If the issue is (1), then Edit the SMTP settings and re-enter the password again post which the Test email should work. If the issue is (2) then you will have to get the right password. 

Hope this helps. 

Friday, 27 January 2017

VDP Restore Fails With "Operation Not Permitted"

Today, a restore of a virtual machine was failing. Infact, restore of all virtual machines were failing. The data was sent to a data domain, but regardless of what the backend storage is, this issue would still occur.

In the restore logs under /usr/local/avamarclient/var with the keyword MOD-EPOCH-vmimage1.log, the following was seen:

2017-01-26T13:11:53.027+04:00 avvcbimage Info <6688>: Process 13007 (/usr/local/avamarclient/bin/avtar) finished (code 1: Operation not permitted)
2017-01-26T13:11:53.027+04:00 avvcbimage Info <7238>: Stream complete: 0 bytes read from stream
2017-01-26T13:11:53.027+04:00 avvcbimage Warning <6690>: CTL workorder "MOD-1485454308259" non-zero exit status 'code 1: Operation not permitted'
2017-01-26T13:11:53.028+04:00 avvcbimage Error <0000>: [IMG0011] Avtar exited with 'code 1: Operation not permitted'
2017-01-26T13:11:53.028+04:00 avvcbimage Info <19647>: Restore Job Wrap-Up Stats:

The avtar log for the same had this:

2017-01-26T13:11:53.025+04:00 avtar Warning <10786>: Cannot use --verbose with --stream=&1, remove the verbose option flag to stop this error

The restore progress used to fail right at 0 percent. 

As the error message stated, we had some additional unwanted entry in the avtar.cmd file. To resolve this:

1. Go to /usr/local/avamarclient/var
2. Open the avtar.cmd file in a vi editor
3. Remove the " --verbose " flag from this and save the file
4. Restart the avagent using:
# service avagent-vmware restart

Post this, the restore should work without issues.

Hope this helps.

Wednesday, 25 January 2017

VDP Verification Job Leaves Undeleted Folders

There was an interesting case opened quite a while back which took almost few months to figure out the solution. The issue was, a backup verification job was performed for a set of VMs. The verification job used to complete successfully, however, it used to leave a folder behind with the name of the VM's verification job ID and in that folder was one file which was the vmxf file of that virtual machine.

If you took a look at the verification job logs on the VDP server, you will not see any errors or warnings. The VM will be test restored, powered on, tested for VM tools heartbeat and then powered off and deleted.

So, on the ESXi datastore, the folder left behind was something similar to this name, VDP_VERIFICATION_Windows_clone_2_1485298729541. And in this directory the file left behind was VDP_VERIFICATION_Windows_clone_2.vmxf

If you look at the hostd.log during this process, you will notice the following:

2017-01-24T23:44:04.127Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.nvram'.

2017-01-24T23:44:04.128Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/vmware.log'.

2017-01-24T23:44:04.128Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/564d7f1f-1606-518d-6b4e-d8689735d481.vmem.WRITELOCK'.

2017-01-24T23:44:04.128Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.vmsd~'.

2017-01-24T23:44:04.128Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.vmsd.lck'.

2017-01-24T23:44:04.129Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteFile dele
ted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.vmsd'.

2017-01-24T23:44:04.129Z info hostd[234C2B70] [Originator@6876 sub=vm:SNAPSHOT: SnapshotDeleteFile deleted '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFI
CATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.vmx opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] lck'.

2017-01-24T23:44:04.130Z info hostd[234C2B70] [Originator@6876 sub=Libs opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] SNAPSHOT: SnapshotDeleteVMInt: Couldn't remove directory '/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541'.

2017-01-24T23:44:04.130Z verbose hostd[234C2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/57ac84be-efa08410-3cf9-549f3515a09a/VDP_VERIFICATION_Windows_clone_2_1485298729541/VDP_VERIFICATION_Windows_clone_2_1485298729541.vmx opID=3a74dd6f-58-d7b3 user=vpxuser:VMWARE.LOCAL\Administrator] Virtual machine delete completed.

If you notice here, the verification job is removing all the test restored files. However, it says it is unable to remove the VDP Verification directory. At this point, if you run a list command in the datastore of ESXi, you will notice there is a vmxf file present. And because of this VDP is unable to remove the verification folder.
During backup, VDP will only backup vmx, nvram, vmdk and flat files. It will not backup the vmxf file. So if it does not backup the vmxf file, it will neither restore the vmxf file. So the question is, what is this vmxf file? Vmxf file was initially created to contain the virtual machine meta data. Then later on, the tools information was added to this file. So what this means is, if you have a virtual machine with VM tools running, you should see a vmxf file associated with that VM. If there is no tools installed on a VM, then there will be no vmxf file.

So, if you perform a ABV of a VM without any tools, then you will not have this vmxf file left behind. So during the ABV, the VDP will test for VM tools heartbeat. So when the VM is powered On, the vmxf file gets created under the ABV VM folder. But since VDP did not create this file, it will neither delete this file. It will delete all the files that was restored by it, leaving this vmxf file under the verification folder. And the clean up in this case has to be done manually.

There is no much use of vmxf file in your production environment. If you delete this file manually and take a new backup of the VM and verify the restore point, the file is still coming back. This means that something in the "vmx" file is causing the "vmxf" file to appear.

The resolution:

1. Power off the virtual machine
2. If you vi the virtual machine's vmx file, you will see the following entry:
extendedConfigFile = "Windows_clone_2.vmxf"

3. Use the vi editor to hash this entry out.

Before edit:
extendedConfigFile = "Windows_clone_2.vmxf"
After edit:
 #extendedConfigFile = "Windows_clone_2.vmxf"

4. Save the file and reload the virtual machine's vmx. To reload, you will have to find the VM ID. To do this, run the below command:
# vim-cmd vmsvc/getallvms
5. Once you get the ID, reload it using:
# vim-cmd vmsvc/reload <vm-id>
6. Remove the vmxf file manually from the virtual machine directory

7. Power On the VM. Wait for the VM to be completely up and the tools status to be Running (Current) Now, if you browse the VM directory, the vmxf is no longer created.

8. Perform another backup for this VM and then perform the verification and this time there will be no stale files left behind.

If you make any changes to the VM tools, such as reinstall or upgrade, then another entry for vmxf will be added in the virtual machine's vmx file and this entire process has to be re-done.

Hope this helps!

Saturday, 21 January 2017

VDP 6.1.3 Backups Hang At 92 Percent When Backing Up VMs On vSAN

Earlier, we had seen an issue where the VDP appliance crashes when we backup a virtual machine residing on a vSAN datastore. This was documented in this VMware KB article here.

On VDP 6.1.3 we use VDDK version 6.5, VixDiskLib (6.5) Release build-4241604. The earlier issue in VDDK 6.0 as mentioned in the above KB article was unable to mount a vSAN sparse delta disk on a proxy server or the VDP server (when using internal proxy) when it was residing on a non vSAN datastore.

In VDDK 6.5, the behavior was changed. The issue still exists but rather than crashing the VDP appliance, the VDDK library crashes causing the backups to fail. The workaround as per the same article was to:

1. If internal proxy was being used, then deploy an external proxy and place it on the vSAN datastore.
2. If space usage is not an issue and internal proxy is being used, move the entire VDP appliance to the vSAN datastore.

However, in some cases this did not work and the backups still used to fail, however the task and events in vSphere would show the progress at 92 percent and mccli activity show --active would not show any progress bytes.

I do not have a vSAN storage in my setup, the back-trace I am using is from couple of cases that I worked on and a community discussion thread which can be found here.

If you view the backup failure vimagew.log under /usr/local/avamarclient/var, you will notice the following back-trace:

2017-01-17T04:02:16.101+04:00 avvcbimage FATAL <5889>: Fatal signal 11 in pid 15065
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  FATAL ERROR: <0001> uapp::handlefatal: Fatal signal 11
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006b93b1
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba127
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba44b
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000006ba53e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 0000000000628810
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c90850
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5050193f
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c277
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5051c75e
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b142
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e5052b678
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecab73
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56ecb123
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004284bc
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004291f9
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00000000004993c4
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 000000000070ac4c
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e56c88806
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  | 00007f2e558cd9bd
2017-01-17T04:02:16.103+04:00 [VcbImageBackupAssistThread]  ERROR: <0001> uapp::handlefatal: aborting program pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage FATAL <5890>: handlefatal: Aborting program with code 176, pid=15065, sig=11
2017-01-17T04:02:16.103+04:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1.

To fix this, change the protocol of backup to nbd. The steps would be:

1. SSH into VDP and browse to the below location:
# cd /usr/local/avamarclient/var
2. Edit the file called as avvcbimageAll.cmd using a vi editor

3. Add the below parameter into this file:

4. Save the file and restart the avagent service using:
 # service avagent-vmware restart

Now your backups should complete successfully. I will update this thread when a fix is released for this bug.
Hope this helps.

Monday, 16 January 2017

Slow GUI Response On VDP 6.1.3

I recently ran into an issue, where the Backup Job tab was extremely slow in loading the jobs and when I say extremely slow it was taking forever to load the jobs. This was the same with the other tabs as well in the Web Client VDP GUI.

In the axis2.log under /usr/local/avamar/var/mc/server_log the following was logged:

2017-01-16 12:57:35,105 [1690894503@qtp-1786872722-26] ERROR org.apache.axis2.transport.http.AxisServlet  - Java heap space
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuffer.append(Unknown Source)
        at java.io.StringWriter.write(Unknown Source)
        at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1103)
        at com.ctc.wstx.sw.BufferingXmlWriter.fastWriteRaw(BufferingXmlWriter.java:1114)
        at com.ctc.wstx.sw.BufferingXmlWriter.writeStartTagEnd(BufferingXmlWriter.java:743)
        at com.ctc.wstx.sw.BaseNsStreamWriter.closeStartElement(BaseNsStreamWriter.java:388)
        at com.ctc.wstx.sw.BaseStreamWriter.writeCharacters(BaseStreamWriter.java:446)
        at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.writeCharacters(XMLStreamWriterWrapper.java:100)
        at org.apache.axiom.om.impl.MTOMXMLStreamWriter.writeCharacters(MTOMXMLStreamWriter.java:289)
        at org.apache.axis2.databinding.utils.writer.MTOMAwareXMLSerializer.writeCharacters(MTOMAwareXMLSerializer.java:139)
        at com.avamar.mc.sdk10.EventMoref.serialize(EventMoref.java:155)
        at com.avamar.mc.sdk10.EventMoref.serialize(EventMoref.java:78)
        at com.avamar.mc.sdk10.ActivityInfo.serialize(ActivityInfo.java:889)
        at com.avamar.mc.sdk10.ActivityInfo.serialize(ActivityInfo.java:198)
        at com.avamar.mc.sdk10.ArrayOfActivityInfo.serialize(ArrayOfActivityInfo.java:216)
        at com.avamar.mc.sdk10.TaskInfo.serialize(TaskInfo.java:630)
        at com.avamar.mc.sdk10.DynamicValue.serialize(DynamicValue.java:244)
        at com.avamar.mc.sdk10.DynamicValue.serialize(DynamicValue.java:152)
        at com.avamar.mc.sdk10.ArrayOfDynamicValues.serialize(ArrayOfDynamicValues.java:216)
        at com.avamar.mc.sdk10.ArrayOfDynamicValues.serialize(ArrayOfDynamicValues.java:160)
        at com.avamar.mc.sdk10.GetDynamicPropertyResponse.serialize(GetDynamicPropertyResponse.java:165)
        at com.avamar.mc.sdk10.GetDynamicPropertyResponse.serialize(GetDynamicPropertyResponse.java:109)
        at com.avamar.mc.sdk10.GetDynamicPropertyResponse$1.serialize(GetDynamicPropertyResponse.java:97)
        at org.apache.axis2.databinding.ADBDataSource.serialize(ADBDataSource.java:93)
        at org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerialize(OMSourcedElementImpl.java:692)
        at org.apache.axiom.om.impl.util.OMSerializerUtil.serializeChildren(OMSerializerUtil.java:556)
        at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:874)
        at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:230)
2017-01-16 12:58:20,038 [1690894503@qtp-1786872722-26] ERROR org.apache.axis2.transport.http.AxisServlet  - Java heap space
java.lang.OutOfMemoryError: Java heap space
2017-01-16 12:59:07,070 [486089829@qtp-1786872722-25] ERROR org.apache.axis2.transport.http.AxisServlet  - Java heap space
java.lang.OutOfMemoryError: Java heap space

Because of this the backup jobs never used to run and the mccli commands took forever to report the outputs.
The solution was to obviously increase the Java heap memory. The steps would be to:
(Backup the files before editing them)

1. Browse to the following location:
# cd /usr/local/avamar/var/mc/server_data/prefs
2. Make a backup of the mcserver.xml file before editing it. Open the mcserver.xml file using a vi editor
# vi mcserver.xml
3. Locate the following line. <entry key="maxJavaHeap" value="-Xmx1G" /> and change the value from 1G to 2G

Before Edit:
<entry key="maxJavaHeap" value="-Xmx1G" />
After Edit:
<entry key="maxJavaHeap" value="-Xmx2G" />

4. Save this file

5. Go to the following location
# cd /usr/local/avamar/lib/
6. Make a copy of mcserver.xml file in this location and open it in vi editor and edit the same parameter in this file too:

Before Edit:
<entry key="maxJavaHeap" value="-Xmx1G" merge="newvalue" />
After Edit:
<entry key="maxJavaHeap" value="-Xmx2G" merge="newvalue" />

7. Save the file

8. Switch to admin mode of VDP sudo su - admin and restart MCS using:
# mcserver.sh --restart

Post this, the GUI should show some relief in terms of loading and you should no longer see the java heap error in axis2.log

Hope this helps.

Tuesday, 10 January 2017

Avamar Virtual Edition 7.1: Failed To Communicate To vCenter During Client Registration

Once you setup your Avamar Server, you will proceed to add the VMware vCenter as a client to this. You will provide the vCenter IP/FQDN along with the administrator user credentials and the default https port 443. However, it errors out stating:

Failed to communicate to vCenter. Unable to find valid certification path to the vCenter. 

The error is due to Avamar not being able to acknowledge the VMware vCenter certificate warning. To do this, we will have to force the avamar to accept the vCenter certificate. 

Login to SSH of the Avamar Server and edit the mcserver.xml file:
# vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

Locate the vcenter certificate ignore parameter by searching for /cert in the vi editor. You will notice the below line.

Change this value from false to true and save the file.

Change the access to admin mode using sudo su - admin and run the below script to restart MCS.
# mcserver.sh --restart

Post this, you should be able to add this vCenter as a client to avamar successfully

That should be it.

Sunday, 8 January 2017

Part 5: Creating Recovery Plans In SRM 6.1

Part 4: Creating protection groups for virtual machines in SRM 6.1

Once you create a protection group, it's time to create a recovery plan. When you want to perform a DR test or a test recovery, it is the recovery plan that you will execute. A recovery plan is tasked to run a set of steps in a particular order to fail over the VMs or test the failover to the recovery site. You cannot change the workflow of the recovery plan, however you can customize by adding your required checks and tasks in between.

Select the production site in SRM inventory and under Summary Tab select Create a recovery plan.

Provide a name for the recovery plan and an optional description and click Next.

Select the recovery site where you want the VMs to failover to and click Next.

The Group type will be VM protection groups and then select the required protection groups to be added to this recovery plan. Only the VMs in the protection group added to the recovery plan will be failed over in an event of disaster. Click Next.

We have something called as Test Recovery. Test recovery does a test failover of the protected VMs to the recovery site without impacting the production VMs working or network identity. A test network or a bubble network (A network with no uplinks) will be created on the recovery site and these VMs will be placed there and bough up to verify if the recovery plan is working good. Keep the default auto create settings and click Next.

Review your recovery plan settings and click Finish to complete the create recovery plan wizard.

If you select the protected site, Related Objects and Recovery plans you can see this recovery plan being listed.

If you select the Recovery Plans in the Site Recovery Inventory, you will see the status of the plan and their related details.

Before you test your recovery, you will have to configure this recovery plan. Browse to, Recovery Plans, Related Objects, Virtual Machines. The VMs available under this recovery plan will be listed. Right click the virtual machine and select Configure Recovery

There are two options here, Recovery properties and IP customization.

The recovery properties discusses the order of VM startup, VM dependencies and additional steps that has to be carried out during and after Power On.

Since I just have one virtual machine in this recovery plan, the priority and the dependencies does not really matter. Set these options as to your requirement.

In the IP Customization option, you will provide the network details for the virtual machine in the Primary and the Recovery Site.

Select Configure Protection and you will be asked to configure IP settings of the VM in protected site. If you have VM tools running on this machine (Recommended), then click Retrieve and it will auto populate the IP settings. Click DNS option and enter the DNS IP and the domain name manually. Click OK to complete. The same steps has to be performed in the Recovery Site too under Configure Recovery, however, all the IP details has to be entered manually (If DHCP is not used) since there are no VM tools or powered On VM on the recovery site.

Once both are complete, you should see the below information in the IP Customization section. Click OK to finish configuring VM recovery.

Once this is performed for all the virtual machines in the recovery plan, the plan customization is complete and ready to be tested. You can also use the DR IP Customization tool to configure VM recovery settings.

In the next article, we will have a look at testing a recovery plan.

Part 4: Creating Virtual Machine Protection Groups In SRM 6.1

Part 3: Configuring Inventory Mappings in SRM 6.1

Once you configure inventory mappings, you will then have to configure protection groups. In protection group you will add the required virtual machines to be failed over by SRM in case of a disaster event.

Select the protected site and in the Summary tab under Guide to configuring SRM click Create a protection group.

Specify a name for this protection group that you will be creating and a description (not mandatory)

You will get to choose the direction for the protection group and the protection group type. In this, vcenter-prod is my production site and vcenter-dr is my recovery site.

I am using a vSphere Replication appliance, host based replication appliance, hence the replication group type would be vSphere Replication so that we can choose the VMs being replicated by this.
You cannot have vR and array-based replication VMs in the same protection group.

I have one virtual machine being replicate by vR. Check the required virtual machine and click Next.

Review the settings and click Finish to complete creating the protection group.

Under the production site, Related Objects, Protection Groups, you will be able to see the protection group listed.

Under the Protection Group option in the SRM inventory, you will see the same protection group listed.

And now in the recovery site, under the Recovered resource pool (selected during Inventory Mapping) you will be able to see the placeholder virtual machine.

With this, we have successfully created a protection group for virtual machines.

Part 5: Creating Recovery Plan in SRM 6.1

Thursday, 5 January 2017

Part 3: Configuring Inventory Mappings In SRM 6.1

Once the SRM sites are paired, you will then have to configure Inventory Mappings on the production site. Inventory mappings basically tell, these are the resources that will be available for the virtual machines in the protected site and these are another set of resources for the same VMs in the recovery site when failed over. Only when inventory mappings are established successfully you will be able to create placeholder virtual machines. If inventory mappings are not performed, then each VM has to be configured individually for resource availability after a fail-over.

In the below screenshot you can see that inventory mappings can be created for resource pools, folders, networks and datastores. There is only a Green check for Site Pairing as this was done in the previous article.

First we will be configuring resource mappings. Click the Create Resource Mappings. In the Production vCenter section, select the Production resource Pool. The virtual machine Router which we replicated earlier resides in this resource pool. Select the appropriate resource pool on the recovery site as well. Click Add Mappings and you should now see the direction of mapping in the bottom section. Click Next.

If you would like to establish re-protection, that is to fail back from the recovery site to the Protected Site, then check the option in Prepare reverse mappings. This is optional and right now, I do not want reverse mapping and I will leave it unchecked. Click Finish.

Now back in the summary tab,the resource mapping would be green checked and next we will be configuring Folder Mappings. In the Summary tab section, click Create Folder Mappings. I will manually configure this, so select Prepare mappings manually. Click Next.

Similarly like resource mapping, select the Folder that is required on the protected site where the protected virtual machines are and select the appropriate folder in the recovery site and click Add Mappings. Once the direction is displayed, click Next.

I will not be configuring any reverse mappings, so will leave this option unchecked and click Finish.

Once the Folder mapping is green checked, click Create Network Mapping and again this will be a manual configuration. Click Next.

In the same way, select the network on the Protected site where the protected VMs will be and an appropriate network on the recovery site and click Add Mappings. Then click Next.

Test networks would be created by default if you would like to test your recovery plans. The production VMs would be running as they are and a failed over instance of these VMs will be bought up on an isolated test network on the recovery site. Click Next.

Again, no reverse mappings here. I will simply click Finish.

With this the inventory mapping for resource, network and folder should be completed.

Next we will be configuring the Placeholder Datastore.

For every virtual machine in protected site, SRM will create a Placeholder VM on the recovery site. This placeholder VM will reside on a datastore and that datastore will be called as placeholder datastore. Once you specify this placeholder datastore, SRM will create VM files on that datastore in recovery site and uses them to register placeholder VMs on the recovery site inventory.

Two key requirements for placeholder datastore:
1. If there is a cluster in recovery site, the placeholder datastore must be visible to all the hosts in that cluster
2. You cannot use a replicated datastore as a placeholder datastore.

Select the Recovery Site in the SRM inventory and click Configure Placeholder Datastore. From the list below, select the required datastore to be used for placeholder and click OK.

If you would like to establish re-protection and fail-back, then you will have to select the placeholder datastore on the production site as well. You will have to click the protected site in the SRM inventory and configure placeholder datastore again.

With this, the inventory mapping will be completed.

Part 4: Creating virtual machine protection groups in SRM 6.1

Wednesday, 4 January 2017

VDP 6.1.3 - ESXi 5.1 Compatibility Issues

The VMware interoperability matrix says VDP 6.1.3 is compatible with ESXi 5.1. However, if you backup a VM when VDP is running on a 5.1 ESXi it will fail. I tried this with the following setup.

I deployed a 6.1.3 VDP on an ESXi 5.1 Build 1065491 and created a virtual machine on this same host. Then, I ran an on demand backup for this VM and it failed immediately. The backup job log had the following entry:

2017-01-04T21:13:36.723-05:-30 avvcbimage Warning <16041>: VDDK:SSL: Unknown SSL Error
2017-01-04T21:13:36.723-05:-30 avvcbimage Info <16041>: VDDK:SSL Error: error:14077102:SSL routines:SSL23_GET_SERVER_HELLO:unsupported protocol
2017-01-04T21:13:36.723-05:-30 avvcbimage Warning <16041>: VDDK:SSL: connect failed (1)
2017-01-04T21:13:36.723-05:-30 avvcbimage Info <16041>: VDDK:CnxAuthdConnect: Returning false because SSL_ConnectAndVerify failed
2017-01-04T21:13:36.724-05:-30 avvcbimage Info <16041>: VDDK:CnxConnectAuthd: Returning false because CnxAuthdConnect failed
2017-01-04T21:13:36.724-05:-30 avvcbimage Info <16041>: VDDK:Cnx_Connect: Returning false because CnxConnectAuthd failed
2017-01-04T21:13:36.724-05:-30 avvcbimage Info <16041>: VDDK:Cnx_Connect: Error message:
2017-01-04T21:13:36.724-05:-30 avvcbimage Warning <16041>: VDDK:[NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error:
2017-01-04T21:13:36.742-05:-30 avvcbimage Warning <16041>: VDDK:SSL: Unknown SSL Error
2017-01-04T21:13:36.705-05:-30 avvcbimage Info <16041>: VDDK:NBD_ClientOpen: attempting to create connection to vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@
2017-01-04T21:13:36.723-05:-30 avvcbimage Warning <16041>: VDDK:SSL: Unknown SSL Error
2017-01-04T21:13:36.723-05:-30 avvcbimage Info <16041>: VDDK:SSL Error: error:14077102:SSL routines:SSL23_GET_SERVER_HELLO:unsupported protocol
2017-01-04T21:13:36.723-05:-30 avvcbimage Warning <16041>: VDDK:SSL: connect failed (1)
2017-01-04T21:13:36.743-05:-30 avvcbimage Info <16041>: VDDK:DISKLIB-DSCPTR: : "vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@" : Failed to open NBD extent.
2017-01-04T21:13:36.743-05:-30 avvcbimage Info <16041>: VDDK:DISKLIB-LINK  : "vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@" : failed to open (NBD_ERR_NETWORK_CONNECT).
2017-01-04T21:13:36.743-05:-30 avvcbimage Info <16041>: VDDK:DISKLIB-CHAIN : "vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@" : failed to open (NBD_ERR_NETWORK_CONNECT).
2017-01-04T21:13:36.743-05:-30 avvcbimage Info <16041>: VDDK:DISKLIB-LIB   : Failed to open 'vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@' with flags 0x1e NBD_ERR_NETWORK_CONNECT (2338).
2017-01-04T21:13:36.780-05:-30 avvcbimage Info <16041>: VDDK:NBD_ClientOpen: attempting to create connection to vpxa-nfcssl://[datastore1 (2)] Thick/Thick.vmdk@

The VDDK disk release on 6.1.3 VDP is:
2017-01-04T21:13:27.244-05:-30 avvcbimage Info <16041>: VDDK:VMware VixDiskLib (6.5) Release build-4241604

Refer to Backward compatibility of TLS in the VDDK release article here.

Recommended Fix:
Upgrade to 5.5 U3e or later.

On the VDP appliance, edit the below file:
# vi /etc/vmware/config
Add the below line:

Save the file. There is no need to restart any services. Re-run the backup job and the backups should now complete successfully. 

Part 2: Pairing Sites in Site Recovery Manager 6.1

Part 1: Installing Site Recovery Manager

In this article we will see how to pair the two SRM sites we installed previously. So if you see here, I have two SRM sites, Production and a DR site. Soon after a fresh install these sites will not be paired and you will see the message "Site is not paired" for both the Production and Recovery Site. In order for failover to take place the Site needs to be paired. Click the option Pair Site in the center screen.

Enter the Platform Services Controller of the DR site and click Next.

The SRM plugin extension would be com.vmware.vcDr unless you chose to create a custom Plugin ID during install. Once the PSC detail is given in previous step, the vCenter corresponding to it will be populated automatically. Select the vCenter and provide the SSO user credentials for authentication and click Finish.

If you are presented with a certificate warning, click Yes to proceed. The pairing should now be completed.

Now, you can see the Sites have been paired from the Summary tab. And also the paired site details will be populated in the "Paired Site" section.

That's it.

Part 3: Configuring Inventory Mappings in SRM 6.1

Tuesday, 3 January 2017

Part 3: Recover A VM Using vSphere Replication

Part 2: Pairing vR Sites and configuring replication for a virtual machine

In this article, we will be performing a recovery of replicated virtual machine using vSphere replication. To perform a recovery, you will have to select the target vCenter (vCenter-DR in my case), select Monitor and Incoming replication.

You will see the below screen at this point and you will notice a big red button with a play symbol. This would be the recovery option. Select this icon.

You will then be presented with the type of recovery you would like to do

Recover with recent changes: This first option will need to have the source VM powered down. Before initiating the recovery process it will sync the recent changes with the source VM, so the recovered VM will be up-to-date.

Use latest available data: If you would not like to power down the source or if the source is unavailable or corrupted, you will choose this option. Here, it will make use of the recent replicated data to recover the virtual machine.

We will be using the second option to recover the virtual machine. In this wizard you will have to choose a destination folder to restore this virtual machine to.

You will then have to select the ESXi host and (if available) a resource pool to recover this virtual machine to.

You will have an option to keep the recovered VM Powered on or off. Depending on your requirement you can select this, and click Finish to begin the recovery process.

Once the recovery is complete, the virtual machine will be now available in the target site, and all the VM files that were named as hbr.UUID.vmdk (VM files that were replicated) will be renamed to the actual virtual machine files)

The status of the replication will now switch to Recovered and there will be no more active replication for this virtual machine.

Resuming Replication After Replication: Reprotect and Failback.

In most scenarios, once a VM is recovered you would like to re-establish the replication the other way to ensure there is a new replicated instance in case if this recovered virtual machine fails at some point. This is called as reverse replication or reprotection.

Initially, the replication was from vcenter-prod to vcenter-dr with the virtual machine residing on the vcenter-prod. Post a recovery, the virtual machine is now running on vcenter-dr. So, now the replication direction changes from vcenter-dr to vcenter-prod.

You will have to first stop the current configured replication for the virtual machine. On the target site, under incoming replication (Above screenshot), right click the VM with status as recovered and select Stop. Then, the virtual machine on the source has to be unregistered (Remove from inventory) on the source side. Once the replication is stopped and the source (old) virtual machine is unregistered, you will then have to reconfigure the replication. The process is same as discussed in Part 2 of this article.

The only difference is, when you select a destination datastore for the replication data to reside you will receive the following message. Select Use Existing. With this option, it will inform you that there are already a set of drives available on the target site and they will be the replication seeds. A initial Full Sync will still occur, but it will not be a copy of data, it will be just a check of the hash to ensure the validity. Once this is done, the new data will then be replicated first, and then replicated according to your set RPO.

Once the replication status goes to OK, you will have a valid replicated instance of the virtual machine at the new target site ready to be recovered.

Performing A Manual Recovery.

Until now, you saw vSphere Replication taking care of all recovery operation. But for some reason, the vCenter is down and you would like to recover a critical virtual machine. If vCenter is down, you cannot manage your vSphere Replication. Then in this case, we will be performing a manual recovery.

From the SSH of the ESXi host, you can see the VM files that are replicated:
# cd /vmfs/volumes/54ed030d-cd8f4a16-9fef-ac162d7a2fa0/Router

-rw-------    1 root     root        8.5K Jan  3 08:24 hbrcfg.GID-c3732b6f-de63-4c55-a830-a4437d91a143.4.nvram.8
-rw-------    1 root     root        3.1K Jan  3 08:24 hbrcfg.GID-c3732b6f-de63-4c55-a830-a4437d91a143.4.vmx.7
-rw-------    1 root     root       84.0K Jan  3 08:24 hbrdisk.RDID-297047a6-c7d0-4322-b290-bb610582daf1.5.59562057314158-delta.vmdk
-rw-------    1 root     root         368 Jan  3 08:24 hbrdisk.RDID-297047a6-c7d0-4322-b290-bb610582daf1.5.59562057314158.vmdk
You will have to rename these VM files to vmdk, flat.vmdk, vmx, nvram extensions. So, create a new folder under the datastore directory.
# cd /vmfs/volumes/54ed030d-cd8f4a16-9fef-ac162d7a2fa0/
# mkdir Rec
Pause the replication and copy / clone the vmdk to the new location using vmkfstools -i
# cd /vmfs/volumes/54ed030d-cd8f4a16-9fef-ac162d7a2fa0/Router
# vmkfstools -i hbrdisk.RDID-297047a6-c7d0-4322-b290-bb610582daf1.5.59562057314158.vmdk -d thin /vmfs/volumes/54ed030d-cd8f4a16-9fef-ac162d7a2fa0/Rec/Rec.vmdk
You will see the following output:
Destination disk format: VMFS thin-provisioned
Cloning disk 'hbrdisk.RDID-297047a6-c7d0-4322-b290-bb610582daf1.5.59562057314158.vmdk'...
Clone: 100% done.

Copy / Rename the vmx and nvram files using the below command:
# cp -a hbrcfg.GID-c3732b6f-de63-4c55-a830-a4437d91a143.4.vmx.7 /vmfs/volumes/54ed030d-cd8f4a16-
# cp -a hbrcfg.GID-c3732b6f-de63-4c55-a830-a4437d91a143.4.nvram.8 /vmfs/volumes/54ed030d-cd8f4a1
Finally, register the VM from the command line using:
# vim-cmd solo/registervm /vmfs/volumes/54ed030d-cd8f4a16-9fef-ac162d7a2fa0/Rec/Rec.vmx
If the registration was successful there will be a VM ID allocated as the output and you can verify the same in the vSphere client.

That's pretty much it.

Monday, 2 January 2017

When Nothing Is Left, Avtar Restore To The Rescue

There are multiple ways to restore a virtual machine in vSphere Data Protection.

When all of these fail, there is another option to restore a virtual machine. I am not sure about what it is called as, I refer to it as command line restore of a virtual machine using avtar.

**Before proceeding, please do not perform this in your production environment as the process is pretty tricky and can cause data loss if not done right. This is last of a last resort that we need to stick to. If restores are failing, the first step would be to fix it. Involve a VMware resource to perform this. That's as much as I can say. Post that, it's your call and risk**

The steps are pretty simple, you just need to be sure and careful on what is being selected. I ran into this issue while working on one of the cases logged with us. I cannot use the output from the session, so I had to reproduce this in my lab.

So, having said that. Let's have a look at the setup. I have a virtual machine on one of my ESXi host, and the name of the VM is Jump. It is a Windows box, with one virtual hard drive of 40 GB. The SCSI controller used here is 0:0. Then, I have a 512 GB of VDP deployed which has 4 drives. The SCSI controllers by default are, 0:0, 1:0, 2:0. 3:0.

With this, let's have a look at the steps:

1. It is always good to restore this disk to a new VM rather than to an existing VM because it reduces to complexity and risks by a large factor. This is because, let's say your VM has 8 drives and drive 6 and 8 has gone corrupt and there is no other means of restore available now. If you perform the avtar level restore, it is quite confusing on what disk has to be chosen and you might end up re-writing a different VMDK.

So to be safe create a new VM with a new hard disk with the same type of provisioning as the old one. Though it is not a hard requirement for the new drive to have similar provisioning, it would reduce the post restore process by a great deal. Like you would not have to SVmotion the drives to change the provisioning.

Now, when you create this new VMDK, please use a unique SCSI controller. Also, the drive created should be at least 1 GB more than the source disk. If my Source disk was 40 GB, I will create this new VMDK as 41 GB. The SCSI controller used here should be the same as any of the existing drives on the original VM or the VDP VM. Once the disk is created, keep the VM powered off, and add the same disk to the VDP appliance as well.
Basically, you will Edit Settings on the VDP appliance > Add > Hard Disk > Use existing hard disk and browse the datastore where this VM resides and add the hard drive. While adding the drive, use the same SCSI controller that was used on the newly created VM.

This would finish the step 1. Now switch to the command line of the VDP appliance for further process.

2. We will have to obtain the LabelNum of the backup existing for this VM, so that you can restore the contents from. To do this, first you will have to verify if the client is available in the GSAN. To do this, run the below command:
# avmgr getl --path=/vcenter-prod.happycow.local/VirtualMachines

The output will be similar to:
1  Request succeeded
1  Jump_UqrwzzeV6zMpRI8yfCBqgQ  location: c3d109f23e18075b48f680f0821730b417260427      pswd: e06a865b7bf4d0aadf90be28de519b8c0681354e

I just have one virtual machine in this VDP, hence one output in the GSAN. Now to get the labelNum of backups for this Jump VM, run the below command:
# avmgr getb --path=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ --format=xml

The output will be similar to:
1  Request succeeded
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<backuplist version="3.0">
  <backuplistrec flags="32768001" labelnum="1" label="Jump-1483351122072" created="1483351973" roothash="a6711baf9a0db97be019109cb7ea177ec7a8035e" totalbytes="42949988352.00" ispresentbytes="0.00" pidnum="3016" percentnew="17" expires="1488535122" created_prectime="0x1d264e0c7ce7616" partial="0" retentiontype="daily,weekly,monthly,yearly" backuptype="Full" ddrindex="0" locked="1" direct_restore="1"/>

LabelNum=1 specifies this is the first backup of the virtual machine. If I back this VM one more time and run the same command we will have two <backuplist> available and the labelNum counter would be incremented to 2 and so on.

3. We will have to list out the files available for this VM. It should list out the vmx, vmdk, flat.vmdk and the nvram files for this VM backup. The command would be:
# avtar --list --labelnum=1 --path=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ

The output will be similar to:
avtar Info <5551>: Command Line: /usr/local/avamar/bin/avtar.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --password=**************** --vardir=/usr/local/avamar/var --server=vdp-dest --id=root --bindir=/usr/local/avamar/bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --list --sequencenumber=1 --account=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ
avtar Info <7977>: Starting at 2017-01-03 00:25:06 IST [avtar Oct 14 2016 05:53:11 7.2.180-118 Linux-x86_64]
avtar Info <6555>: Initializing connection
avtar Info <5552>: Connecting to Avamar Server (vdp-dest)
avtar Info <5554>: Connecting to one node in each datacenter
avtar Info <5583>: Login User: "root", Domain: "default", Account: "/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ"
avtar Info <5580>: Logging in on connection 0 (server 0)
avtar Info <5582>: Avamar Server login successful
avtar Info <10632>: Using Client-ID='c3d109f23e18075b48f680f0821730b417260427'
avtar Info <5550>: Successfully logged into Avamar Server [7.2.80-118]
avtar Info <8745>: Backup from Linux host "/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ" (vdp-dest.happycow.local) with plugin 3016 - Windows VMWare Image
avtar Info <5538>: Backup #1 label "Jump-1483351122072" timestamp 2017-01-02 15:42:53 IST, 9 files, 40.00 GB
avtar Info <40113>: Backup #1 created by avtar version 7.2.180-118
VMConfiguration/avamar vm configuration.xml
VMConfiguration/snapshot description.xml
avtar Info <5314>: Command completed (exit code 0: success)

4. The VMDK file obtained from the above avtar command should be accessible. To verify this, run the below command:
# avtar -x --path=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ --labelnum=1 -O VMFiles/1/virtdisk-descriptor.vmdk

The output would be similar to:
avtar Info <5551>: Command Line: /usr/local/avamar/bin/avtar.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --password=**************** --vardir=/usr/local/avamar/var --server=vdp-dest --id=root --bindir=/usr/local/avamar/bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc -x --account=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ --sequencenumber=1 -O VMFiles/1/virtdisk-descriptor.vmdk
avtar Info <7977>: Starting at 2017-01-03 00:28:29 IST [avtar Oct 14 2016 05:53:11 7.2.180-118 Linux-x86_64]
avtar Info <6555>: Initializing connection
avtar Info <5552>: Connecting to Avamar Server (vdp-dest)
avtar Info <5554>: Connecting to one node in each datacenter
avtar Info <5583>: Login User: "root", Domain: "default", Account: "/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ"
avtar Info <5580>: Logging in on connection 0 (server 0)
avtar Info <5582>: Avamar Server login successful
avtar Info <10632>: Using Client-ID='c3d109f23e18075b48f680f0821730b417260427'
avtar Info <5550>: Successfully logged into Avamar Server [7.2.80-118]
avtar Info <5295>: Starting restore at 2017-01-03 00:28:29 IST as "root" on "vdp-dest.happycow.local" (4 CPUs) [7.2.180-118]
avtar Info <40113>: Backup #1 created by avtar version 7.2.180-118
avtar Info <5949>: Backup file system character encoding is UTF-8.
avtar Info <8745>: Backup from Linux host "/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ" (vdp-dest.happycow.local) with plugin 3016 - Windows VMWare Image
avtar Info <5538>: Backup #1 label "Jump-1483351122072" timestamp 2017-01-02 15:42:53 IST, 9 files, 40.00 GB
avtar Info <5291>: Estimated size for "VMFiles/1/virtdisk-descriptor.vmdk" is 463 bytes
# comment this is an avamar backup

# Extent description
RW 83886080 VMFS "virtdisk-flat.vmdk"

# The Disk Data Base
dbb.adapterType = "lsilogic"
dbb.geometry.cylinders = "5221"
dbb.geometry.heads = "255"
dbb.geometry.sectors = "63"
dbb.longContentID = "9c70cdf008a0d44ace4aa9d83340427c"
dbb.thinProvisioned = "1"
dbb.toolsVersion = "10246"
dbb.uuid = "60 00 C2 98 b6 dc 27 49-cf 44 c6 73 c4 42 e2 84"
dbb.virtualHWVersion = "11"
avtar Info <5267>: Restore of "VMFiles/1/virtdisk-descriptor.vmdk" completed
avtar Info <7925>: Restored 463 bytes from selection(s) with 463 bytes in 1 files
avtar Info <6090>: Restored 463 bytes in 0.01 minutes: 4.271 MB/hour (9,673 files/hour)
avtar Info <7883>: Finished at 2017-01-03 00:28:30 IST, Elapsed time: 0000h:00m:00s
avtar Info <6645>: Not sending wrapup anywhere.
avtar Info <5314>: Command completed (exit code 0: success)

The RW section would describe the size of the VMDK. 83886080 x 512 = 42949672960 bytes, corresponds to 41943040 KB, which is 40960 MB which translates to 40 GB.

5. So now on the new VM I have a 41 GB drive with SCSI 1:1 created and this is attached to the VDP appliance with the same 1:1 controller. Rescan for storage using the below command:
# echo "- - -" > /sys/class/scsi_host/host1/scan

Here "- - -" defines the three values stored inside host*/scan i.e. channel number, SCSI target ID, and LUN values. We are simply replacing the values with wild cards so that it can detect new changes attached to the Linux box. This procedure will add LUNs, but not remove them.

6. Run fdisk -l and the new device should not be detected as a formatted partition. You should be seeing the below output:

Disk /dev/sde doesn't contain a valid partition table

7. Now the next command is restore. This command will start as soon as you hit enter. It will not give you any option to proceed with a yes or no prompt. So be careful on what is entered here before you proceed.

The command would be:
avtar -x --nostdout --account=/vcenter-prod.happycow.local/VirtualMachines/Jump_UqrwzzeV6zMpRI8yfCBqgQ --labelnum=1 -O VMFiles/1/virtdisk-flat.vmdk > /dev/sde

The labelnum can differ depending on your requirement. 
This will not show any output or any progress of the restore. If the VMDK created was a thin provisioned drive, then you can login to ESXi and run the below command:
# watch -n1 stat vm-name-flat.vmdk

This will refresh the output of the VMDK every 1 second. 

The output should be similar to:
Every 1s: stat Jump1-flat.vmdk                                                                                                                                                                                           2017-01-02 20:41:32
File: Jump1-flat.vmdk
Size: 44023414784     Blocks: 305152     IO Block: 131072 regular file
Device: 61f328bc5c1ebe26h/7058029830484180518d  Inode: 226524548   Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-01-02 20:39:58.000000000
Modify: 2017-01-02 20:39:58.000000000
Change: 2017-01-02 20:41:30.000000000

And this should be refreshing until the block size correlates to 40 GB. To calculate this, you need to Blocks x 512 = Size in Bytes.

8. Now, detach this drive from the VDP appliance. So in the end, you should have your new VM powered off and this drive attached to it. Power On the VM and you should be able to see the data.