Friday, 19 May 2017

Unable To Install SQL/Exchange Backup Agent on vSphere Data Protection

When you upgrade your VDP from 5.8.x to/ot 6.0.x to/or 6.1.x you will need to uninstall the old backup agent, re download the new agent from the 6.1.x Configuration Tab and install it. However, this installation will fail with the below error:

This is because the client was not deactivated once the agent was uninstalled. Until you deactivate the client associated with the old plugin you will not be able to install the new one.

On the machine where you are installing the client, if you browse to C:\Program Files\avp\var you will have a log file called avagent.log and if you notice the error here, you will see:

2017-05-19 14:09:37 avagent Error <6210>: Unable to register '/clients/VDPApps\sqlvm' with MCS at 192.x.x.x:28001
2017-05-19 14:09:37 avagent Error <6163>: Registration error: MCS returned error code 110, '* 110 * Re-registration of account /clients/VDPApps/sqlvm.happycow.local is not permitted'
2017-05-19 14:09:37 avagent Error <7531>: Unable to register /clients/VDPApps/sqlvm with Administrator 192.x.x.x:28001

So, to deactivate this client, from the SSH of the VDP run the below command:
# mccli client edit --activated=false --name=/clients/VDPApps/<client-fqdn> 

Then retry the install of the agent and it should complete successfully.

Hope this helps.

Shell Command: Understanding tee

By default when you run a command, the output is reflected onto stdout. Consider this simple example,
# echo "Hello World"

The output would be:
Hello World

The output is displayed on stdout which is my terminal.

In some cases, we will have a need to run the command, view the output and simultaneously save the output to a log file. This is where tee comes in. Consider tee as a fork in a road. The output goes both ways, one to stdout and the other to a file

# some command | tee <file_1> <file_2>.....<file_n>

The output of some command is displayed on stdout and stored in one or more files.

tee comes with a couple of switches. The most simple tee command is:
# echo "Hello World" | tee file.txt

The output is Hello World on stdout and if you cat file.txt you will see the output Hello World
If you run the same command with a different echo text to the same file, then the existing content is replaced.

So if you want to save the existing content and append new output, then use the -a switch
# echo "Hello World" | tee file.txt
# echo "Today is a good day" | tee -a file.txt

When you run the first command, the output on stdout is Hello World and the same output is seen when you cat file.txt

When you run the second command, the output on stdout is Today is a good day and the output when you cat the file.txt is:
Hello World
Today is a good day

Next is tee with -i or --ignore-interrupts which ignores the SIGINT which would be your Ctrl+C
Sigint sends an interrupt signal to terminate a foreground process, generally when you press Ctrl+C, but you want tee to be terminated gracefully. In short, -i ignores Ctrl+C SIGINT

Shell Script To Monitor vSphere Data Protection 6.x Health Status

If you have issues with your VDP service / system functionality and would like a quick overview of how your maintenance services are doing, then look no further, for here is a script.

This script does the following task:

1. Check the core, maintenance, scheduler and management services of VDP
2. Check for VDP System Status
3. Provide an overview of VDP server information
4. Perform checks for Checkpoint, Garbage Collection and HFS check
5. Display storage information
6. Provide client information
7. Perform vCenter port and connectivity tests
8. (Optional) Generate log bundle for VMware Support
9. (Optional) email the report output for VMware Support.

The script zipped file can be downloaded from here along with read me. Please go through the read me file before running the script. Do not run the script as root, for you shall not pass!!

Script download:


Wednesday, 3 May 2017

Shell Script To Deploy vSphere Data Protection

Deploying vSphere Data Protection is always easy from the GUI. But with the 6.5 vCenter release there are a handful of issues with ova deployment. Hence, we use either ovf tool to deploy the VDP or we use govc CLI. If you use govc CLI, you can refer this article here.

Instead of that, you can now use the below script to deploy VDP. Download the script and the Read Me guide from my github repo here:

The script:

echo -e "*****************************************"
echo -e "*   This script is written by Suhas G   *"
echo -e "*            *"
echo -e "*****************************************"
echo -e "This Script Deploys vSphere Data Protection"
echo -e "\nDownloading govc for Linux"
curl -L | gunzip -d > /usr/local/bin/govc
chmod +x /usr/local/bin/govc
echo -e "\nThe version of govc is"
govc version
echo -e "\nEnter the requested details"
read -p "Specify vCenter FQDN or IP Address: " GOVC_URL_a
read -p "Specify administrator username: " GOVC_USERNAME_a
read -p "Specify the password: " GOVC_PASSWORD_a
read -p "Specify the Datastore name: " GOVC_DATASTORE_a
read -p "Specify Network Portgroup: " GOVC_NETWORK_a
read -p "If needed Specify resource pool name: " GOVC_RESOURCE_POOL_a


echo -e "\n\nThe JSON Specification file is as below"
echo -e "Edit the file to enter your deployment details\n"
sleep 5
govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool
govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool > /vdp.json
vi /vdp.json
echo -e "Deploying VDP\n"
govc import.ova -options=/vdp.json /vSphereDataProtection-6.1.3.ova
echo "\n\nDeployment Complete. All Done!"
Hope this helps!

Tuesday, 2 May 2017

Script For VDP Proxy Registration Issue

Earlier in this article here we saw the issue where post an upgrade of VDP the ignore_vc_certificate parameter is flipped to false because of which the proxy registration fails. If you would like to edit the files manually you can follow that article. Else, you can use the attached script in this one.
The script basically changes the parameter, restarts the MCS and re-registers the internal proxy to vCenter.

Download the script, place it in your /home/directory of VDP appliance and run the script while logged in as admin. Provide the script a+x permissions.

You can download the script from github here:

The script:

echo -e "***************************************"
echo -e "* This script is written by Suhas G  *"
echo -e "*          *"
echo -e "***************************************"
echo -e "This script fixes vCenter certificate parameter for proxy\n"
echo -e "The current value for ignore_vc_cert is\n"
value=$(cat /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml | grep ignore_vc_cert | cut -d = -f 3 | cut -d " " -f 1)
echo $value
read -p "Proceed with fix?: " choice

case "$choice" in
y|Y )
sed -i -e 's/entry key="ignore_vc_cert" value="false"/entry key="ignore_vc_cert" value="true"/g'  /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
echo -e "\nThe new value for ignore_vc_cert is"
value=$(cat /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml | grep ignore_vc_cert | cut -d = -f 3 | cut -d " " -f 1)
echo $value
echo -e "\nProceeding to restart MCS" --restart --verbose
echo -e "\nProceeding to re-register internal proxy"
sudo /usr/local/avamarclient/etc/ --start
echo -e "\nAll done!";;
n|N )
echo -e "\nExiting script";;
If there are any issues reach out to me. Hope this helps.

Wednesday, 26 April 2017

VDP Error 1009 When Connecting To Web Client

When you try connecting to a specific VDP appliance from the web client, you will run into the generic Web Client Flash Error.

This error is generally encountered when there is an issue with VMware Tools on the VDP appliance. If you select the VDP VM and go to summary tab, you will notice that there are no tools running and there is no IP address for this VM.

The fix would be to uninstall VMware tools from VDP and re-install it back. To uninstall the VMware tools, run the script which is located under /usr/bin/
Once this script completes, the tools no longer exist on VDP. So next, you will have to re-install it back.

Right click the VM > Guest > Install or Upgrade VMtools. Then SSH into the appliance and run the below set of commands:
# mkdir /mnt/cdrom

# mount /dev/cdrom /mnt/cdrom

# cp /mnt/cdrom/VMwareTools-10.0.6-3560309.tar.gz /tmp
(This might vary depending on version of tools)

# cd /tmp

# tar -zxvf VMwareTools-10.0.6-3560309.tar.gz

# cd vmware-tools-distrib/
The VMtools install script has a ton of options if you go with manual install. The best is to choose default install by running the below command:

# ./ -d default

This will consider all the default settings and complete the tools install and you should see the VM tools status in the VM summary page as running and the appliance should connect successfully to the web client.

Hope this helps.

Tuesday, 25 April 2017

Unable To Reinstall SRM With Existing Database "The selected vCenter server does not match the one used in the previous installation"

There are scenarios where you will have to reinstall a vCenter site. When a vCenter site is reinstalled the SRM solution connected to it has to be reinstalled. The reinstall is usually done with an existing database is because you would like to save your protection groups and recovery plans without the need to recreate them all. However, this reinstall with existing database would fail with the following error:

The selected vCenter server does not match the one used in the previous installation

This is because, every vCenter has a vCenter UUID with it. In this case, the vCenter being used was 6.0 and SRM was 6.1.

If you look at the instance.cfg file. It is located under:

Installation directory/VMware/vCenterServer/vmware-vpx/cfg

In this instance.cfg you will have the vCenter UUID.

When SRM is registered to vCenter, this instanceUUid is saved in the SRM database. Now, when the vCenter is reinstalled, you will have a new instanceUuid which does not match the one stored in SRM Database and the installation fails.

So, you will have to manually update the SRM database with this instanceUuid prior to installation.

The table you need to look into SRM is pd_localsite. If you connect to your DB and view this table, you will see something as below. I am using Embedded Postgres for SRM, so the query would be:

select * from pd_localsite;

The output:
 db_id |    mo_id    | ref_count |            name             | vcaddress | vcport | vccertthumbprint |                 uuid                 |                        siteurl                        |            vcinstanceuuid            |    domain    |                                  sucertificate
     1 | DrLocalSite |         1 | vcenter-prod.happycow.local |           |      0 |                  | 0a063f18-a09c-4126-9556-db5bd30c37dd | https://psc-prod.happycow.local:443/lookupservice/sdk | c4c2202e-4acd-4b35-a2da-0947a3429658 | vmware.local | C:\Program Files\VMware\VMware vCenter Site Re
covery Manager\bin\
(1 row)

So the vcinstanceuuid needs to be matching your instanceUuid of instance.cfg 

Once this is done, proceed with the SRM installation and it will work just fine. 

Tuesday, 18 April 2017

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

In this part, we will replicate VDP backup data to Avamar and then migrate the jobs. First, to replicate the data from VDP to Avamar, we create a regular replication job. Let's have a look at this.

Connect to the VDP appliance from the web client and click the Restore tab. To give you a brief overview, here is the restore points for my VM-A

Click the Replication tab and click Replication job actions. and click Create New Replication Job.

You will be presented with the below wizard. I want to replicate my current Image backups, so the first option and click Next.

I will replicate the individual client, and select VM-A and then click Next.

Select which backup schedule you would like to replicate. This is as per your need and then click Next.

Enter the AVE FQDN/IP as the hostname, the port for replication is 29000. The username to be used is repluser and the password for repluser is the one you setup during the Avamar configuration. Click Verify Authentication. Once the connection test is completed successfully, click Next.

Select how frequently you need to replicate. This does not really matter here, because it is going to be one time replication, because post the migration source VDP will be in a unusable state. Click Next.

Select a retention policy for the replicated backup as per your requirement and click Next.

Provide a name to this replication job and then click Next and then Finish the wizard.

Once the job is created, select the job and click Replicate Now to begin the task.

Once the replication is completed, you can see the replicated data in the Avamar end.

You will still notice there is a red X for Backup Jobs as the jobs are not transferred. It is only the VDP backup data that is replicated to Avamar.

Next we will begin migrating the jobs. To do this, open a browser and go to:

Once on this page, click Avamar Migration and click Migrate.

Check the Terms and conditions to proceed further and click Next.

Provide the AVE FQDN, the root credentials would be for Avamar root and not the OS root password. Click Verify Authentication.

Once the authentication succeeds, click OK and then click Next.

You can select the backup jobs you need to migrate here. 

You can switch the tab to Backup Verification and select the required verification jobs and then click Next.

A validation will be performed and if everything is successful, click Next.

You will be prompted for migration. Click Yes to begin the migration.

You can then monitor the migration progress.

Then you can review the migration status. Click Close.

Now the status is, Jobs have been migrated and the appliance is disabled.

The same is seen in the VDP page in web client as well.

On the Avamar end, you can see the jobs have now been migrated successfully.

So, in the end, the VDP would be used only for performing restore operations with the existing data. And once new valid backups are performed by Avamar, you can discard VDP completely.

Hope this series was helpful. Let me know your thoughts in the comments.

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

Previously we configured AVE services. Next we need to download the Avamar Client, install it and then configure a vCenter and deploy a proxy. Let's look at this step by step.

Let's start off by downloading the Avamar Client. To get this, go to the below URL:

You should see the below screen:

Click Downloads and you should see the below screen. First download the jre.exe and have it installed on the machine where you plan to run the Avamar Client. Then download AvamarConsoleMultipleWindows.exe

Run the installer, it's a pretty straightforward wizard to follow.

Once the client installation is done, open the Avamar client and you will be presented with the below login screen. The Server would be FQDN or IP of your AVE. The Username would be MCUser and the Password for the same with / Domain.

Once the login is successful, we will see the below Avamar management UI.

Notice there are no backup jobs, hence a red X for Backup Jobs tab. Now, we will have to configure vCenter to the AVE. Click Administration from the management UI and you will see a new window:

The vCenter client, should always be added to the / domain. So select the ave and then click Actions > Account Management and then click New Client.

You will see the following section. Select the Client Type as VMware vCenter. Provide the vCenter FQDN or IP for Client name. For username provide an admin user defined on the vCenter level. Disable vCenter Certificate parameter before performing this. You can find this step in this link here.
Click OK and the vCenter should be registered successfully to the AVE machine.

You will notice the vcenter-prod (my vCenter) under the ave domain added as a client.

Next, you will have to deploy a proxy, without this your migration pre-check will fail. In the same window, click VMware and Proxy Deployment Manager.

Here you can input your required setting and create a recommendation.

Click Apply and then click New Proxy and you will be presented with the below window. Enter a name for the proxy. The domain would be your /vcenter-fqdn/VirtualMachines. Provide an IP for the proxy, datastore and a network. Please note the proxy should be resolvable before performing the deployment. Click Save.

The proxy deployment will begin and you can monitor the progress here as well as the vCenter Recent Tasks pane.

This should be it. Until now, you have deployed AVE ovf, configured it, downloaded the client and configured it to a vCenter and added a proxy.

Next we will look at the migration of data and jobs from VDP to AVE.

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

Monday, 17 April 2017

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

In part 1 we saw how to deploy and setup the basic Avamar 7.2 machine. In this article, we will have the appliance configured for usage.

Open a browser and type in:

This should bring up the below screen and you can login with root as the user and changeme as the password.

Post the login you will see the below page. Here on the top left corner you will see a lock icon. Click this and enter the password as Supp0rtHarV1.  Once unlocked, click Install to begin the package installation.

The initialization will take a few minutes to complete and once done you will be presented with the below window. Here, you will need to fill all the fields with the red exclamation marks.

Under Server Settings, the Avamar Server Address should be the hostname and select an appropriate time zone.

The number of storage node I will be using is 0, just like how my VDP is setup. A single node server. This is the node you see when you run status.dpn

Fill in the remaining fields and click Continue on the bottom right. This should begin the package installation like seen below:

Once the configuration is complete you will see the below task. At this point, your AVE setup is complete.

Now you can SSH into the AVE machine with admin credentials and run the below command to verify system and service status.

admin@ave:~/>: dpnctl status
Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: suspended.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.

admin@ave:~/>: status.dpn
Tue Apr 18 00:49:01 IST 2017  [AVE] Mon Apr 17 19:19:01 2017 UTC (Initialized Mon Apr 17 17:51:04 2017 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0   7.2.1-32  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.49 5209    33104   0.0%   0%(onl:8  )  0%(onl:8  )  0%(onl:8  )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1492451464@00:50:56:9A:52:F6

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20170417180550 finished Mon Apr 17 23:36:09 2017 after 00m 19s (OK)
No GC yet
No hfscheck yet

Maintenance windows scheduler capacity profile is active.
  WARNING: Scheduler is WAITING TO START until Wed Apr 19 20:30:00 2017 IST.
  Next backup window start time: Thu Apr 20 20:00:00 2017 IST
  Next maintenance window start time: Thu Apr 20 08:00:00 2017 IST

That should be it with the configuration part. Next, we will look into how to Configure AVE to vCenter Server. 

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-1: Deploying Avamar Virtual Edition 7.2

Since VMware announced, the end of vSphere Data Protection, there is a choice to migrate existing deployment to EMC Avamar. More about EOA can be found in this link here.

In this article, we will be looking at deploying Avamar Virtual Edition 7.2. You can go ahead and download the required version of AVE from EMC download portal. The version I will be using is Avamar 7.2.1.

Login to Web Client or vSphere Client, Select the ESXi host where you want to deploy your AVE and select File > Deploy OVF. Browse the location for the AVE download and add the file. Click Next.

Review the details of the OVF template and click Next.

Accept the EULA and click Next.

Provide a name for this AVE virtual machine. Click Next.

If available and required select a resource pool in which you want to place this VM. Click Next.

Select the datastore where you want to deploy this. Remember the vmdk bundled with AVE is just the OVF, the data drives are configured later just like a VDP appliance. Click Next.

Select a disk provisioning type. Thick provision is recommended. Click Next.

Select a network where this AVE should be connected to. Click Next.

Review the changes and click Finish. Do not check Power On after deployment, because there are couple of steps to be done once the OVF deployment is completed.

Just like VDP, the AVE comes with 4 supported available backup storage. You can refer the below table to size your AVE accordingly.

Once you choose the deployment type, you will need to refer the below table to plan the drive sizes. Just like in VDP a 512GB deployment will have 3 drives of 256 GB each. The additional space is for the checkpoint maintenance overhead.

So the rule goes like:
Total size = GSAN capacity + 1/2 of GSAN capacity.

GSAN capacity would be the actual space for storing backup data.

So go ahead and add three disks manually (Depending on your AVE configuration) to this VM. Only Thick Provisioning is supported for AVE. I will be using Thin because of space constraints.

Once the drives are added, power on the AVE virtual machine.

The default login is root and changeme

First, we will have to configure network settings for the AVE machine. Post the login to AVE from VM console, run yast2 to begin the networking configuration. You will see a similar interface:

Select Network Devices and then Network Services to begin the network configuration wizard and you should see something similar:

You will need to Set IP Configuration in Overview, Hostname / DNS settings and Gateway under Routing.

Once the appliance is configured with network, restart the guest and then verify the network by ping and nslookup. If this works good proceed to Part 2 in the below link.

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2