Friday, 14 April 2017

Failed To Start Internal Proxy In VDP 6.x

Mostly after an upgrade most of your backups fail with a status of "No eligible proxies" or "No data"
You will not be able to run on demand backups in some cases and this would fail with an error "Adhoc Backup Request Error - Exception"

root@vdp-dest:/data01/home/admin/#: mccli client backup-dataset --domain=/vcenter-prod.happycow.local/VirtualMachines --name=VM-C
1,22253,Client Adhoc Backup Request Error - Exception.

If you try to enable Internal proxy from the vdp-configure page, it will fail with the below error:


In the vdr-configure.log you will notice the following:

2017-04-15 03:50:52,463 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
2017-04-15 03:50:52,463 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Error <7531>: Unable to register clients/vdp-dest with Administrator 127.0.0.1:28001
2017-04-15 03:50:52,464 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl:  'Could not reconcile proxy with vCenter.' (203)
2017-04-15 03:50:52,464 ERROR [pool-22-thread-1]-cmdline.RuntimeExecImpl: avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log

You will see vCenter connections down if you run the below command:
# mccli server show-services

You will something similar to:

0,23000,CLI command completed successfully.
Name                               Status
---------------------------------- -----------------------------
Hostname                           vdp-dest.happycow.local
IP Address                         10.109.10.167
Load Average                       0.24
Last Administrator Datastore Flush 2017-04-15 04:45:00 IST
PostgreSQL database                Running
Web Services                       Error
Web Restore Disk Space Available   256,417,868K
Login Manager                      Running
snmp sub-agent                     Disabled
ConnectEMC                         Disabled
snmp daemon                        Disabled
ssh daemon                         Running
Data Domain SNMP Manager           Not Running
Remote Backup Manager Service      Running
RabbitMQ                           Not Running
Replication cron job               Not Running
/vcenter-prod.happycow.local       5 vCenter connection(s) down.

If you try to register proxy from the command line using the below command, it will fail as well. 
# /usr/local/avamarclient/etc/initproxy.sh start

avagent.d Info: Stopping Avamar Client Agent (avagent-vmware)...
avagent.d Info: Client Agent stopped.
avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
avagent Error <7531>: Unable to register clients/vdp-dest with Administrator 127.0.0.1:28001
 'Could not reconcile proxy with vCenter.' (203)
avagent.d Info: Client activation error.
avagent Info <5008>: Logging to /usr/local/avamarclient/var/avagent.log
avagent Info <5417>: daemonized as process id 351
avagent.d Info: Client Agent started.

Registration Failed.
initproxy.sh FAIL: registerproxy failed

The cause:
This is because, there is a key called as "ignore_vc_cert" which will be flipped to false. The VDP will always be waiting for process to acknowledge the certificate warning which will never work and hence the proxy fails to start.

The fix:
1. Run the below command to verify the key value:
# grep -i ignore /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

The output should be similar to:
     <entry key="ddr_ignore_snmp_errors" value="false" />
     <entry key="email_logs_tar_cmd" value="tar -cz --atime-preserve=system --dereference -- ignore-failed-read --one-file-system --absolute-names" />
      <entry key="ignore_vc_cert" value="false" />

2. Edit this mcserver.xml file and replace the ignore_vc_cert value to true and save the file

3. Switch to admin mode of VDP (sudo su - admin) and restart the mcs using:
# mcserver.sh --restart

4. Register the internal proxy from GUI and it should work successfully and none of the vCenter connections will be reported as down.

Hope this helps.