Tuesday, 27 December 2016

VDP Upgrade To 6.1.3 Fails: Installation Of The Package Stalled

vSphere Data Protection 6.1.3 is a pretty stable release in the 6.x chain. However, there was an issue getting to this version from a prior release. This issue is specifically seen while upgrading VDP from 6.1 to 6.1.3. The upgrade fails approximately around 60 percent, with an error "Installation of the package stalled" and it prompts you to revert to the snapshot taken prior to the upgrade.

The avinstaller.log has the following:

INFO: Working on task: Starting MCS (74 of 124) id: 10312
Dec 12, 2016 1:07:32 PM com.avamar.avinstaller.monitor.PollMessageSender sendMessage
INFO: PollMessageSender sent: <UpdateMessage><Progress>60</Progress><TaskName>Starting MCS (74 of 124)</TaskName><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><TaskId>10312</TaskId><Timestamp>2016/12/12-13:07:32.00489</Timestamp></UpdateMessage>
INFO: Package Location in process: /data01/avamar/repo/temp/vSphereDataProtection-6.1.3.avp_1481542766965
<UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:07:38.00626</Timestamp><Content>Starting MCS ...</Content></UpdateMessage>
Dec 12, 2016 1:07:38 PM com.avamar.avinstaller.process.TaskTimeManager resetTimer
INFO: startTime: 1481544458653
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener receiving message
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener forwarding message: <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<UpdateMessage><Content><![CDATA["mcserver.sh --start", exit status=1 (error)]]></Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.process.MessageReceiver receiveMessage
INFO: MessageReceiver receive message: <UpdateMessage><Timestamp>2016/12/12-13:08:38.00890</Timestamp><Content>"mcserver.sh --start", exit status=1 (error)</Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.monitor.PollMessageSender sendMessage
INFO: PollMessageSender sent: <UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:08:38.00890</Timestamp><Content>"mcserver.sh --start", exit status=1 (error)</Content></UpdateMessage>
Dec 12, 2016 1:08:38 PM com.avamar.avinstaller.process.TaskTimeManager resetTimer
INFO: startTime: 1481544518935
Dec 12, 2016 1:08:43 PM com.avamar.avinstaller.monitor.MessageListener doPost
INFO: MessageListener forwarding message: <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<UpdateMessage><Content><![CDATA[----->start MCS failed - operation failed]]></Content></UpdateMessage>
<UpdateMessage><ProcessKey>VSphereDataProtection20161020Oct101476964022</ProcessKey><ProcessInstanceId>VSphereDataProtection20161020Oct101476964022.10007</ProcessInstanceId><Timestamp>2016/12/12-13:08:43.00689</Timestamp><Content>-----&gt;start MCS failed - operation failed</Content></UpdateMessage>

Dec 12, 2016 1:08:44 PM com.avamar.avinstaller.process.handler.ServerScriptHandler$TaskProcessor run
WARNING: From err out: 12/12 13:08:38 error: "mcserver.sh --start", exit status=1 (error)
12/12 13:08:43 error: ----->start MCS failed - operation failed

Dec 12, 2016 1:50:04 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit stderr:
Dec 12, 2016 2:50:00 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush command: /usr/local/avamar/bin/avinstaller.pl --flush
Dec 12, 2016 2:50:03 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit code: 1
Dec 12, 2016 2:50:03 PM com.avamar.avinstaller.util.FlushTimer run
INFO: Flush exit stdout: Flushing Avinstaller...
INFO: Flushing AvInstaller.
INFO: AvInstaller flushed.

Unfortunately there is no fix available for this issue. This is a known issue while upgrading from 6.1 to 6.1.3. See the release notes here

The workaround is to upgrade from 6.1 to 6.1.2 and then upgrade the 6.1.2 to 6.1.3
If this workaround fails, please raise a ticket with VMware Support. I cannot provide the resolution here due to confidentiality.

Hope this helps.