Monday, 22 February 2016

vCenter Appliance upgrade to 6.0 fails with "Internal error occurs during VMware vCenter Server Database pre-upgrade checks"

Written by Suhas Savkoor

Today while upgrading vCenter appliance from 5.1 U3 to 6.0, it failed with the error:

Error: Internal error occurs during VMware vCenter Server Database pre-upgrade checks

Upon checking the log file, CollectRequirements_com.vmware.vcdb_date.log file, this is what I came across:

2016-02-20T19:37:44.353Z INFO vcdb Retrieving DB type...
2016-02-20T19:37:44.353Z INFO vcdb DB type retrieved: PostgreSQL
2016-02-20T19:37:44.366Z INFO vcdb Retrieving DB user...
2016-02-20T19:37:44.395Z INFO vcdb DB user retrieved: vc
2016-02-20T19:37:44.407Z INFO vcdb Retrieving DB password...
2016-02-20T19:37:44.433Z INFO vcdb DB password retrieved: ****
2016-02-20T19:37:44.456Z INFO vcdb Validating source embedded schema...
2016-02-20T19:38:00.481Z INFO vcdb Souce embedded schema validation completed.
2016-02-20T19:38:00.498Z WARNING transport.local BAD REQUEST: Cannot execute ['/opt/vmware/vpostgres/1.0/bin/psql', '-U', 'postgres', '-d', 'VCDB', '-t', '-c', "\n SELECT ceil(sum(pg_relation_size(C.oid)) / (1024*1024)) as disk_size\n FROM pg_class C\n LEFT JOIN pg_namespace N\n ON N.oid = C.relnamespace\n WHERE nspname IN ('vc', 'vpx') and relkind in ('r', 't')\n \n AND relname NOT LIKE 'vpx_hist_stat%'\n AND relname NOT LIKE 'vpx_sample_time%'\n AND relname NOT LIKE 'vpx_event%'\n AND relname NOT LIKE 'vpx_task%'\n AND relname NOT LIKE 'vpx_property_bulletin%'\n "]. Error: [Errno 2] No such file or directory
2016-02-20T19:38:00.498Z ERROR __main__ Upgrade Phase 'vcdb:CollectRequirements' failed. Exception: Traceback (most recent call last): File "/var/tmp/vmware-upgrade-requirements/cis-upgrade-runner/payload/", line 379, in main executionResult = systemExtension(exeContext)

The reference it makes to while saying, Cannot execute is /opt/vmware/vpostgres/1.0/bin/psql

Now, when you run:
ls -l /opt/vmware/vpostgres/1.0/
You see:
total 8
drwxr-xr-x 2 root root 4096 Jan 4 2016 lib
drwxr-xr-x 5 root root 4096 Jan 4 2016 shar

There is no directory called bin under 1.0

When you run:
ls -l /opt/vmware/vpostgres/
You see:
drwxr-xr-x 4 root root 4096 Jan 4 2016 1.0
drwxr-xr-x 7 root root 4096 Jan 4 2016 9.0
lrwxrwxrwx 1 root root 3 Jan 4 2016 current -> 9.0

So, the execution python script is calling a wrong path causing a failure in the upgrade. VMware should fix this call in their python script.

You can address this by creating a symbolic link to the correct path by running the below command:
"ln -s /opt/vmware/vpostgres/current/bin /opt/vmware/vpostgres/1.0/bin" 
Run the upgrade again and this time there should be no issues.


Wednesday, 17 February 2016

Safely Remove Hardware For Virtual Machine?

Written by Suhas Savkoor

When you have a hard disk given to a virtual machine with a SCSI controller, or a Network Interface Card provided to it, and when you login to this virtual machine, you will see our old friend, "Safely Remove Hardware" When you click this, your SCSI controller, your network card, and USB if mounted, will be displayed. Now, if you click the remove button by some mistake, the respective device will be unmounted and removed from the virtual machine. Too bad of a scenario eh? Well, not to worry, virtual machine advanced settings will always come to the rescue. 

1. Power OFF the virtual machine on which you want to disable the Safely Remove Hardware option. 
2. Right click the machine and click Edit Settings
3. Click Options, select General and click Configuration Parameters.
4. Click Add Row and under Namedevices.hotplug Under Value: false
5. Click OK.

Now power ON the virtual machine and this time the safely remove hardware option is no longer present. There was one more question that was asked:

"If I have CPU and memory hot-add option enabled will this parameter affect that operation? Also will I have issues adding disk and network card to the virtual machine once this parameter is set?"

The answer is, No. The add devices, and CPU/Memory hot-add (if enabled) will not be affected by this parameter. 
So, unplug those at your will. Life is too short for safely remove hardware!


Tuesday, 16 February 2016

Certificate Status Alarm Triggered After Upgrading vCenter To 6.0

Written by Suhas Savkoor

So, recently I have received a handful of cases, where the certificate status alarm is triggered, when the existing 5.x environment is upgraded to 6.x. You can acknowledge and clear out the alarm, however after certain days, this alert is triggered back. When this alert is triggered, the vCenter health service enters a warning state. 

When you login to Web Client, click the certificate in the address bar and view certificate information, then you will notice that the there is quite some time for the certificate to expire. 

However, when checking the the vpxd logs for the vCenter:
vCenter Server vpxd log location:
C:\Program Data\VMware\vCenter Server\logs\vmware-vpx
You will see the below logging:
2015-07-17T09:05:18.767+02:00 warning vpxd[05124] [Originator@6876 sub=Main opID=CheckCertificateExpiry-312b91e5] [Vpxd::VecsUtil::CheckCertificatesFromStore] Certificate [Subject: CN=SMS-120924141331507,O=VMware] from store SMS will expire on 2014-09-24 12:13:31.000. 

2015-07-17T09:48:15.592+02:00 warning vpxd[03620] [Originator@6876 sub=[SSO][GroupcheckAdapter] opID=7413e525] [FindAllParentGroups] Maybe SSO Groupcheck is expired, trying to re-login. Exception: class Sso::Fault::NotAuthenticated::Exception(sso.fault.NotAuthenticated)

Here the Storage Monitoring Service (SMS) 5.5 certificate is still in the VECS (VMware Endpoint Certificate Store) and has expired. This certificate is no longer used in version vCenter Server 6.0.

How to remove it?

1. Login to the vCenter machine and open a command prompt in Administrative mode
2. Change the directory to:
cd "C:\Program Files\VMware\vCenter Server\vmafdd"
3. Run the below command to display the cert for SMS service.
vecs-cli entry list --store SMS --text | more
The output is something like below:

Alias : sms_self_signed
Entry type :    Private Key
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=VMware, CN=SMS-120612174523625
            Not Before: Jun 12 21:45:23 2011 GMT
            Not After : Jun 12 21:45:23 2014 GMT
        Subject: O=VMware, CN=SMS-120612174523625
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)

Here the alias is: sms_self_signed

4. Make a note of the cert alias and run the below command to remove the cert:
vecs-cli entry delete --store SMS --alias <certificate_alias>
5. Restart the Web Client service. If there is an alert triggered prior to this procedure, clear it and monitor your environment if the alert re-appears.

99.99 percent, you are in the green zone sergeant!!

Tuesday, 9 February 2016

Client Integration Plugin Fails To Work On MAC OSX 10.11

Written by Suhas Savkoor

So I have been running into Client Integration Plugin issues on 6.0 for quite some time now. Windows as well as MAC OSX. 
Uninstalling and re-installing the plugin will not do much good in OSX because, every time you try to use the Plugin it refers to a path which does not exist. This is because after an OSX upgrade to 10.11, the symbolic link is broken and this has to be re-done in order the get the plugin working again. 

How to do this?
1. Open a Terminal in OSX and run the set of commands below, one by one:
sudo mkdir -p /build/toolchain/mac32/openssl-1.0.1p/lib

cd /build/toolchain/mac32/openssl-1.0.1p/lib/

sudo ln -s /Applications/VMware\ Client\ Integration\ libssl.1.0.1.dylib

sudo ln -s /Applications/VMware\ Client\ Integration\ libcrypto.1.0.1.dylib

/Applications/VMware\ Client\ Integration\ —install
2. Clear the browser cache and the client integration plugin must start to work again. (This step might not be required in some cases, however it's best to be done to remove any older cached entries for the plugin)

However, in my scenario, certain functions of the plugin, like "Import OVF", "Upload To Datastore" functions were working good. Other functions like "Use Windows session credentials" did not work and the message stating "Download Client-Integration Plugin" still persisted.

The fix worked good on both Chrome and Firefox. I read a couple of articles where the upload to datastore function, when executed caused the browser to hang and crash, which I haven't experienced until now.

If you guys have any thoughts on this, let me know!

Saturday, 6 February 2016

VMware vExpert 2016!

Written by Suhas Savkoor

So VMware announced the 2016 vExpert results on February 5 2016 and I am honoured to say that I have been awarded this title. A year and half into VMware and the work has certainly paid off. Congratulations to everyone who were awarded this award again and to those who were selected for the first time. 

What is VMware vExpert?

The VMware vExpert program is VMware's global evangelism and advocacy program. The program is designed to put VMware's marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware's roadmap. VMware will provide you with a unique vExpert ID that will allow insights into analytic to help understand customer trends to assist you and keep your advocacy activities on track.

Thank you VMware and cannot wait for 2017!

Click this link here for the list of all 2016 VMware vExperts. Ctrl+F away to find your name!


Monday, 1 February 2016

Upgrade: VCSA 6.0 External PSC to 6.0 U1

Written by Suhas Savkoor

In the previous article, we saw how to deploy a vCenter appliance 6.0 with an external PSC

In this article, a simple one, we will see how to upgrade this 6.0 GA appliance with external PSC to 6.0 U1. Since, 6.0 GA appliance does not have the Web GUI page for management, the patching and upgrading has to be done via command line. 

The upgrade process is very simple and similar to upgrading 6.0 appliance with embedded PSC.
I will not be attaching any screenshot to this, since it does not require any. 

Step 1:
Now, with an external PSC, you will have two virtual machines. One virtual machine is for a PSC (Platform Services Controller) and the other virtual machine is for vCenter Server. 
Now you have to mount the ISO on the vCenter Server virtual machine. If you try to mount the ISO on the PSC machine and run the upgrade command, it is going to fail.

Step 2:
Once the ISO is mounted to the vCenter machine, open a SSH (Putty) to this vCenter machine and login with root credentials. 
There is no requirement to open the bash shell. 
Once logged in to SSH simply run the below command to directly apply the patches to the appliance:
Command> software-packages install --iso --acceptEulas
This will apply the patches and once done, you need to reboot the appliance using the below command:
Command> ​shutdown reboot -r “Updated to vCenter Server 6.0 Update 1”

Step 3:
If you do not want to directly apply the patches (Step 2), but stage the patches first and then apply it, then there will be a different route that will be followed:

First, you need to stage patches using the below command:
Command> software-packages stage --iso --acceptEulas
Then you can review the staged patches:
Command> software-packages list --staged
Then you will have to apply these staged patches:
Command> software-packages install --staged
Finally, you will have to reboot the vCenter appliance:
Command> ​shutdown reboot -r “Updated to vCenter Server 6.0 Update 1”

Step 4:
Verify the update. 
From the SSH of the vCenter server enabled bash shell and then run the below command:
# vpxd -v
This will give you the build number of the appliance. Correlate this build number to the release version using this KB article.

Also, you can verify this by opening a browser and going to https://<vcenter_IP>:5480
Since this is 6.0 U1 you will now have access to VAMI page (Web GUI management page)