Saturday, 30 January 2016

Installing vCenter Appliance 6.0 With External PSC

Written by Suhas Savkoor

Previously, we had seen how to install Appliance 6.0 with embedded PSC. In this article, we will deploy an appliance 6.0 with external PSC. Where the platform service controller resides as one virtual machine and the vCenter node resides as another virtual machine.

  • As always, ensure the DNS entry is set up for the virtual machines in forward and reverse lookup zones. 
  • There will be two virtual machines deployed, one for PSC and the other one for vCenter node. They both have their own IP address and own FQDN, and the forward and reverse lookup must be configured for each one of them prior to proceeding with the deployment. 
  • A windows machine that acts as a mount point for the ISO.

Deploying Platform Services Controller:

Step 1:
Mount the vCenter 6.0 appliance ISO on a Windows machine and install the client integration plugin from the " vcsa "  folder.
Once the client integration plugin is installed go to root directory of the ISO and open the vcsa-setup.html file (preferably using IE/Firefox) You will come across the below screen.

Click the Install option to begin the setup wizard.

Step 2:
Accept the EULA terms and conditions and proceed Next

Step 3:
  • Provide the details of the ESXi host on which you want to deploy the PSC node. 
  • FQDN or IP address of the ESXi host; root user and its password. 

Step 4:
  • Give a short name to this appliance, this name can be anything. However, I prefer to keep this name same as the short name that I added in the DNS record. 
  • Provide a new password for the root user of the appliance.

Step 5:
  • You will come across the deployment type screen. Since this is an external deployment, we will go with the second part. And since PSC contains the SSO, certificates, lciesing and other services, this node has to be deployed first. 
  • Select Install Platform Services Controller

Step 6:
  • This is the first node that we are deploying, hence select the Create a new SSO domain option.
  • Enter a new password for the SSO user of the appliance.
  • SSO domain name: This has to be a unique domain name. It can be anything, here I am using the "vsphere.local" as my SSO domain. The SSO user then would beadministrator@vsphere.local
  • SSO site name: Any site name that is required.
*Note* If you have an existing domain in your active directory, say,, DO NOT use this domain as your SSO domain. This will prevent you from adding users under to domain to the vCenter for management. 

    Step 7:
    Select a datastore where the PSC node should reside.

    Step 8:
    • Fill out the Network Settings for the PSC node
    • Network label: Where the appliance should reside on your vSwitch/DVswitch port-group
    • IP address of the appliance and FQDN of the appliance. Again, this has to be reflected in the DNS entry.
    • Subnet, gateway and DNS server entry. Enabled SSH if required at the very last.

    Proceed Next and begin the installation. Once the installation is complete, you will be asked to manually start the vCenter Server install. 

    Deploying vCenter Server Node:

    Step 1:
    Click the Install button on the web page and begin the installation again.
    *Step 1 to 4 remain the same*

    Step 2:
    In the Deployment Type select the Install  vCenter Server option and proceed Next

    Step 3:
    • We need to join this vCenter to previously deployed PSC node. 
    • Enter the PSC node's FQDN (Preferred than IP address)
    • vCenter SSO password that was configured in the previous steps for authentication
    • Leave the port at 443

    Step 4:
    Select the size of the appliance depending on the inventory size of your environment. 

    Step 5:
    Select a datastore on which the vCenter node should reside.

    Step 6:
    Select the type of database for vCenter. vPostgres would be an express database for the appliance unless you have an external Oracle database. SQL DB is not yet supported for appliance.

    Step 7:
    Enter the vCenter node's network details

    Proceed Next and begin the installation. 
    Once the deployment is complete, login to vCenter using Windows Client or Web Client and verify it is working good. Since I have deployed a GA version of the appliance you do not have the Web GUI management page for the appliance. This is only available from 6.0 Update 1 onward.

    Wednesday, 27 January 2016

    View ESXi Logs From The Web Page

    Written by Suhas Savkoor

    Short article for how to view ESXi logs from a browser than opening a Putty to the host. One method, is to use the Embedded Host Client. If not, then:

    1. Open a browser
    2. Enter:
    3. When asked for credentials, enter root and the password.
    4. You will see the below page for logs:

    *Note: This is not a real time logging, and needs to be refereshed for any recent changes*
    SSH all the way, but why not something on the side!

    Monday, 25 January 2016

    Connecting A ESXi 5.5 U3b Host To A vCenter With A Lower Version.

    Written by Suhas Savkoor

    Lately, I have been seeing more and more cases with VMware Support regarding, "Unable to connect an ESXi host to a vCenter after upgrading it to 5.5 U3b" The common error you get when you try adding this host to a vCenter is

    " Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management service on this host is not responding "

    This is because, with 5.5 U3b SSLv3 is disabled, however, on a vCenter that is on a version lower than 5.5 U3b has SSLv3 enabled.
    To resolve this, the best method would be to upgrade your vCenter to 5.5 U3b.

    If you do not want to upgrade vCenter, then you can enable SSLv3 on that upgraded ESXi host. However, this is not a best practice and is not recommended by VMware, because it defeats the purpose of the upgrade. However, if you want to, then you can proceed with the below steps:

    There are two parts were you need to enable SSLv3:

    A) Enabling SSLv3 for Hostd - Port 443
    1. Open a SSH to this ESXi host.
    2. Browse to this location using the below command:
    # cd /etc/vmware/rhttpproxy
    3. Backup the config file:
    # cp config.xml config.xml.bak
    4. Edit the file using the below command (Press i to begin edit)
    # vi config.xml
    Locate the <vmacore>, then locate the <ssl> Under <ssl> add the following entry:
    5.Save the file by pressing Esc and then typing :wq!

    B) Enabling SSLv3 for Port 902 (Required to connect to vCenter)
    1. From the same SSH of the host, run the below command:
    # esxcli system settings advanced set -o /UserVars/VMAuthdDisabledProtocols -s ""

    Restart the rhhtpproxy using the below command:
    # /etc/init.d/rhttpproxy restart
    That's it, now you can connect this ESXi 5.5 U3b host to a lower version of vCenter. However, again, this would not be a recommended practice as this will expose the host to SSLv3 POODLE vulnerability.

    Thursday, 21 January 2016

    Configuring Serial Port As A Loop-back COM Port For The Same Virtual Machine.

    Written by Suhas Savkoor

    If you have missed out on "How to configure COM ports between two virtual machines" video, here is the link to it.

    In this article, we will see how to configure COM port on the same virtual machine. This is used in a scenario like; You have an application that monitors a set of readings in your environment. When the readings cross a particular threshold, then it has to generate a signal or send some information to an alerting system using a COM port.

    Now, in Windows, when you go to device manager and expand the Ports option, you can see that there are two COM ports; COM1 and COM2. These are always there irrespective of whether you have configured serial port or not for that specific virtual machine.

    Configuring serial port for the same machine:

    1. Power OFF the virtual machine for which you are trying to configure this COM port.
    2. Go to Edit Settings and Click Add. Here select Serial Port and click Next. 
    3. Select Output to named pipe and click Next
    4. The pipe name should be of the format: \\.\pipe\<pipe_name>
    5. Near End: Server; Far End: A process
    6. Create another COM port for this same virtual machine. Click Add. Select Serial Port and Output to named pipe option again.
    7. Here the pipe name should be the same as the one with first serial port.
    8. Configuration for second serial port; Near End: Client; Far End: A process
    9. Click OK

    Testing COM port setup:

    1. Power ON the virtual machine and open CMD in administrative mode, and open Putty to COM1 in serial.
    2. Type the following command in CMD:
    echo text > COM2
    3. In the Putty you can see the message "text" being echoed.
    4. The Putty is on COM2 and CMD on COM1. Hence the Putty is listening for incoming traffic on COM1 and the CMD is sending the message to COM2, which is why Putty is opened in COM1 serial.

    Simple, isn't it?

    Update Manager Service Crashes During A Scan Operation On An ESXi Host

    Written by Suhas Savkoor

    Today, I came across an issue with Update Manager while working on a support request. The environment, comprised of two vCenter in linked mode and each of them having their own Update Manager server. The update manager was installed on a machine of their own. The second vCenter was running well and good, however, on the first vCenter there was an issue with the "Scan" operation.

    Whenever a baseline was attached to any of the hosts under this vCenter, and a scan operation was performed, the progress would go to 10 percent, stop there for few minutes and then the vSphere Update Manager service used to stop and crash causing the VUM to lose connectivity with the vCenter.

    Upon on reviewing the logs for the failure, vmware-vum-server-log4cpp.log, I noticed the following:
    Error accessing stagepath C:/ProgramData/VMware/VMware Update Manager/Data/host_upgrade_packages/esxi-upgrade-ryvdmfvtoz type 1 error 0/The operation completed
    This means that the patch store and the DB are not in-sync

    When I browse C:\ProgramData\VMware\VMware Update Manager\Data, I do not see the host_upgrade_packages folder, and the scan is failing because it is unable to find this folder.
    The install directory may vary depending on your installation settings.

    What can be done?

    1. If you have your old update manager (Rarely happens), then you can copy paste this folder into this directory and the scan will work good!

    If not, then we will have to re-initialize the update manager database.
    **Re-initializing the database will clear out the database for update manager, which means, if you had any custom baselines and patches downloaded, they will be lost**

    Steps to Re-initialize the VUM database:

    1. First Login to SQL management studio hosting this update manager database. Expand Database > Right click the VUM database > All Tasks > Backup. Back this database to a disk.
    2. Stop the Update Manager service from services.msc
    3. Open a command prompt in elevated permission mode (Administrative mode) change the drive to the VUM installed disk drive and run the below command:
    cd "C:\Program Files (x86)\VMware\Infrastructure\Update Manager\" 
    4. Then run the below command to re-initialize the database:
    vciInstallUtils.exe -O dbcreate -C . -L . 
    (Both . should be used)

    5. Once command has executed restart the Update Manager service.
    6. Login to vCenter > Select ESXi host > Update Manager > Admin View
    7. Under Configuration tab select Download Settings and download the patches again. If you want to add your custom baselines, then you can go ahead and do so
    8. Go back to Compliance View and Attach and Scan, and this time the operation should succeed!

    Wednesday, 20 January 2016

    "No Network Adapters Found" For Nested ESXi 6.0 Host

    Written by Suhas Savkoor

    While setting up a nested ESXi 6.0 host, you will come across the following error during the installation:

    Now, it says no network adapter found for this virtual machine on where I am trying to install the ESXi 6.0. Funny, because during creation of the virtual machine I have specified one network card of the e1000 type.
    Now, if I SSH to the actual ESXi hosting this virtual ESXi, I see that there is a network adapter "e1000" listed in the .vmx file of the virtual machine. And this virtual machine resides on the appropriate network.

    Now the funny thing about this is, the virtual machine that I created for this ESXi was allocated with 2GB of RAM. The minimum memory requirement for a 6.0 ESXi host is a 4GB RAM. I did not receive a warning during the installation, which is quite weird. However, upon changing the memory to 4GB I was able to proceed with the installation successfully.

    Well, there you go!

    Tuesday, 19 January 2016

    Changing The Network Adapter Type of A Virtual Machine Without Removing The NIC

    Written by Suhas Savkoor

    If we have a virtual machine with the NIC given to it as E1000 and we want to change this NIC to VMXNET3, then from the Edit Settings on the virtual machine we will select the Network adapter. Here you will notice that you do not have an option to change the Adapter Type.
    The classic step we would follow is to login to the virtual machine and make a note of the network settings. We then, remove the Network adapter from the Edit Settings of the virtual machine. Once the NIC is removed, we will go ahead and a new NIC, and while adding a new adapter, we get the choice of choosing the adapter type. Once the adapter is added, we login back to the VM and re-populate the network settings.
    This all works good, however, removing the NIC and adding a new one will change the MAC address of the device. Every network adapter will have a MAC address which will be listed under the adapter type option in the Edit Settings of the virtual machine. If an application is dependent on the MAC address, for example a VM hosting telephone IVR operation, this might break as it uses the MAC address of the device. In scenarios like this, we will have to reconfigure the application.

    The other way to change the network adapter type is:

    1. Power OFF the required virtual machine.
    2. Take a SSH (Putty) to the host where this virtual machine resides. Change the directory to the virtual machine's directory.
    3. Open the virtual machine's .vmx file using the vi editor
    # vi <vm_name>.vmx
    4. Locate the following line
    ethernet0.virtualDev = "e1000"
    Press " i " to begin edit and change the e1000 to vmxnet3 (Retain the quotes and text is case sensitive). Press Esc and type :wq! to save and exit the file.

    5. Remove the virtual machine from Inventory.
    6. Browse the datastore where this VM resides and right click the .vmx file and add this vmx file of the virtual machine to the inventory.
    7. Go back to Edit Settings of the virtual machine, select the network adapter and you will see the updated adapter type with the same MAC address.

    You just saved a MAC address!

    Error Licensing A vCenter: License File Not Found.

    Written by Suhas Savkoor

    When you are adding a license to your vCenter 5.x, you receive the error:

    Diagnostic message: License File Not Found

    This error message is received if the license you are adding is a 6.0 license. A 5.5 vCenter cannot be licensed with a 6.0 key. 

    To downgrade your license key you can follow this article here.
    Once the license key is downgraded to 5.x, you will receive a new key, for the downgraded version, you can then apply this license to your 5.5 vCenter successfully.

    Sunday, 17 January 2016

    Port 80 Already In use During vCenter Installation

    Written by Suhas Savkoor

    During vCenter installation, you will come across the port details wizard page and when you click Next, you will receive an error message similar to the one below:

    " The following port numbers are either invalid or already in use. VMware VirtualCenter HTTP Port: 80 "

    You can have a look at this KB article here, however, sometimes even though all the conditions in this article are met, you might still receive the above mentioned error.

    How do we get past this?

    1. Open a command prompt in elevated mode (As administrator) on the machine where you are installing the vCenter Server. Run the below command:
    netstat -ano | find "80"
    This command tells which process is currently listening on port 80. You will see a Process ID (PID) of 4 which is running on this port blocking the installation of vCenter.
    2. Open Task Manager on the same machine. Navigate to Processes tab. Click View and click Select Columns and make sure PID is checked and then select OK
    3. Now, if it is a normal application or IIS, disable it or uninstall. If it is a System Process: PID 4 - you need to disable the HTTP.sys driver
    4. On the same machine, open regedit (Start > regedit)
    5. Go to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP
    6. Change the value of " start " to 4 which means disabled
    7. Reboot the machine and perform the installation again.

    Fresh installation of a 5.5 vCenter: Simple Install

    Written by Suhas Savkoor

    In this article we will see how to install a fresh instance of 5.5 vCenter. Then proceeding further we will be upgrading this vCenter to 5.5 U3b and then to 6.0. This 5.5 installation is a simple install, where all the vCenter components like Single Sign On, Web Client, Inventory Service and vCenter Server reside on the same machine. 
    The database being used here is a SQL Server 2008 R2 SP2. Before you proceed with any installation, there are a couple of pre-requisites that has to be satisfied. These are:
    • Make sure the hardware and software requirements for vCenter are met. This can be found in this link here.
    • The next thing you need to question is what type of database are we using. If it is an embedded SQL express, then it comes pre-packaged with the vCenter ISO and there is no need to worry about the compatibility. If you are using an external SQL then you need to check if the SQL version you are using is compatible with the vCenter version being deployed. You can use VMware Interoperability Matrix to check the compatibility of database. 
    • Next, the Windows machine being used for vCenter deployment should be added to a domain, and the Forward and Reverse Lookup Zone must be configured for this machine.
    Step 1:
    Creating The vCenter Database.

    **If you are using SQL Server (External) then you will have to follow this step to create a database and establish the connection. If it is a SQL express, then the DB is installed automatically during vCenter installation**

    Login to SQL management studio, right click Database and select New Database. 

    It will prompt you to enter a name for the database and provide the name for the vCenter database. Once created, expand the database section and verify that you can see your database in there

    Step 2:
    Create an ODBC connection

    **The Open Database Connectivity (ODBC) is required to be created on the vCenter machine, so that the vCenter can talk to its respective external SQL database. vCenter uses a 64 bit Data Source Name (DSN)**

    **The SQL Native Client that is being chosen should be 10. A lower version of SQL native client will result in failure during vCenter installation wizard**

    Login to the Windows machine being prepared for vCenter Installation. Click Start > Run, and type ODBC. Open the 64 bit ODBC. Click System DSN.

    • Click Add and select a SQL native client version and click Finish. Now we will have to enter the ODBC configuration details. 
    • The Name for the DSN can be any user defined name, if required you can enter a Description.
    • The Server would be the SQL server that you are connecting to. Click Next

    Here you will have to provide an authentication mode for the database. As in the first step I am using the sa account as the DB user, I will provide the sa authentication for my vCenter database. The user that you are going to use to setup your ODBC connection must have sysadmin rights on your database end, else the installation of vCenter will fail at the ODBC wizard page.

    In the Next page you will have to check the "Change the default database to" option and from the drop-down select the database that you created earlier. Keep the other options to default and proceed Next. In the next option, change the language if needed, otherwise keep all the options to default and proceed to Finish.

    Once finished, you can review your settings and then click Test Connection. This should result in Test completed Successfully message. Do not proceed further if the test fails. 

    This takes care of setting up to connectivity of vCenter to database part. 

    Step 3:
    Download the required version of vCenter 5.5 from the VMware download portal and mount this ISO on to this windows machine. Since this is a simple install (All vCenter components residing on the same machine), the installation order is Single Sign On > Web Client > Inventory Service > vCenter Server.

    SSO Installation:
    The first component as discussed would be to install the Single Sign On. On the ISO that is mounted, under custom install, select Single Sign On. You can also proceed with the Simple Install option, which will automate the installation of all the components and prompt you when each component is being installed.

    Click Next and accept the End User License Agreement.

    Click Next and it will check if the machine has met the pre-requisites. Like joining a domain, and having a resolvable DNS entry.

    Once the pre-requisites are met, click Next. Here it will ask you for the SSO deployment type. If this is a first instance of Single Sign On that is being deployed and if it is being deployed for a single vCenter, then select Standalone Single Sign On.

    Click Next and it will ask you to create a new password for the SSO administrator user. Provide this credentials and document this, as this will be required for future management.

    Click Next, and it will ask for a site name. This site name can be anything, however, this has to be documented, because during link mode setup of the vCenter it will ask for this site name.

    Click Next and you will come across the port used for SSO service. This port has to be open on the machine and firewall in order for SSO to work.

    Click Next and then you will be asked for a directory where you want to install the Single Sign On component. Select a directory, or leave it for default values. Click Next, review the configuration and proceed to Install the SSO component.

    Web Client Installation:
    The next component would be install vSphere Web Client which would be a web interface to manage your vCenter environment.
    From the same ISO, if the installtion mode is Simple Install, then you will be automatically prompted at this point. If it is custom, then select Web Client and click install.

    Select a directory where you want to install the Web Client.

    Now, the default https port for web client is 9443. If you want to provide a custom port, then change the https port value, however, record this port number as it is necessary to access the web client. If you do not want to change the port number now, and would like to do so in the future, then you will have to follow this article.

    Provide the SSO password that was created during the Single Sign On setup.

    The web client needs certificates for authentication as well. VMware installs default certificates for Web Client. If needed, you can replace them later either with CA Signed Certificates or from a third party CA authority. Wild card certificates are not supported in VMware.

    Finish the installation of the Web Client component.

    Inventory Service Installation: 
    Inventory Service stores vCenter Server application and inventory data, which lets you search and access inventory objects across linked vCenter Server instances. If your search is not working in vCenter, then there is probably some issue with Inventory Service database.

    Accept the EULA and proceed Next

    Choose a directory as to where you want to install this component. Click Next

    Enter the vCenter's FQDN for which this Inventory Service needs to be registered.

    Review the ports being used for this service. Again, if being changed, then you will have to make a note of the custom port numbers.

    Select an Inventory size, this depends upon your environment size. Choose accordingly.
    You can change your JVM size after installation as well, again, if required!

    Click Next. Provide the SSO credentials for authentication and begin the installation.

    vCenter Server Installation:
    This would be the last core component to be installed for a VMware vCenter environment setup. Accept the EULA and proceed Next.

    You can enter a license key during the installation or you can enter the license key once the vCenter is setup.

    Since we are using an external SQL database, we will choose the second option " Use an existing supported database "  and then from the drop-down select the DSN name that was created under 64 Bit ODBC. If you do not see your DSN listed under drop-down, then enter it manually proceed Next. If you are using an express SQL, then you will not follow the initial steps in this article for creating a vCenter database. You will check this first option which will proceed with installation of SQL express and creating a database, all by itself.

    When you click Next you will be asked for database authentication. Provide the same user which was used to create the DSN (Must have sysadmin privileges)

    In the next page enter the information for VMware vCenter Server service. This can be a local authentication or a domain authentication. If your SQL service is domain then it's best to have a domain authentication for vCenter Server service as well.

    If this is a fresh deployment of a vCenter then select " Standalone instance " If you are setting up another vCenter and want to join these two vCenters in linked mode, then the second option has to be chosen. You can also join two vCenters in linked mode even after two standalone instances are deployed.

    Check the ports required for vCenter services and make sure these are open and proceed Next

    Again, select the inventory JVM size for vCenter server component.

    Provide the SSO credentials for authentication.

    Now, you will get an option to choose the SSO user name. You can give administrator@vsphere.local or any other name. Please make sure that you do not give your existing domain name as your SSO username, as this will cause issues during adding users to vCenter.

    Since vCenter is dependent on Inventory Service, the IS was installed first and the URL for IS was created and auto-populated in the next step.

    In the final step, provide a directory to install vCenter component and begin the installation.

    Once the installation completes, the vCenter deployment is complete. From the same ISO install vSphere Client. After installtion, open the vSphere Client and enter the IP address of the vCenter, SSO username/password and login to verify the setup is working good.
    Open a browser and navigate to Web Client URL, https://<vcenter_IP>:9443 to verify Web Client is functioning good!

    That's all!