Thursday, 8 February 2018

Unable To Connect VDP To vCenter: "login returned a response status of 204 No Content"

When connecting a newly deployed VDP or an existing VDP to the web client, you might run into the following error:

This is a very generic message and if you have a look at the web client logs, you will notice the following back trace:

[2018-02-08T09:03:57.295Z] [WARN ] http-bio-9090-exec-5         70000222 100008 200003 org.springframework.flex.core.DefaultExceptionLogger The following exception occurred during request processing by the BlazeDS MessageBroker and will be serialized back to the client:  flex.messaging.MessageException: com.sun.jersey.api.client.UniformInterfaceException : POST https://10.10.0.12:8543/vdr-server/auth/login returned a response status of 204 No Content
        at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:444)
        at com.vmware.vise.messaging.remoting.JavaAdapterEx.invoke(JavaAdapterEx.java:50)
        at flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:183)


Caused by: com.sun.jersey.api.client.UniformInterfaceException: POST https://10.10.0.12:8543/vdr-server/auth/login returned a response status of 204 No Content
        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:609)
        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:586)
        at com.emc.vdp2.api.impl.BaseApi.convertToFlexException(BaseApi.java:184)

Looking further into the vdr-server.log, you will notice this:

2018-02-08 10:04:44,850 ERROR [http-nio-8543-exec-9]-rest.AuthenticationService: Failed To Get VDR Info
java.lang.NullPointerException
        at com.emc.vdp2.common.appliance.ApplianceServiceImpl.getApplianceState(ApplianceServiceImpl.java:47)
        at com.emc.vdp2.services.VdrInfoServiceImpl.getVdrInfo(VdrInfoServiceImpl.java:171)
        at com.emc.vdp2.services.ApplianceInfoServiceWS.getVdrInfo(ApplianceInfoServiceWS.java:50)
        at com.emc.vdp2.rest.AuthenticationService.login(AuthenticationService.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

The next piece of stack trace might vary, but if you see the above messages then you are bound to see a similar trace as below:

2018-02-08 10:04:44,727 INFO  [http-nio-8543-exec-9]-rest.AuthenticationService: Logging into appliance with type: vdp
2018-02-08 10:04:44,768 INFO  [http-nio-8543-exec-9]-connection.Mcsdk10StubManager: ServiceInstanceMoref desc=Service Id: urn:uuid:9FBE7B2DFEF05750401518080690404 name=urn:uuid:9FBE7B2DFEF05750401518080690404 value=SERVICE
2018-02-08 10:04:44,771 INFO  [http-nio-8543-exec-9]-connection.McAccessManager: Creating new mcsdk stub handler for connection key: [2091248218, Service Id: urn:uuid:9FBE7B2DFEF05750401518080690404] on Thread: [http-nio-8543-exec-9]
2018-02-08 10:04:44,849 ERROR [http-nio-8543-exec-9]-db.ApplianceStateDAO: ApplianceStateDAO.getApplianceState failed to execute ApplianceState query.
java.sql.SQLException: ERROR: relation "appliance_state" does not exist Query: select * from appliance_state Parameters: []
        at org.apache.commons.dbutils.AbstractQueryRunner.rethrow(AbstractQueryRunner.java:320)
        at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:349)
        at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:305)

Right after it initiates Authentication, it queries the vdr database. And in this case, appliance_state, table is missing from vdrdb. 

To connect to vdrdb on VDP, run:
# psql -p 5555 -U admin vdrdb

Type \d to list all tables. You should see 26 tables here:

                          List of relations
 Schema |                  Name                   |   Type   | Owner
--------+-----------------------------------------+----------+-------
 public | appliance_state                         | table    | admin
 public | compatibility                           | table    | admin
 public | container_group_membership              | table    | admin
 public | container_group_membership_id_seq       | sequence | admin
 public | email_report_settings                   | table    | admin
 public | entity_display_path                     | table    | admin
 public | entity_display_path_id_seq              | sequence | admin
 public | esx_hosts                               | table    | admin
 public | esx_hosts_id_seq                        | sequence | admin
 public | group_app_client_targets                | table    | admin
 public | group_app_client_targets_id_seq         | sequence | admin
 public | identity                                | table    | admin
 public | identity_id_seq                         | sequence | admin
 public | job_migration_history                   | table    | admin
 public | job_migration_history_id_seq            | sequence | admin
 public | locked_backup_retentions                | table    | admin
 public | mc_activity_monitor                     | table    | admin
 public | mc_replication_activity_monitor         | table    | admin
 public | user_log                                | table    | admin
 public | user_log_id_seq                         | sequence | admin
 public | v_vm_group_membership_by_container      | view     | admin
 public | vcenter_event_monitor                   | table    | admin
 public | vdp_migration_history                   | table    | admin
 public | vdp_migration_history_id_seq            | sequence | admin
 public | vm_group_membership_by_container        | table    | admin
 public | vm_group_membership_by_container_id_seq | sequence | admin

If you are missing one or more tables, the vdr service is not initialized and the connection fails.
To recreate the missing tables, open a case with VMware Support. I had to fix these tables manually. If someone has a better way, I'm open for suggestions.

Post recreating the tables, restart the tomcat service using:
# emwebapp.sh --restart
That's it!