vRA 7.5 – Hostname Change: Host has been updated successfully, but is not updated in SSO

(The same procedure applies for vRA 7.6)

Use Case

The other day I was working at a customer who wanted to move his vRA environment to an external F5 Load Balancer VIP with a new FQDN in a different namespace (example.com). The environment was currently using an NSX Load Balancer and configured with Custom Signed Certificates of an internal domain (example.local). The new FQDN would not be part of this internal domain’s namespace.

The Old Days – vRA 7.3

Back in the old days of vRA 7.3 you had the ‘Host Settings’ and ‘SSL Configuration’ on the same page. So when you updated the host name of your vRA Cluster, you had the option to also immediately update the SSL Certificates at the same time as shown in the screenshot below:

vRA 7.3.png

In vRA 7.5 the ‘Host Settings’ page has no ‘SSL Configuration’ listed anymore. This has been moved to the ‘Certificates’ tab as shown below:

Schermafbeelding 2019-05-31 om 11.48.41

So I was keen to find out what happened if we updated the hostname with the new FQDN 🙂

*note: all these tests were conducted in a Dev/Test/Lab environment.

Prerequisites

The customer was responsible for creating the F5 Load Balancer VIP & the necessary DNS records + Firewall rules based on our input.

The following was in place:

Verified health of vRA environment before proceeding
F5 Load Balancer VIP with the necessary Monitors, Server Pools, …
Based on VMware Documentation: https://docs.vmware.com/en/vRealize-Automation/7.5/vrealize-automation-load-balancing.pdf
DNS Records, taking into account Split Brain DNS
Firewall Rules so that the necessary components (e.g.: vRB, vIDM) could communicate with the new FQDN
Connectivity towards the solution tested / verified before proceeding
Snapshots taken of the environment
Determined the Master vRA Appliance via VAMI
Unregister vRB (if applicable)
Make sure to know / write down your SSO Admin Default Tenant, User & Password. You’ll be needing it to reconfigure SSO (mandatory step after host name change).

vRA 7.5 Cluster – Host name change

Log on to the VAMI of the master vRA Appliance
Go to ‘VRA‘ – ‘Host Settings‘ and select ‘Update Host‘
Enter the new FQDN of the External F5 Load Balancer VIP
Click ‘Save Settings‘
Now vRA will initiate the process of changing the host name. It will take a while, the status of the process is being shown all the time. See below:
In my case the process failed at ‘Operation Completed on: 40 %, Step: Synchronizing SSO Services‘. It was hanging on this step a long time, afterwards it showed the following error message in the VAMI:

Error: Host has been updated successfully, but is not updated in SSO
*NOTE: Please notice here that your new FQDN does show up under ‘Host Name’ on this page. (deleted from my screenshot).
This resulted in the cluster being in an inconsistent state because it failed to update / sync the SSO service(s) with the new host name. A lot of services could not be started / were not showing up as ‘REGISTERED’ in the VAMI interface.
Analyzing different logs on the vRA Appliance, I stumbled on the following in the ‘Horizon.log‘ file:
019-05-08T13:15:43,525 ERROR (tomcat-http–58) [vsphere.local;86317d63-1063-4941-be9a-15f4122c4fe9;127.0.0.1;] com.vmware.horizon.common.api.token.SuiteToken – Couldn’t get suite token public key from tenant.

javax.net.ssl.SSLPeerUnverifiedException: Certificate for <NEW HOSTNAME> doesn’t match any of the subject alternative names: [OLD SAN 1, OLD SAN 1, OLD SAN 2, …, <IP ADDRESS>]
Apparently vRA 7.5 is not updating the certificates with the new FQDN when changing the host name of the vRA Cluster. Hence it will not trust the different components using certificates because these certificates do not contain the new FQDN as Subject Alternative Name.
We can maybe consider it being a ‘chicken or the egg’ problem. Because when you want to generate a new self signed certificate, it will take the host name on the ‘Host Settings’ page as the Common Name. The Common Name is then grayed out, so you cannot modify it beforehand to represent the new host name. You actually first need to save the Host Name with ‘Update Host’ on the ‘Host Settings’ page and let it fail to be able to generate a new Self Signed Certificate containing the new host name as Common Name. So that’s what we did.
As highlighted above, your vRA Cluster is now in an inconsistent state. Move over to the ‘Certificates‘ page in the VAMI and make sure ‘vRA‘ is selected as ‘Component Type’.

You’ll notice that your certificate is still using the old FQDN as ‘Common Name’. If you now generate a new Self Signed Certificate, it will take the new FQDN Host Name set on the ‘Host Settings‘ page as common name.
Click ‘Generate Certificate‘ and select ‘Save Settings‘. Make sure to let the process complete and you’ll notice that your new certificate now contains the newly set FQDN as ‘Common Name’. Let’s try the Host Name renaming again!
In the VAMI go to ‘Host Settings‘, select ‘Update Host‘, confirm that the FQDN set there still matches your newly set FQDN and is the same as the one shown in the ‘Common Name’ section under the ‘Certificates’ tab.
Click ‘Save Settings‘ and lat the ‘Update Host Process‘ finish. You’ll notice that it will now go through each stage of the process and not fail at 40%. In the end you’ll see the following message:

Success: Host Settings are updated. Make sure to re-configure the SSO settings.
Go over to the ‘SSO‘ tab fill in the initial SSO Admin User (usually administrator@vsphere.local) password and click ‘Save Settings‘. After saving the settings, the process should be finished like this:
Now the Host Name Change has been finished in the VAMI interface, check if all the services show up as ‘REGISTERED‘ in the VAMI.
If ok, it’s time to repoint some elements in vRA to the new FQDN!
Browse to the default tenant (vcac) by using your new FQDN with your tenant administrator credentials
Go to ‘Administration‘
Select ‘Directories Management‘
Go to ‘Identity Providers‘
Select your ‘Active Directory Identity Provider‘ and update the ‘IdP Hostname’ with your new FQDN as shown below (if applicable):
Register vRB back with vRA by going to the vRB’s VAMI interface. (if applicable)
Log on to vRA with the default tenant as infrastructure administrator
Go to ‘Infrastructure‘ – ‘Endpoints‘ and update your embedded vRO endpoint to point ti to the new FQDN as shown below (Address):
Optionally you can perform a ‘Data Collection‘ (under ‘Actions’) on this vRO endpoint to validate if it’s working.
Now we could update the environment with the new certificates from the different namespace by importing them.