Upgrading to vRealize Operations 8.6

vRealize Operations 8.6 was GA on 12^th Oct 2021 and I would consider this as a major release looking at the long list of new features and enhancements available. A public maintenance release of v8.6.2 was further released on 21st Dec 2021 which addresses some security vulnerabilities disclosed and also some performance issues.

Below are some of the key features which makes it a compelling reason to upgrade to vRealize Operations 8.6.2 :

1. Resolves CVE-2021-22033. For more information on this vulnerability and their impact on VMware products, see VMSA-2021-0021.
2. Resolves CVE-2021-44228, CVE-2021-45046. For more information on this vulnerability and their impact on VMware products, see VMSA-2021-0028.5
3. The End Point Operations Management Solution and the related plugins have been deprecated and will not be supported..
4. Complete revamp of product navigation for better user experience and findability. See vRealize 8.6 UI Navigation and vRealize Operations 8.6 Administration.
5. New Sustainability home page and sustainability dashboards to help identify CO2 emissions and power savings. Refer to Sustainability dashboards in vRealize Operations 8.6
6. Costing Enhancements – New Cost homepage, ability to import and export cost drivers, pricing card for AVS, GCVE.
7. Capacity Enhancements – New ‘Peak-focused’ risk level in Capacity Settings, ability to fine-tune Idle VMs definition, VM shortfall for clusters
8. VSAN HCI mesh integration for simplified troubleshooting.
9. Fully customizable email notification templates.
10. Ability to apply perpetual and subscription licenses together on the same instance of vROps 8.6. 
11. New metrics and properties added in vRealize Operations 8.6 – Metrics added in vRealize Operations 8.6

For the full list of new features and enhancements, please refer to  https://blogs.vmware.com/management/2021/10/whats-new-in-vrealize-operations-8-6.html

Important Note before planning for upgrade:

• The minimum hardware version required for vRealize Operations 8.x releases is version 11. If your vRealize Operations virtual appliance had a hardware version earlier than 11, you must first upgrade to hardware version 11 on vRealize Operations virtual appliance and then upgrade to vRealize Operations 8.6.
• Support for Internet Explorer has been dropped from vRealize Operations 8.4 onwards.
• Direct upgrade from vRealize Operations 7.5 to vRealize Operations 8.6 is not supported. For upgrades from vRops 7.x,  please upgrade to vRops 8.5 as first step ( Follow steps from : https://thinkvirtuals.com/2021/08/01/upgrading-to-vrealize-operations-8-5/ ) and then follow with upgrade to vRops 8.6.
• The End Point Operations Management Solution and the related plugins have been deprecated and will not be supported.
• Upgrading to vRealize Operations 8.6, resets out-of-the-box content as part of the software upgrade process. This implies that the user modifications made to default content such as alert definitions, symptom definitions, recommendations, policies, views, dashboards, widgets, and reports are overwritten. You need to clone or backup the content before you upgrade to vRealize Operations 8.6.

vRealize Operations 8.6 upgrades can be done through vRealize Suite Lifecycle Manager 8.6 and several blogs are available on this process, I followed this article – https://www.brockpeterson.com/post/vrealize-8-6-updates-via-vrslcm

Below are the steps documented for the process for upgrading directly from vRealize Operations.

Down-time required	Yes
Time Required	Roughly 2 to 3 hours for the environment of 2 x Large Nodes vROps ( This is a rough estimate and it can vary considerably with several environmental factors )
Impacts during upgrade	No Metrics will be collected during the upgrade window.
Roll-back Plan	Snapshots need to be taken ( with vROps Cluster Offline – very important) prior to initiating upgrade and this snapshot should be used as a roll-back mechanism for any failures reported during upgrade steps.

We will be following the below steps for the upgrade activity:

Phase 1: Running Pre-upgrade Assessment tool – Time required: 15 minutes – Strongly Recommended to do this for any upgrades to identify metrics and contents that could be impacted with the upgrade.

1. Download upgrade assessment tool : vRealize Operations 8.6 – Pre-Upgrade Readiness Assessment Tool  ( Name of File: APUAT-8.6.0.18696447.pak, Size of File: 6.04 MB )
2. Log in to the master node vRealize Operations Manager Administrator interface of your cluster at https://master-node-FQDN-or-IP-address/admin.
3. Click Software Update in the left panel.
4. Click Install a Software Update in the main panel.
5. Follow the steps in the wizard to locate and install your PAK file. ( Check Install the PAK file even if it is already installed )
6. Install the Upgrade Assessment Tool.
7. Wait for the software update to complete.
8. Access to Pre-upgrade Readiness assessment report  ( Refer to KB 85833) – steps as below:

6. Navigate to the Support > Support Bundles tab.
7. Download the light support bundle that was generated from the installation of the Upgrade Assessment Tool.
8.  In the downloaded support bundle, open the cluster_timestamp_nodeaddress/nodeaddress_timestamp_nodeaddress/apuat-data/report/index.html file.
9. A list of all potentially impacted user content , if any , will be displayed in the “Removed / Discontinued Metrics “ tab
10. In the tab of “System Validation Checks” , you will find the report of system validation checks for upgrade and recommendations , if any. Follow the recommendations as suggested.

9. If there are any remediations suggested for “System Validation Checks” in the above step , run the pre-upgrade assessment tool again after remediation tasks are completed. This is to make sure that all the upgrade pre-requisites are met before proceeding to the next phase of upgrade.
10. Proceed to the next phase ONLY after all remediation activities, if required, and all the system validation checks are green. 

Phase 2 : Upgrading to vROps 8.6  from vROps 8.x

1. Recommended to remove or update all impacted Management Packs, if any, prior to initiating the upgrade.
2. Review the vROps 8.6 sizing for the environment prior to upgrade using vRops sizing calculator OR follow the sizing guidelines from KB 85832. Make sure that the vROps nodes are sized based on the recommendations.
3. There are two different upgrade files available for upgrading to vRealize Operations 8.6 depending on the deployment of vRealize Operations with cloud proxy or not. Download the following upgrade files – Use the download link from myvmware portal

Type of Deployment	File Name	Build#	File Size	File Type
vRealize Operations without Cloud Proxy	vRealize_Operations_Manager-VA-8.x-to-8.6.2.19081810.pak	19081810	2.44 GB	.pak
vRealize Operations with Cloud Proxy	vRealize_Operations_Manager_With_CP-8.x-to-8.6.2.19081810.pak	19081810	3.37 GB	.pak

4. Make sure the root logins to all the vROps nodes are available.
5. Make sure passwords for “root” account and “admin” account are reset and is not expired or not locked (On all nodes) – check by logging into SSH console of every node with root and admin accounts . If you have not logged into SSH earlier , the default password will be blank and just press enter at password prompt. After this it will ask for setting a new password.
6. Make sure vROps cluster’s  “admin” account’s  password is not expired or locked – you may change from vROps admin portal.
7. Make sure all the vROps nodes have sufficient disk space for upgrade (Minimum 5% free disk space required for each node): Check with command “df -h” from SSH session to each node.
8. Optional step: If you have a large vROps cluster with more than 4 nodes and deployed across firewalls, the upgrade process can be made faster by pre-copying upgrade files to all nodes. Pre-copy the upgrade files to all nodes except Master Node – follow steps as in KB 2127895
9. Take the vROps cluster offline from Admin UI.
10. Shut down the nodes in the sequence of Remote Collectors , Data nodes , Replica Node and Master Node.
11. Take the snapshot of all the nodes.
12. Power on the nodes in the sequence of Master Node , Replica Node , Data nodes and Remote collectors
13. Initiate vROps In-place upgrade from Admin UI. 
14. Once the upgrade is completed, verify functionality. If all functionality is reported correctly, take the cluster offline, shutdown nodes and delete the snapshots. Once snapshots are removed, power on the nodes in sequence of Master Node , Replica Node , Data Nodes and Remote Collectors
15. Once all nodes are powered on , bring the vROps cluster Online.

You may also please refer to short video on these steps for upgrading vRealize Operations, by my friend Basem Shendy – Youtube Video: Step by Step Upgrade for vRealize Operations

Additional resources :

KB 2001476 – How to reset the root password in vRealize Operations

KB 2078313 – How to reset the admin password in vRealize Operations Manager

Quick access to Commands available in vRops Nodes:

To reset vRops Cluster Admin password from root console :

$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsSetAdminPassword.py –reset

To reset root or admin account in OS – to be run on all nodes

passwd root

passwd admin

To unlock the admin and root accounts in vRops – to be run on all nodes

pam_tally2 –user admin –reset
pam_tally2 –user root –reset

To validate if the accounts are locked  – to be run on all nodes:

passwd -S root

passwd -S admin

( If the account is not locked, you will see PS next to the username.  If the account is locked, you will see LK next to the username )