Invoking the vRealize Automation API – Part I

This post was inspired by a desire to speed up the prep time of my demos. We use nested demo environments hosted inside vCloud Director. The nested environments have resource limitations and we sometimes have to shut down unused VMs in a demo environment to ensure that other components get enough resources to execute. I also wanted to do as little prep work inside vRA as possible – automatically launch blueprints so I have a few managed VMs to show off. My idea was to write a PowerShell script that could be easily launched from the desktop.

First, I did a simple install of vRA in my home lab (this was back in May, vRA 7.2).  I’d like to thank my friend Eric Shanks for his fantastic vRA7 guide available at The IT Hollow. His posts have been extremely valuable in helping get my lab environment working. When I built my environment, I used the same Windows 2012 template machine for both my IAAS box as my SQL Server. This ended up being a major source of trouble for me, which I will detail later.

This week, I started following Eric’s guide to configure vRA. I got it to the point of creating a new tenant and got AD authentication working. Then I tried using the vCenter endpoint that had been created but the logs were throwing SSL errors. I deleted it and recreated it, which was successful, but then I saw logs in Infrastructure>Monitoring>Logs that said it was looking for something named ‘vCenter’. So I deleted the endpoint again and named it vCenter.  I tried a bunch of stuff including deleting and recreating, then I got other errors and eventually got it to work and I saw my compute resources under the vCenter endpoint.

I moved on to try making a Fabric Group, but I could only select my lab cluster, it didn’t have any resources in it, I couldn’t assign any compute or storage. I went back to the logs and found “DataBaseStatsService: ignoring exception:  Error executing query usp_SelectAgent  Inner Exception: Error executing query usp_SelectAgentCapabilities”

I googled the error and came up with this Communities page as well as KB543238
They both pointed me to MSDTC being a problem. But the KB seemingly only applied to vRA 6.x. I followed the communities post and tried uninstalling and reinstalling MSDTC, but no success.

At this point I wondered if I was hitting some 7.2 bug. Since 7.3 was out, I ran an upgrade. The vRA appliance and IAAS box upgraded without issue.  As soon as I logged back in, the vCenter Endpoint wasn’t working at all. The log was full of errors saying “Failed to connect to the endpoint. To validate that a secure connection can be established to this endpoint, go to the vSphere endpoint on the Endpoints page and click the Test Connection button. Inner Exception: Certificate is not trusted (RemoteCertificateChainErrors).”

Per the vRA 7.3 Release Notes, certificate validation is turned on. Not wanting to mess around with signed certificate replacement in the lab,  I got around this problem by downloading the root CA certificate from the homepage of my VCSA, and installing it in the Trusted Root Certification Authorities bucket on the IAAS box. Making this change brought me back to the usp_SelectAgent error. I logged into SQL and tried to see if I could execute the usp_SelectAgent stored procedure, which worked fine.

Having debugged the problems for the better part of two days at this point, I went for help, which thankfully came quickly in our internal message board. My problem was definitely the MSDTC – even if you Sysprep a box, it doesn’t reset the MSDTC unique CID – so the IAAS box was unable to communicate with the SQL server.

I followed this procedure to reset the CID on both SQL and IAAS:

1. Stop the Manager Service.
2. Stop the SQL Server service.
3. Open a command prompt on the machine with the Manager Service and issue the following command:
msdtc -uninstall
4. Open a registry editor on the Manager Service and delete the following keys if they exist:

HKLM/Software/Microsoft/Software/MSDTC
HKLM/System/CurrentControlSet/Services/MSDTC
HKEY_CLASSES_ROOTCID

5. Reboot the machine with the Manager Service.
6. Open a command prompt on the machine with the Manager Service and issue the following command:
msdtc -install
7. Perform steps 3-6 on the machine running the SQL Server.
This procedure generates new CID values for MSDTC on both servers.

After this procedure was completed, everything worked and I was able to continue my vRA configuration without issue.

In Part II, I will cover how I learned some basic vRA API operations.

 

 

Custom Dashboards in vROps 6

I was doing a vRealize Operations demo with a customer today and they had a specific request on how to see some CPU data. They wanted to see a list of physical hosts, CPU utilization metrics on each of those hosts, and then be able to drill into specific CPU stats for the VMs running on the host. We will create a custom dashboard to easily display this information.

Here’s the finished product first. On the top left, we want all of our hosts and clusters to display. When you click on a host or cluster, we want the host metrics to show up in the top right box. Then, we want all of the VMs in the host or cluster to show up in the bottom left box with VM-specific metrics.

18-DashboardResults2-ViewHost

First, we want to create a new custom view for the host CPU metrics. There are many out-of-the-box views inside vROps – you can use any of them, or create your own. We will see both methods in this post – a custom view for the host metrics, and an out-of-the-box view for the VM metrics.

To create a new view, go into Views and click the green plus.

1-NewView

Name the view and add a description.

2-ViewName

Pick how our data will display – in this case we’ll want the data displayed in a list format with columns.

3-ListView

We want vSphere host metrics, which come from the vCenter Adapter. We pick vCenter Adapter, then Host System

4-SubjectsvCenterAdapter

5-Subjects-HostSystem

We want 3 host metrics to show. CPU Demand will show me how much CPU the VMs on the host are demanding. CPU Capacity Usage will show me how much CPU is actually used. CPU Demand can be higher than CPU Capacity Used due to limits, either directly on the VM or imposed by resource pools. There are resource pool limits in this test environment, so we might expect to see higher CPU demand than usage. The final metric we want is CPU Contention. We drag them from the metrics on the right to the include box in the middle.6-Metrics

Finally, we pick the availability settings for our new view. We want to be able to include it in Dashboards, so we make sure that box is checked. A couple other boxes are checked by default – we leave them checked.

7-Visibility

 

Now we create a new dashboard from the Home screen, click on Actions, then Create Dashboard.

8-CreateDashboard

Name the dashboard and provide a description.9-NameDashboard

We’re going to add 3 widgets to our dashboard. First, we drag an Object List widget into the top left corner. We then drag a View widget into the top right and bottom left.

10-AddWidgets

 

 

Now, we customize the Object List. Click on the Edit button.

11-EditObjectList

We name the Object List. We only want vSphere Hosts and Clusters showing up, so we expand the Object Types option.

12-ModifyObjectList

We want Cluster Compute Resources and Host System. We click on the first one, then Ctrl-Click to highlight both.

13-SelectClusterResource14-SelectHostSystem

After these changes, we save the object list.

Now, we edit the View widget on the top right. We name it Host CPU Summary, then pick our Custom CPU view that we created at the beginning of this post.

15-HostCPUSummary

We edit the bottom left view widget. We name it VM CPU Details, and we pick a standard view called Virtual Machine CPU Diagnose List.

16-VMCPUDiagnose

Finally, we modify the Widget interactions. When we select a host or cluster object in the Host / Cluster list box, we want it to change the two view boxes. We configure the Widget Interactions to use the Host / Cluster List selection as the data source, and we have it feed the Host CPU Summary and VM CPU Details view boxes. Click Apply Interactions to save the interactions.

10a-WidgetInteraction

 

In our completed dashboard, we click on the demo-mgmt Cluster. All of the hosts in the cluster show up in the Host CPU Summary box. All of the VMs in the cluster show up in the VM CPU Details box.

17-DashboardResults1-ViewCluster

This is an example of clicking a single host – only the metrics for the one host show up in the Host CPU Summary box, and the VMs running on that one host show up in the VM CPU Details box.

18-DashboardResults2-ViewHost

Here we see more of the metrics available in the Virtual Machine CPU Diagnose List view. Again, we could have created a custom view for the widget instead – it all depends on what metrics you want to show.

19-DashboardResults3-ViewHost

Here is a link to the zipfile containing the JSON dashboard definition and the XML definition for the Custom CPU view that we created.

vROps Custom CPU exported objects

Custom Groups and Policies in vROps 6

This post is based on our Hands-On Lab HOL-SDC-1610, which you can use for free at http://labs.hol.vmware.com

This shows you how to create a custom monitoring policy in vROps 6.

First, this is what my cluster looks like in the lab vCenter Web Client. In this scenario, we want to have a custom monitoring policy for all VMs in Cluster Site A because they are critical VMs and need a more aggressive monitoring policy. We want to change the memory % contention object to give us an alert at a lower percentage of contention.

1-Site-A-VMs

 

We go into Custom Groups from inside vROps and click on the plus to add a new group

2-CustomGroups

 

We name the group “VMs in Cluster A Prod”, pick a Group Type of “Function”, and for now pick the Default Policy. There are various group types – in this case we are separating the VMs based on function (Critical Prod).  We check the Keep group membership up to date box. This ensures that new VMs added to the cluster get picked up by the group.

We want to select VMs, so the Object Type is Virtual Machine. We want to select VMs based on the cluster that they’re on. In the vROps nav tree, VMs are descendants of a cluster. We set the object criteria to Relationship, Descendant of, and contains. We set the nav tree dropdown to “vSphere Hosts and Clusters”

3-NewGroup

 

The name box autofills as we type – Cluster Site A appears, we click on it to fill the box. We now have our custom group of all VMs inside Cluster Site A.

4-NewGroup_2

 

We now move into the Policy Library. The default policy is indicated with a priority of “D”. The concept of inheritance lets you have a master default policy, and you can then override a few specific settings based on group membership.

5-DefaultPolicy

 

We’re going to create a new policy for Cluster A and base it on the Default policy.

6-NewPolicy

 

We jump down to the Alert/Symptom Definitions.

6a-SymptomDef

 

To easily find  our symptom, we can pick vCenter Adapter>Virtual Machine from this dropdown, and then use “memory” on the filter box to find all VM-related memory Symptoms.

7-MemoryPolicy

 

Here, I’ve changed the State to Local, and Override, then changed the threshold from 10 to 8. Any VMs bound to this policy will alert when the memory contention reaches 8% instead of the default of 10%.

8-OverrrideContention

The final step is to select our groups that will use the new policy. We check the box for our VMs in Cluster A Prod custom group.

9-PickGroup

Here is the Default policy with its subordinate policies. In Lab 1610, there is also another subordinate policy for a specific VM, linux-App-02a. This is an example of how granular you can get with your policies, getting down to overriding settings even for a specific VM.

10-Policy

 

We have a YouTube video on this topic as well: Customize Operational Policies in vRealize Operations Manager

Custom Views and Dashboards in vRealize Operations

This post covers a few of the most common questions my customers ask me as I demonstrate what you can do with vROps. I’m going to take you through an example of needing to frequently check the CPU ready % of your VMs – this was my customer’s most recent request, but know that you can make this happen for any metric collected by vROps.

First, we’re going to create a custom view for CPU Ready % by going to Content>Views, then clicking on the green Plus to add a new View.

1-AddView

I named this one “Custom-CPU Ready” and gave it a description.

2-CustomCPUReady

Next, pick what our View looks like. In this case, I want to see all of the data in a list format, so I pick List.

3-Presentation

Now to select the subjects – these are the objects that the View will be looking at. We want CPU Ready % on Virtual Machines, so we pick the vCenter Adapter and scroll down until we find Virtual Machine.

4a-Subjects

 

4b-Subjects

We now need to find the CPU Ready % metric

5-CPUData

Double-click on it when you find it in the list on the left, it will then appear in the Data section. Change the Sort order to descending because we want to see the VM with the highest CPU ready on top.

6-SelectReady

The Availability options let you control where inside vROps the View will be usable. I lef the defaults.

7-Visibility

You now see the custom view in the Views list.

8-CustomView

How can we use our brand new view? We want to see the CPU ready for all VMs in the Production cluster. Go to Environment, then drill down into the vSphere World until you reach the Production cluster. Click on the Details tab, you can then scroll down and find the custom View that we created. Click on it and all of your VMs show up, sorted by highest CPU ready.

9-ShowReady

 

Let’s say this is a metric that you look at at daily or multiple times a day. You can create a custom dashboard so the metric is immediately visible if you’re using vROps Advanced.

To create a new dashboard, from the Home menu, click Actions, then Create Dashboard

1-AddDashboard

Name the dashboard and select a layout.

2-NameDashboard

We want to show a View in the widget, so we drag View over into the right pane.

2a-WidgetList

Click the Edit icon in the blank View to customize it.

3-EditWidget

Click on Self Provider to allow us to specify the Production Cluster object on the left, then select our Custom CPU Ready View on the right and click Save.

4-AddWidget

The dashboard is now ready. The CPU Ready for the Production VMs will now show up in the dashboard.

5-FinalDashboard