Invoking the vRealize Automation API – Part II

Posted on July 10, 2017 by pkremer

In Part I, I talked about why I wanted to learn API calls in vRA and how I got my lab environment working. In Part II, I will talk about how I learned how to make an API call.

I relied heavily on Grant Orchard’s getting started guides. I have linked to Part I, II, and III below, with my explanations of how I used his blog to achieve my goal.

Part I – Getting Started [grantorchard.com]

I couldn’t figure out how to browse through API calls because I wasn’t seeing what Grant was showing. It took me forever to realize that at the very bottom of the page, you can click on Show/Hide – then the API calls appear and you can drill into each one for full details.

Part II – Building Your First API Call [grantorchard.com]

Grant wrote:
Before we start, perform the following steps.
1. Download Postman.
2. Import this Postman collection of the vRA 7.2 API.
3. Import this Postman environment variables file.
4. Open up the API docs at https://{{vra-fqdn}}/component-registry/services/docs

Postman? What’s Postman? It’s a GUI tool to issue API calls.

What’s a Postman collection? It’s a group of API calls that you can easily click on in the GUI.

I can easily search a Collection for the API I want. In this case, I know I want to get the Bearer token (Grant explains this, it’s how the API requests are authenticated), so I search for ‘token’. I click on the “returns a token associated with the provided credentials” and it opens up the request complete with the proper URL. It saves me from having to manually piece together the API calls and paste them into Postman.

You’ve probably noticed {{vra-fqdn}} in the URL. It’s not just a placeholder. It’s an environment variable.

Grant provided a bunch of environment variables in his post – you can import the environment variables and change them to match your lab environment. You can reference these variables inside Postman.

Following Grants example, I opened the token API in Postman. The ‘Tests’ section saves the Bearer token in a variable named “token”.

Part III – Requesting a Catalog Item [grantorchard.com]

Grant’s post said the API I needed was ‘entitledCatalogItemViews’. You can see that I’m using the {{vra-fqdn}} variable in the URL as well as passing the Bearer {{token}} value. One problem I ran into is that you must have a space between Bearer and {{token}}.

Hit Send and my results come back. I have only one blueprint, a Linked Clone blueprint with Photon Linux in it. You can see two links – one for the GET: Request Template, and the other for the POST: Submit Request. The Request Template will return an example set of JSON showing you how to make the POST call to start the Blueprint.

Now I open another Postman tab and paste in the Request Template URL. Add the proper header for Authorization, and hit Send.

This is just a subset of the JSON I got back. I left the tab open and launched a new tab.

In the new tab, I used the URL in the Submit Request response that I got above. I did the same Authorization header as used previously, and pasted the Template JSON from above into the Body field.

After pressing Send, I got this response in the Body. You can see a Request ID as well as a state of “Submitted”

There is also an API where you can check on the state of a request. You can see now that the state has changed from Submitted to In Progress. You can keep

You can see my request in progress inside vRA

You can also see activity in the vSphere Web Client.

You can continue checking on the provisioning status by clicking Send in Postman. You would do the same thing programmatically – periodically ping the API for this asynchronous request to determine when it has completed. We now see that the status code is Successful instead of In Progress

I now have a new Item in vRA.

Now that I know the correct APIs to use, and that they work as expected in my lab environment, I can get to work calling them from Powershell. Part III of the series will document this process.

Invoking the vRealize Automation API – Part I

Posted on July 7, 2017 by pkremer

This post was inspired by a desire to speed up the prep time of my demos. We use nested demo environments hosted inside vCloud Director. The nested environments have resource limitations and we sometimes have to shut down unused VMs in a demo environment to ensure that other components get enough resources to execute. I also wanted to do as little prep work inside vRA as possible – automatically launch blueprints so I have a few managed VMs to show off. My idea was to write a PowerShell script that could be easily launched from the desktop.

First, I did a simple install of vRA in my home lab (this was back in May, vRA 7.2). I’d like to thank my friend Eric Shanks for his fantastic vRA7 guide available at The IT Hollow. His posts have been extremely valuable in helping get my lab environment working. When I built my environment, I used the same Windows 2012 template machine for both my IAAS box as my SQL Server. This ended up being a major source of trouble for me, which I will detail later.

This week, I started following Eric’s guide to configure vRA. I got it to the point of creating a new tenant and got AD authentication working. Then I tried using the vCenter endpoint that had been created but the logs were throwing SSL errors. I deleted it and recreated it, which was successful, but then I saw logs in Infrastructure>Monitoring>Logs that said it was looking for something named ‘vCenter’. So I deleted the endpoint again and named it vCenter. I tried a bunch of stuff including deleting and recreating, then I got other errors and eventually got it to work and I saw my compute resources under the vCenter endpoint.

I moved on to try making a Fabric Group, but I could only select my lab cluster, it didn’t have any resources in it, I couldn’t assign any compute or storage. I went back to the logs and found “DataBaseStatsService: ignoring exception: Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities”

I googled the error and came up with this Communities page as well as KB543238
They both pointed me to MSDTC being a problem. But the KB seemingly only applied to vRA 6.x. I followed the communities post and tried uninstalling and reinstalling MSDTC, but no success.

At this point I wondered if I was hitting some 7.2 bug. Since 7.3 was out, I ran an upgrade. The vRA appliance and IAAS box upgraded without issue. As soon as I logged back in, the vCenter Endpoint wasn’t working at all. The log was full of errors saying “Failed to connect to the endpoint. To validate that a secure connection can be established to this endpoint, go to the vSphere endpoint on the Endpoints page and click the Test Connection button. Inner Exception: Certificate is not trusted (RemoteCertificateChainErrors).”

Per the vRA 7.3 Release Notes, certificate validation is turned on. Not wanting to mess around with signed certificate replacement in the lab, I got around this problem by downloading the root CA certificate from the homepage of my VCSA, and installing it in the Trusted Root Certification Authorities bucket on the IAAS box. Making this change brought me back to the usp_SelectAgent error. I logged into SQL and tried to see if I could execute the usp_SelectAgent stored procedure, which worked fine.

Having debugged the problems for the better part of two days at this point, I went for help, which thankfully came quickly in our internal message board. My problem was definitely the MSDTC – even if you Sysprep a box, it doesn’t reset the MSDTC unique CID – so the IAAS box was unable to communicate with the SQL server.

I followed this procedure to reset the CID on both SQL and IAAS:

1. Stop the Manager Service.
2. Stop the SQL Server service.
3. Open a command prompt on the machine with the Manager Service and issue the following command:
msdtc -uninstall
4. Open a registry editor on the Manager Service and delete the following keys if they exist:

HKLM/Software/Microsoft/Software/MSDTC
HKLM/System/CurrentControlSet/Services/MSDTC
HKEY_CLASSES_ROOTCID

5. Reboot the machine with the Manager Service.
6. Open a command prompt on the machine with the Manager Service and issue the following command:
msdtc -install
7. Perform steps 3-6 on the machine running the SQL Server.
This procedure generates new CID values for MSDTC on both servers.

After this procedure was completed, everything worked and I was able to continue my vRA configuration without issue.

In Part II, I will cover how I learned some basic vRA API operations.

New job, same company

Posted on April 21, 2017 by pkremer

I’m excited to announce my promotion to the newly formed Dell Execution Team (DET) at VMware, effective May 8, 2017! No, we’re not a team of highly trained assassins. The Dell Execution Team helps drive VMware’s software-defined datacenter message through the strength of Dell Technologies. I’ve already done a few DET meetings with customers and I’m beyond thrilled with customer response. Although I’m leaving my role as a Commercial Systems Engineer, I do get to remain in the same excellent Commercial organization at VMware – new manager, same director.

We are actively hiring for my old job! Check out the job posting. If you’re in Chicago and have great VMware skills, hit me up! If I know you, I’ll refer you.

VCIX6 designation clarifications

Posted on December 20, 2016 by pkremer

There is a lot of confusion out there regarding the upgrade paths, the VCIX6, and underlying VCP6 requirements to achieve certification. The Certification folks are working on clarifying language on our website and accurate instructions for our customer-facing employees behind the certification@vmware.com alias.

I am writing this point from the standpoint of the Datacenter Virtualization exam, since that is the track that I am following for my VCDX attempt. If you’re in a different track, the same policy applies for your specific track.

If you are brand new to the DCV track, you have to pass the VCP6-DCV exam. You can’t use a VCP6-DTM to start up the DCV track

If you are a VCAP5-DCA, you can pass the VCAP6-DCV Design exam to achieve VCIX6-DCV designation

If you are a VCAP5-DCD, you can pass the VCAP6-DCV Administration exam to achieve the VCIX6-DCV designation

If you are both a VCAP5-DCA and VCAP5-DCD, you can take either the VCAP6-DCV Design *OR* VCAP6-DCV Administration exam to achieve your VCIX6-DCV designation

Your VCP in the Datacenter Virtualization track must be valid (unexpired). VCAP5 holders with a valid VCP do NOT have to take the VCP6-DCV exam to sit a VCAP6 exam.

Passing the VCAP6-DCV Design or Administration exam extends the expiration date of your VCP for 2 additional years

Achieving the VCIX6-DCV designation will not give you the underlying VCP6-DCV certification.

The VCIX6-DCV is the only prerequisite for VCDX. You DO NOT need a VCP6-DCV certification

VCAP6-DCV Design exam 3V0-622 – Rescore

Posted on November 30, 2016 by pkremer

Update December 6, 2016

The rescore process is complete, all results have been posted to Cert Manager.

Update December 5, 2016

The batch processing at Pearson continues to fail. The certification team is manually updating all score results. This will take a considerable amount of time, but they are making good progress. The hope is to have all rescore results posted by the end of the day Pacific time on December 6th.

Original Post November 30, 2016

Exam takers who failed the 3V0-622 received a notice from Pearson that the exam was under review and might be rescored. The date in this email was that a rescore was expected by November 20th. We are obviously well beyond that date and people are still anxiously awaiting results of the rescore. I am among those waiting for news.

I have volunteered some of my time with the certification team as a SME to help develop exam content (not for 3V0-622). It’s given me insight into just how extraordinarily time consuming it is to create a legally defensible certification exam. No portion of the process is simple. It’s quite similar to putting code into production – even the slightest change means you have to run your entire battery of testing before promoting code. Any hiccup means re-running your tests from the beginning.

I have spoken internally with the Certification team at VMware regarding the staus of 3V0-622. They are doing everything they can to get the rescores out. However, you cannot magically make the processes work faster – the whole process from end-to-end takes 3-4 days. QA processes take the amount of time they take and cannot be rushed or skipped. Pearson has encountered a number of technical difficulties with the exam drivers and have had to run the process multiple times. Progress was further impeded by various resources being unavailable due to the Thanksgiving holiday last week.

At this point we are hoping for exam results to be available online on Friday December 2nd.

I’ve been silent lately

Posted on August 5, 2016 by pkremer

I have’t posted a blog entry in more than 4 months, and there’s a reason why! I’ve been working on a large charitable project sponsored by the VMware Foundation and it’s taken up all of my available time. VMware has an employee-led program called Kernel & Cache, allowing an employee to nominate a nonprofit to receive cash, software, and free engineering hours for a capacity building project. I designed the project and led a team of 11 VMware engineers on the deployment of the largest Kernel & Cache project in VMware’s history. I am using this project’s design as the basis for my VCDX attempt.

I have been invited to speak about the project at the Chicago VMUG Usercon on September 22nd at the Stephens Convention Center – I would love it if you register for and attend the conference! https://www.vmug.com/chivmug The UserCons are day-long events that are kind of like a mini-VMworld. This free event will have many sessions on VMware technology,a vendor exchange full of great partner technology, and the opportunity to network with your peers. Hope to see you out there!

Custom Dashboards in vROps 6

Posted on April 1, 2016 by pkremer

I was doing a vRealize Operations demo with a customer today and they had a specific request on how to see some CPU data. They wanted to see a list of physical hosts, CPU utilization metrics on each of those hosts, and then be able to drill into specific CPU stats for the VMs running on the host. We will create a custom dashboard to easily display this information.

Here’s the finished product first. On the top left, we want all of our hosts and clusters to display. When you click on a host or cluster, we want the host metrics to show up in the top right box. Then, we want all of the VMs in the host or cluster to show up in the bottom left box with VM-specific metrics.

First, we want to create a new custom view for the host CPU metrics. There are many out-of-the-box views inside vROps – you can use any of them, or create your own. We will see both methods in this post – a custom view for the host metrics, and an out-of-the-box view for the VM metrics.

To create a new view, go into Views and click the green plus.

Name the view and add a description.

Pick how our data will display – in this case we’ll want the data displayed in a list format with columns.

We want vSphere host metrics, which come from the vCenter Adapter. We pick vCenter Adapter, then Host System

We want 3 host metrics to show. CPU Demand will show me how much CPU the VMs on the host are demanding. CPU Capacity Usage will show me how much CPU is actually used. CPU Demand can be higher than CPU Capacity Used due to limits, either directly on the VM or imposed by resource pools. There are resource pool limits in this test environment, so we might expect to see higher CPU demand than usage. The final metric we want is CPU Contention. We drag them from the metrics on the right to the include box in the middle.

Finally, we pick the availability settings for our new view. We want to be able to include it in Dashboards, so we make sure that box is checked. A couple other boxes are checked by default – we leave them checked.

Now we create a new dashboard from the Home screen, click on Actions, then Create Dashboard.

Name the dashboard and provide a description.

We’re going to add 3 widgets to our dashboard. First, we drag an Object List widget into the top left corner. We then drag a View widget into the top right and bottom left.

Now, we customize the Object List. Click on the Edit button.

We name the Object List. We only want vSphere Hosts and Clusters showing up, so we expand the Object Types option.

We want Cluster Compute Resources and Host System. We click on the first one, then Ctrl-Click to highlight both.

After these changes, we save the object list.

Now, we edit the View widget on the top right. We name it Host CPU Summary, then pick our Custom CPU view that we created at the beginning of this post.

We edit the bottom left view widget. We name it VM CPU Details, and we pick a standard view called Virtual Machine CPU Diagnose List.

Finally, we modify the Widget interactions. When we select a host or cluster object in the Host / Cluster list box, we want it to change the two view boxes. We configure the Widget Interactions to use the Host / Cluster List selection as the data source, and we have it feed the Host CPU Summary and VM CPU Details view boxes. Click Apply Interactions to save the interactions.

In our completed dashboard, we click on the demo-mgmt Cluster. All of the hosts in the cluster show up in the Host CPU Summary box. All of the VMs in the cluster show up in the VM CPU Details box.

This is an example of clicking a single host – only the metrics for the one host show up in the Host CPU Summary box, and the VMs running on that one host show up in the VM CPU Details box.

Here we see more of the metrics available in the Virtual Machine CPU Diagnose List view. Again, we could have created a custom view for the widget instead – it all depends on what metrics you want to show.

Here is a link to the zipfile containing the JSON dashboard definition and the XML definition for the Custom CPU view that we created.

vROps Custom CPU exported objects

Custom Groups and Policies in vROps 6

Posted on February 12, 2016 by pkremer

This post is based on our Hands-On Lab HOL-SDC-1610, which you can use for free at http://labs.hol.vmware.com

This shows you how to create a custom monitoring policy in vROps 6.

First, this is what my cluster looks like in the lab vCenter Web Client. In this scenario, we want to have a custom monitoring policy for all VMs in Cluster Site A because they are critical VMs and need a more aggressive monitoring policy. We want to change the memory % contention object to give us an alert at a lower percentage of contention.

We go into Custom Groups from inside vROps and click on the plus to add a new group

We name the group “VMs in Cluster A Prod”, pick a Group Type of “Function”, and for now pick the Default Policy. There are various group types – in this case we are separating the VMs based on function (Critical Prod). We check the Keep group membership up to date box. This ensures that new VMs added to the cluster get picked up by the group.

We want to select VMs, so the Object Type is Virtual Machine. We want to select VMs based on the cluster that they’re on. In the vROps nav tree, VMs are descendants of a cluster. We set the object criteria to Relationship, Descendant of, and contains. We set the nav tree dropdown to “vSphere Hosts and Clusters”

The name box autofills as we type – Cluster Site A appears, we click on it to fill the box. We now have our custom group of all VMs inside Cluster Site A.

We now move into the Policy Library. The default policy is indicated with a priority of “D”. The concept of inheritance lets you have a master default policy, and you can then override a few specific settings based on group membership.

We’re going to create a new policy for Cluster A and base it on the Default policy.

We jump down to the Alert/Symptom Definitions.

To easily find our symptom, we can pick vCenter Adapter>Virtual Machine from this dropdown, and then use “memory” on the filter box to find all VM-related memory Symptoms.

Here, I’ve changed the State to Local, and Override, then changed the threshold from 10 to 8. Any VMs bound to this policy will alert when the memory contention reaches 8% instead of the default of 10%.

The final step is to select our groups that will use the new policy. We check the box for our VMs in Cluster A Prod custom group.

Here is the Default policy with its subordinate policies. In Lab 1610, there is also another subordinate policy for a specific VM, linux-App-02a. This is an example of how granular you can get with your policies, getting down to overriding settings even for a specific VM.

We have a YouTube video on this topic as well: Customize Operational Policies in vRealize Operations Manager

Workspace One screenshots

Posted on February 9, 2016 by pkremer

Today, VMware announced the launch of Workspace One and I wanted to throw a couple of screenshots out there. As a field engineer, I use Horizon Workspace every day to access my work applications. I’ve been using Workspace One for the last month and I’m happy with how responsive it is.

This is the Android URL to get the Workspace One App on your phone:
https://play.google.com/store/apps/details?id=com.airwatch.vmworkspace&hl=en

And the Apple AppStore:
https://itunes.apple.com/us/app/vmware-workspace-one/id1031603080?mt=8

This is what my workspace looks like in Google Chrome. I’ve got the Favorites showing, which are the 6 primary apps I use at VMware. Our catalog is full of many dozens of apps, it’s nice to have a quick Favorites list.

This is the Workspace One app on my iPhone. It’s an almost identical look and feel, and the favorites I set while in Chrome are the same favorites on my iPhone.

At VMware, we use two factor authentication to access Workspace One. However, I only had to enter my credentials with RSA key once. After that, I can go back into the app with my Touch ID stored on the iPhone.

Using snapshots to back up SQL under heavy IOPS loads

Posted on December 11, 2015 by pkremer

I find this problem coming up frequently in the field – you’ve virtualized your SQL server and you’re trying to back it up with your snapshot-based software. In many cases, the SQL server is under such a heavy load that it’s impossible to commit the snapshot after the backup is taken. There’s just too much IO demand. You end up having to take a SQL outage to stop IO long enough to get the snapshot committed.

Here’s one strategy for setting up your virtual SQL servers to avoid this problem altogether. It uses a disk mode called independent persistent. An independent persistent disk is excluded from snapshots – all data written to an independent persistent disk is immediately committed, even if a snapshot is active on the VM. By placing SQL datafile and logfile drives in independent persistent mode, they will never be snapshot, eliminating the problem of having to commit a post-backup snapshot.

Here’s a disk layout that I’ve used for SQL servers. These drives are set to standard mode, so a snapshot picks them up.

C:\ – 40GB, SCSI 0:0
D:\ – 30GB, SQL Binaries SCSI 0:1
L:\ – Logs, 1GB SCSI 1:0
O:\ – Datafiles, 1GB SCSI 2:0
Y:\ – Backups, 1GB SCSI 3:0
Y:\SQLBAK01 – SCSI3:1, 2TB+ mounted filesystem under Y:\

Your backup drive is limited to 2TB -512B if you’re using vSphere 5.1 or earlier, but can go up to 62TB in later versions of vSphere.

L:\Logs01 – SCSI 1:1, independent persistent, variable size, mounted filesystem under L:
O:\SQLData01 – SCSI 2:1, independent persistent, variable size, mounted filesystem under O:\

Part of why we used mountpoints was for consistency – no matter what, L: was always logs, O: was always SQL data, and Y: was always backups. There were no questions as to whether a particular SQL server had a certain number of drives for a specific purpose – the entire structure was under a single, consistent drive letter.

Depending on the workload, O:\SQLData01 might have only 1 heavily used database on a single LUN, or it might have a bunch of small databases. When we needed another one, we’d attach another mountpoint O:\SQLData02 on SCSI 2:2, L:\Logs02 on SCSI 1:2, Y:\SQLBAK02 on SCSI 3:2. Nightly backup jobs wrote SQL backups out to Y:\. Since the Y drives are all in standard mode, backup jobs picked up the dumps in the normal snapshotting process.

If you had a complete loss of the entire SQL VM, you could restore from backup and you’d still have the L:, O:, and Y: drives with their mountpoints (although they might not have any disk attached to them), and you’d have to restore the database from the SQL dumps on Y:\. Depending on what the nature of VM loss was, you may have to spend some time manually fixing the mounts.

It took a little bit of work to maintain, but our backups worked every time. Part of setting up a new database was that the DBAs wrote a restore script and stored it in the root of Y: which got backed up as part of the snapshot. Once the VM came back from Veeam restore, the DBAs would bring up SQL, hit the restore scripts, and we were off and running. You also need to coordinate your DBA’s backup schedule carefully with your backup software schedule – what you don’t want is to have backups being written to the Y: drive at the same time you’ve got an active snapshot in place – you could easily fill up the datastore if your backups are large enough. Some backup software allows you to execute pre-job scripting, it’s a fairly simple task to add some code in there to check if an active SQL backup was running. If so, postpone your backup snapshot and try again later.

Patrick Kremer

Technical perspectives from a VMware Field Engineer

Category Archives: VMware