Monitoring VMware Cloud on AWS maintenance with Notification Gateway webhooks

Part of the SDDC upgrade process is replacing SSL certificates. This can cause problems with other services that rely on the SDDC’s vCenter as the new certificate has not been accepted. The result can be an outage until the new certificate is accepted.

For on-prem SSL certificate replacements, this process is relatively easy. The vSphere admin replaces the certs and then immediately notifies the other teams to accept it.

For VMware Cloud on AWS, the customer does not control the timing of the upgrades. How can the customer know when maintenance phases are complete? You can certainly watch your inbox for emails, but what if you want deeper integration? For this, we look at webhooks in the VMC Notification Gateway.

William Lam has written an excellent blog post on using the Notification Gateway API. I used all of his work to configure my SDDC’s NGW for this post.

We make heavy use of Slack at VMware, so I decided I wanted to have SDDC events appear in a Slack channel. I first create a new Slack channel.

I go to https://api.slack.com/ and create a new Slack app

I want my app to accept an incoming webhook

Turn on incoming webhooks.

Because I do not have administrative rights, I have to wait for admin approval for my new webhook

Slackbot notifies me when my webhook request has been approved

Now I add the app to Slack

I don’t need an org-wide installation, just my workspace is fine.

I pick the channel I want to post to.

The integration is complete!

I grab the webhook URL from the incoming webhooks – I will need this to configure the notification gateway.

Time to use the PowerCLI modules referenced in William’s post.

Install-Module VMware.VMC.notification
Install-Module VMware.VimAutomation.VMC
Import-Module VMware.VMC.notification
Import-Module VMware.VimAutomation.VMC

You need a VMC on AWS API token in order to complete this step.

$RefreshToken='[your token]'
Connect-VMC -RefreshToken $RefreshToken

Server                         User
------                         ----
vmc.vmware.com                 pkremer

Connect-VmcNotification -OrgName "VMC-SET-AMER-DEMO" -RefreshToken $RefreshToken

headers                                Server
-------                                ------
{Content-Type, csp-auth-token, Accept} https://vmc.vmware.com/vmc/ng/api/orgs/[redacted]

Let’s see what notifications are possible in the documentation. I find 2 that I am particularly interested in:

• VRTvCenterUpgradeCompleted – vCenter upgrade completed successfully.
• SddcSSLCertificationReplacement – A vCenter and NSX SSL certificate update window has been scheduled for your SDDC.

I also find some interesting activity events like SDDC-PROVISION and SDDC-DESTROY that I want to include. I create my notification parameters and then create the webhook.

$vmcSlackNotificationParams = @{
    ClientId = "vmc-sddc-slack-notification";
    WebhookURL = "https://hooks.slack.com/services/[redacted]";
    NotificationEvents = @("VRTvCenterUpgradeCompleted","SddcSSLCertificationReplacement","SDDC-PROVISION","SDDC-DELETE");
    }
New-VmcNotificationWebhook @vmcSlackNotificationParams

user_id              : bf091344-a880-34c5-ad85-b66dbf8f0e1c
user_name            : pkremer@vmware.com
created              : 2/4/2021 11:36:06 PM
version              : 1
id                   : [redacted]
updated_by_user_id   : [redacted]
updated_by_user_name : pkremer@vmware.com
updated              : 2/4/2021 11:36:06 PM
client_id            : vmc-sddc-slack-notification
org_id               : [redacted]
status               : ACTIVE
web_hook_info        : @{callback_uri=https://hooks.slack.com/services/[redacted];
                       subscribe_filter=; subscribed_events=System.Object[]; sign_key=[redacted]}

Success! Now, any time any of the specified events fire, I will see a Slack notification. Let’s test it. I deploy a test SDDC.

The event shows up in Slack.

I delete the test SDDC.

It also shows up in Slack.

We have now successfully integrated the VMC NGW with Slack!

UPDATE Feb 11, 2021 –

VMware.VMC.Notification has a Test-VMCNotificationWebhook command that sends a test event, letting you test your webhook without waiting for an event to actually fire. After the PowerCLI module was written, the API changed and broke the Test command. After getting an explanation from engineering, I was able to push a fix for the Test command. To use the Test command, first make sure you’ve updated to version 1.0.2 of VMware.VMC.Notification.

PS C:\Users\pkremer\Documents\git\sddc-webhook> get-module vmware.vmc.notification
 ModuleType Version    PreRelease Name                                ExportedCommands
 ---------- -------    ---------- ----                                ----------------
 Script     1.0.2                 vmware.vmc.notification             {Connect-VmcNotification, Get-VmcNotificationEvent, Get-VmcNotificationWeb…

Then you can send any test event like this:

PS C:\Users\pkremer\Documents\git\sddc-webhook> Test-VmcNotificationWebhook -id cdcf9649-[redacted] -EventId CLUSTER-CREATE     
 Successfully sent test webhook and recieved acknowledgement

Now the test notification shows up in Slack.

In Part II, I show you how to do the same thing with Office365 webhooks.