vCenter Event Broker Appliance – Part X – Building the Appliance OVA from source code

Posted on by
Reply

In Part IX of this series, we deployed the datastore usage sample function into our appliance. In this post, we discuss how to build the appliance OVA from source code. This blog post would not have been possible without William Lam spending his valuable time teaching the VEBA team about the build process.

The VEBA appliance is built periodically, when enough features have been added to warrant a release. However, changes are committed frequently to the development branch. Sometimes, you want to deploy a new feature but don’t want to wait for a new release. Or, in my case, I wanted to make a change to the options shown while deploying the OVA. This requires rebuilding the appliance. Everything you need to package the appliance for use is available in the event broker repository.

Step 1 – Build machine

First, you need a build machine. You cannot use Windows for this task. I run a Windows laptop, so I ended up building a Ubuntu build server in my homelab. Here are all of the packages I added in order to be able to build an OVA.

#Git
apt install git-all

#Unzip
apt install unzip

# OpenFaas command line utility faas-cli
curl -sSL https://cli.openfaas.com | sudo sh

#Packer utility from Hashicorp
wget https://releases.hashicorp.com/packer/1.5.6/packer_1.5.6_linux_amd64.zip
unzip packer_1.5.6_linux_amd64.zip -d /usr/local/bin

#Download ovftool from My VMware
#https://my.vmware.com/group/vmware/get-download?downloadGroup=OVFTOOL440
# I used WinSCP to copy the file to the Linux VM. I’m beginning to see why lots of # #developers like Macs

chmod 744 VMware-ovftool-4.4.0-15722219-lin.x86_64.bundle
./VMware-ovftool-4.4.0-15722219-lin.x86_64.bundle

#Install PowerShell 7.0
# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7
# The documentation tells you to do the following commands:
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
apt-get update
add-apt-repository universe
apt-get install -y powershell

#But the commands result in this error:
#Reading package lists… Done
#Building dependency tree
#Reading state information… Done
#Some packages could not be installed. This may mean that you have
#requested an impossible situation or if you are using the unstable
#distribution that some required packages have not yet been created
#or been moved out of Incoming.
#The following information may help to resolve the situation:

#The following packages have unmet dependencies:
#powershell : Depends: libssl1.0.0 but it is not installable
#Depends: libicu60 but it is not installable
#E: Unable to correct problems, you have held broken packages.

# This problem is fixed in RC https://github.com/PowerShell/PowerShell/releases/tag/v7.0.0-rc.1
# Download the release candidate code, extract it, and add a path to $PATH

# Install PowerCLI
install-module vmware.powercli

# Configure Git
root@build01:~# git config –global user.name “Patrick Kremer”
root@build01:~# git config –global user.email “pkremer@vmware.com”

# Configure git password cache for github
# https://help.github.com/en/github/using-git/caching-your-github-password-in-git

git config –global credential.helper cache
git config –global credential.helper ‘cache –timeout=86400’

# Clone my fork, add the upstream repo
git clone https://github.com/kremerpatrick/vcenter-event-broker-appliance.git
git remote add upstream https://github.com/vmware-samples/vcenter-event-broker-appliance.git

Step 2 – Clone the repo

You need to clone the development repo. If you are not familiar with git, take a look at Part V. If you’ve been following the examples and have the code, but just need to make sure you have the latest copy of the development branch, look at Part VI.

Step 3 – Build an ESXi host

If you’ve still got a 6.7 host around, you’re in luck. But as of the date of this blog post, you cannot build the VEBA appliance against a 7.0 host. The build process relies on VNC, which was removed from ESXi in 7.0.  My lab is on 7.0, so I had to build a nested ESXi.  I know for a fact that I’ve gotten the native MAC learning feature working in my lab, but I could not get it to work nesting 6.7 inside 7.0. I’m not sure why it wouldn’t work, so I ended up reverting to the tried-and-true promiscuous mode for the outer portgroup, and an untagged portgroup for the inner one. In this screenshot, esx03 is a VM running on physical hosts in cluster CL1, but I added it as a host outside the cluster in vCenter.

Step 4 – Edit JSON files

In the root of the vcenter-event-broker-appliance directory are photon.json and photon-builder.json. You must edit them to match your environment

root@build01:/var/git/vcenter-event-broker-appliance# cat photon-builder.json
{
  "builder_host": "192.168.30.14",
  "builder_host_username": "root",
  "builder_host_password": "VMware1!",
  "builder_host_datastore": "sm-vsanDatastore",
  "builder_host_portgroup": "VM Network"
}

builder_host is the nested ESXi 6.7 host. Make sure to enter the correct datastore, it’s easy to make a typo so consider copy-pasting the value. You need to keep the portgroup “VM Network”, so you will need to put a portgroup named “VM Network” on your nested ESXi host. Otherwise your own portgroup name will be built into the appliance. Also, the automation scripts in the /test folder will break because they’re looking for “VM Network” to be in the appliance.

Now look at the top of photon.json. Adjust the file to point to the vCenter managing your nested ESXi host. 

root@build01:/var/git/vcenter-event-broker-appliance# more photon.json
{
  "variables": {
    "veba_ovf_template": "photon.xml.template",
    "ovftool_deploy_vcenter": "192.168.30.200",
    "ovftool_deploy_vcenter_username": "administrator@vsphere.local",
    "ovftool_deploy_vcenter_password": "VMware1!"
  },

You can also look at photon-version.json – unless you work for VMware, you probably won’t be releasing VEBA, so you won’t really need to adjust this file. But it’s important to note that the output of the build process will look like an official build. If you build the OVA straight out of the development branch, you’ll end up with a appliance name like vCenter_Event_Broker_Appliance_v0.4.0.ova – this of course does not match the actual release v0.4.0, so don’t get yourself confused with binaries downloaded on your machine.

root@build01:/var/git/vcenter-event-broker-appliance# cat photon-version.json
{
  "version": "0.4.0",
  "description": "Photon Build for vCenter Event Broker Appliance",
  "vm_name": "vCenter_Event_Broker_Appliance",
  "iso_checksum": "f6619bcff94cef63d0d6d7ead7dd3878816ebfa6a1ef5717175bb0d08d4ccc719e4ec7daa7db3c5dc07ea3547fc24412b4dc6827a4ac332ada9d5bfc842c4229",
  "iso_checksum_type": "sha512",
  "iso_url": "http://dl.bintray.com/vmware/photon/3.0/Rev2/iso/Update1/photon-3.0-a0f216d.iso",
  "numvcpus": "2",
  "ramsize": "8192",
  "guest_username": "root",
  "guest_password": "##FILL-IN-SECURE-PASSWORD##"
}

We start the build with build.sh

build.sh release

It will take some time the first time to download photon your system. It caches the iso so future builds aren’t as slow.

The script will also sit for some time on the ‘Starting HTTP server’ message. If you look at your nested host you will eventually see a disk being built

This is what is running behind the HTTP server – the build script has created a PXE boot host

The VM will eventually start booting

If you want to see the PXE boot happening live, you will quickly need to establish a console session on the VM as it boots. You will see packer typing PXE boot commands over VNC. The VM will PXE boot, run commands, then reboot into Photon and end up at a command prompt

The script completes the rest of the configuration via SSH.

The build will progress until completion.

The OVA file will be in the output-vmware-iso folder. You can see here that I renamed it with my initials so I don’t get confused by the official 0.4.0 ova.

root@build01:/var/git/vcenter-event-broker-appliance/output-vmware-iso# ls -al
total 1232992
drwxr-xr-x 2 root root 4096 May 15 23:49 .
drwxr-xr-x 16 root root 4096 May 18 20:43 ..
-rw——- 1 root root 1262570496 May 15 23:44 vCenter_Event_Broker_Appliance_v0.4.0-pk.ova

That’s it for building VEBA. In the next post we will look at the underlying change that made it necessary for me to build the OVA – how to change the options in the deployment GUI.

Troubleshooting vSphere 7 Installation – Home Lab Edition

Posted on by
Reply

My homelab boots off of cheap USB sticks and has done so since 2016.

I easily upgraded my vCenter (VCSA for the win), but could not get Update Manager to upgrade my hosts, it kept throwing an ‘unable to run upgrade script’ error. I tried command-line upgrade without success, then I tried dropping the ISO image on the USB stick with Rufus, booting off the stick and having the installer overwrite the boot image with the installation. I’ve done this same process many times over the past 4 years without issue. However, it kept failing this time. The installation would progress to various percentages, but never beyond 70%

I’ve been working with VMware products since the 3.0 days and I never knew this, thank you to William Lam for his troubleshooting suggestion in VMware’s slack channels. When the progress bar of the installation screen is up, you can press Alt+F1 and get a command prompt. Then you can log in with root and no password. From there you can navigate the filesystem and tail logs, including the install log and vmkernel log. I saw lots of storage errors in the vmkernel log, timeouts and retries.

I then yanked one of the SSD drives in my Synology out and popped it into the host. vSphere 7 installed without issue. Looks like I either need to buy some high-end USB drives, or maybe use this problem to justify swapping out some of the smaller drives in the Synology with larger ones. Then I can rotate the smaller ones into service as boot drives in the hosts.

vCenter Event Broker Appliance – Part VIII – Basic Troubleshooting Techniques

Posted on by
4

In Part VII of this series, we deployed a second sample function – the Host Maintenance functions written in PowerShell. In this post, we discuss some basic troubleshooting techniques. This post was updated on 2020-03-07 with updated screenshots for the VEBA 0.3 release.

The VEBA is running Kubernetes.  Going into this project, I knew nothing about Kubernetes – and still don’t know very much. But I’ve learned enough to be able to do some basic troubleshooting. In the end it’s all a series of Linux commands.

You might want to SSH to the appliance instead of using the console. To do this, log in as root on the console, then execute the command systemctl start sshd.  SSH will run, but not automatically start on the next reboot.

VEBA code is running in Kubernetes pods, and those pods are running in namespaces. kubectl lets you run commands against Kubernetes. So first we list out namespaces with kubectl get namespaces

One of the namespaces you will see is openfaas. All of your functions will be running inside their own container. We issue the command kubectl get pods -n openfaas and see all of our pods that are running OpenFaaS.

PROTIP: kubectl get pods -A gives you all the pods and their associated namespaces in one command.

kubectl logs gets us logfiles from the pods. The VMware Event Router pod is responsible for communicating with vCenter. Obviously these logs could contain interesting information if you’re trying to troubleshoot VEBA communications with vCenter.

kubectl logs vmware-event-router-5dd9c8f858-nph7k -n vmware –follow

 

Here is the output in a screengrab and also pasted.  Note that we see events topics firing (AlarmStatusChangedEvent) and successful interception and function invocation. Rather than trying to dig through API documentation, one great way to figure out how to react to a vCenter event is to start tailing the logfile with –follow – this is essentially the same as tail-f . Then perform an action on vCenter and see which topic fired. You can then build a function to react to the event.

[OpenFaaS] 2020/03/08 00:28:19 invoking function(s) on topic: AlarmStatusChangedEvent
2020/03/08 00:28:19 Invoke function: powershell-datastore-usage
2020/03/08 00:28:20 Syncing topic map

[OpenFaaS] 2020/03/08 00:28:21 processing event [1] of type *types.AlarmStatusChangedEvent from source https://******************************/sdk: &{AlarmEve nt:{Event:{DynamicData:{} Key:8689704 ChainId:8689704 CreatedTime:2020-03-08 00:28:15.875458 +0000 UTC UserName: Datacenter:0xa84c560 ComputeResource:<nil> Host:<nil> V m:<nil> Ds:0xa84c6c0 Net:<nil> Dvs:<nil> FullFormattedMessage:Alarm ‘Datastore usage on disk’ on WorkloadDatastore changed from Gray to Green ChangeTag:} Alarm:{EntityE ventArgument:{EventArgument:{DynamicData:{}} Name:Datastore usage on disk} Alarm:Alarm:alarm-7}} Source:{EntityEventArgument:{EventArgument:{DynamicData:{}} Name:Datace nters} Entity:Folder:group-d1} Entity:{EntityEventArgument:{EventArgument:{DynamicData:{}} Name:WorkloadDatastore} Entity:Datastore:datastore-60} From:gray To:green}

 

In the above log output, we can see that OpenFaaS caught a VM powering up, and invoked the function powershell-datastore-usage.  But we don’t know what happened inside that function.

PROTIP: you can also use –since=2m for the last two minutes of logs, or –tail=20 for the last 20 lines of log.

Let’s look directly into the function logs. Your functions run in the openfaas-fn namespace. We list out the pods.
kubectl get pods -n openfaas-fn

We then look at the logs with:
kubectl logs -n openfaas-fn powershell-datastore-usage-847d5c7875-286hv

Looking at the logs, we see the function firing.

 

Let’s look at a function that I broke intentionally. I update my secret with a configuration that has a bad password – you can test this behavior with any of the sample functions. We happen to be using the pytag function below.

 

Now I want to look at the logs for my pytag function. Again, the commands are

kubectl get namespace
kubectl get pods -n openfaas-fn
kubectl logs pytag-fn-f66d6cffc-x5ghk -n openfaas-fn

An astute reader might notice that my pytag container changed names between the previous screenshot and the current one. The content for this post was written in 2 sessions – when I came back for the second screenshot, I had redeployed the function numerous times. This means I got a new pod with a new name.You can always find the pod name with the get pods command.

Here are the logs when I try to power up a VM after I pushed the broken vcconfig secret:

The log clues us in to an unauthorized error. It doesn’t specifically say password error, but you know there’s something wrong with authentication – at least you have a place to start troubleshooting

2019/12/29 00:46:07 Path /
{“status”: “500”, “message”: “could not connect to vCenter 401 Client Error: Unauthorized for url: https://vc01.ad.patrickkremer.com/rest/com/vmware/cis/session”}

 

Now let’s fix it – we update the secret with the correct password

You can see in the logfile there was one more authentication failure before the new secret was picked up. I’m not sure at the time I posted this exactly what governs picking up a new secret, but it is important to know that it is not instant.

 

In Part IX, we deploy the datastore usage alarms sample function.

vCenter Event Broker Appliance – Part IX – Deploying the Datastore Usage Email sample script in VMC

Posted on by
3

In Part VIII, we discussed basic troubleshooting techniques for VEBA. In this post we will deploy a new sample script.

There is a new VEBA sample script in 0.3 that enables you to send emails based on a datastore alarm. Although the functionality is built into on-prem vCenter, you have very little control over the email content. It is not possible to configure email alarms in VMC on AWS because the customer is locked out of that configuration area in vCenter. The sample script is written in PowerShell/PowerCLI.

I wrote extensive documentation on deploying a PowerShell script into VEBA in Part VII . If you don’t already know how to clone a repository with git and how to work with OpenFaaS, check out Part VII. Here I am just going to show the config files and config changes necessary.

We find the code in examples/powercli/datastore-usage-email

NOTE: If you send a large volume of emails from VMC, you need to open a ticket with VMware support to unthrottle SMTP on the shadow VPC that contains your SDDC. AWS by default throttles emails sent out of all their VPCs. You can simply pop open the chat window in the VMC console and request the change. VMware will file a ticket with AWS support and you will be unthrottled.

The default configuration is set to work with GMail authenticated SMTP. However, many customers run SMTP relays either on-prem or up in VMC for various email related needs. You can change the SMTP_PORT to 25 and leave SMTP_USERNAME and SMTP_PASSWORD blank to use standard SMTP.

Here’s how I changed it to get it work using VMware’s SMTP relay to deliver to my corp email address.

 

My modified stack.yml looks like this

Create the secret and push the function with faas-cli

faas-cli secret create vc-hostmaint-config –from-file=vc-hostmaint-config.json –tls-no-verify

faas-cli up –tls-no-verify

Now we need to cause a storage alarm. We need to find the default Datastore Usage on Disk alarm and edit it

We change the warning level very low, here I change it to 7% to force an alarm. Then I hit next to the end of the config wizard.

The datastore shows a warning. Now we perform the same Edit operation and set the warning percentage back to 70%. The datastore warning should clear.

If everything worked, you should have 2 emails in your inbox – a warning email, and back to normal email.

If you don’t see the emails, check your spam folder – you may have to whitelist the emails depending on your spam settings.

If you have issues, troubleshoot the VMware event router logs and the email function logs as show in the troubleshooting section Part VIII .

vCenter Event Broker Appliance – Part VII – Deploy the Sample Host Maintenance Function

Posted on by
3

In Part VI of this series, we showed how to sync a fork of our VEBA repository to the upstream repository maintained by VMware.  Back in Part IV, we deployed our first sample function. In this post, we will deploy another sample function – the host maintenance function. This post was updated on 2020-03-07 to include screenshots for the VEBA 0.3 release.

Our first sample function was written in Python. As of the date of this post, the other available samples are all in PowerShell. We will be working with the hostmaint-alarms function. This function will disable alarm actions when you pop a host into maintenance mode. No more alerts for a host that you’re doing maintenance on!

We had a problem in the 0.2 release with secret names colliding. Here is the stack.yml file for our python tagging function

Here is the sample file we used to generate the secrets named ‘vcconfig’.

Here is the stack.yml for our host maintenance alarms function in the 0.2 release

We no longer have this issue in the 0.3 release as we named our secret vc-hostmaint-config.

Here is the vcconfig.json file we use to configure the secret named ‘vcconfig’ for our PowerShell function.

A problem happens when you use secrets of the same name for scripts that aren’t expecting the same secrets format. In 0.2, we had a secret named vcconfig used for both functions, but the secret files have a completely different configuration. Neither script can read the other’s secret because they weren’t programmed to do so. The TOML secret file is a configuration file format popular with Python. The PowerShell secret file is simple JSON.  This means that we will need to change the secrets file to a different name, one for the Python script and one for PowerShell. Note that it doesn’t have to be this way – there’s nothing stopping a script writer from using TOML format for PowerShell and JSON for Python – all that matters is how how the script is written. You could write your scripts to use a single secret format and they could all share a single secret.

We now need to change the sample script to point to a different secrets file. In order to do that, we need create a new secret using our vcconfig.json file.

After editing the file to point to our environment, we push it into the VEBA appliance. I name it ‘vcconfig-hostmaint’ but you can name it whatever you want. To match the current 0.3 script, you should name it ‘vc-hostmaint-config’. If you match what’s in the script, you don’t have to rebuild any container images – the default container image will work. But there are many reasons why you would need to rebuild the container image. Any time you want to improve the existing functions, or write your own, you will need to build your own container image.  This post will continue on showing how to finish deploying by rebuilding the container image.

To create the secret file, remember you need to log in with faas-cli first, for a refresher look at Part IV of this series.

Now that we have our secrets file, we need to change our code to use it.

First we edit the first line of script.ps1 in the handler folder. We need to change the secret name to whatever we named it in the cli – here, I change it to: vcconfig-hostmaint

Looking again at the stack.yml file, we have additional problems. We built a new secret, so we can change the secrets: section to point to vcconfig-hostmaint. Our gateway needs to point to our VEBA appliance. Then we need to worry about our image. Because we changed PowerShell code, we have to rebuild the container image that runs the code. Otherwise, the container that launches is the default container that ships with the VEBA appliance.

 

We sign up for a Docker account

 

Now we download and install Docker Desktop. The installation is simple, you can find the official installation documentation here.

 

After installation there’s a little whale icon in my system tray

I right-click and log in with the account I just created

 

 

Now when I right-click the whale, it shows that I’m signed in.

Now we edit the stack.yml file.  We make sure our gateway is pointing to our VEBA. We change the secrets to point to our new secret. And we change the image name – the account name gets changed to the Docker account that we just created.

Now we need to build, push, and deploy our new container image. The faas-cli function creation documentation shows us the commands we need to use.

First, we need to log in to Docker. Since I had no experience with Docker, it took me forever to figure out this seemingly simple task. I tried many different ways, but all failed with an Unauthorized error.

The simple answer is to make sure you’re logged into the Docker desktop

Then you issue a docker login command with no arguments. Just docker login.

 

We now issue the faas-cli build -f stack.yml command.

Multiple screens of information scroll past, but we end with a successful build.

Now we push the containers into the Docker registry with faas-cli push -f stack.yml

Now we deploy the container to our VEBA with faas-cli deploy -f stack.yml –tls-no-verify

PROTIP: Once you understand what the 3 commands do – build, push, deploy – you can use a nifty OpenFaaS shortcut command: faas-cli up –tls-no-verify
This command runs build, push, and deploy in sequence, automatically.

Now we log into vCenter and look at the Alarm actions on a host. They are currently enabled.

 

After entering maintenance mode, the alarm actions have been disabled by our function

 

Now we exit maintenance mode and our function enables alarm actions. Success!

 

Finally, we want to verify that we did not break our other Python function, the one that tags a VM when it powers on. We check our test VM and see it has no tags.

After power on, a tag has been applied. Now we have both functions working!

We have now successfully deployed our PowerShell host maintenance function! In Part VIII, we will look at some troubleshooting techniques.

 

vCenter Event Broker Appliance – Part IV – Deploying the First Sample Function

Posted on by
3

In Part III of this series, we created our vCenter tag and cloned the code repository from Github. In Part IV, we will deploy our function and watch it fire in response to a vCenter event. This post was updated 2020-03-07 with new screenshots for the VEBA 0.3 release.

The python sample function is sitting in the root path /examples/python/tagging. The sample function will cause a vCenter tag to be applied to a VM when the VM powers on.
First we want to modify vcconfig.toml

Here we need to change the server to our vCenter server, put in the correct username/password, and change the tag URN to match the URN we retrieved in Part III

To quickly review, one way to get a Tag’s URN is via the Get-Tag PowerCLI cmdlet

 

vcconfig.toml now looks like this

We continue with the OpenFaaS instructions in the getting started documentation.

When the correct password is used, we should get this:

If you get an error, a number of things could be the cause:

  • The VM might not be fully online; it can take a while for services to come up after initial boot, particularly in a homelab
  • You might be running into a hostname problem – you must set the OPENFAAS_URL to the exact name you used when you deployed the OVF. If you deployed as the FQDN, you have to use the FQDN here. If you deployed with the short name, you must use the short name here.
  • You are typing the incorrect credentials

Next, we want to create a secret named ‘vcconfig’ in faas-cli so we’re not keeping credentials in plaintext files.

Next, we edit stack.yml.  The gateway has to change to your VEBA VM – make sure it matches the name you deployed. We also need to look at the topic. The documentation warns us that if we’re trying to monitor a VM powered on event in a DRS cluster, the topic is actually DrsVmPoweredOnEvent. We change the topic: entry to match.

My changed stack.yml looks like this:

We now issue a faas-cli template pull command (only required for the first deployment), and then deploy the function with:
faas-cli deploy -f stack.yml –tls-no-verify

 

Now the moment of truth, time to test our function! The expected behavior is that when powered on, a VM gets the alert-power-operations tag assigned to it. First we check my test VM to see if it has any tags – it has none.

After we power on the VM, has the tag been assigned?

Success! Note that you may have to hit the refresh button to get the tag to appear in the GUI.

That’s it for your first sample function!

Since this product is open source, you can make changes to it. Even if you’re not a developer, a great way to contribute is to keep documentation up-to-date. In part V, we will explore the basics of using git to commit code back to the repository. Even if you’re not a developer, you can learn the basics of git.

vCenter Event Broker Appliance – Part Ia – AWS EventBridge Deployment

Posted on by
2

Deploying VEBA for AWS EventBridge

Part I of this series covers deploying VEBA. As of the 0.3 VEBA release, we support 2 event processors – built-in OpenFaaS, and external AWS EventBridge. This post covers how to configure AWS EventBridge.

William Lam has a good EventBridge blog showing his setup, and the EventBridge documentation has detailed instructions. I won’t repeat all that content, but I will show you my setup.

First I created a new EventBridge event bus.

Then I created an event forwarding rule

It’s important to make sure you have a filter – I suggest using the following event pattern for testing. You can always change it later to fit whatever events you’re trying to respond to. But if you don’t filter, EventBridge will react to every single event that comes out of your vCenter. If you’re running a Prod vCenter, that’s going to be a LOT of traffic.

I picked these two events because they’re relatively low traffic – most customers aren’t popping their hosts in and out of maintenance mode thousands of times a day.

When you deploy VEBA with the EventBridge processor, you have to enter the following settings in the EventBridge. Access key and secret are straightforward, the same IAM credentials that you’d create for programmatic access to any AWS service.  Event Bus Name needs to match the event bus you created. The region is easily found by clicking on the region name in the AWS console. The ARN is just a copy-paste from the Rule page in the AWS console – just make sure you copy the Rule ARN and not the Event Bus ARN. In my case, this:  arn:aws:events:us-east-1:##########:rule/SET-VMC-EventBus/VEBA-event-forwarding

When you finish deploying VEBA and it boots, you can look at the logfiles to see what’s happening with the Event Broker. Or obviously you can just put a host in maintenance mode and look in CloudWatch to see if a log appears.

To look on the VEBA – Enable SSH on the VEBA by following the instructions in Part VIII, or just do it on the console of the VM.

Type kubectl get pods -A and look for the event router pod

Now type kubectl logs vmware-event-router-5dd9c8f858-7b9pk -n vmware –follow
You are doing the Kubernetes equivalent of tailing a logfile.

Near the top, you see the Event Router connecting to AWS EventBridge and pulling down the configured filters. You then see the EventRouter begin receiving vCenter events

Let’s put a host into maintenance mode and see what happens.

The event was intercepted and sent to EventBridge

Now we take the host out of maintenance mode and we also see the event logged

Let’s check CloudWatch and see if our events were logged – they were!

You can return to Part I of this series for more information on deployment, or move on to Part II to begin working on the prerequisites for deploying code samples to VEBA.

 

vCenter Event Broker Appliance – Part I – Deployment

Posted on by
3

Introduction

I became aware of the vCenter Event Broker Appliance Fling (VEBA) in December, 2019. The VEBA fling is open source code released by VMware which allows customers to easily create event-driven automation based on vCenter Server Events. You can think of it as a way to run scripts based on alarm events – but you’re not limited to only the alarm events exposed in the vCenter GUI. Instead, you have the ability to respond to ANY event in vCenter.

Did you know that an event fires when a user account is created? Or when an alarm is created or reconfigured? How about when a distributed virtual switch gets upgraded or when DRS migrates a VM?  There are more than 1,000 events that fire in vCenter;  you can trap the events and execute code in response using VEBA. Want to send an email and directly notify PagerDuty via API call for an event? It’s possible with VEBA. VEBA frees you from the constraints of the alarms GUI, both in terms of events that you can monitor as well as actions you can take.

VEBA is a client of the vSphere API, just like PowerCLI. VEBA connects to vCenter to listen for events. There is no configuration stored in the vCenter itself, and you can (and should!) use a read-only account to connect VEBA to your vCenter. It’s possible for more than one VEBA to listen to the same single vCenter the same way multiple users can interact via PowerCLI with the same vCenter.

For more details, make sure to check out the VMworld session replay “If This Then That” for vSphere- The Power of Event-Driven Automation (CODE1379UR)

Installing the VEBA – Updated on March 3, 2020 to show the 0.3 release

If you notice screenshots alternating between an appliance named veba01 and veba02, it’s because I alternate between the two as I work with multiple VEBA versions.

The instructions can be found on the VEBA Getting Started page.

First we download the OVF appliance from https://flings.vmware.com/vcenter-event-broker-appliance and deploy it to our cluster


VMC on AWS tip: As with all workload VMs, you can only deploy on the WorkloadDatastore

I learned from my first failed 0.1 VEBA deployment to read the instructions – it says the netmask should be in CIDR format, but I put in 255.255.255.0. In 0.3, the subnet mask/network prefix is a dropdown, eliminating this problem.

Pay attention to DNS and NTP settings – they are space separated in the OVF, the instructions have said ‘space separated’ since 0.2 but it’s important to enter them correctly here. NTP settings were new in 0.2

Proxy support was new in 0.2. I don’t have a proxy server in the lab so I am not configuring it here.

These settings are all straightforward. Definitely disable TLS verification if you’re running in a homelab. One gotcha are passwords – I used blank passwords for the purposes of this screenshot, but you can move ahead with blank passwords.  This resulted in a failure after the appliance booted because it couldn’t connect to vCenter. On the plus side it lead to my capture of how to troubleshoot failures, which is shown further down in this post.

PROTIP: If you’re brand new to this process, it’s easiest to get this working with an administrator account. It is best practice to use a read-only vCenter account. You can easily change this credential after everything is working by following the procedure at the bottom of this post – edit /root/event-router/config.json (file moved in version 0.4) /root/config/event-router-config.json and restart the pod as shown in the directions.

New in 0.3, you can decide whether you want to use AWS EventBridge or OpenFaaS as your event processor.  EventBridge setup is covered in Part Ia of this series. For this setup, we choose OpenFaaS.

Since we’re not configuring EventBridge, we leave this section blank.

The POD CIDR network is an important setting new in 0.2. This network is how the containers inside the appliance communicate with each other. If the IP that you assign to the appliance is in the POD CIDR subnet, the containers will not come up and you will be unable to use the appliance. You must change either the POD CIDR address or the appliance’s IP address so they do not overlap.

This is what the VEBA looks like during first boot

If you end up with this, [IP] in brackets instead of your hostname, something has failed. Did you use a subnet mask instead of CIDR? Did you put comma-separated DNS instead of space? Did you put in the incorrect gateway? If you enabled debugging at deploy time, you can look at /var/log/boostrap-debug.log for detailed debug logs to help you pinpoint the error. If not, see what you can find in /var/log/boostrap.log

There is an entire troubleshooting section for this series. When the pod wasn’t working despite successful deployment, I needed to determine why.

Enabling SSH on the appliance is covered in the troubleshooting section. You could also use the VM console without enabling SSH.

kubectl get pods -A lists out the available pods

The pods that is crashing is the event router pod.
kubectl logs vmware-event-router-5dd9c8f858-q57h6 -n vmware will show us the logs for the event router pod. We can see that the vCenter credentials are incorrect.

We need to edit the JSON config file for Event Router.

vi /root/config/event-router-config.json

Oops. The password is blank

Fix the password

Delete and recreate the pod secret

kubectl -n vmware delete secret event-router-config
kubectl -n vmware create secret generic event-router-config –from-file=event-router-config.json

Get the current pod name with
kubectl get pods -A

Delete the event router pod

kubectl -n vmware delete pod vmware-event-router-5dd9c8f858-zsfq5

The event router pod will automatically recreate itself.  Get the new name with
kubectl get pods -A

Now check out the logs with
kubectl logs vmware-event-router-5dd9c8f858-pcq2v -n vmware

We see the event router successfully connecting and beginning to receive events from vCenter

Now we have a VEBA appliance and need to configure it with functions to respond to events. Note that it may take some time for the appliance to respond to http requests, particularly in small homelab. It took about 4 minutes for it to fully come up after succesful boot in my heavily overloaded homelab. One additional gotcha: the name you input at OVF deployment time is the name encoded in the SSL certificate, so if you input “veba01” as your hostname, you cannnot then reach it with https://veba01.fqdn.com.

In part II of this series, I will demonstrate how I configured my laptop with the prereqs to get the first sample function working in VEBA.

vCenter Event Broker Appliance – Part VI – Syncing Your Fork

Posted on by
2

In Part V of this series, we looked at how to contribute to the VEBA project. In this short post, we will show how to sync your fork back to the upstream repository – the project that you originally forked from.

If you work with an open source fork for any length of time, you will inevitably come to a situation where your fork is out of sync with the upstream repo. Other people have contributed to the upstream repo, and your code is now out of sync. You could issue a pull request to pull the upstream into your fork, but then you are still out of sync – your fork will be 1 commit ahead of the upstream.

The following post shows how I used Github’s Syncing a Fork post to sync my repo with the upstream VEBA project.

First, my fork in Github shows that it’s 5 commits behind the upstream repository vmware-samples:master.

In order to fix this, I need the upstream’s clone URL

We add the URL as an upstream branch

Fetch the upstream code and then check out our local master branch

 

Merge the upstream changes into my local branch

We then push the local copy into the Github repo

Our fork is now even with the upstream repo.

In Part VII, we will look at deploying additional VEBA sample functions.