I became aware of the VMware Event Broker Appliance Fling (VEBA) Fling in December, 2019. The VEBA Fling is open source code released by VMware which allows customers to easily create event-driven automation based on vCenter Server Events. You can think of it as a way to run scripts based on alarm events – but you’re not limited to only the alarm events exposed in the vCenter GUI. Instead, you have the ability to respond to ANY event in vCenter.
Did you know that an event fires when a user account is created? Or when an alarm is created or reconfigured? How about when a distributed virtual switch gets upgraded or when DRS migrates a VM? There are more than 1,000 events that fire in vCenter; you can trap the events and execute code in response using VEBA. Want to send an email and directly notify PagerDuty via API call for an event? It’s possible with VEBA. VEBA frees you from the constraints of the alarms GUI, both in terms of events that you can monitor as well as actions you can take.
VEBA is a client of the vSphere API, just like PowerCLI. VEBA connects to vCenter to listen for events. There is no configuration stored in the vCenter itself, and you can (and should!) use a read-only account to connect VEBA to your vCenter. It’s possible for more than one VEBA to listen to the same single vCenter the same way multiple users can interact via PowerCLI with the same vCenter.
For more details, make sure to check out the VMworld session replay “If This Then That” for vSphere- The Power of Event-Driven Automation (CODE1379UR)
Installing VEBA with Knative
The product documentation can be found on the VEBA website.
First we download the OVF appliance from https://flings.vmware.com/vcenter-event-broker-appliance and deploy it to our cluster
I am deploying a prerelease version of v0.6, so the version number still shows 0.5
I don’t have a proxy server so I leave all of the settings blank
This is the root password to log into the appliance. You generally need it only for troubleshooting. If you want the SSH daemon to auto-start, check the box. It is more secure to leave it unchecked.
These settings are all straightforward. Definitely disable TLS verification if you’re running in a homelab. One gotcha are passwords – you can move forward with blank passwords – if you use a blank vCenter password, the appliance won’t be able to connect to vCenter to receive events. It is best practice to use a read-only account – we don’t need any additional permissions to read the vCenter logs
These settings are for the new 0.6 feature – a vCenter user interface plugin! Using the plugin is not a requirement, but it’s a great way to easily deploy functions. The account you specify here needs permissions to register a plugin with vCenter:
- Register and Update Extension – Install Plugin
- Manage Plugins – Update Plugin
Definitely disable TLS verification in your homelab – and maybe you like to run Prod a little dangerously too?
Here we pick Knative, and then skip to the Knative section
Knative comes installed in the VEBA appliance – you do not need to specify any of these settings unless you want to use your own Knative instance.
Finally, we have networking settings, you won’t need to change any of these unless you have a conflict with the ranges.
The appliance boots up and configures itself.
Finished! This is what the console should look like if the deployment succeeds.
If you want to look around on the appliance, you can either log directly onto the console as root or SSH if you’ve enabled the SSH daemon. It is best practice to leave SSH disabled, the recommended method for administering Kubernetes on this appliance is via remote kubectl. We cover installation and setup of kubectl in Part II.
View all the running pods with
kubectl get pods -A
root@veba04 [ ~ ]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE contour-external contour-5869594b-4mtnh 1/1 Running 0 7m21s contour-external contour-5869594b-tk8tg 1/1 Running 0 7m21s contour-external contour-certgen-v1.10.0-pl8qz 0/1 Completed 0 7m21s contour-external envoy-s4ngd 2/2 Running 0 7m21s contour-internal contour-5d47766fd8-5ggc8 1/1 Running 0 7m20s contour-internal contour-5d47766fd8-95wws 1/1 Running 0 7m20s contour-internal contour-certgen-v1.10.0-dv6hn 0/1 Completed 0 7m21s contour-internal envoy-mxm5p 2/2 Running 0 7m20s knative-eventing eventing-controller-658f454d9d-x7ps2 1/1 Running 0 6m42s knative-eventing eventing-webhook-69fdcdf8d4-9tsj4 1/1 Running 0 6m42s knative-eventing rabbitmq-broker-controller-88fc96b44-gww84 1/1 Running 0 6m31s knative-serving activator-85cd6f6f9-4452r 1/1 Running 0 7m46s knative-serving autoscaler-7959969587-lvd9d 1/1 Running 0 7m45s knative-serving contour-ingress-controller-6d5777577c-sscjm 1/1 Running 0 7m19s knative-serving controller-577558f799-h8rcb 1/1 Running 0 7m45s knative-serving webhook-78f446786-km7qw 1/1 Running 0 7m45s kube-system antrea-agent-t7hgq 2/2 Running 0 7m46s kube-system antrea-controller-849fff8c5d-flp9n 1/1 Running 0 7m46s kube-system coredns-74ff55c5b-4w6x8 1/1 Running 0 7m46s kube-system coredns-74ff55c5b-bgm9b 1/1 Running 0 7m46s kube-system etcd-veba04 1/1 Running 0 7m51s kube-system kube-apiserver-veba04 1/1 Running 0 7m51s kube-system kube-controller-manager-veba04 1/1 Running 0 7m51s kube-system kube-proxy-r5zmd 1/1 Running 0 7m46s kube-system kube-scheduler-veba04 1/1 Running 0 7m51s local-path-storage local-path-provisioner-5696dbb894-qc9kk 1/1 Running 0 7m45s rabbitmq-system rabbitmq-cluster-operator-7bbbb8d559-frthd 1/1 Running 0 6m32s vmware-functions default-broker-ingress-5c98bf68bc-2fhhd 1/1 Running 0 5m7s vmware-system tinywww-dd88dc7db-rjjf2 1/1 Running 0 6m23s vmware-system veba-rabbit-server-0 1/1 Running 0 6m24s vmware-system veba-ui-677b77dfcf-tss5t 1/1 Running 0 6m19s vmware-system vmware-event-router-b85dd9d87-g9cm5 1/1 Running 4 6m26s
They should all be running except for the ones that say certgen. We are most interested in the one named vmware-event-router. The extra text – ‘-b85dd9d87-g9cm5’ in this case – will be unique for every deployment.
The event router process is what connects to vCenter Server and reads all of the events. Let’s check the logs with:
kubectl logs -n vmware-system vmware-event-router-b85dd9d87-g9cm5
We see that the event router has started up and is receiving events from our vCenter server.
Our set up worked, but what if yours failed?
kubectl get pods -A
You might log in and find the event router pod in a crashed state.
Check the pod logs to see if you can figure out the problem
kubectl logs -n vmware-system vmware-event-router-b85dd9d87-jhgtx
The logs show us that we failed to log into the vCenter. Time to check the configuration file.
The event router pod configuration file is located at /root/config/event-router-config.yaml
cd /root/config ls -al
We see the .yaml file is there. Edit the .yaml file with vi
Our error message makes it obvious that we have a bad username or password. I intentionally put in a bad password to cause this error. But you might have made a mistake in the vCenter URL, or you may have forgotten to disable SSL checks in your homelab. Make the necessary change to the config file and save it.
We now need to recreate the Kubernetes secret – this is where Kubernetes securely stores credentials
kubectl -n vmware-system get secret
event-router-config is the one we need to rebuild
Delete the event-router-config secret
kubectl -n vmware-system delete secret event-router-config
Recreate the secret
kubectl -n vmware-system create secret generic event-router-config --from-file=event-router-config.yaml
Now we need to delete the crashing pod. Doing this will cause Kubernetes to recreate the pod with our new secret.
Find the broken pod
kubectl get pods -A
The pod is named ‘vmware-event-router-b85dd9d87-jhgtx’. Delete the pod.
kubectl -n vmware-system delete pod vmware-event-router-b85dd9d87-jhgtx
Now look for the new pod
kubectl get pods -A
This time the pod is running. Check the logs
kubectl logs -n vmware-system vmware-event-router-b85dd9d87-5tw7x
That’s all for this post. In Part II, we look at preqreqs to deploy functions.