vCenter Event Broker Appliance – Part VIII – Basic Troubleshooting Techniques

In Part VII of this series, we deployed a second sample function – the Host Maintenance functions written in PowerShell. In this post, we discuss some basic troubleshooting techniques.

The VEBA is running Kubernetes.  Going into this project, I knew nothing about Kubernetes – and still don’t know very much. But I’ve learned enough to be able to do some basic troubleshooting. In the end it’s all a series of Linux commands.

VEBA code is running in Kubernetes pods, and those pods are running in namespaces. kubectl lets you run commands against Kubernetes. So first we list out namespaces with kubectl get namespaces

One of the namespaces you will see is openfaas. All of your functions will be running inside their own container. We issue the command kubectl get pods -n openfaas and see all of our pods that are running OpenFaaS.

kubectl logs gets us logfiles from the pods. The vCenter Connector pod is responsible for communicating with vCenter. Obviously these logs could contain interesting information if you’re trying to troubleshoot VEBA communications with vCenter.
kubectl logs vcenter-connector-566c9b7494-87168 -n openfaas

 

Here is the output in a screengrab and also pasted.  Note that we see events topics firing (drs.vm.powered.on) and successful interception and function invocation. One thing you could do is to perform an action on vCenter and then look in this logfile to figure out which topic fired. You can then build a function to react to the event.

2019/12/28 04:43:30 Event [0] &{{{{{} 76868 76864 2019-12-28 04:43:29.634 +0000 UTC VSPHERE.LOCAL\Administrator 0xc0001d7590 0xc0001d75c0 0xc0001d75f0 0xc0001d7620 <nil> <nil> <nil> DRS powered On testvm2 on esx02.ad.patrickkremer.com in HomeLab } false}}}

2019/12/28 04:43:30 message: {“topic”:”drs.vm.powered.on”,”category”:”info”,”source”:”vc01.ad.patrickkremer.com”,”userName”:”VSPHERE.LOCAL\\Administrator”,”createdTime”:”2019-12-28T04:43:29.634Z”,”objectName”:”testvm2″,”managedObjectReference”:{“Type”:”VirtualMachine”,”Value”:”vm-61″}}

2019/12/28 04:43:30 Message on topic: drs.vm.powered.on

2019/12/28 04:43:30 Invoke function: pytag-fn

2019/12/28 04:43:31 Response [200] from pytag-fn

In the above log output, we can see that OpenFaaS caught a VM powering up, and invoked the pytag-fn. But we don’t know what happened inside that function.

 

Let’s look directly into the function logs. Your functions run in the openfaas-fn namespace. We list out the pods.
kubectl get pods -n openfaas-fn

Looking at the logs, we see the tag succeeded.

 

Now let’s break it. I update my secret with a configuration that has a bad password.

 

Now I want to look at the logs for my pytag function. Again, the commands are

kubectl get namespace
kubectl get pods -n openfaas-fn
kubectl logs pytag-fn-f66d6cffc-x5ghk -n openfaas-fn

An astute reader might notice that my pytag container changed names between the previous screenshot and the current one. The content for this post was written in 2 sessions – when I came back for the second screenshot, I had redeployed the function numerous times. This means I got a new pod with a new name.You can always find the pod name with the get pods command.

Here are the logs when I try to power up a VM after I pushed the broken vcconfig secret:

The log clues us in to an unauthorized error. It doesn’t specifically say password error, but you know there’s something wrong with authentication – at least you have a place to start troubleshooting

2019/12/29 00:46:07 Path /
{“status”: “500”, “message”: “could not connect to vCenter 401 Client Error: Unauthorized for url: https://vc01.ad.patrickkremer.com/rest/com/vmware/cis/session”}

 

Now let’s fix it – we update the secret with the correct password

You can see in the logfile there was one more authentication failure before the new secret was picked up. I’m not sure at the time I posted this exactly what governs picking up a new secret, but it is important to know that it is not instant.

A useful flag for the log command is –follow
kubectl logs –follow vcenter-connector-566c9b7494-nkcw5 -n openfaas

This is essentially the equivalent of tail -f – a realtime view of the logfile. The screen will continue to scroll as new logs appear in the logfile.

 

 

 

 

1 thought on “vCenter Event Broker Appliance – Part VIII – Basic Troubleshooting Techniques

  1. Pingback: vCenter Event Broker Appliance – Part VII – Deploy the Sample Host Maintenance Function | Patrick Kremer

Leave a Reply

Your email address will not be published. Required fields are marked *