Had a request to show all privileges assigned to CloudAdmin in VMC. Here is a PowerCLI one-liner to dump a list of permissions.
Get-VIRole “CloudAdmin” | Get-VIPrivilege | Select-object ParentGroup, Description | Sort-Object ParentGroup | Export-Csv permissions.csv
Here’s the output file: permissions.csv
In Part VII of this series, we deployed a second sample function – the Host Maintenance functions written in PowerShell. In this post, we discuss some basic troubleshooting techniques.
The VEBA is running Kubernetes. Going into this project, I knew nothing about Kubernetes – and still don’t know very much. But I’ve learned enough to be able to do some basic troubleshooting. In the end it’s all a series of Linux commands.
VEBA code is running in Kubernetes pods, and those pods are running in namespaces. kubectl lets you run commands against Kubernetes. So first we list out namespaces with kubectl get namespaces
One of the namespaces you will see is openfaas. All of your functions will be running inside their own container. We issue the command kubectl get pods -n openfaas and see all of our pods that are running OpenFaaS.
kubectl logs gets us logfiles from the pods. The vCenter Connector pod is responsible for communicating with vCenter. Obviously these logs could contain interesting information if you’re trying to troubleshoot VEBA communications with vCenter.
kubectl logs vcenter-connector-566c9b7494-87168 -n openfaas
Here is the output in a screengrab and also pasted. Note that we see events topics firing (drs.vm.powered.on) and successful interception and function invocation. One thing you could do is to perform an action on vCenter and then look in this logfile to figure out which topic fired. You can then build a function to react to the event.
2019/12/28 04:43:30 Event [0] &{{{{{} 76868 76864 2019-12-28 04:43:29.634 +0000 UTC VSPHERE.LOCAL\Administrator 0xc0001d7590 0xc0001d75c0 0xc0001d75f0 0xc0001d7620 <nil> <nil> <nil> DRS powered On testvm2 on esx02.ad.patrickkremer.com in HomeLab } false}}}
2019/12/28 04:43:30 message: {“topic”:”drs.vm.powered.on”,”category”:”info”,”source”:”vc01.ad.patrickkremer.com”,”userName”:”VSPHERE.LOCAL\\Administrator”,”createdTime”:”2019-12-28T04:43:29.634Z”,”objectName”:”testvm2″,”managedObjectReference”:{“Type”:”VirtualMachine”,”Value”:”vm-61″}}
2019/12/28 04:43:30 Message on topic: drs.vm.powered.on
2019/12/28 04:43:30 Invoke function: pytag-fn
2019/12/28 04:43:31 Response [200] from pytag-fn
In the above log output, we can see that OpenFaaS caught a VM powering up, and invoked the pytag-fn. But we don’t know what happened inside that function.
Let’s look directly into the function logs. Your functions run in the openfaas-fn namespace. We list out the pods.
kubectl get pods -n openfaas-fn
Looking at the logs, we see the tag succeeded.
Now let’s break it. I update my secret with a configuration that has a bad password.
Now I want to look at the logs for my pytag function. Again, the commands are
kubectl get namespace
kubectl get pods -n openfaas-fn
kubectl logs pytag-fn-f66d6cffc-x5ghk -n openfaas-fn
An astute reader might notice that my pytag container changed names between the previous screenshot and the current one. The content for this post was written in 2 sessions – when I came back for the second screenshot, I had redeployed the function numerous times. This means I got a new pod with a new name.You can always find the pod name with the get pods command.
Here are the logs when I try to power up a VM after I pushed the broken vcconfig secret:
The log clues us in to an unauthorized error. It doesn’t specifically say password error, but you know there’s something wrong with authentication – at least you have a place to start troubleshooting
2019/12/29 00:46:07 Path /
{“status”: “500”, “message”: “could not connect to vCenter 401 Client Error: Unauthorized for url: https://vc01.ad.patrickkremer.com/rest/com/vmware/cis/session”}
Now let’s fix it – we update the secret with the correct password
You can see in the logfile there was one more authentication failure before the new secret was picked up. I’m not sure at the time I posted this exactly what governs picking up a new secret, but it is important to know that it is not instant.
A useful flag for the log command is –follow
kubectl logs –follow vcenter-connector-566c9b7494-nkcw5 -n openfaas
This is essentially the equivalent of tail -f – a realtime view of the logfile. The screen will continue to scroll as new logs appear in the logfile.
In Part VI of this series, we showed how to sync a fork of our VEBA repository to the upstream repository maintained by VMware. Back in Part IV, we deployed our first sample function. In this post, we will deploy another sample function – the host maintenance function.
Our first sample function was written in Python. As of the date of this post, the other available samples are all in PowerShell. We will be working with the hostmaint-alarms function. This function will disable alarm actions when you pop a host into maintenance mode. No more alerts for a host that you’re doing maintenance on!
But first, we have a problem. Here is the stack.yml file for our python tagging function
Here is the sample file we used to generate the secrets named ‘vcconfig’.
Here is the stack.yml for our host maintenance alarms function
Here is the vcconfig.json file we use to configure the secret named ‘vcconfig’ for our PowerShell function.
Do you see the problem? We have a secret named vcconfig used for both functions, but the secret files have a completely different configuration. Neither script can read the other’s secret because they weren’t programmed to do so. The TOML secret file is a configuration file format popular with Python. The PowerShell secret file is simple JSON. This means that we will need to change the secrets file to a different name, one for the Python script and one for PowerShell. Note that it doesn’t have to be this way – there’s nothing stopping a script writer from using TOML format for PowerShell and JSON for Python – all that matters is how how the script is written. You could write your scripts to use a single secret format and they could all share a single secret.
We now need to change the sample script to point to a different secrets file. In order to do that, we need create a new secret using our vcconfig.json file. The file looks like this:
After editing the file to point to our environment, we push it into the VEBA appliance. I name it ‘vcconfig-hostmaint’ but you can name it whatever you want. Remember you need to log in with faas-cli first, for a refresher look at Part IV of this series.
Now that we have our secrets file, we need to change our code to use it.
First we edit the first line of script.ps1 in the handler folder. We need to change the secret name to whatever we named it in the cli – here, I change it to: vcconfig-hostmaint
Looking again at the stack.yml file, we have additional problems. We built a new secret, so we can change the secrets: section to point to vcconfig-hostmaint. Our gateway needs to point to our VEBA appliance. Then we need to worry about our image. Because we changed PowerShell code, we have to rebuild the container image that runs the code. Otherwise, the container that launches is the default container that ships with the VEBA appliance.
We sign up for a Docker account
Now we download and install Docker Desktop. The installation is simple, you can find the official installation documentation here.
After installation there’s a little whale icon in my system tray
I right-click and log in with the account I just created
Now when I right-click the whale, it shows that I’m signed in.
Now we edit the stack.yml file. We make sure our gateway is pointing to our VEBA. We change the secrets to point to our new secret. And we change the image name – the account name gets changed to the Docker account that we just created.
Now we need to build, push, and deploy our new container image. The faas-cli function creation documentation shows us the commands we need to use.
First, we need to log in to Docker. Since I had no experience with Docker, it took me forever to figure out this seemingly simple task. I tried many different ways, but all failed with an Unauthorized error.
The simple answer is to make sure you’re logged into the Docker desktop
Then you issue a docker login command with no arguments. Just docker login.
We now issue the faas-cli build -f stack.yml command.
Multiple screens of information scroll past, but we end with a successful build.
Now we push the containers into the Docker registry with faas-cli push -f stack.yml
Now we deploy the container to our VEBA with faas-cli deploy -f stack.yml –tls-no-verify
PROTIP: Once you understand what the 3 commands do – build, push, deploy – you can use a nifty OpenFaaS shortcut command: faas-cli up –tls-no-verify
This command runs build, push, and deploy in sequence, automatically.
Now we log into vCenter and look at the Alarm actions on a host. They are currently enabled.
After entering maintenance mode, the alarm actions have been disabled by our function
Now we exit maintenance mode and our function enables alarm actions. Success!
Finally, we want to verify that we did not break our other Python function, the one that tags a VM when it powers on. We check our test VM and see it has no tags.
After power on, a tag has been applied. Now we have both functions working!
We have now successfully deployed our PowerShell host maintenance function! In Part VIII, we will look at some troubleshooting techniques.
In Part V of this series, we looked at how to contribute to the VEBA project. In this short post, we will show how to sync your fork back to the upstream repository – the project that you originally forked from.
If you work with an open source fork for any length of time, you will inevitably come to a situation where your fork is out of sync with the upstream repo. Other people have contributed to the upstream repo, and your code is now out of sync. You could issue a pull request to pull the upstream into your fork, but then you are still out of sync – your fork will be 1 commit ahead of the upstream.
The following post shows how I used Github’s Syncing a Fork post to sync my repo with the upstream VEBA project.
First, my fork in Github shows that it’s 5 commits behind the upstream repository vmware-samples:master.
In order to fix this, I need the upstream’s clone URL
We add the URL as an upstream branch
Fetch the upstream code and then check out our local master branch
Merge the upstream changes into my local branch
We then push the local copy into the Github repo
Our fork is now even with the upstream repo.
In Part VII, we will look at deploying additional VEBA sample functions.
In Part IV of this series, we successfully deployed our sample function, it fired when a VM powered up and correctly tagged the VM.
In this post, we will explore contributing to the vCenter Event Broker Appliance Open Source project. Open Source is obviously reliant on contributions from the community. One way to contribute is using your skills as a developer. But if you’re not a developer, you can still contribute in the form of documentation. When I started working with VEBA, I knew nothing about git. You can do some extremely complex things with Git – but the basics of making a change to a file and submitting it back for inclusion into this project are easy to learn.
Note: this post assumes you’ve already installed Git. Take a look at Part II if you don’t have Git installed.
For further learning outside of this blog post, I strongly recommend Commitmas, a great vBrownBag series on how to use git. Thank you to Kyle Ruddy for the suggestion.
The first thing you need is a Github account. Once you’ve signed up and verified your account, open up your Git Bash and use the git config –global command to set the user.name and user.email variables to match your name and verified e-mail address from Github. You will need this to match in order to sign your code when you commit it.
There are a few basic git operations that you need to understand:
As I was going through the documentation, I read a file named getting-started.md. An .md file is a markdown file, you will find most documentation in Github written in this format.
A few of the parts of this getting started file were incorrect. When I tried to use the referenced stack.yml file, I received an error saying that the provider name was invalid, and instead should be ‘openfaas’. I changed “name: faas” to “name: openfaas” and it worked.
I then found a typo of “read_debuge” instead of “read_debug”.
Finally, there was a mistake in the faas-cli commands at the bottom of this screenshot. The second command says to use “faas” when it should have read “faas-cli”.
I wanted to fix these and save them back to the main VEBA repository so others could benefit from my updates.
Because I don’t have rights to directly modify code in the VEBA repository, I need to fork the project. You can see the Fork button on the top righthand corner.
After clicking on the fork button, I end up with a copy of the code in my own personal account – Github tells you on the top lefthand corner that this is the vcenter-event-broker-appliance repository under my account, kremerpatrick, but it’s forked from the vmware-samples account
Now that I have a repository in my own personal account, I’m going to clone it to my local workstation
Note in this clone command I call my local folder “vcenter-event-broker-appliance4” because I have many different copies of the VEBA project as I played around with git.
I edit the getting-started.md file and make the 3 fixes – provider name, read_debug, and faas-cli. I save the file
Now I issue a git diff command. This shows me the differences between the files I edited and the code in my local repository. You can see all 3 of my changes, removed code is in red, and added code is in green.
Everything looks good in the diff. Time to commit the code. I issue the command
git commit -a -s
-a stands for “all”, meaning we want to commit all changed files. I only changed one in this case, but if you changed multiple files, the -a switch is one way to commit all of them.
-s means that I’m signing the file with the user.name and user.email variables that we populated above.
When I issue the command, git pops open my text editor of choice. I need to write a comment documenting my changes. I write it, save it, then close the text editor.
The commit is now complete. As expected, 1 file changed, and 3 lines of code changed in that file.
Now the code is committed locally, and I need to push the code up to my personal Github repository
I go back to Github and find my repository
I click on my repository and Github reports that my branch is 1 commit ahead of the base repository. This is expected as I performed 1 commit.
Now I need to ask the maintainers of the VEBA repository to merge my change into their repository. This is called a pull request (PR). I click on the “New pull request” button in my repository.
Because my repository is forked from the base VMware repository, git knows what to compare my commit against. It shows me a diff, and that diff matches the diff that I ran myself in git – all 3 of my changes are there. Everything looks OK so I click “Create pull request”.
This is the last step to filing a PR – I write a summary and explanation of my fixes. In my case, I had already fixed these errors elsewhere in the code, but I missed getting-started.md. I fill out the fields and click Create pull request.
My changes are now ready to merge. The repository owners are now notified that there is a new PR to approve. They can respond to me with requests for changes, or they can commit my change.
All done! We have now made changes to an open source repository – everybody who clones this repository after the PR is merged will be able to take advantage of the changes I sent over.
In Part VI of this series, we will look at how to sync our fork back to the upstream repository.
In Part III of this series, we created our vCenter tag and cloned the code repository from Github. In Part IV, we will deploy our function and watch it fire in response to a vCenter event.
The python sample function is sitting in the root path /examples/python/tagging. The sample function will cause a vCenter tag to be applied to a VM when the VM powers on.
First we want to modify vcconfig.toml
Here we need to change the server to our vCenter server, put in the correct username/password, and change the tag URN to match the URN we retrieved in Part III
To quickly review, one way to get a Tag’s URN is via the Get-Tag PowerCLI cmdlet
vcconfig.toml now looks like this
We continue with the OpenFaaS instructions in the getting started documentation.
When the correct password is used, we should get this:
If you get an error, a number of things could be the cause:
Next, we want to create a secret named ‘vcconfig’ in faas-cli so we’re not keeping credentials in plaintext files.
Next, we edit stack.yml. The gateway has to change to your VEBA VM – make sure it matches the name you deployed. We also need to look at the topic. The documentation warns us that if we’re trying to monitor a VM powered on event in a DRS cluster, the topic is actually drs.vm.powered.on. We change the topic: entry to match.
My changed stack.yml looks like this:
We now issue a faas-cli template pull command (only required for the first deployment), and then deploy the function with:
faas-cli deploy -f stack.yml –tls-no-verify
Now the moment of truth, time to test our function! The expected behavior is that when powered on, a VM gets the alert-power-operations tag assigned to it. First we check my test VM to see if it has any tags – it has none.
After we power on the VM, has the tag been assigned?
Success! Note that you may have to hit the refresh button to get the tag to appear in the GUI.
That’s it for your first sample function!
Since this product is open source, you can make changes to it. Even if you’re not a developer, a great way to contribute is to keep documentation up-to-date. In part V, we will explore the basics of using git to commit code back to the repository. Even if you’re not a developer, you can learn the basics of git.
In Part I of this series, we explored how to deploy the VEBA appliance. In Part II, we looked at setting up a Windows workstation with the prereqs for interacting with VEBA. Part III focuses on working with vCenter tags and cloning a repository. The sample function will cause a vCenter tag to be applied to a VM when the VM powers on. At the end of Part III, you should have a vCenter tag defined, the tag’s URN, and a copy of the sample code from Github.
If you don’t want to learn about govmomi, you do not need to learn it for the code sample to work. You can create a tag named anything you like in the vCenter GUI. You can then use the get-tag PowerCLI cmdlet to retrieve the needed tag URN for the function.
In the getting started documentation, the Categories and tags section tells us to set 2 environment variables:
export GOVC_INSECURE=true
export GOVC_URL=’https://vcuser:vcpassword@vcenter.ip’
I set them to match my environment
govc tags.ls will list out all of the tags defined in your vCenter. If you have a brand new lab vCenter, this command will return nothing – but it should also not error out. For my lab, it shows the only tag that I have defined: alert-power-operations
I confirm the command was accurate by looking at the Tags & Custom Attributes area of vCenter
A few things about govc – you can invoke help to list all available operations – this is just one page of results
You can also show help on a specific command.
Note that this help dynamically reads environment variables – my lab’s vCenter is showing up as GOVC_URL in this help output.
The VEBA documentation says to take note of the tag’s URN when you create it with govc. We can use the tags.info command to get the URN of an existing tag without having to create it using govc.
You can retrieve the same information from PowerCLI if you’re more familiar with it – you don’t even need to use govc
Next, we need to clone the sample repository. There will be more on git later in this series, but for now you just need to be able to clone a repository and use a text editor.
I create an empty git folder, right-click inside of the folder and choose Git Bash here.
The documentation says to run the following commands:
git clone https://github.com/vmware-samples/vcenter-event-broker-appliance
cd vcenter-event-broker-appliance/examples/python/tagging
Git clone creates a copy of the VEBA sample code repository on your local system. Note that I passed a folder argument in the git clone command. By default, git would have created a folder called vcenter-event-broker-appliance and cloned the code inside of it. But I already have 2 copies of the repository for various testing reasons. I pass the clone command a folder argument of vcenter-event-broker-appliance3. Git clones the code for me and I CD to the new directory
The code is now downloaded to my laptop. In Part IV, we will customize the code for my lab environment and deploy the sample function to the VEBA appliance
In Part I of this series, we explored how to deploy the VEBA appliance. In Part II, we look at setting up a Windows workstation with the prereqs for interacting with VEBA.
We begin with the Function Deployment section of the Getting Started Guide shown on the VEBA Fling instructions page. The guide lists 3 prereqs to getting started: git, faas-cli, and govc. Going into this I have no idea what any of that means. I know what git is, meaning I know the definition of source control and that git is one of many choices for source control. But I’ve never used it, and don’t know anything about faas-cli or govc
I start with the git download page. As I go through this, I seem to find myself at a disadvantage trying to do code on Windows – everything seems to be focused around Mac and Linux. But I successfully installed the Windows binary on my laptop.
I leave the defaults here
I know the basics of getting around vi, but spend a lot more time in Notepad++, so I switch to it here.
This seems to be an OK option – I’m not sure what people typically use. Through the sample code deployment I was only calling git from the Git Bash anyway, so I guess it doesn’t make a difference.
I don’t think I have a need to use Windows CAs so I kept OpenSSL.
I leave this default since it was the recommended Windows setting
I don’t like the Windows default console windows, I figured MinTTY couldn’t be any worse – I select it.
I leave all these default.
The next preqeq is faas-cli. I have no idea what this is, so I click on the link and read about it. faas-cli is a command line interface for OpenFaaS – a serverless functions framework for Docker and Kubernetes. OpenFaaS is built into the VEBA appliance – it is how you publish your custom code to run when an event happens.
For Windows, all I did is go to the releases page and download faas-cli.exe.
I decide to create a specific folder for any executables related to this project, so I created a folder and added that folder to my PATH variable. This way I can invoke the executable no matter where I am in the filesystem.
The final prereq is something called govc, part of govmomi. govmomi is a Go library for interacting with the vSphere APIs. govc is a CLI interface to the library. I did not know why govmomi existed when I started documenting my work with VEBA – thanks to Michael Gasch for explaining “govmomi is a library many VMware products and OSS projects use to provide a vSphere API library for the Go programming language.”
I was able to find and download govc_windows_386.exe at the govmomi releases page. I put it in the same folder as faas-cli.exe
We have now installed all of the prereqs for running our first sample function. In Part III, we will set up vCenter tags for the sample function and download the VEBA sample code to our local workstation with git.
I recently became aware of the vCenter Event Broker Appliance Fling, Open Source code released by VMware which allows customers to easily create event-driven automation based on vCenter Server Events. You can think of it as a way to run scripts based on alarm events – but you’re not limited to only the alarm events exposed in the vCenter GUI. Instead, you have the ability to respond to ANY event in vCenter.
Did you know that an event fires when a user account is created? Or when an alarm is created or reconfigured? How about when a distributed virtual switch gets upgraded or when DRS migrates a VM? There are more than 1,000 events that fire in vCenter; you can trap the events and execute code in response using VEBA. Want to send an email and directly notify PagerDuty via API call for an event? It’s possible with VEBA. VEBA frees you from the constraints of the alarms GUI, both in terms of events that you can monitor as well as actions you can take.
VEBA is a client of the vSphere API, just like PowerCLI. VEBA connects to vCenter to listen for events. There is no configuration stored in the vCenter itself, and you can (and should!) use a read-only account to connect VEBA to your vCenter. It’s possible for more than one VEBA to listen to the same single vCenter the same way multiple users can interact via PowerCLI with the same vCenter.
For more details, make sure to check out the VMworld session replay “If This Then That” for vSphere- The Power of Event-Driven Automation (CODE1379UR)
If you notice screenshots alternating between an appliance named veba01 and veba02, it’s because veba02 was my v0.1 appliance and veba01 is my new v0.2 appliance. I realize the naming is backwards, but veba02 was actually my second v0.1 appliance and I re-used the old veba01 IP and DNS entry.
The instructions can be found on the VEBA Getting Started page.
First we download the OVF appliance from https://flings.vmware.com/vcenter-event-broker-appliance and deploy it to our cluster
I learned from my first failed VEBA deployment to read the instructions – it says the netmask should be in CIDR format, but I put in 255.255.255.0. Make sure you’re using CIDR format.
Another potential gotcha is DNS settings – DNS is space separated in the OVF, although the 0.2 version guides you with a prompt for space separated.
Both NTP and proxy support are new in 0.2. I don’t have a proxy server in the lab so I am not configuring it here.
The POD CIDR network is an important setting new in 0.2. This network is how the containers inside the appliance communicate with each other. If the IP that you assign to the appliance is in the POD CIDR subnet, the containers will not come up and you will be unable to use the appliance. You must change either the POD CIDR address or the appliance’s IP address so they do not overlap.
This is what the VEBA looks like during first boot
If you end up with this, [IP] in brackets instead of your hostname, something has failed. Did you use a subnet mask instead of CIDR? Did you put comma-separated DNS instead of space? Did you put in the incorrect gateway? If you enabled debugging at deploy time, you can look at /var/log/boostrap-debug.log for detailed debug logs to help you pinpoint the error. If not, see what you can find in /var/log/boostrap.log
If you made a configuration error and the initial setup was unsuccessful, you can make changes to the vApp properties and reboot. Note that changes made to the vApp properties after the first successful boot of the VEBA will not alter the appliance configuration.
In this case, the debug log showed me that DNS lookups were failing, and I used comma-separated DNS. I also used the wrong gateway, so I had to make multiple fixes before setup succeeded. The VM must be powered off to make changes to the vApp Options. I clicked on guestinfo.dns, then clicked the “SET VALUE” option, and correctly entered my DNS servers with space separation. Note that in future releases of the VEBA appliance, the configuration screen will notify you that you need to use space-separated DNS.
This represents a successful VEBA boot.
Now we have a VEBA appliance and need to configure it with functions to respond to events. Note that it may take some time for the appliance to respond to http requests, particularly in small homelab. It took about 4 minutes for it to fully come up after succesful boot in my heavily overloaded homelab. One additional gotcha: the name you input at OVF deployment time is the name encoded in the SSL certificate, so if you input “veba01” as your hostname, you cannnot then reach it with https://veba01.fqdn.com.
In part II of this series, I will demonstrate how I configured my laptop with the prereqs to get the first sample function working in VEBA.
We have an internal tool in Slack where you can see how long you’ve been employed relative to other VMware employees i.e. X-employee account created on yyyy-mm-dd and has been here longer than XY% of employees. After looking at that, I wondered just how quickly we’ve grown over the years. So I grabbed employee counts reported in our 10-K filings at ir.vmware.com and graphed them. We just keep growing!
