A quick NSX microsegmentation example

This short post demonstrates the power of NSX. My example is a DMZ full of webservers – you don’t want any of your webservers talking to each other. If one of your webservers happens to be compromised, you don’t want the attacker to then have an internal launching pad to attack the rest of the webservers. They only need to communicate with your application or database servers.

We’ll use my lab’s Compute Cluster A as an a sample. Just pretend it’s a DMZ cluster with only webservers in it.

Compute Cluster A


I’ve inserted a rule into my Layer 3 ruleset and named it “Isolate all DMZ Servers”. In my traffic source, you can see that you’re not stuck with IP addresses or groups of IP addresses like a traditional firewall – you can use your vCenter groupings like Clusters, Datacenters, Resource Pools, or Security Tags to name a few.

Rule Source

I add Computer Cluster A as the source of my traffic. I do the same for the destination.

NSX Source Cluster


My rule is now ready to publish. As soon as I hit publish changes, all traffic from any VM in this cluster will be blocked if it’s destined for any other VM in this cluster.

Ready to publish


Note that these were only Layer3 rules – so we’re secured traffic going between subnets. However, nothing’s stopping webservers on the same subnet from talking to each other. No worries here though, we can implement the same rule at layer 2.

Once this rule gets published, even VMs that are layer 2 adjacent in this cluster will be unable to communicate with each other!

NSX layer 2 block

This is clearly not a complete firewall policy as our default rule is to allow all. We’d have to do more work to allow traffic through to our application or database servers, and we’d probably want to switch our default rule to deny all. However, because these rules are tied to Virtual Center objects and not IP addresses, security policies apply immediately upon VM creation. There is no lag time between VM creation and application of the firewalling policy – it is instantaneous!  Anybody who’s worked in a large enterprise knows it can take weeks or months before a firewall change request is pushed into production.

Of course, you still have flexibility to write IP-to-IP rules, but once you start working with Virtual Center objects and VM tags, you’ll never want to go back.

Enable and Disable GPOs with PowerShell

Post Updated April 13, 2015

I received a comment below saying the BSonPosh link to Microsoft was dead. It appears that Microsoft has retired the code modules. It also looks like they have a native PowerShell equivalent, examples of how to use it are here.

If you don’t want to modify the script to use the native Microsoft method, I do still have my original download of the modules. Here’s a link to the original BSonPosh modules BSonPosh.zip

Original Post Feb 19, 2011

We had a need to enable and disable groups of GPOs on a recurring basis and wanted to automate the process.

This script relies on the BSonPosh module. It also relies on the Windows PowerShell Group Policy cmdlets. The Group Policy cmdlets are on Windows 2008 R2 DCs, a server with the GPMC installed, or Windows 7 with the RSAT installed.

To use the script, create a text file with the name of each GPO that you want to control via the script. The script takes 2 parameters, whether to enable or disable the GPOs, and the name of the textfile with the list of GPOs.

	#Enabled or Disabled, whether you want the GPOs enabled or disabled
	[string]$GPOStatus = $(Throw '$GPOStatus is required'),
	#List of GPOs to enable/disable
	[string]$GPOList =  $(Throw '$GPOList is required')
	$GPO_DISABLED = "AllSettingsDisabled"
	$GPO_ENABLED  = "AllSettingsEnabled"
	#Change the specified GPO's GpoStatus property
	function SetGPOStatus( [string]$GPOName, [string]$Status )
		$gpo=Get-GPO $GPOName -server $PDC.ServerName -errorAction SilentlyContinue
		if ( $gpo -eq $null )
			write-host "Could not locate" $GPOName
			$gpo.GpoStatus = $Status
			Write-Host "Set"$gpo.DisplayName"to"$gpo.GpoStatus
	#Attempt to load a module with Import-Module
	function TryImportModule( [string]$ModuleName )
		if ( (Get-Module $ModuleName ) -eq $null )
			Import-Module $ModuleName
			if ( (Get-Module $ModuleName ) -eq $null )
				Write-Host "Unable to load module" $ModuleName
				return $false
		return $true
	# Microsoft module to manage Group Policy
	$retval = TryImportModule "grouppolicy"
	if ( $retval -eq $false )
	# Community module that will help retrieve FSMO roles
	$retval = TryImportModule "bsonposh"
	if ( $retval -eq $false )
	# Modify the GPOs on the server with the PDC Master FSMO role
	$PDC = Get-Fsmo -role "PDCMaster" -errorAction SilentlyContinue
	if ( $PDC -eq $null )
		write-host "Could not locate PDC Master"
	# Validate Status flag input
	if ( $GPOStatus.ToLower() -eq "disabled" )
		$SetFlag = $GPO_DISABLED
	elseif ( $GPOStatus.ToLower() -eq "enabled" )
		$SetFlag = $GPO_ENABLED
		Write-Host "Invalid value '$GPOStatus' for paramGPOStatus. Allowed values: [Disabled|Enabled]".
	# Ensure we actually have a list of GPOs in our text file
	if ( (Test-Path $GPOList) -eq $false )
		write-host "Could not locate"$GPOList
		$AllGPOs = Get-Content $GPOList
		if ( $AllGPOs -eq $null )
			write-Host $GPOList" is empty."
		foreach ( $myGPO in $AllGPOs )
			if ( $myGPO.SubString(0,1) -ne "#" ) #Allows comments in the text file
				SetGPOStatus $myGPO $SetFlag

Example usage: .\SetGPOStatus.ps1 -GPOStatus “Disabled” -GPOList “gpolist.txt”

Alzheimer’s Association – Forgotten Donation

Chris Wahl just put up this blog post showing the donation of royalties from his book, Networking for VMware Administrators. I won my copy for free and at the time I promised to donate to the Alzheimer’s association. I failed to do so, but I have rectified that today.

Below is my personal donation along with VMware’s matching gift. $31.41 is the minimum donation to receive a matching gift with VMware’s matching program.  alz-donation alz-matching

Chicago parking enforcement lies, hits me with bogus ticket

Update May 21, 2015

I received a letter from the City of Chicago saying the ticket was dismissed. Unfortunately, there was no mention of disciplining the officer, but at least I’m not on the hook for the ticket.

Original Post: March 15, 2015

I received a $100 parking ticket in Chicago the other day. However, I wasn’t parked illegally. I’m sure I’m one of thousands of people who have been hit with bogus parking tickets. However, the officer who did this isn’t as sneaky as he thinks he is. Maybe it’s a she, I don’t know. In any event, Officer Miles, ID #639, has presented false evidence.

Here’s the citation. I have obscured the full ticket number as well as my license plate.

The actual citation

The actual citation

Here is the picture of the sign that I was allegedly violating.  Officer Miles took this picture and attached it to my citation. Note the timestamp.

Officer Evidence #1


This is a picture that Officer Miles took of my license plate. Note the timestamp – 5 minutes and 10 seconds after the first picture.Officer Evidence #2

For starters, these pictures prove nothing. They don’t prove that I was parked illegally, all they prove is that Officer Miles took a picture of a sign, and then 5 minutes and 10 seconds later took a picture of my license plate.

Here are the pictures I took when I found the ticket on my car. In this one, you can see my vehicle is clearly in front of the pole with a sign pointing to a tow zone in the other direction. If there was a parking restriction in effect, it seems that a great place to put the sign would be on the same pole that shows a parking restriction in the other direction.


Here’s a look at the other side of the pole. This proves that there is only 1 sign, the tow zone, and it’s pointing in the opposite direction. There is no sign showing a 9-6PM restriction.


Here’s a picture shooting up the street with my car on the left, right next to the tow zone pole. There are no other sign poles as far as you can see.

My final picture was taken right next to my passenger door. You can see much more clearly down the block – no signs – and you can also see the lampposts. They look nothing like the lamppost in the picture provided by Officer Miles.


I believe the officer falsified the ticket. He snapped a photograph of the parking restriction sign on another street, then snapped a picture of my vehicle more than 5 minutes later. I am contesting the ticket and hope the judge sees the evidence the same way I see it.


Custom Views and Dashboards in vRealize Operations

This post covers a few of the most common questions my customers ask me as I demonstrate what you can do with vROps. I’m going to take you through an example of needing to frequently check the CPU ready % of your VMs – this was my customer’s most recent request, but know that you can make this happen for any metric collected by vROps.

First, we’re going to create a custom view for CPU Ready % by going to Content>Views, then clicking on the green Plus to add a new View.


I named this one “Custom-CPU Ready” and gave it a description.


Next, pick what our View looks like. In this case, I want to see all of the data in a list format, so I pick List.


Now to select the subjects – these are the objects that the View will be looking at. We want CPU Ready % on Virtual Machines, so we pick the vCenter Adapter and scroll down until we find Virtual Machine.




We now need to find the CPU Ready % metric


Double-click on it when you find it in the list on the left, it will then appear in the Data section. Change the Sort order to descending because we want to see the VM with the highest CPU ready on top.


The Availability options let you control where inside vROps the View will be usable. I lef the defaults.


You now see the custom view in the Views list.


How can we use our brand new view? We want to see the CPU ready for all VMs in the Production cluster. Go to Environment, then drill down into the vSphere World until you reach the Production cluster. Click on the Details tab, you can then scroll down and find the custom View that we created. Click on it and all of your VMs show up, sorted by highest CPU ready.



Let’s say this is a metric that you look at at daily or multiple times a day. You can create a custom dashboard so the metric is immediately visible if you’re using vROps Advanced.

To create a new dashboard, from the Home menu, click Actions, then Create Dashboard


Name the dashboard and select a layout.


We want to show a View in the widget, so we drag View over into the right pane.


Click the Edit icon in the blank View to customize it.


Click on Self Provider to allow us to specify the Production Cluster object on the left, then select our Custom CPU Ready View on the right and click Save.


The dashboard is now ready. The CPU Ready for the Production VMs will now show up in the dashboard.






VMware Hands-on Labs

It always surprises me how few customers are aware of the Hands-on Labs. The labs are full installations of VMware software running in VMware’s cloud, accessible from any browser and completely free for anyone to use.

Each self-paced lab is guided with a step-by-step lab manual. You can follow the manual from start to finish for a complete look at the specific software product. Alternatively, you can focus on specific learning areas due to the modular structure of the lab manual. You can even ignore the manual completely and use the lab as a playground for self-directed study.

You can give the labs a try here: http://labs.hol.vmware.com/

Moving VMs to a different vCenter

I had to move a number of clusters into a different Virtual Center and I didn’t want to have to deal with manually moving VMs into their correct folders. In my case I happened to have matching folder structures in both vCenters and I didn’t have to worry about creating an identical folder structure on the target vCenter. All I need to do was to record the current folder location and move the VM to the correct folder in the new vCenter.

I first run this script against the source cluster in the source vCenter. It generates a CSV file with the VM name and the VM folder name

$VMCollection = @()
Connect-VIServer "Source-vCenter
$CLUSTERNAME = "MySourceCluster"
$vms = Get-Cluster $CLUSTERNAME | Get-VM
foreach ( $vm in $vms )
	$Details = New-Object PSObject
	$Details | Add-Member -Name Name -Value $vm.Name -membertype NoteProperty 
	$Details | Add-Member -Name Folder -Value $vm.Folder -membertype NoteProperty
	$VMCollection += $Details
$VMCollection | Export-CSV "folders.csv"

Once the first script is run, I disconnect each host from the old vCenter and add it into a corresponding cluster in the new vCenter. I can now run this command aginst the new vCenter to ensure the VMs go back into their original folders.

Connect-VIServer "Dest-vCenter"
$vmlist = Import-CSV "folders.csv"
foreach ( $vm in $vmlist )
	Move-VM -VM $vm.Name -Destination $vm.Folder

The parent virtual disk has been modified since the child was created

Some VMs in my environment had virtual-mode RDMs on them, along with multiple nested snapshots. Some of the RDMs were subsequently extended at the storage array level, but the storage team didn’t realize there was an active snapshot on the virtual-mode RDMs. This resulted in immediate shutdown of the VMs and a vSphere client error “The parent virtual disk has been modified since the child was created” when attempting to power them back on.

I had done a little bit of work dealing with broken snapshot chains before, but the change in RDM size was outside of my wheelhouse, so we logged a call with VMware support. I learned some very handy debugging techniques from them and thought I’d share that information here. I went back into our test environment and recreated the situation that caused the problem.

In this example screenshot, we have a VM with no snapshot in place and we run vmkfstools –q –v10  against the vmdk file
-q means query, -v10 is verbosity level 10

The command opens up the disk, checks for errors, and reports back to you.



In the second example, I’ve taken a snapshot of the VM. I’m now passing the snapshot VMDK into the vmkfstools command. You can see the command opening up the snapshot file, then opening up the base disk.




In the third example, I  pass it the snapshot vmdk for a virtual-mode RDM on the same VM –  it traverses the snapshot chain and also correctly reports that the VMDK is a non-passthrough raw device mapping, which means virtual mode RDM.



Part of the problem that happened was the size of the RDM changed (increased size) but the snapshot pointed to the wrong smaller size.  However, even without any changes to the storage, a corrupted snapshot chain can  happen  during an out-of-space situation.

I have intentionally introduced a drive geometry mismatch in my test VM below – note that the value after RW in the snapshot TEST-RDM_1-00003.vmdk  is 1 less than the value in the base disk  TEST-RDM_1.vmdk



Now if I run it through the vmkfstools command, it reports the error that we were seeing in the vSphere client in Production when trying to boot the VMs – “The parent virtual disk has been modified since the child was created”. But the debugging mode gives you an additional clue that the vSphere client does not give– it says that the capacity of each link is different, and it even gives you the values (20368672 != 23068671).

The fix was to follow the entire chain of snapshots and ensure everything was consistent. Start with the most current snap in the chain. The “parentCID” value must be equal to the “CID” value in the next snapshot in the chain. The next snapshot in the chain is listed in the “parentFileNameHint”.  So TEST-RDM_1-00003.vmdk is looking for a ParentCID value of 72861eac, and it expects to see that in the file TEST-RDM_1.vmdk.

If you open up Test-RDM_1.vmdk, you see a CID value of 72861eac – this is correct.  You also see an RW value of 23068672. Since this file is the base RDM, this is the correct value. The value in the snapshot is incorrect, so you have to go back and change it to match.  All snapshots in the chain must match in the same way.



I change the RW value in the snapshot back to match  23068672 – my vmkfstools command succeeds, and I’m also able to delete the snapshot from the vSphere client6_vmkfstools


Zoning a SAN – The Danger Zone

Updated May 3, 2014

My autographed copy has arrived!


Updated January 6, 2014

Per @StevePantol‘s comment below, I won a free signed copy of his and @ChrisWahl‘s Networking for VMware Administrators, due out at the end of March!

Original Post: January 1, 2014
On New Year’s Eve, @ChrisWahl tweeted:

ChrisWahl - Dange rZone


@StevePantol responded with:

Steve Pantol Lyrics

Challenge accepted. I officially submit my SAN-themed Danger Zone parody lyrics. To refresh your memory, here is a link to the actual lyrics.

Setting up the switches
Nexus or an MDS
Cables are all ready
One task left before you go

Building fibre channel zones
Adding hosts into the zones

Typin’ world wide names
Lost up in a sea of hex
Thanks for FCNS
Without it you’d be truly vexed

Building fibre channel zones
You will start
Adding hosts into the zones

You’ll never get the port online
Until you bind it to the vfc
You’ll have to keep up with the fight
Until the VSAN membership is right

Hoping to see flogi
Praying to be done and free
The longer that it takes
The greater the insanity

Building fibre channel zones
You finished
Adding hosts into the zones