Life and death monitoring

The Brookfield Zoo posted this statement on Facebook today and also emailed it out to all members.

On July 10, there was a drop in the oxygen level at Brookfield Zoo’s Stingray Bay habitat. Veterinary staff was promptly on the scene to provide medical treatment to the affected stingrays. Additionally, immediate action was taken by animal care staff to rectify the situation and get the levels back to normal. Despite tireless efforts by staff, all the animals, which included four southern stingrays and 50 cownose rays, succumbed.

“We are devastated by the tragic loss of these animals,” said Bill Zeigler, senior vice president of animal programs for the Chicago Zoological Society, which operates the zoo. “Our staff did everything possible to try and save the animals, but the situation could not be reversed.”

Staff is currently analyzing the life support system to determine the exact cause of the malfunction. At this time, the Chicago Zoological Society has made the decision to not reopen the summer-long temporary exhibit for the remainder of the season. The popular exhibit has been operating since 2007.

The zoo posted a further clarification on what kind of monitoring was used in the enclosure.

Brookfield Zoo 15 minute monitoring - Facebook

Brookfield Zoo 15 minute monitoring – Facebook

I’m not a zoologist, I have no experience with monitoring systems for animals. But I do have vast experience monitoring critical services. A simple Google search finds $200 monitors for home aquariums that take readings every 6 seconds. It seems reasonable to assume that a commercial system would offer something similar.

In IT, we take painstaking care to ensure that our critical servers stay online.  Most of what we do has nothing to do with life-and-death. Even though we’re only protecting our company financials and reputation, any halfway decent system that I stand up has:

  • Independent power feeds from separate areas of the building
  • Dual power distribution units, dual power on all equipment
  • N+1 architecture to sustain the loss of one host in the cluster
  • Redundant storage controllers with redundant paths
  • Battery backup
  • Appropriate monitoring

In general, a monitoring interval of 5 minutes is the maximum I would ever allow for a Production server. Critical servers could be monitored as frequently as 1 minute. Load balancers watch services as frequently as every 5-10 seconds. All of this work is done to ensure availability of the services.

Anything can go wrong and it’s possible that a more frequent monitoring interval would not have made a difference in this case. But at first glance, a 15 minute interval seems negligent. If I can monitor my goldfish every 6 seconds, it seems that the zoo should have been monitoring the rays more closely. A 15 minute monitoring interval means that you can’t expect a human response for at LEAST 20 minutes, and that doesn’t seem sufficient when the lives of 54 animals depend on it.

A quick NSX microsegmentation example

This short post demonstrates the power of NSX. My example is a DMZ full of webservers – you don’t want any of your webservers talking to each other. If one of your webservers happens to be compromised, you don’t want the attacker to then have an internal launching pad to attack the rest of the webservers. They only need to communicate with your application or database servers.

We’ll use my lab’s Compute Cluster A as an a sample. Just pretend it’s a DMZ cluster with only webservers in it.

Compute Cluster A

 

I’ve inserted a rule into my Layer 3 ruleset and named it “Isolate all DMZ Servers”. In my traffic source, you can see that you’re not stuck with IP addresses or groups of IP addresses like a traditional firewall – you can use your vCenter groupings like Clusters, Datacenters, Resource Pools, or Security Tags to name a few.

Rule Source

I add Computer Cluster A as the source of my traffic. I do the same for the destination.

NSX Source Cluster

 

My rule is now ready to publish. As soon as I hit publish changes, all traffic from any VM in this cluster will be blocked if it’s destined for any other VM in this cluster.

Ready to publish

 

Note that these were only Layer3 rules – so we’re secured traffic going between subnets. However, nothing’s stopping webservers on the same subnet from talking to each other. No worries here though, we can implement the same rule at layer 2.

Once this rule gets published, even VMs that are layer 2 adjacent in this cluster will be unable to communicate with each other!

NSX layer 2 block

This is clearly not a complete firewall policy as our default rule is to allow all. We’d have to do more work to allow traffic through to our application or database servers, and we’d probably want to switch our default rule to deny all. However, because these rules are tied to Virtual Center objects and not IP addresses, security policies apply immediately upon VM creation. There is no lag time between VM creation and application of the firewalling policy – it is instantaneous!  Anybody who’s worked in a large enterprise knows it can take weeks or months before a firewall change request is pushed into production.

Of course, you still have flexibility to write IP-to-IP rules, but once you start working with Virtual Center objects and VM tags, you’ll never want to go back.

Enable and Disable GPOs with PowerShell

Post Updated April 13, 2015

I received a comment below saying the BSonPosh link to Microsoft was dead. It appears that Microsoft has retired the code modules. It also looks like they have a native PowerShell equivalent, examples of how to use it are here.

If you don’t want to modify the script to use the native Microsoft method, I do still have my original download of the modules. Here’s a link to the original BSonPosh modules BSonPosh.zip

Original Post Feb 19, 2011

We had a need to enable and disable groups of GPOs on a recurring basis and wanted to automate the process.

This script relies on the BSonPosh module. It also relies on the Windows PowerShell Group Policy cmdlets. The Group Policy cmdlets are on Windows 2008 R2 DCs, a server with the GPMC installed, or Windows 7 with the RSAT installed.

To use the script, create a text file with the name of each GPO that you want to control via the script. The script takes 2 parameters, whether to enable or disable the GPOs, and the name of the textfile with the list of GPOs.

Param(
	#Enabled or Disabled, whether you want the GPOs enabled or disabled
	[string]$GPOStatus = $(Throw '$GPOStatus is required'),
 
	#List of GPOs to enable/disable
	[string]$GPOList =  $(Throw '$GPOList is required')
)
Process
{
	$GPO_DISABLED = "AllSettingsDisabled"
	$GPO_ENABLED  = "AllSettingsEnabled"
 
	#Change the specified GPO's GpoStatus property
	function SetGPOStatus( [string]$GPOName, [string]$Status )
	{
		$gpo=Get-GPO $GPOName -server $PDC.ServerName -errorAction SilentlyContinue
		if ( $gpo -eq $null )
		{
			write-host "Could not locate" $GPOName
		}
		else
		{
			$gpo.GpoStatus = $Status
 
			Write-Host "Set"$gpo.DisplayName"to"$gpo.GpoStatus
		}
	}
 
	#Attempt to load a module with Import-Module
	function TryImportModule( [string]$ModuleName )
	{
		if ( (Get-Module $ModuleName ) -eq $null )
		{
			Import-Module $ModuleName
			if ( (Get-Module $ModuleName ) -eq $null )
			{
				Write-Host "Unable to load module" $ModuleName
				return $false
			}
		}
		return $true
	}
 
	# Microsoft module to manage Group Policy
	$retval = TryImportModule "grouppolicy"
	if ( $retval -eq $false )
	{
		return
	}
 
	# Community module that will help retrieve FSMO roles
	$retval = TryImportModule "bsonposh"
	if ( $retval -eq $false )
	{
		return
	}
 
	# Modify the GPOs on the server with the PDC Master FSMO role
	$PDC = Get-Fsmo -role "PDCMaster" -errorAction SilentlyContinue
	if ( $PDC -eq $null )
	{
		write-host "Could not locate PDC Master"
		return
	}
 
	# Validate Status flag input
	if ( $GPOStatus.ToLower() -eq "disabled" )
	{
		$SetFlag = $GPO_DISABLED
	}
	elseif ( $GPOStatus.ToLower() -eq "enabled" )
	{
		$SetFlag = $GPO_ENABLED
	}
	else
	{
		Write-Host "Invalid value '$GPOStatus' for paramGPOStatus. Allowed values: [Disabled|Enabled]".
		return
	}
 
	# Ensure we actually have a list of GPOs in our text file
	if ( (Test-Path $GPOList) -eq $false )
	{
		write-host "Could not locate"$GPOList
		return
	}
	else
	{
		$AllGPOs = Get-Content $GPOList
		if ( $AllGPOs -eq $null )
		{
			write-Host $GPOList" is empty."
			return
		}
		foreach ( $myGPO in $AllGPOs )
		{
			if ( $myGPO.SubString(0,1) -ne "#" ) #Allows comments in the text file
			{
				SetGPOStatus $myGPO $SetFlag
			}
		}
	}	
 
}

Example usage: .\SetGPOStatus.ps1 -GPOStatus “Disabled” -GPOList “gpolist.txt”

Alzheimer’s Association – Forgotten Donation

Chris Wahl just put up this blog post showing the donation of royalties from his book, Networking for VMware Administrators. I won my copy for free and at the time I promised to donate to the Alzheimer’s association. I failed to do so, but I have rectified that today.

Below is my personal donation along with VMware’s matching gift. $31.41 is the minimum donation to receive a matching gift with VMware’s matching program.  alz-donation alz-matching

Chicago parking enforcement lies, hits me with bogus ticket

Update May 21, 2015

I received a letter from the City of Chicago saying the ticket was dismissed. Unfortunately, there was no mention of disciplining the officer, but at least I’m not on the hook for the ticket.

Original Post: March 15, 2015

I received a $100 parking ticket in Chicago the other day. However, I wasn’t parked illegally. I’m sure I’m one of thousands of people who have been hit with bogus parking tickets. However, the officer who did this isn’t as sneaky as he thinks he is. Maybe it’s a she, I don’t know. In any event, Officer Miles, ID #639, has presented false evidence.

Here’s the citation. I have obscured the full ticket number as well as my license plate.

The actual citation

The actual citation

Here is the picture of the sign that I was allegedly violating.  Officer Miles took this picture and attached it to my citation. Note the timestamp.

Officer Evidence #1

 

This is a picture that Officer Miles took of my license plate. Note the timestamp – 5 minutes and 10 seconds after the first picture.Officer Evidence #2

For starters, these pictures prove nothing. They don’t prove that I was parked illegally, all they prove is that Officer Miles took a picture of a sign, and then 5 minutes and 10 seconds later took a picture of my license plate.

Here are the pictures I took when I found the ticket on my car. In this one, you can see my vehicle is clearly in front of the pole with a sign pointing to a tow zone in the other direction. If there was a parking restriction in effect, it seems that a great place to put the sign would be on the same pole that shows a parking restriction in the other direction.

Exhibit1

Here’s a look at the other side of the pole. This proves that there is only 1 sign, the tow zone, and it’s pointing in the opposite direction. There is no sign showing a 9-6PM restriction.

Exhibit2

Here’s a picture shooting up the street with my car on the left, right next to the tow zone pole. There are no other sign poles as far as you can see.
Exhibit3

My final picture was taken right next to my passenger door. You can see much more clearly down the block – no signs – and you can also see the lampposts. They look nothing like the lamppost in the picture provided by Officer Miles.

 

Exhibit4
I believe the officer falsified the ticket. He snapped a photograph of the parking restriction sign on another street, then snapped a picture of my vehicle more than 5 minutes later. I am contesting the ticket and hope the judge sees the evidence the same way I see it.

 

Custom Views and Dashboards in vRealize Operations

This post covers a few of the most common questions my customers ask me as I demonstrate what you can do with vROps. I’m going to take you through an example of needing to frequently check the CPU ready % of your VMs – this was my customer’s most recent request, but know that you can make this happen for any metric collected by vROps.

First, we’re going to create a custom view for CPU Ready % by going to Content>Views, then clicking on the green Plus to add a new View.

1-AddView

I named this one “Custom-CPU Ready” and gave it a description.

2-CustomCPUReady

Next, pick what our View looks like. In this case, I want to see all of the data in a list format, so I pick List.

3-Presentation

Now to select the subjects – these are the objects that the View will be looking at. We want CPU Ready % on Virtual Machines, so we pick the vCenter Adapter and scroll down until we find Virtual Machine.

4a-Subjects

 

4b-Subjects

We now need to find the CPU Ready % metric

5-CPUData

Double-click on it when you find it in the list on the left, it will then appear in the Data section. Change the Sort order to descending because we want to see the VM with the highest CPU ready on top.

6-SelectReady

The Availability options let you control where inside vROps the View will be usable. I lef the defaults.

7-Visibility

You now see the custom view in the Views list.

8-CustomView

How can we use our brand new view? We want to see the CPU ready for all VMs in the Production cluster. Go to Environment, then drill down into the vSphere World until you reach the Production cluster. Click on the Details tab, you can then scroll down and find the custom View that we created. Click on it and all of your VMs show up, sorted by highest CPU ready.

9-ShowReady

 

Let’s say this is a metric that you look at at daily or multiple times a day. You can create a custom dashboard so the metric is immediately visible if you’re using vROps Advanced.

To create a new dashboard, from the Home menu, click Actions, then Create Dashboard

1-AddDashboard

Name the dashboard and select a layout.

2-NameDashboard

We want to show a View in the widget, so we drag View over into the right pane.

2a-WidgetList

Click the Edit icon in the blank View to customize it.

3-EditWidget

Click on Self Provider to allow us to specify the Production Cluster object on the left, then select our Custom CPU Ready View on the right and click Save.

4-AddWidget

The dashboard is now ready. The CPU Ready for the Production VMs will now show up in the dashboard.

5-FinalDashboard

 

 

 

 

VMware Hands-on Labs

It always surprises me how few customers are aware of the Hands-on Labs. The labs are full installations of VMware software running in VMware’s cloud, accessible from any browser and completely free for anyone to use.

Each self-paced lab is guided with a step-by-step lab manual. You can follow the manual from start to finish for a complete look at the specific software product. Alternatively, you can focus on specific learning areas due to the modular structure of the lab manual. You can even ignore the manual completely and use the lab as a playground for self-directed study.

You can give the labs a try here: http://labs.hol.vmware.com/

Moving VMs to a different vCenter

I had to move a number of clusters into a different Virtual Center and I didn’t want to have to deal with manually moving VMs into their correct folders. In my case I happened to have matching folder structures in both vCenters and I didn’t have to worry about creating an identical folder structure on the target vCenter. All I need to do was to record the current folder location and move the VM to the correct folder in the new vCenter.

I first run this script against the source cluster in the source vCenter. It generates a CSV file with the VM name and the VM folder name

$VMCollection = @()
Connect-VIServer "Source-vCenter
$CLUSTERNAME = "MySourceCluster"
 
$vms = Get-Cluster $CLUSTERNAME | Get-VM
foreach ( $vm in $vms )
{
	$Details = New-Object PSObject
	$Details | Add-Member -Name Name -Value $vm.Name -membertype NoteProperty 
	$Details | Add-Member -Name Folder -Value $vm.Folder -membertype NoteProperty
	$VMCollection += $Details
}
 
$VMCollection
$VMCollection | Export-CSV "folders.csv"

Once the first script is run, I disconnect each host from the old vCenter and add it into a corresponding cluster in the new vCenter. I can now run this command aginst the new vCenter to ensure the VMs go back into their original folders.

Connect-VIServer "Dest-vCenter"
$vmlist = Import-CSV "folders.csv"
 
foreach ( $vm in $vmlist )
{
	$vm.Name
	$vm.Folder
	Move-VM -VM $vm.Name -Destination $vm.Folder
}