# My VCAP5-DTD exam experience

I took the VCAP5-DTD beta exam on January 3rd, 2013. Like many people, I received the welcome news today that I passed the exam.

I’m laughing a little to myself as I write this post because my certification folder contains a log of my studying. I downloaded the beta blueprint on December 17, 2012, but I already had Microsoft exams scheduled for December 28th.  I did no studying for this VCAP until the day before the exam, January 2rd, where you can clearly see my feverish morning download activity. I will say though that I have several years of View deployments under my belt, so my knowledge on the engineering side was up-to-date and at the front of my mind.

I downloaded every PDF referenced in the exam blueprint, and I already had most of the product documentation already downloaded. I am primarily a delivery engineer, but to be successful on the exam you need to put on your designer’s hat. I tried to keep that in mind as I pored through the PDFs – it does make a difference because different information will stand out if you actively look for design elements.

My exam was just after lunch and it was well over an hour away, so I left early and brought my Kindle. I continued going through the PDFs until exam time. The sheer volume of information you have to read through makes VMware design exams quite difficult. I suggest reading the answers before you read the question – this helps you identify clues in the question. There are detailed descriptions requiring 6 or more paragraphs of reading just to answer a single multiple choice question.

The GA version of the exam has 115 questions and 6 diagramming scenarios. Keep track of the number of diagramming questions you get so you can budget your time appropriately. You should not spend any more than 15 minutes on a diagram. Keep in mind that 15 * 6 = 90 minutes, leaving you only 105 minutes to answer 109 questions. The pace you have to sustain is mentally exhausting. The beta was even more difficult with 131  questions, plus the expectation to provide comment feedback on the questions.

I found the diagramming questions to be even more involved than the DCD questions.. I’d say the tool was a bit better behaved than the DCD exam, but not by much. It’s easy to get sucked in to a design scenario and waste far too much time. Remember that you’re not designing the perfect system, it just has to be good enough to meet the stated requirements.

# Fear and Loathing of Pearson Vue

Computer testing vendor Pearson Vue suffered a massive outage this past week – at least most people would call it an outage. Pearson Vue’s spin team tried to say their systems were 100% up, only slow, but countless posts online contradict this.

The issues were first acknowledged on the company’s Facebook page.

An entire day goes by and they claim the issue is fixed.

But shortly thereafter, another acknowledgement of an ongoing problem.

On April 24th, another acknowledgement of a problem.

A second generic post again on April 24th.

I first learned of this outage when I walked into a Vue testing center for an exam on April 24, only to discover that they were unable to deliver because Vue’s servers were not accessible. The center called in to Vue, and customer service said all their systems were frozen and nothing could be done.

Pearson Vue put up another April 24th post suggesting that users try scheduling during non-peak hours.

On April 25th came the first of many outright lies posted by Pearson Vue.

This leads you to believe the system is up but slow. This was not the case. I tried many times to log in without success, as did others such as this Facebook poster.

Here is the rest of the FAQ from April 25th

I called multiple times, only to be told by customer service that they could not log in. This happened to people worldwide, here are a few of the many posts on Facebook.

Testing centers were not able to deliver exams, either.

Later in the day on April 25th came a post with another outright lie saying “our systems are operational, just not optimal”

That post prompted me to post the following, which was not replied to or acknowledged in any way.

On April 26th, a series of posts came out saying that engineers had found the problem and they were bringing the system back to expected performance levels.

A Facebook post directly under the above message shows a user who still can’t schedule an exam using customer service – the timestamp on this is April 28th, 9:30AM CDT.

On April 28th at 10:30 AM CDT, Pearson Vue had the audacity to ask users to stress test the system for them.

The user impact of this outage has been massive. It was more of an inconvenience for me. But for others, there were signifiant impacts in time, expense, and even their ability to work.

Here is one Facebook post from a user who has no Pearson Vue facility in their country. They have to get a visa to leave the country to sit an exam. In order to get a visa, they have to make an appointment with their embassy. Once they get their appointment, they have to register for the exam and bring printed confirmation. Unable to register for over a week, this user loses the embassy appointment.

Did Vue suffer data loss on top of the outage?

A user needing to test for starting a job next week.

I know for a fact that I saw dozens more posts with similar problems – physicians unable to go to board exams, nurses unable to work because of results delays. I wish I had thought to screencap more while this was going on, but I didn’t. It appears that those posts were either eaten up by Facebook (yeah right) or deleted by Pearson (likely, but can never be proven).  At least one user wrote a post confirming post removal. None of my posts were deleted.

As an IT professional, I find this outage appalling. The company states this was started by an upgrade. Every place I’ve ever worked at upgrades during off hours and rolls back on failure. Pearson deployed a faulty upgrade then forced its users to pay the price while programmers scrambled desperately to fix their poorly written code. Pearson Vue’s suggestion that they carefully planned and tested their upgrades is nonsense. A proper load test reveals these kinds of failures. Their post from today ‘inviting’ us to load test their fixed system also points to the fact that they are unable or unwilling to test their own systems.

The fact that this upgrade caused a global outage for both scheduling and test delivery demonstraties critical failures of their architecture. Are the same webservers used for scheduling also used for exam delivery? Could a breach of vue.com could then result in the theft of exam content?  Or instead are they separate servers connected to the same backend database? In any event, their architecture is an abomination. The global failure to deliver exams points to only two possibilities. Do all global Vue delivery centers connect to the same datacenter, meaning the ability to deliver exams globally relies on a single point of failure? If so, this is a catastrophic design flaw. If they do have datacenter redundancy, then they deployed their upgrade across the entire system at the same time. This demonstrates atrocious planning. Why would you risk multiple datacenters with the same upgrade?

Pearson Vue is a billion dollar, Fortune 500 corporation. This kind of an outage is both unacceptable and inexcusable. Considering the power that Vue wields over people’s careers, it’s frightening to witness the depth of ineptitude demonstrated in this disaster.

# Moving PVS VMs from e1000 to VMXNET3 network adapter

A client needed to remove the e1000 NIC from all VMs in a PVS pool and replace it with the VMXNET3 adapter. PVS VMs are registered by MAC address – replacing the NIC means a new MAC, and PVS has to be updated to allow the VM to boot.

I needed a script to remove the old e1000 NIC, add a new VMXNET3 NIC, and register the new NIC’s MAC with PVS. I knew I would easily accomplish the VM changes with PowerCLI, but I didn’t know what options there were with Citrix. I found what I needed in MCLIPSSNapin, a PowerShell snap-in installed on all PVS servers. The snap-in gives you Powershell control over just about anything you need to do on a PVS server.

I didn’t want to install PowerCLI on the production PVS servers, and I didn’t want to install PVS somewhere else or try manually copying files over. I decided I needed one script to swap out the NICs and dump a list of VMs and MAC address to a text file. Then a second script to read the text file and make the PVS changes.

First, the PowerCLI script. We put the desktop pool into maintenance mode with all desktops shut down. It takes about 10 seconds per VM to execute this script.

Param(
[switch] $WhatIf , [switch]$IgnoreErrors
,
[ValidateSet("e1000","vmxnet3")]
[string]
$NICToReplace = "e1000" ) # vCenter folder containing the VMs to update$FOLDER_NAME = "YourFolder"

# vCenter Name
$VCENTER_NAME = "YourvCenter" #The portgroup that the replacement NIC will be connected to$VLAN_NAME = "VLAN10"

#If you want all VMs in $FOLDER_NAME, leave$VMFilter empty. Otherwise, set it to a pipe-delimited list of VM names
$VMFilter = "" #$VMFilter = "DESKTOP001|DESKTOP002"

$LOG_FILE_NAME = "debug.log" Connect-VIServer$VCENTER_NAME

$NICToSet = "e1000" if ($NICToReplace -eq "e1000" )
{
$NICToSet = "vmxnet3" } elseif ($NICToReplace -eq "vmxnet3" )
{
$NICTOSet = "e1000" } function LogThis { Param([string]$LogText,
[string] $color = "Gray") Process { write-host -ForegroundColor$color $LogText Add-Content -Path$LOG_FILE_NAME $LogText } } if ( Test-Path$LOG_FILE_NAME )
{
Remove-Item $LOG_FILE_NAME }$errStatus = $false$warnStatus = $false$msg = ""

if ( $VMFilter.Length -eq 0 ) {$vms = Get-Folder $FOLDER_NAME | Get-VM } else {$vms = Get-Folder $FOLDER_NAME | Get-VM | Where{$_.Name -match $VMFilter } } foreach ($vm in $vms) {$vm.Name
$msg = "" if ($vm.NetworkAdapters[0] -eq $null ) {$errStatus = $true$msg = "No NIC found on " + $vm.Name LogThis$msg "Red"

}
else
{
if ( ($vm.NetworkAdapters | Measure-Object).Count -gt 1) {$errStatus = $true msg = "Multiple NICs found on " +$vm.Name
LogThis $msg "Red" } else { if ($vm.NetworkAdapters[0].type -ne $NICToReplace ) {$warnStatus = $true$msg = "NIC is not " + $NICToReplace + ", found" +$vm.NetworkAdapters[0].type + " on " + $vm.Name LogThis$msg "Yellow"
}

LogThis $vm.Name,$vm.NetworkAdapters[0].MacAddress

}

}

}

if ( $errStatus =$true -and $IgnoreErrors -ne$true)
{
LogThis "Errors found, please correct and rerun the script." "Red"

}
else
{
if ( $warnStatus =$true )
{
LogThis "Warnings were found, continuing." "Yellow"
}
foreach ( $vm in$vms )
{
if ( $WhatIf -eq$true )
{
$msg = "Whatif switch enabled, would have added " +$NICToSet + " NIC to " + $vm.Name LogThis$msg
}
else
{
$vm.NetworkAdapters[0] | Remove-NetworkAdapter -confirm:$false
$vm | New-NetworkAdapter -NetworkName$VLAN_NAME -StartConnected -Type $NICToSet -confirm:$false
}
}

if ( $VMFilter.Length -eq 0 ) {$vms = Get-Folder $FOLDER_NAME | Get-VM } else {$vms = Get-Folder $FOLDER_NAME | Get-VM | Where{$_.Name -match $VMFilter } } LogThis("Replaced MAC addresses:") foreach ($vm in $vms ) { LogThis$vm.Name,$vm.NetworkAdapters[0].MacAddress } }  The script offers a -Whatif switch so you can run it in test mode without actually replacing the NIC. It writes all its output to$LOG_FILE_NAME. First it logs the VMs with their old MAC, then the replaced MAC. The output looks something like this:
VD0001 00:50:56:90:00:0a
VD0002 00:50:56:90:00:0b
VD0003 00:50:56:90:00:0c
VD0004 00:50:56:b8:00:0d
VD0005 00:50:56:b8:00:0e
VD0001 00:50:56:90:57:1b
VD0002 00:50:56:90:57:1c
VD0003 00:50:56:90:57:1d
VD0004 00:50:56:90:57:1e
VD0005 00:50:56:90:57:1f

Scan the logfile for any problems in the top section. The data after “Replaced MAC addresses:” is what the PVS server needs. Copy this over to the PVS host. Now we need to use MCLIPSSnapin, but first we have to register the DLL. I followed this Citrix blog for syntax:
“C:\Windows\Microsoft.NET\Framework64\v2.0.50727\installutil.exe” “C:\Program Files\Citrix\Provisioning Services Console\McliPSSnapIn.dll”

I copied the VM names and new MAC addresses to a text file vmlist.txt and put it on my PVS server, in the same folder as the following PowerShell script. It runs very quickly, it takes only a few seconds even if you are updating hundreds of VMs.

Add-PSSnapIn mclipssnapin
$vmlist = get-content "vmlist.txt" foreach ($row in $vmlist) {$vmname=$row.Split(" ")[0]$macaddress=$row.Split(" ")[1]$vmname
$macaddress Mcli-Set Device –p devicename=$vmname –r devicemac=$macaddress }  Now, replace the PVS pool’s image with one that is prepared for a VMXNET3 adapter and boot the pool. Migration complete! # Is It Time To Remove the VCP Class Requirement – Rebuttal This post is a rebuttal of @networkingnerd‘s blog post Is It Time To Remove the VCP Class Requirement. I would like to acknowledge that it’s easy for me to have the perspective I do as a VCP holder since version 3. I’ve already got it, so I naturally want it to remain valuable. The fact that my employer at the time paid for the class has opened up an entire career path for me that would have otherwise been closed. But I believe the VCP cert remains fairly elite specifically because of the course requirement. First, consider Microsoft’s certifications. As a 15-year veteran of the IT industry, I believe I am qualified to state unequivocally that Microsoft certifications are utterly worthless. This is partially because of the proliferation of braindumps. There is no knowledge requirement whatsover to sit the Microsoft exams. You don’t even need to look at a Microsoft product to pass a Microsoft test – go memorize a braindump and pass the test. We’ve all encountered paper MCSEs – their existence completely devalues the certification. I consider the MCSE nothing more than a little checkbox on some recruiter’s wish list. I would go so far as to say that Microsoft’s test are specifically geared towards memorizers – they acutally encourage braindumping by focusing on irrelevant details and not on core skills. Passing a Microsoft exam has everything to do with memorization and almost nothing to do with your skill as a Windows admin. On the other hand, to sit the VCP exam you have to go through a week of training. At the very least, you’ve touched the software. You installed it. You configured it. You (wait for it)… managed it. Obviously there are braindumps out there for the VCP exam too, but everybody starts with the same core of knowledge. The VCP exams have improved to a point where they are not memorize-and-regurgitate. A person who has worked with the product actually stands a reasonable chance of passing the exam. Quoted directly from the blog post: For those that say that not taking the class devalues the cert, ask yourself one question. Why does VMware only require the class for new VCPs? Why are VCPs in good standing allowed to take the test with no class requirement and get certified on a new version? If all the value is in the class, then all VCPs should be required to take a What’s New class before they can get upgraded. If the value is truly in the class, no one should be exempt from taking it. For most VCPs, this is not a pleasant thought. Many that I talked to said, “But I’ve already paid to go to the class. Why should I pay again?” This just speaks to my point that the value isn’t in the class, it’s in the knowledge. Besides VMware Education, who cares where people acquire the knowledge and experience? Isn’t a home lab just as good as the ones that VMware built. There is a tiny window of opportunity after the release of new vSphere edition to go take the exam without a course requirement. Those of us who are able to pass the exam in that small window are the people who do exactly as you say – we are downloading and installing the software in our labs. We are putting in the time to make sure that our knowledge of the newest features is up to par. Many of us partipate in alpha and beta programs, spending far more time with the software than a typical certification candidate. Some of us participate in the certification beta program, where we have just a couple of short weeks to study for and book the exam. I’ve put in quite a few late nights prepping for beta exams. VMware forces us to learn the new features by putting a time limit on the upgrade period. We have a foundation of knowledge that was created by the original class that we took. There isn’t enough time for braindumps to leak out there, and the vast majority of upgraders wouldn’t use one anyhow. VMware can be reasonably certain that a VCP upgrader without the class really is taking the time to learn the new features. @networkingnerd is correct, the value IS in the knowledge, but the focus is ensuring that every VCP candidate starts with the same core of knowledge. @networkingnerd suggests an alternative lower level certification such as a VCA with a much less expensive course requirement. I think it’s an interesting idea, but I don’t know how you’d put it into practice. I’m not sure what a 1-day class could prepare you for. It’s one thing for experienced vSphere admins to attend a 2-day What’s New class. But what could you really teach and test on? Just installing vSphere? There’s not a whole lot of value for an engineer who can install but not configure. Again quoting from the article: Employers don’t see the return on investment for a$3,000US class, especially if the person that they are going to send already has the knowledge shared in the class. That barrier to entry is causing VMware to lose out on the visbility that having a lot of VCPs can bring.

This suggests that the entry-level certification from the leader in virtualization is somehow not well-known. While I would agree that the VCAP-level certifications do not enjoy the same level of name recognition as the CCNP, I talk to seniors in college who know what the VCP is. There is no lack of awareness of the VCP certification. I also agree that it’s ridiculous to send a VMware admin who has years of experience to the Install Configure Manage class. That’s why the Optimize and Scale and the Fast Track classes exist.

I don’t believe dropping the course requirement would do anything to enhance VMware’s market share. The number of VCP individuals has long since reached a critical mass.  Nobody is going to avoid buying vSphere because of a lack of VCPs qualified to administer the environment. While I agree that Hyper-V poses a credible threat, Microsoft is just now shipping features that vSphere has had for years. Hyper-V will start to capture the SMB market, but it will be a long time before it has the chance to unseat vSphere in the enterprise.

# My EMC E10-001 exam experience

I passed the EMC E10-001 exam on Thursday, giving me the EMCISA certification. My primary study material was the insanely long-titled Information Storage and Management: Storing, Managing, and Protecting Digital Information in Classic, Virtualized, and Cloud Environments. I went with the Kindle edition to save money as well as the ability to read the book anywhere – phone, laptop, or Kindle.

Everything you need to know for exam success is in this book. I found the first few chapters to be basic IT skills review, but it soon moved into material that I was hazy on - FCoE, object-based storage, and array-based replication technology. It’s a fairly lengthy read at over 500 pages, but I went through the book once, did my Kindle highlighting, then headed over to the practice test.

The practice test lets you know what you got wrong, but not the correct answer. I went back and studied up on the ones that I got wrong. When I got a 90% on the practice test, I booked the exam. Since I’d never taken an EMC exam, I wasn’t sure whether I’d get Microsoft-style pure memorization questions, or Cisco-style conceptual questions. I was pleased to find the EMC questions were more like a Cisco exam. I found the questions to be straightforward, no material beyond what is covered in the book and no trick questions.

With the EMCISA certification knocked out, the path is now clear to pursue a Specialist-level EMC certification.

# My MCSA 2012 upgrade exam experience – MS 70-417

I passed the Microsoft 70-417 exam today; it wasn’t my first attempt. I don’t pursue many Microsoft certifications because I find them to be of questionable value. The use of braindumps is rampant in IT certification, particularly true in the case of Microsoft exams. I feel this is in part due to the ridculous nature of most of the questions. There are always going to be people who simply don’t want to learn, the so-called “paper MCSEs”. But unlike exams from other vendors such as VMware and Cisco, experience with Microsoft’s products has nothing to do with passing their exams. Microsoft’s inability to write decent exam questions is truly baffling to me.

I had many questions that forced me to try and remember where in the GUI certain options were. As an example, think about the advanced NTFS permission list. Is “Create files / write data” before or after “Create folders / append data” in the permission list? You probably don’t know and you surely don’t care. If you do happen to know, it doesn’t make you a better administrator. It doesn’t demonstrate much of anything other than a photographic memory. The 417 exam is loaded with these kinds of worthless questions. Even worse is trying to remember a particular switch for a PowerShell command – that’s why cmdlets are self-documenting.

The 70-417 exam is divided into 3 parts, the 70-410 (Install and Configure), the 70-411 (Administering) and the 70-412 (Advanced). My exam had 61 questions and you have to take each section invidually. Once you complete a section, there is no going back. I would never have passed this exam without J.C. Mackin’s Exam Ref 70-417: Upgrading Your Skills to MCSA Windows Server® 2012 and Timothy Warner’s 70-417 videos on CBTnuggets.

I’m normally proud and excited when I pass an exam, but not this time. I mostly feel anger toward the Microsoft Certification team for wasting my time. The exam hasn’t measured my actual skill with Server 2012, all it’s done is measure how well I can remember trivia. I’m happy to have this one behind me. I doubt I’ll look at any more Microsoft exams until this MCSA expires.

# Windows 2012 DHCP – Migration and Fault Tolerance

A long overdue feature in Windows Server 2012 is the ability to have a truly distributed DHCP infrastructure. You used to have to use the 80/20 split scope configuration which was at best administratively burdensome, and at worst unmanageable for a large enterprise. Alternatively, you’d have to run a Windows cluster for DHCP – who actually wants to do that? Windows Server 2012 now offers distributed DHCP failover.

I’ll start by talking about migrating your DHCP configuration from Windows 2008 to Windows 2012. First, you obviously have to install Windows 2012 and add the DHCP role. Microsoft has made it ridiculously easy to export and import the configuration with PowerShell – you can complete the process with 2 commands.

Open a PowerShell window on your 2012 server and run
Export-DhcpServer -ComputerName your2008DHCPserver.foo.com -Leases -File C:\path\to\exportfilename.xml -verbose

Exporting Windows 2008 DHCP configuration

A bit of advice from my lab environment – if you have a clock mismatch between domain controllers, you won’t be able to run these remote Powershell commands. If you’re getting strange errors in your lab, check the clocks on your DCs. My 2012 DC was many hours behind my 2008 DC and I couldn’t run any commands. I fixed the clock and the problem disappeared.

When your export is complete, run the import.
Import-Dhcpserver -Leases -File C:\path\to\exportfilename.xml -BackupPath C:\windows\temp\ -verbose

Importing DHCP configuration from a Windows 2008 export

Import of 2008 DHCP configuration completed.

The 2008 imported scopes on the 2012 DHCP server.

Moving on to configuring the failover -

Configuring failover on a DHCP scope

Not much to see here, but if you don’t have any scopes visible you aren’t going to be able to go much farther. One possible solution to the problem is in this post.

First screen of the failover wizard

Specify another Windows 2012 server with the DHCP role already installed.

Specifying a failover partner in the failover wizard

The failover relationship must be named. Maximum client lead time refers to amount of time the surviving DHCP host will wait before assuming complete control over the scopes. For mode, you can pick either load balanced or hot standby – basically an active-active or active-passive relationship.

Defining the failover relationship

Finish will kick off the process.

Last screen of the failover wizard

Failover progresses to completion

The scope and leases are now visible from the secondary DHCP server.

The failover scope as seen on the secondary DHCP server

I will now shut down the primary DHCP host and leave the secondary host online. You can see my Linux guest with the DHCP address of 192.168.237.201.  After shutting down primary DHCP, I reboot this Linux guest.

Client IP prior to reboot

The guest comes back with the same .201 IP address, this time served by the secondary DHCP host.

Client IP after reboot – DHCP served by failover host.

Now I boot a second Linux guest and it draws a .220 IP address from the secondary DHCP host.

A second client VM booted with primary DHCP server disabled

Here are the leases as seen from the secondary DHCP host. Both of the Linux hostnames are the same because I didn’t bother customizing them in the guest. However, the MAC addresses are different and each VM is drawing the correct IP.

DHCP leases on failover host

I now bring the primary DHCP host back up. Immediately after boot, you can see that the lease for the .220 IP is not listed.

DHCP leases on DC2 after powerup

We could wait for the hosts to do this themselves, but we’re impatient and force a replication.

Forcing scope replication to pull leases from failover host

Now the .220 lease is visible from the primary host.

DHCP leases on primary host

Now we take down the secondary DHCP host, leaving only the primary online. Reboot both of the Linux guests and they both correctly draw the .201 and .220 IP address.

DHCP IP on client 1 after reboot

DHCP IP on client 2 after reboot

Once you have this set up you can configure your network infrastructure with multiple DHCP addresses. Vendors have various names, but Cisco refers to this as the ip helper address. DHCP requests will always be sent to both DHCP hosts, but each scope is active on one host at a time. One DHCP request will be acknowledged and the other will be ignored.

# VMware View Composer starts, but does no work.

I worked on a client outage over the weekend, Virutal Center and View Composer were down. It started with a disk full situation on the SQL server hosting the vCenter, Composer, and Events databases. The client was shut down for winter break, so the Composer outage was not noticed for several days. After fixing the SQL Server disk space problem, everything came back up. I was able to restart all services and they appeared to be running. Composer started without issue, but it didn’t respond to any commands – any operations I requested in View Manager were ignored. I didn’t find any obvious errors in the logs.

I ran through the troubleshooting options in KB1030698 without finding any issues. I validated the SDK was responding by going to https://vcenteripaddress/sdk/vimService.wsdl . I couldn’t find any cause for the outage, so I opened up a Sev-1 ticket with VMware Support.

The support tech concluded that a problem with the ADAM database was preventing Composer from doing the job. He had me shut down all but one connection broker, then restart the View services on the remaining broker. At this point, commands issued on the broker were obeyed by Composer. We deleted or refreshed all of the desktops listed under Problem Desktops. Once we were sure that the ADAM database reflected the true state of the environment as reflected in vCenter, we restarted the other brokers. They synced databases and the problem was resolved.

# Windows 2012 DHCP Failover – Cannot enable

I tried setting up the new Windows 2012 DHCP failover and got it working, but I couldn’t get some of my scopes that I’d imported from 2008 to 2012 working. If I built the same scope (or what I THOUGHT was the same scope) from scratch in 2012, I could configure failover.

I posted in the Microsoft forums but didn’t find resolution. However, one month later somebody posted the solution on my thread. My 2008 scopes happened to have both DHCP and BOOTP enabled, and you can’t enable failover on a scope set to both DHCP and BOOTP.