I’ve made a couple of minor corrections to the networking section of the vReference Card:

Port groups per vSS = 256

Hosts per vDS = 350

Thanks to FerFables, Daniel M, Rene Reitinger, and Alex for pointing out these two typos!

As usual, you can grab the latest copy over here.

I think I might have stumbled across an interesting design conflict.

UCS Boot from iSCSI SAN support

Cisco UCS manager 2.0  now offers the ability for their blade servers to boot from iSCSI SAN.  In the release notes it states:

iSCSI Boot - iSCSI boot enables a server to boot its operating system from an iSCSI target machine located remotely over a network.

Sounds good to me. I know a lot of blade aficionados were looking forward to this addition, as Boot from SAN and Blades are a popular combination. Digging a little deeper, it appears that during the install of non-Windows OSes the NICs offers an iBFT setup, which to me indicates they are considered “Dependent HW NICs” in VMware parlance. The adapters are configured with iSCSI settings in card’s firmware and handle some offload, but they are more similar to SW initiators than outright iSCSI HBAs in that the VMkernel is still responsible for most of the day-to-day storage traffic. From the latest UCS Manger 2.0 Configuration Guide, page 392 states:

The iBFT works at the OS installation software level and might not work with HBA mode (also known as TCP offload). Whether iBFT works with HBA mode depends on the OS capabilities during installation.

followed on page 393 by:

only Windows OS supports HBA mode during installation

VMware ESXi 5.0 boot from iSCSI SAN support

Now we flick over to the VMware vSphere 5.0 Storage Guide, on page 100:

With independent hardware iSCSI only, you can place the diagnostic partition on the boot LUN. If you configure the diagnostic partition in the boot LUN, this LUN cannot be shared across multiple hosts. If a separate LUN is used for the diagnostic partition, it can be shared by multiple hosts. If you boot from SAN using iBFT, you cannot set up a diagnostic partition on a SAN LUN.

VMware vSphere 5.0 Dump Collector

Lastly we need to refer to VMware KB article 2000781 regarding support of its Dump Collector tool which states:

The vSphere ESXi 5.0 Network Dump Collector feature is supported only with Standard vSwitches and cannot be used on a VMkernel network interface connected to a vSphere Distributed Switch or Cisco Nexus 1000 Switch.

Do the hokey cokey and turn around

So still following along? I’ll join the dots now so you can see where I’m going with all this… Once you’ve installed your shiny new UCS chassis and blades, you see the freshly-released boot from iSCSI SAN support and decide to install the latest and greatest ESXi 5.0 as your hypervisor of choice.  Unfortunately VMware doesn’t create a diagnostic partition during the install because ESXi sees the iSCSI adapter using iBFT. No problems you think, you can setup the new centralized Dump Collector to make sure those diagnostic dumps don’t get lost during a kernel panic.  Bang, but you’re using a Distributed Switch – uh oh spaghettios.  Let’s face it, I would think it’s very likely that the sort of datacenter that uses Cisco UCS blades, and the sort of environment that would consider Boot from SAN ESXi installs, are likely to be using vDS or 1000v switches in their configuration.

Now I realize that not having a diagnostic partition is not the end of the world. You can still install and run ESXi fine without it.  However, if you are using UCS with ESXi and were thinking about Boot from iSCSI as an option, then you should realize that your likely not capturing the kernel dumps. I’m sure that is not what most folk expect. Just a curious design quirk that might be useful to highlight.

Design Workaround

There is a design workaround for this.  You could create a separate 110MB partition on each blade’s local disk and redirect the dumps there. But that kinda defeats the point doesn’t it?  Or you could use a shared SAN LUN and point all your hosts there. Just remember to be quick and grab that dump immediately after a crash or the next host crash will overwrite it. Not great options I agree, but if you *really* want to go this way…

Or take Gabe’s advice and go with Auto Deploy.

Background

VMware have recommended for quite some time that we stick to multicast when configuring NLB (MS’s Network Load Balancing) where possible:

VMware recommends that you use multicast mode, because unicast mode forces the physical switches on the LAN to broadcast all Network Load Balancing traffic to every machine on the LAN.

If you need to use unicast, then to prevent port flooding you should change the Port Group’s “Notify Switches” policy to No - the default being Yes.

Windows 2008 R2 Failover Clustering

According to this white paper from Microsoft,

Multicast functionality has been discontinued in Windows Server 2008 failover clustering, and cluster communications now use User Datagram Protocol (UDP) unicast.

So Microsoft clustering gurus, does this mean for Window 2008 R2 Failover Clusters we should also change the “Notify Switches” policy off? Is the recommended setting for MS NLB clustering now applicable to MS’s latest version of MSCS?

I’ve been working away on both the ESXi Host and ESXi Install sections for the vReference card, and I came across something I found interesting about the all new Auto Deploy tool. Here’s a quote from the penultimate paragraph on Page 68 of the current Installation and Setup Guide PDF for vSphere 5:

If the vCenter Server system is unavailable, the host contacts the Auto Deploy server for image profiles and host profiles and the host reboots. However, Auto Deploy cannot set up vSphere distributed switches if vCenter Server is unavailable, and virtual machines are assigned to hosts only if they participate in an HA cluster. Until the host is reconnected to vCenter Server and the host profile is applied, the switch cannot be created and, because the host is in maintenance mode, virtual machines cannot start.

So if you are running a fully virtualized environment, and planning to use Auto Deploy to build and configure all the hosts via Image Profiles and Host Profiles, then you need think twice about the design. Imagine you were ever faced with a complete power outage in your datacenter. Now in this day and age, you’d hope that this never happens. However, considering the number of complete outages I’ve seen at sites, I know I wouldn’t bet my job against it never happening.

So here’s the scenario. Everything powers off, all at once. You hit the power button on the servers. The hosts boot up, but stay in Maintenance Mode because they can’t hit the vCenter VM or Auto Deploy VM for their Host Profile. In Maintenance Mode the VMs won’t power on. The vDS switch cannot be created. You can’t power on your vCenter VM. You can’t power on your Auto Deploy VM.

Now, I’m not saying that you couldn’t get out of this situation if you knew what you were doing. Presumably you could recreate some Standard vSwitches from the ESXi Shell and force the host out of Maintenance Mode. And through good prior planning you’d already pinned your vCenter VM to a set host so you knew which one to start working on.

So how do you design around this? A physical server, a separate management cluster, a remote secondary Auto Deploy instance, …

This is certainly something to consider carefully before jumping into a full-scale Auto Deploy rollout.

Update: Michael Webster (AKA @vcdxnz001) just sent in the following addtional Auto Deploy design consideration. vShield App isn’t supported with Auto Deploy.

Update 2: VMware has released a new video-based technical note explaining how to build a Highly Available Auto Deploy Infrastructure. Their recommended path is to create a separate management cluster in which the hosts are not deployed via Auto Deploy.  In the video, they call-out the following services as important to segragate:

Infrastructure VMs

• vCenter
• Active Directory
• DNS

PXE Boot Infrastructure

• TFTP
• DHCP

Auto Deploy Environment

• PowerCLI
• Auto Deploy
• vCenter

Highly Available Auto Deploy Infrastructure