I’ve recently been thinking about the practicalities of PXE booting ESXi servers.  Sounds great, but how do you make this work in a typical environment?

Using trunked connections on ESXi hosts is very much common place.  It’s likely that your ESXi’s Management Network connection, which by default will be your first onboard NIC (vmnic0), is connected to a trunked uplink switch port.  Probably the most popular configuration is bonding your Management Network with your vMotion vmknic on a vSwitch with two trunk uplinks which includes vmnic0.  The drive towards 10GbE and cable consolidation only increases the likelihood that your vmnic0 will patched into a trunked port.

VMware are starting to pursue solutions using servers’ ability to PXE boot.  The potential to PXE boot into an installation routine is not a new concept.  VMware’s AutoDeploy and the recently announced PXE Manager fling, uses this technique. In fact not only PXE booting the install, but actually PXE booting the OS itself via the network, or stateless as it is being referred to (although this term really defines something specific, not just PXE booting).

The question comes – how do I PXE boot my servers which are connected to trunked interfaces on the switch?  If your servers are physically connected to a trunked connection, then a standard PXE boot won’t tag the traffic appropriately (tell me if I’m wrong – is this something you can set in a server BIOS these days?)  You don’t want to re-patch a server’s network cables if you have to quickly rebuild it.  Or if you are PXE booting (stateless) then you’d have to do this for each reboot.  And you don’t want to trouble your Network Admin to change it back to an access port every time.

This is where I think Native VLANs can help out.  As a vSphere server guy, what I know about Native VLANs is VMware’s advice that you avoid tagging traffic with VLAN 1, because this is what Cisco set as the default Native VLAN for switches. When thinking about VLAN IDs for your trunked ESXi ports, you just choose something other than 1.  But Native VLANs could provide a solution to the problem of PXE booting on trunks.

If the interface for your vmnic0 has a Native VLAN, then when the server tries to PXE boot, it can get out onto the network.  If untagged traffic is being received on a switch’s trunked interface, then it will assume it is for that interface’s Native VLAN.  You could have the Native VLAN set as the same VLAN as your Management Network subnet.  Then it will PXE boot straight on to the same subnet that it will get once the Management Network is brought up.  Alternatively, if you only want to PXE boot into an installer, you could set your Native VLAN to a special build subnet.  Once the server is built, then the Management Network traffic is tagged back on to your regular trunked VLAN.

So what do you think?  Feasible, secure enough, any potential issues? Or do you have other ways you set this up in your environment that you can recommend to everyone?