Too frequently I hear the expression n+1 as a model for ESX clusters to provide High Availability. If you EVER expect to patch ESX servers without VM downtime then you need at least(†) n+2. When running your clusters to only n+1, you can never safely put one of your hosts in Maintenance Mode; not if High Availability is important to you.
Footnote: If you don’t understand the importance of HA slot sizes, go learn.
The default GUI install of ESX4 makes the /tmp partition 1GB and even then it is only categorized as optional. I’ve been asked several times why you’d want to make /tmp any bigger. If it fills up you just clear it out, right?
Well here’s a good reason. It seems that VUM (vCenter Update Manager) uses /tmp. When you stage updates, VUM copies all the patches to the folder /tmp/updatecache. It does the sensible thing and checks that there is enough space first, but if it can’t then it tries to create a ramdisk. I don’t think I’m that keen on my server’s ram being tied up with patches. Sometimes you might want to stage the patches days in advance of an outage. I’d hope that the ESX is clever enough to dump the ramdisk if there was any sort of memory contention, but still.
Anyway, with ESX3 I know the patches could accumulate to quite a size (a couple of GBs if you left them a few months). I hear ESX4 is better in this regard, however I would suggest keeping at least 2GB for /tmp during the install.
VUM isn’t a crucial service. You can always manually copy patches to a different partition, but VUM (especially the new staging feature) is a real time-saver so I know I’ll be making sure there is plenty of space in /tmp.
Dudley passed me the latest version of the Firewall diagram. Go and grab it:
Here’s what’s new since the last download:
What’s new in v3:
Now synchronized with “VMware Network Ports Compendium v3″
What’s new in v213:
Change port range in VUM to 9000-9100 (and not 9000-9010)
What’s new in v212:
Added SRM Port 9007 for WSDL, SOAL
Changed SRM Port 443 to Port 80 for Communication with remote vCenter Server
He’s also given me access to his “source” document. It’s a spreadsheet which makes looking for a specific port when troubleshooting much easier. You can grab yourself a copy here:
Dudley maintains on online version here: http://webbrain.com/brainpage/brain/89EFA582-2C35-F6A2-9ED1-7AD4810266C2/
Here’s a strange one I’ve come across in vCenter 2.5. You have a user, who is a member of an AD group, which has been assigned the Administrator role in vCenter over a Datacenter (or a folder, cluster, host, …) – but not at the root level. Got that? That user can do everything that you would expect an administrator to be able to, at that level.
However once that user generates tasks in vCenter, it seems they can’t cancel them. From what I’ve found, canceling tasks seems to be a Global permission and is only allowed if your administrative permissions are set at the top of the tree. Even though the task was created by them and is a task within their Datacenter.
Has anyone else seen this and come to a sensible work around? Does it happen in vCenter 4.0?
Configuring MSCS (MicroSoft Clustering Service) in the VMware world is a complicated process. I’ve setup many MSCS solutions on VMware, and I still cringe when a customer demands it as a solution. It works, but every time I do it there are always so many little challenges.
I’ll try to describe what creates the most common misunderstanding, as best I see it. Keep in mind, this advice is for ESX 3.x. I haven’t looked too closely at how vSphere4 handles it, but I don’t think it’s that different. Also, I’m very willing to be corrected if you think I’m misrepresenting things.
There are 2 different settings, which sound very similar:
They are distinct, and just because you chose a Virtual RDM, doesn’t mean the SCSI controller should necessarily be set to Virtual .
Let’s deal with the disks first. I stand by the table on my reference card. The critical deciding factors are the host configuration, need for snapshots and if you need SCSI target software to run. The hosts can either be:
Now the SCSI bus sharing setting is different. It often gets missed, because you don’t manually add the second controller (in fact you can’t). You need to go back to the settings after you have added the first shared disk. There are 3 settings here:
So, the problem can really lie in two areas:
Here is the latest 3.5 PDF for MSCS:
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_mscs.pdf
Add to the mix, you need to understand Boot from SAN, Independent disks, Persistent/Nonpersistent, VMDK disk types, e.g. eagerzeroedthick & additional SCSI controllers. And its always changing; back in the days of ESX2, they called things pass-though and non-pass-through RDMs. This is just to setup the hardware, wait until you have to configure the disks and cluster!
It’s definitely a rats nest, but I don’t blame VMware. MSCS is a fairly complex beast, and is very touchy when it comes to its shared storage. I’m sure VMware provided MSCS because its customers demanded it, but you can tell they certainly don’t want to promote its use. Hopefully, the new Fault Tolerance features will draw most architects away from MSCS.
According to the vSphere 4.0 release notes (http://www.vmware.com/support/vsphere4/doc/vsp_esx40_vc40_rel_notes.html )
vCenter Server 4.0 becomes unresponsive in large environments if managing ESX Server 3.5 hosts prior to ESX 3.5 patch 10
vCenter Server 4.0 can become unresponsive in large environments after 30 days if it manages any ESX Server 3.5 hosts prior to ESX Server 3.5 patch 10.
Workaround: Upgrade to ESX Server 3.5 Update 4 if you are running ESX Server 3.5 with vCenter Server 4.0.
What exactly is “patch 10”? How do you know what build level qualifies? Is there a single patch that can be applied?
If you upgrade to vCenter 4 and things become unresponsive after 30 days, patching all your disparate ESX servers isn’t going to be something that most large organizations can do quickly. I’d say this is an important one.
I posed the question to my local SE, and several VMware contacts I have, but no-one seems to know. Can anyone out there on the tubes clarify this one?
I had a another problem with Storage VMotion yesterday and found out a couple of interesting things.
Firstly, there is a now a Knowledge Base article explaining how to recreate vmdk header files if they are missing. This was news to me, so here’s the link:
http://kb.vmware.com/kb/1004232
Secondly, was discovering a sightly different approach to recovering a failed Storage VMotion (DMotion). My previous experiences had involved something along these lines:
http://communities.vmware.com/message/999890#999890
Which basically breaks down to creating another snapshot, so that you will then be able to force a vmware-cmd …/vmname.vmx removesnapshots
However this approach was messy, didn’t always commit properly and required editing the vmx file.
So yesterday, when facing a similar problem, I saw it resolved in slightly different way. We started by checking the linking of parentCID to CID in the vmdk header files, as we had an issue with all the different DMotion snapshot files. Then to commit the snapshots, we used vmkfstools -i <last_snapshot.vmdk> <destination.vmdk> to clone the disk to another file. By sending the clone command to the last snapshot header file, it knew to roll all the chained snapshots, along with the original disk, into this new copy.
Obviously this method requires extra space for the second copy and can take longer, but you have the advantage that the original are untouched.
Update:
VMware has just released a new KB article covering the whole process: http://kb.vmware.com/kb/1007849
Here’s an old trick which works great on ESX servers (thanks to this article on Linux Journal for reminding me). It turns your prompt different colours to highlight when you are logged in as root.
To make the prompt red when you’re running as root add this to /root/.bashrc:
PS1=’\[\e[31m\]\u@\h:\w#\[\e[m\] ‘
To make the prompt green when running as a normal user add this to ~/.bashrc:
PS1=’\[\e[32m\]\u@\h:\w\$\[\e[m\] ‘
Update: If you want to add this to a kickstart script, do this:
# Help identify when logged in as root
echo “PS1=’\[\e[31m\]\u@\h:\w#\[\e[m\]‘” >> /root/.bashrc
echo “PS1=’\[\e[32m\]\u@\h:\w#\[\e[m\]‘” >> /etc/skel/.bashrc
I wanted to blog here about a silly mistake I made yesterday.
I need to expand a VM’s disk and was using the ol’ vmkfstools -X. However, I obviously do this all too flippantly and forgot to check if there was any snapshots. oops. That sunken feeling.
I corrupted the vmdk and the snapshot vmsd file.
Here’s how I fixed it all.
Firstly I couldn’t run anything against the original vmdk file, as it kept saying the file was locked by the host. I unregistered the VM from VC, put the ESX server into maintenance mode (which moved all the VMs off to other hosts in the clustering – fully enabled DRS) and power cycled the host. This remove the lock on the file thankfully.
The original vmdk had been 20GB before I tried expanding it to 30GB. So I created a new 20GB vmdk:
# vmkfstools -c 20G -d thick silver1.vmdk
I then copied the contents of the broken 30GB disk into the new one:
# dd if=silver-flat.vmdk of=silver1-flat.vmdk bs=1048576 count=20480
Moved the old vmdk and renamed the new one:
# vmkfstools -E silver.vmdk silver.vmdk.old
# vmkfstools -E silver1.vmdk silver.vmdk
All the snapshots rely upon a CID and a parentCID in their -00000x.vmdk files. I opened both snapshot disks to record the details.
# cat silver-000001.vmdk
# cat silver-000002.vmdk
Basically, 000002 parentCID is 000001′s CID. In turn 000001 parentID was set to point to the original vmdk. So I had to edit the new vmdk file and set this CID.
# vi silver.vmdk
I was then able to power on the VM. Phew!
However the snapshots were still screwed up. When I tries to commit the snapshots with:
# vmware-cmd silver.vmx removesnapshots
It thought it didn’t have any snapshots, but it was clearly using the two additional vmdk disks. The vmsd file was also corrupted. So I renamed the existing one:
# mv silver.vmsd silver.vmsd.old
Then I created a new snapshot, to re-generate a new vmsd file:
# vmware-cmd silver.vmx createsnapshot recreate_vmsd
And finally I could commit the snapshots:
# vmware-cmd silver.vmx removesnapshots
Now all that was left was to increase the disk:
#vmkfstools -X 30G silver.vmdk
I guess I won’t be doing that in a hurry again 
