I want a template replicator vApp…

Here’s the problem:

Companies that have more than one office split by a WAN link, have problems keeping their templates in sync.  There are two common approaches to this:
1) When updating a template, touch each site and update the same template in each location.
2) Update a master template and copy it out to each site.
Neither of these solutions scale very well, when you have multiple templates and lots of remote sites.

I dream of a vApp which you can deploy each site, and that is aware of the other instances at each site.  The vApp’s sole purpose is to watch for changes in a local template store on the designated “master” appliance, and replicate out those changes to all connected instances (at each sites).  It would be nice if those changes were just block-level changes.  The templates can sit in the vApp themselves on an NFS export, to be mounted by local ESXi hosts, so the templates can be deployed and updated straight from the vApp.

What do you think?  Obviously these tasks can be offloaded to storage array replication, DataDomain type devices, etc.  But I’d like a native tool that could be used anywhere, and wouldn’t rely on specific equipment.  Let me know in the comments below if you already have any groovy tools or rsync scripts that you use to do this automagically.

Hopefully, the VMware labs can create me a new fling for Christmas

I thought I’d write a quick message about this, as much for my own reference.  Maybe someone else out will find it useful one day. Its purpose is to document the changes made by the esx 3.5 to 4.x rollback script.

Normally when you do 3.5 > 4.x upgrade, if the upgrade fails then it automatically rolls the changes back for you.  Assuming that the install completes successfully, you can do one of two things afterwards.  You can manually remove the old 3.5 cruft with the cleanup-esx3 shell script.  Or if you decide there is something not quite right with the new 4.x install you can manually roll the server back to 3.5 by running the rollback-to-esx3.  All pretty straightforward stuff really.

However, I was just in a situation where I had to boot back into a 3.5 install (until you run the cleanup script, the 3.5 boot options remains in the grub boot menu), and I wanted to run the rollback script.  Normally you run the rollback script when you have booted up into the 4.x install.  But in this case I couldn’t boot into the 4.x image (following a failed patching session) so couldn’t get to the rollback script which lives in /usr/sbin/ of the esxconsole.vmdk file, which 3.5 doesn’t mount.

The reason was the server had recently been upgraded to 4.1 fine, but when subsequently applying the latest 4.1 U1 patches the upgrade went belly-up. This particular server was built by a consultant with only a 100MB boot partition.  The ESX 4 kernel images are around 25MB and aren’t removed automatically, so the /boot partition can fill up after a couple of patching sessions.

The type of nonsensical VUM error messages I was getting were:
HostPatchESXUpdateFailure
HostUpgradeIncompatible
RemediateFailure
HostUpgradePrecheckTestFailBootStorage <whinge> this is the one that made me check the filesystem usage, but you’d think they could just say explicitly why and save us all a lot of head-scratching</whinge>

Most of the vCenter error messages are much improved these days.  However VUM messages are still dreadful.

Fortunately I had access to another recently upgraded host, which I hadn’t patched with 4.1U1, so I could get to the rollback script.  I just made the same changes manually.  (Actually I didn’t remove the files, but moved to the /tmp directory)

Once I’d done that, I was able to rescan the host in VUM again and run the upgrade from 3.5 to 4.1 again.  Then I could run the cleanup script, and then upgrade it to 4.1U1.

So here are the contents of the rollback-to-esx3 script:

rm -rf /boot/config-2.6.*

rm -rf /boot/initrd-2.6.*

rm -rf /boot/initrd.img

rm -rf /boot/System.map-2.6.*

rm -rf /boot/vmlinuz-2.6.*

rm -rf /boot/vmlinuz

rm -rf /boot/trouble

cp /boot/grub/grub.conf.esx3 /boot/grub/grub.conf        ## make a copy first

And for completeness, here is the cleanup-esx3 script:

rm /usr/sbin/rollback-to-esx3

sed -i -e '/^# BEGIN migrated entries/,/^# END migrated entries/d' /etc/fstab

# Remove old ESX v3 titles in grub.conf

sed -i -e '/^# BEGIN ESX v3 title/,/^# END ESX v3 title/d' /boot/grub/grub.conf

# Remove old ESX v3 boot files:

rm -f /boot/initrd-2.4.21-58.ELvmnix.img-dbg

rm -f /boot/initrd-2.4.21-58.ELvmnix.img

rm -f /boot/System.map

rm -f /boot/System.map-2.4.21-58.ELvmnix

rm -f /boot/config-2.4.21-58.ELvmnix

rm -f /boot/kernel.h

rm -f /boot/vmlinuz-2.4.21-58.ELvmnix

rm -f /boot/initrd-2.4.21-58.ELvmnix.img-sc

rm -f /boot/vmlinux-2.4.21-58.ELvmnix

Following our announcement last week about the new VMware vSphere Design book, I wanted to follow-up on some of the feedback we’ve received.

Firstly, and I know I speak for Scott and Maish here, we are extremely grateful for all the kind words and thoroughly encouraged by those of you who let us know via email, blog comments and twitter that you had already ordered your copy.  All three of us were very enthused about writing this book, because we felt it would fill a gap that exists at the moment.  I sincerely hope that everyone in the VMware community who grabs themselves a copy finds it useful.

There are a few recurring questions that we’ve been asked, so I’ll try to answer them here (let me know in the comments if there are others, and I’ll update the post):

Will there be an electronic/kindle/nook/etc version available?

Yes.  The publishers will release an electronic version at the same time that the paper copy leaves the warehouse.  It is up to each retailer to convert it into their chosen electronic format.  We expect the electronic versions to be available from your favourite retailers around the same time as the printed version.

Does the book focus on vSphere 4.0 or 4.1? Does it cover feature XXX in 4.1?

The book was conceived before the release of version 4.1, and in fact the writing had begun by the time 4.1 was released.  Personally I had completed one chapter and was partway through a second when 4.1 hit.  However, I went back and made sure I “retrofitted” those to include anything new from 4.1.  So all the chapters I wrote certainly cover 4.1, and I also tried to make note of where there was a difference between the two.

The book certainly stands for both versions and I fully expect the majority of it to be relevant for vSphere 4+1, as the book tries to focus on conceptual ideas first.

Will the book be available in the UK/Europe/Netherlands/Timbuktu?

This book is distributed by a regular book publisher (Sybex, part of the Wiley group), not a self-publishing group.  As such it should be available for order via any reputable book retailer.  Like any technical book, you are more likely to find it on their online store first.  If you don’t find it on the shelf of your local bricks ‘n mortar bookshop, they should be able to order it for you.

Does it prepare me for my VCDX defence/application or VCAP-DCD exam?

This book is not specifically written for either the VCDX process or the VCAP-DCD exam.  It does discuss vSphere design in-depth, and therefore should be valuable while considering the design process.  Primarily this book is aimed as a practical guide to understanding vSphere design, not a study guide.  If it does help you, then that’s great.

——-

I’m immensely proud to have it finished, and to have been able to work in such great company as Scott, Maish and Jason Boche (who was our technical editor).  Until it’s actually out there, here is a peek at the chapters:

1. An introduction to designing vSphere environments
2. ESX versus ESXi
3. Designing the management layer
4. Server hardware
5. Networking
6. Storage
7. Virtual machine design
8. Datacenter designs
9. Designing with security in mind
10. Monitoring and capacity planning
11. Bringing it all together (design case-study)

——-

Now that I’ve actually finished working on it, it affords me some more time to get back to my (ir)regular blogging.  One thing I noted whilst writing the book was there was occasionally not enough pages (and unfortunately never enough time) to always dive as deeply into some areas that I’d have liked.  There were a number of areas I realized that weren’t covered in much depth elsewhere and I’d love to explore them more.  So, as time permits, I’m going to concentrate on some of those and dig into things a bit.  I’ve already got a couple of posts brewing; I’m just working out my plan for them now.  All published on my blog here.  Now, I just need to find the hours to write them

##################

The VMware vSphere Design book is now available for pre-order on Amazon and will be in the stores around the middle of March 2011.  Pre-order your copy today:

This is a joint post by three prominent writers in the virtualization community: Maish Saidel-Keesing, Scott Lowe and Forbes Guthrie.

For the past 6 months we have worked together on a new book.  This has been kept pretty quiet, but it’s now time to make it public.

Previous VMware vSphere books have focused on how to master the technology, deep-diving into certain elements and giving tips & tricks that help you manage your virtual infrastructure.  But we felt there was something missing in all these books – how to design an infrastructure.  For example:

- What kind of servers should I use?
- Which storage protocol: NFS, iSCSI, FC?
- How to scale a vCenter server appropriately?

The three of us collaborated on the book, not only to explain how to configure each element of your infrastructure, but to make you think about all the options available, and how each choice can impact the overall design. It should help you find the right solution for your environment – because no “one size fits all”.

It is the only book focused on designing VMware vSphere implementations. It’s written for engineers and architects who plan, install, maintain and optimize vSphere solutions.

The book details the overall design process, server hardware selection, network layout, security considerations, storage infrastructure, virtual machine design, and more. We debate the merits of scaling up servers versus scaling out, ESX versus ESXi hypervisors, vSwitches versus dvSwitches, and FC, FCoE, iSCSI or NFS storage. We show you which tools can be used to monitor, to plan, to manage, to deploy and to secure your vSphere landscape. We run through the design decisions that a typical company might face, and question the choices you come to. The book is packed with real-world proven strategies. VMware vSphere Design examines how the virtualization architecture for your company should ideally look – be it a newly deployed environment or optimizing the existing infrastructure.

We would like to thank Jason Boche for acting as the technical editor for the book.

We hope you enjoy reading this book, as much as we enjoyed writing it.
Maish, Scott and Forbes.

The VMware vSphere Design book is now available for pre-order on Amazon and will be in the stores around the middle of March 2011.  Pre-order your copy today:

This post is in response to Gabe’s recent post:

Large Pages, Transparent Page Sharing and how they influence the consolidation ratio

and then Frank Denneman’s reply here: Re: Impact of Large Pages on Consolidation ratios

Finished reading? Ok, let’s continue…

Let me try to summarize very briefly what happens.  When Large Pages are not considered, Transparent Page Sharing (TPS) runs across a host’s VMs periodically (every 60 minutes by default) and reclaims identical 4KB pages – think of it like de-duping your memory.  When using Large Pages (2MB), TPS does not try to reclaim those identical pages because of the “cost” of comparing these much larger pages.  That is until there is memory contention, at which point the Large Pages are broken down into 4KB blocks and identical pages are shared.

Large Pages have been proven to offer performance improvements.  The strategy that VMware are following is technically advantageous – why incur the expense of figuring out which Large Pages are identical if the host doesn’t need to reclaim memory.  If it can back all VM memory requests with physical memory, then it doesn’t need to worry about reclamation yet.

However, the problem I see is that of perception.  If you run lots of VMs with very similar memory usage on a host, you would expect to see the advantage of TPS kicking in and suitable memory savings.  However if the Large Pages don’t get shared until the host thinks it is under pressure, you won’t see those savings until there appears to be a problem.  The host will wait until it hits 94% memory usage (6% free), before it deems itself under memory contention and starts to break those Large Pages into smaller 4KB ones.  So in a VM environment with very similar VMs, you are consistently going to run your host at around 94% memory used.  This isn’t a technical issue.  All those identical memory pages can still be reclaimed, just as before, and you are gaining a performance gain of Large Pages.  This is a perception issue.

Most vSphere administrators probably don’t realize this, their manager almost certainly don’t realize this.  All they see is their hosts running out of memory – they have less than 10% memory free! Time for some more hosts.  And even for those that understand that when they get to this level there is the potential that there is still more potential memory available, it’s not that apparent how much saving they could expect and how far to push it.  Do we hide all memory usage in the vSphere client until it hits only 4% or 2% free?  Obviously not. I think this is a measurement issue.

I’ve always avoided twitter.  Until now.  I’ve got something to say this time and want as many people to hear about it as possible.  The last one of the top 25 virtualization bloggers is joining the twitter ranks.

You might have noticed that for the last 6 months my blogging content has fallen off dramatically.  This was an unfortunate side-effect of all the work I have poured into a new project.  I hope my part of this new endeavour benefits our community – its something I’m pretty proud of.

Although I have blogged for a few years now and hopefully created useful content for VMware community, I have never joined the twitter gang.  I made a conscious effort to concentrate on areas I could add-value and was mindful of spreading myself too thin.  We all suffer from information overload these days, drinking from the fire hose.  For that reason, I purposefully don’t “do” Facebook, LinkedIn, IRC channels. I avoid spending too much time on forums and keep my RSS feeds to a minimum.  Keeping up to date is crucial in our industry, but being able to skim the cream and not get caught up in the noise is a tricky tightrope.  As essential (and enjoyable) as it is to consume information, I believe creating it is so much more valuable.

Now I’m not quite ready to let you all know what I’ve been up to, but I will use my spanking brand new twitter account to reveal all, sometime in the next couple of weeks.  So why not follow me on this space-aged newfangled twitter thing and be the first to find out what is happening.  Don’t fear, I don’t expect to flooding the twitterverse with my daily routine, just the odd tweet to let you know about new blog posts, new reference cards, or anything vSphere related that doesn’t warrant a full blog post.

If you already know what I’m about to reveal, then please don’t spoil the fun and keep it to yourself.  It’s probably not too difficult to figure out, and super-sleuths (like you Mr E. Sloof ) are banned from playing.

Here is my vSphere 4.1 card blown up to poster size.  It’s finally time to spruce up your office and get rid of that old Baywatch one with the curled ends.

Help yourself.  Thanks again Dudley.

In case you’ve not seen it before, here is my View & ThinApp Desktop Card as a poster as well.

And remember, if you haven’t already voted; get yourself over to vsphere-land and show your support for your favourite VMware bloggers for all their hard work over the year.  It will only take 2 minutes of your time, and everyone on that list deserves a big pat on the back as far as I’m concerned . It’s your turn to pay it forward!

I’ve already had a couple of great bits of feedback about the new vReference card for 4.1.

Valerio spotted a handful of typos and Christian Meier point me to some licensing changes.  I’ve updated the card and you’ll now find the download is for version 2.3.2.

So here is a couple of things new to me:

• vMotion is now included with the Essential Plus and Standard versions.
• Thin provisioning which used to be included in all versions (including the free ESXi version, Essential and Essential Plus) as can be seen on the table here, is now only included in Advanced and above according to this new licensing PDF.  I hope this is just a documentation error.  If not then I consider this quite a loss for the smaller businesses who probably won’t have access to advanced storage arrays.

Sorry about the update so soon after its release, but I thought you’d all rather have the correct information as soon as possible.  As ever, the vReference card page is always the place to go for the latest version – don’t link friends to the card directly as these links get out of date.  I had over 100 downloads for the old version 1.0 and 0.x cards just in the last week alone.  I released 2.0 around a year ago!

I’ve also added a link on the vSphere 4 card page to the last card supporting vSphere 4.0.  I realised that some of you are still probably supporting and maybe even deploying/designing around this version.  If any of you are still working with 3.5, you can still find the old VI3 card on its page here.