Since VMware View was released in November 2009 and Citrix XenDesktop 4 in the same month, you could be mistaken that this is a game of tennis at Wimbledon with all the hype about each product. I first started working with desktop virtualisation in a VDI sense (not terminal server based) when VMware VDM 2.0 beta was released in 2007 (now VMware View), and for a moment VMware seemed to be ahead of the game. Prior to that, I had to put up with Citrix Desktop Broker which was made end-of-life a short while after its release. Then Citrix released XenDesktop 2.0, and with desktop provisioning making storage issues a thing of the past thanks to Citrix Provisioning Server (formerly an Ardence product), the ball seemed to be firmly back in Citrix’s court.

Having stayed with Citrix through to the current XenDesktop 4, I must admit I have been tempted by the offering VMware View 4 is making. VMware View 4 now has a new display protocol called PCoIP which according to VMware, will outperform the Citrix ICA protocol.

VMware have also put a strong emphasis on the ease of installation VMware View 4 has to offer, compared to Microsoft RDS (Remote Desktop Services) and Citrix XenDesktop, in addition to some significant performance improvements. What has got me excited though is the actual virtualisation platform (vSphere for Desktops) which has the same features as vSphere 4 Enterprise Plus. Whilst it has the same features as vSphere 4 Enterprise Plus, it has been configured specifically for hosting desktops and doesn’t require you to purchase a separate host license for the virtualisation platform. Citrix XenDesktop similarly bundles XenServer as part of their packages.

Some of the other new features include ClearType font rendering, multi-monitor, 1920×1200 display resolution and even horizontal & vertical monitor support. Also according to VMware, a 10,000 hosted desktop environment would require around 125 servers, whereas Citrix would require 239 servers. Don’t shoot the messenger though as I do feel there is a little ‘marketing’ spin on this, because I haven’t seen evidence that this is based on identical hardware requirements. Citrix XenServer is not something to be overlooked.

In terms of licensing VMware have a purely concurrent user model which means that you need a license for each connected (concurrent) user. Licenses are purchased in packs, for example the View Enterprise 100 pack that contains vCenter Server standard, and vSphere 4 for Desktops with an unlimited number of hosts. I remember when I was using VMware VDM 2.0, cost would prohibitive due to the fact I needed to buy VMware ESX licenses to host my desktops. Now this isn’t the case it seems they are matching up with Citrix on the cost model too.

As expected Citrix also offer concurrent licensing, with XenDesktop Standard, Advanced, Enterprise and Platinum editions all licensed based on the number of concurrent virtual desktops in use.

So who is the winner in this game of VDI tennis?

Honestly I think it is so close I wouldn’t like to say one way or another. If you already have a team of Citrix expertise in your department then go with Citrix XenDesktop 4. On the other hand if you are free to decide which way to go then check out VMware View 4. What I will say is that Citrix do have other technologies that can compliment your VDI solution, such as the Access Gateway with Advanced Access Control, EdgeSight for XenApp and NetScaler. If you are building an enterprise scale VDI solution then these products would also be part of the overall mix, which may bring Citrix into the lead.

A recent post on the VMware blogs discussed the acquisition of RTO Software, here is an excerpt from the blog:

“This is an exciting technology. For those unfamiliar with the specifics of what Virtual Profiles actually does, the technology seamlessly virtualizes, caches and synchronizes a desktop user’s roaming profile, while improving both the performance and data integrity of the profile. When a user logs on, instead of monolithically delivering the entire user profile and making the user wait for all of it, Virtual Profiles performs a “just-in-time” delivery.”

You can read the full article here: http://blogs.vmware.com/view-point/2010/02/vmware-to-acquire-rto-software.html
You can find more information on VMware View here: http://www.vmware.com/view
VMware View 4 Licensing FAQ: http://www.vmware.com/files/pdf/VMware-View4-Pricing-Licensing-Support-FAQ.pdf

When you power on a virtual machine with VMware ESX VI3 or vSphere it may reach 95% progress in the tasks, but when you open the console you’ll notice the error: “Unable to connect to the MKS: vmx connection handshake failed for mks of /vmfs/volumes…”.

The MKS is the virtual Mouse, Keyboard, Screen (aka. KVM – Keyboard, Video, Mouse), and the error is simply unable to map this to the guest OS, hence the power on will fail.

This issue isn’t very common, but should be relatively easy to resolve. First check the following:

1) Edit the settings of the virtual machine, click the Options tab and make sure that the guest operating system version is correct. This can cause this error, so make sure you select the correct OS. E.g. “Microsoft Windows Server 2008 R2 (64-bit)”.
2) Try removing the virtual machine from the vCenter inventory and re-adding it.
3) Try restarting the VMware management agents (service mgmt-vmware restart and service vmware-vpxa restart).

I have only had this issue occur twice and both times it was related to the guest operating system type being set incorrectly.

I passed my VCP4 back in December, and after many months of waiting I finally received my VCP certificate on vSphere 4 today!

I’m also working on some study material for those of you that are aiming for a VMware VCP certificate. If you subscribe to my RSS feed, you won’t miss out when it’s published!

Prior to VMware ESX 4 (vSphere) it was not possible to clone a running virtual machine to another datacenter, or to a virtual machine data store that wasn’t available on the source host. Now with VMware vSphere it is possible to clone a running virtual machine to another datacenter, even if the destination data store is not presented to the source host.

How does this work when the target hosts volume isn’t even accessible on the originating server?

The answer is in the Service Console network. When source and destination disk volumes are present, then it will initiate a disk to disk transfer (over iSCSI or fibre channel). However, when the target storage is unavailable it will now use the Service Console network to transfer the data. Magic!

This is an excellent feature, as prior to ESX 4 I had to temporarily present the target volume to the source host during the migration or cloning process. Now I can keep storage between my datacenters seperate.

Why the Service Console Network?

Providing the service console network is used for management only, this is the best option. You certainly wouldn’t want the VMkernal network to be saturated with traffic generated from a storage migration, as this would impact VMotion.

Thanks again to VMware for another excellent feature!

Government ICT strategy for 2010:

“Traditionally, the public sector has relied on commercial off-the-shelf (COTS) software or bespoke developments from global providers. This restricts the ability of the public sector to reuse solutions, reduces flexibility to manage assets efficiently and prevents government organisations from switching suppliers. The Open Source, Open Standards, Reuse Strategy provides government’s approach to open source alternatives that meet public sector requirements. Government already commits to using only open standards for documentation. The ICT Strategy will build capability within the public sector to increase the amount of open source code and software in use and to make it available for reuse elsewhere.”

If the government already commits to using only open standard for documentation, then how much you spending our document management and CRM applications?

Here are some alternatives:

• http://www.openkm.com
• http://www.alfresco.com
• http://www.sugarcrm.com/crm

With budget cuts and reduction in public spending, this certainly makes a lot of sense. Don’t fall into the trap of believing any myths about Open Source software!

1) Open Source doesn’t just mean it’s limited to Linux.
2) Open Source is 100% reliable. If it wasn’t then 90% of the internet wouldn’t be reliable as it’s based on Open Source standards (LAMP, Apache, BIND… TCP/IP!)
3) Large enterprises and government organisations are now taking this seriously.
4) Support is actually vast, with massive online communities and support & maintenance agreements often available at a fraction of the price of COTS.
5) It’s not over-complicated. In fact quite the opposite.

I’ve said my piece and will wait with baited anticipation. I am sure that many organisations still fear Open Source, but isn’t it time to change your mind?

Virtualcenter looses connectivity to an ESX or vSphere host, and all of the virtual machines that are running on the host show as ‘disconnected’. You will also see that the host has ‘not responding’ in brackets next to it’s name.

This one is very simple to fix, as it is usually caused by the host agent service (mgmt-vmware) failing due to a dead process.

First, try and restart the mgmt-vmware service:

# service mgmt-vmware restart

If you find this is hanging when trying to restart the host agent, then you’ll need to kill off the process causing the issue. Open another console session and do the following:

# ps -ef | grep hostd

This will output a list of processes using hostd similar to the following:

root 23955 1 0 10:42 pts/1 00:00:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u
root 23961 23955 4 10:42 ? 00:00:15 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u
root 24211 23422 0 10:48 pts/1 00:00:00 grep hostd

If you look at the output carefully you’ll see that the first process is using the vmware-watchdog, this is fine, but the second line is using hostd (config.xml -u). This is the culprit, so lets kill the process. By the way, your virtual machines will continue to run so don’t worry about that.

# kill -9 23961

You’ll now find that the hostd service will start and after a few seconds your host and virtual machines will become available again in vCenter.

In 1987 when I was just 11 years old, I used my first real PC, other than the Sinclair Spectrum and Commodore 64 (1984) of course… an Intel 80386 Personal Computer with a green screen CRT display. Bundled with it was a book called ‘80386 Programmer’s Reference Manual‘.  Back then I was fascinated with computer programming and machine code, and although little of the book was understood at the time, much of it is still relevant today. The book was published in 1986, and one of the sections covered processor privilege levels.

Privilege levels determine what rights the procedure being executed has when issuing instructions, or accessing I/O ports and memory addresses. There are 4 privilege levels for x86 processor architecture, and these are called rings. The four rings (0 to 3) are designed to protect the hardware resources, keeping user applications at level 3 (the least privilege) and the operating system kernel at ring 0 (the most privileged).

In 1998 VMware used a technique called binary translation that allowed the operating system to operate at ring 1, whilst (unknown to the OS) the VMM (Virtual Machine Monitor) is running at ring 0 with the highest privileges on the hardware resources. You can read more on that <here>. What is important, is that this approach enabled full virtualisation of any x86 operating system on VMware Virtual Infrastructure. Interestingly, Virtual Iron which Oracle acquired then ditched back in 2009, didn’t use binary translation but still managed to achieve full virtualisation using what they labelled as ‘native virtualisation’.

Then in 2006 Intel and AMD released hardware virtualisation technology (VT or AMD-V) in their processors that overcomes the need for using binary translation techniques. This introduced a new level (or mode) aptly names ‘root mode’ that sits under ring 0.

I imagine that one day in the distant future, this technology will evolve to such as level it could be like trying to understand molecular modelling!

Got r00t?

It’s Monday morning and you arrive late at the office thanks to the trains being delayed yet again. At that particular moment in time as you grab your morning coffee, several hundred users have already logged in and started launching their email client, web applications, and a myriad of documents and spreadsheets. So far this sounds like any other morning, but what I didn’t mention was the fact that just 30 minutes before you arrived at the office, a water from a pipe in the ceiling started to leak into the rack containing your SAN’s disk array.

What a nightmare. Not only has the water managed to get into both SAN controllers, but it has caused the trip switch for that rack to shut off. But wait… not a single user has called to say they can’t access their applications or data. Thanks to storage mirroring between two SAN arrays in separate racks, the business has continued to operate and all of the servers are now communicating with your secondary array. Seamless.

Walking back to your desk with your morning coffee, your phone receives the first SMS message. Here it is, an alert from your monitoring system to say that the primary storage array is offline. “09:24 PRI-SAN01 offline, critical.”

At this point, it would certainly be pertinent to discuss best practice for data centre design, environment monitoring, and DR procedures. To achieve a solid DR solution for your infrastructure you must have the basics in place before anything else. This means your DR strategy has got to be reviewed on a regular basis, and business continuity planning must be in motion with all areas of the business. Without a solid continuity plan, your DR might not serve the actual needs of the business. The focus on this article is disaster recovery for your SAN rather than business continuity, but BCP must never be ignored. Lets rewind back to the implementation of a highly available SAN architecture, it’s far more interesting!

In 2005 I started to look at how storage mirroring can protect your data in this type of situation, and also provide you with ‘zero downtime’ maintenance windows for your SAN. Over the past few years, storage vendors have been implementing mirroring, thin provisioning, snapshots, and asynchronous replication for remote sites in entry level SAN solutions, not just the large enterprise offerings. Don’t be fooled into thinking that designing a highly available SAN architecture is limited to those with massive budgets. There are other solutions, such as SANmelody or SANsymphony by Datacore, that allow you to present your existing disk arrays or SAN to storage servers. It is far more cost effective than upgrading your entire SAN hardware, and you can even increase performance by using the storage servers RAM for your write cache.

Datacore SAN software is what I have been working with, in conjuction with EMC and HP SAN storage over the past few years. The main reason being that we can present storage from different SAN vendors, and create pooled storage that can then be partitioned up into virtual volumes (or LUN’s) for your application servers. On top of that we gain mirroring, thin provisioning, snapshots, and other features that our HP and EMC didn’t have without an expensive upgrade. Datacore are releasing SANSymphony-V in 2010, which I’ve had the pleasure of using in a technology preview recently. Datacore were talking about storage virtualisation back in 1999, so I’d certainly recommend you speak to them about what they can offer.

Lets familiarise ourselves with some key storage technologies:

Synchronous storage mirroring – When data is written to the primary array it is also written to the secondary array. Will require a high speed link between both arrays, such as fibre channel or iSCSI. This provides high availability for your SAN, but can double up on the storage cost in some situations.

Asynchronous mirroring – SAN replication to a DR site or remote office. Will replicate data in the background, using queuing, buffering and scheduling to the remote site. Typically used over WAN connections.

Snapshots – The ability to take a ‘point-in-time’ snapshot of your data. Very useful in a DR scenario, and for testing.

SAN Building Blocks for Disaster Recovery & High Availability

To set the scene I’ll use a typical IT infrastructure that you would find in most SME organisations. They have already implemented virtualisation for at least 50% of the server infrastructure, and have a midrange SAN from a well known vendor using fibre channel. SAN capacity is up to 8TB which contains a mix of virtual machine, database, and file store LUN’s. The majority of servers are running Microsoft Windows Server 2008, and some Linux servers for key network services.

The diagram shown here (1.0) is certainly simplified, but represents the core components of most SME infrastructures.

Using this example you’ll see that virtualisation is already in place having implemented VMware with High Availability, and additional high availability has been implemented with a Microsoft SQL database cluster. There is enough capacity to support a single host failure using VMware high availability, but there are still some physical application servers that are yet to be virtualised. Given this is a typical SME infrastructure, lets also imagine that the SAN has dual controllers, and it’s connected to a fibre fabric consisting of two core fibre channel switches (A and B). This is a very good situation to be in as we have most of the servers virtualised, SQL databases are stored on the SAN, in addition to file server storage for our shared drives.

Implementing on-site HA (High Availability) using synchronous mirroring, even to another building with a fibre link between the two, gives this environment an excellent level of resilience. However synchronous mirrors do have some pitfalls, mainly due to the cost as you need twice the amount of storage as the solution is split into two. One rack will contain a SAN with an 8TB array, and the other rack will contain another SAN with an 8TB array with mirroring between the two. You will then need to decide on the level of disk redundancy within each array as you could use a basic RAID0 stripe, given the fact you have mirroring between separate arrays. I personally prefer to stick with RAID5 arrays, even though they are mirrored between two arrays.

An asynchronous mirror is where true disaster recovery comes into play. By selecting key SAN Lun’s (or data volumes) to be replicated to a remote site you can specify which databases, virtual machines or file stores are part of the replication. This does introduce an extra layer of complexity though, which you don’t get with synchronous mirrors. First of all you need to have a suitable location / site for the destination SAN, unless you consider using a co-location service with an ISP. Depending on how much the replicated data changes, the link between these sites could be very busy so bandwidth is a consideration. That being said, a 20Mb private circuit between two sites around 40 miles away should be in a fairly realistic price bracket. If you are using a co-location provider, they should be able to provide this for you. As a rough estimate, I would say £10,000 to £20,000 per annum for a 20Mb link in the UK.

Adding further complexity to the asynchronous mirroring solution is what to do with the destination data in the event of a disaster (or DR test). When a SAN Lun is first presented to an application server, whether that is a VMware host, Windows or Linux host, it will need to write a disk signature to the disk (LUN). When using asynchronous mirrors, the destination LUN (at the DR site) will have exactly the same signature. In this case you must make sure the disk isn’t resignatured by the application servers at the remote site. VMware servers (ESX and vSphere) have an advanced option to disable resignaturing, whereas Windows servers shouldn’t cause an issue unless they are part of a cluster.

When testing your DR site with the replicated data, it is recommended that snapshots are used to take a ‘point in time’ snapshot of the destination volume. The snapshot volume is then presented to the application servers at the DR site, leaving the replication of live data to continue. Using asynchronous mirroring and snapshots provide the ability to carry our DR tests without impacting the live environment, so can be done during normal business hours in most cases.

Summary

Storage replication and snapshot technology certainly provide the key ingredients to form part of your DR solution, but there are still important factors to consider. Do you want high availability, replication to another site, or both? Does your existing SAN support these technologies, or should you consider an upgrade? Obviously your budget is going to be a major factor, and I’m not here to lecture you on ‘what would it cost if you actually had a disaster’, you can make that decision!

If you decide to adopt mirroring and snapshot technologies as part of your DR solution and you are already running a virtual infrastructure, then you are on your way to an excellent DR solution. There are some technical complexities you need to be aware of, but if you have a good knowledge in these areas they are only minor factors.

If you are in need of a CAT5 crossover cable, but can’t find one then it’s very easy to make one providing you have the right tools.

This is what you need:
Length of CAT5 (or CAT5e / 6) cable
Wire cutters
RJ45 connectors
RJ45 crimper

The drawing below shows you what each end of the crossover cable should look like:

The cable on the right of the drawing is the crossover end, notice that pins 1 & 3 have been crossed over, and pins 2 & 6 have also been swapped over. If you were to make a straight through cable, you obviously would make it the same both ends. In fact, it doesn’t matter which colours are used on each pin but we’re sticking to standards and good practice!

Tip: If you haven’t got the tools, but you need a quick crossover cable then cut the wire 6 inches from one of the ends, and solder the wires together ensuring you’ve swapped 1/3 and 2/6. Some may frown at this, but it works and it’s a good quick fix.

Why do you need a crossover cable?

The transmit wires (pins 1 and 2) on one computer need to be connected to the Receive wires (pins 3 and 6) on the other computer. If you use a hub/switch, it performs this ‘crossover’ automatically. Also bear in mind that if you’re connecting two hubs together you’ll also need a crossover cable. Most hubs/switches do have a small switch next to one of the ports, enabling you to use a straight through cable.

Start the VMware Tools installation

Using the VMware VI Client, right click on the Linux guest and select Install/Upgrade VMware Tools.

Mounting the CD-ROM

You’ll need to mount the CD-ROM on the Linux guest.
# mount /dev/cdrom /mnt

Change the working directory to /mnt
# cd /mnt
Using the RPM installer, install VMware tools.
# rpm -Uhv VMwareTools-3.5.0-153875.i386.rpm
Unmount the CD-ROM
# umount /dev/cdrom
Starting VMware tools and make sure it starts on boot.
# service vmware-tools start
# chkconfig vmware-tools on
Run the Vmware tools configuration tool. * This will shutdown the ethernet interface, so do this from the console.

# vmware-config-tools.pl

That’s it!

Troubleshooting:

“Trying to find a suitable vmmemctl module for your running kernel.”

“None of the pre-built vmmemctl modules for VMware Tools is suitable for your running kernel. Do you want this program to try to build the vmmemctl module for your system (you need to have a C compiler installed on your system)?”

If you are using CentOS 5 like me, then you’ll need to install GCC and kernel-devel otherwise it will fail. Also, make sure your kernel version matches the kernel-devel version. This error will also present itself if your are running Xen virtualisation.

# yum install gcc kernel-devel
# yum upgrade kernel kernel-devel

Finally, reboot to load the updated kernel and try again.