If you are running VMware ESXi 4.1 then you can now activate SSH access (or Remote Tech Support) easily which is documented here: http://www.thomasmaurer.ch/2010/07/activated-ssh-on-esxi-4-1/

However, if you are working on ESXi 3.5 or 4.0 then you need to enter unsupported mode:

1) Press ALT & F1 to access the console.
2) Type: unsupported (you won’t see any characters on screen), then press Enter.
3) Edit /etc/inetd.conf, find the line starting with #SSH and remove the #. See my note below on using Vi with ESXi
4) Exit Vi (Type :wq)
5) Restart the management services: /sbin/services.sh restart
6) If you are running ESXi 3.5 Update 2 then you will also have to kill the inetd process. Type ps | grep inetd to find the process ID and then kill it with kill -HUP <ID>

Using Vi with ESXi

I don’t use anything other than Vi when editing files in Linux, but for most Linux users Nano is the editor of choice. Don’t fret, Vi is not that difficult, and the following steps will be enough to open a file, make some changes, save and quit Vi.

1) Open /etc/inetd.conf with Vi:
# vi /etc/inetd.conf

2) Jump straight to the line starting with SSH: Type /ssh and it will jump to the first occurrence of ssh. A bit like search in Word.

3) Make sure the cursor is at the start of the line starting with #SSH and press x to delete a character (delete #)

4) Press Escape to make sure it is not in editing mode, and type :wq and press enter. This writes and quits.

Accidently killed inetd without the -HUP switch?

I once used kill -9 instead of -HUP which stops and restarts the process. If you cannot afford to restart the ESXi host there is a way of starting inetd with ESXi:

# cat /var/run/inetd.pid

This will give you a process ID (a very long number)

# kill -HUP <ID>

It is not uncommon for system administrators to clone virtual servers or take an image of physical servers running Windows Server 2008 these days. There are plenty of tools to do that these days (Ghost, Acronis, Platespin for P2V conversions, etc.) If this is something you do regularly then you won’t be unfamiliar with Sysprep or NewSID, but according to Mark Russinovich at Microsoft, the SID doesn’t matter and Sysinternals have now retired NewSID (written by Mark). NewSID isn’t supported in Windows Server 2008 and the only option now is to use Sysprep. Whilst the facts presented on Mark’s blog are correct, I have personally seen many issues cloning or imaging Windows Server 2008 machines that haven’t been cloned with Sysprep first. Let me present a typical scenario that would cause problems here.

Scenario:

1) Build a Windows Server 2008 R2 server, apply patches and various tweaks.
2) Shutdown the server and take an image (or clone it to a virtual machine template). Note: I haven’t used Sysprep!
3) Deploy two new servers from the image or template. Promote one to a domain controller and add the other one to the domain as a member server.

In this scenario the first problem I would encounter is that any domain users that are a member of Domain Admins will not have the appropriate permissions to access PowerShell or Computer Management. The default Administrator account would work fine. Secondly, if I try and ping the domain controller I would get the following error:

C:\Users\User1>ping LAB-DC01
Unable to contact IP driver. General failure.

So the SID really does matter. Prior to taking your clone or image, just remember to use Sysprep as follows:

1) Run Sysprep (on Windows Server 2008 this is located in c:\Windows\System32\Sysprep\Sysprep.exe)
2) Ensure ‘System Out-of-Box Experience (OOBE)’ is selected
3) Tick the ‘Generalize’ option (this resets the SID)
4) Select ‘Shutdown’ from the Shutdown Options.
5) Once the machine has shutdown, take your image and you are good to go!

Earlier this week VMware released vSphere 4.1, a significant new version of the industry leading vSphere 4 virtualisation hypervisor. What is interesting is that from September 2010, vCenter management products will be licensed on a per VM basis. This includes: Chargeback, Site Recovery Manager, AppSpeed  and CapacityIQ  which are currently priced per processor. Products that won’t be included in this licensing change are: vCenter Lab Manager, vCenter Lifecycle Manager, vCenter Server Heartbeat, vCenter Server, and VMware vSphere (ESX).

The new licensing model will be sold in packs, with a minimum of a 25 VM pack. VMware vCenter will remain as a per server license.

“As virtualization becomes more pervasive, the VM has become the unit of measure for the data center in terms of cost accounting,” Bogomil Balkansky, VMware’s vice president of product marketing, told RCPU over lunch recently. The new model, which breaks from a CPU-based licensing structure, “allows customers to be a lot more granular,” Balkansky said.

New pricing has also been announced as part of vSphere 4.1, which offers stacks of new features for enterprise users, and importantly allows VMware to offer some of its more dated innovations at SMB prices. vMotion for example is now included in Standard and Essentials Plus vSphere editions, although the price has increased. Customers that have purchased support for the Standard Edition of vSphere 4.0 will be able to use vMotion at no extra cost.

For the enterprise customer, vSphere 4.1 has a new I/O control for both storage and networking. With this QoS (Quality of Service) style prioritisation for virtual machines, we can now guarantee storage and network IOPS for a virtual machine. Previously all virtual machines on the same ESX server had the same level of access to storage and network traffic, and there was no option to prioritse this. QoS metrics now move with the virtual machine when vMotion is active.

The vSphere 4.1 product is available as a free upgrade for those users buying support for vSphere 4.0 and will be available as a download as of today.

http://www.vmware.com/support/vsphere4/doc/vsp_41_new_feat.html

Update: If you are already using VMware vCenter Lab Manager 4 in your vSphere environment, and you plan to upgrade to vSphere 4.1 then you must upgrade to VMware Lab Manager 4.0.2.

A few weeks ago I was reading about a recent VDI implementation that Lancashire Police had recently undertaken, where they deployed 4000 virtual desktops to address compliance issues. Their first step was to gain control of the desktop first, gaining an understanding of the applications used, and any bespoke or specialist applications being used. It is often the case that applications are installed that the IT department knows nothing about.

Most will agree that server virtualisation has reached a high level of technical maturity thanks to VMware vSphere, Hyper-V R2 and Citrix XenServer (mostly thanks to VMware). We have come to expect that all server virtualisation technologies now include live virtual machine migration with VMotion, XenMotion, and Hyper-V R2 Live Migration. Server virtualisation has addressed issues such as server sprawl, energy consumption and deployment times for new servers.

Virtual Desktop Infrastructure (VDI) is not a new concept, and Citrix had a Desktop Broker before they acquired XenSource back in 2006. The Citrix Desktop Broker product that had a Session Manager running on Presentation Server 4, was not anywhere near what we expect from a VDI solution these days. Citrix subsequently dropped the Desktop Broker (thankfully, as it was terrible) and in 2008 XenDesktop 2.0 beta was unleashed. VMware were not ignoring this and released VMware VDM 2.0, which was a real competitor to the Citrix offering. For 2010, VDI technology has improved massively and issues like virtual desktops not responding to power commands, or non-registration are becoming less likely.

A few weeks ago (March 2009) Microsoft announced RemoteFX which is a technology they acquired from Calista that essentially improves RDP. PCoIP from VMware and enhancements from Citrix with ICA, have also emerged to deliver a more ‘rich’ graphically intensive environment. Do all of these really deliver good performance over a WAN? An interesting article on DABCC discusses this in more detail: http://www.dabcc.com/article.aspx?id=14000

We have now reached a stage in the world of VDI evolution where desktop virtualization is being seriously considered by most organizations wishing to manage their desktops and regain control. I like to think that VDI is an evolutionary step from Citrix environments or Terminal Services. Before jumping into a VDI implementation though, think about how you currently manage your desktops. Do you have an accurate picture of the applications used around the organisation? If not, then perhaps a software asset management exercise would be the most logical first step.

When planning your VDI solution, try not to think too much about desktop virtualisation but instead focus on delivering applications, gaining control of desktops, and enforcing compliance throughout your estate. Staff in your organisation will also have to get used to the idea that their physical PC is no longer ‘theirs’ but a corporate managed desktop.

I mentioned software asset management (SAM), as this is also an integral part of delivering VDI. Try to gain a firm understating of which applications are used around the business, and by whom. Performance requirements of different teams and departments should also be part of your planning phase. It may be the case that your finance department have a higher specification workstation due to the financial software they are running.

Here is a quick summary of things to consider when planning your VDI project:

• Don’t underestimate the importance of storage, especially throughput (I/O). The more spindles the better. If you want the ultimate in VDI storage then look at solid state drives (SSD).
• Choose your hypervisor provider. The main contenders are VMware vSphere for Desktops, Citrix XenServer, or Microsoft Hyper-V R2.
• Connection brokering is now an integral part of VMware View or Citrix XenDesktop, but research this carefully as there are other providers out there.
• Be clear on the security requirements. Each delivery protocol is slightly different (ICA, PCoIP, RemoteFX / RDP).
• Do you need features such as XenMotion / VMotion, DRS and HA? This will determine which edition you choose (E.g. Premium or Enterprise).
• Decide how you are going to deliver your applications. VMware ThinApp is unique in the sense that it offers total virtualisation for the application, but other alternatives such as Microsoft App-V or streaming applications from XenApp are available.
• Decide what server hardware you are going to host your virtual desktops on. Lots of memory is key, but try and keep costs down.
• Will you be requiring persistent or non-persistent desktops? In other words, each time a user logs out do you want to preserve the state of their desktop, or give them a fresh build each time they log on?
• Keep login time down to an absolute minimum. This can be achived by eliminating traditional roaming profiles and using profile management software such as AppSense, RTO Virtual Profiles (now owned by VMware) or Citrix Profile Management.
• Consider using folder redirection for the desktop, and My Documents.
• Make sure you have a solid Software Asset Management (SAM) process in place.

It was announced last week that Microsoft have updated their VECD licensing model for VDI (Virtual Desktop Infrastructure). These changes are welcome news as you no longer require a VECD for SA license, because under the new changes Software Assurance (SA) now incorporates virtual desktop access rights. In addition to that you can now benefit from roaming usage rights that allows non-corporate home computers to connect to a virtual desktop.

A new VDA (Virtual Desktop Access) license has also been introduced as a device based model to allow customers to purchase a virtual desktop access right per device. This can be used with non-Microsoft devices such as Linux thin-clients, or third party devices.

These changes will take effect from 1st July 2010:

• Software Assurance (SA) now includes virtual desktop access and roaming rights for home PC’s. VECD for SA is no longer required.
• The new VDA (Virtual Desktop Access) license is a per device license that retails at approximately $100 per/annum. This is available through Microsoft Volume License programs.

I touched on licensing in part 1 of the Hosting the Desktop series, and it doesn’t matter whether your VDI project is using Microsoft Hyper-V, Citrix XenDesktop or VMware View, you still need to make sure you are licensed with Microsoft to run Windows clients. Unless of course you are running a Linux operating system on your virtual desktops!

Since VMware View was released in November 2009 and Citrix XenDesktop 4 in the same month, you could be mistaken that this is a game of tennis at Wimbledon with all the hype about each product. I first started working with desktop virtualisation in a VDI sense (not terminal server based) when VMware VDM 2.0 beta was released in 2007 (now VMware View), and for a moment VMware seemed to be ahead of the game. Prior to that, I had to put up with Citrix Desktop Broker which was made end-of-life a short while after its release. Then Citrix released XenDesktop 2.0, and with desktop provisioning making storage issues a thing of the past thanks to Citrix Provisioning Server (formerly an Ardence product), the ball seemed to be firmly back in Citrix’s court.

Having stayed with Citrix through to the current XenDesktop 4, I must admit I have been tempted by the offering VMware View 4 is making. VMware View 4 now has a new display protocol called PCoIP which according to VMware, will outperform the Citrix ICA protocol.

VMware have also put a strong emphasis on the ease of installation VMware View 4 has to offer, compared to Microsoft RDS (Remote Desktop Services) and Citrix XenDesktop, in addition to some significant performance improvements. What has got me excited though is the actual virtualisation platform (vSphere for Desktops) which has the same features as vSphere 4 Enterprise Plus. Whilst it has the same features as vSphere 4 Enterprise Plus, it has been configured specifically for hosting desktops and doesn’t require you to purchase a separate host license for the virtualisation platform. Citrix XenDesktop similarly bundles XenServer as part of their packages.

Some of the other new features include ClearType font rendering, multi-monitor, 1920×1200 display resolution and even horizontal & vertical monitor support. Also according to VMware, a 10,000 hosted desktop environment would require around 125 servers, whereas Citrix would require 239 servers. Don’t shoot the messenger though as I do feel there is a little ‘marketing’ spin on this, because I haven’t seen evidence that this is based on identical hardware requirements. Citrix XenServer is not something to be overlooked.

In terms of licensing VMware have a purely concurrent user model which means that you need a license for each connected (concurrent) user. Licenses are purchased in packs, for example the View Enterprise 100 pack that contains vCenter Server standard, and vSphere 4 for Desktops with an unlimited number of hosts. I remember when I was using VMware VDM 2.0, cost would prohibitive due to the fact I needed to buy VMware ESX licenses to host my desktops. Now this isn’t the case it seems they are matching up with Citrix on the cost model too.

As expected Citrix also offer concurrent licensing, with XenDesktop Standard, Advanced, Enterprise and Platinum editions all licensed based on the number of concurrent virtual desktops in use.

So who is the winner in this game of VDI tennis?

Honestly I think it is so close I wouldn’t like to say one way or another. If you already have a team of Citrix expertise in your department then go with Citrix XenDesktop 4. On the other hand if you are free to decide which way to go then check out VMware View 4. What I will say is that Citrix do have other technologies that can compliment your VDI solution, such as the Access Gateway with Advanced Access Control, EdgeSight for XenApp and NetScaler. If you are building an enterprise scale VDI solution then these products would also be part of the overall mix, which may bring Citrix into the lead.

A recent post on the VMware blogs discussed the acquisition of RTO Software, here is an excerpt from the blog:

“This is an exciting technology. For those unfamiliar with the specifics of what Virtual Profiles actually does, the technology seamlessly virtualizes, caches and synchronizes a desktop user’s roaming profile, while improving both the performance and data integrity of the profile. When a user logs on, instead of monolithically delivering the entire user profile and making the user wait for all of it, Virtual Profiles performs a “just-in-time” delivery.”

You can read the full article here: http://blogs.vmware.com/view-point/2010/02/vmware-to-acquire-rto-software.html
You can find more information on VMware View here: http://www.vmware.com/view
VMware View 4 Licensing FAQ: http://www.vmware.com/files/pdf/VMware-View4-Pricing-Licensing-Support-FAQ.pdf

When you power on a virtual machine with VMware ESX VI3 or vSphere it may reach 95% progress in the tasks, but when you open the console you’ll notice the error: “Unable to connect to the MKS: vmx connection handshake failed for mks of /vmfs/volumes…”.

The MKS is the virtual Mouse, Keyboard, Screen (aka. KVM – Keyboard, Video, Mouse), and the error is simply unable to map this to the guest OS, hence the power on will fail.

This issue isn’t very common, but should be relatively easy to resolve. First check the following:

1) Edit the settings of the virtual machine, click the Options tab and make sure that the guest operating system version is correct. This can cause this error, so make sure you select the correct OS. E.g. “Microsoft Windows Server 2008 R2 (64-bit)”.
2) Try removing the virtual machine from the vCenter inventory and re-adding it.
3) Try restarting the VMware management agents (service mgmt-vmware restart and service vmware-vpxa restart).

I have only had this issue occur twice and both times it was related to the guest operating system type being set incorrectly.

I passed my VCP4 back in December, and after many months of waiting I finally received my VCP certificate on vSphere 4 today!

I’m also working on some study material for those of you that are aiming for a VMware VCP certificate. If you subscribe to my RSS feed, you won’t miss out when it’s published!

Prior to VMware ESX 4 (vSphere) it was not possible to clone a running virtual machine to another datacenter, or to a virtual machine data store that wasn’t available on the source host. Now with VMware vSphere it is possible to clone a running virtual machine to another datacenter, even if the destination data store is not presented to the source host.

How does this work when the target hosts volume isn’t even accessible on the originating server?

The answer is in the Service Console network. When source and destination disk volumes are present, then it will initiate a disk to disk transfer (over iSCSI or fibre channel). However, when the target storage is unavailable it will now use the Service Console network to transfer the data. Magic!

This is an excellent feature, as prior to ESX 4 I had to temporarily present the target volume to the source host during the migration or cloning process. Now I can keep storage between my datacenters seperate.

Why the Service Console Network?

Providing the service console network is used for management only, this is the best option. You certainly wouldn’t want the VMkernal network to be saturated with traffic generated from a storage migration, as this would impact VMotion.

Thanks again to VMware for another excellent feature!

Virtualcenter looses connectivity to an ESX or vSphere host, and all of the virtual machines that are running on the host show as ‘disconnected’. You will also see that the host has ‘not responding’ in brackets next to it’s name.

This one is very simple to fix, as it is usually caused by the host agent service (mgmt-vmware) failing due to a dead process.

First, try and restart the mgmt-vmware service:

# service mgmt-vmware restart

If you find this is hanging when trying to restart the host agent, then you’ll need to kill off the process causing the issue. Open another console session and do the following:

# ps -ef | grep hostd

This will output a list of processes using hostd similar to the following:

root 23955 1 0 10:42 pts/1 00:00:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u
root 23961 23955 4 10:42 ? 00:00:15 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u
root 24211 23422 0 10:48 pts/1 00:00:00 grep hostd

If you look at the output carefully you’ll see that the first process is using the vmware-watchdog, this is fine, but the second line is using hostd (config.xml -u). This is the culprit, so lets kill the process. By the way, your virtual machines will continue to run so don’t worry about that.

# kill -9 23961

You’ll now find that the hostd service will start and after a few seconds your host and virtual machines will become available again in vCenter.