If you are running VMware ESXi 4.1 then you can now activate SSH access (or Remote Tech Support) easily which is documented here: http://www.thomasmaurer.ch/2010/07/activated-ssh-on-esxi-4-1/

However, if you are working on ESXi 3.5 or 4.0 then you need to enter unsupported mode:

1) Press ALT & F1 to access the console.
2) Type: unsupported (you won’t see any characters on screen), then press Enter.
3) Edit /etc/inetd.conf, find the line starting with #SSH and remove the #. See my note below on using Vi with ESXi
4) Exit Vi (Type :wq)
5) Restart the management services: /sbin/services.sh restart
6) If you are running ESXi 3.5 Update 2 then you will also have to kill the inetd process. Type ps | grep inetd to find the process ID and then kill it with kill -HUP <ID>

Using Vi with ESXi

I don’t use anything other than Vi when editing files in Linux, but for most Linux users Nano is the editor of choice. Don’t fret, Vi is not that difficult, and the following steps will be enough to open a file, make some changes, save and quit Vi.

1) Open /etc/inetd.conf with Vi:
# vi /etc/inetd.conf

2) Jump straight to the line starting with SSH: Type /ssh and it will jump to the first occurrence of ssh. A bit like search in Word.

3) Make sure the cursor is at the start of the line starting with #SSH and press x to delete a character (delete #)

4) Press Escape to make sure it is not in editing mode, and type :wq and press enter. This writes and quits.

Accidently killed inetd without the -HUP switch?

I once used kill -9 instead of -HUP which stops and restarts the process. If you cannot afford to restart the ESXi host there is a way of starting inetd with ESXi:

# cat /var/run/inetd.pid

This will give you a process ID (a very long number)

# kill -HUP <ID>

Earlier this week VMware released vSphere 4.1, a significant new version of the industry leading vSphere 4 virtualisation hypervisor. What is interesting is that from September 2010, vCenter management products will be licensed on a per VM basis. This includes: Chargeback, Site Recovery Manager, AppSpeed  and CapacityIQ  which are currently priced per processor. Products that won’t be included in this licensing change are: vCenter Lab Manager, vCenter Lifecycle Manager, vCenter Server Heartbeat, vCenter Server, and VMware vSphere (ESX).

The new licensing model will be sold in packs, with a minimum of a 25 VM pack. VMware vCenter will remain as a per server license.

“As virtualization becomes more pervasive, the VM has become the unit of measure for the data center in terms of cost accounting,” Bogomil Balkansky, VMware’s vice president of product marketing, told RCPU over lunch recently. The new model, which breaks from a CPU-based licensing structure, “allows customers to be a lot more granular,” Balkansky said.

New pricing has also been announced as part of vSphere 4.1, which offers stacks of new features for enterprise users, and importantly allows VMware to offer some of its more dated innovations at SMB prices. vMotion for example is now included in Standard and Essentials Plus vSphere editions, although the price has increased. Customers that have purchased support for the Standard Edition of vSphere 4.0 will be able to use vMotion at no extra cost.

For the enterprise customer, vSphere 4.1 has a new I/O control for both storage and networking. With this QoS (Quality of Service) style prioritisation for virtual machines, we can now guarantee storage and network IOPS for a virtual machine. Previously all virtual machines on the same ESX server had the same level of access to storage and network traffic, and there was no option to prioritse this. QoS metrics now move with the virtual machine when vMotion is active.

The vSphere 4.1 product is available as a free upgrade for those users buying support for vSphere 4.0 and will be available as a download as of today.

http://www.vmware.com/support/vsphere4/doc/vsp_41_new_feat.html

Update: If you are already using VMware vCenter Lab Manager 4 in your vSphere environment, and you plan to upgrade to vSphere 4.1 then you must upgrade to VMware Lab Manager 4.0.2.

This is bit of a random post, but I wanted to share some of the reasons I joined IT in the first place. Lets face it, IT infrastructure hardware has evolved very well in the past 10 years. Pictured to the left is the HP c7000 blade chassis, and this was actually taken in March 2010 when  I was rolling out VMware vSphere on HP BL460 G6 blades. What I like about the BL460 G6 is that it can run 6 x NIC’s (two on board and a four port card) in addition to an HBA mezzanine card. Top that off with excellent BladeSystem management, ILO and Insight Manager, you can’t get much better. These blades here have 48GB RAM per host and two quad-core Intel x5550 processors. At half-height, you can pack 16 of these in a single chassis. Excellent for consolidation, both in physical terms and virtual!

I’ve been working within large data centres since 2004, prior to that I had only worked in what I call ‘server rooms’. As an infrastructure manager / technical specialist, I pride myself in delivering first class technical solutions which I typically sum up in the following value statement:

“Providing a solid – reliable IT infrastructure, to meet the existing and future needs of the business, incorporating expert design, solutions and a truly professional level of service.”

It’s not all about flashy hardware though, as the statement above suggest it is about the service provided to your organisation. It’s probably easy for me to say, as I have worked over the past years introducing virtualisaton, VDI, storage and other infrastructure technologies, which in turn gives you a personal sense of pride. Your success is ultimately measured by the service provided, which not only incorporates uptime (HA and DR) but quality of service, speedy performance and uninterrupted service even during maintenance windows.

If you are interested in the uptime of data centre’s then you can find out more by visiting the Uptime Institute or by reading up on tier 1 data centre’s (according to TIA-942) thatl adopt N+1 resilience with cooling and power, and secure access control which meets strict security requirements.

Keep your data centre tidy, and feel free to comment or share your own pics!

Ray

Prior to VMware ESX 4 (vSphere) it was not possible to clone a running virtual machine to another datacenter, or to a virtual machine data store that wasn’t available on the source host. Now with VMware vSphere it is possible to clone a running virtual machine to another datacenter, even if the destination data store is not presented to the source host.

How does this work when the target hosts volume isn’t even accessible on the originating server?

The answer is in the Service Console network. When source and destination disk volumes are present, then it will initiate a disk to disk transfer (over iSCSI or fibre channel). However, when the target storage is unavailable it will now use the Service Console network to transfer the data. Magic!

This is an excellent feature, as prior to ESX 4 I had to temporarily present the target volume to the source host during the migration or cloning process. Now I can keep storage between my datacenters seperate.

Why the Service Console Network?

Providing the service console network is used for management only, this is the best option. You certainly wouldn’t want the VMkernal network to be saturated with traffic generated from a storage migration, as this would impact VMotion.

Thanks again to VMware for another excellent feature!

Virtualcenter looses connectivity to an ESX or vSphere host, and all of the virtual machines that are running on the host show as ‘disconnected’. You will also see that the host has ‘not responding’ in brackets next to it’s name.

This one is very simple to fix, as it is usually caused by the host agent service (mgmt-vmware) failing due to a dead process.

First, try and restart the mgmt-vmware service:

# service mgmt-vmware restart

If you find this is hanging when trying to restart the host agent, then you’ll need to kill off the process causing the issue. Open another console session and do the following:

# ps -ef | grep hostd

This will output a list of processes using hostd similar to the following:

root 23955 1 0 10:42 pts/1 00:00:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u
root 23961 23955 4 10:42 ? 00:00:15 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u
root 24211 23422 0 10:48 pts/1 00:00:00 grep hostd

If you look at the output carefully you’ll see that the first process is using the vmware-watchdog, this is fine, but the second line is using hostd (config.xml -u). This is the culprit, so lets kill the process. By the way, your virtual machines will continue to run so don’t worry about that.

# kill -9 23961

You’ll now find that the hostd service will start and after a few seconds your host and virtual machines will become available again in vCenter.

Snapshots are a fantastic way of providing a quick and reliable method of rolling back the state of a virtual machine, should something go astray following an patch or update. VMware VCB also uses virtual machine snapshots to quiesce the VM prior to taking the backup data.

However, in larger environments where there may be tens or hundreds of VMware ESX servers, snapshots can also be a pain in the backside if there is no control over who is using them. Why? Because snapshots work by creating a delta VMDK that records the changes in blocks, a process called copy-on-write (COW). Over time the delta VMDK file will grow, and depending on the level of I/O within the VM it could grow faster on some virtual machines and not others.

The danger only presents itself if the datastore where the VMDK resides reaches it’s capacity. When this happens, virtual machines that are not thin-provisioned should continue to run with no problems, but think about these situations:

1) You have other virtual machines in the same datastore using snapshots.
2) You have one or more virtual machines on thin-provisioned disks.
3) You have powered off virtual machines, that need to be powered on.

In all of the above scenarios if the datastore is full then the affected virtual machines will be suspended (paused). Virtual machines with thick-provisioned disks will continue to operate as the VMDK already has the full allocated of storage space available. Virtual machines that are powered off, and need to be powered back on will fail as they won’t have enough disk space to create the virtual swap file.

The simple rules to follow to avoid these situations is:

1) Allow enough free disk space in each datastore for the snapshots to reside. If snapshots are only ever created during a VCB backup, and the virtual machines have little I/O then you won’t need as much space as busy environments.
2) Monitor the use of snapshots in your environment. I’ve included a command you can run to find snapshots below.
3) Don’t forget about snapshots and leave them in place. They will get bigger!
4) Change the permissions to restrict who can take virtual machine snapshots. The fewer the better, and try to establish some internal procedures to make IT staff aware of the potential issues.

A simple, yet useful command you can run on each ESX server to find snapshots is this:

Login to your ESX server and change the working directory to /vmfs/volumes. Then use the grep command to find vmsn (VMware Snapshot) files.

# cd /vmfs/volumes
# ls -Rh | grep "vmsn"

This can be really useful, especially when you don’t know who is creating snapshots and forgetting they are there!