It was announced this morning that the VDCD511 (VCAP5-DCD beta exam) is available to take from 13th February to 2nd March 2012. You can take it at VMware Partner Exchange 2012 in Las Vegas. When the final exam is released, for a limited time you won’t need to have a VCP5 certification as a pre-requisite. Even if you are not planning on taking this exam during the beta invitations, it’s a great opportunity to get studying. The blueprint does subtly differ from the VCAP4-DCD, and as with the previous exam you’ll be expected to understand the VMware design methodology.

It is important to have a working technical knowledge of vSphere 5, but you need to apply this technical knowledge to a design (Conceptual > Logical > Physical > Implementation Planning).

Here is the exam blueprint in a summarised form:

Section 1 – Create a vSphere Conceptual Design
1.1 – Gather and analyze business requirements
1.2 – Gather and analyze application requirements
1.3 – Determine Risks, Constraints, and Assumptions

Section 2 – Create a vSphere Logical Design from an Existing Conceptual Design
2.1 – Map Business Requirements to the Logical Design
2.2 – Map Service Dependencies
2.3 – Build Availability Requirements into the Logical Design
2.4 – Build Manageability Requirements into the Logical Design
2.5 – Build Performance Requirements into the Logical Design
2.6 – Build Recoverability Requirements into the Logical Design
2.7 – Build Security Requirements into the Logical Design

Section 3 – Create a vSphere Physical Design from an Existing Logical Design
3.1 – Transition from a Logical Design to a vSphere 5 Physical Design
3.2 – Create a vSphere 5 Physical Network Design from an Existing Logical Design
3.3 – Create a vSphere 5 Physical Storage Design from an Existing Logical Design
3.4 – Determine Appropriate Compute Resources for a vSphere 5 Physical Design
3.5 – Determine Virtual Machine Configuration for a vSphere 5 Physical Design
3.6 – Determine Datacenter Management Options for a vSphere 5 Physical Design

Section 4 – Implementation Planning
4.1 – Create an Execute a Validation Plan
4.2 – Create an Implementation Plan
4.3 – Create an Installation Guide

If you are new to the VCAP exams, then I would personally recommend that you take the VCAP5-DCA (no details of this exam yet) as I feel it is important that you have the technical knowledge required to apply to a design. Sure if you have solid technical experience with VMware products then it doesn’t really matter which exam you take first. I would strongly recommend taking the vSphere 5: Design Workshop course (3 day), and the course manual will go through the design process, technical design elements of vSphere 5.

Recommended Study Areas:

• Design Methodology (see blueprint)
• Storage Design
• VMFS Maximums
• iSCSI, NFS, Fibre Channel
• vSA (vSphere Storage Appliance)
• Storage DRS
• Storage I/O Control (SIOC)
• Network Design
• Networking Fundamentals
• VLANS / PVLANS
• Distributed Virtual Switch
• NIC Teaming
• Redundancy design options
• Network I/O Control (NIOC)
• DirectPath I/O
• Converged Networking and FCoE
• CDP
• Compute Design (Host)
• CPU (NUMA, Maximums, vCPU per core, etc)
• Host Configuration (PCI Slot, BIOS, Security)
• Virtual Machine
• Configuration Maximums
• Virtual Machine Security
• Optimal configuration (swap file, vCPU, Memory)
• vCenter
• Database
• Requirements
• vCenter Server Appliance (requirements, configuration)
• vCenter Heartbeat
• Security (SSL certificates)
• Linked Mode
• Cluster (HA/DRS)
• Resource Pools
• Admission Control
• Fault Tolerance (FT)
• HA Heartbeat Redundancy
• DPM

Recommended Reading:

• vSphere 5 Networking
• VMware vSphere 5 Clustering Technical Deepdive
• vSphere 5 Design Workshop Course Book

Whilst working on a Vblock 300 implementation a few weeks ago I had an interesting conversation with one of the network architects at VCE and we discussed the subject of best practices surrounding 10Gb and 1Gb networking. Traditionally with 1Gb networking it is best practice to separate traffic on your ESX/ESXi hosts with vSwitches (or dvPortGroups) dedicated to each type of traffic (vMotion, Management, Storage, production networking) and typically designs will contain 6 to 8 NIC’s per host. With the introduction of 10Gb networking, I’ve noticed that some implementations have neglected to include some important design considerations regarding the use of 10Gb networking. Lets say for that we present 4 x 10Gb NIC’s to each host (these are vNIC’s in the Cisco UCS world) or we can present 6 x 1Gb NIC’s  using traditional methods of separating the traffic into various dvportGroups. Which is best?  Can we get away with just 2 x 10Gb NIC’s or do we need more? The key consideration here isn’t how many NIC’s (or vNIC’s) are presented to each host, but rather how much network bandwidth is available to each traffic type (i.e. vMotion, FT Logging, VM traffic) and critically how we control it.

Best practice designs often take a while to stick in the mind and more often than not we forget the all important question of WHY it is best practice in the first place, we are all human after all.

Here is what I mean; Going back a few years I remember explaining to friends and colleagues why vMotion, Storage, Management and production VM traffic needs to be kept physically separate (not just on different VLAN’s). Now don’t get me wrong there was always plenty of documentation to support this, but there were still a high proportion of IT professionals still using a single vSwitch and only two 1Gb uplinks in their designs. A few years later it became widely accepted as best practice to separate the traffic, but now with 10Gb networking we have similar problem and one magic ingredient is often overlooked, and that ingredient is traffic shaping!

Shaping Up

The reason we need to introduce network traffic shaping with 10Gb networking is because uplinks can (and will) get saturated. For some reason some folks think that the limits of 10Gb will never be reached and a pair of 10Gb NIC’s on each host (with no traffic shaping) will suffice. A good example of where saturation can occur is vMotion, see the graph on the left (click to enlarge). vSphere 4.0 vMotion traffic can achieve up to 2.6Gbps per vMotion (maximum of two concurrent) so it would never max out a 10Gb link, but since vSphere 4.1 a single vMotion can now achieve up to 8Gbps, and can concurrently perform eight vMotions. In fact using vMotion on vSphere 5 can theoretically achieve up to 40Gbps with multi-NIC vMotion (using multiple vmKernel interfaces across four adapters) – by the way I’m not aware of this being tested, but it would be interesting to see it in action. Now that is pretty impressive if you want super fast vMotion, but lets not sacrifice other traffic such as our storage (NFS, iSCSI), FT logging or production networking. The question I always ask is what are the requirements we’re putting into our design? There is no right or wrong answer of how our traffic needs to be prioritised but there are some good guidelines available.

So, enough talk… lets design our 10Gb converged infrastructure for vSphere!

There are a number of methods you can use to prioritise your network traffic, and these are (in my opinion) the most popular:

• Network I/O Control (NetIOC) with vSphere 4.1 (or higher) vNetwork Distributed Switch
• Cisco Nexus 1000V – QoS
• Cisco UCS – QoS (on the Virtual Interface Card)

Lets look at each of them in more detail before I conclude with my recommendations.

NetIOC

Since vSphere 4.0 we have been able to configure traffic shaping policies on the vDS (Ingress and Egress) and we are able to control Average Bandwidth, Peak Bandwidth and Burst size. This is no match for traffic shaping for 10GB networking. NetIOC, however, which was made available in vSphere 4.1 (with Enterprise Plus) allows for a form of QoS tagging (Quality of Service) using ‘shares’ and shapes traffic leaving the host. The diagram (left) from What’s New in VMware vSphere 4.1 Networking illustrates this.

Shares and limits define the prioritisation for the following pre-defined network resource pools:

• vMotion
• iSCSI
• NFS
• FT Logging
• Management
• Host based replication (vSphere 5)
• VM network traffic

Note: vSphere 5 NetIOC also allows for user defined network resource pools.

What I really like about NetIOC is that it uses the concept of Shares and Limits which you may already be familiar with (I.e CPU and Memory), and it’s very easy to use and setup (you access it from the Resource Configuration tab of your dVS). VMware have published an excellent whitepaper on NetIOC which I recommend reading, even if you’re not planning on using it, as it details some of the reasons for traffic shaping that I’m outlining in this blog post.

Cisco Nexus 1000V – QoS

Whilst I’m not a networking expert, my background in previous roles meant that I was heavily involved with supporting customers using Cisco products, managing 4500 and 6500 switches, Cisco PIX (yes it was a long time ago!) and ASA, and general networking. As a VMware consultant I find the Cisco Nexus 1000V to be extremely powerful and I see a growing number of organisations using it which is fantastic as I’m a big fan of Cisco products. Network folk will love using the  Cisco Nexus 1000V as it hands control back to them and it is what they know and love, especially shaping traffic with QoS. Since version 1.4 the N1KV has a feature called CBWFQ (Class-Based Weighted Fair Queuing) which is similar to using shares and essentially allows you to define traffic classes based on match criteria. It also ties in very well with Cisco UCS (see the next section). If you are familiar with Cisco products then using class-maps and matching them to a protocol is nothing new, and if you’re new to Cisco products then it’s simply awesome

The following pre-defined matches are available:

• n1k_control
• n1k_mgmt
• n1k_packet
• vmw_ft
• vmw_iscsi
• vmw_mgmt
• vmw_nfs
• vmw_vmotion

It goes without saying that you can create your own user-defined matches, but the main differentiation between using the Nexus 1000V and NetIOC is where and  how the traffic shaping takes place. Essentially the Nexus 1000V will mark CoS tags on upstream traffic, NetIOC won’t do this, so ingress network traffic could potentially saturate the 10Gb adapters. Brad Hedlund has an excellent post on exactly this topic, and I have included an snippet from his article below which highlights some important points:

… the Nexus 1000V can apply a bandwidth transmission policy while at the same time marking the traffic for consistent treatment in the upstream network. In contrast VMware Network I/O Control does not mark any traffic. As a result the upstream network must either independently re-classify the traffic or not do anything at all and just “let it rip”. If you do the later “let it rip” approach you will have a situation in which there are no controls in how the network transmits traffic to a server. As a result you can either overwhelm your server with receive traffic and defeat the purpose of implementing NetIOC to begin with. Or, you can implement receive based rate-limiters on every VMware host in fear of excessive traffic and again defeat the purpose of intelligent bandwidth sharing that NetIOC sets out to accomplish.

Cisco UCS Virtual Interface Card (Palo) – QoS 

If you are using Cisco UCS then you can configure QoS with the Palo / VIC (Virtual Interface Card). There are in fact two ways of doing this; one is to present vNIC’s to the hosts each with a QoS policy defined, or you can leverage the Nexus 1000V using that to define the CoS tags, and the Cisco UCS will honor those tags rather than the VIC defined CoS. Each has it’s benefits and will again depend on your requirements. If you use the Nexus 1000V as described before, but use the QoS capability of the Virtual Interface Card (Host Control setting), you only need to present two vNIC’s to each host, and the Virtual Interface Card will still process the QoS buffers taking full advantage of the Cisco hardware.

Configuring QoS on Cisco UCS Virtual Interface Card is pretty straight forward to do, as it has six pre-defined system priorities:

• Platinum
• Gold
• Silver
• Bronze
• Best effort
• Fibre Channel (notice it is aware of fibre channel, unlike Nexus 1000V and NetIOC)

Essentially you create your own QoS Policies and select a priority (E.g. Platinum or Gold) for each QoS Policy you define, along with options such as burst (in Bytes), rate (Kbps) and Host Control (setting to Full allows for use of Nexus 1000V). Each QoS Policy you create is then assigned to a vNIC template (one per vNIC). For example, you might end up with six vNIC’s:

• vNIC vmnic0-mgmt
• vNIC vmnic1-mgmt
• vNIC vmnic2-VM
• vNIC vmnic3-VM
• vNIC vmnic4-vmotion
• vNIC vmnic5-vmotion

This would present 6 NIC’s to each host (3 per fabric interconnect via the IO module), unless you are also leveraging the Nexus 1000V in which case your QoS is configured on the N1KV and you only need to present 2 x 10Gb NIC’s (one for each fabric interconnect). In addition you would create a QoS policy for your fibre channel storage, and that’s it, simple.

Conclusion

There is no single design or best practice that fits all, but there is guidance that will help you make the right choices for your environment. Neither the Cisco Nexus 1000V or NetIOC are not able to prioritise FCoE traffic as essentially this is fibre channel as far as they are concerned, so keep this in mind when using QoS on the Nexus 1000V or with NetIOC. You don’t want to end up with a situation where you’re CoS tagging traffic but FCoE could still impede on the 10Gb link.

The Nexus 1000V is really suited well when network management is the responsibility of the network team, or when you require not only QoS on egress traffic (leaving the host) but want to apply CoS tags for ingress. In addition, using the Nexus 1000V for QoS tagging means that you only need to present 2 x NIC’s to each host whereas using Cisco UCS Palo card (Virtual Interface Card) for QoS you’ll need to present 6 or more NIC’s for each type of traffic (on each fabric interconnect).

IMHO it is better to have implemented traffic shaping than use nothing at all, so if you choose to use NetIOC then you’ve made an excellent choice. This is where some say NetIOC only solves part of the problem, so why wouldn’t you want to shape both egress and ingress traffic? Just bear in mind what the differences are as I’ve outlined.

Whichever method you use to shape your network traffic for 10Gb networks, here are some guidelines and key considerations:

• Use QoS!
• vMotion requires a minimum of 1Gb and it will spike (peak in bandwidth).
• Set aside 3Gb for storage networking (NFS, iSCSI)
• Use 2Gb for FT Logging
• Use 2.5Gb for production VM traffic (depending on your production networking load)
• Use 0.5Gb for Management
• If FCoE is on the same 10Gb links then leave room for this when prioritising your network traffic (Cisco Nexus 1000V or NetIOC cannot see FCoE).
• NetIOC allows for shaping on egress network traffic using shares (minimum bandwidth) and limits (Mbps)
• Use the Nexus 1000V to define the CoS tags
• In this scenario you only need 2 x vNICs per host
• Set Host Control to FULL to allow the Cisco UCS to use CoS tags defined in the Nexus 1000V
• If using Cisco UCS Virtual Interface Card, then define QoS policies for each traffic type
• Use at least 3 x vNIC’s per fabric interconnect (total of 6 x vNICs), this will provide a pair of vNICs for vMotion, VM network and Management.
• Set Host Control to None

The important thing to remember is the values stated here are for guidance only and have been derived from various sources and feedback from customer implementations I have worked on. The illustration to the left shows how a 10Gb interface can be prioritised for vSphere traffic. If your design doesn’t require NFS or FT Logging then you can tweak these values to your own requirements, it really depends on each environment. Doing nothing at all with vSphere 4.1 or later is not recommended as you now know, because vMotion can end up saturating the 10Gb links.

I’ve focused on the Nexus 1000V, Cisco UCS Virtual Interface Card and VMware NetIOC. I have chosen to focus on these three technology sets because of their growing popularity and my present involvement working on Vblock implementations. However, I hope you can apply this knowledge to your designs or existing infrastructure.

I’d like to leave you with some additional links on the subject which you may find useful:

• UCS vNIC Calculation
• Cisco UCS Info by Aaron Delp
• Concept Behind vNIC and vHBA
• VMware 10GE QoS Design Deep Dive with Cisco UCS and Nexus by Brad Hedlund
• Configuring Cisco QoS Weighted Fair Queuing
• Nexus 1000V Best Practices on Cisco UCS B and C Series

Since the days when exams were written with chalk and slates and blog posts were cave paintings, it is obligatory to share the experience of taking exams within the community. I hope the title didn’t get you too excited as I signed an NDA and really can’t tell you how to pass this exam. But, what I can do is give you advice and help you focus your study where it really matters. For starters, if you are reading this then you are probably wondering about the VCP5 and the VCAP5 exams. I wouldn’t blame you if you are opting to hold on for the release of the VCAP5 exams, but as it stands whilst I write this post we have no idea when they will be released. It’s likely to be next year sometime, but that is a pure guess. Gregg over at TheSaffaGeek has already started compiling some material to help you with studying for the VCAP 5 exams. However, if you have decided to jump straight in and sit the VCAP-DCD4 (VDCD410) exam then here are my thoughts.

This exam is HARD-ass. There are a few peeps that say it’s easy, but I personally found this harder than the VCAP-DCA due to the shear number of questions (113 in total as stated in the blueprint). If you are a native English speaker then you get 3 hours 45 minutes (4 hours for non-native), and the key to passing this exam is primarily being able to skim-read a case study or scenario and understand design requirements, constraints, risks, assumptions and translate these into one or more of the possible answers. If you spend time reading each question in detail then you are likely to run out of time. It’s also a hard exam to study for because it tests your general experience with vSphere and design knowledge, so you are not just remembering where something is configured.

I don’t want to repeat what a Google search will already reveal as there are plenty of VCAP-DCD study resources out there (I’ve provided my favorite links below), but one significant study aid is the VMware vSphere Design Workshop course, especially if you are not familiar with the VMware design methodology. Another valuable resource is the VMware vSphere HA and DRS Technical Deepdive by Duncan Epping and Frank Denneman. I’m not just saying that because they’re great guys, this book really did help me with the exam. Understanding some of the advanced settings and HA best practices is an important skill to learn, and this book covers much of what you need to know.

You will be tested on some technical knowledge, but the main factor required to pass this exam is based on your ability to map business requirements to a valid vSphere solution. Unlike the VCAP-DCA exam where you need to learn how to configure multipathing options from the command line for example, the VCAP-DCD requires you to understand which circumstances you would need to do this and which options you  need to configure based on customer requirements.

Do you need a home lab for this exam?   No. For the VCAP-DCA you certainly need a lab environment but the DCD exam tests your understanding of vSphere in-depth and many of the best practices surrounding it. Time to start reading I’m afraid!

The exam format contains a mix of multiple choice, drag-and-drop and design (Visio style) questions. If you do lots of vSphere designs and you use Visio a lot then you’ll be fine with this, but one free tool that worked very well for me is the drawing app with Google Docs. If you have a Google account, just head to Docs and select Create > Drawing. This is actually an excellent little tool and alternative to Visio and you can practice some vSphere designs. I even managed to get vNIC’s looking like a NIC card, good fun too!

My final tip is not to panic. Whilst I said this exam is HARD-ass, it is not impossible. If you fail it?   Then at least you have experience of the exam and know where to focus your learning for next time.

Good luck!!

My VCAP-DCD Study Guides and Resources

http://thesaffageek.co.uk/vcap-dca-dcd/
http://www.vmware.com/resources/techresources/10166
http://www.virtualsmb.net/?p=109
http://vmwaredevotee.com/2010/10/11/cpu-performance-monitoring-on-esx-and-vms/
http://www.virtualizationteam.com/virtualization-vmware/certification-virtualization-vmware/vcap-dcd/my-vcap-dcd-journey-tips.html
http://www.seancrookston.com/2011/09/13/vcap-dcd-tips-and-exam-experience/
http://www.seancrookston.com/vcap-dcd-index/
http://professionalvmware.com/2011/09/brownbag-follow-up-vcap-dcd-objective-1-jason-boche/
http://communities.vmware.com/docs/DOC-9279
http://mylearn.vmware.com/courseware/82525/VCAPDCD_Tutorial.swf

Today I took the VCP510 exam and passed, so I thought I would share my thoughts on this latest certification by VMware. Firstly, I’m in the middle of the VCAP exams, having just completed the VCAP-DCA4 and will soon start studying for the VCAP-DCD4. Why take the VCP exam based on a new version now?   Well, mainly because I’ve already been on the VMware vSphere: What’s New [V5.0] course and most of the contents are still fresh in memory. Secondly having passed the VCAP-DCA4 exam, with the next one being design focused I can pretty much focus on design best practices regardless of the version (although there are obvious differences).

After spending a week working down-under in Perth, I managed to get some time off this week to recover from the jet lag, so what better way than spending a few days refreshing my memory and doing LOTS of lab work with vSphere 5!

Much of my focus was around the vSphere Storage Appliance (vSA), vCenter Server Appliance and Auto Deploy and I must admit after reading some other comments I didn’t focus too much on configuration maximums (still glanced over them).

I can’t go into the specifics of the exam, as per the usual NDA, but here are my thoughts…

It’s Difficult

This was much tougher than I thought, perhaps a combination of jet lag, and maybe VCAP-DCA/DCD4 study floating around in my head didn’t help. However, don’t underestimate it. It is a little more difficult than previous VCP exams (3/4). There were a good few questions that I marked for review and couldn’t decide between some of the options, but certainly more hands-on lab work would have helped. I was OK with the time allocated (you get 90 minutes to answer 85 questions), but I do tend to rush quickly through each question and review them all at the end.

Preparation Material

One tip which will save your ass is to spend some time reading up on the new features, but more importantly implementing them for yourself!  Many questions are aimed at your experience with the product, not just specifics like config maximums and what you’ve read in the PDF’s. I use Dropbox and store all of the vSphere 5 documents so I have them wherever I go. If you are lucky enough to have a vSphere 5 environment to hand then spend most of your time implementing the new features, understand how to configure them and what the limitations are.

My second tip is to read the vSphere 5 Clustering Technical Deepdive by Duncan Epping and Frank Denneman. I purchased the Kindle version from Amazon so I can read on my iPad, but you can get it in paperback too. This contains a lot of valuable information for this exam, especially as HA is completely new and you will likely get tested in this area.

There is also a vSphere 5 training series by TrainSignal which is excellent especially if you are new to virtualisation and vSphere 5 is your first step into the world of cloud computing.

Final Thoughts

VMware continue to have the best certifications in the industry which are highly respected and sought after, and having done various other VMware exams in the past it is clear than they are continuing to push us individuals further and further which maintains a very high standard of competency in our industry. If you are serious about a career in virtualisation then if you only listen to one thing I’ve said here, then it is this… Get a home lab!  If not for the VCP5, you’ll certainly need it for the VCAP-DCA exam, but most of all you’ll get a lot of satisfaction out of working in your own virtual infrastructure that YOU have created.

Good luck!

Also, check out StratoGen who are a leading cloud provider, a VMware Enterprise Service Provider and have some hardcore skills to including my pal Simon Greaves who recently joined them. If you need VMware Hosting, then these are the people you need to speak to!

I’m actually really looking forward to the winter this year, and it seems that 5 is the lucky number. The new iPhone 5 will be released, rumours suggest this will be on or around 15th October, and then there is the iPad 3 (ok, it’s not 5 but worth mentioning!) which should have a far superior retina display which will be great for reading PDF’s and studying with! I’m still using the iPhone 3GS and iPad 1 so my Xmas list is easy this year, whether I get what I want depends on how good I am!

In the virtual world, at least inside the cloud, VMware have released vSphere 5 and View 5, in addition to many other new products. With the up and coming release of Microsoft Windows 8 with it’s incredible boot time thanks to the way it shuts down in a hybrid shutdown / sleep state, I can only imagine the benefits this will bring to desktop virtualisation and end-user computing as a whole.

So what’s the deal with vSphere 5?  Well, it’s a big deal indeed. VMware are no longer offering ESX with it’s service console, so you’ll have to use ESXi. The difference? Well, ESXi doesn’t have a service console (as such) as like ESXi 4 it has a DCUI (Direct Console User Interface) which is very limited but there if needed. Management of ESXi should be performed using the vMA (vSphere Management Assistant) which by the way is excellent and you’ll begin to wonder why on earth you needed ESX’s service console in the first place  Needless to say you can automate many tasks with PowerCLI. Here are some of the new features of vSphere 5:

• VMFS 5 (now with unified 1MB block size and volume size of 60TB supported!)
• Storage DRS
• ESXi firewall
• vSphere 5 Web Client (you’ve gotta see this, it’s great!)
• Virtual hardware version 8
• vSphere Storage Appliance (vSA)
• … the list goes on!

The VCP510 exam (vSphere 5) is now available and if you take this before 29th February 2012 you won’t need to take any course prerequisite which is good to hear. After that date though you must take the pre-qualifying course. My colleague Simon Long at VMware has put together a VCP5 practice exam (good work dude!) so definately check this out.

So what’s next?   Well for me at least, I’ve got plenty of work to do. I need to site my VCAP-DCD exam, but I’m mega busy working on Vblock deployments with VCE at the moment. It’s good though as I get to travel the world and I’m off to Australia soon. The downside is studying for any exam is something I have to do in my sleep. I think 2012 will be a jam packed year for sure, with even more amazing products from VMware and even more certifications to achieve!

Networking is a critical component of any virtual infrastructure, and often it’s the management networks that are overlooked. Back in the day before virtualisation, management networks were considered less important than production, and it wasn’t too much of a big deal as it only provided console access (E.g. iLO, DRAC), SNMP monitoring, web interfaces, and so on – it just didn’t impact production. I have noticed that this mindset has crept into some vSphere designs where management interfaces lack any form of redundancy. Why is this so important?   Well for starters ESXi uses the Management network for vMotion, Fault Tolerance (FT) and HA. In fact if your management interface has no redundancy you’ll get a warning as described in VMware KB 1004700. Without redundancy on your management networks, these features will not work. In addition, you may be faced with mixed ESX and ESXi environments where the management networks are different between ESX/ESXi. Management on ESX uses the Service Console network, but ESXi uses a vmKernel network called Management Network. Duncan Epping of Yellow Bricks has an excellent article on VMware HA here, also check out Frank Denneman and Duncan Epping’s HA and DRS Technical Deepdive book is an excellent read and I recommend this especially if you’re studying for the VCAP-DCA.

Knowledge Required

• Identify common virtual switch configurations

Key Focus Areas

• Determine use cases for and apply IPv6
• Configure NetQueue
• Configure SNMP
• Determine use cases for and apply VMware DirectPath I/O (this is covered in Implement and Manage Storage, part 1)
• Migrate a vSS network to a Hybrid or Full vDS solution – Coming soon
• Configure vSS and vDS settings using command line tools - Coming soon
• Analyse command line output to identify vSS and vDS configuration details - Coming soon

Key Materials (VMware PDF’s & KB articles)

1. vSphere Command-Line Interface Installation and Scripting Guide
2. VMware vNetwork Distributed Switch Migration and Configuration
3. ESX Configuration Guide
4. ESXi Configuration Guide
5. IPv6 Support in vSphere by Eric Siebert
6. VMworld 2009: vSphere Networking Deepdive

Determine use cases for and apply IPv6

So what is the use case for IPv6?  I can think of only two scenarios, one is that you’re insane and have run out of IPv4 addresses, the other is the company infrastructure policy requires all systems to have IPv6 enabled, not that unreasonable eh?    Good job it’s easy to configure

Remember that even when IPv6 is enabled on your ESX/ESXi host, your network infrastructure must support it. This includes your DNS and DHCP servers, routers, switches, etc.

1. Using the vSphere Client, select your ESX/ESXi host and go to the configuration tab, select Networking and click Properties.

2. Tick ‘Enable IPv6 support on this host system’, click OK and reboot your host.

3. Have a coffee and explain to your colleagues how IPv6 works

You can also enable IPv6 from the DCUI, after setup completes by pressing F2 > Configure Management Network > IPv6 Configuration, then enabling IPv6:

Alternatively you can enable IPv6 from the command line. Follow these steps:

1. Enable IPv6 for the vmKernel

# esxcfg-vmknic -6 true

2. Enable IPv6 for the Service Console (ESX only)

# esxcfg-vswif -6 true

3. Reboot the host for the changes to take effect.

You can tell when IPv6 is enabled because the DCUI screen shows the management URL in IPv4 and IPv6 format:

You will also see IPv6 settings in the Management Network properties:

Configure NetQueue

When the physical NIC on the ESX/ESXi host sends or receives packets (TX/RX) it enters a single TX or RX queue which the vmKernel schedules on a single CPU. Without NetQueue, the result that is noticed on 10GbE NIC’s is the throughput isn’t optimal and the single TX or RX queue causes a bottleneck. NetQueue allows for multiple TX/RX queues which are scheduled on multiple CPU’s. A single VM that places heavy demand on bandwidth will still have it’s queue processed on a single CPU, but other VM’s have their own queue processed by a different CPU. The result of using NetQueue is 10GbE interfaces will be able to maximise the throughput available.

Note: It is enabled by default if your hardware supports it. Your physical NIC must support NetQueue in addition to your system supporting MSI-X.

Advantages:

• Lower CPU utilisation
• Higher network throughput, especially on 10GbE interfaces
• Better scaling and load balancing across CPU’s

Requirements:

• NIC hardware must support NetQueue
• System hardware must support MSI-X

You can enable or disable NetQueue in the Advanced Settings of your host:

It can also be enabled or disabled with the esxcfg-advcfg command:

# esxcfg-advcfg -k 1 netNetqueueEnabled (1 to enable, or 0 to disable)

Configure SNMP

You can configure an ESX/ESXi host with SNMP is minutes, it’s that easy. If you want to test this in your home lab then you’ll need an SNMP receiver, typically a monitoring tool, or you can download an evaluation of Kiwi Syslog which you can use instead as it has an SNMP receiver service. Using Kiwi Syslog, go to File > Setup > Inputs > SNTP. Tick ‘Listen for SNMP traps” and click OK.

Using the vMA run the following commands to point the ESX/ESXi host at an SNMP server and set the community string, enable SNMP (disabled by default), and then run a test. Kiwi Syslog will show the traps received so you can see it working.

# vicfg-snmp --server core-esx.home.lab --username root --password ******* -t 192.168.4.40/public
# vicfg-snmp --server core-esx.home.lab -enable
# vicfg-snmp --server core-esx.home.lab --username root --password  -test

Update: VCAP-DCA4 no. 386 (cool!)

Juggling work, travel, kids, and personal time, whilst trying to study for any exam is no easy feat especially when you have a family that demands your undivided attention. I’m sure many of you are all in the same boat. Despite that I managed to squeeze in a lot of time into my home lab and study for the VCAP-DCA exam, so it is possible (just damn difficult!).   So to set the scene, I woke up early in a hotel in Ireland yesterday following an intense but enjoyable week working on Vblock, and I noticed an email from VMware certification on my phone…   Opening that email and the attachment with the results seemed to take forever, but what a relief to see that I passed.

I can’t share specific details of the exam as I’d be tracked down and assassinated, but I can share my experience and study tips to help you.

The Exam:

Unlike the majority of IT exams, this is based on a live lab environment which you have access to along with a set of tasks that require you to put your skills and experience to the test. The blueprint states that there are approximately 40 questions / tasks that you are required to complete, I had slightly less, around 34 I think. Some tasks are independant of the others but some must be completed before you are able to perform later tasks. The exam lab environment was very similar to my home lab so I strongly recommend you invest in a home lab setup if you are serious about taking this exam. Unless of course you have access to a lab environment at work.

Tip 1: Don’t underestimate the content of the blueprint, and if it’s on the blueprint then it’s likely to be in the exam! Highlight areas of the blueprint that you are confident with, and those areas you are weak. For me, I hadn’t spent much time with vCenter Orchestrator or PowerCLI so that was my focus. I love the vMA and felt comfortable with that so I was able to spend less time studying this topic.

Know Your Enemy…. The Clock!

When you start the exam you’ll first be required to complete the short survey based on your skills and exposure to various technologies. Following that you can start the exam. On the top right hand corner you’ll see the time remaining, and if you are a native English speaker you will have 3 hours 30 minutes (4 hours for non native speakers). On clicking the begin button the PC I was using crashed and I had to get the test centre administrator to get me setup on another PC… which also crashed!  After some time waiting for Pearson VUE support to sort out the issue, I lost some valuable time (only a few minutes) so I felt I had to run through the tasks quickly to make up for it. This brings me on to my second tip:

Tip 2: Pick the low hanging fruit first, then go back and tackle the big ones.

By the way you’ll get access to VMware PDF’s, a plastic writing card and a pen. Before starting the exam I recommend you write 1 – 40 on the plastic card so you can quickly mark them off as you progress through the exam. Every minute counts!

Do as many tasks as you can that you are comfortable with, these will give you valuable points. If you think a task is going to require some thinking and more time then make a note and move on. This way when you finish the tasks you can go back and tackle the more difficult ones. I did this and reduced eight outstanding tasks down to two very quickly, but then I ran out of time. I didn’t feel as bad because I knew the remaining two tasks would have taken me another 40 minutes but at least I looked at all of the tasks.

Know your stuff!

This seems an obvious statement, but you should be comfortable with performing any task on the blueprint. Sure you may not be an expert in every single area, but you should be able to carry out what is required on the blueprint.

My Experience

Overall I really enjoyed taking this exam, but I’m glad I’ve passed and don’t have to take it again. I spent around 5 hours a day for 5 days leading up to the exam, to practice carrying out tasks in my home lab. Most of this was late into the night once my wife and daughter were fast asleep!  I’ve heard comments from others that have taken this exam saying that the environment was very slow and opening the PDF’s took a long time. To be honest I found it was acceptable considering it’s accessing a live vSphere environment somewhere (probably in the US) and it didn’t impact my experience in any negative way.

I only used the PDF’s once to look up a setting to ensure I got the syntax correct, but other than that try not to rely on the PDF’s as it’ll cost you valuable time. Some commands and PowerCLI can easily be typed incorrectly or worse still your mind can go blank. If you’ve got any advanced settings, PowerCLI or vMA commands that you think you may need then write them down on the plastic card before starting the exam. This will save valuable time, and when under pressure you won’t have to worry about forgetting the small details. Is it -n or -d, or both?  You get the picture.

Final Tip… no. 3:

If you haven’t taken this exam yet, just remember… don’t panic!  What’s the worst thing that can happen?   If you fail then you’ve got the exam experience and will be able to focus your study on what you need to pass. It’ll take around 10 business days to get the result (3 weeks for me), but it’s what makes the VMware and the community we all participate in so great.

Oh and if you can get yourself booked on the VMware vSphere Troubleshooting course then it will go a long way to helping you pass this exam. It’s a lab intensive course and very enjoyable. Alternatively check out what’s on offer by TrainSignal.

Good luck!

This section on storage continues on from section 1.2 in the blueprint (Manage Storage Capacity in a vSphere Environment) which at the time of writing these study notes, I haven’t completed yet. I felt that managing multipathing and PSA plugins deserves more attention, at least for me anyway. This is very command line heavy but remembering that during the VCAP-DCA exam documentation is provided (see key materials below), and the fact you can use command line help makes this a little less scary. Just try and remember what you need to achieve and have a good idea of which commands are used!

Knowledge Required

• Explain the Pluggable Storage Architecture (PSA) layout

Key Focus Areas

• Install and Configure PSA plug-ins
• Understand different multipathing policy functionalities
• Perform command line configuration of multipathing options
• Change a multipath policy
• Configure Software iSCSI port binding

Key Materials (VMware PDF’s & KB articles)

1. iSCSI SAN Configuration Guide
2. Fibre Channel SAN Configuration Guide
3. vSphere vSphere Command-Line Interface Installation and Scripting Guide

Install and Configure PSA plug-ins

The most common PSA plug-in is EMC PowerPath which is what I’ll base this on. You can use the vMA (which I prefer) or vCLI.

1. Download EMC PowerPath for VMware from https://powerlink.emc.com (you will need to register for an account).
2. Extract the .zip file and copy EMCPower.VMWARE.5.4.SP2.b298.zip from the folder to your vMA (use WinSCP).
3. Place the host into maintenance mode.
4. Make sure that there is no previous installation of PowerPath/VE:

# vihostupdate --query --server <hostname>

5. Install the PowerPath plugin:

# vihostupdate --server <hostname> --install --bundle EMCPower.VMWARE.5.4.SP2.b298.zip

6. Reboot the server.
7. If you run vihostupdate query again you should see that PowerPath is installed.

Understand different multipathing policy functionalities

The knowledge required according to the VCAP-DCA blueprint on this topic is to understand Pluggable Storage Architecture (PSA). The following sections will detail how to use the CLI, so grab a coffee as we dive into PSA!

As part of the VMkernel the PSA provides multipathing using either the Native Multipathing Plugin (NMP) or third-party multipathing plugins (MPP) such as EMC PowerPath. The NMP has two plugins it uses which are key to this topic; SATP (Storage Array Type Plugins) and PSP (Path Selection Plugins). If you haven’t seen it already, there is a well used diagram on these by VMware, which you’ll find on page 24 of the Fibre Channel SAN Configuration Guide:

The SATP (Storage Array Type Plugins) are responsible for monitoring the health of each path, reports changes to the state of each path, and will activate a passive path if required (active-passive arrays). Ed Grigson of vExperienced said the SATP is like the traffic cop, which is a good analogy.

The PSP (Path Selection Plugins) are responsible for choosing which path to use for I/O requests. The following PSP’s are available:

• Most Recently Used (VMW_PSP_MRU) – Uses the path that was most recently used to a given device.
• Fixed (VMW_PSP_FIXED) – Uses a preferred path unless it’s down in which case it will select an alternative path at random. If a preferred path isn’t configured then it will used the fist path discovered at boot time. * See my note on path thrashing with this policy.
• Round Robin (VMW_PSP_RR) – Enables load-balancing across the available paths using a rotating method.
• VMW_PSP_FIXED_AP – Fixed functionality for active-passive and ALUA arrays.

Perform command line configuration of multipathing options

The VCAP-DCA blueprint for section 1.3 only mentions use of esxcli, so this will be the focus of these study notes. Remember when using esxcli with vMA, you’ll need to add --server <hostname>, otherwise it will target the localhost (even if you’ve done vifptarget -s). I’ve also completed a section on esxcli corestorage which you’ll find in part two of my Implement and Manage Storage guide. Here are the available esxcli namespaces:

• corestorage (covered here)
• network
• nmp *
• swiscsi (detailed in the next section, see below)
• vaai
• vms

* esxcli nmp relevant for this section of the VCAP-DCA blueprint.

Using esxcli nmp is straight forward enough, just type it alone and it’ll present you with a list of available objects (see screenshot):

As this section of the blueprint is multipathing and PSA plugins, we’ll get started with listing the paths available on the ESX/ESXi host:

# esxcli nmp path list | less (can pipe to more, but I prefer less

You can also list the devices controlled by the nmp plugin:

# esxcli nmp device list | more

Setting the PSA Multipathing Policy with esxcli

1. First, list the path selection plugins available:

# esxcli nmp psp list

2. Next, we’ll list the available devices so we can choose which one to set the multipath policy against. Tip: If you know the naa ID of the device, you can pipe this command to grep (see screenshot).

# esxcli nmp device list | grep <naa ID>

3. Next, we’ll set the multipathing policy for a given device.

# esxcli nmp device setpolicy --device naa.<ID> --psp VMW_PSP_???

So you can see with three commands, we can list the psp plugins, list the devices and use grep to filter to a specific device, then set the policy to another psp plugin.

Tip: If you get stuck in the exam, remember this is detailed in the vSphere Command Line Reference (now vSphere vSphere Command-Line Interface Installation and Scripting Guide).

Configure Software iSCSI Port Binding

To configure port binding for software iSCSI you’ll need to create two VMkernel ports and bind them to separate NIC’s. There are two ways of doing this, which you’ll see on page 38 of the iSCSI SAN Configuration Guide; a single vSwitch with two VMKernel ports and two physical NIC’s, or a separate vSwitch for each VMkernel port, each with it’s own NIC. It really doesn’t matter which you choose as the next step will be to bind the iSCSI ports on each NIC to each iSCSI adapter. Once you’ve done this, configure your software iSCSI adapter as usual. If you are not sure how to do this then refer to the iSCSI SAN Configuration Guide. Once your iSCSI LUN is presented we’ll do the rest from the command line.

1. Check which HBA your software iSCSI adapter is using from the vSphere Client:

2. From the command line, run the esxcli swisci command to confirm now NIC’s are already bound to the adapter.

# esxcli swiscsi nic list -d <vmhba> (E.g. esxcli swiscsi nic list -d vmhba35)

3. Next, we’ll add one of our VMkernel ports (E.g. vmk1) to the software iSCSI adapter (E.g vmhba35).

# esxcli swiscsi nic add -n vmk1 -d vmhba35

4. Do the same for the second VMkernel port.

# esxcli swiscsi nic add -n vmk2 -d vmhba35

5. Next, run the esxcli swiscsi command again to check the port bindings:

# esxcli swiscsi nic list -d vmhba35

6. Rescan the vmhba (E.g. vmhba35).

# esxcfg-rescan vmhba35

That’s it, both VMkernel ports are now bound to the software iSCSI adapter.

TIP: Use esxtop/resxtop to see traffic flowing through these VMkernel ports. In the following screenshot, I’ve started a storage vMotion to an iSCSI datastore and you can see it is using vmk1. Normally you would see two separate NIC’s under the TEAM-PNIC column, but  I just wanted to show you that you can see the traffic flowing in esxtop. Also, if you have two vmnic’s assigned and one fails then the TEAM-PNIC column will show which one has failed.

I remember when I first started using Microsoft Cluster Servers with SQL 2000 and Exchange 2003, and I had plenty of experiences (good and bad) especially once when I lost the quorum disk and I was due to go on holiday the next day!   When I saw this topic on the VCAP-DCA blueprint I thought ‘oh’. Funnily enough, whilst I have had plenty of experience with physical clusters, I’ve never had to implement clustering in a vSphere environment. Due to certain complexities of MSCS you can achieve unwanted downtime (such as when I lost the quorum disk, or through mis-configuration) and can be a headache in itself. I’m not sure to what level the VCAP-DCA exam will require us to configure MSCS, but I am confident that it’s just configure the vSphere environment and not the other things you would normally have to do (SAN zoning, shared quorum disk, etc).

The first and only document I will use for this section on MSCS is Setup for Failover Clustering and Microsoft Cluster Service, and it’s only 36 pages so don’t worry you won’t be spending the next three weeks on MSCS alone! In fact, the VCAP-DCA blueprint lists each section in order on this document so it’s a safe bet!

Knowledge Required

• Identify MSCS clustering solution requirements
• Identify the three supported MSCS configurations

Key Focus Areas

• Configure Virtual Machine hardware to support cluster type and guest OS
• Configure a MSCS cluster on a single ESX/ESXi Host
• Configure a MSCS cluster across ESX/ESXi Hosts
• Configure standby host clustering

Key Materials (VMware PDF’s & KB articles)

1. MSCS is Setup for Failover Clustering and Microsoft Cluster Service

Configure Virtual Machine hardware to support cluster type and guest OS

Features such as vMotion are not supported, or running the virtual machines in a HA / DRS cluster (prior to vSphere 4.1). Let me list out the key points for configuring a virtual machine in a Microsoft cluster setup:

• Fibre channel storage only (no NFS or iSCSI) * Local storage can be used on ‘cluster-in-a-box’
• LSI SCSI adapter (LSI SAS for Windows 2008)
• Thick provisioned disks only
• VM hardware version 7

Regardless of virtual or physical hardware, each cluster node must have at two NIC’s; one for the public network (LAN) and the other for the cluster heartbeat (private network).

Limitations:

• iSCSI, NFS and FCoE are not supported
• Mixed versions of ESX/ESXi not supported
• Fault Tolerance (FT) not supported
• vMotion no supported
• N-Port ID Virtualisation (NPIV) not supported
• NMP with Round Robin not supported
• Must have VM hardware version 7
• HA/DRS Clusters are only supported on vSphere 4.1 and above.

Configure a MSCS cluster on a single ESX/ESXi Host

Here are the steps required:

1. Create the first node (VM) with 2 x vNIC’s and the required OS.
2. Connect one NIC to the public network and the other to the private network for the cluster heartbeat (this could be an internal only vSwitch)
3. Install Windows Server on the first node.
4. Clone the first node (or create a template) to create a second node.
5. On the guest operating system customisation, make sure to select Generate New Security ID (SID).
6. Finish cloning the second node.

Create a Quorum Disk

1. Select the first node and add a new hard disk (1GB should be enough), make sure to select ‘Support clustering features such as Fault Tolerance’.
2. Select a location for the quorum disk, shared storage or local. * If using shared storage, the second node must match the exact same location.

2. Select a new SCSI device (E.g. SCSI 1.0)

3. Finish the wizard.
4. Edit the SCSI Controller type and make sure it’s set to LSI Logic Parallel (Windows Server 2003) or LSI Logic SAS (Windows Server 2008), and set SCSI Bus Sharing to Virtual.

Adding the Quorum Disk to the Second Cluster Node
1. Go to the virtual machine settings of the second node and add a hard disk.
2. Select ‘Use an existing virtual disk’.
3. Select the same virtual device node as the first virtual machine (E.g SCSI 1:o).
4. Browse to the location of the quorum disk.

Configure a MSCS cluster across ESX/ESXi Hosts

This is very similar to the single box solution so I won’t detail every step. Each node resides on a separate ESX/ESXi host. Take note of the following differences:

• Create the quorum disk as an RDM in physical compatibility mode.
• The ESX/ESXi host must have at least 3 physical NIC’s; Two for the MSCS cluster (one public and one private heartbeat) and one for ESX Service Console (or ESXi VMkernel for Management).
• SCSI Bus Sharing must be set to Physical (not virtual, because we’re clustering between two physical ESX/ESXi hosts).

Steps:

1. Create the first virtual machine (first node) as before, but don’t add the shared quorum disk yet.
2. Clone the first node to the second node, and place it on the second ESX/ESXi host.
3. Generate a new Security ID (SID) as before.
4. Add the quorum RDM to the first node (physical compatibility mode) with a new virtual SCSI device (E.g. SCSI 1:0).
5. Set SCSI bus sharing to Physical.
6. Add the quorum RDM to the second node, same process.
7. Optionally add any other shared storage to each node.

Configure standby host clustering

With this configuration, the first node is a physical machine and the second is virtual. Here are the differences you need to be aware of:

• Create the physical server first and attach the storage before creating the second (virtual) node.
• Create the second (virtual) node and add the RDM that is presented to the first (physical) server using physical mode.
• Configure the virtual adapters for the second node with the private heartbeat and public network (LAN).
• Set SCSI bus sharing to Physical.

This is the second part to my VCAP-DCA study guide on storage (section 1.1 of the blueprint). As mentioned in part 1, I intend to focus these study notes on what you need to know with essential learning points. There is a big section on LUN masking using PSA related commands in this part of my study guide, and also an introduction to analysing performance stats with esxtop. Whilst esxtop is covered in many of the VCAP-DCA blueprint sections, 1.1 is the first section it is mentioned in regards to assessing storage performance, so at the very end of this post I have included the topic ‘Using ESXTOP for Storage Performance Analysis’.

Implement and Manage Complex Storage Solutions

Part 1 | Part 2

Knowledge Required

• Identify RAID levels
• Identify supported HBA types
• Identify virtual disk format types

Key Focus Areas

• VMware DirectPath I/O (see part 1)
  
• NPIV (see part 1)
• Storage Best Practices (see part 1)
• Raw Device Mapping (see part 1)
• Storage filters
• VMFS resignaturing
• LUN masking using PSA-related commands
• I/O workloads

Key Materials (VMware PDF’s & KB articles)

1. Configuration Examples and Troubleshooting for VMDirectPath
2. Configuring VMDirectPath I/O pass-through devices on an ESX host
3. How to Configure NPIV on VMware vSphere 4.0
4. VMware Storage Best Practices
5. Best Practices for Configuring Virtual Storage
6. Performance Best Practices for VMware vSphere 4.0
7. VMware Multipathing with the SAN Volume Controller and the Causes of SCSI-2 Reservation Conflicts
8. Fibre Channel SAN Configuration Guide
9. iSCSI SAN Configuration Guide
10. Performance Characterization of VMFS and RDM Using a SAN
11. VMware VMFS Volume Management
12. ESX Configuration Guide (RDM, Storage Filters, Resignaturing)
13. Masking a LUN from ESX and ESXi 4.x using the MASK_PATH plug-in
14. vSphere Command-Line Interface Installation and Scripting Guide (good section on LUN masking)
15. Identifying disks when working with VMware ESX
16. Storage Workload Characterization and Consolidation in Virtualized Environments
    
17. Using vscsiStats for Storage Performance Analysis
18. VIFS reference document
19. VMware KB 900: Moving or Copying Virtual Disks in a VMware Environment

Storage Filters

Storage filters are used by default with vCenter to avoid storage corruption by retrieving only the storage devices or LUN’s that can (or should) be used. VMware KB article 1010513 details how to switch off vCenter storage filters, and the ESX Configuration guide (page 133) also contains details on how to do this.

This explanation is taken from the aforementioned KB article:

This LUN filtering mechanism helps prevent LUN corruption that might occur if the following conditions are not met:

• The same LUN cannot be used for a VMFS datastore and RDM simultaneously.
• Two virtual machines cannot have access to the same LUN using two different RDM mapping files.

Follow these steps to switch off vCenter storage filters:

1. Within the vSphere client, go to Administration > vCenter Server Settings, then click on Advanced Settings.

2. Depending on the storage filter you want to add, enter one or both of the following values:

config.vpxd.filter.rdmFilter; false – This filters RDM’s that are already assigned to a VM.
config.vpxd.filter.vmfsFilter; false – This will filter LUN’s with an existing VMFS volume.

Duncan Epping also has an excellent blog post on storage filters, where he details two other filters that can be used:

config.vpxd.filter.hostRescanFilter – Disables the automatic rescan that occurs after a VMFS datastore is added.
config.vpxd.filter.SameHostAndTransportsFilter – This will filter LUN’s that do not have the same masking applied as the original VMFS volume. Also filters which LUN’s are available to add as an extent.

I actually have an excellent use case for the Host Rescan Filter because I experienced some horrendous problems a few years ago where this saved the day. I had a VI 3.5 cluster which was hosted externally and there was a fault with the SAN. Each time a datastore was added, which incidentally was weekly due to DR testing, it caused the database LUN to go down and an externally hosted web application went offline. Not good. By adding config.vpxd.filter.hostRescanFilter to the vCenter advanced settings, it prevented an automatic rescan on all of the hosts when a datastore is added.

VMFS Resignaturing

The topic of VMFS volume resignaturing will usually occur when you have taken a LUN snapshot, for example a DR site may have a replicated LUN, and you need to present an existing VMFS volume to an ESX host (this is done by taking a snapshot of the replicated LUN). This changed since VI 3.5 and it was made easier with ESX/ESXi 4 as described in VMware KB article 1011387. I would recommend that you read page 120 of the ESX Configuration guide, as this contains more information on resignaturing a VMFS volume.

For the VCAP-DCA lab you should understand what to do when presented with a replica VMFS LUN as you’ll have three options:

1. Keep the existing datastore (do not change the signature)
2. Assign a new signature
3. Format the disk (this will create a new VMFS volume)

You need to learn how to list volumes, resignature or mount without resignaturing which is detailed in KB article 1011387. Here are a summary of commands you need to know:
esxcfg-volume -l – This will list all volumes detected as snapshots (replica LUN’s)

Mount the volume without resignaturing the LUN (take the VMFS UUID or label from the output of esxcfg-volume -l):
esxcfg-volume -m <VMFS UUID|label> (Use -M to keep the LUN mounted after host reboot)

Mount the volume and resignature the LUN:
esxcfg-volume -r <VMFS UUID|label>

LUN Masking and PSA Commands

LUN masking is usually performed on the storage array (remember that in the VCP exam?), NetApp uses Initiator Groups, EMC Clariion uses Storage Groups, but you can also perform masking on the ESX/ESXi host using esxcli corestorage claimrule.  Read page 57 of the vSphere Command-Line Interface Installation and Scripting Guide, this is listed in the key learning materials section above (14) and VMware KB article 1009449 also describes masking a LUN from ESX and ESXi 4.x using the MASK_PATH plug-in.

This is a long topic and a complicated process, especially if you plan to perform this on multiple ESX hosts. I’m not sure if this needs to be performed in the VCAP-DCA exam, but I would strongly recommend doing this a few times in your home lab (if you have one).

Two things you should do first:

1. Check what multipath plugins are currently installed on your ESX/ESXi host with: esxcfg-mpath -G

You should see NMP and MASK_PATH

2. Check what the next available rule ID is, using: esxcli corestorage claimrule list

The claimrules for MASK_PATH will start at 101, so as you can see from my screenshot the next rule will be 102.

3. Add the mask rule with:

esxcli corestorage claimrule add –rule <number> -t location –A <hba_adapter> -C <channel> -T <target> -L <LUN> -P MASK_PATH

Example: esxcli corestorage claimrule add --rule 102 -t location vmhba35 -C 0 -T 0 -L 10 -P MASK_PATH

To get the location of the path you want to mask, I use esxcfg-scsidevs --vmfs to find the NAA device and then use esxcfg-mpath -L: to get the paths. Choose the path you want to mask.

i. List the datastores and get the NAA device of the datastore:

esxcfg-scsidevs --vmfs

ii. List the paths for the datastore and that will give you the location:

esxcfg-mpath -L | grep <naa_device>

 4. Verify that the rule has been added successfully:

Note: You’ll see the new rule listed under class type ‘File’.

esxcli corestorage claimrule list

5. Load the claimrules, and list them to see that the new rule is in runtime.

esxcli corestorage claimrule load

esxcli corestorage claimrule list

6. Next we need to disassociate our path from the PSA plug-in which is owned by NMP, this will associate it with MASK_PATH.

esxcli corestorage claiming reclaim -d <naa.ID>

7. Verify that our device is not used by the host, and the LUN is not active.

esxcfg-mpath -L | grep <naa.ID>

esxcfg-scsidevs --vmfs (the device should no longer be listed).

I/O Workloads

This is the final subject on the Implement and Manage Storage section, but measuring I/O workloads is an important part of administering a vSphere environment, not just a topic you should learn just for the VCAP-DCA exam. A great place to start is vscsiStats, a tool that is available on ESX and ESXi (prior to ESXi 4.1 this had to be downloaded to the host). vscsiStats measures I/O on the VMDK (virtual machine SCSI disk) so the underlying storage architecture can be NFS, Fibre Channel, or iSCSI. ESXTOP is another obvious choice for measuring performance, in particular IO workloads but it lacks somewhat as it only provides latency for F/C and iSCSI (not NFS). See the community document listed under my key materials above (17) for further info on vscsiStats.

Note: Section 6.4 of the VCAP-DCA blueprint also has ESXTOP and vscsiStats listed as the skills required for troubleshooting storage performance and connectivity. I feel these are best introduced here and then re-visited again when you start study for 6.4.

Using vscsiStats

vscsiStats will measure:

• I/O Length (size)
• Seek distance (Understand disk seek time)
• Outstanding I/O’s (IO queues)
• Latency (ms) *Most useful statistic
• Interarrival (Measures the time of read/write IO requests and the arrival in Microseconds to the virtual disk)

Using vscsiStats is really easy, and involves the following steps:

1. List the available virtual machines with vscsiStats -l

vscsiStats -l

Note: Find the virtual machine you want to collect statistics for and make a note (or copy) the worldGroupID number.

2. Next, start vscsiStats and specifiy the worldGroupID:

vscsiStats -s -w <worldGroupID>

3. Use vscsiStats -p to specify the metric and print the statistics:

vscsiStats -p latency (type vscsiStats on it’s own to see the help and other options)

Using ESXTOP for Storage Performance Analysis

ESXTOP is a tool based on the ‘top’ command in Linux, but is specifically geared towards ESX/ESXi performance metrics. It’s all over the VCAP-DCA blueprint so you must be comfortable using esxtop for the exam or you might be in trouble. The best place to start is, well… running esxtop!  Seriously it can be a bit daunting at first, but try and use it in a lab environment and understand what the stats are telling you.

Here are the top things you should know about esxtop:

1. Interactive mode (default)
2. Custom configuration – saving your configuration to a file (W to save c to load)
3. Batch mode (esxtop -b <filename>)
4. Identifying I/O latency

esxtop Configuration Files

Once you are familiar with esxtop, you need to know how to save your configuration file using W. You might want to do this because you’ve changed the order of some columns or added some additional fields. The default filename is esxtop41rc (just press ENTER to save the default) and this will load each time at default, otherwise you can press c and load your own configuration.

Interactive mode (default)

When you run esxtop, by default it’s running in interactive mode. Press h (for help) and you’ll see the help screen (below). I often refer to this to remind myself how to switch from cpu (default) to virutal machine disk stats by pressing v.

Batch Mode (exporting to CSV)

The first thing you need to do is configure esxtop in interactive mode to display the fields and columns you want to export. Once you’ve done this save your configuration, following the steps above, and then use the -b option for batch mode and specify a csv filename to use. We will also specify the a delay (-d) in seconds between the samples taken, and a number of iterations (-n) otherwise esxtop will run until you press CTRL & C. See the following:

esxtop -b -d 5-n 20 > filename.csv (this will sample every 5 seconds, 20 times).

Identify I/O Latency (Fibre channel & iSCSI only)

This is one of my favorite sections and uses of esxtop, it really shows how powerful it is. Run esxtop in interactive mode then press u to switch to ‘disk device’ and press f to change the fields so you can see the following metrics (turn off F and G).

• DAVG/cmd – Device average latency (should be under 25, preferable 15-20)
• KAVG/cmd – Kernel average latency (should be less than 2, or zero)
• GAVG/cmd – Guest average latency, this is the sum of DAVG and KAVG (should be less than 25)
• QAVG/cmd – Queue average latency (should be zero)

The VMware communities document Interpreting esxtop Statistics has more information on this under section 4.2, as does Duncan Epping’s esxtop page. Simon Greaves has recently posted on esxtop. Finally, read VMware KB article 1008205 which summarises the steps to monitor I/O latency with esxtop.

This is part 2 of the VCAP-DCA Study Guide – Storage, go back to Part 1 for:

• VMware DirectPath I/O
  
• NPIV
• Storage Best Practices
• Raw Device Mapping