iSCSI Multi-pathing and Jumbo Frames on vSphere

Storage multi-pathing is an important technology that's been built into VMware for fibre channel SANs for some time. It's handled automatically for FC (and hardware iSCSI actually): just put more than one HBA into your server and off you go. This is not the case for software iSCSI however. There is only one software iSCSI initiator inside the vmkernel, so you can't do multi-pathing, right? Hmm... read on.

Lets assume that you have a setup like this:

(picture)

iSCSI Server Address: 192.168.19.1
ESX vmk port IP: 192.168.19.11 (vmk1)
vmnic4
vmnic5

Many VMware administrators think that a setup like this will give them improved performance. And many VMware administrators are wrong.

If you were to set this up in the GUI, you would have a NIC team. Both network cards would be bound to the same vmkernel port. This means that the software iSCSI initiator would show only one path to the storage. IF one of the NICs failed the you're good. But there is no performance gain by using two NICs in the setup. Using the vSphere GUI, you can't do multi-pathing for iSCSI. You have to get down and dirty with the command line. But when you do, you'll be rewarded with improved performance. So I'm going to show you how. I'm also going to look at how to enable Jumbo frames.

We're going to build this new setup using the same physical layout as the example above. No wires need to move and the IP addresses will stay the same. This is important Good News if you're thinking of improving the performance of an existing implementation. The end result will even look the similar in the GUI to the untrained eye. :-)

Firstly, place the host im maintenance mode. This means that you'll not have any running virtual machines onboard. This is important as we're going to chop your ESX servers connection to its storage temporarily. If you do this with running VMs, you'll end up with delayed writes at the very least and probably some crashed VMs. You've been warned.

Remove the vmkernel port from vSwitch1 (or your iSCSI vSwitch of course). You should now have an empty vSwitch with two vmnics attached. SSH to you ESX server and su to root. If you're using ESXi, you'll need to do this with vCLI. Instead of esxcfg-xxx, you'll need to use vicfg-xxx. The reason we're doing this in the command line is to enable jumbo frames.

We need to enable jumbo frames and then create two port groups on your empty vSwitch:

# esxcfg-vswitch -m 9000 vSwitch1
# esxcfg-vswitch -A "iSCSI Path One" vSwitch1
# esxcfg-vswitch -A "iSCSI Path Two" vSwitch1

Then we're going to create a vmkernel port for each of our port groups. These need to be enabled for jumbo frames which can only be done at creation time:

# esxcfg-vmknic -a -i 192.168.19.11 -n 255.255.255.0 -m 9000 "iSCSI Path One"
# esxcfg-vmknic -a -i 192.168.19.12 -n 255.255.255.0 -m 9000 "iSCSI Path Two"

When we go back to the GUI and hit the refresh link, you see that we've got two new vmkernel ports on our vSwitch. Great, but we're not done. The problem at the moment is that the NIC team actually gets in the way. We're going to override the vSwitch properties for each individual vmkernel port. Go into the properties of the vSwitch and verify that the vSwitch shows both vmnics as Active:

Then, go to the "iSCSI Path One" vmkernel port and configure it to use only one of the two vmnics. The other should be placed in "Unused":

Guess what comes next? You need to configure the other "iSCSI Path Two" vmkernel port to utilise only the other vmnic, again placing the spare in "Unused":

Nearly there. Again in the GUI, check the vmhba number of your iSCSI adapter. Mine was vmhba35 but yours may not be. So, now go to the command line again and type the following command, adjusting for your own vmk and vmhba numbers:

# esxcli swiscsi nic add -n vmk0 -d vmhba35
# esxcli swiscsi nic add -n vmk1 -d vmhba35

This has enabled iSCSI multi-pathing using the esxcli command to influence the vSphere PSA into action. To check, run the following:

# esxcli swiscsi nic list -d vmhba35

A key point to look for are that both of these entries must have a MAC address assigned. If they don't, you probably forgot to assign a single unique vmnic to each vmkernel port in the vSwitch. A second point is to note that jumbo frames are enabled for each vmkernel port.

Now you're ready to go and sort out your software iSCSI initiator. At the moment, it still thinks it has only one path to each LUN. You could just do a rescan. If you jumped ahead and did this automatically then well done! You're clearly a savvy VMadmin. However, you'll find that you get more paths than you'd expect. This is because of some old dross that we need to clear out. If this is a new install and you've never connected to iSCSI storage before: just click the Rescan link and move onto setting fixed paths below.

To clear out the old paths go inside the properties of your iSCSI initiator and choose the "Static Paths". You need to remove each of these in turn. Don;t worry if it takes a very , very long time to chew through each one. It just does. Once this horrible long task is finished, close those properties and click the Rescan link. You should be rewarded with two paths per LUN.

Now it's time to choose which of these paths you want to use. I'd recommend the fixed (preferred) path option, although you may wish to try Round Robin. If I'm fixing paths, I give some thought to the amount of IO activity on each LUN and try to balance these across available paths. Obviously.

Once way to see that this is working is to check the performance stats for the two vmnics that you're using for iSCSI in the vSphere client. You should see both vmnics doing some work now. Multi-pathing FTW! :-)

The side benefit of creating this setup is that you've also got jumbo frame support built in. So now you can enable jumbo frames on your switch(es) and storage array and off you go. For instance, to save you some searching:

HP Switches enable jumbo frame support per VLAN. If your iSCSI VLAN was 88, you'd type:

config# vlan 88 jumbo
config# sh vlan 88 (just to check)

Cisco switches enable jumbo frames for the whole switch or on a per interface basis depending on the exact model and IOS/CatOS version. You can find out exactly how for your switch here.

Remember that jumbo frames only work if the whole end to end path supports them. That means ESX -> Switch(es) -> Storage Array all need jumbo frames support.