Poor iSCSI Design = Problems

Most days, I really, really love working with VMware. When the system is working sweetly and the VMs are all happily migrating about in their little swarms, then I'm happy too. But when the dark clouds of instability start to mass on the horizon, it can be a hard day at the office. This was nothing to do with VMware and everything to do with bad design.

While P2V'ing some servers today, I had some issues with the backend iSCSI network that is propping up the implementation that I'm working on. My client has had this storage network (I refuse to call it a SAN) designed and implemented by another company before they even considered virtualisation. It's based on Lefthand iSCSI servers, so it should be fine for the job. Only the way in which it's been implemented has been badly thought out.

The iSCSI array is split across two comms rooms and the data is replicated between the two halves of the array. This is working fine. The traffic from the physical servers reaches the iSCSI network (a different VLAN) using a second gigabit network card implanted in each server. So far so good.

The problem is that the iSCSI traffic is sharing network switches (including the core switch) with normal traffic. If something big and bad happens on the storage network - say, some idiot P2V'ing a whole server - then this impacts operational network traffic. There is fibre between comms rooms, but this goes via the core switch. It's quite old and doesn't seem to have a very large backplane. Also, no link aggregation has been done so even though there are two fibres the maximum bandwidth between arrays is one gigabit. Spanning tree has shut down one of the links, as it should. This is also true of the two NICs on the back of each storage array.

I could rearrange the whole thing and make it run sweetly: if I had been engaged to do this work! There isn't time in my project to get this done. But it sorely needs doing. A long day today and a longer one tomorrow with more network smashing P2Vs to do. :-(