Possible fix found MSA1500sc halting problem

Thanks to Eric at Genesis Hosting for finding a possible fix for our MSA1500sc and VMWare problem. He has had the same thing happen to him and has gotten further with HP and VMWare then I did for finding a fix.

His server hardware is a bit different than mine but the SAN and FC components are almost the same. Here are the various things that need doing to fix the problem:

• Make sure the FC HBA have the latest firmware installed


• We are both using Brocade switches, so the firmware needs to be v5.31 or higher


• Even if you only have a single controller in the MSA1500sc, upgrade to v7.00 of the firmware


• Most importantly, set the FC switch ports that your HBAs and MSA1500 are connected to to 2Gbps and unselect any unused port types. In my case I set the two MSA1500 connections to F-Type. I was unable to modify the type setting for the HP BL480p FC ports for some reason.

Eric's testing seems to indicate that the problem occurs if the SCSI Queue depth gets to high. You can view this by using the MSA1500 CLI and entering SHOW TASKSTATS.

You can change the FC port speed and type will VMs are running but you probably want to do it when there is no disk load on the SAN.

Eric also pointed out that instead of having to shut down all of the VMs and reboot the MSA1500 when the problem does occur you can go on to the CLI and enter DISABLE THIS_CONTROLLER REBOOT. If you have two controllers you would also have to do DISABLE OTHER_CONTROLLER REBOOT. This may hang or cause problems with the VMs but you should be able to reboot them without having to reboot the ESX hosts.

I have made all of the changes already except for the firmware on the Brocade switches. I have scheduled some downtime this weekend and will do the upgrade then as well as a reboot of everything and then I will leave it and see what happens.

MSA1500cs periodically grinding to a halt with VMWare ESX 3.5

Last night for the third time since going live with VMWare in Sept. 2007 we had an issue where the MSA1500cs SAN that acts as the datastore for our 3 host ESX cluster ground to a halt. As the MSA1500 has no ability to log what it is doing this is a tricky one to figure out.

The first time it happened, I though it was because I had set the multipathing on the ESX hosts to fixed. Apparently this is a not the suggested configuration for the MSA1500 as even though it has the v7.00 Active/Active firmware on it, ESX doesn't see it as a true Active/Active setup. So I changed the path setup to Most Recently Used (MRU) and left it to run.

The second time it happened was shortly after upgrading to v3.5 which caused High Availability (HA) on one of the hosts to go nuts and fill up the log partition.

Last night it happened again but nothing had been changed on the setup for weeks. Basically, the VMs grind to a halt and then become unavailable and then some or all of the LUNs presented to the hosts disappear. The only way I've found to fix it is to shut the ESX hosts down (which takes about 30 minutes each as it is trying to gracefully flush the cache to the LUNs but they aren't there) and then power down the MSA1500 controller shelf and bring everything backup again in the correct order.

I opened a ticket with HP about it today but they couldn't see anything from the CLI show tech_info dump I sent them (which I guess they wouldn't as the thing had been power cycled!) and suggested that it could be the drivers or firmware in the BL480c servers that we are using or a firmware upgrade may be needed in the Brocade fiber channel switches.

As I had the time last night, I also applied the last batch of v3.5 patches.

So far it has happened after about 50-60 days of uptime so I think for now we will be proactive and do a manual shut down of everything on a weekend every 6 weeks or so.

Virtual Virtualization lab

I'm doing some preparation for some cross training I'm doing on Friday on VMWare Virtual Infrastructure 3. I've used some old servers to create a lab environment consisting of a virtual SAN (using SANMelody) on one to act as an iSCSI target, another server as a VirtualCenter server and two old Dell PE2650s as VMWare ESX 3.5 servers. This works fine for VMotion, DRS, HA and all the other good stuff that VMWare supports but still takes 4 physical boxes, two of which need to be multiprocessor or multicore and have a fair amount of RAM. In a pinch you could get this down to 3 servers if you combined the VC server and the SANMelody server onto one box.

Then I got to thinking, I know people have installed ESX Server as a VM under VMWare Server or GSX Server so would it be possible to create all four machines as VMs on one physical box? This would let me give the two people I am training their own server (the PE2650s) and they could run through the entire setup on their own.

I'm going to have a go myself today and see what happens.