Installing and booting VMware ESXi over FCoE on an IBM BladeCenter H with Brocade CNA

Hardware:
– IBM BladeCenter H
– HS22V and HX5 blades with Brocade 2 port 10GbE Converged Network Adapter, no local drives
– IBM Storwize V7000

At the time of writing the Brocade 2 port 10GbE Converged Network Adapter for IBM BladeCenter (81Y1650) is supported on VMware ESXi up to version 4.1.

To install VMware on the blades, a custom VMware ESXi 4.1 ISO image with Brocade CNA drivers is required.

The Brocade 2 port 10GbE Converged Network Adapter for IBM BladeCenter drivers are available on Brocade web site:

http://www.brocade.com/services-support/drivers-downloads/adapters/IBM_BLADECENTER.page

Download the latest Adapter Driver Package for WMware ESX 4.0/4.1 (brocade_driver_esx4x_v2-3-0-2.tar.gz at the time of writing).

I used the VMware KB Adding third-party device drivers to ESXi 4.1 to include the Brocade CNA drivers into the installation media.

First, download and install the VMware VIBDDI Image Customization Tool for ESXi 4.1. I recommend using 32-bit version for Red Hat Enterprise Linux 5. Anyway, vtar binary is 32-bit only inside the 64-bit package.

Copy the ESXi 4.1 installation media ISO image to /opt/vmware/vibddi/iso.

Mount (loop) the ISO:

# mount -o loop /opt/vmware/vibddi/iso/<VMware-VMvisor-Installer-iso> /mnt

Decompress the imagedd.bz2 file:

# bunzip2 < /mnt/imagedd.bz2 > imagedd

Umount the ISO image:

# umount /mnt

View the contents of the ESXi image:

# vibddi -i imagedd -q
< oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM) >
< oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM) >
< vmware-esx-firmware (4.1.0-1.4.348481) >
< CMPI-1.0 >
< VMW_CMPI_CPP-1.0.5 >
< vmkapi_1_1_0_0 >
< vmknexus1kvapi-1-4 >
< vmkepsecapi_1.0.0.0 >
< DriverAPI-9.0 >
< DriverAPI-9.1 >
< vmware-esx-tools-light (4.1.0-1.4.348481) >

Extract the network and Fibre Channel HBA offline bundle ZIP files (BCD-bna-*offline_bundle-*.zip and BCD-bfa-*-offline_bundle-*.zip) from the two ISO files inside the Brocade Adapter Driver Package for WMware ESX 4.0/4.1.

Add the Fibre Channel HBA driver (bfa) to imagedd:

# vibddi -i imagedd -o BCD-bfa-*-offline_bundle-*.zip
Setting up configuration for esxupdate
This may take a few seconds...
Checking bootbank filesystems (pre-install)
Installing offline bundle: BCD-bfa-*-offline_bundle-*.zip
Update operation completed successfully
Checking bootbank filesystems (post-install)

Confirm the driver is in place (look for vmware-esx-drivers-scsi-bfa) in the query output:

# vibddi -i imagedd -q

Add the CNA network driver (bna) to imagedd and generate the custom installation image (-g flag):

# vibddi -i imagedd -o BCD-bna-*-offline_bundle-*.zip -g
Setting up configuration for esxupdate
This may take a few seconds...
Checking bootbank filesystems (pre-install)
Installing offline bundle: BCD-bna-*-offline_bundle-*.zip
Update operation completed successfully
Checking bootbank filesystems (post-install)
Generating customized ISO: /opt/vmware/vibddi/iso/VMware-VMvisor-*.x86_64.iso
This may take a few minutes...
Generated new ISO: /opt/vmware/vibddi/iso/custom/VMware-VMvisor-*.iso

Confirm both drivers are in place (look for vmware-esx-drivers-scsi-bfa and vmware-esx-drivers-net-bna) in the query output:

# vibddi -i imagedd -q

The VMware ESXi 4.1 custom installation image with Brocade 2 port 10GbE Converged Network Adapter for IBM BladeCenter is in /opt/vmware/vibddi/iso/custom/VMware-VMvisor-*.iso. Use the image to install ESXi 4.1 on the blades with Blade Tasks / Remote Control / Remote Console under BladeCenter H AMM.

Check about vHBAs and other PCI devices may stop responding in ESX/ESXi 4.1 when using Interrupt Remapping.

One comment.

  1. Hi Cristian!

    As per the ServerProven page for HS22V and Storwize V7000, the following CNA cards are supported:
    – Qlogic 10Gb CNA (CFFh)
    http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/blade/7871/ibm_2076Cxxx.html

    IBM refused to support us with Brocade CNA and V7000

    Best regards.
    Oleg.