Infiniband setup with VMware and DSS V7

Article ID: 2563

Last updated: 24 Jun, 2014

We want to thank the PAConnect Team for the hard work on putting this together and contributing this for everyone to use as a guide.

Written and developed by:
Kenneth Lindsay & Jessy Marker
MCSE,MCTS,ACE
www.paconnect.com

Infiniband for VMWare

* Download PuTTY and WinSCP, install them on a laptop or PC

* Download the Infiniband Files for VMware 5.1, 5.5 included in this email

* Physically install the Infiniband card in the VMWare host

* Login to the VMWare host with vSphere Client

* Go to Configuration > Security Profile > Enable to Start and Stop with Host ESXi Shell and SSH

* Open PuTTY and connect to the VMhost with root user

* Run cmd's:

esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
reboot

* Open PuTTY again and connect to VMhost with root user

* Open WinSCP and connect to the VMhost with root user

* Copy all files from Infiniband Files to the host under /tmp
Infiniband_Files_for_VMware_5_1_5_5.zip

* Go back to PuTTY and Run Commands:

cd /tmp

-- Install 1.6.1 driver ---
unzip mlx4_en-mlnx-1.6.1.2-471530.zip
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip --no-sig-check

--- Install OFED ---
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip --no-sig-check

--- Install OpenSM ---
esxcli software vib install -v /tmp/ib-opensm-3.3.16-64.x86_64.vib --no-sig-check

--- Optional - Install MFT Tools ---
esxcli software vib install -v /tmp/mlnx-mft-2.7.1-1.vib --no-sig-check
esxcli software vib install -d /tmp/MLNX-MST-ESX-2.0.0.0.zip --no-sig-check

--- Reboot to apply newly installed packages
reboot

--- Enable 4k MTU ---
esxcli system module parameters set -m=mlx4_core -p=mtu_4k=1

--- Set up opensm ---
cd /tmp
echo "Default=0x7fff,ipoib,mtu=5:ALL=full;" > partitions.conf
cp partitions.conf /scratch/opensm/<adapter_1_hca>/ *** This will be found using WinSCP and browsing to /scratch/opensm/
cp partitions.conf /scratch/opensm/<adapter_2_hca>/ *** This will be found using WinSCP and browsing to /scratch/opensm/

* Open the VMware vSphere Client

* Assign infiniband Port 0 to a new vSwitch and call the vSwitch something like IBDatastore

* Assign the other infiniband Port 1 to another new vSwitch called IBvMotion

* Assign both vSwitches an IP Address and change the MTU on both the switch and the port group to 2044

**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as Open-E Infiniband IP subnet

How To upgrade Firmware in the ESX Host:

This process will use PuTTY and WinSCP

*Open PuTTY and WinSCP, connect to the host

* with PuTTY run the cmd to load the mst software

vmkload_mod mst

ls dev

**The above ls dev will list a device mt25418_pciconf# this is what you will use to identify the Infiniband Card if more than 1 is installed.

/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q *This cmd will work if only 1 infiniband card is installed in the host, if you have more than 1 please confirm the pciconf number as stated. The cmd will show you everything about the Infiniband card including the firmware. Firmware above v 2.8 tends to cause issues with VMWare, I have included the firmware for the HP MHGH28-XTC for v 2.7 and v 2.8.

* Use WinSCP to transfer the firmware image to /tmp
kb_upload/file/Firmware_for_Infiniband_HP_448397-B21.zip

* Run one of the cmd's below in PuTTY depending on firmware version you want:

Firmware Version 2.7
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_7_000-448397-B21.bin -qq -allow_psid_change -y b

Firmware Version 2.8
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_8_0000-448397-B21.bin -qq -allow_psid_change -y b

/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q **This will Verify the Update. Check to confirm the firmware updated

reboot

OPEN-E Storage SAN Setup:

*Get Hardware

* Install Open-e DSS 7

* Configure Software RAID

* Configure Volume Group - Auto Configure iSCSI Target

* Configure Infiniband NIC with IP address and MTU of 2044

**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as VMware Infiniband IP subnet