We want to thank the PAConnect Team for the hard work on putting this together and contributing this for everyone to use as a guide.
Written and developed by:
Kenneth Lindsay & Jessy Marker
MCSE,MCTS,ACE
www.paconnect.com
Infiniband for VMWare
* Download PuTTY and WinSCP, install them on a laptop or PC
* Download the Infiniband Files for VMware 5.1, 5.5 included in this email
* Physically install the Infiniband card in the VMWare host
* Login to the VMWare host with vSphere Client
* Go to Configuration > Security Profile > Enable to Start and Stop with Host ESXi Shell and SSH
* Open PuTTY and connect to the VMhost with root user
* Run cmd's:
esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
reboot
* Open PuTTY again and connect to VMhost with root user
* Open WinSCP and connect to the VMhost with root user
* Copy all files from Infiniband Files to the host under /tmp
Infiniband_Files_for_VMware_5_1_5_5.zip
* Go back to PuTTY and Run Commands:
cd /tmp
-- Install 1.6.1 driver ---
unzip mlx4_en-mlnx-1.6.1.2-471530.zip
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip --no-sig-check
--- Install OFED ---
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip --no-sig-check
--- Install OpenSM ---
esxcli software vib install -v /tmp/ib-opensm-3.3.16-64.x86_64.vib --no-sig-check
--- Optional - Install MFT Tools ---
esxcli software vib install -v /tmp/mlnx-mft-2.7.1-1.vib --no-sig-check
esxcli software vib install -d /tmp/MLNX-MST-ESX-2.0.0.0.zip --no-sig-check
--- Reboot to apply newly installed packages
reboot
--- Enable 4k MTU ---
esxcli system module parameters set -m=mlx4_core -p=mtu_4k=1
--- Set up opensm ---
cd /tmp
echo "Default=0x7fff,ipoib,mtu=5:ALL=full;" > partitions.conf
cp partitions.conf /scratch/opensm/<adapter_1_hca>/ *** This will be found using WinSCP and browsing to /scratch/opensm/
cp partitions.conf /scratch/opensm/<adapter_2_hca>/ *** This will be found using WinSCP and browsing to /scratch/opensm/
* Open the VMware vSphere Client
* Assign infiniband Port 0 to a new vSwitch and call the vSwitch something like IBDatastore
* Assign the other infiniband Port 1 to another new vSwitch called IBvMotion
* Assign both vSwitches an IP Address and change the MTU on both the switch and the port group to 2044
**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as Open-E Infiniband IP subnet
How To upgrade Firmware in the ESX Host:
This process will use PuTTY and WinSCP
*Open PuTTY and WinSCP, connect to the host
* with PuTTY run the cmd to load the mst software
vmkload_mod mst
ls dev
**The above ls dev will list a device mt25418_pciconf# this is what you will use to identify the Infiniband Card if more than 1 is installed.
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q *This cmd will work if only 1 infiniband card is installed in the host, if you have more than 1 please confirm the pciconf number as stated. The cmd will show you everything about the Infiniband card including the firmware. Firmware above v 2.8 tends to cause issues with VMWare, I have included the firmware for the HP MHGH28-XTC for v 2.7 and v 2.8.
* Use WinSCP to transfer the firmware image to /tmp
kb_upload/file/Firmware_for_Infiniband_HP_448397-B21.zip
* Run one of the cmd's below in PuTTY depending on firmware version you want:
Firmware Version 2.7
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_7_000-448397-B21.bin -qq -allow_psid_change -y b
Firmware Version 2.8
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_8_0000-448397-B21.bin -qq -allow_psid_change -y b
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q **This will Verify the Update. Check to confirm the firmware updated
reboot
OPEN-E Storage SAN Setup:
*Get Hardware
* Install Open-e DSS 7
* Configure Software RAID
* Configure Volume Group - Auto Configure iSCSI Target
* Configure Infiniband NIC with IP address and MTU of 2044
**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as VMware Infiniband IP subnet
