Open-E Knowledgebase

Infiniband setup with VMware and DSS V7

Article ID: 2563
Last updated: 24 Jun, 2014

We want to thank the PAConnect Team for the hard work on putting this together and contributing this for everyone to use as a guide.

Written and developed by:
Kenneth Lindsay & Jessy Marker
MCSE,MCTS,ACE
www.paconnect.com

Infiniband for VMWare
 
* Download PuTTY and WinSCP, install them on a laptop or PC
 
* Download the Infiniband Files for VMware 5.1, 5.5 included in this email
 
* Physically install the Infiniband card in the VMWare host
 
* Login to the VMWare host with vSphere Client
 
* Go to Configuration > Security Profile > Enable to Start and Stop with Host ESXi Shell and SSH
 
* Open PuTTY and connect to the VMhost with root user
 
* Run cmd's:
 
esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
reboot
 
* Open PuTTY again and connect to VMhost with root user
 
* Open WinSCP and connect to the VMhost with root user
 
* Copy all files from Infiniband Files to the host under /tmp
Infiniband_Files_for_VMware_5_1_5_5.zip

* Go back to PuTTY and Run Commands:
 
cd /tmp
 

-- Install 1.6.1 driver ---
unzip mlx4_en-mlnx-1.6.1.2-471530.zip
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip --no-sig-check


--- Install OFED --- 
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.0.zip --no-sig-check


--- Install OpenSM --- 
esxcli software vib install -v /tmp/ib-opensm-3.3.16-64.x86_64.vib --no-sig-check


--- Optional - Install MFT Tools ---
esxcli software vib install -v /tmp/mlnx-mft-2.7.1-1.vib --no-sig-check
esxcli software vib install -d /tmp/MLNX-MST-ESX-2.0.0.0.zip --no-sig-check

--- Reboot to apply newly installed packages
reboot

--- Enable 4k MTU ---
esxcli system module parameters set -m=mlx4_core -p=mtu_4k=1


--- Set up opensm ---
cd /tmp
echo "Default=0x7fff,ipoib,mtu=5:ALL=full;" > partitions.conf
cp partitions.conf  /scratch/opensm/<adapter_1_hca>/   *** This will be found using WinSCP and browsing to /scratch/opensm/
cp partitions.conf  /scratch/opensm/<adapter_2_hca>/   *** This will be found using WinSCP and browsing to /scratch/opensm/

* Open the VMware vSphere Client

* Assign infiniband Port 0 to a new vSwitch and call the vSwitch something like IBDatastore

* Assign the other infiniband Port 1 to another new vSwitch called IBvMotion

* Assign both vSwitches an IP Address and change the MTU on both the switch and the port group to 2044

**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as Open-E Infiniband IP subnet

 

How To upgrade Firmware in the ESX Host:

 

This process will use PuTTY and WinSCP
 
*Open PuTTY and WinSCP, connect to the host
 
* with PuTTY run the cmd to load the mst software
 
vmkload_mod mst
 
ls dev
 

 


**The above ls dev will list a device mt25418_pciconf# this is what you will use to identify the Infiniband Card if more than 1 is installed.
 
 
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q        *This cmd will work if only 1 infiniband card is installed in the host, if you have more than 1 please confirm the pciconf number as stated. The cmd will show you everything about the Infiniband card including the firmware. Firmware above v 2.8 tends to cause issues with VMWare, I have included the firmware for the HP MHGH28-XTC for v 2.7 and v 2.8.
 
* Use WinSCP to transfer the firmware image to /tmp
kb_upload/file/Firmware_for_Infiniband_HP_448397-B21.zip

 
* Run one of the cmd's below in PuTTY depending on firmware version you want:
 
Firmware Version 2.7
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_7_000-448397-B21.bin -qq -allow_psid_change -y b
 
Firmware Version 2.8
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 -i /tmp/fw-25408-2_8_0000-448397-B21.bin -qq -allow_psid_change -y b
 
/opt/mellanox/bin/flint -d /dev/mt25418_pciconf0 q  **This will Verify the Update. Check to confirm the firmware updated
 
reboot
 
 
OPEN-E Storage SAN Setup:
 
*Get Hardware
 
* Install Open-e DSS 7
 
* Configure Software RAID
 
* Configure Volume Group - Auto Configure iSCSI Target
 
* Configure Infiniband NIC with IP address and MTU of 2044
 
**Remember Infiniband NIC's should be configured with a different subnet than the LAN but the Same as VMware Infiniband IP subnet
 

This article was:   Helpful | Not helpful Report an issue


Article ID: 2563
Last updated: 24 Jun, 2014
Revision: 1
Views: 3266
Posted: 24 Jun, 2014 by Rafinski G.
Updated: 24 Jun, 2014 by Rafinski G.
print  Print email  Subscribe email  Email to friend share  Share pool  Add to pool
Also listed in
folder Troubleshooting -> Hardware -> Miscellaneous

Prev     Next
Intel 10GB Network card is not configurable       Meaning of the dropped packets

The Knowledge base is managed by Open-E data storage software company.