Mellanox ConnectX-3 Firmware Flashing and Configuration for Both Ethernet and InfiniBand in 2021

Introduction

Although old, Mellanox ConnectX-3 has been a good card with decent performance and a good price on second-hand markets. This makes it the straightforward choice for a high-performance home or lab network. (If you intend to run InfiniBand on ESXi 7.0 or higher, go for CX4.) Sometimes you would find a card with a strange firmware or one not compatible with your existent gear. Luckily you can change the port configuration on all CX3 cards with a simple firmware flash. Here’s how.

Disclaimer:

  • Flashing firmware may brick your card and I’m not responsible for it
  • Do not power off or reboot your computer during firmware flashing

Preparation

I’ll assume you are using Windows Server 2019 with all the latest patches; all programs are available under Linux too so you can apply all the instructions for a Linux server with only minor deviations.

All the download links are for ConnectX-3 non-pro version; If you have a ConnectX-2 or ConnectX-3 Pro, you need to download a different firmware.

Downloads

Download and install Mellanox firmware tools. Install ALL the components (some of them are not selected by default).

Download the ConnectX-3 VPI firmware package. EN firmware can only run Ethernet; VPI firmware run both Ethernet and IB. Thus, VPI firmware is always recommended.

Check Your Card

First of all, we need the card’s machine-dependent device ID. This device ID is used for all the commands below.

In this case, the card’s ID is mt4099_pci_cr0 . Then we make sure the card communicates:

Now we are good to go.

Firmware Backup

ALWAYS make a backup so you can go back when something went wrong.

Manually open all the txt & ini files to make sure they are not empty and no error messages exist.

Firmware Flashing

Unzip the firmware downloaded from Mellanox, change the file extension from .bin to .mlx , then verify the firmware is OK:

Make sure the PSID from the first command is the target PSID (model number) you want; and make sure the second command outputs “Image is bootable”. Then we proceed to actually burn the card. (For demonstration purposes, I burned firmware with a different PSID.)

Reboot to use the new firmware.

FAQ/Miscellaneous

Firmware Compatibility?

In theory, all firmware for the same chip are compatible with all the cards with that chip. In practice, only flash a firmware that contains the same physical configuration as your card. For example, if your card have one physical port, do not flash any 2-port firmware. If your card have SFP+ ports, do not flash any QSFP firmware.

How to Switch Port Protocol (Ethernet/IB)

In devmgmt.msc , expand System Devices, double click on Mellanox ConnectX-3 VPI (MT04099) Network Adapter, go to Port Protocol tab, uncheck HW Defaults, select your desired port type. It should look like this:

Unable to Switch Port Protocol

If the port protocol is locked to Ethernet or IB and the options are all grayed out, it is likely that some boot-time configuration is set incorrectly. First read them out:

For link type fields, 1 means IB only, 2 means Ethernet only, 3 means online configurable. In this case, we configure them all to 3:

Reboot the machine and profit.

SR-IOV?

Then reboot. For detailed instructions, see the official documentation.

DO NOT configure too many VFs (e.g. NUM_OF_VFS=64 ) as the card might become IB only on that configuration even if it is in VPI mode (mode 3).

“Compile” the Firmware?

It’s known that old CX3 firmware need to be “compiled” (basically you bake a configuration into the firmware) then be flashed. New stock firmware doesn’t need this process anymore. For the completeness of this article, I’ll put the commands below.

The ini files are available for download here.

OEM Card Firmware?

If you really want OEM (HP, IBM, etc.) ConnectX-3 firmware for some reason, download from here.

Bricked Card?

If you unfortunately have a bricked card, power off, try to find the DFU pins (2-pin jumper on the card), short them, then power on the computer. The card will boot in DFU mode and you might be able to flash a new firmware onto it. Then power off the computer, disconnect DFU jumper, power on again.

References

One thought on “Mellanox ConnectX-3 Firmware Flashing and Configuration for Both Ethernet and InfiniBand in 2021

  1. loebl

    Nice post, but doesn’t work on ConnectX-3 model MCX311A:
    That seems to be the one model in the ConnectX-3 series that is limited in capabilities.

    https://community.mellanox.com/s/question/0D51T00006lDPyFSAW/multiple-issues-with-connectx3-cx311axcat-firmware

    It cannot be flashed with any other image due to insufficient space, the on-board config cannot be changed (I wanted to activate SRIOV), not through mlxconfig (“Failed to query device”), not through FlexBoot (“insufficient permission”), and applying a patched firmware with the config setup up also does not bring the desired effect.

    It’s probably still a fine 10G card, but I wanted to toy around with the IO virtualization feature, so I am probably going to switch it out.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *