Introduction
Although old, Mellanox ConnectX-3 has been a good card with decent performance and a good price on second-hand markets. This makes it the straightforward choice for a high-performance home or lab network. (If you intend to run InfiniBand on ESXi 7.0 or higher, go for CX4.) Sometimes you would find a card with a strange firmware or one not compatible with your existent gear. Luckily you can change the port configuration on all CX3 cards with a simple firmware flash. Here’s how.
Disclaimer:
- Flashing firmware may brick your card and I’m not responsible for it
- Do not power off or reboot your computer during firmware flashing
Preparation
I’ll assume you are using Windows Server 2019 with all the latest patches; all programs are available under Linux too so you can apply all the instructions for a Linux server with only minor deviations.
All the download links are for ConnectX-3 non-pro version; If you have a ConnectX-2 or ConnectX-3 Pro, you need to download a different firmware.
Downloads
Download and install Mellanox firmware tools. Install ALL the components (some of them are not selected by default).
Download the ConnectX-3 VPI firmware package. EN firmware can only run Ethernet; VPI firmware run both Ethernet and IB. Thus, VPI firmware is always recommended.
Check Your Card
First of all, we need the card’s machine-dependent device ID. This device ID is used for all the commands below.
|
1 2 3 4 5 |
PS C:\Users\Administrator> mst status MST devices: ------------ mt4099_pci_cr0 mt4099_pciconf0 |
In this case, the card’s ID is mt4099_pci_cr0 . Then we make sure the card communicates:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
PS C:\Users\Administrator> flint -d mt4099_pci_cr0 query Image type: FS2 FW Version: 2.42.5000 FW Release Date: 5.9.2017 Product Version: 02.42.50.00 Rom Info: type=PXE version=3.4.752 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: [redacted] MACs: [redacted] VSD: PSID: MT_1090120019 |
Now we are good to go.
Firmware Backup
ALWAYS make a backup so you can go back when something went wrong.
|
1 2 3 4 5 6 |
flint -d <id> query full > flint_query.txt flint -d <id> hw query > flint_hwinfo.txt flint -d <id> ri orig_firmware.mlx flint -d <id> dc orig_firmware.ini flint -d <id> rrom orig_rom.mlx mlxburn -d <id> -vpd > orig_vpd.txt |
Manually open all the txt & ini files to make sure they are not empty and no error messages exist.
Firmware Flashing
Unzip the firmware downloaded from Mellanox, change the file extension from .bin to .mlx , then verify the firmware is OK:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
PS C:\Users\Administrator\Desktop> flint -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx query full Image type: FS2 FW Version: 2.42.5000 FW Release Date: 5.9.2017 MIC Version: 2.0.0 Config Sectors: 1 PRS Name: cx3-1_MCX354A_fdr_09v.prs Product Version: 02.42.50.00 Rom Info: type=PXE version=3.4.752 Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050 MACs: 0002c9000001 0002c9000002 VSD: n/a PSID: MT_1090120019 PS C:\Users\Administrator\Desktop> flint -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx verify FS2 failsafe image. Start address: 0x0. Chunk size 0x80000: NOTE: The addresses below are contiguous logical addresses. Physical addresses on flash may be different, based on the image start address and chunk size /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK /0x00002980-0x00003923 (0x000fa4)/ (Configuration) - OK /0x00003924-0x0001d14b (0x019828)/ (ROM) - OK /0x0001d14c-0x0001d18f (0x000044)/ (GUID) - OK /0x0001d190-0x0001d31b (0x00018c)/ (Image Info) - OK /0x0001d31c-0x0002a6b3 (0x00d398)/ (DDR) - OK /0x0002a6b4-0x0002b707 (0x001054)/ (DDR) - OK /0x0002b708-0x0002bad7 (0x0003d0)/ (DDR) - OK /0x0002bad8-0x00069aaf (0x03dfd8)/ (DDR) - OK /0x00069ab0-0x0006e933 (0x004e84)/ (DDR) - OK /0x0006e934-0x00072e07 (0x0044d4)/ (DDR) - OK /0x00072e08-0x000738ff (0x000af8)/ (DDR) - OK /0x00073900-0x000a29f3 (0x02f0f4)/ (DDR) - OK /0x000a29f4-0x000a659f (0x003bac)/ (DDR) - OK /0x000a65a0-0x000bb493 (0x014ef4)/ (DDR) - OK /0x000bb494-0x000bb59b (0x000108)/ (DDR) - OK /0x000bb59c-0x000c061f (0x005084)/ (DDR) - OK /0x000c0620-0x000c1e1b (0x0017fc)/ (Configuration) - OK /0x000c1e1c-0x000c1e8f (0x000074)/ (Jump addresses) - OK /0x000c1e90-0x000c280f (0x000980)/ (FW Configuration) - OK /0x00000000-0x000c280f (0x0c2810)/ (Full Image) - OK -I- FW image verification succeeded. Image is bootable. |
Make sure the PSID from the first command is the target PSID (model number) you want; and make sure the second command outputs “Image is bootable”. Then we proceed to actually burn the card. (For demonstration purposes, I burned firmware with a different PSID.)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
PS C:\Users\Administrator\Desktop> flint -d mt4099_pci_cr0 -i .\fw-ConnectX3-rel-2_42_5000-MCX354A-QCB_Ax-FlexBoot-3.4.752.mlx -allow_psid_change burn Current FW version on flash: 2.42.5000 New FW version: 2.42.5000 Note: The new FW version is the same as the current FW version on flash. Do you want to continue ? (y/n) [n] : y You are about to replace current PSID on flash - "MT_1090120019" with a different PSID - "MT_1090110018". Note: It is highly recommended not to change the PSID. Do you want to continue ? (y/n) [n] : y Burning FS2 FW image without signatures - OK Restoring signature - OK |
Reboot to use the new firmware.
FAQ/Miscellaneous
Firmware Compatibility?
In theory, all firmware for the same chip are compatible with all the cards with that chip. In practice, only flash a firmware that contains the same physical configuration as your card. For example, if your card have one physical port, do not flash any 2-port firmware. If your card have SFP+ ports, do not flash any QSFP firmware.
How to Switch Port Protocol (Ethernet/IB)
In devmgmt.msc , expand System Devices, double click on Mellanox ConnectX-3 VPI (MT04099) Network Adapter, go to Port Protocol tab, uncheck HW Defaults, select your desired port type. It should look like this:
Unable to Switch Port Protocol
If the port protocol is locked to Ethernet or IB and the options are all grayed out, it is likely that some boot-time configuration is set incorrectly. First read them out:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 query Device #1: ---------- Device type: ConnectX3 Device: mt4099_pci_cr0 Configurations: Next Boot SRIOV_EN False(0) NUM_OF_VFS 8 LINK_TYPE_P1 IB(1) LINK_TYPE_P2 IB(1) LOG_BAR_SIZE 3 BOOT_PKEY_P1 0 BOOT_PKEY_P2 0 BOOT_OPTION_ROM_EN_P1 True(1) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 True(1) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0) CQ_TIMESTAMP True(1) |
For link type fields, 1 means IB only, 2 means Ethernet only, 3 means online configurable. In this case, we configure them all to 3:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 set LINK_TYPE_P1=3 LINK_TYPE_P2=3 Device #1: ---------- Device type: ConnectX3 Device: mt4099_pci_cr0 Configurations: Next Boot New LINK_TYPE_P1 IB(1) VPI(3) LINK_TYPE_P2 IB(1) VPI(3) Apply new Configuration? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations. |
Reboot the machine and profit.
SR-IOV？
|
1 |
mlxconfig -d <id> set SRIOV_EN=1 NUM_OF_VFS=8 |
Then reboot. For detailed instructions, see the official documentation.
DO NOT configure too many VFs (e.g. NUM_OF_VFS=64 ) as the card might become IB only on that configuration even if it is in VPI mode (mode 3).
“Compile” the Firmware?
It’s known that old CX3 firmware need to be “compiled” (basically you bake a configuration into the firmware) then be flashed. New stock firmware doesn’t need this process anymore. For the completeness of this article, I’ll put the commands below.
|
1 2 3 4 5 6 7 8 9 |
# Compile mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.mlx # Verify flint -i mlnx_firmware.mlx query full flint -i mlnx_firmware.mlx verify # Flash flint -d <id> -i mlnx_firmware.mlx -allow_psid_change burn |
The ini files are available for download here.
OEM Card Firmware?
If you really want OEM (HP, IBM, etc.) ConnectX-3 firmware for some reason, download from here.
Bricked Card?
If you unfortunately have a bricked card, power off, try to find the DFU pins (2-pin jumper on the card), short them, then power on the computer. The card will boot in DFU mode and you might be able to flash a new firmware onto it. Then power off the computer, disconnect DFU jumper, power on again.
Nice post, but doesn’t work on ConnectX-3 model MCX311A:
That seems to be the one model in the ConnectX-3 series that is limited in capabilities.
https://community.mellanox.com/s/question/0D51T00006lDPyFSAW/multiple-issues-with-connectx3-cx311axcat-firmware
It cannot be flashed with any other image due to insufficient space, the on-board config cannot be changed (I wanted to activate SRIOV), not through mlxconfig (“Failed to query device”), not through FlexBoot (“insufficient permission”), and applying a patched firmware with the config setup up also does not bring the desired effect.
It’s probably still a fine 10G card, but I wanted to toy around with the IO virtualization feature, so I am probably going to switch it out.
I stumbled over your post when looking at how to add some lines to an .ini config file and have a request: could you please shed some light on why you think it is OK to use .mlx and .bin files interchangeably (rename .bin to .mlx and export .mlx where flint ri writes a .bin)?
There has been a discussion of enabling 56Gb over back-to-back Ethernet connections and some participants are alienated by your approach 😉
Here is the discussion:
https://forums.servethehome.com/index.php?threads/help-getting-56gb-on-connectx-3-pros.40427/post-394256
One thing to watch out for, that I was recently bit by, is if you’re using Connectx-3 cards, be careful what version of the firmware tools you use. The latest versions no longer support Connectx-3, and provide some truly useless error messages.
The last firmware tools that works with ConnectX-3 is 4.22.1-406-LTS
If you have no cards listed when you run mst status, and you get “device not supported” when running other commands, make sure your firmware tools isn’t too new.
Excellent post, thanks so much!!
This helped me out with a different problem. I bought two Mellanox MCX312A-XCBT ConnectX-3 Ethernet Dual SFP+ cards. Installed into two Windows systems (W11 and W10). Initially the card worked fine in the W11 system. Installed the MLX drivers and during that process, it asked to update the firmware to 2.42. Stupidly I said yes. After reboot, I kept getting Windows driver error 43. Uninstalled, reinstalled but no luck. Checked the system log files which said this:
Mellanox ConnectX-3 VPI (MT04099) Network Adapter (PCI bus 3, device 0, function 0): SR-IOV cannot be enabled because FW does not support SR-IOV. In order to resolve this issue please re-burn FW, having added parameters related to SR-IOV support.
Native_3_0_0: Execution of FW command failed. op 0xfff, status 0x1, errno -5, token 0xffff, in_modifier 0x100, op_modifier 0, in_param 189ff000.
Native_3_0_0: MAP_FA command failed with error -5.
The adapter card is non-functional.
Most likely a FW problem.
Please burn the last FW and restart the mlx4_bus driver.
Native_3_0_0: Driver startup failed because the hca could not be initialized.
So I followed your instruction, re-flashed the firmware with older 2.40 rebooted and voila, working again!
Interestingly, the W10 machine has no problems with the newer firmware. Gigabyte Z97 board, whereas the W11 system is an Aorus Z390.
Thanks again, you saved the day!