Preparation<\/h1>\n
I’ll assume you are using Windows Server 2019 with all the latest patches; all programs are available under Linux too so you can apply all the instructions for a Linux server with only minor deviations.<\/p>\n
All the download links are for ConnectX-3 non-pro version; If you have a ConnectX-2 or ConnectX-3 Pro, you need to download a different firmware.<\/p>\n
Downloads<\/h2>\n
Download and install Mellanox firmware tools<\/a>. Install ALL the components (some of them are not selected by default).<\/p>\n Download the ConnectX-3 VPI firmware package<\/a>. EN firmware can only run Ethernet; VPI firmware run both Ethernet and IB. Thus, VPI firmware is always recommended.<\/p>\n First of all, we need the card’s machine-dependent device ID. This device ID is used for all the commands below.<\/p>\n In this case, the card’s ID is mt4099_pci_cr0<\/span> . Then we make sure the card communicates:<\/p>\n Now we are good to go.<\/p>\n ALWAYS make a backup so you can go back when something went wrong.<\/p>\n Manually open all the txt & ini files to make sure they are not empty and no error messages exist.<\/p>\n Unzip the firmware downloaded from Mellanox, change the file extension from .bin<\/span> to .mlx<\/span> , then verify the firmware is OK:<\/p>\n Make sure the PSID from the first command is the target PSID (model number) you want; and make sure the second command outputs “Image is bootable”. Then we proceed to actually burn the card. (For demonstration purposes, I burned firmware with a different PSID.)<\/p>\n Reboot to use the new firmware.<\/p>\n In theory, all firmware for the same chip are compatible with all the cards with that chip. In practice, only flash a firmware that contains the same physical configuration as your card. For example, if your card have one physical port, do not flash any 2-port firmware. If your card have SFP+ ports, do not flash any QSFP firmware.<\/p>\n In devmgmt.msc<\/span> , expand System Devices, double click on Mellanox ConnectX-3 VPI (MT04099) Network Adapter, go to Port Protocol tab, uncheck HW Defaults, select your desired port type. It should look like this:<\/p>\n If the port protocol is locked to Ethernet or IB and the options are all grayed out, it is likely that some boot-time configuration is set incorrectly. First read them out:<\/p>\n For link type fields, 1 means IB only, 2 means Ethernet only, 3 means online configurable. In this case, we configure them all to 3:<\/p>\n Reboot the machine and profit.<\/p>\n Then reboot. For detailed instructions, see the official documentation<\/a>.<\/p>\n DO NOT configure too many VFs (e.g. NUM_OF_VFS=64<\/span> ) as the card might become IB only on that configuration even if it is in VPI mode (mode 3).<\/p>\n It’s known that old CX3 firmware need to be “compiled” (basically you bake a configuration into the firmware) then be flashed. New stock firmware doesn’t need this process anymore. For the completeness of this article, I’ll put the commands below.<\/p>\n The ini files are available for download here<\/a>.<\/p>\n If you really want OEM (HP, IBM, etc.) ConnectX-3 firmware for some reason, download from here<\/a>.<\/p>\n If you unfortunately have a bricked card, power off, try to find the DFU pins (2-pin jumper on the card), short them, then power on the computer. The card will boot in DFU mode and you might be able to flash a new firmware onto it. Then power off the computer, disconnect DFU jumper, power on again.<\/p>\n Introduction Although old, Mellanox ConnectX-3 has been a good card with decent performance and a good price on second-hand markets. This makes it the straightforward choice for a high-performance home or lab network. (If you intend to run InfiniBand on ESXi 7.0 or higher, go for CX4.) Sometimes you would find a card with a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-281","post","type-post","status-publish","format-standard","hentry","category-networking"],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/281"}],"collection":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/comments?post=281"}],"version-history":[{"count":12,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/281\/revisions"}],"predecessor-version":[{"id":294,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/281\/revisions\/294"}],"wp:attachment":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/media?parent=281"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/categories?post=281"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/tags?post=281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Check Your Card<\/h2>\n
PS C:\\Users\\Administrator> mst status\r\nMST devices:\r\n------------\r\n mt4099_pci_cr0\r\n mt4099_pciconf0<\/pre>\n
PS C:\\Users\\Administrator> flint -d mt4099_pci_cr0 query\r\nImage type: FS2\r\nFW Version: 2.42.5000\r\nFW Release Date: 5.9.2017\r\nProduct Version: 02.42.50.00\r\nRom Info: type=PXE version=3.4.752\r\nDevice ID: 4099\r\nDescription: Node Port1 Port2 Sys image\r\nGUIDs: [redacted]\r\nMACs: [redacted]\r\nVSD:\r\nPSID: MT_1090120019<\/pre>\n
Firmware Backup<\/h1>\n
flint -d <id> query full > flint_query.txt\r\nflint -d <id> hw query > flint_hwinfo.txt\r\nflint -d <id> ri orig_firmware.mlx\r\nflint -d <id> dc orig_firmware.ini\r\nflint -d <id> rrom orig_rom.mlx\r\nmlxburn -d <id> -vpd > orig_vpd.txt<\/pre>\n
Firmware Flashing<\/h1>\n
PS C:\\Users\\Administrator\\Desktop> flint -i .\\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx query full\r\nImage type: FS2\r\nFW Version: 2.42.5000\r\nFW Release Date: 5.9.2017\r\nMIC Version: 2.0.0\r\nConfig Sectors: 1\r\nPRS Name: cx3-1_MCX354A_fdr_09v.prs\r\nProduct Version: 02.42.50.00\r\nRom Info: type=PXE version=3.4.752\r\nDevice ID: 4099\r\nDescription: Node Port1 Port2 Sys image\r\nGUIDs: 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050\r\nMACs: 0002c9000001 0002c9000002\r\nVSD: n\/a\r\nPSID: MT_1090120019\r\nPS C:\\Users\\Administrator\\Desktop> flint -i .\\fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.mlx verify\r\n\r\n FS2 failsafe image. Start address: 0x0. Chunk size 0x80000:\r\n\r\n NOTE: The addresses below are contiguous logical addresses. Physical addresses on\r\n flash may be different, based on the image start address and chunk size\r\n\r\n \/0x00000038-0x0000065b (0x000624)\/ (BOOT2) - OK\r\n \/0x0000065c-0x0000297f (0x002324)\/ (BOOT2) - OK\r\n \/0x00002980-0x00003923 (0x000fa4)\/ (Configuration) - OK\r\n \/0x00003924-0x0001d14b (0x019828)\/ (ROM) - OK\r\n \/0x0001d14c-0x0001d18f (0x000044)\/ (GUID) - OK\r\n \/0x0001d190-0x0001d31b (0x00018c)\/ (Image Info) - OK\r\n \/0x0001d31c-0x0002a6b3 (0x00d398)\/ (DDR) - OK\r\n \/0x0002a6b4-0x0002b707 (0x001054)\/ (DDR) - OK\r\n \/0x0002b708-0x0002bad7 (0x0003d0)\/ (DDR) - OK\r\n \/0x0002bad8-0x00069aaf (0x03dfd8)\/ (DDR) - OK\r\n \/0x00069ab0-0x0006e933 (0x004e84)\/ (DDR) - OK\r\n \/0x0006e934-0x00072e07 (0x0044d4)\/ (DDR) - OK\r\n \/0x00072e08-0x000738ff (0x000af8)\/ (DDR) - OK\r\n \/0x00073900-0x000a29f3 (0x02f0f4)\/ (DDR) - OK\r\n \/0x000a29f4-0x000a659f (0x003bac)\/ (DDR) - OK\r\n \/0x000a65a0-0x000bb493 (0x014ef4)\/ (DDR) - OK\r\n \/0x000bb494-0x000bb59b (0x000108)\/ (DDR) - OK\r\n \/0x000bb59c-0x000c061f (0x005084)\/ (DDR) - OK\r\n \/0x000c0620-0x000c1e1b (0x0017fc)\/ (Configuration) - OK\r\n \/0x000c1e1c-0x000c1e8f (0x000074)\/ (Jump addresses) - OK\r\n \/0x000c1e90-0x000c280f (0x000980)\/ (FW Configuration) - OK\r\n \/0x00000000-0x000c280f (0x0c2810)\/ (Full Image) - OK\r\n\r\n-I- FW image verification succeeded. Image is bootable.\r\n<\/pre>\n
PS C:\\Users\\Administrator\\Desktop> flint -d mt4099_pci_cr0 -i .\\fw-ConnectX3-rel-2_42_5000-MCX354A-QCB_Ax-FlexBoot-3.4.752.mlx -allow_psid_change burn\r\n\r\n Current FW version on flash: 2.42.5000\r\n New FW version: 2.42.5000\r\n\r\n Note: The new FW version is the same as the current FW version on flash.\r\n\r\n Do you want to continue ? (y\/n) [n] : y\r\n\r\n\r\n You are about to replace current PSID on flash - \"MT_1090120019\" with a different PSID - \"MT_1090110018\".\r\n Note: It is highly recommended not to change the PSID.\r\n\r\n Do you want to continue ? (y\/n) [n] : y\r\nBurning FS2 FW image without signatures - OK\r\nRestoring signature - OK<\/pre>\n
FAQ\/Miscellaneous<\/h1>\n
Firmware Compatibility?<\/h2>\n
How to Switch Port Protocol (Ethernet\/IB)<\/h2>\n
![\"\"](\"https:\/\/blog.swineson.me\/en\/wp-content\/uploads\/sites\/2\/2021\/06\/mlx-cx3-port-protocol-260x300.png\")
<\/a><\/p>\nUnable to Switch Port Protocol<\/h2>\n
PS C:\\Users\\Administrator\\Desktop> mlxconfig -d mt4099_pci_cr0 query\r\n\r\nDevice #1:\r\n----------\r\n\r\nDevice type: ConnectX3\r\nDevice: mt4099_pci_cr0\r\n\r\nConfigurations: Next Boot\r\n SRIOV_EN False(0)\r\n NUM_OF_VFS 8\r\n LINK_TYPE_P1 IB(1)\r\n LINK_TYPE_P2 IB(1)\r\n LOG_BAR_SIZE 3\r\n BOOT_PKEY_P1 0\r\n BOOT_PKEY_P2 0\r\n BOOT_OPTION_ROM_EN_P1 True(1)\r\n BOOT_VLAN_EN_P1 False(0)\r\n BOOT_RETRY_CNT_P1 0\r\n LEGACY_BOOT_PROTOCOL_P1 PXE(1)\r\n BOOT_VLAN_P1 1\r\n BOOT_OPTION_ROM_EN_P2 True(1)\r\n BOOT_VLAN_EN_P2 False(0)\r\n BOOT_RETRY_CNT_P2 0\r\n LEGACY_BOOT_PROTOCOL_P2 PXE(1)\r\n BOOT_VLAN_P2 1\r\n IP_VER_P1 IPv4(0)\r\n IP_VER_P2 IPv4(0)\r\n CQ_TIMESTAMP True(1)<\/pre>\n
PS C:\\Users\\Administrator\\Desktop> mlxconfig -d mt4099_pci_cr0 set LINK_TYPE_P1=3 LINK_TYPE_P2=3\r\n\r\nDevice #1:\r\n----------\r\n\r\nDevice type: ConnectX3\r\nDevice: mt4099_pci_cr0\r\n\r\nConfigurations: Next Boot New\r\n LINK_TYPE_P1 IB(1) VPI(3)\r\n LINK_TYPE_P2 IB(1) VPI(3)\r\n\r\n Apply new Configuration? (y\/n) [n] : y\r\nApplying... Done!\r\n-I- Please reboot machine to load new configurations.<\/pre>\n
SR-IOV\uff1f<\/h2>\n
mlxconfig -d <id> set SRIOV_EN=1 NUM_OF_VFS=8<\/pre>\n
“Compile” the Firmware?<\/h2>\n
# Compile\r\nmlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.mlx\r\n\r\n# Verify\r\nflint -i mlnx_firmware.mlx query full\r\nflint -i mlnx_firmware.mlx verify\r\n\r\n# Flash\r\nflint -d <id> -i mlnx_firmware.mlx -allow_psid_change burn<\/pre>\n
OEM Card Firmware?<\/h2>\n
Bricked Card?<\/h2>\n
References<\/h1>\n
\n