Device configuration - Mellanox/scalablefunctions GitHub Wiki
Scalable functions are disabled by default in the device. User must enable them when they want to use it.
Scalable functions have very little device configuration. It includes enabling total number of scalable functions and PCI BAR size.
Scalable function device (firmware) configuration is done using mlxconfig tool.
To configure 236 SFs per PCI device with BAR size of 1MBytes:
$ mlxconfig -d 0000:03:00.0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10
Note: SF configuration is a per PCI device config. So if 2nd PCI function needs SFs, user must enable it explicitly on 2nd PF as well. Such as:
$ mlxconfig -d 0000:03:00.1 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10
1. Total/maximum number of scalable functions per PCI PF This is controlled using PF_TOTAL_SF.
2. PCI BAR size per scalable function This is controlled using PF_SF_BAR_SIZE.
When using DPU Bluefield-2, DPU software hotplug and unplug a scalable function in the external (a.k.a bare metal) host system. To use scalable functions in the external host system which managed by DPU software, user must perform SF configuration in the external bare-metal host PF where containers use the SF.
| Device | Minimum firmware version | Maximum scalable functions (on eswitch PF) | Maximum scalable functions (on bare-metal host PF) | Maximum virtual functions (on bare-metal host PF) | Config (mlxconfig) on eswitch PF | Config (mlxconfig) on bare-metal host PF |
|---|---|---|---|---|---|---|
| DPU Bluefield-3 | 24.31.1012 | 2 | 1000 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=2 PF_SF_BAR_SIZE=10 SRIOV_EN=0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=1000 PF_SF_BAR_SIZE=10 SRIOV_EN=0 NUM_OF_VFS=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=2047 |
| DPU Bluefield-2 | 24.31.1012 | 500 | 0 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=0 SRIOV_EN=0 NUM_OF_VFS=0 |
| DPU Bluefield-2 | 24.31.1012 | 2 | 500 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=2 PF_SF_BAR_SIZE=10 SRIOV_EN=0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=0 NUM_OF_VFS=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=2047 |
| DPU Bluefield-2 | 24.31.1012 | 500 | 0 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=0 SRIOV_EN=0 NUM_OF_VFS=0 |
| DPU Bluefield-2 | 24.31.1012 | 480 | 4 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=480 PF_SF_BAR_SIZE=10 SRIOV_EN=0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=4 SRIOV_EN=1 NUM_OF_VFS=16 |
| Device | Minimum firmware version | Maximum scalable functions | Maximum virtual functions | Config (mlxconfig) on PF |
|---|---|---|---|---|
| ConnectX-7 | 22.31.1012 | 500 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=0 NUM_OF_VFS=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=2047 (Should be done only on one PF on a dual PF NIC). |
| ConnectX-6-DX | 22.31.1012 | 236 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=16 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=1024 |
| ConnectX-6-DX | 22.31.1012 | 500 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=0 NUM_OF_VFS=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=2047 (Should be done only on one PF on a dual PF NIC). |
| ConnectX-6-DX | 22.31.1012 | 236 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=16 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=1024 |
| ConnectX-6-DX | 22.31.1012 | 253 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=253 PF_SF_BAR_SIZE=10 SRIOV_EN=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=1024 |
| ConnectX-6-DX | 22.31.1012 | 0 | 16 | PF_TOTAL_SF=0 PER_PF_NUM_SF=1 SRIOV_EN=1 NUM_OF_VFS=16 |
| ConnectX-6 | 20.31.1012 | 236 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=16 |
| ConnectX-6-LX | 26.31.1012 | 236 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=16 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=512 |
| ConnectX-6-LX | 26.31.1012 | 500 | 0 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=500 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=0 NUM_PF_MSIX_VALID=0 PF_NUM_PF_MSIX_VALID=1 PF_NUM_PF_MSIX=512 |
| ConnectX-5 | 16.31.1012 | 236 | 16 | PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10 SRIOV_EN=1 NUM_OF_VFS=16 |