The Supermicro X10SDV-4C-TLN2F is the motherboard I use for my primary NAS. It has a Xeon D-1521 SOC which provides a reasonable amount of compute power for a NAS without using an unreasonable amount of power (for legacy enterprise hardware). Performance wise, it is actually similar to an Intel N100, although that chip uses less power. The D-1521 has multiple benefits though. It supports 128G ECC RAM, has 32 PCIe lanes, and comes on servre boards with features such as IPMI. It also has built-in 10G networking.

NIC Instability

Unfortunately, the built-in 10G networking proved to be somewhat unstable for me. This mainly presented itself as a complete inability to connect to the NAS (via the web UI, Samba, etc.) for an hour or more after reboots. Even after that, connections were somewhat unreliable for hours to days and would occasionally fail/connect slowly even after an extended period of time. Although this did seem to get better the longer the NAS had been powered on.

Another way this presented itself was in alerts for communication errors with the Eaton UPS it is connected to. I originally assumed this was an issue with the UPS, however, the backup TrueNAS server I'm using (a Dell R430) has no problems at all. Combined with the other networking issues the TrueNAS server was having, I eventually narrowed this down to a networking issue on the X10SDV-4C-TLN2F itself. After some searching, I found a Reddit thread and a TinkerTry post describing the same problem.

Work Around

I tried a number of troubleshooting option from within TrueNAS (using ethtool, the ip command suite, looking at dmesg messages, etc.) but, as far as I could tell, TrueNAS thought the NIC was working perfectly. I eventually checked the logs of the network switch it was connected to and found that the port was going through a Listening ➔ Learning ➔ Forwarding ➔ Disabled (Port Down) loop every few seconds.

This made it seem like a STP issue. I know I had a couple bridge interfaces setup on TrueNAS, but I checked and STP was disabled on those interfaces (so it wasn't a case of them "fighting"). I then configured the port on the switch as an "edge port" to tell the switch that another switch is not going to be connected there. In my case, I'm using a Ruckus ICX7250 switch. The commands were:

enable
configure terminal
interface ethernet 1/2/3
spanning-tree 802-1w admin-edge-port
stp-bpdu-guard

That last option is suppored to stop rogue loops if something else is plugged in later.

Unfortunately, at this point I was still seeing the same log behavior on the switch and the network was still unstable. So I disabled spanning tree on the port entirely.

enable
configure terminal
interface ethernet 1/2/3
no spanning-tree

This resolved the STP messages on the switch, but I was instead getting UP ➔ DOWN flapping on that port. According to Google Gemini:

If the port still cycles with no spanning-tree and no loop-detection, you are dealing with a known hardware quirk involving the Intel X552 10G MAC/PHY integration on Xeon D-1500 boards and the ixgbe Linux driver.

During the TrueNAS Scale Linux boot sequence, the ixgbe driver resets the physical network interface multiple times as it loads the driver, initializes SR-IOV (if active), and attaches the interface to the Linux bridge. On the X552, this causes the SFP+ laser/PHY to completely power down and power back up repeatedly.

The Ruckus switch sees the physical light signal disappear, marks the port as Disabled (Port Down), and when the light comes back, it starts the STP timer from scratch, leading to an endless hardware synchronization loop.

How to stabilize it from the Ruckus side: You can tell the Ruckus switch to stop auto-negotiating the link speed and force it to stay up, which prevents the transceiver from resetting its state when the driver blinks.

This was accomplished with the following commands on the switch:

enable
configure terminal
interface ethernet 1/2/3
speed-duplex 10g-full

This significantly reduced the UP ➔ DOWN flapping in the switch logs, although it still happened occasionally. It also made the TrueNAS host reachable immediately after boot, although it did still take several minutes for the network to completely stabilize. That being said, I was only seeing ~30% packet drop rate during this time, instead of nearly 100%, and the worst of it was only a few minutes, rather than hours. This seems to have made the problem significantly less egregious and it now seems to work well enough.

The switch config must also be written to flash memory to persist across reboots:

write memory

Permanent Fix

Apparently, according to the Reddit and TinkerTry posts (linked above), there is a Supermicro firmware patch that should (hopefully) fix this permanently. Specifically, this should be SDV23B. It's a little hard to tell if I already have this or not, but apparently I need to contact Supermicro support to get a copy of it and to then flash it using USB connected storage. This is what Google Gemini had to say:

The Supermicro / Xeon-D Microcode Patch (The Permanent Fix) The Intel X552 integrated MAC on Xeon D-1500 series motherboards (specifically Supermicro X10SDV series) has a physical timing bug involving the PHY core clocks.
Supermicro released a critical 10GbE microcode/PHY firmware patch specifically to address this issue where the interface permanently flaps or experiences packet loss when linked to enterprise switches (like Cisco and Ruckus).
How to identify: Check your motherboard model. If it is a Supermicro X10SDV-4C-TLN2F (or similar), this applies to you.
The Solution:

1. Go to Supermicro’s support site or open a ticket with them.
2. Request the 10Gbe PHY Microcode Update for your specific board (the historical fix bundle was often named SDV23B.zip).
3. You will need to boot the server into a DOS/UEFI environment via a USB drive to flash the Intel X552 PHY layer microcode.
   
   Once that motherboard PHY layer patch is applied, the Intel X552 behaves normally, auto-negotiation will work correctly, and the link will stop flapping on boot.