Chip-to-chip interconnect technologies are crucial components in modern computing systems, enabling high-speed communication between different processors, memory units, and other components within a computer or data center. As computing demands continue to grow, efficient interconnects have become increasingly important for system performance, scalability, and power efficiency. This article provides an overview of chip-to-chip interconnect technologies, with a particular focus on NVIDIA's NVLink and its applications in high performance computing.

Source: https://8kassociation.com/industry-info/8k-news/new-chip-scale-interconnect-advancements-to-boost-processing-including-8k-ecosystem/

Overview

Chip-to-chip interconnects are communication links that allow different semiconductor chips or dies to exchange data within a package or system. These interconnects are essential for:

1. Performance: They directly impact system performance by determining how quickly data can be transferred between components.
2. Scalability: Efficient interconnects allow for easier scaling of computing systems.
3. Power Efficiency: Well-designed interconnects can significantly reduce power consumption.
4. Flexibility: Advanced interconnects enable more flexible system architectures, facilitating modular chip designs (chiplets).

As traditional methods of improving processor performance through silicon scaling have slowed, chip designers have turned to multi-chip modules and chiplet-based designs to continue advancing performance. This approach requires efficient chip-to-chip communication, driving the development of advanced interconnect technologies. Chip-to-chip interconnects are essential for scaling up computational power by connecting multiple CPUs and GPUs. They enable faster training of large AI models, improve performance for data-intensive scientific simulations, and allow for more efficient parallel processing across multiple chips. As computing demands continue to grow, especially in fields like AI and machine learning, efficient chip-to-chip communication becomes increasingly critical for pushing the boundaries of what's possible in computational capabilities.

Figure 2: Node architecture using different interconnect technologies

Source: https://arxiv.org/html/2408.14090v1

Implementation of Chip-to-Chip Interconnects

1. Purpose:

Chip-to-chip interconnects allow for the transfer of data, power, and control signals between multiple chips, enabling them to work together efficiently in complex systems.

2. Types of Interconnects:

There are two main categories:

• On-chip interconnects: Connect different functional modules within a single chip
• Off-chip interconnects: Connect separate chips on a circuit board or within a package

3. Communication Methods:

• Serial communication: Data is sent one bit at a time over a single channel
• Parallel communication: Multiple bits are sent simultaneously over parallel channels

4. Key Features:

• High Bandwidth: Modern interconnects like NVLink 4.0 can provide up to 900 GB/s of bidirectional bandwidth per GPU.
• Low Latency: Minimizing delay in data transfer is crucial for system performance.
• Scalability: Ability to connect multiple chips efficiently.
• Power Efficiency: Reducing power consumption while maintaining high performance.

5. Design: (example with PCIe architecture)

• Transaction Layer: The uppermost layer, responsible for creating and managing PCI Express request and completion transactions. It generates and processes Transaction Layer Packets (TLPs), which carry data and commands between devices.
• Data Link Layer: The middle layer, ensuring data integrity and managing flow control. It adds sequence numbers and error checking (LCRC) to TLPs, handles packet acknowledgment and retransmission, and manages Data Link Layer Packets (DLLPs) for link-specific operations.
• Physical Layer: The lowest layer, divided into logical and electrical sub-blocks. The logical sub-block handles data scrambling, encoding, and packet framing. The electrical sub-block manages the actual transmission of data, including serialization/deserialization, clock recovery, and differential signaling. These layers work together to provide reliable, high-speed communication between PCIe devices. The Transaction Layer creates data packets, the Data Link Layer adds reliability features, and the Physical Layer converts data into electrical signals for transmission over the PCIe link.

Figure 3: PCIe architecture includes application, transaction, data link and physical layers.

Source: https://blog.teledynelecroy.com/2021/07/anatomy-of-pcie-link.html

Types of Chip-to-Chip Interconnects

PCI Express (PCIe)

PCI Express (Peripheral Component Interconnect Express) has been the industry standard for chip-to-chip communication in personal computers and servers for many years. PCIe is a serial expansion bus standard that connects a computer to one or more peripheral devices.

Key features of PCIe include:

• Lower latency and higher data transfer rates compared to parallel buses
• Point-to-point connections for each device
• Scalability from one to 32 separate lanes
• Backward compatibility between different PCIe versions

The latest PCIe 5.0 specification provides up to 32 GT/s (Gigatransfers per second) per lane. PCIe continues to evolve, with each new generation roughly doubling the bandwidth of the previous one.

Universal Chiplet Interconnect Express (UCIe)

UCIe is an open specification for die-to-die interconnects, developed collaboratively by major industry players including AMD, Arm, Intel, and NVIDIA. UCIe aims to standardize chiplet interconnects, enabling interoperability between chiplets from different manufacturers.

Key features of UCIe include:

• Physical layer based on PCIe 6.0, supporting up to 32 GT/s
• Protocol layer based on Compute Express Link (CXL)
• Scalability for future versions with higher bandwidth and 3D packaging support

UCIe 1.0 was released in March 2022, with version 1.1 following in August 2023. The standard continues to evolve, with UCIe 2.0 released in August 2024, adding support for 3D packaging and improved system-level solutions.

Intel's QuickPath Interconnect (QPI) and UltraPath Interconnect (UPI)

These technologies were developed by Intel for high-speed point-to-point links, primarily used in multi-processor systems.

AMD's Infinity Fabric

AMD's proprietary interconnect technology used in their CPUs and GPUs, designed to provide high bandwidth and low latency communication.

NVIDIA NVLink

NVLink is NVIDIA's high-speed interconnect technology, specifically designed for GPU-to-GPU and GPU-to-CPU communication. It offers significantly higher bandwidth compared to PCIe.

Figure 4: Blackwell GPU in production showing physical NVlink devices

Source: https://wccftech.com/nvidia-blackwell-up-running-nvlink-upgraded-1-4-tbps-first-ever-fp4-genai-image/

NVLINK

NVLink is NVIDIA's proprietary high-speed interconnect technology designed for efficient communication between GPUs and between GPUs and CPUs. It addresses the limitations of traditional PCIe connections by offering significantly higher bandwidth, lower latency, and more efficient data transfer.

Key features:

1. High bandwidth: Up to 900 GB/s bidirectional bandwidth per GPU with NVLink 4.0
2. Scalability: Multiple links per device, allowing for GPU clusters
3. Mesh networking: Direct communication between devices
4. Support for GPU-to-GPU and GPU-to-CPU communication

NVLink has evolved through several generations, with each iteration improving bandwidth and increasing the number of links per GPU. The latest NVLink 4.0 provides 50 GB/s per link, with 18 links on Hopper GPUs.

The technology's architecture differs from PCIe by allowing multiple links per device and using mesh networking for direct communication. This design enables extremely high aggregate bandwidth, crucial for AI, machine learning, and high-performance computing applications.

Benefits:

1. Efficient multi-GPU systems for parallel processing
2. Faster training of large AI models
3. Support for complex scientific simulations and data analysis
4. Real-time processing of large datasets

NVIDIA has implemented NVLink in various high-performance computing systems, such as the DGX series, demonstrating its scalability and performance benefits in multi-GPU configurations.

NVLINK-C2C

NVLink-C2C (Chip-to-Chip) is an advanced interconnect technology developed by NVIDIA, designed to enable high-speed, coherent communication between different chips and dies in computing systems. This ultra-fast chip-to-chip and die-to-die interconnect allows custom dies to coherently interconnect with NVIDIA's GPUs, CPUs, DPUs, NICs, and SOCs. The technology offers significant improvements over PCIe Gen 5, including up to 25 times more energy efficiency and 90 times more area efficiency, while enabling coherent interconnect bandwidth of 900 GB/s or higher.

A key feature of NVLink-C2C is its use of single-ended signaling at 40Gbps/pin, which contributes to its high performance and efficiency. The technology supports the Arm AMBA Coherent Hub Interface (AMBA CHI) protocol, with NVIDIA and Arm collaborating to enhance this protocol for fully coherent and secure accelerators. NVLink-C2C is used in NVIDIA's Grace Superchip family and the Grace Hopper Superchip, demonstrating its capabilities in high-performance computing applications. NVIDIA has also opened NVLink-C2C for semi-custom silicon-level integration, potentially enabling new levels of performance and efficiency in AI, machine learning, and data analytics applications.

As AI and machine learning models become increasingly complex and data-intensive, NVLink's ability to enable direct GPU-to-GPU communication at unprecedented speeds has become crucial for improving system performance and reducing processing times in multi-GPU systems.

Source: https://developer.nvidia.com/blog/deploying-retrieval-augmented-generation-applications-on-nvidia-gh200-delivers-accelerated-performance/

Comparison of Interconnect Technologies

While PCIe remains the most widely used interconnect standard, specialized technologies offer significant advantages in certain scenarios.

Technology	Bandwidth	Latency	Scalability	Typical Applications	Unique Characteristics
PCIe 5.0	32 GT/s	10 ns	High	General purpose	Industry standard, PAM4 signaling
UCIe 1.0	32 GT/s	2 ns	High	Chiplet integration	Interoperability between chiplet vendors
Intel QPI	25.6 GB/s	130 ns	Medium	Multi-CPU systems	Proprietary Intel technology
Intel UPI	41.6 GB/s	100 ns	Medium	Multi-CPU systems	Successor to QPI
AMD IF	100 GB/s	5 ns	Medium	AMD CPUs and GPUs	Scalable network-on-chip architecture
NVLink 4.0	900 GB/s	5 ns	High	GPU-intensive tasks	Point-to-point GPU connectivity

Note: Bandwidth and latency values are approximate and may vary based on specific implementations and configurations.

Each interconnect technology has its strengths and specific use cases. NVLink stands out with the highest bandwidth, making it ideal for GPU-intensive workloads, while UCIe offers the lowest latency and promises interoperability between different chiplet vendors. PCIe remains the most widely adopted standard, offering a balance of performance and compatibility, while Intel's and AMD's proprietary solutions cater to their respective CPU architectures.

Conclusion

Chip-to-chip interconnect technologies are a critical component of modern computing systems, enabling the high-speed communication necessary for today's data-intensive applications. While PCIe remains the industry standard, specialized technologies like NVIDIA's NVLink offer significant performance advantages for specific use cases. The emergence of open standards like UCIe points towards a future of greater interoperability and innovation in chip-to-chip communication. As computing demands continue to grow, we can expect further advancements in interconnect technologies, driving the next generation of high-performance computing systems.

References

[1] https://hpcat.seas.gwu.edu/Research06.html

[2] https://en.wikipedia.org/wiki/NVLink

[3] https://en.wikipedia.org/wiki/UCIe

[4] https://blog.teledynelecroy.com/2021/07/anatomy-of-pcie-link.html

[5] https://www.techtarget.com/searchdatacenter/definition/PCI-Express

[6] https://www.amax.com/unleashing-next-level-gpu-performance-with-nvidia-nvlink/

[7] https://blogs.nvidia.com/blog/what-is-nvidia-nvlink/

[8] https://siliconangle.com/2024/08/16/nvlink-nvswitch-nvidias-secret-weapon-ai-wars/

[9] https://en.wikipedia.org/wiki/NVLink