Docker Networking: IPVLAN and SRi‐OV VF - fordsfords/fordsfords.github.io GitHub Wiki

I don't have good expertise on Docker/Kubernetes, especially as they relate to performance.

Note: Much of the following information is from ChatGPT, and is therefore suspect. Consider this information as a starting point for further research.

Table of Contents A customer's experience IPVLAN SRI-OV VF Default Docker Network SR-IOV and Kubernetes SR-IOV and Cloud

A customer reached out to us for help with loss as they experimented with network plugins for Docker that improve network performance.

By default, Docker implements a virtual bridge with NAT to give the containers an IP address that can get off-box. But this adds software layers and therefore higher latency and lower throughput.

The network plugins for Multus/IPVLAN and SRI-OV FVF get rid of the bridge and NAT.

According to the customer:

• They don't see loss when using host network.
• They do see loss when they use Multus/IPVLAN or MACVLAN interface plugins.
• The loss goes away when they use SRI-OV VF.

Using tools they determined that the loss with Multus/IPVLAN is not UM's fault. They are going with SRI-OV VF.

Note: the following information is from ChatGPT, and is therefore suspect. Consider this information as a starting point for further research.

Here's some info from ChatGPT about IPVLAN.

In the context of Docker networking, the IPVLAN plugin is a networking plugin that allows for the creation of virtual interfaces (veth pairs) within containers, providing each container with its own unique network namespace and IP address. IPVLAN operates at Layer 2 of the OSI model, enabling containers to directly communicate with the host and other containers on the same subnet without the need for NAT (Network Address Translation).

Here are some key features and characteristics of the IPVLAN plugin:

1. Layer 2 Isolation: IPVLAN provides network isolation at Layer 2, meaning that each container has its own MAC address and can communicate directly with other containers and the host.

2. Efficient Packet Forwarding: IPVLAN uses efficient packet forwarding mechanisms, reducing overhead and improving performance compared to traditional bridge-based networking.

3. Shared Subnet: Containers attached to the same IPVLAN virtual network share the same subnet, allowing them to communicate as if they were connected to the same physical switch.

4. Native Performance: Since IPVLAN operates at Layer 2, it offers native performance without the overhead associated with Layer 3 routing or NAT.

5. Scaling: IPVLAN is suitable for scenarios where large numbers of containers need to communicate efficiently within the same subnet, making it a good choice for containerized applications with microservices architectures.

Overall, the IPVLAN plugin provides a lightweight and efficient networking solution for Docker containers, enabling high-performance communication within the same subnet.

IPVLAN does not require special NICs to work. IPVLAN operates at Layer 2 of the OSI model and leverages existing networking infrastructure without the need for specialized hardware support.

IPVLAN works by creating virtual interfaces (veth pairs) within containers, which communicate with the host's networking stack. These virtual interfaces use the host's physical NIC to send and receive network packets, but they operate independently within the container's network namespace.

Since IPVLAN operates at Layer 2 and relies on standard networking mechanisms, it can work with any network interface card (NIC) that supports basic Ethernet functionality. There are no specific hardware requirements or dependencies for IPVLAN, making it a widely compatible option for container networking within Docker and Kubernetes environments.

Here's some info from ChatGPT about SRI-OV VF.

Single Root I/O Virtualization (SR-IOV) Virtual Function (VF) plugins for Docker containers.

SR-IOV is a technology that allows a single physical network interface card (NIC) to present itself as multiple virtual NICs to virtual machines (VMs) or containers. Each of these virtual NICs is called a Virtual Function (VF). This technology improves network performance and efficiency by allowing VMs or containers to directly access hardware resources without going through the hypervisor or host operating system for networking tasks.

In the context of Docker containers, SR-IOV VF plugins provide integration between Docker and SR-IOV-enabled NICs. These plugins enable Docker containers to leverage the benefits of SR-IOV technology, such as improved network performance and reduced CPU overhead.

With SR-IOV VF plugins, Docker containers can be assigned specific VFs of an SR-IOV-enabled NIC, allowing them to bypass the virtual switch layer and communicate directly with the physical NIC. This direct access to hardware resources can result in lower latency and higher throughput for network-intensive containerized applications.

Overall, SR-IOV VF plugins for Docker containers enable the use of SR-IOV technology within containerized environments, providing enhanced network performance and efficiency.

Here's some info from ChatGPT about the default Docker network with virtual bridge and NAT.

By default, when you install Docker, it creates a virtual bridge network called docker0, and it uses NAT (Network Address Translation) to manage communication between containers and the external network.

Here's how it typically works:

1. docker0 Bridge: Docker creates a virtual Ethernet bridge called docker0 when it starts. This bridge acts like a virtual switch, allowing containers to communicate with each other and with the host system.

2. NAT: When Docker containers are started, they are connected to the docker0 bridge. Docker sets up NAT rules so that containers can communicate with the external network (Internet) through the host system's IP address. Outgoing traffic from containers is masqueraded with the host's IP address, and incoming traffic is routed back to the appropriate container based on port mappings.

This default networking setup provides a simple and effective way for containers to communicate with each other and with external systems. However, it may not be suitable for all scenarios, especially when advanced networking features such as direct Layer 2 communication or SR-IOV are required. That's where plugins like IPVLAN or SR-IOV VF plugins come into play, offering more flexibility and performance optimizations for specific use cases.

In the default Docker networking setup with a single bridge (like docker0) and NAT, Docker handles IP address allocation internally. Containers are assigned IP addresses from a private subnet managed by Docker's internal DHCP server. From the perspective of system and network administrators, they typically don't need to allocate IP addresses manually to individual containers.

However, when using networking plugins like IPVLAN or SR-IOV VF, the allocation of IP addresses to virtual NICs may become the responsibility of system and network administrators.

For example:

1. IPVLAN: With IPVLAN, each container typically gets its own unique IP address within the same subnet. These IP addresses need to be managed and allocated by the administrators, either manually or through DHCP.

2. SR-IOV VF: Similarly, with SR-IOV, each Virtual Function (VF) may require its own IP address, and these addresses need to be allocated and managed by the administrators.

In both cases, the network configuration becomes more complex compared to the default Docker networking setup, as administrators need to ensure proper IP address allocation and management for the virtual NICs used by containers.

Here's some info from ChatGPT about SR-IOV and Kubernetes.

Integrating SR-IOV with Kubernetes involves several steps to configure both the underlying infrastructure and Kubernetes itself to support SR-IOV-enabled devices. Here's a high-level overview of the process:

1. Infrastructure Preparation:

• Ensure that your hardware supports SR-IOV and that the required drivers are installed on the host machines.
• Configure the SR-IOV Virtual Functions (VFs) on the network interface cards (NICs) of the host machines.
• Enable SR-IOV in the BIOS settings of the servers if necessary.

2. Kubernetes Setup:

• Install and configure Kubernetes on your cluster according to your requirements. This might involve using a tool like kubeadm, kops, or another Kubernetes distribution.
• Deploy the Container Network Interface (CNI) plugin that supports SR-IOV. CNI plugins like SR-IOV CNI or Multus CNI are commonly used for this purpose. These plugins enable Kubernetes to assign SR-IOV VFs to pods.

3. SR-IOV Device Plugin:

• Install the SR-IOV device plugin on your Kubernetes cluster. This plugin advertises the available SR-IOV VFs to the Kubernetes scheduler so that pods requesting SR-IOV resources can be scheduled onto nodes with the appropriate hardware.

4. Resource Allocation:

• Define Kubernetes Resource Custom Resource Definitions (CRDs) or other configuration objects to specify SR-IOV resources and policies. This might include defining SR-IOV network policies, allocating specific VFs to pods, or specifying Quality of Service (QoS) requirements.

5. Pod Configuration:

• Configure your pod specifications to request SR-IOV VFs as needed. This involves specifying the SR-IOV network interfaces in the pod's network configuration.

6. Validation and Testing:

• Validate the SR-IOV integration by deploying pods that use SR-IOV VFs and testing their connectivity and performance.

7. Monitoring and Maintenance:

• Monitor the SR-IOV-enabled pods and the underlying infrastructure to ensure proper operation and performance.
• Perform regular maintenance tasks such as updating drivers, plugins, and Kubernetes itself to keep the environment up to date and secure.

Integrating SR-IOV with Kubernetes requires coordination between system administrators, network administrators, and Kubernetes administrators to properly configure and manage the infrastructure and Kubernetes components. Additionally, it's essential to test and validate the setup thoroughly to ensure reliable operation and performance.

Here's some info from ChatGPT about SR-IOV and Cloud.

Support for SR-IOV in cloud environments varies depending on the cloud service provider and the specific infrastructure they offer. Some cloud providers do offer support for SR-IOV, particularly in instances or virtual machines that are optimized for high-performance networking scenarios.

Here's a brief overview of SR-IOV support in major cloud providers:

1. Amazon Web Services (AWS): AWS offers Enhanced Networking, which provides higher network performance and lower latency compared to traditional network interfaces. Enhanced Networking is available on certain instance types and supports SR-IOV. AWS also offers Elastic Fabric Adapter (EFA), which is a network interface for HPC (High Performance Computing) workloads that supports SR-IOV.

2. Microsoft Azure: Azure supports SR-IOV on certain Virtual Machine (VM) instances through the use of Accelerated Networking. Accelerated Networking provides high-performance networking capabilities by bypassing the host and virtual switch layers and directly attaching virtual NICs to the hardware NICs using SR-IOV.

3. Google Cloud Platform (GCP): GCP offers Network Interface (NIC) optimization for certain VM instances, providing higher throughput and lower latency for network-intensive workloads. While GCP doesn't explicitly mention SR-IOV support, its NIC optimization features may include similar capabilities to SR-IOV.

4. IBM Cloud: IBM Cloud offers SR-IOV support on certain bare metal and virtual server instances, allowing users to leverage SR-IOV for improved network performance and efficiency.

It's important to check the documentation and specifications provided by each cloud provider to determine the availability of SR-IOV support and which instance types or VM families offer it. Additionally, keep in mind that while SR-IOV can provide performance benefits, its availability may be limited to specific instance types or regions within a cloud provider's infrastructure.