Whitebox Switches

Overview

RBFS is a network operating system (NOS) running on whitebox or bare-metal switches. But what exactly is a whitebox switch?

A whitebox switch is a network device that uses generic, off-the-shelf hardware components, allowing for flexibility and cost savings compared to proprietary hardware. Whitebox switches typically consist of several key components, each playing a crucial role in the functioning of the switch. Here are the primary components of a whitebox switch:

  • The chassis houses all the internal components of the switch. Whitebox switches can come in various form factors, such as 1U rack-mounted units or larger chassis-based designs.

  • The Printed Circuit Board (PCB) is the medium to interconnect the various components with each other. In personal computers, PCBs are often called motherboard.

  • The Central Processing Unit (CPU) including memory (for more detail see next section).

  • The Switching fabric (for more detail see next section).

  • Physical interfaces including management ports

  • The Power Supply Unit (PSU) provides electrical power to the switch, ensuring its continuous operation. Whitebox switches may have redundant PSUs for improved reliability.

  • The fans are used to maintain the operating temperature within safe limits as the switch components generate quite a lot of heat.

  • Optionally, the Baseboard Management Controller (BMC) (for more detail see next section).

whitebox overview
Figure 1. Generic Whitebox Switch Architecture

The basic idea behind whitebox switches is disaggregation, i.e., decoupling of hardware and software components. Disaggregation offers more flexibility and innovation and breaks the vendor lock-in of traditional networking devices.

The terms whitebox switch and bare-metal switch are often used interchangeably. Technically the difference is that a whitebox switch has an operating system installed on it, while the bare-metal switch has not. However, the whitebox switch is still an open switch, because the operating system and hardware are not integrated and bundled like traditional switches, which is equivalent to buying a bare metal switch and an operating system in a package.

Component Details

CPU and Memory

The Central Processing Unit (CPU) is responsible for executing control plane functions of the switch. It runs the network operating system (NOS) which is often based on Open Network Linux (ONL) and additional applications running on top of it.

RtBrick Full Stack (RBFS) is a disaggregated and open network operating system and runs as an Ubuntu container on the Open Network Linux operating system on white boxes which can perform Layer 2 and Layer 3 switching.

In addition, the CPU manages the overall operation of the switch. For instance, temperature sensors, power supplies and fans are controlled by the CPU using either I2C (Inter-Integrated Circuit) or SMBus (System Management Bus) communication protocol.

The CPU is connected to different types of memory including

  • DRAM (Dynamic Random Access Memory) for temporary data storage and fast access, e.g., application memory and routing tables.

  • Hard disks or solid-state disks (SSD) for storing firmware and software as well as log files

  • Boot flash to store essential data to start-up the device.

There are a couple of management interfaces that are directly connected to the CPU for management purposes, e.g., out-of-band Ethernet port, a serial console, or an USB port.

Some whitebox switch vendors offer module CPU cards that allow flexibility in the CPU and memory configuration.

Switching ASIC and Networking Interfaces

A switching ASIC (Application-Specific Integrated Circuit) is a specialized integrated circuit designed for the specific purpose of forwarding and processing data packets, thus, responsible for the high-speed packet switching and routing functions that enable data to move efficiently within a network. The switching ASIC is optimized to rapidly inspect incoming data packets and make forwarding decisions based on packet header information like destination MAC addresses or IP addresses by consulting specialized lookup tables that are typically stored in on-chip memory for fast access. It determines the appropriate egress port for each packet to ensure it reaches its destination. Depending on the type of ASIC, additional functions are supported, e.g., VLAN handling, queueing and scheduling of packets, etc.

Switching ASICs are engineered for high-speed data processing and packet forwarding at line rates. As ASICs are a key component in networking devices, they are designed for energy efficiency.

The switching ASIC is physically connected to the CPU via a PCI-E bus and programmed using either the ASIC API or a software development kit (SDK) provided by the ASIC vendor.

The network interfaces of a switching ASICs are implemented using serial high speed communication protocol called SerDes (Serializer/Deserializer). SerDes links use pulse code modulation (PAM) and are available in different speeds depending on the ASIC in use. Physical network ports of the whitebox switch are connected to the switching ASIC via SerDes links. When you want to multiplex multiple physical interfaces, such as multiple 10GbE interfaces, to a single 40 Gbps SerDes interface, you typically use a technology called gearbox.

Switching ASICs can be deployed standalone, i.e., the switch consists of only one single switching ASIC, or as part of a switching fabric consisting of multiple switching ASICs.

Examples of ASICs include

  • Broadcom StrataXGS family, including Tomahawk and Trident ASICs

  • Broadcom StrataDNX series including Jericho and Qumran ASICs

  • Mellanox (now part of NVIDIA) Spectrum family

  • Barefoot Tofino (now part of Intel)

As of today, RBFS only supports Broadcom Qumran 2c (Q2c), Qumran 2a (Q2a) and Qumran AX (QAX) based whitebox switches. For a list of supported hardware, please refer to the Platform Overview.

The Open Compute Project is an open-source initiative that provides reference designs for various data center and networking equipment, including whitebox switches. These designs cover hardware, software, and management aspects of whitebox networking.

BMC

The Baseboard Management Controller (BMC) is an optional component available in some whitebox switches. The BMC is a specialized microcontroller responsible for out-of-band management tasks. It allows for remote monitoring and control, even when the main CPU is offline. It’s important to note that the BMC and the CPU operate independently. The BMC has its own firmware and processing capabilities, allowing it to function even if the main CPU is not operational. For instance, the BMC is able to monitor temperature, voltage, fan speed, and system health using the I2C/SMBus.

Most often, the BMC can be reached via the Ethernet port or serial console interface.

The BMC is physically connected to the CPU via the LPC bus (Low Pin Count). As communication protocol IPMI (Intelligent Platform Management Interface) is used which is a standardized, out-of-band management interface that provides a set of common commands for system management and monitoring.

Conclusion

Most vertically integrated switching and routing platforms are loaded with features the most of consumers will never use resulting in a high price. Thus, the overall cost of white box switch is lower than those branded equipment. In addition, whitebox switches can easily be upgraded and updated re-using existing software.