How Far Can You Go with Top-of-Rack?

In a previous blog, we discussed how SFP direct attach cables (DACs) will support most enterprise server speeds in top-of-rack (ToR) switch-to-server deployments well into the future, with their ability to support downlink speeds of 25 Gig via SFP28 DACs and the potential for 50 Gig via emerging SFP56 DACs. But the truth is that larger cloud and enterprise data centers are already seeing the need for 100 Gig server speeds, with the need for even faster speeds expected in the future.

But will ToR deployments continue to support switch-to-server links at these next-generation speeds? Let’s take a closer look at the technology and key considerations involved.

How Do We Get There?

The ability to support faster transmission speeds has a lot to do with binary encoding schemes used to convert data into digital signals. While we won’t delve into the physics behind encoding, it’s essentially the process of turning data into binary bits (i.e., 1’s and 0’s) via discrete voltage levels. The most common encoding scheme that has long been used in data transmission is non-return-to-zero (NRZ), which uses two different voltage levels for the two binary digits where positive voltage represents a “1” and negative voltage represents “0” (also referred to as a two-level pulse amplitude modulation, or PAM2). NRZ encoding has significantly evolved over the past few decades and is primarily used to support bit rates of 1, 10, and 25 Gb/s per lane in data center links.

When we look at small form-factor pluggable technology, single-lane SFP+ and SFP28 high-speed interconnects that support 10 and 25 Gig respectively are based on NRZ encoding. For higher speeds, 4-lane QSFP+ and QSFP28 interconnects that support 40 and 100 Gig are also based on NRZ-QSFP+ at 10 Gb/s bit rate per lane and QSFP28 at 25 Gb/s bit rate per lane. Technically, because NRZ can support a bit rate of 50 Gb/s, it seems logical that a single-lane SFP interconnect would support 50 Gig and a 4-lane QSFP interconnect would support 200 Gig using NRZ encoding. However, with NRZ speeds above a 25 Gb/s bit rate, channel loss becomes a problem. Enter four-level pulse amplitude modulation, or PAM4.

PAM4 encoding offers twice the bit rate per the same signal period of NRZ by using four voltage levels instead of two, supporting 50 and 100 Gb/s bit rates without an increase in channel loss. For small form-factor pluggable technology, PAM4 now gives us single-lane SFP56 interconnects for 50 Gig and four-lane QSFP56 interconnects for 200 Gig. PAM4 is also what enables 400 Gig applications-the double-density 8-lane QSFP-DD form factor relies on the PAM4 50 Gb/s bit rate to achieve 400 Gig (i.e., 50 Gb/s X 8 lanes), which is ideal for switch-to-switch deployments. Unfortunately, the increased throughput offered by PAM4 comes a cost.

What the FEC?

PAM4 encoding is far more susceptible to noise than NRZ. To improve performance and counteract any potential errors cause by increased noise, PAM4 signals use advanced forward error correction (FEC). FEC works by adding redundant data that the receiver can check and use to correct errors and recover the original data without the need for signal retransmission. PAM4 requires FEC, but it adds latency, typically in the order of 100 to 500 milliseconds (ms).

While there are low-latency FEC developments happening behind the scenes to try and cut the delay by as much as 50%, the fact remains that some applications will simply not tolerate the delay. For applications like financial trading, edge computing, interactive gaming, video conferencing, virtual and augmented reality, artificial intelligence, real-time monitoring, and data analytics, any latency over 100 ms can impact performance. In gaming for example, latency over 100 ms means a noticeable lag for players. For data centers looking to support these applications, latency in switch-to-server connections is a consideration.

Due to the added latency of FEC with PAM4, the highest-speed and lowest-latency option is currently the 4-lane QSFP28 DAC that supports 100 Gig using NRZ and a 25 Gb/s bit rate that does not require FEC up to 3-meters. While most enterprise data centers are just starting to shift to 25 Gig server connections using single-lane SFP28 DACs, 4-lane QSFP28 DACs enable migration to high-speed, low-latency 100 Gig server connections to support emerging real-time applications.

What are the Options?

For existing NRZ encoding, Siemon currently offers several options for using high-speed interconnects in the data center to support from 10 to 100 Gig switch-to-server links. These include direct connections in short-reach (1 to 3 meter) links for in-cabinet ToR deployments using DACs or in longer-reach (1 to 20 meter) links for cabinet-to-cabinet deployments (e.g., end-of-row) using active optical cables (AOCs). For breakout applications where a single switch port connects to multiple lower-speed servers, Siemon also offers a variety of hybrid breakout assemblies for both DACs and AOCs. As highlighted in a previous blog, when choosing between the DACs and AOCs, it’s important to consider density, distance, power consumption, scalability and interoperability, as well as overall cost and availability. Siemon’s current offering includes the following:

• SFP+ DACs and AOCs for 10 Gig links
• SFP28 DACs and AOCs for 25 Gig links
• QSFP+ DACs and AOCs for 40 Gig links
• QSFP28 DACs and AOCs for 100 Gig links
• QSFP+ to 4 SFP+ DACs and AOCs for 4X10 Gig breakout links
• QSFP28 to 4 SFP28 DACs and AOCs for 4X25 Gig breakout links

More to Come

With the introduction of PAM4 encoding, DACs and AOCs will enable future 200 and 400 Gig links. You can rest assured that Siemon has an eye on the market with plans to introduce PAM4 high-speed interconnects when these higher-speed switch-to-server links come to fruition. Future PAM4 options to be on the lookout for include:

• QSFP56 DACs and AOCs for 200 Gig links
• QSFPDD to 2 QSFP56 DACs and AOCs for 2×200 Gig breakout links

And it doesn’t stop there. While PAM4 to NRZ conversion technology can be used with AOCs to support a 4X100 Gig breakout application using QSFPDD-to-QSFP28 hybrid assemblies, cost is always a factor. Developments are therefore underway for more cost-efficient PAM4-to-PAM4 breakout applications with DACs using a QSFPDD to two-lane (i.e., double density) SFP interface and a PAM4 50 Gb/s bit rate. However, it is still unclear which double-density SFP interface, either the SFP-DD or the DSFP, will become the predominant connector for this 4X100 Gig breakout solution. So what does this all mean for your data center?

The key takeaway is that the introduction of PAM4 encoding technology enables DACs and AOCs to support from 10 to 400 Gig direct links, including the lowest-latency 100 Gig option of QSFP28 DACs that use NRZ encoding technology for emerging real-time applications. And that means that ToR switch-to-server deployments with DACs are here to stay and will get you where you need to go.