Want 400GbE in a Blade Server? Here’s How

A few years ago, the thought of needing multiple 25GbE ports on a blade server was not even a consideration.  In fact, I did a survey 6 years ago and got an overwhelming response that only a couple of 10GbE ports was needed for VMware virtualization efforts.  Today there is now an option to have 8 x 25GbE ports in a 2 CPU blade server – but there are some things you need to know.

About the Broadcom 57504 Quad Port 25GbE MX Mezzanine Adapter

Late last month, Dell Technologies announced a quad port 10/25GbE Broadcom 57504 network card mezzanine for the PowerEdge MX740c and MX840c blade servers.  I’ve listed the key features you’ll find on this card below, followed by a few things to note:

  • x16 PCI Express v4.0
  • Flexible, power-efficient 25 GbE MX Mezzanine solution
  • Line-rate bidirectional concurrent throughput on all ports
  • BroadSAFE® technology provides Silicon Root of Trust, secure boot, secure firmware, and secure key storage for the industry’s most-secure Ethernet adapter
  • TruFlow™ Flow Accelerator— programmable packet processing in hardware boosts application Performance
  • Maximizes performance while minimizing latency across full range of frame sizes
  • Support for Data Plane Developer Kit (DPDK) framework
  • Low-latency for RDMA over Converged Ethernet (RoCE) v1 and v2
  • OpenManage Enterprise-Modular embedded systems management integration
  • TruManage™ for enterprise-scale manageability
  • Single Root I/O Virtualization (SR-IOV) with Function Level Reset (FLR) support
  • Tunnel-aware stateless offloads for VXLAN, NVGRE, Geneve
  • NPAR EP support for up to 16 Physical Functions (PFs)
  • Reliability and interoperability assured by Broadcom’s market-proven Ethernet expertise

There are a few things to point out about the features listed above:

It Doesn’t Support PCIe Gen 4

Although the card supports PCIe v4.0, the current 2nd Generation Intel Xeon SP processor that ships on the PowerEdge MX740c and MX840c servers does not support PCIe Gen 4, so the card will run at PCIe Gen 3 speeds.

It Runs at Full Line Rate Speeds

Each of the 4 x 25GbE ports runs at full line-rate bidirectional speeds.  There are no oversubscribed ports (more on this below.)

It Has Both SR-IOV and NPAR

NPAR, or Network Partioning, has been a technology that Broadcom started offering many years ago.  In summary, it takes a single 25GbE port and offers up to 4 physical functions (PFs) per port.  (Read more about how NPAR works here.) Since this card has 4 ports, that equals 16 physical ports that will be presented to the O/S or hypervisor.  The biggest thing to understand is that this capability occurs at the server hardware level.  It does not require any specific blade server switch.

The Quad Port Card Requires Using BOTH Ports on the MX7116n

The PowerEdge MX scalable architecture uses a pair of Fabric Switch Engines (MX9116n)  and Fabric Extender Modules (MX7116n).  The MX7116n has two 200GbE ports that each provide single 200Gb (8 x 25GbE) lanes that connect the NICs per server to the MX9116n switch engines at full 25GbE connectivity.  With 2 x 25GbE NICs, only 1 of the 2 ports of the MX7116n was needed.  In a redundant pair, the MX7116n first port provided the 2 x 25GbE connections required per dual CPU blade server.  The addition of a QUAD port NIC means we now have to use the 2nd port on each MX7116n as shown in the diagram.  This means adding in another pair of 200GbE uplinks to your MX9116n switch pair.

You Can Get Up to 400GbE Per Blade Server

If you want to use both Fabric A and Fabric B in the PowerEdge MX7000 chassis, you can get 8 x 25GbE NICs (200GbE) per PowerEdge MX740c blade server.  Need more?  Move to the double-wide 4 processor PowerEdge MX840c and get 16 x 25GbE NICs (400GbE) per blade server.

 

Kevin Houston is the founder and Editor-in-Chief of BladesMadeSimple.com.  He has over 23 years of experience in the x86 server marketplace.  Since 1997 Kevin has worked at several resellers in the Atlanta area, and has a vast array of competitive x86 server knowledge and certifications as well as an in-depth understanding of VMware virtualization.  Kevin has worked at Dell since August 2011 supporting enterprise server sales as a Principal Technologist.   Follow him on Twitter @Kevin_Houston

 Disclaimer: The views presented in this blog are personal views and may or may not reflect any of the contributors’ employer’s positions. Furthermore, the content is not reviewed, approved or published by any employer.