Cisco Nexus 5000 bridges the network gap

A switch that promises to deliver the services of Ethernet and FC over the same wire without packet losses and without appreciable latency is certainly worth reviewing.

In addition to 10G Ethernet ports, our test unit mounted some native FC ports, which made possible running tests to evaluate its behavior when emulating a native FC switch. Other items in our test plan were exploring the management features of the Nexus 5000 and running performance benchmarks to measure latency, I/O operations, and data rate.

The Nexus 5020 is a 2U rack mounted unit and packs in that small space an astonishing number of sockets: 40 to be precise. Each socket can host Ethernet ports running at 10G. Using an optional expansion module (the switch has room for two), you can extend connectivity with six more 10G Ethernet ports, eight more FC ports, or a combo module with four FC and four 10G Ethernet ports.

However, those sockets don't need to be completely filled. For example, our test unit had only 15 10G ports and 4 FC ports active. At review time the Nexus 5000 offered support for all FC connectivity speeds, up to but not including 8G.

Typically, you would deploy the 5020 in the same rack where your app servers reside, or in an adjacent rack. Considering a resilient configuration with two 10G connections for each server, two Nexus 5000 can connect up 40 servers and still have room for more ports with the expansion modules.

The front of the 5000 hosts five large, always spinning and rather noisy fans. With only one power supply (a configuration with dual PSU is also available) we measured around 465 watts absorbed by the switch. Interestingly, the Nexus kept running when we removed one of the fans but, as weI had been warned, shut down automatically when weremoved a second fan. However, the remaining three fans kept spinning to keep the internal electronics cool.

Policing with a policy
Obviously, the most important novelty that the Nexus 5000 brings to a datacenter and the greatest differentiator with other, single protocol switches is that Ethernet and FC are just two supported applications that you monitor and control from the same administrative interface.

With that in mind it's easy to understand why the Nexus runs a new OS, the NX-OS, which, according to Cisco, inherits and brings together the best features of their Ethernet-focused IOS and their FC focused SAN-OS.

To access the OS features administrators can choose between a powerful CLI or the GUI-based Fabric Manager. We used the plural because the administrative tasks of the switch can be easily divided between multiple roles, each with a different login and confined to a specific environment, as defined by and under the supervision of a super admin. That's a critical and much-needed option if you plan to bring multiple administrative domains and their administrators under the same banner.

This and other configuration setting of the Nexus 5000 are policy-driven, which makes for easy and transparent management. Another remarkable feature is that you can define classes of service that logically isolate different applications.

For example, after logging in to the switch, a simple command such as “sh policy-map interface Ethernet 1/1” listed all traffic statistics on that port, grouped for each CoS (class of service) and listing separated numbers for inbound and outbound packets.

Combining a certain CoS with a proper policy, an admin can not only monitor what traffic is running on the switch but can also automatically control where packets are routed and how. Load balancing is a typical application where that combination of policy and QoS shines, but there are others — for example, automatically assigning packets with different MTU to different classes of traffic.

The NX-OS makes easy some otherwise challenging settings, such as mirroring the traffic flowing on one interface to another on the same or on a different VLAN. A similar setting can be useful for sensitive applications such as surveillance and remote monitoring, but can also help test the impact of a new application on a production VLAN.

Defining a correct policy can help also make sure that FC traffic, or any other traffic running on the 5000, will never drop a frame. Dropping a frame is obviously a mortal sin if a storage device is at one end of the connection, but other performance-sensitive applications can benefit from uninterrupted transport.

How to fill up a 10G line
If setting that policy up was easy, testing that it was actually working was a bit more complicated and called for using the powerful features of IP Performance Tester, a traffic generator system by Ixia. One of the problems we had to solve was how to create significant traffic on my 10G connections, which is where IP Performance Tester, luckily already installed in our test system, was called to action.

For the PFC test, the Ixia system was set to generate enough traffic to cause a level of congestion which would have translated, without PFC, into losing packets. The switch under test passed this test with aplomb and without losses, proving that not only FC but also Ethernet can be a reliable, lossless protocol.

Of the many test scripts we ran on the Nexus 5000 this was, without any doubt, the most significant. The switch offers many powerful features, including guaranteed rate of traffic, automatic bandwidth management, and automated traffic span.

However, PFC is what legitimates FCoE as a viable convergence protocol that can bridge the gap between application servers and storage, and it makes the Nexus 5000 a much-needed component in datacenter consolidation projects.

One last question remained still unanswered in my evaluation: The Nexus 5000 had proven to have the features needed to be the connection point between servers and storage in a unified environment, but did the machine have enough bandwidth and responsiveness for the job?

To answer those we moved the testing to a different setting where the Nexus 5020 was connected to 8 hosts running NetPipe.

NetPipe is a remarkable performance benchmark tool that works particularly well with switches because you can measure end-to-end (host-to-host) performance and record (in Excel-compatible format) how those results vary when using different data transfers sizes.

In essence you can set NetPipe to use one way or bidirectional data transfers and increase the data transfer size gradually within a range., recording the transfer rate in megabytes per second and the latency in microseconds..

A step for network consolidation
When reviewing products such as the Nexus 5000 that bear the first implementation of an innovative technology is often difficult to maintain judgments about of the technology separated from that about the solution. Which is probably why, at the end of our evaluation, we tend to think of the Nexus 5020 and of FCoE as a whole — which they are, because at the moment there is no other switch that let you implement the new protocol.

However, even if we break apart the two, each piece has merits of its own. We like the unified view that FCoE brings to network transport and I like the speed and feather-light impact that the Nexus 5020 brings to that union.

Obviously the Nexus 5000 is a first version product and however well rounded, it's easy to predict that future versions will move up the bar even further. As for the technology, perhaps the greatest endorsement that FCoE received is that Brocade is planning to ship a Nexus 5000 rival solution, based on FCoE by year's end. Obviously the old “if you can't beat them, join them” battle cry of competition is still alive and well in the storage world.