Network World Clear Choice Test: 10G ToR Data Center Switches

Test Methodology

Version 2010011800. Copyright 1999-2010 by Network Test Inc. Vendors are encouraged to comment on this document and any other aspect of test methodology. Network Test and Network World reserve the right to change test parameters at any time.

PDF version: http://networktest.com/tor09/tor09meth.pdf

1 Executive summary

This document describes benchmarking procedures for 10-gigabit Ethernet top-of-rack data center switches. Test results are published in the 18 January 2009 issue of Network World.

Given that Network World’s readership is comprised of enterprise network managers, the key emphases of this project will be performance, manageability, usability and features in an enterprise context. As described in detail below, tests cover the following areas:

· Switch features

· Switch usability

· Power consumption

· MAC address capacity

· Forward pressure

· Unicast throughput/latency/jitter/sequencing

· Multicast throughput/latency/jitter/sequencing

· Multicast group capacity

· Multicast group join/leave delay

· Link aggregation hashing fairness

This document is organized as follows. This section introduces the tests to be conducted. Section 2 describes the test bed. Section 3 describes the tests to be performed. Section 4 provides a change log.

2 The test bed

This section discusses requirements of systems under test and introduces the test equipment to be used.

2.1 Devices under test

Participating vendors should supply the following:

One data center switch equipped with at least 24 10-gigabit Ethernet ports.

At least 26 optical transceivers for the 10-gigabit ports. We assume the use of 10GBASE-SR XFP transceivers. Two extra transceivers are requested for sparing.

2.2 Test instruments

2.2.1 Spirent TestCenter

The primary instrument for performance assessment in this project is Spirent TestCenter. Spirent has supplied a 9U test chassis equipped with 8-port HyperMetrics CV 10-gigabit Ethernet modules with XFP 10GBase-SR transceivers.

We use Spirent TestCenter Application and firmware version 3.33.

All Spirent TestCenter interfaces use a single clock source. The deficit idle count (DIC) mechanism in 10-gigabit Ethernet is enabled on all ports, and all ports are configured to use a maximum transmission unit (MTU) of 9216 bytes.

2.2.2 Fluke True-rms Clamp Meter 335

The power consumption measurement instrument for this project is a Fluke True-rms Clamp Meter 335. Power consumption tests also use a WaveTek Meterman ELS2 line splitter to avoid the need to split power cords.

3 Test procedures

This section describes the test procedures. This document follows the template given in most methodology documents of the IETF’s benchmarking working group, describing the following for each test:

· the test objective(s);

· the configuration to be used;

· the procedure to be used;

· the test metrics to be recorded;

· reporting requirements.

3.1 Switch features

3.1.1 Objective

To determine the feature set supported by the DUT

3.1.2 Test bed configuration

Not applicable

3.1.3 Procedure

We ask participating vendors to complete a features questionnaire listing various attributes supported by the DUT. Examples of such attributes include the number and type of physical interfaces; layer-2 and layer-3 switching and routing protocols; VLAN support; spanning tree support; discovery protocol support; anti-spoofing and anti-DOS protection mechanisms; and management methods.

The questionnaire will examine what distinguishes a “data center switch” from a “switch,” focusing on redundancy features and support for data center protocols such as data center bridging protocols and Fibre Channel over Ethernet (if supported).

The questionnaire includes space for vendors to describe features not covered by the various questions.

Network World will publish the results of the features questionnaire, usually in its online edition. The publication should include a caveat that responses are supplied by vendors, and not all features have been verified by Network World.

3.1.4 Metrics

Features supported

3.1.5 Reporting requirements

Features questionnaire

3.2 Switch usability

3.2.1 Objectives

To determine ease of use for common switch management functions

3.2.2 Test bed configuration

This test requires no specific configuration. Subjective impressions of switch usability are gathered in the course of performing all tests described in this document.

3.2.3 Procedure

This is a subjective assessment of each switch’s manageability. In the course of conducting the other tests described in this document, testers will gather impressions about the intuitiveness of the switch’s user interface. Testers also will focus on common management tasks such as examining switch statistics and error counters; provisioning VLANs; and enabling and disabling various L2 and L3 functions.

Unlike objective test results, Usability impressions are not shared with vendors prior to test publication.

3.2.4 Metrics

N/A

3.2.5 Reporting requirements

Usability impressions, included in final test report

3.3 Power consumption

3.3.1 Objectives

To determine the power consumption of the DUT when idle

To determine the power consumption of the DUT when fully loaded

3.3.2 Test bed configuration

This test uses the following equipment:

· Fluke 335 True-RMS clamp meter

· WaveTek ELS2 AC line splitter

· Spirent TestCenter chassis

The DUT plugs into the line splitter and the clamp meter measures power consumption through the line splitter. The Spirent TestCenter chassis attaches to all 24 10G Ethernet interfaces of the DUT for a “fully loaded” test, and to 12 10G Ethernet interfaces for a “half-loaded” test. In the 12-port test case, the DUT should be populated with only 12 10GBase-SR transceivers.

This test will measure power consumption when idle and again when fully loaded, both in the 12- and 24-port cases.

Test traffic will comprise 64-byte UDP/IP frames offered in a fully meshed pattern.

3.3.3 Procedure

1. Using the clamp meter and leads, measure AC voltage from the power outlet. We refer to this measurement as V. If necessary, repeat for all power outlets used by the DUT.

2. Plug the DUT into the line splitter and verify the system has booted up.

3. Place the clamp meter jaws around the “10X” receptacle of the line splitter.

4. The clamp meter will display AC amps drawn by the DUT times 10. We refer to this figure as 10A.

5. Derive idle-DUT power consumption in watts (W) using the formula W = V * (10A/10).

6. Using Spirent TestCenter, offer 64-byte frames to all 24 interfaces at line rate for at least 60 seconds.

7. Repeat steps 3-5 to determine maximum-load power consumption with 24 ports.

8. For devices with multiple power supplies, repeat all previous steps for each power supply. Add wattage from each power supply to determine total system power consumption.

9. Remove cables and transceivers from 12 interfaces and repeat steps 3-8 above for power measurement with 12-port fully meshed traffic pattern.

3.3.4 Metrics

Supplied power (volts AC)

Idle power consumption (watts)

24-port maximum-load power consumption (watts)

12-port maximum load power consumption (watts)

3.3.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.4 MAC address capacity

3.4.1 Objectives

To determine the MAC address caching capacity of the DUT as defined in RFC 2285, section 3.8.1.

3.4.2 Test bed configuration

This test can be configured from the RFC 2889 wizard on Spirent TestCenter.

As described in RFC 2889, this test involves three ports: a learning port, a test port and a monitor port.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

To avoid contention, the wire should be “quiet” except for test traffic. Spanning tree, LLDP, SNMP, proprietary discovery mechanisms and all other management traffic should be disabled for this test.

This test involves a binary search to determine the maximum number of MAC addresses the DUT can learn without flooding. Vendors must declare the expected MAC address capacity of the DUT before the test, allowing Network Test to configure appropriate minimum, maximum and initial MAC address counts to be used in the binary search.

The Spirent TestCenter test instrument MUST offer traffic at a rate less than or equal to the DUT’s MAC address learning rate and MUST use an aging time equal to twice the MAC address aging time set on the DUT.

For example, assume the DUT can learn MAC addresses at a rate of at least 1,000 addresses/second, and that the DUT has a stated capacity of 10,000 MAC addresses.

In this case, the DUT and Spirent TestCenter would be configured with these values:

DUT aging time: 60 seconds

Spirent TestCenter MAC address contents: RFC 4814 pseudorandom

Spirent TestCenter minimum addresses: 1

Spirent TestCenter maximum addresses: 10,000

Spirent TestCenter initial addresses: 2,500

Spirent TestCenter aging time: 120 seconds

Spirent TestCenter learning rate: 1,000 addresses/second

Spirent TestCenter frame size: 64 bytes

Spirent TestCenter latency type: Not meaningful (Latency not measured as part of this test)

Spirent TestCenter offered load: 10% of line rate

Spirent TestCenter test duration: 10 seconds

These are sample values. Actual parameters may be adjusted depending on DUT MAC address capacity and learning rate. Note that the initial address count is 2,500 addresses, or 25% of the maximum count. Regardless of actual maximum count, the initial address count must be 25% of the maximum.

3.4.3 Procedure

This procedure is adapted from RFC 2889, section 5.7.3.

The Learning port (Lport) transmits learning frames to the DUT with varying source addresses and a fixed destination address corresponding to the address of the device connected to the Test port (Tport) of the DUT. By receiving frames with varying source addresses, the DUT should learn these new addresses.

The Test port (Tport) of the DUT acts as the receiving port for the learning frames. Test frames will be transmitted back to the addresses learned on the Learning port. The Monitoring port (Mport) on the DUT acts as a monitoring port to listen for flooded or mis-forwarded frames.

Using a binary search approach, the test targets the exact number of addresses supported with consistent 60-second test iterations. Due to the aging time of DUT address tables, each iteration may take some time during the waiting period for the addresses to clear; as noted in “Test bed configuration” the test instrument’s aging time MUST be twice that of the DUT’s. Once the high and low values of N meet, then the threshold of address handling has been found.

To smooth out variability among results, repeat above procedure five times. Discard the lowest and highest results and average the remaining three.

3.4.4 Metrics

MAC address capacity

3.4.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.5 Forward pressure

3.5.1 Objectives

To determine if the DUT applies forward pressure as defined in RFCs 2285 and 2889.

3.5.2 Test bed configuration

This test involves all 24 ports of the DUT. We configure Spirent TestCenter to offer fully meshed traffic among all interfaces. RFC 2285 describes traffic orientation and distribution.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

Test traffic offered to all ports will have 25 MAC addresses per port, and will use pseudorandom MAC addresses as described in RFC 4814. The DUT should be configured to support switching of jumbo frames.

The DUT must be configured so that entries in its bridging table will not age out during the test. MAC address aging should be disabled altogether, if the switch supports that feature.

The DUT must be configured to disable spanning tree, routing protocols, multicast and any other protocols that might put control-plane traffic on the wire during the test duration. The goal of this test is to determine maximum data-plane performance, and the existence of even one extra frame other than test traffic can lead to frame loss.

3.5.3 Procedure

The test instrument offers fully meshed traffic to each port of the DUT with an interframe gap of 88 bits, for an interframe gap (IFG) of 11 bytes. This will apply forward pressure to the DUT and overload it at a rate of one byte per frame. The forwarding rate on all destination ports of the DUT is measured. The measured forwarding rate should not exceed the medium's maximum theoretical utilization (MOL).

3.5.4 Metrics

Forward pressure (true/false)

3.5.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.6 Unicast throughput / latency / jitter / sequencing

3.6.1 Objectives

To determine the unicast throughput rate of the DUT as defined in RFC 1242

To determine the average and maximum unicast latency of the DUT as defined in RFC 1242

To determine the average and maximum unicast jitter of the DUT as defined in RFC 1242

To determine frames in and out of sequence at the unicast throughput rate

3.6.2 Test bed configuration

This test involves all 24 ports of the DUT. We configure Spirent TestCenter to offer fully meshed traffic among all interfaces. RFC 2285 describes traffic orientation and distribution.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

The DUT must be configured so that entries in its bridging table will not age out during the test. MAC address aging should be disabled altogether, if the switch supports that feature.

As required by RFC 1242, latency will be measured using the LILO method for store-and-forward devices and the FIFO method for cut-through devices. Time permitting, we may also repeat these tests using FILO timestamping for all devices at the throughput rate for a given frame length.

These are sample values. Some values, especially throughput intended loads, will be modified depending on DUT capability.

3.6.3 Procedure

1. We perform a learning run to populate the DUT’s bridging table. Results from this run are not recorded.

2. Using a binary search algorithm, we offer fully meshed streams of test traffic to all interfaces for 60 seconds to determine the throughput rate and, at the throughput rate, latency, jitter and frames received out of sequence.

3. We repeat the previous step for each of the following Ethernet frame lengths: 64, 108, 256, 1518 and 9216 bytes.

3.6.4 Metrics

Throughput (frames/second)

Average and maximum latency (nanoseconds or microseconds)

Average and maximum jitter (nanoseconds or microseconds)

Frames out of sequence (frame count)

3.6.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.7 Multicast group capacity

3.7.1 Objectives

To determine the maximum number of IGMP multicast groups the DUT can support while maintaining the ability to forward multicast frames to all multicast groups registered to the DUT

3.7.2 Test bed configuration

This test can be configured from the RFC 3918 wizard on Spirent TestCenter.

This test involves 24 ports: One transmitter port and 23 subscriber ports

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

All DUT ports should be members of VLAN 1. The DUT should be configured to support IGMPv2 snooping and to act as an IGMPv2 querier with an address of 10.0.0.1/16.

Spirent TestCenter should be configured to support IGMPv2 multicast transmitters and receivers, with one transmitter per multicast group. Multicast group addresses will begin at 225.0.1.0. The Spirent TestCenter test instrument will emulate one host per port, using the address range 10.0.0.2-10.0.0.25/16 inclusive, and will use RFC 4814 pseudorandom MAC addresses. The test instrument MUST offer IGMP reports at a rate less than or equal to the multicast learning rate of the DUT.

The DUT must be configured to disable spanning tree and unicast routing protocols.

3.7.3 Procedure

Each subscriber port will join 100 multicast groups (or some number other than 100 if the switch vendor declares what number of IGMP groups the DUT supports). After a delay of 60 seconds, Spirent TestCenter will transmit 1,518-byte multicast traffic to all subscribers in all groups at an aggregate rate of 10 percent of 10G line rate for a duration of 10 seconds. If Spirent TestCenter receives one or more frames per group, the iteration is said to pass at the current capacity, per RFC 3918. If the monitor port receives multicast traffic and/or if one or more multicast groups on the test port fails to receive traffic, the iteration is said to fail.

The test instrument will use a binary search algorithm to determine the maximum number of groups supported. The test stops iterating when one or more of the egress interfaces fails to forward traffic on one or more of the configured multicast groups.

3.7.4 Metrics

Total number of multicast group addresses successfully forwarded through the DUT

3.7.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.8 IGMP group join/leave delay

3.8.1 Objectives

To determine the time duration it takes the DUT to start forwarding multicast frames from the time a successful IGMP report has been issued to the DUT

3.8.2 Test bed configuration

This test can be configured from the RFC 3918 wizard on Spirent TestCenter.

This test involves all 24 ports of the DUT: One transmitter port, 22 subscriber ports for all 989 multicast groups defined and one monitor port.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

All DUT ports should be members of VLAN 1. The DUT should be configured to support IGMPv2 snooping and to act as an IGMPv2 querier with an address of 10.0.0.1/16. The Spirent TestCenter test instrument will emulate one host per port, using the address range 10.0.0.2-10.0.0.25/16 inclusive, and will use RFC 4814 pseudorandom MAC addresses. The test instrument MUST offer IGMP reports at a rate less than or equal to the multicast learning rate of the DUT.

The total number of multicast groups will be 989, which is the closest integer multiple to 1,000 possible with 23 subscriber ports. Multicast group addresses will begin at 225.0.1.0, and will increment using a mask of 0.0.0.1. All subscribers will join all groups. The test instrument MUST offer IGMP reports at a rate less than or equal to the multicast learning rate of the DUT.

The DUT must be configured to disable spanning tree and unicast routing protocols.

3.8.3 Procedure

1. The test instrument will perform layer-2 learning on all ports to populate the DUT’s MAC address forwarding table.

2. The test instrument will offer 256-byte frames to one port, destined to all multicast group addresses on all other ports in a one-to-many pattern at an intended load of 10 percent of line rate.

3. At least 15 seconds after the test instrument begins offering traffic on all ports, emulated hosts will join all multicast groups sourced from all other ports. The interval between transmission of each join message and the DUT forwarding traffic to that IGMP subscriber is the group join delay.

4. At least 15 seconds after the completion of sending IGMP join messages, all emulated hosts will send IGMP leave messages for all groups. The interval between transmission of each leave message and the DUT ceasing to forward traffic to that IGMP subscriber is the group leave delay.

5. Repeat steps 1-4 five times. To smooth out variability in test results, discard results from trial iterations with the highest and lowest average join delay. Average the results of all remaining iterations.

3.8.4 Metrics

Average/maximum group join delay (microseconds)

Average/maximum group leave delay (microseconds)

Packet loss (percent of expected frames)

3.8.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.9 Multicast throughput, average/max latency, jitter (23 ports)

3.9.1 Objectives

To determine the multicast throughput rate of the DUT as defined in RFC 1242 and RFC 2432

To determine the average and maximum multicast latency of the DUT as defined in RFC 1242

To determine the average and maximum multicast jitter of the DUT as defined in RFC 1242

To determine frames in and out of sequence at the multicast throughput rate

3.9.2 Test bed configuration

This test can be configured from the RFC 3918 wizard on Spirent TestCenter.

This test involves all 24 ports of the DUT, with one port acting as a multicast source and all remaining 23 ports acting as multicast receivers for all 989 IGMP groups defined.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

The DUT must be configured to disable spanning tree and unicast routing protocols.

3.9.3 Procedure

1. Prior to running any tests, we perform L2 learning on all 24 active ports to populate the DUT’s bridging table. The test instrument emulates one host per port.

2. The test instrument sends IGMP join messages from subscribers on 23 ports to 989 IGMPv2 groups, starting at 225.0.1.0 and incrementing by 0.0.0.1. The join rate must not exceed the multicast learning rate of the DUT.

3. Before offering test traffic, the testers will verify that all expected subscribers have joined all groups. If the DUT cannot operate with 989 IGMP groups, the test will be conducted with fewer groups (with the appropriate asterisk applied to test results).

4. Using a binary search algorithm, the test instrument offers test traffic to the first DUT port, destined to all 989 multicast group addresses on all other ports to determine the throughput rate and, at the throughput rate, average/maximum latency, average/maximum jitter and frames received out of sequence (if any). The test duration is 60 seconds.

5. We repeat the previous step for each of the following Ethernet frame lengths: 64, 108, 256, 1518 and 9216 bytes.

6. Testers may spot-check for flooding by rerunning some or all above tests with 22 subscriber ports, with the last port used as a monitor.

3.9.4 Metrics

Aggregated multicast throughput (frames/second)

Aggregated multicast forwarding rate (frames/second)

Average and maximum latency (nanoseconds or microseconds)

Average and maximum jitter (nanoseconds or microseconds)

Frames out of sequence (frame count)

Flooded frames (pass/fail)

3.9.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

3.10 Link aggregation hashing fairness

3.10.1 Objectives

To determine the distribution of pseudorandom traffic across an eight-member link aggregation group
To determine the distribution of pseudorandom traffic across a seven-member link aggregation group

3.10.2 Test bed configuration

This test involves up to16 ports on both the DUT and Spirent TestCenter. Eight ports on each side function as “edge” ports. On the DUT, the remaining eight ports should be configured as a single link aggregation group (LAG) running link aggregation control protocol (LACP) in active mode. LACP timing should be set to slow (updates every 30 seconds). A second test with a seven-member LAG will verify LACP functionality and hashing fairness.

Spirent TestCenter has a “device behind device” capability that allows it to emulate both a LAG and hosts behind the LAG. Test traffic will be unidirectional (from edge ports to LAG) so that hashing across the LAG is performed only by the DUT.

Deficit idle count (DIC) MUST be enabled on 10-gigabit Ethernet switch ports and on the Spirent TestCenter test instrument.

Test traffic offered to all ports will have 75 MAC addresses per port, and will use pseudorandom MAC addresses as described in RFC 4814. Test traffic shall consist of raw Ethernet frames with no upper-layer IP or UDP/TCP headers. The DUT should be configured to support switching of jumbo frames.

The DUT must be configured so that entries in its bridging table will not age out during the test. MAC address aging should be disabled altogether, if the switch supports that feature.

3.10.3 Procedure

1. On the test instrument, manually start LACP and verify all eight LAG members are in “Up” state.

2. Manually conduct “L2 learning” and “L2 Rx learning” for all stream blocks.

3. Offer 64-byte unicast frames in unidirectional pattern for a 60-second duration at 10 percent of line rate, or at the throughput rate if the DUT’s link aggregation throughput is less than 10 percent of line rate.

4. At the end of the test, note the received frame counters on each of the eight link aggregation group members on the test instrument. These counters indicate distribution of traffic across the LAG.

5. Disable first interface of LAG on test instrument, and repeat all previous steps with seven-member LAG.

3.10.4 Metrics

8-member LAG distribution (histogram)

8-member LAG distribution (standard deviation)
7-member LAG distribution (histogram)

7-member LAG distribution (standard deviation)

3.10.5 Reporting requirements

DUT configuration and software version

Spirent TestCenter configuration and software version

Test results

4 Change history

Version 2010011800

18 January 2010

Section 1: Specified publication date of 18 January 2009

Section 3.4.2: Changed initial address count to 25% of maximum (was 100%); added text about starting at 25% of maximum

Section 3.4.3: Added text about five-trial trimming method

Section 3.7.2: MGC test involves one Tx and 23 Rx ports (was one Tx, 22 Rx and one monitor port)

Section 3.8.2: JLD test involves one Tx, 22 Rx and one monitor port

Section 3.8.3: Added step 5 describing five-trial trimming method

Version 2009103001

30 October 2009

Section 1, 3.5 (former), 3.11: Cut MAC address learning rate test; Cut link aggregation throughput/latency jitter test

Section 2.2.1: Upgraded to Spirent TestCenter version 3.33 to address issues with multicast testing

Sections 3.3.2-3: Confirmed 12- and 24-port versions of power consumption test (was only on a time-permitting basis in previous version)

Section 3.4.3: Changed test duration from 10 to 60 seconds, per Spirent TestCenter RFC 2889 wizard usage.

Section 3.6.3: Reduced test duration from 300 to 60 seconds

Section 3.7.2: Increased port count from three to 24 (one transmitter port, 22 subscriber ports, and one monitor port)

Sections 3.8.2-3, 3.9.2-3: Changed topology from many-to-many to one-to-many; changed group count from 600 to 989

Section 3.9.3: Added spot-check for flooding of multicast traffic

Sections 3.10.2-4: Added seven-member LAG tests

19 October 2009

Initial public release