Network World lab test: High-end enterprise switch/routers
Scheduled for publication in late March 2001
Test methodology
v. 1.22 Copyright Ó 2000 by Network Test Inc. Vendors are encouraged to comment on this document and any other aspect of test methodology. Network Test reserves the right to change the parameters of this test at any time.
By David Newman
Please forward comments to dnewman@networktest.com
This document describes the methodology to be used in a comparison of high-end enterprise switch/routers This test has four main areas:
This document is organized as follows. This section introduces the test. Section 2 describes device requirements and test equipment. Section 3 describes test procedures. Section 4 describes the change history of this document.
This section discusses requirements of systems under test and introduces the test equipment to be used.
Participating vendors must supply two chassis, each equipped as follows:
The principal test instrument for this project is the SmartBits traffic generator/analyzer manufactured by Spirent Communications Inc. (Chatsworth, Calif.). Spirent’s SmartBits 6000 chassis will be equipped with the company’s new LAN-3201 and/or LAN-6201 interfaces. These cards have 1000Base-SX gigabit Ethernet interfaces and layer 2/3/4 capabilities.
The SmartBits hardware will use SmartFlow and SmartWindow software and scripts custom-developed for this project.
For each routine in this section, this document describes:
· the test objective;
A primary design goal of this methodology is to complete all events in 2.5 working days per vendor.
Determine forwarding rate/packet loss, latency, and jitter of single blade
Determine forwarding rate/packet loss, latency, and jitter of single chassis
One chassis equipped with at least 32 gigabit Ethernet (1000Base-SX) interfaces
Separate IP subnet configured for each interface, as follows:
Interface 1: 10.1.0.0/16
Interface 2 10.20.0/16
Interface 3 10.3.0.0/16
…
Interface 32: 10.32.0.0/16
Test traffic will represent 224
unique host IP addresses per subnet. The
host IP addresses on each subnet will use the IP addresses 10.x.0.2 through 10.x.0.225.
The test traffic shall consist of 64- and 1,518-byte IP packets[1] (offered in separate runs) using a bidirectional traffic orientation and fully meshed distribution. See RFC 2285 for definitions of traffic orientation and distribution.
Offer traffic to 8 interfaces on the same blade. Determine throughput, latency, jitter, packets received in sequence, and packet loss at maximum offered load.
Repeat test with fully meshed traffic offered to all 32 interfaces on all blades. This test will use 217 (not 224) unique host IP addresses per subnet. (In a 32-interface full mesh, each interface addresses 31 destination interfaces. The Smartbits requires that the number of hosts must be an integer multiple of the number of destination interfaces – ergo, 217 rather than 224.)
Throughput
Latency
Jitter
Packets received in sequence
Packet loss at maximum forwarding rate
Determine maximum number of gigabit Ethernet links that can be aggregated
Determine throughput, latency, and jitter using aggregated links
Determine ability to dynamically add individual links to an existing aggregate link
Determine ability to dynamically drop individual links from an existing aggregate link
On each of two chassis, aggregate as many links into a single aggregated link as the device will support, up to a maximum that shall not exceed ½ the number of interfaces per chassis.
Use remaining interfaces for input and output of test traffic.
The test traffic shall consist of 64- and 1,518-byte IP packets (offered in separate runs) using a bidirectional traffic orientation and partially meshed distribution. See RFC 2285 for definitions of traffic orientation and partial mesh.
Configure the device under test to create an aggregate link consisting of the maximum number of individual links supported. Offer test traffic to all non-aggregated interfaces of each chassis, destined for all non-aggregated interfaces of the opposite chassis. Measure throughput by following the procedure described in RFC 2544, section 26.1.
If a device supports multiple aggregated links, repeat above procedure with maximum number of active aggregated links.
To test addition of an individual link to an existing aggregated link: Configure an aggregated link consisting of L-1 links, where L represents the maximum number of individual links that can be aggregated into one larger unit. Offer test traffic to all non-aggregated interfaces of each chassis, destined for non-aggregated interfaces of the opposite chassis. The offered load shall not exceed the channel capacity of the aggregated link. While test traffic is being offered, configure the device to add one link to the aggregated link. The test duration must exceed the link addition by at least 30 seconds. Record any packet loss and/or drop in throughput during the test duration.
To test removal of an individual link to existing aggregated link: Configure an aggregated link consisting of L links, where L represents the maximum number of individual links that can be aggregated into one larger unit. Offer test traffic to all non-aggregated interfaces of each chassis, destined for all non-aggregated interfaces of the opposite chassis. The offered load shall not exceed the capacity of L-1 links in the aggregated link. While test traffic is being offered, configure the device to remove one link from the aggregated link. The test duration must exceed the link addition by at least 30 seconds. Record any packet loss and/or drop in throughput during the test duration.
Maximum number of links that can be aggregated
Throughput
Latency
Jitter
Packets received in sequence
To determine failover time upon failure of primary link
To determine failover time upon failure of primary aggregated link
2 chassis with one primary link (single gigE) and secondary link (single gigE) configured between chassis
For devices that support failover across aggregated links, 2 chassis with one primary aggregated link and one secondary aggregated link
Offer traffic at a rate of 1,000,000 pps. Physically remove the primary backbone link. Note the time for traffic to be rerouted over the primary link.
Repeat the procedure for devices supporting failover across aggregated links.
Packet loss
Switchover time (determined from packet loss)
Case 1: Determine whether devices under test allocate fixed bandwidth allocations to given traffic classes during periods of congestion
Case 2: Determine whether devices under test allocate any available bandwidth to a given traffic class during periods of congestion
Two chassis connected via one gigabit Ethernet circuit. For devices that support link aggregation, this test will be repeated with an aggregated link consisting of 8 gigabit Ethernet links.
The test traffic shall consist of 128-byte IP packets offered in a bidirectional orientation and a partially meshed distribution. See RFC 2285 for definitions of traffic orientation and partial mesh.
The test traffic shall be divided into three classes, distinguished by different TCP port numbers:
Priority |
Traffic type |
Destination TCP port |
High |
DLSw |
2065 |
Medium |
Web |
80 |
Low |
FTP |
20 |
The Smartbits will offer all packets with an IP precedence value set to 1. The devices under test must reclassify traffic using IP precedence levels 7 for high priority traffic, 3 for medium-priority traffic, and 1 for low-priority traffic.
To create congestion, the aggregate offered load of all classes of test traffic will exceed the capacity of the backbone link (or aggregated link) by a ratio of 2:1.
For all tests, traffic will be offered to the device under test in a high/medium/low ratio 1:10:10 for all classes.
Devices under test must be configured to enforce the following rules:
Case 1: Offer all three classes of traffic to 7 interfaces on chassis A in a high/medium/low ratio of 1:10:10, at a rate to oversubscribe channel capacity by a factor of 2. Observe output ratio on destination interfaces; an ideal result is a ratio of approximately 2:12:7 between high, medium and low classes of traffic.
The traffic classes should be distributed as follows:
Traffic class |
Aggregate offered load (Mbit/s) |
Target aggregate output load (Mbit/s) |
High |
82.367 |
82.367 |
Medium |
823.668 |
494.201 |
Low |
823.668 |
288.284 |
Total packets |
1,729.704 |
864.852 |
Total packets, preamble and interframe gap |
2,000.000 |
1,000.000 |
Case 2: Offer medium- and low-priority classes of traffic to two interfaces on chassis A in a medium/low ratio of 10:10, at a rate to oversubscribe channel capacity by a factor of 2. Observe output ratio on destination interfaces; an ideal result is a ratio of 5:3 between medium and low classes of traffic.
The traffic classes should be distributed as follows:
Traffic class |
Aggregate offered load (Mbit/s) |
Target aggregate output load (Mbit/s) |
Medium |
864.852 |
540.532 |
Low |
864.852 |
324.320 |
Total packets |
1,729.704 |
864.852 |
Total packets, preamble and interframe gap |
2,000.000 |
1,000.000 |
Case 1: Throughput of high-, medium-, and low-priority traffic
Ratio of high-, medium-, and low-priority traffic
Packets delivered in sequence
Case 2: Throughput of medium- and low-priority traffic
Ratio of medium- and low-priority traffic
Packets delivered in sequence
Version 1.22
Date: 22 May 2001
Fixed typo in HTML title
Version 1.21
Date: 27 February 2001
Fixed typo in introduction
Changed text of section 3.4.3, case 2, to indicate only medium- and low-priority traffic is offered
Version 1.20
Date: 9 February 2001
Changed partial mesh to full mesh error in section 3.1.2
Changed target data rates in section 3.4 to compensate for Ethernet interframe gap and packet preamble
Version: 1.12
Date: January 2001
Corrected typos in version 1.11
Version: 1.11
Date: January 2001
Changed section 3.1.2 to reflect full mesh
Version 1.10
Date: December 2000
Added target data rates in section 3.4 on QOS
Version 1.0
Date: November 2000
Initial release
[1] All references to packet lengths in this document refer to IP over Ethernet. We measure packet length from the first byte of the Ethernet MAC header to the last byte of the CRC. Unless otherwise specified, IP packets consist of IP headers but not upper-layer headers (e.g., TCP, UDP, HTTP, etc.).
