Network Test :: Benchmarking Services Network Test Methodologies::Results Contact Information About Network Test

Light Reading/Byte & Switch Lab Test: 2-Gbit/s SAN Fabric Switches

Published 17 January 2002

Test Methodology

 

v. 2.01 Copyright 2001-2 by Network Test Inc. We welcome comments and suggestions about this document and any other aspect of test methodology. Network Test reserves the right to change the parameters of this test at any time.

 

By David Newman

 

Please forward comments to dnewman@networktest.com

 

1         Executive summary

This document describes the methodology to be used in a comparison of 2-Gbit/s SAN fabric switches. All devices must support the new 2-Gbit/s specification for Fibre Channel interfaces.

 

The test focuses mainly on device performance. In all, there are six sets of tests:

 

In addition to the performance tests, we ask participating vendors to complete a features questionnaire. We also ask vendors to supply prices for systems as tested. 

 

This document is organized as follows. This section introduces the test. Section 2 describes device requirements and test equipment. Section 3 describes test procedures. Section 4 describes the change history of this document.

 

2         The test bed

This section discusses requirements of systems under test and introduces the test equipment to be used.

2.1        Devices under test

Participating vendors must supply four SAN fabric switches, each equipped as follows:

 

2.2        Test hardware

The principal test instrument for this project is the SmartBits traffic generator/analyzer manufactured by Spirent Communications Inc. (Chatsworth, Calif.). Spirent’s SmartBits 6000 chassis is equipped with the company’s new FBC-3602A Fibre Channel cards. The Fibre Channel cards emulate N/NL ports and perform automatic rate-sensing between 1 and 2 Gbit/s. More information about the SmartBits system and Fibre Channel cards is available here:

 

http://www.netcomsystems.com/solutions/products/applications/pdf/FBC/index.htm

2.3        Test software

The test application is Spirent’s SmartFabric v. 1.00.16. This application is similar to Spirent’s SmartFlow, in that it allows users to define a very large number (tens to hundreds of thousands) of flows through a graphical interface.

 

SmartFabric uses API v. 2.01.01.00 and chassis library v. 3.10-207.

 

3         Test procedures

For each routine in this section, this document describes:

 

        the test objective;

        the configuration to be used;

        the procedure to be used;

        the test metrics to be recorded.

 

3.1        Single-switch baselines

3.1.1        Objective

To describe the basic forwarding and delay characteristics of a single fabric switch.

 

3.1.2        Test bed configuration

The figure below illustrates the test bed topology. We attach SmartBits test interfaces to 16 interfaces on the device under test.

 

 

 

 

 

 

 

 

 

 

 

 

 


Note that the test bed figure describes only the physical topology of the test bed. As described in sections 3.1.3, 3.4, and 3.5, the logical traffic flows involve multiple many-to-one and many-to-few flows to create a fully meshed traffic pattern.

 

Vendors must configure each interface to support 2-Gbit/s speeds and point-to-point connections to N/NL-port (test) interfaces.

 

3.1.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers frames to the 16 switch interfaces in a bidirectional, fully meshed traffic pattern (traffic offered to each interface is destined for all other interfaces). We measure throughput as well as frames in sequence.

 

We repeat these tests with 60- and 2,148-byte[1] frames.

 

For each frame size, we repeat the test with burst counts of 1 and 8. A burst count describes the number of frames the SmartBits offers to a given destination before moving on to the next destination. With a burst count of 1, the SmartBits offers 1 frame to destination port A, 1 frame to port B, and so on; when the burst count is 8, the SmartBits offers 8 frames to port A, 8 frames to port B, and so on.

 

For throughput measurements, the test duration is 30 seconds.

 

At the throughput level, we also measure latency and latency distribution. To avoid any potential clock drift, the test duration for these delay-related measurements is 1 millisecond.

 

3.1.4         Metrics

Throughput (Mbyte/s)

Latency

Latency distribution (histograms)

Jitter

Frames in sequence

Frames out of sequence

3.2        Multi-switch fabric baselines

3.2.1        Objective

To describe the basic forwarding and delay characteristics of four interconnected fabric switches handling a fully meshed traffic distribution

 

3.2.2        Test bed configuration

The figure below illustrates the test bed topology. We attach SmartBits test interfaces to 10 interfaces on each of the four devices under test. Vendors must configure interswitch links (ISLs) to carry traffic to and from all switches. Vendors should use trunking (also known as link aggregation) if possible to make single logical interfaces from multiple ISLs.

 

Note that there are 2 links between switches (or 6 ISLs total per switch), while there are 10 connections to the test equipment. Despite this imbalance in physical ports, there is no oversubscription of ISLs or ISL trunks. By careful use of SmartBits flows and streams, there is no oversubscription of any port on the test bed.

 

Vendors must configure each interface to support 2-Gbit/s speeds and point-to-point connections to test interfaces.

 

3.2.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers frames to 10 switch interfaces in a bidirectional, fully meshed traffic pattern. We measure throughput as well as frames in sequence.

 

We repeat these tests with 60- and 2,148-byte frames.

 

For each frame size, we repeat the test with burst counts of 1 and 8. A burst count describes the number of frames the SmartBits offers to a given destination before moving on to the next destination. With a burst count of 1, the SmartBits offers 1 frame to destination port A, 1 frame to port B, and so on; when the burst count is 8, the SmartBits offers 8 frames to port A, 8 frames to port B, and so on.

 

For throughput measurements, the test duration is 30 seconds.

 

At the throughput level, we also measure latency and latency distribution. To avoid potential clock drift, the test duration for these delay-related measurements is 1 millisecond.

 

3.2.4         Metrics

Throughput (Mbyte/s)

Minimum, average, and maximum latency

Latency distribution (histograms)

Frames in sequence

Frames out of sequence

 

3.3        Congestion control (head-of-line blocking)

3.3.1        Objective

To determine whether congestion on one output interface affects the forwarding rate or latency on an uncongested output interface.

 

3.3.2        Test bed configuration

The figure below illustrates the test bed topology. Note that only 4 interfaces are involved in this test.

 

3.3.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers traffic to interface if0 at line rate. All of the traffic offered to if0 is destined for the SmartBits interface behind if2. At the same time, the SmartBits also offers traffic to interface if1. Some 50 percent of this traffic is destined for the SmartBits interface behind if2, while the balance is destined for the SmartBits interface behind if3. Thus, an overload of 150 percent exists at if2 while there is no overload at if3.

 

We examine frame loss and latency of traffic from if2 and if3, and compare if3 results with those from the baseline tests.

We repeat this test with 60- and 2,148-byte frames.

 

The test duration is 30 seconds.

 

3.3.4        Metrics

Throughput at congested port (Mbyte/s)

Throughput at uncongested port (Mbyte/s)

Minimum, average, and maximum latency at uncongested port

Latency distribution at uncongested port

3.4        1- and 2-Gbit/s many-to-one tests

3.4.1        Objective

To determine forwarding and delay characteristics of a switch moving traffic from 15 1-Gbit/s interface to 1 2-Gbit/s interfaces.

 

3.4.2        Test bed configuration

The figure below illustrates the test bed topology. Note that the left-hand interface is configured at 2 Gbit/s and all others are configured for 1-Gbit/s rates.

 

3.4.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers traffic at 1/15 of 2-Gbit/s line rate to each of 15 1-Gbit/s interface of the device under test. The traffic is destined for 1 SmartBits interface behind a switch port configured at 2-Gbit/s rates. (The offered load is evenly distributed across the 15 1-Gbit/s interfaces, such that each interface receives 1/15 of 2 Gbit/s’ worth of traffic.) We measure throughput as well as frames in sequence.

 

We repeat this test with 60- and 2,148-byte frames.

 

For each frame size, we repeat the test with burst counts of 1 and 8. A burst count describes the number of frames the SmartBits offers to a given destination before moving on to the next destination. With a burst count of 1, the SmartBits offers 1 frame to destination port A, 1 frame to port B, and so on; when the burst count is 8, the SmartBits offers 8 frames to port A, 8 frames to port B, and so on.

 

For throughput measurements, the test duration is 30 seconds.

 

At the throughput level, we also measure latency and latency distribution. To avoid any potential clock drift, the test duration for these delay-related measurements is 1 millisecond.

 

3.4.4        Metrics

Throughput (Mbyte/s)

Minimum, average, and maximum latency

Latency distribution (histograms)

Frames in sequence

Frames out of sequence

3.5        1- and 2-Gbit/s many-to-few tests

3.5.1        Objective

To determine forwarding and delay characteristics of a switch moving traffic from 12 1-Gbit/s interfaces to 4 2-Gbit/s interfaces.

 

3.5.2        Test bed configuration

The figure below illustrates the test bed topology. Note that the left-hand interfaces are configured at 2 Gbit/s and all others are configured for 1-Gbit/s rates.

 

3.5.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers traffic to each of 12 1-Gbit/s interface of the device under test. The traffic is destined for 4 SmartBits interfaces behind a switch port configured at 2-Gbit/s rates. (The offered load is evenly distributed across the 12 1-Gbit/s interfaces, such that each interface receives 1/12 of 2 Gbit/s’ worth of traffic.) We measure throughput as well as frames in sequence.

 

We repeat this test with 60- and 2,148-byte frames.

 

For each frame size, we repeat the test with burst counts of 1 and 8. A burst count describes the number of frames the SmartBits offers to a given destination before moving on to the next destination. With a burst count of 1, the SmartBits offers 1 frame to destination port A, 1 frame to port B, and so on; when the burst count is 8, the SmartBits offers 8 frames to port A, 8 frames to port B, and so on.

 

For throughput measurements, the test duration is 30 seconds.

 

At the throughput level, we also measure latency and latency distribution. To avoid any potential clock drift, the test duration for these delay-related measurements is 1 millisecond.

 

3.5.4        Metrics

Throughput (Mbyte/s)

Minimum, average, and maximum latency

Latency distribution (histograms)

Frames in sequence

Frames out of sequence

3.6        Switch failover times

3.6.1        Objective

To determine failover time upon failure of primary link

To determine failover time upon failure of primary trunk (aggregated link)

 

3.6.2        Test bed configuration

This test requires 2 chassis with 1 primary and 1 secondary ISL configured between chassis.

 

For devices that support failover across trunks (aggregated links), the tests requires 2 chassis with one primary trunk and one secondary trunk. Each trunk will consist of 4 physical ISLs.

 

Vendors configure all interfaces to operate at 2-Gbit/s rates.

 

3.6.3        Procedure

After SmartBits completes fabric login (FLOGI), it offers traffic at a rate of 1 million frames per second. Then we physically remove the primary backbone ISL cabling.

 

We derive failover time from frame loss. At a rate of 1 million fps, each lost frame counts for 1 microsecond of failover time.

 

We repeat the procedure for devices supporting failover across aggregated links.

 

We use 128-byte frames for this test. At 2-Gbit/s rates, the 1 million fps rate is equivalent to approximately 71 percent utilization.

 

3.6.4        Metrics

Frame loss

Failover time (derived from frame loss)

 

4         Change history

Version 2.01

Date: 17 January 2002

Added publication date to document title

Version 2.00

Date: 31 October 2001

In sections 3.1.3 and 3.2.3, removed references to loss in throughput measurements

 

In sections 3.1.3 and 3.2.3, added description of 1- and 8-frame burst counts

 

In sections 3.1.3 and 3.2.3, revised description of delay-related measurements to note that test duration is now 1 millisecond (was 30 seconds)

 

In section 3.2.1, clarified objective of four-switch baseline test

 

In section 3.3.4, changed metrics to throughput and latency; deleted references to loss

 

In section 3.4, deleted one-to-many test; section 3.4 is now many-to-one test

 

In section 3.4.4, changed metrics to match those of baselines (section 3.1.4)

 

Added section 3.5, a many-to-few test

 

Version 1.11

Date: 10 August 2001

In sections 3.2.2 and 3.2.3, corrected edge interface references to indicate 10 connections to SmartBits for each switch

 

Version 1.10

Date 1 August 2001

In section 2.1, changed product requirements to indicate vendors must supply GBICs only for their devices, not for test equipment

 

In section 2.1, added requirement that vendors supply cabling for interswitch links (ISLs)

 

In section 3.1.2, added text distinguishing physical from logical topology

 

In section 3.2.2, changed physical topology to indicate 10 edge and 6 core interfaces per switch

 

In section 3.2.2, changed text to clarify that new topology does not create oversubscription

 

Version 1.01

Date: 25 July 2001

In section 3.2.2, corrected reference to test interfaces

 

In section 3.2.3, changed text to note fully meshed traffic distribution

 

Version 1.0

Date: 24 July 2001

Initial release



[1] All references to frame length in this document include headers and CRC. The payload in a 2,148-byte header is 2,112 bytes, the maximum allowed in Fibre Channel.

Network Test Footer. Privacy Policy. Contact us to learn more.