Network World Clear Choice Test: 802.11n for the Enterprise

Scheduled for publication in Network World in fall 2008

Test methodology

 

Version 2008071001. Copyright 1999-2008 by Network Test Inc. Vendors are encouraged to comment on this document and any other aspect of test methodology. Network Test and Network World reserve the right to change test parameters at any time.

 

This document is available in PDF format here.

 

1       Executive summary

This test seeks to answer one central question: How fast do 802.11n networks run in an enterprise context?

 

That means testing with multiple 802.11n access points and running separate tests for controller-based and standalone systems.  ÒFastÓ doesnÕt just mean measuring throughput; it also involves measurements of latency and jitter, key considerations for voice and video over wireless.

 

There will be two sets of performance tests: RFC benchmarking and ÒWiMixÓ

tests involving multiple types of enterprise traffic. We also consider non-performance criteria including features, usability, and price/performance.

 

This document is organized as follows. This section introduces the tests to be conducted. Section 2 describes the test bed. Section 3 describes the tests to be performed. Section 4 provides a change log.

 

2       The test bed

2.1     System under test (SUT)

The system under test should support an enterprise installation of 8 802.11n Wireless Access Points (APs) spread across the premises and 2 Gigabit Copper Ethernet links.

 

Each of the 8 APs will be placed in its own RF shielded enclosure and shielded from each other in order to reduce RF interference.  The RF shielded enclosure will be cabled directly to the WT-90s using multiple cables.  Each RF cable will pass a single spatial stream.  For a 2x2 AP, two RF cables will be used; for a 3x3 AP, three cables will be used.  The cables will be terminated with SMA connectors.

 

For dual APs that combine both radio frequencies into one SMA connection on the AP, a 2:1 splitter will be used inside the enclosure to allow two VeriWave WBW2000 WaveBlades (test instrument ports) to be connected to the single port of the AP.  Ports A of two WaveBlades will be connected to one splitter, Ports B of the same two WaveBlades will be connected to a separate splitter, and so on.

 

The APs should support 802.11a, 802.11b, 802.11g and 802.11n.  On the 5GHz band, tests will be run with 40MHz channel bandwidth.  On the 2.4GHZ band, tests will be run with 20MHz channel bandwidth.  For the co-existence tests, expect legacy clients to be mixed in with the 802.11n clients.  APs should use 802.3af power over Ethernet. Let us know ASAP if your device does not support PoE.

 

All clients will connect to the APs using WPA2 Personal (the preshared key will be ÒVeriwaveÓ).  Clients will be using from one to four SSIDs depending upon the test.   The number of clients per radio, per AP will be set to 20.  The number of Ethernet clients will be equal to or less than the total number of wireless clients.

 

There are no DHCP or Radius servers in the test bed.  Both DHCP and Radius are only used in setting up the test bed, and are not part of the measurement.  Whether a client has a static or dynamic IP address, or whether a client has to authenticate to a Radius server, SHOULD NOT effect the throughput, latency, or service differentiation for that client.

 

We are glad to test products that do not include a wired Ethernet controller. We will optionally supply a gigabit Ethernet switch with 802.3af power over Ethernet support; vendors MUST support power injectors if access points require more power than 802.3af provides. For controller-based systems, controller cost is included in the total price of the system under test.

 

2.1.1    System under test configuration

The test will use static IPv4 addresses across 1 to 4 VLANs (depending upon the test).  VLAN100 is reserved for the SUTÕs management.  Subnet mask is 255.255.252.0 for all VLANs.  The first valid host address of 10.0.x.1 is reserved for the gateways.

 

Name

SSID

Security

IP addresses

VLAN

SUT Management

 

 

10.0.0.0/24

100

SMB

NWSMB

WPA2-PSK

10.0.4.0/22

101

HTTP

NWHTTP

WPA2-PSK

10.0.8.0/22

102

Media Player

NWWMP

WPA2-PSK

10.0.12.0/22

103

VoIP

NWVOIP

WPA2-PSK

10.0.16.0/22

104

 

Every AP, Ethernet controller, and any Management console can be statically configured between 10.0.0.2 and 10.0.0.254 with a subnet mask of 255.255.255.0 and a gateway of 10.0.0.1.  Any traffic on this VLAN will be considered out-of-band and should be keep to a minimum in order not to affect the measurement on other VLANs.

 

Test client IP address will be assigned such that the wired side will occupy the lower half of the IP subnet and wireless clients will occupy the upper half.  There will be a total of 320 IP addresses used by the test traffic.   Throughput and latency tests will place all the IP addresses on one VLAN, where the WiMix tests will spread them over four VLANs.

 

Test traffic will always be configured to run between wired client and wireless clients in the same IP subnet.  Routing between the VLAN is not required by the test and no test traffic will ever need to be routed.  Routing can be turned on for management and diagnostic purposes.

2.2     Test instrument

VeriWaveÕs WT90 is a Wireless LAN and Ethernet performance test platform, capable of stress testing a complete WLAN network, including Access Points (APs) and WLAN Switches. The WT90 provides simultaneous generation of traffic from thousands of 802.11a/b/g/n and Ethernet clients, with line speed analysis of the behavior of the system being tested.  For more information, visit ttp://www.veriwave.com/products/wavetest.asp

3       Test Procedures

3.1     802.11n Throughput

3.1.1    Objective

To determine the SUT throughput (as defined in RFC 1242) for eight APs, each with 20 pure 802.11n clients (not greenfield). Test bed configuration

3.1.2    Test bed configuration

Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

CTS-to-self protection

Disabled

Radio

5GHZ, with 40MHz bandwidth

MCS index

7 for 1x1, 15 for 2x2, 23 for 3x3

802.11n guard interval

Short (400 ns)

A-MPDU aggregation

Enabled

Acceptable Loss

0.1%[1]

Search Maximum

150%

Search Starting Point

10%

Search Minimum

1%

Search Resolution

0.1%

 

3.1.3    Procedures

Using the VeriwaveÕs Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.

 

The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm.  The binary search algorithm uses two parameters in its search:  the maximum passed rate and the maximum failed rate.

 

Between the iterations, the WT-90 will measure the RSSI value of the APÕs beacons.  At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point.  If the RSSI values are not in the range of -40 dBm to -20 dBm, the number of retransmissions may be higher than expected and the reported throughput lower.

 

3.1.4    Metrics

Throughput in bits and frames per second.

 

Direction

88 bytes (bit/s)

512 bytes (bit/s)

1518 bytes (bit/s)

88 bytes (fps)

512 bytes (fps)

1518 bytes (fps)

Downlink

 

 

 

 

 

 

Uplink

 

 

 

 

 

 

Bi-directional

 

 

 

 

 

 

 

3.2     Coexistence Throughput

3.2.1    Objective

To determine the SUT throughput (as defined in RFC 1242) with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).

 

3.2.2    Test bed configuration

Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless APÕs RF band.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

This test will measure throughput of the entire system, concurrently offering traffic to both 2.4- and 5-GHz radios. There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

CTS-to-self protection

Disabled

2.4-GHz radio

20MHz bandwidth

5-GHz radio

40MHz bandwidth

802.11a

Phy Rate  = 54 Mbits

802.11g

Phy Rate  = 54 Mbits

802.11n

MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns)

A-MPDU aggregation

Enabled

Acceptable Loss

0.1%[2]

Search Maximum

150%

Search Starting Point

10%

Search Minimum

1%

Search Resolution

0.1%

 

3.2.3    Procedures

Using the VeriwaveÕs Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.

 

Tests will concurrently determine the throughput rate on the 2.4- and 5-GHz radios.

 

The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm. Between iterations of the binary search, the WT-90 will measure the RSSI value of the APÕs beacons.  At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point.  If the RSSI values are not in the range of 0 dBm to -20 dBm, the number of retransmission may be higher than expected and the reported throughput lower.

 

3.2.4    Metrics

Throughput in bits and frames per second.  802.11n and non-802.11n throughput are reported separately. The throughput test algorithm accounts for clients of different capacities being present in the same test configuration, and will distribute the intended load at any given step of the test in proportion to what each client should be able to forward.  This ensures that 802.11n clients will be configured so as to encourage aggregation, while non-802.11n clients will not be overloaded.  All clients get equal opportunity to access the medium, however.

 

 

Direction

 

88 bytes (bit/s)

512 bytes (bit/s)

1518 bytes (bit/s)

 

88 bytes (fps)

512 bytes (fps)

1518 bytes (fps)

Downlink (802.11n)

 

 

 

 

 

 

Uplink (802.11n)

 

 

 

 

 

 

Bi-directional (802.11n)

 

 

 

 

 

 

Downlink (non-802.11n)

 

 

 

 

 

 

Uplink (non-802.11n)

 

 

 

 

 

 

Bi-directional (non-802.11n)

 

 

 

 

 

 

 

3.3     802.11n Latency

3.3.1    Objective

To determine the SUT latency and jitter for eight APs, each with 20 pure 802.11n clients (not greenfield), at the throughput rate.

 

3.3.2    Test bed configuration

Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Intended Load

Measured Throughput value in 3.1.4

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Test Traffic

Stateless UDP packets

Traffic orientation

Upstream, Downstream, Bi-directional

Channel Model

Bypass

SSID

NWHTTP

CTS-to-self protection

Disabled

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

RF Radio

5GHZ, with 40MHz bandwidth

MCS index

7 for 1x1, 15 for 2x2, 23 for 3x3

802.11n guard interval

Short (400 ns)

A-MPDU aggregation

Enabled

 

3.3.3    Procedures

Using the VeriwaveÕs Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter.

 

3.3.4    Metrics

For each iteration record the Offered load, average latency, maximum latency, and jitter

 

Direction

OLOAD

Avg Latency

Max Latency

Jitter

Downlink 88 bytes

 

 

 

 

Downlink 512 bytes

 

 

 

 

Downlink 1518 bytes

 

 

 

 

Uplink 88 bytes

 

 

 

 

Uplink 512 bytes

 

 

 

 

Uplink 1518 bytes

 

 

 

 

Bi-directional 88 bytes

 

 

 

 

Bi-directional 512 bytes

 

 

 

 

Bi-directional 1518 bytes

 

 

 

 

 

3.4     Coexistence Latency

3.4.1    Objective

To determine the SUT latency and jitter at the throughput rate, with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).

 

3.4.2    Test bed configuration

Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

This is a dual-radio throughput test with traffic being presented on both 2.4GHz and 5-GHz radios at the same time.  There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus there will be 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Intended Load

Measured Throughput value in 3.2.4

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

CTS-to-self protection

Disabled

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

2.4-GHz radio

20MHz bandwidth

5-GHz radio

40MHz bandwidth

802.11a

Phy Rate  = 54Mbits

802.11g

Phy Rate  = 54Mbits

802.11n

MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns)

A-MPDU aggregation

Enabled

 

3.4.3    Procedures

Using the VeriwaveÕs Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter.  The average latency, maximum latency, and jitter will be reported separately between 802.11n and non-802.11n clients.

 

3.4.4    Metrics

Record the Offered load, average latency, maximum latency, and jitter is a separate table for each of the Traffic orientations. 

 

3.4.4.1 Downlink (Wired to Wireless)

 

Client

Radio

Frame Size

OLOAD

Avg Latency

Max Latency

Jitter

802.11g

2.4GHz

88 bytes

 

 

 

 

802.11n

2.4GHz

88 bytes