Network World Clear Choice Test: 802.11n for the Enterprise

Scheduled for publication in Network World in fall 2008

Test methodology

 

Version 2008071001. Copyright 1999-2008 by Network Test Inc. Vendors are encouraged to comment on this document and any other aspect of test methodology. Network Test and Network World reserve the right to change test parameters at any time.

 

This document is available in PDF format here.

 

1       Executive summary

This test seeks to answer one central question: How fast do 802.11n networks run in an enterprise context?

 

That means testing with multiple 802.11n access points and running separate tests for controller-based and standalone systems.  “Fast” doesn’t just mean measuring throughput; it also involves measurements of latency and jitter, key considerations for voice and video over wireless.

 

There will be two sets of performance tests: RFC benchmarking and “WiMix”

tests involving multiple types of enterprise traffic. We also consider non-performance criteria including features, usability, and price/performance.

 

This document is organized as follows. This section introduces the tests to be conducted. Section 2 describes the test bed. Section 3 describes the tests to be performed. Section 4 provides a change log.

 

2       The test bed

2.1     System under test (SUT)

The system under test should support an enterprise installation of 8 802.11n Wireless Access Points (APs) spread across the premises and 2 Gigabit Copper Ethernet links.

 

Each of the 8 APs will be placed in its own RF shielded enclosure and shielded from each other in order to reduce RF interference.  The RF shielded enclosure will be cabled directly to the WT-90s using multiple cables.  Each RF cable will pass a single spatial stream.  For a 2x2 AP, two RF cables will be used; for a 3x3 AP, three cables will be used.  The cables will be terminated with SMA connectors.

 

For dual APs that combine both radio frequencies into one SMA connection on the AP, a 2:1 splitter will be used inside the enclosure to allow two VeriWave WBW2000 WaveBlades (test instrument ports) to be connected to the single port of the AP.  Ports A of two WaveBlades will be connected to one splitter, Ports B of the same two WaveBlades will be connected to a separate splitter, and so on.

 

The APs should support 802.11a, 802.11b, 802.11g and 802.11n.  On the 5GHz band, tests will be run with 40MHz channel bandwidth.  On the 2.4GHZ band, tests will be run with 20MHz channel bandwidth.  For the co-existence tests, expect legacy clients to be mixed in with the 802.11n clients.  APs should use 802.3af power over Ethernet. Let us know ASAP if your device does not support PoE.

 

All clients will connect to the APs using WPA2 Personal (the preshared key will be “Veriwave”).  Clients will be using from one to four SSIDs depending upon the test.   The number of clients per radio, per AP will be set to 20.  The number of Ethernet clients will be equal to or less than the total number of wireless clients.

 

There are no DHCP or Radius servers in the test bed.  Both DHCP and Radius are only used in setting up the test bed, and are not part of the measurement.  Whether a client has a static or dynamic IP address, or whether a client has to authenticate to a Radius server, SHOULD NOT effect the throughput, latency, or service differentiation for that client.

 

We are glad to test products that do not include a wired Ethernet controller. We will optionally supply a gigabit Ethernet switch with 802.3af power over Ethernet support; vendors MUST support power injectors if access points require more power than 802.3af provides. For controller-based systems, controller cost is included in the total price of the system under test.

 

2.1.1    System under test configuration

The test will use static IPv4 addresses across 1 to 4 VLANs (depending upon the test).  VLAN100 is reserved for the SUT’s management.  Subnet mask is 255.255.252.0 for all VLANs.  The first valid host address of 10.0.x.1 is reserved for the gateways.

 

Name

SSID

Security

IP addresses

VLAN

SUT Management

 

 

10.0.0.0/24

100

SMB

NWSMB

WPA2-PSK

10.0.4.0/22

101

HTTP

NWHTTP

WPA2-PSK

10.0.8.0/22

102

Media Player

NWWMP

WPA2-PSK

10.0.12.0/22

103

VoIP

NWVOIP

WPA2-PSK

10.0.16.0/22

104

 

Every AP, Ethernet controller, and any Management console can be statically configured between 10.0.0.2 and 10.0.0.254 with a subnet mask of 255.255.255.0 and a gateway of 10.0.0.1.  Any traffic on this VLAN will be considered out-of-band and should be keep to a minimum in order not to affect the measurement on other VLANs.

 

Test client IP address will be assigned such that the wired side will occupy the lower half of the IP subnet and wireless clients will occupy the upper half.  There will be a total of 320 IP addresses used by the test traffic.   Throughput and latency tests will place all the IP addresses on one VLAN, where the WiMix tests will spread them over four VLANs.

 

Test traffic will always be configured to run between wired client and wireless clients in the same IP subnet.  Routing between the VLAN is not required by the test and no test traffic will ever need to be routed.  Routing can be turned on for management and diagnostic purposes.

2.2     Test instrument

VeriWave’s WT90 is a Wireless LAN and Ethernet performance test platform, capable of stress testing a complete WLAN network, including Access Points (APs) and WLAN Switches. The WT90 provides simultaneous generation of traffic from thousands of 802.11a/b/g/n and Ethernet clients, with line speed analysis of the behavior of the system being tested.  For more information, visit ttp://www.veriwave.com/products/wavetest.asp

3       Test Procedures

3.1     802.11n Throughput

3.1.1    Objective

To determine the SUT throughput (as defined in RFC 1242) for eight APs, each with 20 pure 802.11n clients (not greenfield). Test bed configuration

3.1.2    Test bed configuration

Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

CTS-to-self protection

Disabled

Radio

5GHZ, with 40MHz bandwidth

MCS index

7 for 1x1, 15 for 2x2, 23 for 3x3

802.11n guard interval

Short (400 ns)

A-MPDU aggregation

Enabled

Acceptable Loss

0.1%[1]

Search Maximum

150%

Search Starting Point

10%

Search Minimum

1%

Search Resolution

0.1%

 

3.1.3    Procedures

Using the Veriwave’s Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.

 

The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm.  The binary search algorithm uses two parameters in its search:  the maximum passed rate and the maximum failed rate.

 

Between the iterations, the WT-90 will measure the RSSI value of the AP’s beacons.  At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point.  If the RSSI values are not in the range of -40 dBm to -20 dBm, the number of retransmissions may be higher than expected and the reported throughput lower.

 

3.1.4    Metrics

Throughput in bits and frames per second.

 

Direction

88 bytes (bit/s)

512 bytes (bit/s)

1518 bytes (bit/s)

88 bytes (fps)

512 bytes (fps)

1518 bytes (fps)

Downlink

 

 

 

 

 

 

Uplink

 

 

 

 

 

 

Bi-directional

 

 

 

 

 

 

 

3.2     Coexistence Throughput

3.2.1    Objective

To determine the SUT throughput (as defined in RFC 1242) with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).

 

3.2.2    Test bed configuration

Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP’s RF band.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

This test will measure throughput of the entire system, concurrently offering traffic to both 2.4- and 5-GHz radios. There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

CTS-to-self protection

Disabled

2.4-GHz radio

20MHz bandwidth

5-GHz radio

40MHz bandwidth

802.11a

Phy Rate  = 54 Mbits

802.11g

Phy Rate  = 54 Mbits

802.11n

MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns)

A-MPDU aggregation

Enabled

Acceptable Loss

0.1%[2]

Search Maximum

150%

Search Starting Point

10%

Search Minimum

1%

Search Resolution

0.1%

 

3.2.3    Procedures

Using the Veriwave’s Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.

 

Tests will concurrently determine the throughput rate on the 2.4- and 5-GHz radios.

 

The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm. Between iterations of the binary search, the WT-90 will measure the RSSI value of the AP’s beacons.  At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point.  If the RSSI values are not in the range of 0 dBm to -20 dBm, the number of retransmission may be higher than expected and the reported throughput lower.

 

3.2.4    Metrics

Throughput in bits and frames per second.  802.11n and non-802.11n throughput are reported separately. The throughput test algorithm accounts for clients of different capacities being present in the same test configuration, and will distribute the intended load at any given step of the test in proportion to what each client should be able to forward.  This ensures that 802.11n clients will be configured so as to encourage aggregation, while non-802.11n clients will not be overloaded.  All clients get equal opportunity to access the medium, however.

 

 

Direction

 

88 bytes (bit/s)

512 bytes (bit/s)

1518 bytes (bit/s)

 

88 bytes (fps)

512 bytes (fps)

1518 bytes (fps)

Downlink (802.11n)

 

 

 

 

 

 

Uplink (802.11n)

 

 

 

 

 

 

Bi-directional (802.11n)

 

 

 

 

 

 

Downlink (non-802.11n)

 

 

 

 

 

 

Uplink (non-802.11n)

 

 

 

 

 

 

Bi-directional (non-802.11n)

 

 

 

 

 

 

 

3.3     802.11n Latency

3.3.1    Objective

To determine the SUT latency and jitter for eight APs, each with 20 pure 802.11n clients (not greenfield), at the throughput rate.

 

3.3.2    Test bed configuration

Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Intended Load

Measured Throughput value in 3.1.4

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Test Traffic

Stateless UDP packets

Traffic orientation

Upstream, Downstream, Bi-directional

Channel Model

Bypass

SSID

NWHTTP

CTS-to-self protection

Disabled

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

RF Radio

5GHZ, with 40MHz bandwidth

MCS index

7 for 1x1, 15 for 2x2, 23 for 3x3

802.11n guard interval

Short (400 ns)

A-MPDU aggregation

Enabled

 

3.3.3    Procedures

Using the Veriwave’s Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter.

 

3.3.4    Metrics

For each iteration record the Offered load, average latency, maximum latency, and jitter

 

Direction

OLOAD

Avg Latency

Max Latency

Jitter

Downlink 88 bytes

 

 

 

 

Downlink 512 bytes

 

 

 

 

Downlink 1518 bytes

 

 

 

 

Uplink 88 bytes

 

 

 

 

Uplink 512 bytes

 

 

 

 

Uplink 1518 bytes

 

 

 

 

Bi-directional 88 bytes

 

 

 

 

Bi-directional 512 bytes

 

 

 

 

Bi-directional 1518 bytes

 

 

 

 

 

3.4     Coexistence Latency

3.4.1    Objective

To determine the SUT latency and jitter at the throughput rate, with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).

 

3.4.2    Test bed configuration

Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients.  There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP.  Test traffic will run between wired client and wireless clients in the same IP subnet.

 

This is a dual-radio throughput test with traffic being presented on both 2.4GHz and 5-GHz radios at the same time.  There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus there will be 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.

 

For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.

 

Test duration

60 seconds

Intended Load

Measured Throughput value in 3.2.4

Test traffic frame sizes

88, 512, and 1,518 bytes (as offered on 802.3 side)

Traffic orientation

Upstream, Downstream, Bi-directional

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

CTS-to-self protection

Disabled

Ethernet VLAN ID

102 (tagged frames offered on Ethernet side)

Security

WPA2-PSK

Preshared Key

Veriwave

2.4-GHz radio

20MHz bandwidth

5-GHz radio

40MHz bandwidth

802.11a

Phy Rate  = 54Mbits

802.11g

Phy Rate  = 54Mbits

802.11n

MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns)

A-MPDU aggregation

Enabled

 

3.4.3    Procedures

Using the Veriwave’s Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter.  The average latency, maximum latency, and jitter will be reported separately between 802.11n and non-802.11n clients.

 

3.4.4    Metrics

Record the Offered load, average latency, maximum latency, and jitter is a separate table for each of the Traffic orientations. 

 

3.4.4.1 Downlink (Wired to Wireless)

 

Client

Radio

Frame Size

OLOAD

Avg Latency

Max Latency

Jitter

802.11g

2.4GHz

88 bytes

 

 

 

 

802.11n

2.4GHz

88 bytes

 

 

 

 

802.11a

5GHz

88 bytes

 

 

 

 

802.11n

5GHz

88 bytes

 

 

 

 

802.11g

2.4GHz

512 bytes

 

 

 

 

802.11n

2.4GHz

512 bytes

 

 

 

 

802.11a

5GHz

512 bytes

 

 

 

 

802.11n

5GHz

512 bytes

 

 

 

 

802.11g

2.4GHz

1518 bytes

 

 

 

 

802.11n

2.4GHz

1518 bytes

 

 

 

 

802.11a

5GHz

1518 bytes

 

 

 

 

802.11n

5GHz

1518 bytes

 

 

 

 

 

3.4.4.2 Uplink (Wired to Wireless)

 

Client

Radio

Frame Size

OLOAD

Avg Latency

Max Latency

Jitter

802.11g

2.4GHz

88 bytes

 

 

 

 

802.11n

2.4GHz

88 bytes

 

 

 

 

802.11a

5GHz

88 bytes

 

 

 

 

802.11n

5GHz

88 bytes

 

 

 

 

802.11g

2.4GHz

512 bytes

 

 

 

 

802.11n

2.4GHz

512 bytes

 

 

 

 

802.11a

5GHz

512 bytes

 

 

 

 

802.11n

5GHz

512 bytes

 

 

 

 

802.11g

2.4GHz

1518 bytes

 

 

 

 

802.11n

2.4GHz

1518 bytes

 

 

 

 

802.11a

5GHz

1518 bytes

 

 

 

 

802.11n

5GHz

1518 bytes

 

 

 

 

 

3.4.4.3 Bi-directional

 

Client

Radio

Frame Size

OLOAD

Avg Latency

Max Latency

Jitter

802.11g

2.4GHz

88 bytes

 

 

 

 

802.11n

2.4GHz

88 bytes

 

 

 

 

802.11a

5GHz

88 bytes

 

 

 

 

802.11n

5GHz

88 bytes

 

 

 

 

802.11g

2.4GHz

512 bytes

 

 

 

 

802.11n

2.4GHz

512 bytes

 

 

 

 

802.11a

5GHz

512 bytes

 

 

 

 

802.11n

5GHz

512 bytes

 

 

 

 

802.11g

2.4GHz

1518 bytes

 

 

 

 

802.11n

2.4GHz

1518 bytes

 

 

 

 

802.11a

5GHz

1518 bytes

 

 

 

 

802.11n

5GHz

1518 bytes

 

 

 

 

 

3.5     WiMix performance

3.5.1    Objectives

To determine the TCP goodput (RFC 2647) and UDP forwarding rates (RFC 2285) of 4 dual-radio APs when forwarding a mix of enterprise applications.

To determine the ability of the SUT to apply QoS enforcement to different traffic classes to meet predefined service levels.

To determine the ability of the SUT to re-mark diff-serv code points (DSCPs)

 

3.5.2    Test bed configuration

There will be a total of four different groups of clients, each running a different type of traffic.  The table below shows each of the client groups along with the desired classification and marking for frames leaving the SUT.  Frames offered to the SUT may or may not be properly marked to the desired classification.

 

Client

SSID

VLAN ID

Classification

DSCP

802.11e

WMM

Laptop group 1

NWSMB

101

Background

10

1

1

Laptop group 2

NWHTTP

102

Best Effort

0

0

0

Laptop group 3

NWWMP

103

Video

34

5

5

Handset group 4

NWVOIP

104

Voice

46

7

7

 

All clients will use WPA2 personal.

 

This is a dual-radio test with traffic presented concurrently to both 2.4- and 5-GHz radios.  The relative percentage of traffic by traffic class presented to the network will remain constant throughout the test. The VeriWave test instrument will establish one client per radio for SMB, HTTP, and Windows Media Player traffic. The VeriWave test instrument will vary the number of VoIP clients as necessary to establish the percentage of the total load as defined in the table below.

 

The 2.4-GHz radio will be set to the following properties:

 

Bandwidth

20MHz

Client type

802.11g and 802.11n

802.11g clients

Phy rate = 54Mbits

802.11n clients

MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

Standard/long (800ns)

CTS-to-self protection

Disabled

A-MPDU aggregation

Enabled

 

The 5-GHz radio will be set to the following properties:

 

Bandwidth

20MHz

Client type

802.11a and 802.11n

802.11a clients

Phy rate = 54Mbits

802.11n clients

MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3

802.11n guard interval

Short (400 ns)

CTS-to-self protection

Disabled

A-MPDU aggregation

Enabled

 

Test traffic will be offered in four different flavors based on the client groups.  Test traffic will run between wired and wireless clients in the same IP subnet.

 

On the wired side, two gigabit Ethernet VeriWave test interfaces offer all four traffic classes to the SUT. The traffic load will be evenly distributed across both cards.

 

Client

802

Traffic class

OLoad

Protocol

Properties

Laptop group 1

.11n

SMB

50%

TCP

Stateful, MSS= 1460, ReceiveWindowSize = 65535

Laptop group 2

.11n

HTTP

25%

TCP

Stateful, MSS= 1460, ReceiveWindowSize = 65535

Laptop group 3

.11n

Windows Media Player

20%

RTP

Downlink,

151 byte frames

Handset group 4

.11a/g

VoIP G711

5%

RTP

Bi-directional,

236 byte frames

3.5.3    Procedures

Using the Veriwave’s Standard WiMix suite:

 

1. Set the offered load to 50 Mbit/s using a single AP. Set the test to run for 30 seconds. (Note that this is just a setup phase to determine final load; the “official” test duration is 300 seconds.)

2. Run a single iteration with properly marked code points.

3. If the SLA for any traffic class fails, repeat step 2 with offered load set at 25 Mbit/s per AP. If this test succeeds, repeat the test using a step algorithm from 26 Mbit/s per AP to 49 Mbit/s per AP in 1-Mbit/s increments, until any traffic class no longer complies with its respective SLA (see table below for SLA values). If this test fails, keep repeating test, decreasing oload by 1 Mbit/s per AP, until all traffic classes comply with their respective SLAs  (see table below for SLA values).

If the SLAs for all traffic classes are met, repeat step 2 with offered load set at 75 Mbit/s per AP. If this test succeeds, repeat step 2 at 100 Mbit/s, and so on in 25 Mbit/s increments until the test fails. Call the failure ILOAD N. Then use a step algorithm to vary the ILOAD from N-25 Mbit/s per AP to N Mbit/s per AP in 1 Mbit/s increments, until any traffic class no longer complies with its respective SLA (see table below for SLA values). If this test fails, keep repeating test, decreasing oload by 1 Mbit/s per AP, until all traffic classes comply with their respective SLAs  (see table below for SLA values).

4. Repeat all previous steps with 8 APs operating concurrently at the ILOAD determined in the previous step. The test duration is 300 seconds.

5. As a sanity check against static bandwidth allocation for high-priority traffic, rerun the test with SMB and HTTP alone (groups 1 and 2). The iLoad will be the highest value from the previous step where Windows Media and VoIP were in compliance with SLA boundaries.

6. Set all the classification to best effort (value = 0) to simulate no marking, and repeat at the final offered load from step 3. Make sure packet capture is enabled, and verify that codepoints are set to the values given in the first table in section 3.5.2 (eg., DSCP 46 for VoIP).

 

3.5.4    Metrics

Report the metrics per class as defined below.  Look at the packet captures to verify the frames exiting the SUT are properly marked.

 

3.5.4.1  Properly Marked Frames

Client

Metric

SLA

Client count

Measured

Laptop group 1

Goodput

5% ILOAD

1

 

Laptop group 2

Goodput

5% ILOAD

1

 

Laptop group 3

Forwarding rate

98% ILOAD

1

 

 

Latency

< 500mS

N/A

 

 

Jitter

< 500mS

N/A

 

 

Packet

Loss

< 2%

N/A

 

Handset group 4

R-Value

> 78

 

 

 

3.5.4.2  Improperly Marked Frames

Client

Metric

SLA

Client count

Measured

Laptop group 1

Goodput

5% ILOAD

1

 

Laptop group 2

Goodput

5% ILOAD

1

 

Laptop group 3

Forwarding rate

98% ILOAD

1

 

 

Latency

< 500mS

N/A

 

 

Jitter

< 500mS

N/A

 

 

Packet

Loss

< 2%

N/A

 

Handset group 4

R-Value

> 78

 

 

 

3.5.4.3 Remarking of DSCP

Client

Expected

Measured

Pass/Fail

Laptop group 1

10

 

 

Laptop group 2

0

 

 

Laptop group 3

34

 

 

Handset group 4

46

 

 

 

3.6     Power over Ethernet consumption

3.6.1    Objectives

To measure the 802.3af power consumption of a single AP while idle and while doing a maximum forwarding rate test.

 

3.6.2    Test bed configuration

This is a dual-radio test with traffic presented to both 2.4GHz and 5-GHz radios at the same time.  There will be 20 802.11n clients on the 2.4-GHz radio, plus another 20 802.11n clients on the 5-GHz radio.

 

A special cat5e cable will be inserted between the AP and the PoE switch.  With this cable, pins 1-2 and 4-5 will be looped clockwise, while pins 3-6 and 7-8 will be looped counter-clockwise.  Loop lengths will maintain the same propagation delay as a normal cable so as not to affect gigabit Ethernet performance.  The cat5e twists will not be changed.

 

Test duration

60 seconds

Test traffic frame sizes

88 and 1518 bytes (as offered on 802.3 side)

Traffic orientation

Downstream

Test Traffic

Stateless UDP packets

Client Contention

Zero

Channel Model

Bypass

SSID

NWHTTP

CTS-to-self protection

Disabled

Security

WPA2-PSK

Preshared Key

Veriwave

2.4-GHz radio

40MHz bandwidth

5-GHz radio

40MHz bandwidth

MCS index

7 for 1x1, 15 for 2x2, 23 for 3x3

802.11n guard interval

Short (400 ns)

A-MPDU aggregation

Enabled

Search Resolution

0.1%

 

3.6.3    Procedures

Run the looped twisted pair wires through a clamp-on DC current probe.  Measure the current while the AP is idle (no test traffic). 

 

Using the Veriwave’s Standard WLAN Benchmarking Test to measure Maximum Forwarding Rate (per RFC 2285).  During the test, record the maximum current reported by the DC current probe.

 

Remember to divide the DC current probe reading by the number of loops through the clamp to get the actual current consumption of the AP.  Additional loops may be used to increase the sensitivity of the DC current probe.

 

3.6.4    Metrics

Power consumption when idle (watts)

Power consumption at maximum forwarding rate (watts)

Maximum forwarding rate (bit/s and frame/s) (Note: This is recorded for completeness, but optionally might not be included in final article depending on space requirements)

3.7     System features

3.7.1    Objective

To determine the feature set supported by the SUT

 

3.7.2    Test bed configuration

Not applicable

 

3.7.3    Procedure

We ask participating vendors to complete a features questionnaire listing various attributes supported by the SUT. The questionnaire includes space for vendors to describe features not covered by the various questions.

 

Network World will publish the results of the features questionnaire, usually in its online edition. The publication should include a caveat that responses are supplied by vendors, and not all features have been verified by Network World.

 

In addition, testers may consider include subjective ratings of system usability and manageability in the features category.

 

3.7.4    Metrics

Features supported

 

3.8     System price/performance

3.8.1    Objective

To determine the relative price/performance offered by each SUT

 

3.8.2    Test bed configuration

Not applicable

 

3.8.3    Procedure

Using a formula that combines US list price with the results from the benchmark and WiMix performance tests described above, testers will compare products based on relative price/performance.

 

3.8.4    Metrics

Price/performance

 

4       Change Log

10 July 2008

All tests

Charged 802.11n guard interval from long/standard (800ns) to short (400 ns) when radio uses 40 MHz of bandwidth

 

Section 2.1

Removed PoE injector requirement for standard PoE-compliant APs

 

Sections 3.2.2, 3.2.3

Changed coexistence test to determine throughput concurrently on 2.4- and 5-GHz radios (previously determined throughput on 5-GHz alone first, then on both radios)

 

Section 3.6.2

Changed 802.11n bandwidth from 20 MHz to 40 MHz

 

19 May 2008

Version 2008051901

Section 2.1

Deleted extraneous word “either” before WPA2

 

Corrected typo (“wither” becomes “whether”)

 

Specified the use of a splitter to connect two test ports to a single AP port for APs that operate dual radios through one interface

 

Sections 3.1.2, 3.2.2, 3.3.2, 3.4.2, 3.5.2, 3.6.2

Specified 800-ns guard interval in test setup parameters

Specified CTS-to-self is disabled in test setup parameters

 

Section 3.5.2

Specified A-MPDU on both 2.4- and 5-GHz bands

Specified that 802.3 Ethernet traffic is offered from two interfaces

 

Section 3.5.3

Added SLA parameters for HTTP and SMB traffic (5% of ILOAD each)

Added initial setup phase with 30-second tests on a single AP

 

Section 3.5.4

Deleted pass/fail columns from tables 3.5.4.1 and 3.5.4.2 (tests recorded are by definition iterations that pass)

 

Section 3.6.2

Added 88-byte frame testing

 

Section 3.6.4

Clarified measurements: both idle and at MFR

Added maximum forwarding rate to metrics

 

12 May 2008

Version 2008051201

Section 3.6.2

Added 2.4-GHz row to parameter table

 

7 May 2008

Version 2008050701

Section 2.1.1

Changed VoIP handset security from “open” to “WPA2-PSK”

 

Section 3.2.4

Clarified measurement for clients running at different rates

Added fps alongside bit/s as a metric

 

Section 3.2.3

Changed acceptable RSSI value to 0 to -20 dBm (was -20 to -40 dBm, but VeriWave 802.11n card now includes built-in 20 dB attenuator)

 

 

6 May 2008

Version 2008050601

Sections 3.1.2, 3.2.2

Changed acceptable loss from 0 to 0.1% to handle weak error-checking of PLCP header

 

5 May 2008

Version 2008050501

Section 3.1.4

Extended results table to report in bits and frames per second

 

Section 3.5.2

Clarified client counts in WiMix test (1 apiece per radio for SMB, HTTP, and Windows Media; variable for VoIP)

 

Section 3.5.3

In step 5, pointed to desired DSCP values for re-marking test

 

Modified search algorithm to range from 1 to 100 Mbit/s

 

Sections 3.5.4.1, 3.5.4.2

Added “client count” columns to results tables

 

21 April 2008

Version 2008042101

Section 2.1

Specified WPA2 Personal for all clients, including VoIP handsets

 

Deleted requirement for including switch cost as part of system under test (controller cost still included)

 

Sections 2.1, 3.6

Corrected IEEE 802.3af reference

 

Sections 3.2.2, 3.2.3

Modified procedure for measuring coexistence throughput (measure 5-GHz throughput alone first; then measure 2.4-GHz throughput while concurrently offering traffic on 5-GHz radios using fixed intended load at 5-GHz throughput rate)

 

Section 3.5.2

Added PHY rates for 802.11g and 802.11a clients

 

Specified WPA2 Personal for all clients, including VoIP handsets

 

Section 3.5.3

Modified offered load to use step algorithm (25 Mbit/s/AP midpoint, then step from 5-25 or 26-50 depending on outcome)

Add sanity check against TDM-like static bandwidth allocation for high-priority traffic

Clarify “pass” metrics (Windows Media and VoIP must fall within SLA boundaries)

 

 

7 April 2008

Version 2008040701

Initial public release



[1] This violates RFC 1242, which defines throughput as a zero-loss condition. As discussed in a previous Network World test, weak error checking in the 802.11 PLCP header requires nonzero loss tolerance in throughput measurement. This design flaw is present in all versions of 802.11 (including 802.11n non-greenfield, which is tested here).

[2] This violates RFC 1242, which defines throughput as a zero-loss condition. As discussed in a previous Network World test, weak error checking in the 802.11 PLCP header requires nonzero loss tolerance in throughput measurement. This design flaw is present in all versions of 802.11 (including 802.11n non-greenfield, which is tested here).