Network World Clear Choice Test: 802.11n for the Enterprise
Scheduled
for publication in Network World in fall 2008
Test
methodology
Version 2008071001. Copyright 1999-2008 by Network Test Inc.
Vendors are encouraged to comment on this document and any other aspect of test
methodology. Network Test and Network World reserve the right to change test
parameters at any time.
This document is available in PDF
format here.
This test seeks to answer one central question: How fast do 802.11n networks run in an enterprise context?
That means testing with multiple 802.11n access points and running separate tests for controller-based and standalone systems. ÒFastÓ doesnÕt just mean measuring throughput; it also involves measurements of latency and jitter, key considerations for voice and video over wireless.
There will be two sets of performance tests: RFC benchmarking and ÒWiMixÓ
tests involving multiple types of enterprise traffic. We also consider non-performance criteria including features, usability, and price/performance.
This document is organized as follows. This section
introduces the tests to be conducted. Section 2 describes the test bed. Section
3 describes the tests to be performed. Section 4 provides a change log.
The system under test should support an enterprise installation of 8 802.11n Wireless Access Points (APs) spread across the premises and 2 Gigabit Copper Ethernet links.
Each of the 8 APs will be placed in its own RF shielded enclosure and shielded from each other in order to reduce RF interference. The RF shielded enclosure will be cabled directly to the WT-90s using multiple cables. Each RF cable will pass a single spatial stream. For a 2x2 AP, two RF cables will be used; for a 3x3 AP, three cables will be used. The cables will be terminated with SMA connectors.
For dual APs that combine both radio frequencies into one SMA connection on the AP, a 2:1 splitter will be used inside the enclosure to allow two VeriWave WBW2000 WaveBlades (test instrument ports) to be connected to the single port of the AP. Ports A of two WaveBlades will be connected to one splitter, Ports B of the same two WaveBlades will be connected to a separate splitter, and so on.
The APs should support 802.11a, 802.11b, 802.11g and 802.11n. On the 5GHz band, tests will be run with 40MHz channel bandwidth. On the 2.4GHZ band, tests will be run with 20MHz channel bandwidth. For the co-existence tests, expect legacy clients to be mixed in with the 802.11n clients. APs should use 802.3af power over Ethernet. Let us know ASAP if your device does not support PoE.
All clients will connect to the APs using WPA2 Personal (the preshared key will be ÒVeriwaveÓ). Clients will be using from one to four SSIDs depending upon the test. The number of clients per radio, per AP will be set to 20. The number of Ethernet clients will be equal to or less than the total number of wireless clients.
There are no DHCP or Radius servers in the test bed. Both DHCP and Radius are only used in setting up the test bed, and are not part of the measurement. Whether a client has a static or dynamic IP address, or whether a client has to authenticate to a Radius server, SHOULD NOT effect the throughput, latency, or service differentiation for that client.
We are glad to test products that do not include a wired Ethernet controller. We will optionally supply a gigabit Ethernet switch with 802.3af power over Ethernet support; vendors MUST support power injectors if access points require more power than 802.3af provides. For controller-based systems, controller cost is included in the total price of the system under test.
The test will use static IPv4 addresses across 1 to 4 VLANs (depending upon the test). VLAN100 is reserved for the SUTÕs management. Subnet mask is 255.255.252.0 for all VLANs. The first valid host address of 10.0.x.1 is reserved for the gateways.
|
Name |
SSID |
Security |
IP addresses |
VLAN |
|
SUT Management |
|
|
10.0.0.0/24 |
100 |
|
SMB |
NWSMB |
WPA2-PSK |
10.0.4.0/22 |
101 |
|
HTTP |
NWHTTP |
WPA2-PSK |
10.0.8.0/22 |
102 |
|
Media Player |
NWWMP |
WPA2-PSK |
10.0.12.0/22 |
103 |
|
VoIP |
NWVOIP |
WPA2-PSK |
10.0.16.0/22 |
104 |
Every AP, Ethernet controller, and any Management console can be statically configured between 10.0.0.2 and 10.0.0.254 with a subnet mask of 255.255.255.0 and a gateway of 10.0.0.1. Any traffic on this VLAN will be considered out-of-band and should be keep to a minimum in order not to affect the measurement on other VLANs.
Test client IP address will be assigned such that the wired side will occupy the lower half of the IP subnet and wireless clients will occupy the upper half. There will be a total of 320 IP addresses used by the test traffic. Throughput and latency tests will place all the IP addresses on one VLAN, where the WiMix tests will spread them over four VLANs.
Test traffic will always be configured to run between wired client and wireless clients in the same IP subnet. Routing between the VLAN is not required by the test and no test traffic will ever need to be routed. Routing can be turned on for management and diagnostic purposes.
VeriWaveÕs WT90 is a Wireless LAN and Ethernet performance test platform, capable of stress testing a complete WLAN network, including Access Points (APs) and WLAN Switches. The WT90 provides simultaneous generation of traffic from thousands of 802.11a/b/g/n and Ethernet clients, with line speed analysis of the behavior of the system being tested. For more information, visit ttp://www.veriwave.com/products/wavetest.asp
To determine the SUT throughput (as defined in RFC 1242) for eight APs, each with 20 pure 802.11n clients (not greenfield). Test bed configuration
Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients. There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP. Test traffic will run between wired client and wireless clients in the same IP subnet.
For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.
|
Test duration |
60 seconds |
|
Test traffic frame sizes |
88, 512, and 1,518 bytes (as offered on 802.3 side) |
|
Traffic orientation |
Upstream, Downstream, Bi-directional |
|
Test Traffic |
Stateless UDP packets |
|
Client Contention |
Zero |
|
Channel Model |
Bypass |
|
SSID |
NWHTTP |
|
Ethernet VLAN ID |
102 (tagged frames offered on Ethernet side) |
|
Security |
WPA2-PSK |
|
Preshared Key |
Veriwave |
|
CTS-to-self protection |
Disabled |
|
Radio |
5GHZ, with 40MHz bandwidth |
|
MCS index |
7 for 1x1, 15 for 2x2, 23 for 3x3 |
|
802.11n guard interval |
Short (400 ns) |
|
A-MPDU aggregation |
Enabled |
|
Acceptable Loss |
0.1%[1] |
|
Search Maximum |
150% |
|
Search Starting Point |
10% |
|
Search Minimum |
1% |
|
Search Resolution |
0.1% |
Using the VeriwaveÕs Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.
The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm. The binary search algorithm uses two parameters in its search: the maximum passed rate and the maximum failed rate.
Between the iterations, the WT-90 will measure the RSSI value of the APÕs beacons. At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point. If the RSSI values are not in the range of -40 dBm to -20 dBm, the number of retransmissions may be higher than expected and the reported throughput lower.
Throughput in bits and frames per second.
|
Direction |
88 bytes (bit/s) |
512 bytes (bit/s) |
1518 bytes (bit/s) |
88 bytes (fps) |
512 bytes (fps) |
1518 bytes (fps) |
|
Downlink |
|
|
|
|
|
|
|
Uplink |
|
|
|
|
|
|
|
Bi-directional |
|
|
|
|
|
|
To determine the SUT throughput (as defined in RFC 1242) with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).
Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients. There will be 80 clients on each wired Ethernet port and 20 clients on each wireless APÕs RF band. Test traffic will run between wired client and wireless clients in the same IP subnet.
This test will measure throughput of the entire system, concurrently offering traffic to both 2.4- and 5-GHz radios. There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.
For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.
|
Test duration |
60 seconds |
|
Test traffic frame sizes |
88, 512, and 1,518 bytes (as offered on 802.3 side) |
|
Traffic orientation |
Upstream, Downstream, Bi-directional |
|
Test Traffic |
Stateless UDP packets |
|
Client Contention |
Zero |
|
Channel Model |
Bypass |
|
SSID |
NWHTTP |
|
Ethernet VLAN ID |
102 (tagged frames offered on Ethernet side) |
|
Security |
WPA2-PSK |
|
Preshared Key |
Veriwave |
|
CTS-to-self protection |
Disabled |
|
2.4-GHz radio |
20MHz bandwidth |
|
5-GHz radio |
40MHz bandwidth |
|
802.11a |
Phy Rate = 54 Mbits |
|
802.11g |
Phy Rate = 54 Mbits |
|
802.11n |
MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3 |
|
802.11n guard interval |
2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns) |
|
A-MPDU aggregation |
Enabled |
|
Acceptable Loss |
0.1%[2] |
|
Search Maximum |
150% |
|
Search Starting Point |
10% |
|
Search Minimum |
1% |
|
Search Resolution |
0.1% |
Using the VeriwaveÕs Standard WLAN Benchmarking Test to measure throughput (per RFC1242), Network Test will determine the maximum offered load each system under test can sustain with zero packet loss.
Tests will concurrently determine the throughput rate on the 2.4- and 5-GHz radios.
The Standard WLAN Benchmarking Test determines the throughput using a binary search algorithm. Between iterations of the binary search, the WT-90 will measure the RSSI value of the APÕs beacons. At the end of the test, the report will contain the minimum, maximum, and average RSSI per access point. If the RSSI values are not in the range of 0 dBm to -20 dBm, the number of retransmission may be higher than expected and the reported throughput lower.
Throughput in bits and frames per second. 802.11n and non-802.11n throughput are reported separately. The throughput test algorithm accounts for clients of different capacities being present in the same test configuration, and will distribute the intended load at any given step of the test in proportion to what each client should be able to forward. This ensures that 802.11n clients will be configured so as to encourage aggregation, while non-802.11n clients will not be overloaded. All clients get equal opportunity to access the medium, however.
|
Direction |
88 bytes (bit/s) |
512 bytes (bit/s) |
1518 bytes (bit/s) |
88 bytes (fps) |
512 bytes (fps) |
1518 bytes (fps) |
|
Downlink (802.11n) |
|
|
|
|
|
|
|
Uplink (802.11n) |
|
|
|
|
|
|
|
Bi-directional (802.11n) |
|
|
|
|
|
|
|
Downlink (non-802.11n) |
|
|
|
|
|
|
|
Uplink (non-802.11n) |
|
|
|
|
|
|
|
Bi-directional (non-802.11n) |
|
|
|
|
|
|
To determine the SUT latency and jitter for eight APs, each with 20 pure 802.11n clients (not greenfield), at the throughput rate.
Topology is non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients. There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP. Test traffic will run between wired client and wireless clients in the same IP subnet.
For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.
|
Test duration |
60 seconds |
|
Intended Load |
Measured Throughput value in 3.1.4 |
|
Test traffic frame sizes |
88, 512, and 1,518 bytes (as offered on 802.3 side) |
|
Test Traffic |
Stateless UDP packets |
|
Traffic orientation |
Upstream, Downstream, Bi-directional |
|
Channel Model |
Bypass |
|
SSID |
NWHTTP |
|
CTS-to-self protection |
Disabled |
|
Ethernet VLAN ID |
102 (tagged frames offered on Ethernet side) |
|
Security |
WPA2-PSK |
|
Preshared Key |
Veriwave |
|
RF Radio |
5GHZ, with 40MHz bandwidth |
|
MCS index |
7 for 1x1, 15 for 2x2, 23 for 3x3 |
|
802.11n guard interval |
Short (400 ns) |
|
A-MPDU aggregation |
Enabled |
Using the VeriwaveÕs Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter.
For each iteration record the Offered load, average latency, maximum latency, and jitter
|
Direction |
OLOAD |
Avg Latency |
Max Latency |
Jitter |
|
Downlink 88 bytes |
|
|
|
|
|
Downlink 512 bytes |
|
|
|
|
|
Downlink 1518 bytes |
|
|
|
|
|
Uplink 88 bytes |
|
|
|
|
|
Uplink 512 bytes |
|
|
|
|
|
Uplink 1518 bytes |
|
|
|
|
|
Bi-directional 88 bytes |
|
|
|
|
|
Bi-directional 512 bytes |
|
|
|
|
|
Bi-directional 1518 bytes |
|
|
|
|
To determine the SUT latency and jitter at the throughput rate, with 4 dual-radio APs, each with 4 802.11a clients, 4 802.11g clients and 32 802.11n clients (split between the radios).
Topology is Non-meshed (as defined in RFC 2285), with all traffic offered between wired Ethernet and wireless LAN clients. There will be 80 clients on each wired Ethernet port and 20 clients on each wireless AP. Test traffic will run between wired client and wireless clients in the same IP subnet.
This is a dual-radio throughput test with traffic being presented on both 2.4GHz and 5-GHz radios at the same time. There will be 4 802.11g clients and 16 802.11n clients on the 2.4-GHz radio, plus there will be 4 802.11a clients and 16 802.11n clients on the 5-GHz radio.
For downstream-only tests, keep-alive frames from the VeriWave clients will be enabled.
|
Test duration |
60 seconds |
|
Intended Load |
Measured Throughput value in 3.2.4 |
|
Test traffic frame sizes |
88, 512, and 1,518 bytes (as offered on 802.3 side) |
|
Traffic orientation |
Upstream, Downstream, Bi-directional |
|
Test Traffic |
Stateless UDP packets |
|
Client Contention |
Zero |
|
Channel Model |
Bypass |
|
SSID |
NWHTTP |
|
CTS-to-self protection |
Disabled |
|
Ethernet VLAN ID |
102 (tagged frames offered on Ethernet side) |
|
Security |
WPA2-PSK |
|
Preshared Key |
Veriwave |
|
2.4-GHz radio |
20MHz bandwidth |
|
5-GHz radio |
40MHz bandwidth |
|
802.11a |
Phy Rate = 54Mbits |
|
802.11g |
Phy Rate = 54Mbits |
|
802.11n |
MCS=15 for 2x2, MCS=23 for 3x3 |
|
802.11n guard interval |
2.4-GHz radio: Standard/long (800ns); 5-GHz radio: Short (400 ns) |
|
A-MPDU aggregation |
Enabled |
Using the VeriwaveÕs Standard WLAN Benchmarking Test, Network Test will measure the minimum latency, maximum latency, average latency and jitter. The average latency, maximum latency, and jitter will be reported separately between 802.11n and non-802.11n clients.
Record the Offered load, average latency, maximum latency, and jitter is a separate table for each of the Traffic orientations.
|
Client |
Radio |
Frame Size |
OLOAD |
Avg Latency |
Max Latency |
Jitter |
|
802.11g |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11a |
5GHz |
88 bytes |
|
|
|
|
|
802.11n |
5GHz |
88 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11a |
5GHz |
512 bytes |
|
|
|
|
|
802.11n |
5GHz |
512 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11a |
5GHz |
1518 bytes |
|
|
|
|
|
802.11n |
5GHz |
1518 bytes |
|
|
|
|
|
Client |
Radio |
Frame Size |
OLOAD |
Avg Latency |
Max Latency |
Jitter |
|
802.11g |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11a |
5GHz |
88 bytes |
|
|
|
|
|
802.11n |
5GHz |
88 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11a |
5GHz |
512 bytes |
|
|
|
|
|
802.11n |
5GHz |
512 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11a |
5GHz |
1518 bytes |
|
|
|
|
|
802.11n |
5GHz |
1518 bytes |
|
|
|
|
|
Client |
Radio |
Frame Size |
OLOAD |
Avg Latency |
Max Latency |
Jitter |
|
802.11g |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
88 bytes |
|
|
|
|
|
802.11a |
5GHz |
88 bytes |
|
|
|
|
|
802.11n |
5GHz |
88 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
512 bytes |
|
|
|
|
|
802.11a |
5GHz |
512 bytes |
|
|
|
|
|
802.11n |
5GHz |
512 bytes |
|
|
|
|
|
802.11g |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11n |
2.4GHz |
1518 bytes |
|
|
|
|
|
802.11a |
5GHz |
1518 bytes |
|
|
|
|
|
802.11n |
5GHz |
1518 bytes |
|
|
|
|
To determine the TCP goodput (RFC 2647) and UDP forwarding rates (RFC 2285) of 4 dual-radio APs when forwarding a mix of enterprise applications.
To determine the ability of the SUT to apply QoS enforcement to different traffic classes to meet predefined service levels.
To determine the ability of the SUT to re-mark diff-serv code points (DSCPs)
There will be a total of four different groups of clients, each running a different type of traffic. The table below shows each of the client groups along with the desired classification and marking for frames leaving the SUT. Frames offered to the SUT may or may not be properly marked to the desired classification.
|
Client |
SSID |
VLAN ID |
Classification |
DSCP |
802.11e |
WMM |
|
Laptop group 1 |
NWSMB |
101 |
Background |
10 |
1 |
1 |
|
Laptop group 2 |
NWHTTP |
102 |
Best Effort |
0 |
0 |
0 |
|
Laptop group 3 |
NWWMP |
103 |
Video |
34 |
5 |
5 |
|
Handset group 4 |
NWVOIP |
104 |
Voice |
46 |
7 |
7 |
All clients will use WPA2 personal.
This is a dual-radio test with traffic presented concurrently to both 2.4- and 5-GHz radios. The relative percentage of traffic by traffic class presented to the network will remain constant throughout the test. The VeriWave test instrument will establish one client per radio for SMB, HTTP, and Windows Media Player traffic. The VeriWave test instrument will vary the number of VoIP clients as necessary to establish the percentage of the total load as defined in the table below.
The 2.4-GHz radio will be set to the following properties:
|
Bandwidth |
20MHz |
|
Client type |
802.11g and 802.11n |
|
802.11g clients |
Phy rate = 54Mbits |
|
802.11n clients |
MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3 |
|
802.11n guard interval |
Standard/long (800ns) |
|
CTS-to-self protection |
Disabled |
|
A-MPDU aggregation |
Enabled |
The 5-GHz radio will be set to the following properties:
|
Bandwidth |
20MHz |
|
Client type |
802.11a and 802.11n |
|
802.11a clients |
Phy rate = 54Mbits |
|
802.11n clients |
MCS=7 for 1x1, MCS=15 for 2x2, MCS=23 for 3x3 |
|
802.11n guard interval |
Short (400 ns) |
|
CTS-to-self protection |
Disabled |
|
A-MPDU aggregation |
Enabled |
Test traffic will be offered in four different flavors based on the client groups. Test traffic will run between wired and wireless clients in the same IP subnet.
On the wired side, two gigabit Ethernet VeriWave test interfaces offer all four traffic classes to the SUT. The traffic load will be evenly distributed across both cards.
|
Client |
802 |
Traffic class |
OLoad |
Protocol |
Properties |
|
Laptop group 1 |
.11n |
SMB |
50% |
TCP |
Stateful,
MSS= 1460, ReceiveWindowSize = 65535 |
|
Laptop group 2 |
.11n |
HTTP |
25% |
TCP |
Stateful,
MSS= 1460, ReceiveWindowSize = 65535 |
|
Laptop group 3 |
.11n |
Windows Media Player |
20% |
RTP |
Downlink,
151
byte frames |
|
Handset group 4 |
.11a/g |
VoIP G711 |
5% |
RTP |
Bi-directional, 236
byte frames |
Using the VeriwaveÕs Standard WiMix suite:
1. Set the offered load to 50 Mbit/s using a single AP. Set the test to run for 30 seconds. (Note that this is just a setup phase to determine final load; the ÒofficialÓ test duration is 300 seconds.)
2. Run a single iteration with properly marked code points.
3. If the SLA for any traffic class fails, repeat step 2 with offered load set at 25 Mbit/s per AP. If this test succeeds, repeat the test using a step algorithm from 26 Mbit/s per AP to 49 Mbit/s per AP in 1-Mbit/s increments, until any traffic class no longer complies with its respective SLA (see table below for SLA values). If this test fails, keep repeating test, decreasing oload by 1 Mbit/s per AP, until all traffic classes comply with their respective SLAs (see table below for SLA values).
If the SLAs for all traffic classes are met, repeat step 2 with offered load set at 75 Mbit/s per AP. If this test succeeds, repeat step 2 at 100 Mbit/s, and so on in 25 Mbit/s increments until the test fails. Call the failure ILOAD N. Then use a step algorithm to vary the ILOAD from N-25 Mbit/s per AP to N Mbit/s per AP in 1 Mbit/s increments, until any traffic class no longer complies with its respective SLA (see table below for SLA values). If this test fails, keep repeating test, decreasing oload by 1 Mbit/s per AP, until all traffic classes comply with their respective SLAs (see table below for SLA values).
4. Repeat all previous steps with 8 APs operating concurrently at the ILOAD determined in the previous step. The test duration is 300 seconds.
5. As a sanity check against static bandwidth allocation for high-priority traffic, rerun the test with SMB and HTTP alone (groups 1 and 2). The iLoad will be the highest value from the previous step where Windows Media and VoIP were in compliance with SLA boundaries.
6. Set all the classification to best effort (value = 0) to simulate no marking, and repeat at the final offered load from step 3. Make sure packet capture is enabled, and verify that codepoints are set to the values given in the first table in section 3.5.2 (eg., DSCP 46 for VoIP).
Report the metrics per class as defined below. Look at the packet captures to verify the frames exiting the SUT are properly marked.
|
Client |
Metric |
SLA |
Client count |
Measured |
|
Laptop group 1 |
Goodput |
5% ILOAD |
1 |
|
|
Laptop group 2 |
Goodput |
5% ILOAD |
1 |
|
|
Laptop group 3 |
Forwarding rate |
98% ILOAD |
1 |
|
|
|
Latency |
< 500mS |
N/A |
|
|
|
Jitter |
< 500mS |
N/A |
|
|
|
Packet Loss |
< 2% |
N/A |
|
|
Handset group 4 |
R-Value |
> 78 |
|
|
|
Client |
Metric |
SLA |
Client count |
Measured |
|
Laptop group 1 |
Goodput |
5% ILOAD |
1 |
|
|
Laptop group 2 |
Goodput |
5% ILOAD |
1 |
|
|
Laptop group 3 |
Forwarding rate |
98% ILOAD |
1 |
|
|
|
Latency |
< 500mS |
N/A |
|
|
|
Jitter |
< 500mS |
N/A |
|
|
|
Packet Loss |
< 2% |
N/A |
|
|
Handset group 4 |
R-Value |
> 78 |
|
|
|
Client |
Expected |
Measured |
Pass/Fail
|
|
Laptop group 1 |
10 |
|
|
|
Laptop group 2 |
0 |
|
|
|
Laptop group 3 |
34 |
|
|
|
Handset group 4 |
46 |
|
|
To measure the 802.3af power consumption of a single AP while idle and while doing a maximum forwarding rate test.
This is a dual-radio test with traffic presented to both 2.4GHz and 5-GHz radios at the same time. There will be 20 802.11n clients on the 2.4-GHz radio, plus another 20 802.11n clients on the 5-GHz radio.
A special cat5e cable will be inserted between the AP and the PoE switch. With this cable, pins 1-2 and 4-5 will be looped clockwise, while pins 3-6 and 7-8 will be looped counter-clockwise. Loop lengths will maintain the same propagation delay as a normal cable so as not to affect gigabit Ethernet performance. The cat5e twists will not be changed.
|
Test duration |
60 seconds |
|
Test traffic frame sizes |
88 and 1518 bytes (as offered on 802.3 side) |
|
Traffic orientation |
Downstream |
|
Test Traffic |
Stateless UDP packets |
|
Client Contention |
Zero |
|
Channel Model |
Bypass |
|
SSID |
NWHTTP |
|
CTS-to-self protection |
Disabled |
|
Security |
WPA2-PSK |
|
Preshared Key |
Veriwave |
|
2.4-GHz radio |
40MHz bandwidth |
|
5-GHz radio |
40MHz bandwidth |
|
MCS index |
7 for 1x1, 15 for 2x2, 23 for 3x3 |
|
802.11n guard interval |
Short (400 ns) |
|
A-MPDU aggregation |
Enabled |
|
Search Resolution |
0.1% |
Run the looped twisted pair wires through a clamp-on DC current probe. Measure the current while the AP is idle (no test traffic).
Using the VeriwaveÕs Standard WLAN Benchmarking Test to measure Maximum Forwarding Rate (per RFC 2285). During the test, record the maximum current reported by the DC current probe.
Remember to divide the DC current probe reading by the number of loops through the clamp to get the actual current consumption of the AP. Additional loops may be used to increase the sensitivity of the DC current probe.
Power consumption when idle (watts)
Power consumption at maximum forwarding rate (watts)
Maximum forwarding rate (bit/s and frame/s) (Note: This is recorded for completeness, but optionally might not be included in final article depending on space requirements)
To determine the feature set supported by the SUT
Not applicable
We ask participating vendors to complete a features questionnaire listing various attributes supported by the SUT. The questionnaire includes space for vendors to describe features not covered by the various questions.
Network World will publish the results of the features questionnaire, usually in its online edition. The publication should include a caveat that responses are supplied by vendors, and not all features have been verified by Network World.
In addition, testers may consider include subjective ratings of system usability and manageability in the features category.
Features supported
To determine the relative price/performance offered by each SUT
Not applicable
Using a formula that combines US list price with the results from the benchmark and WiMix performance tests described above, testers will compare products based on relative price/performance.
Price/performance
10 July 2008
All tests
Charged 802.11n guard interval from long/standard (800ns) to short (400 ns) when radio uses 40 MHz of bandwidth
Section 2.1
Removed PoE injector requirement for standard PoE-compliant APs
Sections 3.2.2, 3.2.3
Changed coexistence test to determine throughput concurrently on 2.4- and 5-GHz radios (previously determined throughput on 5-GHz alone first, then on both radios)
Section 3.6.2
Changed 802.11n bandwidth from 20 MHz to 40 MHz
19 May 2008
Version 2008051901
Section 2.1
Deleted extraneous word ÒeitherÓ before WPA2
Corrected typo (ÒwitherÓ becomes ÒwhetherÓ)
Specified the use of a splitter to connect two test ports to a single AP port for APs that operate dual radios through one interface
Sections 3.1.2, 3.2.2, 3.3.2, 3.4.2, 3.5.2, 3.6.2
Specified 800-ns guard interval in test setup parameters
Specified CTS-to-self is disabled in test setup parameters
Section 3.5.2
Specified A-MPDU on both 2.4- and 5-GHz bands
Specified that 802.3 Ethernet traffic is offered from two interfaces
Section 3.5.3
Added SLA parameters for HTTP and SMB traffic (5% of ILOAD each)
Added initial setup phase with 30-second tests on a single AP
Section 3.5.4
Deleted pass/fail columns from tables 3.5.4.1 and 3.5.4.2 (tests recorded are by definition iterations that pass)
Section 3.6.2
Added 88-byte frame testing
Section 3.6.4
Clarified measurements: both idle and at MFR
Added maximum forwarding rate to metrics
12 May 2008
Version 2008051201
Section 3.6.2
Added 2.4-GHz row to parameter table
7 May 2008
Version 2008050701
Section 2.1.1
Changed VoIP handset security from ÒopenÓ to ÒWPA2-PSKÓ
Section 3.2.4
Clarified measurement for clients running at different rates
Added fps alongside bit/s as a metric
Section 3.2.3
Changed acceptable RSSI value to 0 to -20 dBm (was -20 to -40 dBm, but VeriWave 802.11n card now includes built-in 20 dB attenuator)
6 May 2008
Version 2008050601
Sections 3.1.2, 3.2.2
Changed acceptable loss from 0 to 0.1% to handle weak error-checking of PLCP header
5 May 2008
Version 2008050501
Section 3.1.4
Extended results table to report in bits and frames per second
Section 3.5.2
Clarified client counts in WiMix test (1 apiece per radio for SMB, HTTP, and Windows Media; variable for VoIP)
Section 3.5.3
In step 5, pointed to desired DSCP values for re-marking test
Modified search algorithm to range from 1 to 100 Mbit/s
Sections 3.5.4.1, 3.5.4.2
Added Òclient countÓ columns to results tables
21 April 2008
Version 2008042101
Section 2.1
Specified WPA2 Personal for all clients, including VoIP handsets
Deleted requirement for including switch cost as part of system under test (controller cost still included)
Sections 2.1, 3.6
Corrected IEEE 802.3af reference
Sections 3.2.2, 3.2.3
Modified procedure for measuring coexistence throughput (measure 5-GHz throughput alone first; then measure 2.4-GHz throughput while concurrently offering traffic on 5-GHz radios using fixed intended load at 5-GHz throughput rate)
Section 3.5.2
Added PHY rates for 802.11g and 802.11a clients
Specified WPA2 Personal for all clients, including VoIP handsets
Section 3.5.3
Modified offered load to use step algorithm (25 Mbit/s/AP midpoint, then step from 5-25 or 26-50 depending on outcome)
Add sanity check against TDM-like static bandwidth allocation for high-priority traffic
Clarify ÒpassÓ metrics (Windows Media and VoIP must fall within SLA boundaries)
7 April 2008
Version 2008040701
Initial public release
[1] This violates RFC 1242, which defines throughput as a zero-loss condition. As discussed in a previous Network World test, weak error checking in the 802.11 PLCP header requires nonzero loss tolerance in throughput measurement. This design flaw is present in all versions of 802.11 (including 802.11n non-greenfield, which is tested here).
[2] This violates RFC 1242, which defines throughput as a zero-loss condition. As discussed in a previous Network World test, weak error checking in the 802.11 PLCP header requires nonzero loss tolerance in throughput measurement. This design flaw is present in all versions of 802.11 (including 802.11n non-greenfield, which is tested here).