Data Comm Lab Test: Voice-over-IP Gateways
Publication Date: September 1999
Test Methodology
v. 1.06 Copyright © 1999 CMP Media Inc. Readers are encouraged to comment on this document and any other aspect of test methodology. However, Data Communications and NSTL reserve the right to change test parameters at any time.
This document describes the procedures to be used in evaluating voice-over-IP gateways. NSTL will evaluate gateways in each of four major evaluation categories:
Audio quality will be evaluated both subjectively and objectively. A jury of approximately 20 persons will evaluate the audio quality of sound samples taken from each system under test.
In addition, NSTL will use several measurements to objectively evaluate the impact of network performance on audio quality and vice-versa. This test begins with a "clean" network with no other traffic present, and measures the bandwidth consumed by voice calls and the latency imposed by the network on voice traffic. Then the tests are repeated with the introduction of progressively higher levels of latency, jitter, and packet loss.
The security tests evaluate what measures the systems under test take to secure voice traffic. NSTL will verify that systems supporting encryption using H.245 or proprietary means actually do so. NSTL also will ask vendors to describe gateway capabilities in the areas of authentication, authorization, and accounting.
Finally, Data Comm and NSTL will use responses from a features questionnaire (distributed separately) to compare management capabilities, standards support, and connectivity options among the various gateways.
The Test Bed
The following diagram illustrates the test bed to be used in the Data Comm/NSTL evaluation of VoIP gateways.
The test bed simulates a two-office network in which a headquarters and branch office exchange voice and data traffic over an IP WAN backbone.
For voice connections, telephone handsets at both headquarters and branch office are attached to a Mitel SX200 analog PBX. The PBX interface to the gateway is FXO/FXS using loop-start signaling. NSTL will assume gateways use DS-0 (64-kbit/s) circuits unless otherwise specified.
On the WAN side, the routers are configured with maximum data rates of T1 (1.544 Mbit/s). Gateways may be attached to the routers using T1 or Ethernet connections at the vendor's discretion.
Using a special telephone interface from Hello Direct Inc., a PC with a sound card will be attached to one handset on each side of the test bed. The PC will play three voice recordings: a man speaking, a woman speaking, and a man and woman simultaneously speaking. All three recordings are in English. NSTL recorded each file at a rate of 44,100 samples per second, the standard for CD-quality audio.
((Webmaster--insert Visio picture of test bed here))
Performance Testing
NSTL will conduct the performance tests on three closely-knit performance criteria of the VoIP devices:
NSTL conducts baseline tests of voice traffic on a quiet network, and then repeats the tests using progressively "dirtier" network conditions.
The first test is a jury test, which measures the quality of voice transmission over the IP network using several .WAV files. The selected jurors (approximately 20 NSTL test staff) will grade quality of the voice subjectively on a scale of 1-5, where 1 is poor and 5 is excellent.
The second test measures latency using a stereo .WAV file (sinusoidal wave). To measure latency, NSTL uses Goldwave, a shareware utility that graphically displays soundwaves of audio recordings (the tool is available from www.goldwave.com). A PC with a stereo sound card plays a .WAV file on one side of the test bed using the sound card's left channel, and receives the signal on the other side of the test bed using the sound card's right channel. As noted, NSTL plays sound files recorded at 44,100 samples per second. Goldwave notes the exact sample number where given tones began on each channel. NSTL then compares starting points on left and right channels, subtracts the difference in sample numbers between the two, and divides the remainder by 44,100 to derive the amount of delay added by the IP network.
The third test measures the bandwidth consumption required for the packetization of the voice by a particular VoIP gateway. This test measures bandwidth required for call setup and actual audio transmission. NSTL uses an Internet Advisor protocol analyzer with VoIP module from Hewlett Packard Co. (Palo Alto, Calif.). The Internet Advisor also decodes H.323 and/or SIP call setup routines.
The "Dirty" Network
To simulate congestion in the Internet, NSTL introduces progressively higher amounts of latency, jitter, packet loss, and packet resequencing. To do so, NSTL uses The Cloud, a software-based traffic modulator developed by Shunra Software Inc. (www.shunra.com).
As noted, NSTL first plays the voice recordings on a quiet network to obtain baseline traffic measurements. Then NSTL reruns the tests, introducing progressively higher levels of network "noise." The low- and medium-noise simulations are based on actual traffic measurements taken from sites on the Internet.
In all, there are four scenarios:
The Cloud introduces no latency or packet loss. The gateway uplink/downlink parameters set to 1.544 Mbit/s. NSTL runs this simulation to provide a baseline measurement.
This simulation, based on measurements of www.cigna.com (a Web server apparently in Illinois, as observed from NSTL in Philadelphia) represents a reasonably clean WAN line. In this scenario, NSTL configures The Cloud to introduce an average of 49 milliseconds of latency (minimum 31 ms, maximum 117 ms, standard distribution), and 5 percent random packet loss.
This simulation, based on measurements of www.br.ibm.com (IBM's Web server in Brazil, as observed from NSTL in Philadelphia) represents a moderately congested WAN link. NSTL configures The Cloud to introduce an average of 142 ms of latency (minimum 95 ms, maximum 350 ms, standard distribution), and 12 percent random packet loss.
This simulation represents a highly congested link. NSTL configures The Cloud with to introduce 700ms of latency (in a uniform distribution, with actual latencies ranging from 0 to 700 ms) and 17 percent random packet loss. The Cloud will also create 20 percent out-of-order packets and 20 percent duplicate packets--including those to be included in the 17 percent of dropped packets. This scenario uses a predefined file supplied by Shunra Software and is not based on actual traffic measurements by NSTL.
Security
Encryption
NSTL will verify that devices supporting encryption of voice traffic actually do so. Encryption will be verified using HP's Internet Advisor protocol analyzer, which examines data streams for the H.245 commands transmitted to set up encrypted sessions.
In addition, participating vendors will complete a features questionnaire that covers gateway capabilities in the areas of authentication, authorization, and accounting.
