Troubleshooting Quality of Service for VoIP
The primary goal of Quality of Service (QoS) is to make network service better and more predictable by providing dedicated bandwidth, controlled jitter and latency, and improved loss characteristics. QoS achieves these goals by providing tools for managing network congestion, shaping network traffic, using expensive wide-area links more efficiently, and setting traffic policies across the network.
For information about configuring QoS for voice, refer to the Quality of Service for Voice document. For more information about QoS, refer to the Cisco IOS Quality of Service Solutions Configuration Guide.
This chapter contains the following topics:
Understanding VoIP QoS Issues
When VoIP calls are properly established, the next step is to verify that the voice quality is good. You should consider the following guidelines as you attempt to achieve good voice quality:
•
Understand how much bandwidth a VoIP call consumes with each codec, including Layer 2 and IP/UDP/RTP headers. For more information about VoIP bandwidth consumption, refer to Voice Over IP - Per Call Bandwidth Consumption, document ID 7934.
Understand how much bandwidth a VoIP call consumes with each codec, including Layer 2 and IP/UDP/RTP headers. For more information about VoIP bandwidth consumption, refer to Voice Over IP - Per Call Bandwidth Consumption, document ID 7934.
•Understand the characteristics of the IP network over which the calls travel. For example, the bandwidth of a Frame Relay network at CIR is much different than that above-CIR (or burst), where packets could be dropped or queued in the Frame-Relay cloud. Ensure that delay and jitter are controlled and eliminated as much as possible. One-way transmit delay should not exceed 150 ms (per G.114 recommendation).
Understand the characteristics of the IP network over which the calls travel. For example, the bandwidth of a Frame Relay network at CIR is much different than that above-CIR (or burst), where packets could be dropped or queued in the Frame-Relay cloud. Ensure that delay and jitter are controlled and eliminated as much as possible. One-way transmit delay should not exceed 150 ms (per G.114 recommendation).
•Use a queuing technique that allows VoIP traffic to be identified and prioritized.
Use a queuing technique that allows VoIP traffic to be identified and prioritized.
•When transmitting VoIP over low-speed links, consider using Layer 2 packet fragmentation techniques, such as Multilink Point-to-Point Protocol (MLPPP) with Link Fragmentation and Interleaving (LFI) on point-to-point links, or FRF.12 on Frame Relay links. Fragmentation of larger data packets allows less jitter and delay in transmitting VoIP traffic because the VoIP packets can be interleaved onto the link.
When transmitting VoIP over low-speed links, consider using Layer 2 packet fragmentation techniques, such as Multilink Point-to-Point Protocol (MLPPP) with Link Fragmentation and Interleaving (LFI) on point-to-point links, or FRF.12 on Frame Relay links. Fragmentation of larger data packets allows less jitter and delay in transmitting VoIP traffic because the VoIP packets can be interleaved onto the link.
•Try the call with a different codec and with the voice activity detector (VAD) enabled and disabled to possibly narrow down the issue to the digital signal processor (DSP), as opposed to the IP network.
Try the call with a different codec and with the voice activity detector (VAD) enabled and disabled to possibly narrow down the issue to the digital signal processor (DSP), as opposed to the IP network.
With VoIP, when you are troubleshooting QoS issues, look especially for dropped packets and network bottlenecks that can cause delay and jitter.
Specifically, look for the following:
•Interface drops
Interface drops
•Buffer drops
Buffer drops
•Policy-map drops
Policy-map drops
•Interface congestion
Interface congestion
•Link congestion
Link congestion
Each interface in the path of the VoIP call should be examined and drops and congestion should be eliminated. Also, round-trip delay should be reduced as much as possible. Pings between the VoIP end points give an indication of the round trip delay of a link. The round trip delay should not exceed 300 ms whenever possible. If the delay does have to exceed this value, efforts should also be taken to ensure this delay is constant, so that you do not introduce jitter or variable delay.
When low latency queueing (LLQ) is set up, you can check policy-map drops by looking for packets that match the priority class by entering the show policy interface serial command. This shows how much traffic is matching the priority class and how much bandwidth is used.
Ensure that the Cisco IOS queuing mechanism is placing the VoIP packets within the proper queues. Cisco IOS commands, such as show queue andshow priority can help you to verify queueing.
Measuring QoS
Cisco offers several options for monitoring QoS in networks using VoIP solutions. Cisco offers tools that provide information about the voice quality you are experiencing by measuring delay, jitter, and packet loss. Cisco solutions do not measure voice quality using Perceptual Speech Quality Measurement (PSQM) or some of the new proposed algorithms for voice quality measurement. Tools from outside vendors are available for this purpose.
When implementing service policies using the QoS command line interface (CLI), start with the Cisco Class-Based QoS Configuration and Statistics Management Information Base (MIB). This MIB provides read access to QoS configuration and statistics information for Cisco platforms that support the Modular QoS CLI. Statistics available through this MIB include summary counts and rates by traffic class before and after any configured QoS policies are enforced. In addition, detailed feature-specific statistics are available for select PolicyMap features. See Cisco MIBs for the object IDs.
In addition, Cisco offers the following software tools for monitoring QoS:
•Network monitoring using Cisco Service Assurance Agent (Cisco SSA)—The response time and availability monitoring capabilities of this tool include support for VoIP, QoS, and the World Wide Web. The Cisco SSA is an application-aware synthetic operation agent that monitors network performance by measuring key metrics such as response time, availability, jitter (interpacket delay variance), connect time, throughput, and packet loss. These metrics can be used for troubleshooting, for analysis before problems occur, and for designing future network topologies. This tool is designed more for trending, rather than real-time monitoring. Refer to Using Cisco Service Assurance Agent and Internetwork Performance Monitor to Manage Quality of Service in Voice over IP Networks for more information.
Network monitoring using Cisco Service Assurance Agent (Cisco SSA)—The response time and availability monitoring capabilities of this tool include support for VoIP, QoS, and the World Wide Web. The Cisco SSA is an application-aware synthetic operation agent that monitors network performance by measuring key metrics such as response time, availability, jitter (interpacket delay variance), connect time, throughput, and packet loss. These metrics can be used for troubleshooting, for analysis before problems occur, and for designing future network topologies. This tool is designed more for trending, rather than real-time monitoring. Refer to Using Cisco Service Assurance Agent and Internetwork Performance Monitor to Manage Quality of Service in Voice over IP Networks for more information.
•Cisco Gateway Management Agent (CGMA)—The only real-time management Cisco IOS software agent and protocol for VoIP. The CGMA is a new gateway Cisco IOS agent that provides real-time call-state information for all VoIP calls. CGMA supports a push protocol, in which certain call-state changes result in a message being sent out of CGMA by the gateways. The interface from the CGMA is the Real Time Management Protocol (RTMP). RTMP is a lightweight XML-based protocol that uses TCP as the transport protocol. This solution allows Service Providers to monitor their calls (session initiation protocol (SIP) and H.323 networks), viewing call detail records (CDRs) and trunk utilization in real time. The validated gateways for the CGMA include the Cisco 2600 series, the Cisco 3600 series, and the Cisco 5000 series. The Cisco IOS that has been validated on all gateways is the 12.2(2)XB mainline release.
Cisco Gateway Management Agent (CGMA)—The only real-time management Cisco IOS software agent and protocol for VoIP. The CGMA is a new gateway Cisco IOS agent that provides real-time call-state information for all VoIP calls. CGMA supports a push protocol, in which certain call-state changes result in a message being sent out of CGMA by the gateways. The interface from the CGMA is the Real Time Management Protocol (RTMP). RTMP is a lightweight XML-based protocol that uses TCP as the transport protocol. This solution allows Service Providers to monitor their calls (session initiation protocol (SIP) and H.323 networks), viewing call detail records (CDRs) and trunk utilization in real time. The validated gateways for the CGMA include the Cisco 2600 series, the Cisco 3600 series, and the Cisco 5000 series. The Cisco IOS that has been validated on all gateways is the 12.2(2)XB mainline release.
•CiscoWorks QoS Policy Manager (QPM)—QPM provides a scalable platform for defining, applying, and monitoring QoS policy on a system-wide basis for Cisco devices, including routers and switches.
CiscoWorks QoS Policy Manager (QPM)—QPM provides a scalable platform for defining, applying, and monitoring QoS policy on a system-wide basis for Cisco devices, including routers and switches.
QPM enables you to baseline profile network traffic, create QoS policies at an abstract level, control the deployment of policies, and then monitor QoS to verify intended results. As a centralized tool QPM is used to monitor and provision QoS for groups of interfaces and devices.
QPM provides a web-based intuitive user interface to define QoS policies, and translates those policies into the device's command line interface (CLI) commands.
QPM runs on the CiscoWorks Common Services server, which can be installed as a standalone server, or as an add-on to CD One 5th Edition. CiscoWorks Common Services provides the infrastructure required by QPM to run from the CiscoWorks desktop environment, and also provides management of user roles and privileges, allowing you to control who gets access to specific tasks in QPM.
Common QoS Problems
Common voice quality issues include the following:
Table 44 lists some QoS problems you might encounter after configuring voice ports and suggests some remedies. To configure QoS features on voice ports, see the "Fine-Tuning Analog and Digital Voice Ports" section in the Voice Port Configuration document.
Symptom
|
Suggested Action
|
|---|---|
Telephony device buzzes or does not ring.
|
Use the show voice port command to confirm that ring frequency is configured correctly. It must match the connected telephony equipment and might be country-dependent.
|
Distorted speech.
|
Use the show voice port command to confirm that the cptone keyword setting (also called region tone) is set to us for the United States.
Note
Having a wrong cptone setting can cause faulty voice reproduction during analog-to-digital or digital-to-analog conversions. |
Music on Hold (MOH) is not heard.
|
Reduce the music-threshold level to make VAD less sensitive with the music-threshold voice-port configuration command. Valid entries are from -70 to -30.
|
Background noise is not heard.
|
Enable the comfort-noise command.
|
Long pauses occur in conversation, as if you were speaking on a walkie-talkie.
|
Overall delay is probably excessive; the standard for adequate voice quality is 150 ms one-way transit delay. Measure delay by using ping tests at various times of the day with different network traffic loads. If delay must be reduced, examine propagation delay of signals between the sending and receiving endpoints, voice encoding delay, and the voice packetization time for various VoIP codecs.
|
Jerky or choppy speech.
|
Variable delay, or jitter, is being introduced by congestion in the packet network. There are two possible remedies:
•
Reduce the amount of congestion in your packet network. Pings between VoIP endpoints provide information about the round-trip delay of a link, which should never exceed 300 ms. Network queuing and dropped packets should also be examined.
•
Increase the size of the jitter buffer with the playout-delay command. (See the "Jitter Adjustment" section for more information.) |
Clipped or fuzzy speech.
|
Reduce input gain. (See the "Voice Level Adjustment" section for more information.)
Change the VAD level. Sometimes VAD cuts the sound too early and the speaker's voice is clipped. You can also change the time period that VAD waits for silence.
|
Clipped speech.
|
Reduce the input level at the listener's router. (See the "Voice Level Adjustment" section for more information)
|
Volume too low or missed DTMF.
|
Increase speaker's output level or listener's input level. (See the "Voice Level Adjustment" section for more information.)
|
Echo interval is greater than 25 ms (sounds like a separate voice).
|
Configure the echo-cancel enable command and increase the value for the echo-cancel coveragekeyword. (See the "Echo Adjustment" section for more information.)
|
Too much echo.
|
Reduce the output level at the speaker's voice port. (See the "Voice Level Adjustment" section for more information.)
|
Delay in Voice Networks
Delay is the time it takes for voice packets to travel between two endpoints. Excessive delay can cause quality problems with real-time traffic such as voice. However, because of the speed of network links and the processing power of intermediate devices, some delay is expected.
When listening to speech, the human ear normally accepts up to about 150 ms of delay without noticing it. The ITU G.114 standard recommends no more than 150 ms of one-way delay for a normal voice conversation. Once the delay exceeds 150 ms, a conversation is more like a "walkie-talkie" conversation in which one person must wait for the other to stop speaking before beginning to talk.
Several different types of delay combine to make up the total end-to-end delay associated with voice calls:
•Coder delay—Amount of time used by the digital signal processor (DSP) to compress samples.
Coder delay—Amount of time used by the digital signal processor (DSP) to compress samples.
•Handling delay—Amount of time it takes to process data (adding headers, taking samples, forming packets, and so forth).
Handling delay—Amount of time it takes to process data (adding headers, taking samples, forming packets, and so forth).
•Serialization delay—Fixed delay related to the clock rate on the interface. The amount of delay needed for different frame sizes is shown in Table 45.
Serialization delay—Fixed delay related to the clock rate on the interface. The amount of delay needed for different frame sizes is shown in Table 45.
•Propagation delay—Amount of time it takes the data to physically travel over the media.
Propagation delay—Amount of time it takes the data to physically travel over the media.
•Queuing delay—Amount of time lost due to congestion. Every network device between two endpoints involved in a call is a potential source of queuing or buffering delays. Use a sniffer trace at each hop to see where the delay or jitter is being introduced.
Queuing delay—Amount of time lost due to congestion. Every network device between two endpoints involved in a call is a potential source of queuing or buffering delays. Use a sniffer trace at each hop to see where the delay or jitter is being introduced.
Output drops can occur because of incorrectly configured priority queuing. For information about troubleshooting output drops with priority queueing, refer to Troubleshooting Output Drops with Priority Queueing , document ID 10105.
•Variable delay or jitter—Amount of time that causes the conversation to break and become unintelligible. Jitter is described in the "Jitter Adjustment" section.
Variable delay or jitter—Amount of time that causes the conversation to break and become unintelligible. Jitter is described in the "Jitter Adjustment" section.
You can measure delay by using ping tests at various times of the day with different network traffic loads. Sniffer traces can also be used to find where delay is being introduced in the network. For more information about measuring QoS, see the "Measuring QoS" section.
The recommended ranges for delay are shown in Table 46. These ranges are for connections in which echo is controlled. Echo cancellers are required when one-way delay exceeds 25milliseconds. For more information about echo cancellation, see the "Echo Adjustment" section. These ranges are for total end-to-end delay, not just the delay on the WAN side of a connection. For more information about developing a delay budget, refer to the "Building the Delay Budget" section of the Understanding Delay in Packet Voice Networks white paper.
Packet Loss
Packet loss is a very common cause of voice quality problems on an IP network. In a properly designed network, packet loss should be near zero. Voice codecs can tolerate some degree of packet loss without a dramatic effect on voice quality. Voice packets should be tagged for zero packet loss. On converged voice/data networks, the quantity of the voice calls and link bandwidth should be limited. The bandwidth should be guarenteed for the voice calls by giving priority to voice traffic. Packet loss can have an impact voice quality in several ways:
•Packet loss can grow as call volume increases
Packet loss can grow as call volume increases
•Fax and modem transmissons are greatly affected by packet loss
Fax and modem transmissons are greatly affected by packet loss
A common cause of packet loss is duplex mismatching. This occurs when one side of the connection is set up for full duplex and the other is set up for half duplex. Look at the link between the two endpoints experiencing the packet loss and check the speed and duplex settings for each side.
Although duplex mismatch is common, there are many other opportunities for packet loss in the network. To find the cause of these packet drops, check each interface between two endpoints by using the show interface command. If dropped packets are showing up on any interface, the link might be oversubscribed or there might be other traffic on this link that you are not expecting. In this case, use a network analyzer (or "sniffer") to check what traffic might be congesting the link. For more information about network analyzers, see the "Network Analyzers" section on page 178.
Jitter Adjustment
Delay can cause unnatural starting and stopping of conversations, but variable-length delays (also known as jitter) can cause a conversation to break and become unintelligible. Jitter is not usually a problem with PSTN calls because the bandwidth of calls is fixed and each call has a dedicated circuit for the duration of the call. However, in VoIP networks, data traffic might be bursty, and jitter from the packet network can become an issue. Packets from the same conversation can arrive at different interpacket intervals, especially during times of network congestion, which can disrupt the steady, even delivery needed for voice calls. Cisco voice gateways have built-in jitter buffering to compensate for a certain amount of jitter; the playout-delay command can be used to adjust the jitter buffer.
Normally, the defaults in effect are sufficient for most networks. However, a small playout delay from the jitter buffer can cause lost packets and choppy audio, and a large playout delay can cause unacceptably high overall end-to-end delay.
Note For Cisco IOS Release 12.1(5)T and later, in most cases playout delay should be configured in dial-peer configuration mode on the VoIP dial peer that is on the receiving end of the voice traffic that is to be buffered. This dial peer senses network conditions and relays them to the DSPs, which adjust the jitter buffer as necessary. When multiple applications are configured on the gateway, playout delay should be configured in dial-peer configuration mode. When there are numerous dial peers to configure, it might be simpler to configure playout delay on a voice port. If there are conflicting playout delay configurations on a voice port and also on a dial peer, the dial-peer configuration takes precedence.
For Cisco IOS Release 12.1(5)T and later, in most cases playout delay should be configured in dial-peer configuration mode on the VoIP dial peer that is on the receiving end of the voice traffic that is to be buffered. This dial peer senses network conditions and relays them to the DSPs, which adjust the jitter buffer as necessary. When multiple applications are configured on the gateway, playout delay should be configured in dial-peer configuration mode. When there are numerous dial peers to configure, it might be simpler to configure playout delay on a voice port. If there are conflicting playout delay configurations on a voice port and also on a dial peer, the dial-peer configuration takes precedence.
Jitter is more likely to occur on low-speed links because even a single packet in the queue on a low-speed link can dramatically affect the amount of time a voice packet needs to wait in the queue before being transmitted. Jitter can be an even bigger problem if you do not have priority queuing, such as low latency queuing (LLQ), enabled or configured correctly on your WAN connections. For more information about LLQ, see the Cisco IOS Quality of Service Solutions Configuration Guide. For information about troubleshooting VoIP over Point-to-Point Protocol (PPP) links with QoS for low latency queueing, refer to VoIP over PPP Links with Quality of Service (LLQ / IP RTP Priority, LFI, cRTP), document ID 7111.
Low-speed links also require special considerations when data traffic is also present to ensure that a large data packet does not cause excessive jitter. Generally on WAN links that are 768 kbps or slower, you should use some form of fragmentation and interleaving to ensure that large data packets do not starve smaller voice packets. Even with LLQ enabled, the voice packet must wait if it arrives when a data packet is in the process of being transmitted.
To configure the playout delay jitter buffer, perform the following steps.
SUMMARY STEPS
1. enable
enable
2. configure terminal
configure terminal
3. voice-port slot/port:ds0-group-number
voice-port slot/port:ds0-group-number
4. playout-delay mode {adaptive | fixed}
playout-delay mode {adaptive | fixed}
5. playout-delay {nominal value | maximum value | minimum {default | low | high}}
playout-delay {nominal value | maximum value | minimum {default | low | high}}
6. exit
exit
For more information about jitter, refer to Understanding Jitter in Packet Voice Networks (Cisco IOS Platforms), document ID 18902.
Echo Adjustment
Echo is a common problem, but can sometimes be difficult to troubleshoot. Echo exists in almost any telephone network but problems are more apparent in packet networks. Echo is perceived as a problem when all of the following conditions are true:
•Signal leakage between analog transmit (Tx) and receive (Rx) paths. This leakage is in one of these forms of echo:
Signal leakage between analog transmit (Tx) and receive (Rx) paths. This leakage is in one of these forms of echo:
–Acoustic echo—Acoustic echo is caused by poor acoustic isolation between the earpiece and the microphone in handsets and hands-free devices.
Acoustic echo—Acoustic echo is caused by poor acoustic isolation between the earpiece and the microphone in handsets and hands-free devices.
–Hybrid echo—Hybrid echo is caused by an impedance mismatch in the hybrid circuit, such as a two-wire to four-wire interface. This mismatch causes the Tx signal to appear on the Rx signal.
Hybrid echo—Hybrid echo is caused by an impedance mismatch in the hybrid circuit, such as a two-wire to four-wire interface. This mismatch causes the Tx signal to appear on the Rx signal.
•Perceptible delay between when the originating signal is transmitted and when the echo returns. In most telephones, sidetone helps mask some of the echo. Echos must be delayed by at least 20 milliseconds to be perceived.
Perceptible delay between when the originating signal is transmitted and when the echo returns. In most telephones, sidetone helps mask some of the echo. Echos must be delayed by at least 20 milliseconds to be perceived.
•Sufficient echo amplitude. If the amplitude of the echo is low, it can go unnoticed.
Sufficient echo amplitude. If the amplitude of the echo is low, it can go unnoticed.
The packet segment of the voice connection introduces a significant delay, typically 30 ms in each direction. The introduction of delay causes echoes generated on analog tail circuits that were normally indistinguishable from the side tone to be perceived by the user. The delay introduced by packet voice is unavoidable, therefore, the voice gateways must prevent the echo.
For a description about echo cancellation features and how echo cancellation is implemented on different Cisco platforms and Cisco IOS software releases, refer to the "Information about Echo Cancellation" section in Voice Port Configuration.
To configure echo cancellation, refer to the "How to Configure the Extended G.168 Echo Canceller" section in Voice Port Configuration.
When troubleshooting echo problems, see the following:
Determine Where Echo Is Occurring
When troubleshooting echo, first determine who is being affected by the echo problems.
PSTN Phone Users Hears Echo
In this case, a PSTN phone user hears echo, which is caused by acoustic coupling between the earpiece and the microphone in the IP phone handset. The solution is to use a load ID on the IP phone, which includes echo suppression on the handset and headset. Available load IDs only include echo cancellation on the speaker phone.
IP Phone User Hears Echo
In this case, an IP phone user hears echo caused by hybrids in a PSTN network. The solution is to configure and verify echo cancellation operation on a Cisco IOS gateway. The echo canceller in the voice gateway cancels the echo heard by the IP phone user.
Troubleshooting Echo in Cisco IOS Gateways
It is important to make sure that the echo canceller has enough information to distinguish between echo and voice conversation. The following parameters control the distinction:
•Input level—Signal input gain is performed before the echo canceller has detected the echo.
Input level—Signal input gain is performed before the echo canceller has detected the echo.
•Output level—Signal output attenuation is performed after the echo canceller has seen the original output signal.
Output level—Signal output attenuation is performed after the echo canceller has seen the original output signal.
•Echo canceller coverage—The amount of time the echo canceller retains a signal that has been output. This parameter must be set to a value greater than the time the echo needs to return to the gateway.
Echo canceller coverage—The amount of time the echo canceller retains a signal that has been output. This parameter must be set to a value greater than the time the echo needs to return to the gateway.
To eliminate echo, perform the following steps.
SUMMARY STEPS
1. Verify echo cancellation.
Verify echo cancellation.
2. Configure echo canceller coverage.
Configure echo canceller coverage.
3. Measure the echo and adjust the echo signal level.
Measure the echo and adjust the echo signal level.
4. Verify the impedance configured in the voice port.
Verify the impedance configured in the voice port.
Step 1 Verify that echo cancellation is enabled on the voice port. Echo cancellation is enabled by default on most platforms in most Cisco IOS releases.
Verify that echo cancellation is enabled on the voice port. Echo cancellation is enabled by default on most platforms in most Cisco IOS releases.Gateway(config-voiceport)# echo-cancel
coverage Echo Cancel Coverage
enable Echo Cancel Enable
Note You must enter the shut, then the no shut commands on the voice port for the changes to take effect
You must enter the shut, then the no shut commands on the voice port for the changes to take effect
Step 2 Configure the echo canceller coverage to a value greater than the time the echo needs to return to the gateway. It must be long enough to cover the worst case for your environment, but not longer.
Configure the echo canceller coverage to a value greater than the time the echo needs to return to the gateway. It must be long enough to cover the worst case for your environment, but not longer.Gateway(config-voiceport)# echo-cancel coverage
16 16 milliseconds echo canceller coverage
24 24 milliseconds echo canceller coverage
32 32 milliseconds echo canceller coverage
8 8 milliseconds echo canceller coverage
Note You must enter the shut and no shut commands on the voice port for the changes to take effect.
You must enter the shut and no shut commands on the voice port for the changes to take effect.
The default coverage is set to 8 ms, but it can be increased up to 64 ms, depending on the Cisco IOS software release. If the PSTN delay (tail length) is more than 32 ms, echo cancellers in Cisco IOS gateways cannot cancel the echo.
Step 3 Measure the echo and adjust the echo signal level as required.
Measure the echo and adjust the echo signal level as required.
No comments:
Post a Comment