Part
01
of one
Part
01
What are metrics for comparing VIOP audio call quality
EXECUTIVE SUMMARY
The most common industry standard for measuring VoIP call quality is the Mean Opinion Score (MOS). The MOS is determined by measuring bandwidth, latency, jitter, packet loss, compression, and codecs, with a minimum score of 4.3-of-5 being preferred for VoIP calls. Other measures like R-Factor, Gap and Burst Density, and QoS Prioritization are important, as well. The POLQA, or Perceptual Objective Listening Quality Analysis, is another of the most-current industry standards for measuring call quality. Cisco is known within the VoIP industry as having the most comprehensive evaluation system on the market, versions of which are used by many businesses across the globe.
VOIP CALL QUALITY METRICS: PRIMARY METRICS
According to industry experts, there are a variety of metrics that should be used when evaluating the call quality of a VoIP system. These include measures for three primary categories: (1) Listening Quality, (2) Conversational Quality, and (3) Transmission Quality. For each of these, a variety of measures are included in the overall scoring.
Drilled down to major categories, primary metrics to evaluate include:
• BANDWIDTH (TRANSMISSION/LISTENING): This metric measures the “throughput capacity of your WAN connection,” as noted by JCMTechnologySolutions. This is measured in Mb/second and the number correlate to download / upload speeds. However, they note that, “having a lot of capacity doesn’t always mean fast throughput”.
• LATENCY (TRANSMISSION/LISTENING): Latency is a secondary measurement to bandwidth and measures “how fast your link to another party is,” states JCMTS. The lower your latency score, the better your audio call quality.
• JITTER (TRANSMISSION/LISTENING): Jitter refers to the “timing gaps between data packets being sent out on internet lines,” according to JCMTS. This is measured in millisecond counts (MS) using packet time stamps, or using Real-Time Protocol (RTP) time stamps (not measured in MS). ViaviSolutions explains how this is calculated: The calculation is performed on every packet with the jitter max(imum) being the highest calculated jitter observed by the system conducting the measurements. The latest / most-recent jitter value is also measured. They note that “jitter requires a minimum of 16 packets observed” to obtain a value measurement. So, if your jitter rating is low or nonexistent, your call quality is higher. Fiber connections have the best jitter rate, followed by DSL and DOCSIS (coax), and cellular and satellite services have the worst jitter ratings.
• PACKET LOSS (TRANSMISSION/LISTENING): Packets are the virtual suitcases carrying your data from one place to another, with the loss of these often being the “result of the jitter buffer being overwhelmed,” as noted by ViaviSolutions. Additional reasons for packet loss can include poor wireless-signal quality or the failure of landlines. The best VoIP systems show the lowest (or no) packet loss, as “even just a few percent of packet loss equates to terrible VoIP calls,” as noted by JCMTS.
• VOICE COMPRESSION (TRANSMISSION/LISTENING): Although dated, this book from Cisco provides information on voice compression, which in North America, is measured in u-law (as compared to Europe’s a-law) or in adaptive differential pulse code modulation (ADPCM). SearchUnifiedCommunications notes that compression is used in VoIP “to lower the size of voice packets to improve the effectiveness of transmission”. VoIP uses a variety of compression ratios, from 1:1 to 12:1. The lower the ratio, the better the quality of the call. They note that “The determining factor is available bandwidth, which is related to your VoIP setup and, of course, to the speed of the connection -- either Internet or proprietary -- that ‘carries’ your VoIP session”.
• CODECS (TRANSMISSION/LISTENING): Codec means “Coder/Decoder” and refers to “the algorithm used to covert the analog voice signal into packets on the network, and back again,” as noted by ViaviSolutions. Codecs use a variety of sampling rates for implementing various compression levels. For this measure, lower sampling rates mean lower call quality. However, ViaviSolutions notes that “sometimes a lower sampling rate can reduce contention and prevent worse degradation”. This book from Cisco outlines the Voice Coding Standards (industry-wide) for telephony and packet voice measures, if you’re interested.
Next, let's look at the Mean Opinion Score.
MEAN OPINION SCORE (MOS)
Altogether, the major metrics described above make up the industry average measurement called MOS, or Mean Opinion Score, which measures overall Listening Quality. JCMTS defines the MOS as “a numerical measure to quantify the subjective nature of voice quality as registered by the human ear”. VoIPMechanic states that “this measurement is the result of underlying network attributes that act upon data flow and is useful in predicting call quality and is a good VoIP test tool in determining issues that can affect your VoIP quality and your conversations”. This measurement system (MOS) has been used for decades, and uses the Absolute Category Ranking (1 to 5 scale), with 5 being the best call quality and 1 being the worst.
The higher the score (out of 5), the better the VoIP call quality.
Industry standards note that a minimum score of 4.3-of-5 must be met to have superior-level voice call quality. Any score lower than that equals lower-quality calls, with the lower it goes equaling worse call quality. VoIPMechanic has a great article that outlines exactly what most-affects (negatively and positively) a VoIP MOS score, if you’d like to read more.
VOIP CALL QUALITY METRICS: SECONDARY METRICS
Additionally, there are other measurements which are important to take into consideration, but not necessarily measured by the MOS. These additional measurements include those outlined below:
• R-FACTOR (LISTENING/CONVERSATIONAL): This measures life, real-time user-experience with VoIP phone calls, and impacts transmission quality. This takes into account delay, echo, and recency as measuring factors. This is measured on a scale of 1-to-100, though ViaviSolutions notes that “unavoidable degradation means that 93.2 is the highest reading you will see on an actual VoIP call; scores below 80 typically results in dissatisfied users”.
• GAP & BURST DENSITY (TRANSMISSION): This relates to packet loss. With VoIP calls, some packet loss is inevitable (like from temporary utilization spikes, as an example). Periods where packet loss is minimal (but present) are called “gaps,” while periods where high-packet-loss occurs are called “bursts”. ViaviSolutions notes that “density refers to the rate of packet loss during bursts and gaps”. The burst periods are responsible for low-quality calls as they show the highest packet loss. ViaviSolutions explains: “Understanding the density and duration of bursts and gaps can help you quickly respond to (and sometimes prevent) voice degradation on the VoIP network. For example, an extremely high burst density (20% or more) coupled with extended burst duration times (more than a second or two) can suggest problems with hardware either failing or being completely overwhelmed by traffic. Gap densities climbing over time, coupled with low-density, short-duration 'burstiness' can mean the VoIP network is attempting to service too many calls given the available bandwidth”.
• QUALITY OF SERVICE (QoS) PRIORITIZATION (TRANSMISSION): This is also referred to as the Type of Service (ToS) or Precedence. This involves the routers/switches recognizing and prioritizing traffic based on what each application requires (and how much bandwidth is available). VoIP requires the highest priority level. If this is set incorrectly within the VoIP system, it will be in contention with other network data, which will lead to jitter and packet loss resulting in poor call quality.
ViaviSolutions notes that, “VoIP monitoring tools calculate the MOS and R-factor scores using a formula known as the E-model. Using the statistics it has collected from the network, the analyzer calculates how much the various impairment factors (such as codec compression, jitter, delay, and packet loss) would affect the typical user’s perception of call quality. MOS and R-factor are used to gauge user satisfaction with call quality. MOS levels under 3.5 and R-factor below 80 mean trouble”. Note that other experts say 3.3 is acceptable (as noted/linked above).
MONITORING TECHNIQUES: PESQ & POLQA
WhatWhenHow describes an additional industry-standard objective VoIP call quality measuring technique, “Active monitoring techniques of PESQ” (Perceptual Evaluation of Speech Quality). This perceptual model utilizes human auditory perception and “has proven to the best way of accurately predicting the audibility and annoyance of complex distortions”. This measures one-way voice quality on the half-duplex scale. For detailed information on this, please see the report from WhatWhenHow.
However, it should be noted that POLQA (which was adopted 10 years after PESQ) is the most-recent industry standard for this metric, though which one you will use will depend on the VoIP system you’re using. GL notes that “PESQ-based measurements will still be considered an industry standard for several years, also for reasons of backward compatibility”.
POLQA, or Perceptual Objective Listening Quality Analysis, is noted as having “overcome all known issues and limitations of PESQ, “ states GL. It is “suitable for 3G and 4G networks, VoIP networks and NGN networks delivering HD- quality voice services such as ‘wideband’ and ‘super-wideband’ telephone calls, 7 kHz and 14 kHz frequency range”. To learn all about the POLQA measurement, check out this article from GL.
CISCO VOIP CALL QUALITY MEASUREMENT SYSTEM
One of the best-known systems for measuring VoIP call quality is produced by Cisco. They use the “Cisco Service Assurance Agent (SAA) and Internetwork Performance Monitor (IPM) to measure quality of service (QoS) in Voice over IP (VoIP) networks”. This article and this one outline their metrics and procedures so that you can apply their methodologies to your own call-quality measuring. Additionally, you can find detailed information on all aspects of VoIP quality measures from this book from Cisco. Although a bit dated, it remains an industry-standard for information on this.
Overall, different companies using VoIP measure the call-quality of their systems against each of the measures above. Some utilize proprietary software or other programs available on the market, while others utilize these services through their telephony providers. If you’d like Wonder to research the top-rated software used in VoIP measure-reviews, then we’d be happy to do so, just let us know!
ADDITIONAL HELPFUL INFORMATION
This article from ViaviSolutions lists the “VoIP Top 10 Best Practices” to follow when you are implementing a new VoIP system or troubleshooting your current one. These tips will be of great value to you in your VoIP testing and comparison efforts.
This article from VoIP-Info outlines additional information and provides links to a wide variety of other sources that may be of use and/or interest to you in your own testing.
CONCLUSION/SUMMARY
Metrics for evaluating VoIP call quality have evolved over the last several decades as the related technologies have evolved. Industry standards have emerged including the MOS and POLQA, as well as specific standards for each distinct item needing measurement. The most-common measurements used in determining VoIP call quality include those for Listening Quality, Conversational Quality, and Transmission Quality.