Articles and whitepapers

A Whitepaper - Solving Digital Image Artefacts with Advanced Video Processing (3/10/2006)

By Phuc-Tue Le Dinh and Jacques Patry, Algolith

If you find the picture from your new HDTV set less than perfect, some new video processing technology might be the solution you are looking for.

Television, Analogue vs. Digital

The technology behind what was the centrepiece of American culture is on its last legs. The de-facto standard for the last 40 years, the NTSC-Analogue television system, will soon become obsolete with the promise of a sharper image, higher resolution and more vibrant colours thru Digital Television (DTV).

DTV technology is widely considered the most significant step in TV technology since colour television, but the process of its introduction extends for over more than a decade. Let us put aside for now the politics and legislative activities concerning its adoption and focus on its technical merits. We'll also look into how this new standard, for years to come, will affect the end-user.

The Bandwidth Battle

In theory, DTV's picture quality is undoubtedly superior to traditional Analogue TV: no more 'ghosting', 'snow', 'judder', 'Never The Same Colour', etc. Nevertheless, analogue signals' arguably most glaring weakness is its blurriness and lack of fine image details due to shortcomings in high frequency response, or simply put, in bandwidth. The more detailed an image is or the more resolution it has, the more bandwidth it needs.

Traditionally, this bandwidth limitation has always been transparent to the end user. Long ago, it was agreed upon that 6 megahertz (MHz) of bandwidth from the allowed spectrum would be allocated to each channel for broadcasters in America to provide these analogue TV signals.

This limitation in video bandwidth and its corresponding standard (NTSC) in turn dictated the specifications of the traditional TV set, as well as its picture quality for decades. With the advent of DTV, broadcasters saw a great opportunity to make much better use of their bandwidth (and to ask for more...). Indeed, from their standpoint, one of the sterling advantages of DTV was that it allowed for multiple channels in the same amount of bandwidth, and would later on allow for High-Definition Programming (HDTV).

Everywhere, HDTV

The underlying principle behind HDTV is to recreate the movie experience by increasing the visual field contained in an image. Basically, it means more resolution within a more immersive aspect-ratio (16:9). It also means a surge in technical requirements.

A conventional NTSC signal has 525 lines scanned at 29.97Hz for a 4.2MHz minimum bandwidth to carry the analogue video off of a 6MHz channel [0]. When digitised and compressed, this signal can be recorded on a DVD and its bit rate varies from 2 to 10M/s (adaptive) with an average of 4M/s. For comparison, a typical HDTV feed has roughly 5 times the resolution. All things being equal, the transmitted bit rate should be around 5 times more important to deliver similar performance.

Imagine everyone swapping their cars for motor homes and they are all trying to reach their downtown office at the same time. Unless you could do some major urban reconstruction in a blink of an eye, it's safe to say that you won't be getting to work anytime soon. The challenge is very similar when integrating and delivering HDTV to the consumer. The roads that were built are now simply not big enough.

Whether it's the traditional over-the-air (OTA) broadcast, the cable company's set-top box or the satellite-TV provider, they all have a limited amount of bandwidth to send you all these feeds, to which, they add other bandwidth-intensive services such as interactive broadcasts, subscription channels, TV schedules, etc.

So what is the solution? Compression.

Digital Video Compression

Even if you could magically double your bandwidth, it would still not be enough to meet the needs of the brave new digital world. This is where codecs (Coder/Decoder) come into play. The most commonly used method today to compress digital video data is called MPEG-2.

From current satellite streams and digital cable feeds to off-the-air digital broadcasts, MPEG-2 has now been internationally adopted for a variety of applications.

In a nutshell, MPEG-2 first exploits temporal redundancy through motion estimation and then proceeds to spatially subdivide the image in 8x8 blocks upon which the DCT (Discrete Cosine Transform) is applied to exploit spatial redundancy. Compression is done by quantizing resulting DCT coefficients and re-ordering them to maximize the probability of long runs of zeroes, and then run-length coded. Finally a Huffman encoding scheme is used. The whole process allows for great savings in terms of bit-rate ratio (>10:1).

However, these savings don't come free, and because the codec discards some of the original video information, there can be serious side-effects; MPEG-2 is what we call a lossy codec. It discards image information believed to be of lesser visual importance. The more you want to compress, the further away you get from the look of the original image. Image quality and fidelity now depends on the chosen (or often imposed) level of compression. And since that is directly tied to the available bandwidth, we must ask ourselves when is the video simply too compressed?

Visible Artefacts

Bandwidth restrictions in the digital domain, combined with an aggressive image compression scheme, will manifest themselves differently than in the analogue world.

Usually, the analogue degradation (or noise) will more often than not follow a Gaussian distribution. This distribution's advantage is that it will preserve essential content and mimic our eyes' drop-off. We usually find a constrained analogue image a bit fuzzy but nothing clearly objectionable.

Digital noise follows a different distribution pattern and, more importantly, has a particular shape that the human perception finds unnatural. There are mainly two artefacts who present the latter characteristic when pushing the limits of MPEG-2 (or any DCT block-based codec): Mosquito noise and Blocking artefacts.

- Mosquito noise, a.k.a. Gibbs effect

Mosquito noise is most apparent around artificial or CG (Computer Generated) objects or scrolling credits (lettering) on a plain coloured background. It appears as some haziness and/or shimmering around high-frequency content (sharp transitions between foreground entities and the background or hard edges) and can sometimes be mistaken for ringing. Unfortunately, this peppered effect is also visible around more natural shapes like a human body.

The VIRIS project (a Video Reference Impairment System [1]) defines mosquito noise as follows: 'Form of edge busyness distortion sometimes associated with movement, characterized by moving artefacts and/or blotchy noise patterns superimposed over the objects (resembling a mosquito flying around a person's head and shoulders).'

Mosquito noise

It occurs when reconstructing the image and approximating discarded data by inversing the transform model (iDCT). 'Mosquitoes' can also be found in other areas of an image. For instance, the presence of a very distinct texture or film grain at compression will also introduce mosquito noise. The result will be somewhat similar to random noise; the mosquitoes will seem to blend with the texture or the film grain and can look like original features of the picture.

- Blocking artefacts

Blocking artefacts, as its name suggests, manifest themselves as objectionable and unnatural blocks within an image. Sometimes referred to as macro blocking, it is a picture distortion characterized by the underlying block encoding structure becoming visible.

Blocking artefacts

When pushing the limits of the encoder, the blocks are rather roughly averaged, making them appear as one big pixel. From block to block, the average calculated can vary, and thus creates these well defined borders between blocks.

This effect becomes even more pronounced when there's some fast motion or quick camera movement. Probably the best example for this is during NFL telecasts, where the player carrying the football can quickly turn into some form of low-res, blocky, pixelated Mario Bros. look-alike from the old Nintendo days.

- Pre-Smoothing

Although not part of the compression artefact family, pre-smoothing has worked its way onto this short list of conspicuous digital annoyances. Broadcasters and content providers alike have become increasingly aware of their distribution system's shortcomings. Some of them came up with a rather disputable solution to their bandwidth restrictions: pre-smoothing.

By removing high-frequency content in the picture before putting it through their transmission chain, the encoder has a much easier time to do its job, and the resulting images are less subject to blocking artefacts and mosquito noise. On the other hand, this sometimes excessive filtering also gets rid of all the subtle details and textures of the original image. The football player with his 1- week playoff beard is now cleanly shaved (even when he's not moving) and the Astro-turf turns into a big green mass of Play-doh...

Arguably, one might think that it's not such a bad trade-off, but pre-smoothing is simply an irreversible process. Once they erase the details, one cannot go about re-creating those details at the other end.

Mosquito noise and blocking artefacts, however, can be addressed.

MNR: Algolith's Solution

From an academic standpoint, compression artefacts and their correction have been extensively studied, but until now, there haven't been many tangible solutions for the end-user.

Algolith is one of the first to provide a proprietary real-time solution to mosquito noise and blocking artefacts: Algolith's MNR - MPEG Noise Reducer.

MNR - MPEG Noise Reducer Algolith's MNR is a highly automated, autonomous, efficient and non-iterative technique for post-processing DCT compressed images and is the fruit of more than 10 years of algorithmic research and development.

MNR implements four distinct image processing techniques:

1. Per-pixel temporal recursive noise reduction.
2. Mosquito noise reduction using sophisticated segmentation techniques.
3. BAR - Block Artefact Reduction by detecting, blending and diminishing inherent block structures of DCT block-based compression.
4. Multiple user adjustable image enhancement options using nonlinear filtering.

The essence of the MNR resides in its spatial image analysis module. Each pixel is classified into specific regions of interest; edge, texture, flat and artefact areas are all distinguished. It also looks into the temporal domain to discern motion areas of the picture. Combining all this information determines one of many different filters to be applied.

MNR's spatial image analysis

This adaptability is what sets MNR apart. The MNR is able to address very specific problem areas without hindering the rest of the image. It is probably as important to know when and where to filter as it is to know when and where not to filter. It's with this motto in mind that the MNR was designed. As a result, MNR only enhances the viewing experience and has proven to be a much needed add-on for large screen displays and projection screens.

Its highly adaptive nature also allows for improved picture quality without disrupting the established broadcasting infrastructure. And since MNR has always been designed with real-time realization and hardware feasibility in mind, it can be seamlessly inserted into an end-user's existing home-theatre equipment.

Mosquito noise (left), processed with Algolith's MNR (right)

Blocking artefacts (left), processed with Algolith's BAR (right)

The Progression of Displays

It used to be that the standard NTSC would dictate the specifications of the display. Analogue TVs always had the same resolution for ages with only marginal improvements in visual quality. Nowadays, the inability from the governing bodies to stick and impose one digital standard, for better or worse, has blown-up the specifications of the generic display.

There's no more unique standard resolution to cater to; and, with the rapid rise of new display technologies (LCOS, DLP, etc.), some of the new monitors can now exceed the maximum resolution of the feeds. More importantly newer displays boast higher contrast ratios and are reaching sizes that were unfathomable only a few years ago.

All of these elements put more stress on picture quality as they act as a magnifying glass on potential artefacts. Suddenly, the feeds are being exposed under a very harsh (back-)light and their imperfections, uncovered.

As display technology continues to evolve, remedies to faulty source material, such as Algolith's proprietary MNR, will become more attractive.

Another Perspective

The transition to the digital world is not just restricted to the broadcasting industry. The old VHS tape is also dying a rather quick death (if it's not dead already) at the hands of the all-digital DVD-Video (Digital Versatile Disc). But even this technology is not safe from the needs to compress.

Indeed, including bonus features, extra footage and multiple soundtracks to an already constrained amount of space can make the picture marred with visual artefacts. As more 'value' is added, disk space becomes a premium and a higher compression ratio, the salvation. In turn, consumers complained about the deteriorating visual quality and this has prompted the 'Superbit' collection of DVDs, where the focus is on maximizing the space for the movie feature itself [2].

This move from the DVD publishing industry not only confirms the possible drawbacks of compression, but also highlights the public's awareness to them.

Hence, the normal DVD medium would also benefit from further video processing such as Algolith's MNR solution. In this case, MNR would allow current generation media to stay relevant while we wait for the winner to emerge from the upcoming Blu-ray vs. HD-DVD vs. HVD battle.

Future Outlook

The winner(s) of the new format battle aims at delivering HD content with the highest visual quality, something simply not possible with current DVD technology.

Furthermore, these new formats, in addition to current codecs, will also include next generation ones:

* MPEG-2, enhanced for HD [3].
* VC-1, informal draft name of the SMPTE standard 421M, based on Microsoft's Windows Media Video (WMV) technology.
* MPEG-4/AVC a.k.a. H.264, arguably the most promising.

However, this variety of standards, combined with ongoing licensing issues might result in the same disorder that has plagued DTV from the start. It remains to be seen as to how final image quality will be affected.

Summary

DTV and HDTV were primed to be the Holy Grail in picture quality. Even so, the reality is that we are still far from image nirvana. The need to compress has brought forth several problems of its own and, with ever better display technologies emerging, these issues are becoming more apparent to the average viewer. The industry, as a whole, is well aware of the harsh facts of limited video bandwidth. Advancement in codec efficiency is showing potential as the next great solution in picture improvement. Yet, as the future seems to be determined to increase its bandwidth needs (IPTV, increased interactivity, specialized content, etc.), one has to wonder if innovations in compression technology alone will ever catch up to the consumer's expectations.

As this gap widens, the need for better video processing algorithms will certainly become another battlefield for a clearer tomorrow.

References
[0] Recommendation ITU-R BT.470-6, Conventional Television Systems
[1] ITU-T Recommendation P.930, Principles of a reference impairment system for video, 8/96.
[2]'The SUPERBIT(tm) collection converts the physical space devoted to added value to higher bit rate video (...)', - Sony Pictures.
[3] Press Release, Sony Pictures Home Entertainment Completes First Full- Length Blu-ray Disc, Friday November 18, 5:54 pm ET CULVER CITY, Calif., Nov. 18 /PRNewswire/.

Algolith is a spin-off of a broadcast infrastructure company that helped transition major broadcast networks from analogue to digital transmission. With its wealth of expertise, has become a leading image enhancement solutions company with a range of innovative products and proprietary algorithms that meet the needs of a variety of customers.

www.algolith.com