H264 TESTS (Draft, Nov. 14, 2005)

These are preliminary and incomplete. There are many parameters that can be varied when compressing to H.264. We decided to initially limit them to a few.

Source material was a 70 second clip from a documentary by Eric Archer, shot with a Sony PD-100 to DVCam. The DV file is 4:1:1 color-space, and has a lot of motion, subtle texture gradients, thin diagonal lines, chroma saturation, major luminance changes. In other words, a compression buster.

Our approach to educational videos is that content is more important than visual quality, as long as the quality is acceptable. These tests are NOT representative of the ultimate quality H.264 can achieve, the goal was to "break" the codec, and assess what needs to be done to achieve acceptable quality. This second step was not always performed. The tests include many compression artifacts, parameters have not always been "pulled back" to eliminate them.

In our usual workflow we don't have the time to tweak each parameter for the specific video content at hand, to wring the highest possible quality at the lowest data rate. Our end-goal is to keep our video data rates as low, and our frame size as high, as reasonably possible at 29.97 fps, without introducing too many compression artifacts. These are not Apple QuickTime trailers! We did not perform and pre-processing before compression, aside from deinterlacing when noted.

For the first set of tests (file name starts with QTP), we used QuickTime Player Pro, compressing using File>Export>Movie to QuickTime Movie, and varied the Options. In Video Settings, H.264 was always selected, frame rate was always 29.97 fps, and data rate 500 kbps, unless noted. Unless noted, we used single-pass encoding, and Optimized for Download. We used Frame Reordering. It was not possible to move the quality slider from "High". We already found one mistake, the movies are not fast-start. For this, we should have selected Prepare for Internet Streaming: Fast Start. To change them to fast-start, it will be enough to do a "Save As", they do not need to be recompressed.

For us, single-pass encoding works fine for streaming, we don't use CBR. Thus, the single-pass files can be hinted and streamed. We decided to deliver them by http to make them easier to compare. There are other tools to compress video, we first wanted to see how much can be done with QuickTime Player Pro.

Audio was compressed to: AAC, mono, 32KHz, CBR, 32kbps, Precedence: bit rate, Quality: best.
Computer used was a Dual 2 GHz PowerPC G5 with 1 GB of RAM, QuickTime 7.0.3, OS X 10.4.3
The encoding time (min:sec) is noted under the clip. If you want to compare clips, open two or more, and go to View>Play All Movies. To compare frames, use Window>Show Movie Info, and look at the Current Time while you select each clip.

Click here to go to the movies

Test 1: General Quality and Scrubbing ability
We wanted to get a general idea of the degree of compression artifacts at different frame sizes, at 500 kbps. Obviously, the bigger the frame size, the greater the number of artifacts present. The QTP3 384x288 version is not too bad, but suffers from interlacing problems. It's not possible to deinterlace with QTP Pro.

We wanted to see if there was a difference in scrubbing ability (how fast you can drag the playhead across the timeline and have the video refresh) and visual quality between automatic keyframes and a keyframe every 150. This was tested at 512x384 (QTP1 vs. QTP1b) and 448x336 (QTP2 vs. QTP2b). No significant difference in quality or scrubbing ability was detected. If the clips are to be streamed, it might make a difference, this needs to be tested. We do not yet have the tools to measure where the automatic keyframes were placed, or their frequency.

Test 2: One-Pass vs. Multi-Pass
We tested the 448x336 size, QTP2b (Single-pass) vs. QTP2c (Multiple-pass). The multi-pass encoding time was more than twice as long. Watching the movie while playing, multi-pass quality was a bit better. When the movies where stopped at the same frame, there were usually fewer artifacts in the Multi-Pass version. However, any improvements are overwhelmed by severe interlacing artifacts.

We tried to deinterlace in QTP, by going to Window>Show Movie Properties, selecting the Video Track, Visual Settings, and trying different combinations of High Quality, Single Field and Deinterlace. While these affected the appearance of the master DV file, no effect was found in the compressed video.

Using multiple-pass on a smaller movie, 384x288, which had fewer artifacts than the 448x336 movie, the quality difference between one-pass (QTP3) and multiple-pass (QTP3c) was not very noticeable.

Test 3: Optimizing for 320x240
We compressed at 4 different data rates, 200 through 400 kpbs, files QTP4 through QTP4e. We thought the 350 kbps, multi-pass test was adequate, but there were a few compression artifacts. These were decreased, but not eliminated in the 400 kbps mulit-pass version (QTP4e).

We encoded a 15 fps version at 300 kbps, QTP4f. Still frame quality was about the same as the 29.97 fps 350 kbps version. However, playback was strobe-like, due to high-motion. If data rate has to be limited to 300 kbps at this frame size, the 29.97 version (QTP4c) might be less objectionable than the 15 fps version. Even though it has more compression artifacts, motion is smoother.

There were no interlacing artifacts in the 320x240 videos.
This concluded our tests with QuickTime Player Pro, for now, and we went on to Compressor.

Test 4: Comparing Compressor with QuickTime Player Pro without interlacing
In Compressor, the Video Settings and Audio Settings take you to the same selection windows as when using QTP Pro. However, "Optimized for:" is grayed out, this is set in Compressor. We first tested to see if there was a speed or quality difference between QTP Pro and Compressor for one of the movies. We choose QTP3: 384x288, 500 kbps. The result, CO3, was the same visual quality, and took a little less time to compress.

Test 5: Using Compressor Deinterlacing
In Compressor, in Filters, we tried both the Even (CO3b) and Blur (CO3c) Algorithms. There does not seem to be any appreciable difference in picture quality, however Blur results in a "double image" in areas of high motion when the video was stopped.

Image quality is markedly superior in CO3b to the non-interlaced, single-pass QTP Pro version QTP3, and displays very few artifacts. By increasing the data rate to 550 kbps in CO3d, most were eliminated. We would consider, for our purposes, either version usable.

We have been unable to reliably test multi-pass encoding with Compressor, selecting Multi-Pass is not "sticky", that is, when you close the Video setting window and re-open it, the selection is back to "Single-Pass", no matter what your selections are in Optimized For, Data Rate, and Streaming.

Test 6: Using Compressor's Frame Controls instead of Deinterlacing Filters

According to the manual, deinterlacing in Frame Controls provide better quality than in Filters. This, and other tests, remain to be performed.

If you have any comments on these preliminary tests, you may direct them to Frank Fulchiero, fful@conncoll.edu