The AAC - MP3 - WAV Challenge!

[anamexis]

The MP3 vs. AAC vs. Uncompressed (WAV) Challenge!

This thread got me thinking.. how distinguishable are MP3 and AAC? (and just for kicks, uncompressed audio?)

To attempt to find out, I have assembled a short blind test (yay, science!)

I have taken a 30 second portion from a MGMT song. Three copies were spawned: one left untouched, one converted to 256k AAC, and one converted to 256k MP3. The latter two were then converted back to WAV to mask their source.

So, can you tell the difference?

For each of these sets, rank them in terms of quality from best to worst.
Like this: A,B,C, if you think A is the best and C is the worst.

For those wishing to do the bonus challenge, I've included a Sinatra and Beethoven test as well. But if you're short on time, just do MGMT.

Use your favorite/most comfortable/best known listening environment!

Then, PM me your results. Do not post them. This will keep this test as neutral as possible. I'll post the results, but in a way that keeps the answers ambiguous. I'll also PM you the answers when you PM me your results.

Here are the links (all WAVs, 5.1MB each)
MGMT A
MGMT B
MGMT C


And for the bonus challenge:
Sinatra A
Sinatra B
Sinatra C

Beethoven A
Beethoven B
Beethoven C



Full Methodology for Nerds and Skeptics
1. Downloaded songs in FLAC format. (All 44.1 kHz)
2. Opened in Audacity. Cut 30 second portion out of song.
3. Added 1 second fade-in and -out on either end of song.
4. Saved as 16 bit, 44.1kHz WAV.
5. Opened in MAX.
6a. Converted to MP3, 256kbps CBR, encoding engine quality high using LAME v3.97
6b. Converted to AAC, 256kbps CBR, maximum quality, using Core Audio
7. Opened each converted file in Audacity; saved as WAV.
8. Randomized order and names.

[locojohn]

The MP3 vs. AAC vs. Uncompressed (WAV) Challenge! [...]

Full Methodology for Nerds and Skeptics
1. Downloaded songs in FLAC format. (All 44.1 kHz)
2. Opened in Audacity. Cut 30 second portion out of song.
3. Added 1 second fade-in and -out on either end of song.
4. Saved as 16 bit, 44.1kHz WAV.
5. Opened in MAX.
6a. Converted to MP3, 256kbps CBR, encoding engine high using LAME v3.97
6b. Converted to AAC, 256kbps CBR, maximum quality, using Core Audio
7. Opened each converted file in Audacity; saved as WAV.
8. Randomized order and names.

Note that essential information is missing: command line options for lame.

Andrejs

[anamexis]

Note that essential information is missing: command line options for lame.

Andrejs

It is:

Code: Select all

lame -h -m j -b 256

-h is equivalent of -q2, so the quality could theoretically be higher.

[8O]

hehe - great idea - look forward to seeing the results.

Just another question: can we assume that it's the same A, B and C for the MGMT, Sinatra and Beethoven or is it mixed up each time, if you see what I mean?

[anamexis]

hehe - great idea - look forward to seeing the results.

Just another question: can we assume that it's the same A, B and C for the MGMT, Sinatra and Beethoven or is it mixed up each time, if you see what I mean?

Yeah, they're mixed up each time.
Gotta keep you on your feet

[8O]

Damn! (reaches for his HD-600s in despair... )

[8O]

PM sent! Thanks for providing this test.

This test should be compulsory for all those arguing in the other thread! I look forward to seeing the overall results...

[solacerodgers]

Did you use CBR or VBR forthe mp3 encoding I cant read that small code?

[gtodd876]

Great idea! Thanks for going through he effort to make this happen.

Todd

[Hardtoe]

I will do this tomorrow when I get to the studio and can listen in my regular environment - MGMT may not be the best choice to show the differences in encoding though, as they have VERY heavily limited masters, which removes a lot of the depth that full quality files exhibit - maybe the classical trax will be more revealing...

Thanks for the test...

[anamexis]

The first results are in!
And are quite interesting indeed.
Test subject A, we shall name him, pegged MP3 as having best quality on MGMT and Sinatra, and also pegged WAV as the worst on Beethoven

Test subject B guessed AAC right, but switched MP3 and WAV.

When I get some more results I'll make some pretty pictures with them.

[anamexis]

Did you use CBR or VBR forthe mp3 encoding I cant read that small code?

CBR, I wanted the encoding to be as equivalent as possible.

I will do this tomorrow when I get to the studio and can listen in my regular environment - MGMT may not be the best choice to show the differences in encoding though, as they have VERY heavily limited masters, which removes a lot of the depth that full quality files exhibit - maybe the classical trax will be more revealing...

Agreed regarding MGMT and limited masters, but I also had a point by choosing that - namely that a lot of what people are DJing with and producing are heavily limited masters themselves. Basically, it provides a more real-world setting.

[anamexis]

Holy crap.
With 3 people responding so far, check it out:


WAV hasn't been picked as best-sounding once, and MP3 has been picked as best sounding 80% of the time.

Paging the people from the MP3 vs AAC thread... I want your results

[selthym]

Holy crap.
With 3 people responding so far, check it out:


WAV hasn't been picked as best-sounding once, and MP3 has been picked as best sounding 80% of the time.

Paging the people from the MP3 vs AAC thread... I want your results

Could this be because people are so used to listening to MP3 now, that it sounds the most normal, which is then perceived as the best???

[hereschenes]

Ummm... don't get too excited about the data when your sample size = 3.

I'll send you my results tonight!