The AAC - MP3 - WAV Challenge!

Discussion of music production, audio, equipment and any related topics, either with or without Ableton Live
anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 6:55 pm

The MP3 vs. AAC vs. Uncompressed (WAV) Challenge!

This thread got me thinking.. how distinguishable are MP3 and AAC? (and just for kicks, uncompressed audio?)

To attempt to find out, I have assembled a short blind test (yay, science!)

I have taken a 30 second portion from a MGMT song. Three copies were spawned: one left untouched, one converted to 256k AAC, and one converted to 256k MP3. The latter two were then converted back to WAV to mask their source.

So, can you tell the difference?

For each of these sets, rank them in terms of quality from best to worst.
Like this: A,B,C, if you think A is the best and C is the worst.

For those wishing to do the bonus challenge, I've included a Sinatra and Beethoven test as well. But if you're short on time, just do MGMT.

Use your favorite/most comfortable/best known listening environment!

Then, PM me your results. Do not post them. This will keep this test as neutral as possible. I'll post the results, but in a way that keeps the answers ambiguous. I'll also PM you the answers when you PM me your results.

Here are the links (all WAVs, 5.1MB each)
MGMT A
MGMT B
MGMT C


And for the bonus challenge:
Sinatra A
Sinatra B
Sinatra C

Beethoven A
Beethoven B
Beethoven C



Full Methodology for Nerds and Skeptics
1. Downloaded songs in FLAC format. (All 44.1 kHz)
2. Opened in Audacity. Cut 30 second portion out of song.
3. Added 1 second fade-in and -out on either end of song.
4. Saved as 16 bit, 44.1kHz WAV.
5. Opened in MAX.
6a. Converted to MP3, 256kbps CBR, encoding engine quality high using LAME v3.97
6b. Converted to AAC, 256kbps CBR, maximum quality, using Core Audio
7. Opened each converted file in Audacity; saved as WAV.
8. Randomized order and names.
Last edited by anamexis on Sun Apr 26, 2009 7:16 pm, edited 1 time in total.

locojohn
Posts: 487
Joined: Mon Feb 19, 2007 8:42 am
Location: riga, latvia

Re: The AAC - MP3 - WAV Challenge!

Post by locojohn » Sun Apr 26, 2009 6:58 pm

anamexis wrote:The MP3 vs. AAC vs. Uncompressed (WAV) Challenge! [...]

Full Methodology for Nerds and Skeptics
1. Downloaded songs in FLAC format. (All 44.1 kHz)
2. Opened in Audacity. Cut 30 second portion out of song.
3. Added 1 second fade-in and -out on either end of song.
4. Saved as 16 bit, 44.1kHz WAV.
5. Opened in MAX.
6a. Converted to MP3, 256kbps CBR, encoding engine high using LAME v3.97
6b. Converted to AAC, 256kbps CBR, maximum quality, using Core Audio
7. Opened each converted file in Audacity; saved as WAV.
8. Randomized order and names.
Note that essential information is missing: command line options for lame.

Andrejs
/*
  • the basic tone of life remains the same,
    and in it there are some happy melodies
    and some sad melodies
    - sekito kisen
*/

anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 7:06 pm

locojohn wrote:
Note that essential information is missing: command line options for lame.

Andrejs
It is:

Code: Select all

lame -h -m j -b 256
-h is equivalent of -q2, so the quality could theoretically be higher.

8O
Posts: 5502
Joined: Fri Mar 28, 2008 9:29 am
Location: Berlin

Re: The AAC - MP3 - WAV Challenge!

Post by 8O » Sun Apr 26, 2009 7:13 pm

hehe - great idea - look forward to seeing the results.

Just another question: can we assume that it's the same A, B and C for the MGMT, Sinatra and Beethoven or is it mixed up each time, if you see what I mean?
Image

anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 7:13 pm

8O wrote:hehe - great idea - look forward to seeing the results.

Just another question: can we assume that it's the same A, B and C for the MGMT, Sinatra and Beethoven or is it mixed up each time, if you see what I mean?
Yeah, they're mixed up each time.
Gotta keep you on your feet :)

8O
Posts: 5502
Joined: Fri Mar 28, 2008 9:29 am
Location: Berlin

Re: The AAC - MP3 - WAV Challenge!

Post by 8O » Sun Apr 26, 2009 7:19 pm

Damn! (reaches for his HD-600s in despair... :D )
Image

8O
Posts: 5502
Joined: Fri Mar 28, 2008 9:29 am
Location: Berlin

Re: The AAC - MP3 - WAV Challenge!

Post by 8O » Sun Apr 26, 2009 8:23 pm

PM sent! Thanks for providing this test.

This test should be compulsory for all those arguing in the other thread! I look forward to seeing the overall results...
Image

solacerodgers
Posts: 191
Joined: Wed May 21, 2008 9:35 pm
Location: myrtle beach
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by solacerodgers » Sun Apr 26, 2009 9:02 pm

Did you use CBR or VBR forthe mp3 encoding I cant read that small code?
www.myspace.com/solacerodgers - music mastering and more.

gtodd876
Posts: 180
Joined: Sat Jan 05, 2008 3:00 pm
Location: Virginia
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by gtodd876 » Sun Apr 26, 2009 9:07 pm

Great idea! Thanks for going through he effort to make this happen.

Todd
-----------------------------------------------
Todd Matthews
http://www.facebook.com/mydarlingfury
http://twitter.com/mydarlingfury | http://mydarlingfury.com

Hardtoe
Posts: 182
Joined: Thu Jan 15, 2004 3:50 am

Re: The AAC - MP3 - WAV Challenge!

Post by Hardtoe » Sun Apr 26, 2009 9:28 pm

I will do this tomorrow when I get to the studio and can listen in my regular environment - MGMT may not be the best choice to show the differences in encoding though, as they have VERY heavily limited masters, which removes a lot of the depth that full quality files exhibit - maybe the classical trax will be more revealing...

Thanks for the test...
Can you hear it?

anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 10:14 pm

The first results are in!
And are quite interesting indeed.
Test subject A, we shall name him, pegged MP3 as having best quality on MGMT and Sinatra, and also pegged WAV as the worst on Beethoven :)

Test subject B guessed AAC right, but switched MP3 and WAV.

When I get some more results I'll make some pretty pictures with them.
Last edited by anamexis on Sun Apr 26, 2009 10:19 pm, edited 1 time in total.

anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 10:18 pm

solacerodgers wrote:Did you use CBR or VBR forthe mp3 encoding I cant read that small code?
CBR, I wanted the encoding to be as equivalent as possible.
Hardtoe wrote:I will do this tomorrow when I get to the studio and can listen in my regular environment - MGMT may not be the best choice to show the differences in encoding though, as they have VERY heavily limited masters, which removes a lot of the depth that full quality files exhibit - maybe the classical trax will be more revealing...
Agreed regarding MGMT and limited masters, but I also had a point by choosing that - namely that a lot of what people are DJing with and producing are heavily limited masters themselves. Basically, it provides a more real-world setting.

anamexis
Posts: 582
Joined: Sat Dec 18, 2004 9:43 pm
Location: Oakland
Contact:

Re: The AAC - MP3 - WAV Challenge!

Post by anamexis » Sun Apr 26, 2009 11:25 pm

Holy crap.
With 3 people responding so far, check it out:
Image

WAV hasn't been picked as best-sounding once, and MP3 has been picked as best sounding 80% of the time.

Paging the people from the MP3 vs AAC thread... I want your results :)

selthym
Posts: 380
Joined: Sat Oct 07, 2006 2:20 pm

Re: The AAC - MP3 - WAV Challenge!

Post by selthym » Mon Apr 27, 2009 2:51 am

anamexis wrote:Holy crap.
With 3 people responding so far, check it out:
Image

WAV hasn't been picked as best-sounding once, and MP3 has been picked as best sounding 80% of the time.

Paging the people from the MP3 vs AAC thread... I want your results :)
Could this be because people are so used to listening to MP3 now, that it sounds the most normal, which is then perceived as the best???

hereschenes
Posts: 154
Joined: Mon Mar 20, 2006 9:14 am
Location: Adelaide, SA

Re: The AAC - MP3 - WAV Challenge!

Post by hereschenes » Mon Apr 27, 2009 5:31 am

Ummm... don't get too excited about the data when your sample size = 3.

I'll send you my results tonight! ;)

Post Reply