Seq/DAW benchmark 2011, Live 8.22 vs. Studio1 vs. FL studio

[golemus2]

OK, I have had so much problems with Live that I decided to do a benchmark. I installed many hosts to my machine and did the exactly same test project in each of them. This project reflects what I might have loaded in a typical project, (is maybe a more heavy though), total 40 plugins. In this project I loaded the following plugins to separate tracks:

5 instances of ReFX Nexus
5 instances of Korg M1
1 Kontakt
1 StylusRMX
8 instances of Zebra
5 instances of Sylenth
2 instances of Z3TA+
1 Slayer
1 Absynth
10 instances of Synth 1



My machine is Core2-duo Laptop, Win7-64, 6GB RAM, Fastrack Pro, cost like 650eur so not a high end laptop. I did this with Ableton Live 8.22, FL Studio 9.1, FL Studio 9.94, FL Studio 10 demo, Cubase 5.12, Studio One 1.63. All hosts are 32-bit versions and plugins too. I chose exactly same patches in every host to the plugins. Did exactly same patterns that had simple 1/16 notes in every beat (except I did so that not every plugin plays a note at the same time). After this (still not pressing the play button) I monitored from task manager how much resources the host takes and how much the CPU value shows:

FL Studio 9.1 CPU 19-35 1149MB memory (private working size), 1270MB memory (commit memory), after reloading p-mem 877MB, c-mem 1011MB

Cubase 5.1, CPU 15-25, p-mem 1274, c-mem, 1436, after reloading p-mem 1016, c-mem 1177

Studio One, CPU 22-30, p-mem 1310, c-mem 1440, after reloading p-mem 1038, c-mem 1174

Ableton Live, CPU 0-17, p-mem 1461, c-mem 1588, after reloading, pmem 1187, cmem 1325

FL Studio 9.94beta, CPU 18-29, after reloading pmem 887, cmem 1026



After this I started playing the project in loop (there was only 1 bar loop). I did this first with Studio One. I found out that it was far too heavy for my machine to play it so I deleted every third MIDI clip and tried again. After this Studio One was able to play it smoothly unless I moved some window which caused crackling. CPU was 76-85, memory the same as before. Then while the loop was playing I did this test. I added another Nexus instance under the 40 tracks and measured how long it took before it was loaded, this took abt. 2 seconds. Then I loaded epic pads patch, which took 3.5 seconds (the loop still playing).

I repeated the test with Cubase and I crackled horribly. I had the wrong driver so I changed to M-Audio ASIO, CPU 88-95, and a little bit crackling, almost OK, but not. Then I changed (kind of accidentally) to ASIO4ALL and discovered that it gives less crackling and CPU consumption than with M-Audio drivers!! (CPU 72-90 and no crackling) it had 512 buffer in stead of 256 so maybe this is the reason . Nexus load test, 2sec and patch load 3.5-4sec.

Then I did it with FL Studio 9.1, CPU shows 97 and the crackling is absolutely horrible (sounds like AM radio). I repeated this with FL Studio 9.94beta and 10 demo, which has much less crackling but still it crackles like 5-20 times a second. Nexus load 2s, Nexus patch 3s, in beta Nexus load 2s, Nexus patch 4s.

Then I did this with Live 8.22, CPU shows 95, pmem 1213, cmem 1338. In "crackling" performance, Live is clearly better than FL studio, there is some crackles now and then. Nexus loading takes 11s, more than 5 times longer than in the other hosts!, Nexus patch takes 5s, a bit longer than in other hosts.




The next test I did was I first loaded the program, then the project, then another variation of the project and then I closed the program and measured these four times. This is what I got

Live loading time 9-10s, project1 loading 30s, project 2 loading 32s, closing time 5s

FL Studio 1s, 12.5s, 13.5s, 2s

Cubase 11-12s, 13.5s, 14s, 2.5s

Studio One 12s, 12.5s, 13s, 2s

FL studio 10 beta, 1s, 14s, 14s, 2s
******************************************


then I did a fourth test, i deleted every second clip from the original (so 50% of task in average), and tried everything again. With M-Audio driver (and the 256 buffer). Here are results:

Ableton Live CPU 71-82, crackling now and then, still too much for working.

FL studio 10 CPU 97-98, very much crackling

FL studio 9.94beta, pretty much the same

FL studio 9.1 even worse crackling

Studio One, CPU64-77, no crackling at all, or no slowing in UI at all

Cubase 5, CPU66-77, no crackling at all, no slowing in UI at all






I did these tests because I cannot finish my Live tunes and wanted to know to which platform to switch (and also to quantify to Ableton team that there is still something seriously wrong with Live PC). I tought I would share these results with people. Of course the test is not totally scientific, but I tried to be as precise as possible and to repeat in similar way. What can we learn and conclude from this?

1. FL studio has some serious trouble in VST plugin performance, even FL 10, but 9.1 is even much worse. I heard rumors that FL 8 would be much more efficient. FL studio has clearly the fastest startup time (1 second!, versus 9-12s of other hosts). If I had FL 9.1 I would definitely upgrade to 10 or downgrade to 8 (not sure which is a better option).

2. Ableton Live has pretty good VST plugin performance (not as good as Cubase or S1 though) but suffers from other problems. In addition to the memory consuption crashing ( didn't test if 8.22 crashes as much as 8.21) I think even the bigger problem is sluggishness and loading times. 11 seconds of loading a Nexus to a track vs. 2 seconds in every other host!, this is just one example, it feels that EVERYTHING takes 5.5 times longer than normally when your project is crowded. When I delete a track after pressing delete button usually I have to wait 5-10 seconds before the track actually vanishes from UI. I wonder if this is associated to the events.txt logging (which I wrote of to Ableton forum) or is it a normal feature of Live to create such huge log every time a project loads. And of course the project loading time, 30s of Live versus 12.5-14s of other hosts.

3. FL studio uses RAM most efficiently cmem 1271 and after reload 1011, Ableton Live uses RAM least efficiently, cmem 1588 and after reload 1325.

4. Nothing to blame about Cubase or Studio One, they seem to work as a DAW should. Memory performance average.




I tested also Sonar X1 in a different machine. It seemed to me that it might have even better performance than Cubase (I did a similar test), although I cannot confirm that, but there is another interesting product to wait for It would be interesting to know also that does Cubase 6 have better or worse performance from this view.

Remember this is a view of only one person and one machine. If you have a powerful i7 quad core, maybe none of this matters to you. And it might be totally different story in Win7-32, WinXP and Mac.
Or still another story with 64bit host versions.

Share also your experiences (in this topic or another, how you prefer) if you have done similar tests.

[Khazul]

And this tells us what exactly for normal use?

You really run all that lot nomrally?

[golemus2]

And this tells us what exactly for normal use?

You really run all that lot nomrally?

Not simultaneously. I usually have this many plugs, but they are not playing at the same time. Sometimes I do have a lot of stuff playing even simultaneously if I do doublings and have some pad/soundscapes with multiple different layers. But only in some tunes.

[golemus2]

And this tells us what exactly for normal use?

You really run all that lot nomrally?

Not simultaneously. I usually have this many plugs, but they are not playing at the same time. Sometimes I do have a lot of stuff playing even simultaneously if I do doublings and have some pad/soundscapes with multiple different layers. But only in some tunes.

Of course most music styles would not need a lot to happen simultaneously, but Live gets sluggish from the amount of plugs loaded, this sluggishness is there even when the sequencer is paused.

[Fizmarble]

Thank you for running the test and sharing the results. The Live team needs to step up their game now that most of their competitors have passed or surpassed them in features and performance. Even Pro Tools looks to be doing things right on the software side now. Where Live used to innovate, they are now playing catchup. They have clearly spread themselves too thin and been over-ambitious in the recent past. But we will all have to wait for their next offering to see if it's worth jumping ship. I do believe in the Live team though.