Question about better beta testing.

corygilbert · Post by **corygilbert** » Fri Nov 16, 2007 1:54 pm

OK, so I didn't want to clog up the beta forum with non-bug talk so....
I'm currently in the beta pool and I'd like to ask some of our more seasoned beta testers how you go about trying to find buggy behavior.
I've been doing everything that I normally do, and attempting to work with features that I don't always use, I've not had any problems that could be bug related, which is good, but I figure if the Abes are so kind as to have us beta test, I'd like to actually try to bust the thing a little.
So, are there any techniques that some of you guys use to try to find buggy behavior?
Thanks,

Angstrom · Post by **Angstrom** » Fri Nov 16, 2007 2:58 pm

those of us that were in what you could call 'very early' beta testing did not have to try hard at all, it was quite easy to spot them. Then we just kept an eye on the bugs that we spotted and checked if they got fixed. The things I did was to go buck wild with all the new stuff, sending the sidechains back on themselves, trying to bend a tension instrument to destruction, dragging multiple selections of chains out of the inline mixer to see if I could freak it out. All of these produced terrible errors or crashes.

Right now though I think Ableton just want you to use the app as you normally would, to see if it crashes in your usage situation.
Although doing it from 'end to end' might be best, so from creation to completion. That's the best method, as you'll soon see if something is not right. It will either be an obvious flaw - such as the broken pitch bend in Tension ... or the app will just close on you in a spectacular crash as it fails to communicate with some specific peripheral or other.

andydes · Post by **andydes** » Fri Nov 16, 2007 3:16 pm

Yeah, I wondered about this well.

Decided the best thing to do was just to make a couple of tracks from start to finish and try to use as many different tools and techniques I could think of. No point in me investigating things I don't use normally (video, for instance). ALthough I might break out some old hardware and check out the external instrument insert.

Think they've got enough people in the beta circle now not to worry about having everyone try everything. Another 400 joining next week I think.

longjohns · Post by **longjohns** » Fri Nov 16, 2007 3:27 pm

I am hoping that people will copy their entire library (or a nice big chunk) so that they can _safely_ File Manage the hell out of it with the beta.

It would be great if we could get the FM in a better state! there are still some problems with the new improvements

JDSampo · Post by **JDSampo** » Fri Nov 16, 2007 3:28 pm

I imagine that just using the application as you normally would on your particular combination of computer, operating system, interface, and controllers is helpful. At this late stage in a beta test most of the real bad bugs have been squashed already. By expanding the beta circle Ableton get their stuff tested on far more combinations of hardware than they ever could in-house.

--JD

longjohns · Post by **longjohns** » Fri Nov 16, 2007 3:31 pm

andydes wrote: Think they've got enough people in the beta circle now not to worry about having everyone try everything. Another 400 joining next week I think.

I must admit I've been relying on this rationalization lately. But this attitude scares me.

I just hope that people do put in effort to trying to test things and don't just view it as a chance to play with the new version early.

It's not really fair to sign up for a beta serial and then figure that "oh, someone else will be testing it enough, since there are so many"

longjohns · Post by **longjohns** » Fri Nov 16, 2007 3:33 pm

Another area which people should be testing extensively is MIDI function and timing issues

Angstrom · Post by **Angstrom** » Fri Nov 16, 2007 3:54 pm

the thing is - people beta testing at this stage should just come down to using the app as they normally would ... and that does return worthwhile results.

It wouldn't be massively valuable for a novice to try and comment on aspects of the application which are unfamiliar to them, it is perfectly fine that everyone uses the tools that they would like to use and in the way they would like to use them

Remember that although a small percentage of the beta testers may be appear to be pea brained beatmonkeys

, they are testing the app from that valuable perspective for that user sub-group.
It would be a bit worthless for them to try experimentally scoring an advert, adjusting clocks and setting up an external SMPTE source, just to see if it breaks.

andydes · Post by **andydes** » Fri Nov 16, 2007 3:55 pm

longjohns wrote:
andydes wrote: Think they've got enough people in the beta circle now not to worry about having everyone try everything. Another 400 joining next week I think.
I must admit I've been relying on this rationalization lately. But this attitude scares me.

I just hope that people do put in effort to trying to test things and don't just view it as a chance to play with the new version early.

It's not really fair to sign up for a beta serial and then figure that "oh, someone else will be testing it enough, since there are so many"

What I mean is concentrate on the parts of the program that I'm most familiar with. I'm still going to throw much more at it than I usually would in writing a track.

Going partcualarly nuts with routing, automation and trying out every instrument and effect I can find. When I'm through making this cacophoney of sound, I'll try all the editing and performance functions out.

But I'm not going to do more than a quick check on something like DJing MP3s as there's other people with way more knowledge who'll be looking into this above all else.

Anyway, I can't feel that guilty, this way Ableton have secured one more sale of the suite, which I may or may not have gone for otherwise. They loose nothing by running a public beta test, but we should do our part to ensure a solid release.

andydes · Post by **andydes** » Fri Nov 16, 2007 4:07 pm

Angstrom wrote: Remember that although a small percentage of the beta testers may be appear to be pea brained beatmonkeys , they are testing the app from that valuable perspective for that user sub-group.

I'll have you know I'm very knowledgeable in fields that have absolutely nothing to do with digital audio.

longjohns · Post by **longjohns** » Fri Nov 16, 2007 4:55 pm

My statement had much more to do with my own guilt, rather than pointing the finger at anyone. Now with more people on board I have definitely let myself slack off and attend to other areas of life

I understand that someone may not be able to comment on features which they are not all that familiar with.

But surely we should all make some effort to explore a bit more than usual and push it a bit more than usual.

At least taking a tour of the new feature list and giving those a try..

Angstrom · Post by **Angstrom** » Fri Nov 16, 2007 5:10 pm

there's loads of areas where I would be useless in testing. Midi timing would be one of them because all my outboard gear is in storage.

andydes · Post by **andydes** » Fri Nov 16, 2007 5:16 pm

Likewise, my response was mainly due to my own sense of guilt. I'll feel more justified when I've found a new bug that's not down to operator error. By operator, I mean me, not the plug in. Only just signed up though.

Anyway it's friday night, and I'm not going drinking afterwork. No one can ask for a bigger sacrifice, surely.

peeddrroo · Post by **peeddrroo** » Fri Nov 16, 2007 5:18 pm

yeah, i'd say using as you would normally do is the best testing.
but then i know that i always try -and discover- new stuff while testing. that's why i like it.
i even discovered "new" features that were already in previous versions (such as the ability to delete an automation directly on the plugin)

one area that i'm a bit scared of is the time signature thing. i'm not sure that many ppl test it. would be nice if the ppl who requested it badly could have a go.

Angstrom · Post by **Angstrom** » Fri Nov 16, 2007 5:58 pm

peeddrroo wrote: one area that i'm a bit scared of is the time signature thing. i'm not sure that many ppl test it. would be nice if the ppl who requested it badly could have a go.

I've chased a few bugs in that area, mainly the one where the automation that you send from Session disappeared after recording it into Arrangement. That one is fixed now.
there's not that much to test really. I mainly just drop a marker onto arrangement with the appropriate sig, presto change-o. (I do most of my detailed work in arrangement).
Anyway, I think it's quite robust from what I can tell.