audio to midi realtime patch - possible & feasible with m4l?

[stefan-tiedje]

(I know FFT takes time but we're running very powerful computers these days)

The latency of FFT is completely independent of the power of your computer, its dependent on your frequency resolution you need within your process. All professional plug-ins have to fight the same problem and if they are fft based have the same latency.
A more powerful computer can run more of these processes, but the latency would always be the same...

Pitch detection is a special problem, as you'd typically need several complete cycles to find the correct pitch. The lower your string, the longer it takes. (makes it difficult to calculate the latency, as it depends on the actual pitch)
The pitch detectors built into the the Axon kind of processors are faster, because they use some knowledge about the waveform of the attack of guitar strings, they only work with guitars. And as its a patented technology, there are no equivalent solutions as MSP externals. You get them cheap enough to just place them into your guitar rig though...

Stefan

[Pasha]

(I know FFT takes time but we're running very powerful computers these days)

The latency of FFT is completely independent of the power of your computer, its dependent on your frequency resolution you need within your process. All professional plug-ins have to fight the same problem and if they are fft based have the same latency.
A more powerful computer can run more of these processes, but the latency would always be the same...

Pitch detection is a special problem, as you'd typically need several complete cycles to find the correct pitch. The lower your string, the longer it takes. (makes it difficult to calculate the latency, as it depends on the actual pitch)
The pitch detectors built into the the Axon kind of processors are faster, because they use some knowledge about the waveform of the attack of guitar strings, they only work with guitars. And as its a patented technology, there are no equivalent solutions as MSP externals. You get them cheap enough to just place them into your guitar rig though...

Stefan

Hi Stefan,
Thanks for the fix (can't wait to get home...) and for this explanation.
What I do not get however is that, Zebrify (Urs Heckerman - U-he)
has to get pitch / transient detection and it's incredibly fast.
For what is worth (me being a noobie...) I have read that Roland GR300 considered the faster
Gtr Synth ever, was using Zero Crossing to detect pitch. I guess this can be done in MSP as well
if you're a master...What do you think? I like to learn new things...

- Best
- Pasha

PS : speculation tells that AXONS use FFT for G to higher E string while Zero Crossing for Lower E and A.

[stringtapper]

The pitch detectors built into the the Axon kind of processors are faster, because they use some knowledge about the waveform of the attack of guitar strings, they only work with guitars.

My Chapman Stick disagrees.

[stringtapper]

What I do not get however is that, Zebrify (Urs Heckerman - U-he)
has to get pitch / transient detection and it's incredibly fast.

The comparison here is not MSP vs. Zebrify, but rather Pasha vs. Urs, I think.

[stefan-tiedje]

What I do not get however is that, Zebrify (Urs Heckerman - U-he)
has to get pitch / transient detection and it's incredibly fast.
For what is worth (me being a noobie...) I have read that Roland GR300 considered the faster
Gtr Synth ever, was using Zero Crossing to detect pitch. I guess this can be done in MSP as well
if you're a master...What do you think? I like to learn new things...
PS : speculation tells that AXONS use FFT for G to higher E string while Zero Crossing for Lower E and A.

Zero crossing would also require at least two of them, which would add to a whole cycle, but Axon is faster, its happy with a quarter cycle, because it knows the waveform of a guitar.
With so called guitar synths you have another option, instead of normal oscillators, you can use the original sound of the string, distort it, to get higher harmonics, and add envelopes and filters - then there is no pitch detection necessary. I think at least some of the Rolands do just that.
You could also try to get the attack of the sound, start with an arbitrary pitch (for example the last detected one) and then just correct it to the later detected pitch, that would at least get you a shorter latency rhythmically, and for lower pitches our ears also need more time to detect the pitch correctly...

Stefan

[Pasha]

What I do not get however is that, Zebrify (Urs Heckerman - U-he)
has to get pitch / transient detection and it's incredibly fast.
For what is worth (me being a noobie...) I have read that Roland GR300 considered the faster
Gtr Synth ever, was using Zero Crossing to detect pitch. I guess this can be done in MSP as well
if you're a master...What do you think? I like to learn new things...
PS : speculation tells that AXONS use FFT for G to higher E string while Zero Crossing for Lower E and A.

Zero crossing would also require at least two of them, which would add to a whole cycle, but Axon is faster, its happy with a quarter cycle, because it knows the waveform of a guitar.
With so called guitar synths you have another option, instead of normal oscillators, you can use the original sound of the string, distort it, to get higher harmonics, and add envelopes and filters - then there is no pitch detection necessary. I think at least some of the Rolands do just that.
You could also try to get the attack of the sound, start with an arbitrary pitch (for example the last detected one) and then just correct it to the later detected pitch, that would at least get you a shorter latency rhythmically, and for lower pitches our ears also need more time to detect the pitch correctly...

Stefan

Thanks for reply.
The synthesis you're after It's called HRM (Harmonic Restructure Modeling) and it's inside VG99 unit... however the process as you describe it was inside the GR300 unit and I have used that to inspire the synth sound I made with my guitar and Zebrify, no pitch 2 MIDI needed. However, my guess is that operating on harmonics needs an FFT or a shorter version to manipulate harmonic content. However I still believe that MSP is better for educational and avant-gard proof of concepts than an usable real-time output. I have tested all the possible devices into M4L and all, even the simplest have latency (and somewhat pops & clicks) where Live FX does not and Audio is clean. By judging about the ratio between MIDI devices and Audio Devices you can find around I am tempted to say that Max is great for MIDI (or people do not have the skills for MSP) and not really there for real time audio.
What do you think about my last statement?

- Best
- Pasha

[pid]

By judging about the ratio between MIDI devices and Audio Devices you can find around I am tempted to say that Max is great for MIDI (or people do not have the skills for MSP) and not really there for real time audio

again, it really depends what you use it for. for some people, msp is the only thing that can do what they want. the midi/audio thing is simply indicative of boring/interesting and easy/hard factors that are always at play in these situations.

i think your points are all valid though, definitely. for me though these issues rarely define why i use one software or another. also, fft type work etc is a real leveller; try doing the same thing in pd, csound, rtcmix, chuck, hell even bidule; same issues in all of them.

best

[Pasha]

By judging about the ratio between MIDI devices and Audio Devices you can find around I am tempted to say that Max is great for MIDI (or people do not have the skills for MSP) and not really there for real time audio

again, it really depends what you use it for. for some people, msp is the only thing that can do what they want. the midi/audio thing is simply indicative of boring/interesting and easy/hard factors that are always at play in these situations.

i think your points are all valid though, definitely. for me though these issues rarely define why i use one software or another. also, fft type work etc is a real leveler; try doing the same thing in pd, csound, rtcmix, chuck, hell even bidule; same issues in all of them.

best

Thanks for reply. I liked the FFT as a leveler. Is really true unless you use some trick. Roland, GR4, Electro Harmonix POG/HOG have FFT inside for harmonics and pitch detection, and I think they have patented ways of doing so in a different way. We can emulate something in MSP but not everything...However I'm curious about what IRCAMAX might have in the news for NAMM... Those things IMHO need to the written by using low level code not an interpreted one. Too much level of abstraction and you're done.. latency arises.

- Best
- Pasha

[stefan-tiedje]

By judging about the ratio between MIDI devices and Audio Devices you can find around I am tempted to say that Max is great for MIDI (or people do not have the skills for MSP) and not really there for real time audio.
What do you think about my last statement?

I am biased, I am using Max for almost twenty years, and MSP since it was published the first time. 'til the advent of Max4Live it was my sole tool for real time audio...
If you experience specific latency, this might be something to discuss with the creators of Live and M4L. But then you should setup very specific comparisons. Compare only similar effects. As I explained already, pitch detection is a special field, which could show the differences you experience.
But if you get into MSP, you'll also learn possibilities how to tweak performance issues. There are things like vector sizes, input buffers and alike, which influence your available processing power and your latency. If you need better latency, you will get less power and vice versa. In Max/MSP you can control them to a much higher degree than in Live though.
To find out what causes specific performance differences, needs a good deal of analysis. How different are the processes itself etc.
I am quite happy with what I have. But if you are a Live nerd, you can do some tests yourself.
To find out latency differences, you could measure the latency with an older Live version and the Pluggo Junior collection of VST effects, they come from Cycling and are free. These effects should also be in the M4L collection of devices which come with M4L.
I doubt that the underling engines are much different, but I don't know for sure if there are different vector sizes for example.
Personally I don't care about latencies in the range of the delay between amp and my ear caused by the speed of sound. Anything between 5 and 10 ms is fine, I can adapt my playing to it easily. Just imagine with how much latency an organ player in the church is faced usually
For straight sequencing, latency can be compensated and is no issue, Unless you have effects which have varying latencies. But this has nothing to do with the different device technologies...

Stefan

[stefan-tiedje]

However I'm curious about what IRCAMAX might have in the news for NAMM... Those things IMHO need to the written by using low level code not an interpreted one. Too much level of abstraction and you're done.. latency arises.

I know the guys, if they give it away for 99,-, its just what I would do, use their max externals and place them into M4L devices. Its just that they do the work for you.
They have a bunch of pretty exciting collections, like physical models (modalys), the complete FTM library (with its own learning curve), a super phase vocoder...
The Midi bus seems the most straight forward and easy to achieve device... The first thing I needed when starting with M4L, so I made something like that myself...
The core of the Jimmies is a bit like all these examples which come with Max/MSP btw...
Except for the Jimmies, these collection have special externals, which are coded low level, but this is also true for the externals which come with Max...
Of course there is some overhead within a patching system like Max, but this has nothing to do with expected latency... If you do the same process in MSP and in a dedicated low level language, you might be limited to open fewer instances in the Max/MSP version. But the latency and the sound would be the same...

Stefan

[elt.net]

There is a m4l patch doin audio to midi

http://www.maxforlive.com/library/device.php?id=97

uses the external fiddle~ to analyze the audio, only available currently for MAC, im on WIN.

Hello, it's updated for PC (using pitch~ instead fiddle~).

[luzil]

BIG THX to all for the valuable infos & patches

@elt: works nicely, thx, dont forget to install fftw3.dll to max folder if anybody want to test this on windows
http://web.media.mit.edu/~tristan/MaxMSP/fftw3.dll.zip

on latency:

i tested now several audio-to-midi, audio-to-synth vst plugins available on kvr, free & commercial ones, tested the latency of some with vst plugin analyzer http://www.savioursofsoul.de/Christian/ ... -programs/

The Quality of Pitch detection is pretty variating on Plugin/Plugin itselfs detection parameters (transient, pitch, feedback noise, gate...)

Best results i had with midifier http://www.knzaudio.com/index.php which even tracks pitch bending automation in ur sequencer , not single midi note on/off + pitch. Latency around 50ms. Other audio-to-midi plugins didnt show significant lower latency

Contrary audio-to-synth plugins like already mentioned zebrify or several FREE vst on http://rekkerd.org/fretted-synth/ show also pretty good results tracking pitch while showing latency of 10ms.

Question is were this huge difference comes up and also if i want to compare latency of a m4l plugin with vst, how to do this objectively. Using an short clicking audio sample and render it to arrangment view, can i disable delay/latency compensation in Live between tracks so i can see when zooming in in arrangement view what latency ms distance between the source clicking sample on diff tracks it creates? Im unsure if 40ms are due to routing of the midi stream to the receicing vsti (latency of a normal synth is around 10ms, if vst plugin analyzer is correct) or if this has to do with the discretisation of the pitch information to a midi signal format. But probably this has to be tested further how exact audio-to-synth pitch detection here really is compared to e.g. midifier. I dont think this can be only contributed to URS u-he

Is it from latency pov probably better to code a audio to midi patch as max standalone patch (eventually more performance parameter avaialable) and rewire the midi stream directy on a LIVE midi recording track? Any experience here?

Conclusion

from all this is to first code a audio-to-synth m4l patch, which seems easy to do with example patches here and pitch~ and look up which latency this causes, seems also u can change fft parameter in pitch~ external, so see how this will effect pitch detection quality depending on audio input type. Especially if u want to live trigger a synth with ur voice (variates strongly on person and summing, whistling, beatboxing, absolut pitch...) a single super duper pitch detection algorithm doesnt seem possible and feasible. The many detection parameters on the tested vst and variating quality imply this also. Guitar & human voice a too diff. to put a cap about both So probably u can save up a lot of additional latency optimizing the code to its final purpose (incoming audio, pitch range/accuracy)

Generally i dont think u get worse latency with MSP compared to Reaktor or Synthedit which show alot of realtime audio fx strangley, maybe the max community just needs more time or Max MIDI capabilities are anyway higher than in above SDK so the community focused more on this part. Probably IRCAM wouldnt also develop cpu heavy physical modeling on it as stefan wrote if there where fundamental latency concerns. But then u have too chose the tricky low level programming external path, so for non academic single user this is not feasible mostly

Further thoughts welcome

[Pasha]

What I do not get however is that, Zebrify (Urs Heckerman - U-he)
has to get pitch / transient detection and it's incredibly fast.

The comparison here is not MSP vs. Zebrify, but rather Pasha vs. Urs, I think.

I love Urs! It's the nicest guy and synth/FX programmer in the world! What Urs does is simply my
baseline for comparison!

[Pasha]

BIG THX to all for the valuable infos & patches

@elt: works nicely, thx, dont forget to install fftw3.dll to max folder if anybody want to test this on windows

on latency:

i tested now several audio-to-midi, audio-to-synth vst plugins available on kvr, free & commercial ones, tested the latency of some with vst plugin analyzer http://www.savioursofsoul.de/Christian/ ... -programs/

The Quality of Pitch detection is pretty variating on Plugin/Plugin itselfs detection parameters (transient, pitch, feedback noise, gate...)

Best results i had with midifier http://www.knzaudio.com/index.php which even tracks pitch bending automation in ur sequencer , not single midi note on/off + pitch. Latency around 50ms. Other audio-to-midi plugins didnt show significant lower latency

Contrary audio-to-synth plugins like already mentioned zebrify or several FREE vst on http://rekkerd.org/fretted-synth/ show also pretty good results tracking pitch while showing latency of 10ms.

Question is were this huge difference comes up and also if i want to compare latency of a m4l plugin with vst, how to do this objectively. Using an short clicking audio sample and render it to arrangment view, can i disable delay/latency compensation in Live between tracks so i can see when zooming in in arrangement view what latency ms distance between the source clicking sample on diff tracks it creates? Im unsure if 40ms are due to routing of the midi stream to the receicing vsti (latency of a normal synth is around 10ms, if vst plugin analyzer is correct) or if this has to do with the discretisation of the pitch information to a midi signal format. But probably this has to be tested further how exact audio-to-synth pitch detection here really is compared to e.g. midifier. I dont think this can be only contributed to URS u-he

Is it from latency pov probably better to code a audio to midi patch as max standalone patch (eventually more performance parameter avaialable) and rewire the midi stream directy on a LIVE midi recording track? Any experience here?

Conclusion

from all this is to first code a audio-to-synth m4l patch, which seems easy to do with example patches here and pitch~ and look up which latency this causes, seems also u can change fft parameter in pitch~ external, so see how this will effect pitch detection quality depending on audio input type. Especially if u want to live trigger a synth with ur voice (variates strongly on person and summing, whistling, beatboxing, absolut pitch...) a single super duper pitch detection algorithm doesnt seem possible and feasible. The many detection parameters on the tested vst and variating quality imply this also. Guitar & human voice a too diff. to put a cap about both So probably u can save up a lot of additional latency optimizing the code to its final purpose (incoming audio, pitch range/accuracy)

Generally i dont think u get worse latency with MSP compared to Reaktor or Synthedit which show alot of realtime audio fx strangley, maybe the max community just needs more time or Max MIDI capabilities are anyway higher than in above SDK so the community focused more on this part. Probably IRCAM wouldnt also develop cpu heavy physical modeling on it as stefan wrote if there where fundamental latency concerns. But then u have too chose the tricky low level programming external path, so for non academic single user this is not feasible mostly

Further thoughts welcome

Thank for the summary!
Just wanted to add a little bit just to learn more.
In my experience Zebrify works in three different ways:

1) is very simple and does not use any audio to midi as far as I can tell. Simply get a sound and make it pass through two different VCF and then XMF use one input to drive the XMF via sidechain. With that I have easily created a Brassy synth sound with no latency ot compromises. However that fact that it's monophonic raises some suspect in me..

2) Here you need Audio to MIDI. The structure is the same as before but an OSC or FM OSC is used as the sidechain of the XMF this alters the harmonic structure of your incoming signal that feeds the XMS directly. I also have added some COMB filter in the main path. This works great. (remember to mute the column containing the OSC or FM OSC)

3) Don't like this too much for it works like a synth that produces its note continuosly like a doepfer.
Simply put an OSC on the grid and play.

- Best
- Pasha

[luzil]

@pasha

here is a preset i optimized several times to my voice to trigger a osc like with audio to midi

http://www.mediafire.com/?2v47xix7dvja82r

Tune & Vol of the osc is modulated by pitchtrack and transient in the osc control window. For live use i dont like this too much, as zebrify peaks with small feedback permanently. It seems this have to be setup best for individual voice, feedback noise, mic type, similar to other audio-to-midi plugins. Im also unsure what zebrify really does here, but as in my preset tune & vol is modulation controlled, there has to be some discretisized data which the pitch tracker creates, also u can range the highest and lowest note.

Ur 1) example looks a bit weird to me, but im not deep in zebra, dont know exakt diff. between xmf and vcf. But if u want to trigger a synth with ur voice would u do it my way or do i overlook here something. Would be great if u simply can load a zebra synth preset in zebrify and then modulate all 4 routing lines by pitchtrack & transient. Seems now i would have to make all synth presets again from scratch.