1. Forget the whole "Mixes"-tab in CueMix. Not needed here (though probably could be used too, which would bring in another level of complexity…).
2. Open "MOTU Audio Setup" from your Applications folder. Set "Return Assign" to whatever MOTU output is used as your Mac output (VLC, Safari etc.) / Default Stereo Output in "MOTU Audio Setup" (Analog 1-2 here).
3. Set your Live recording channels "Audio To" to some unused outputs ("Ext Out 3/4" here) - Set the inputs "Audio From" to "Ext In 13/14" (13/14 = MOTU-Stereo Return 1-2 bus.). (11/12 = MOTU Reverb return bus).Warning: the Return inputs can cause feedback loops! DO NOT assign this input to a track that shares the same 828mk3 output pair as the returns. (so in this example NOT Ext Out 1/2, but e.g 3/4 in Live!!!
edit: looks like 11/12 & 13/14 assignments are swapped in the ultralight mk3 - check manual
(You could now add some of the MOTU CueMix DSP effects (EQ/Dynamics/Reverb, if wanted…)
I just tried it with ITunes & had to lower it's output a bit (clipping in Live), though no Enhancer, or EQ was active in iTunes, nor DSP stuff was activated in CueMix. Strange… ???
From the 828mk3 manual:
Return Assign
The Return Assign menu lets you choose any pair of 828mk3 audio outputs. The audio signal from this output pair is then sent back to the computer via the Stereo Return 1-2 bus. This stereo return bus from the 828mk3 appears in your host software alongside all other 828mk3 inputs, wherever your host software lists them.
The 828mk3 stereo return bus can be used for a variety of purposes. For example, you could use it to send a final mix being played through the 828mk3 back to the computer, where you could record it for mastering or archiving purposes.
As another example, you could use the stereo return bus to capture tracks played from your host software, along with live inputs being routed directly through the 828mk3 hardware via CueMix FX (with or without CueMix effects processing on the live inputs).
@dazzer: Dunno about the pre mk3s, but maybe you also have to set it up in MOTU "Audio Setup" (Applications folder) -> "Return Assign"?
Would be helpful to know supamonsta's "MOTU Audio Setup" settings.