I don't have a mac,, but from friends who have etc, you should be able to run at least 8 tracks with loads of vst and vst instruments without any problems - for example..
My PC with old Core Duo (i.e not core 2 duo) 1.6Ghz processors and 2 GB RAM runs at about 4% when idle, and rarely exceeds 60% for what I do i.e.up to 16 tracks two or three vst's on each, and 5 channels of sends and returns. (and mac and PC preformance can't be that different)
has the operating system recognised the ram correctly ? (go to 'about this mac' and check) what soundcard are you using ? this is often the cause ? what latency setting ? what sample rate and what plug-in buffer size (all found in the preferences dialogue) what 'test scenario' are you using ? have you got the dual processor switch in the CPU preferences checked ? could be a firewire bus problem - what devices do you have connnected ? there are so many variables you'll have to give a bit more information to get any useful feedback - have you tried using the search function for the soundcard you're using to see if others have probems ?
Live wouldn't have got to where it is as a DAW if it couldn't run on a computer as well specced as yours ?? sounds like there is another problem. do you have any other programmes running at the same time ?