I've been buying .wav when possible for years (or buy a CD when I go to a show), I convert it to Apple Lossless. It's worth it for me, I can hear a difference from 128 aac. I can only hear a small difference from 256 aac to .wav, but maybe that could be placebo. Maybe I wouldn't be able to point out the 256 in a blind test, but I'm sure the 128, I could.
I never downloaded a 320 mp3 but it's about the same as a 256 aac. I don't think you'd be missing a whole lot. But once you get above that level & go to .wav I think the main advantage is future-proofing the quality of your music collection. I'd rather spend more on the .wav now then consider repurchasing in a few years as hard drive capacity continues to get cheaper & lossy formats become less necessary. You already don't see as many of the 128 downloads like you used to. Who knows, maybe in 10 years, the consumer standard will be back up to CD quality.
I use lossless compression after I buy the .wav & then store the .wav on a backup drive by themselves. But I'm not sure I really even need to be keeping all those original .wav downloads somewhere off my main HD. Isn't there an easy way to convert Apple Lossless or FLAC file back to a .wav without degrading it?