Memory Hell

[inquiry]

So what's the conclusion here? Is this definitely a Leopard/Intel-specific problem with Logic/Leopard/Intel not paging VM in and out properly -- whereas Logic/Tiger/PPC is doing so with exactly the same song?

 

Does it still occur with only 4GB installed on the Leopard machine? (I'd really try a complete wiping of the hard drive, reinstall of just OSX and Logic and the 3rd party plug-ins, with four matching 1GB sticks of Kingston premium RAM -- to eliminate any variables like OS corruption, sub-quality RAM.)

 

Does it ever occur with just Apple plug-ins -- like could you load enough Ultrabeat instances to make it happen on the Leopard machine, while it still isn't happening on the Tiger machine?

 

In general, I wonder why Logic doesn't have a third "RAM usage meter" alongside the CPU and HD meters, showing us when we're loading too much into RAM.

[Kent Sandvik]

So what's the conclusion here? Is this definitely a Leopard/Intel-specific problem with Logic/Leopard/Intel not paging VM in and out properly -- whereas Logic/Tiger/PPC is doing so with exactly the same song?

 

Does it still occur with only 4GB installed on the Leopard machine? (I'd really try a complete wiping of the hard drive, reinstall of just OSX and Logic and the 3rd party plug-ins, with four matching 1GB sticks of Kingston premium RAM -- to eliminate any variables like OS corruption, sub-quality RAM.)

 

Does it ever occur with just Apple plug-ins -- like could you load enough Ultrabeat instances to make it happen on the Leopard machine, while it still isn't happening on the Tiger machine?

 

In general, I wonder why Logic doesn't have a third "RAM usage meter" alongside the CPU and HD meters, showing us when we're loading too much into RAM.

 

The activity monitor shows memory usage, select Logic and you see usage. Even if you hit the theoretical 4Gb you should still get things (barely working), depending on the page hit traffic.

 

Kontakt also has a memory usage meter.

[stevenson]

the conclusion?

 

1. 32bit logic does not have enough internal memory to run even fairly modest 3rd party libraries/plug-ins.

 

2. therefore logic would have to become 64bit which is NOT going to happen for many VERY good reasons, or else the 3rd party plug-ins will have to use the same mmap trick as the exs24.

 

3. the mmap functionality requires a certain amount of internal logic memory to make work. this severely curtails the amount of memory you have for your 3rd party stuff, which is why reducing your physical RAM could actually increase how much you have available for logic. the amount of memory logic sets aside to run mmap is set in your plist file. even a fairly low level programmer could adjust this. i have had mine adjusted and it has made a huge difference.

 

4. the only real solution for those who are working with a fair amount of 3rd party stuff is to use an AU host. this really is not the end of the world and actually brings its own benefits. in this case a 32bit au host has its own 4 Gb memory space which makes far better use of your systems RAM.

 

5. logic is much more stable close to its memory limits than it ever has been before, but it reaches the limit far sooner because of the 512 Mb set aside to run mmap. i stress, if you are running close to the memory limit you need to consider an AU host.

 

6. it absolutely has nothing to do with computer model, intel vs PPC, or whether your RAM is paired, coloured purple or has furry dice hanging off of it. it is all to do with the internal RAM address space limitation with logic.

 

some further thoughts:

 

- generally if you get the unable to save message, it actually does save anyway. look for the file with a tilda ( ~ ) after it.

 

- there are many features in logic which have not been thought all the way through; file management, communication with other DAWs, beat mapping, working with video, the library drawer, score editing, and finally mmap.

 

mmap was clearly meant to address the problem of working with large libraries and is partially a good solution. but the reality is, most modern sound libraries are only available for 3rd parties plug-ins and not available for the exs. missing this point makes the mmap virtually a redundant feature - how can you miss it? simply saying its up to the 3rd party manufacturers to sort out is not helpful to the poor user who is wondering why their expensive macpro with a gazillion gigabytes of RAM falls over sooner and runs no faster than their old coal-powered G5.

 

having spoken to very smart programmers not related to logic development (that i know of) there are a myriad ways of exploiting RAM on your system and certain advantages at least in the short term to remaining a 32bit application. logic developers have not thought all the way through the problem of running large libraries and this annoys me somewhat. an integrated secondary hosting program such as RAX or plogue bidule, created by the logic developers would be much easier and more efficient to manage, more tightly integrated and would make a lot more sense than an obscure programming trick that they didn't even announce when it came out.

 

a big problem of running combined libraries is that when switching songs, the 3rd party samples have to be reloaded. keeping them in plogue bidule running in rewire, means that switching songs takes a fraction of the time it used. the technology exists for logic to run a secondary application very easily and would have been a far more practical and obvious solution than the one they came up with, because the situation as we have it means that people get burned all the time as they come against the memory wall.

[inquiry]

OK, so you want 64-bit computing now, and I want 20GHz processors now (and 40GHz in 18 months). And a flying car. You'll be getting yours sooner with Snow Leopard (OSX 10.6) and all Apple apps getting ported to 64-bit in the next version... or two ; )

 

For revolucian, has just lowering his physical RAM to 4GB solved his problem of opening (and reliably using) his Logic projects from his Tiger machine? You seem to be saying his problems are all due to mmap using an additional 500MB, but does he even have any EXS instances loaded in these projects? Was his Tiger machine even 4GB RAM?

 

BTW #1: if the 3rd party samplers are streaming their samples, why is this even an issue??

 

BTW #2: is mmap 500MB per instance of EXS? And what's EXS's memory footprint normally (just the plug-in)?

[stevenson]

For revolucian, has just lowering his physical RAM to 4GB solved his problem of opening (and reliably using) his Logic projects from his Tiger machine? You seem to be saying his problems are all due to mmap using an additional 500MB, but does he even have any EXS instances loaded in these projects? Was his Tiger machine even 4GB RAM?

 

i don't know. i am partially joking about lowering RAM. you could do the same thing by turning virtual memory off in exs preferences. however, LP8s RAM footprint is much higher than 7 and it does not seem to be quite as efficient at memory management as 7, although at high RAM usages it seems to be more stable. ie you can keep LP8 flying for much longer with big projects than you could with 7.

 

BTW #1: if the 3rd party samplers are streaming their samples, why is this even an issue??

 

the samples have to be mapped and sample headers loaded into RAM. all apple loops occupy RAM and the block of internal RAM a single instance from a 3rd party can occupy quite a lot of RAM. also, because the RAM has to be contiguous, you may run into memory problems even if it seems you have enough memory. imagine an SUV parked inconsiderately. there would be room 3 cars but only 1 big one will fit.

 

BTW #2: is mmap 500MB per instance of EXS? And what's EXS's memory footprint normally (just the plug-in)?

 

no, the 500 Mb is to run mmap - to allow for the external mapping. its not per instance, and i believe (although do not know this for sure but i am 99% certain) even if you do not have any exs instruments loaded but you have virtual memory enabled in logic prefs it will still be on and sucking up that memory.

[ski]

keeping them in plogue bidule running in rewire...

 

Are there any advantages/disadvantages to running plogue bidule via rewire as opposed to IAC?

[Fred B]

keeping them in plogue bidule running in rewire...

 

Are there any advantages/disadvantages to running plogue bidule via rewire as opposed to IAC?

 

ReWire is definitely more powerful than IAC as it delivers audio back into Logic (without latency).

Accordingly, the setup of ReWire is somewhat more complicated.

[ski]

Thanks Fred.

 

I did a little reading up on using ReWire and Logic just now, and it says that it's necessary to create Auxes to return audio to Logic from plugs/standalones that are ReWired. Other than that are there any other complications to know about?

[Fred B]

I did a little reading up on using ReWire and Logic just now, and it says that it's necessary to create Auxes to return audio to Logic from plugs/standalones that are ReWired. Other than that are there any other complications to know about?

Not directly with Logic. However, since the ReWire engine is shared by all audio applications installed on the system there may happen interference from bad implementations. For example, I've once read about issues caused by Melodyne.

[stevenson]

i'll post a screen shot tomorrow, but basically the problem with bidule is that the midi channels don't show volume info. also, and it might be because i don't fully understand it, i have rewire bidules, and midi bidules and they are slightly different. i haven't got my head round it and it might be worth starting a thread on plogue because we are all going to have to get to grips with it if we want to use big libraries (either 8 or 7 - it will always remain an issue).

 

plogue bidule is kind of like logics environment only even more flexible - which also means it is kind of also a little more confusing initially. i would love to have an integrated solution with logic because AU hosting with rewire is definitely the way to go. what a difference it has made to logics performance in 8! its like a whole new computer....

[inquiry]

you could do the same thing by turning virtual memory off in exs preferences.

Have you tested that?

but you have virtual memory enabled in logic prefs it will still be on

You mean in EXS prefs...?

[inquiry]

Well all this is leaving me thinking:

 

- Stevenson is complaining about Logic a bit too much : ) Are there any 64-bit Mac or Win sequencers in existence at the moment? Would you actually be able to use them without 64-bit 3rd party plug-ins, and do those exist yet? (No, no.) Are Pro Tools LE, Performer, Ableton, Cubase, handling 32-bit memory any better than Logic? For 32-bit XP and Vista, you can't even install (and use) more than 3GB physical RAM (or less depending on the video card) on the machine.

 

- The 3rd party plug-in developers are causing a lot of the problem with bloated plug-ins.

 

- Users need to consider plucking the one sample they need out of 5 different sample instruments, and loading those in 1 sampler instance instead of using 5 sample instances loaded with 20GBs of samples they don't need. Perhaps a week with an early 80's AKAI hardware sampler with 2MB RAM (the deluxe model) would teach them some sample frugality.

 

- EXS should let you turn off mmap independently of the general virtual memory preference.

 

- Logic 8 should definitely have better RAM warning and error handling (keep you apprised of RAM usage, disable plug-ins when RAM usage goes over 4GB instead of crashing!).

 

- I'm still wondering though if the crashing (rather than an appropriate error message) occurs when reaching 4GB RAM even with the G5, as the memory addressing system is totally different than the (various) Mac Intel memory systems. And if the crashing happens when you go over the 4GB (even on Intel) if using only native Logic plug-ins.

 

- From what I'm reading an autumn release of 64-bit Logic 9 at the same time as "Snow Leopard" (OSX 10.6) sounds possible. (And... we wonder if they'll have been devoting all their time to recoding in Cocoa for 64-bit instead of implementing bug fixes and new features.) But then you'll still have to wait 2 years for all the 3rd party plug-ins to get ported to 64-bit so you can use them with 64-bit Logic.

[Fred B]

..but basically the problem with bidule is that the midi channels don't show volume info..

Ouch! I always thought that ReWire MIDI tracks would behave like ordinary external MIDI tracks. But apparently that's not true, at least in Logic. So it seems I have to revise my former comment about complications..

[stevenson]

- Stevenson is complaining about Logic a bit too much : ) Are there any 64-bit Mac or Win sequencers in existence at the moment?

 

yes cubase 5 and sonar. only with vista though.

 

 

Would you actually be able to use them without 64-bit 3rd party plug-ins, and do those exist yet? (No, no.

)

 

err.. yes yes, although i am skeptical they will be much chop. they have wrappers that allow you to use 3rd party plug-ins that are still 32bit within the 64bit program.

 

- The 3rd party plug-in developers are causing a lot of the problem with bloated plug-ins.

 

oh for sure. and logics plugs are vastly more efficient in every respect. the truth of that doesn't change the situation.

 

- Users need to consider plucking the one sample they need out of 5 different sample instruments, and loading those in 1 sampler instance instead of using 5 sample instances loaded with 20GBs of samples they don't need. Perhaps a week with an early 80's AKAI hardware sampler with 2MB RAM (the deluxe model) would teach them some sample frugality.

 

hmmm....i do that a bit or commit to audio, but its not always practical. the great thing about the new sample libraries are the range of articulations or velocity layers. take ivory. it 'only' has up to 10 velocity layers per note running for the full length of the sustained note (no loops). i still feel it could do with better resolution than 10 velocity layers.

 

round robin samples for certain articulations for eg arco strings, mean you have 2 sets of samples per velocity layer. if you want the sophistication and detail that can really bring a score alive then you need the samples. factor in ALL the articulations and the detail you would like to have on call as you develop your score and i am afraid you are going to need heaps of samples - and room to put them.

 

- EXS should let you turn off mmap independently of the general virtual memory preference.

 

well, that's not how it works.

 

- I'm still wondering though if the crashing (rather than an appropriate error message) occurs when reaching 4GB RAM even with the G5, as the memory addressing system is totally different than the (various) Mac Intel memory systems. And if the crashing happens when you go over the 4GB (even on Intel) if using only native Logic plug-ins.

 

logic plug ins a much more efficient but you could definitely get logic to crash on any system ppc or intel if you go over 4 GB. regardless of platform you just cannot do it.

 

- From what I'm reading an autumn release of 64-bit Logic 9 at the same time as "Snow Leopard" (OSX 10.6) sounds possible. (And... we wonder if they'll have been devoting all their time to recoding in Cocoa for 64-bit instead of implementing bug fixes and new features.) But then you'll still have to wait 2 years for all the 3rd party plug-ins to get ported to 64-bit so you can use them with 64-bit Logic.

 

i will bet my life logic will not go 64bit. nor particulalry do i want it to. i just want to see a better solution - a more global solution - for the memory limit in logic.

[Fred B]

..but basically the problem with bidule is that the midi channels don't show volume info..

Ouch! I always thought that ReWire MIDI tracks would behave like ordinary external MIDI tracks. But apparently that's not true, at least in Logic. So it seems I have to revise my former comment about complications..

I have just checked using ReWire in Cubase instead of Logic. And there it works as expected: The ReWired device is controlled by normal MIDI tracks and can be used just as if it was a stand-alone MIDI hardware sound module.

 

But Logic is different, unfortunately.

[stevenson]

I have just checked using ReWire in Cubase instead of Logic. And there it works as expected: The ReWired device is controlled by normal MIDI tracks and can be used just as if it was a stand-alone MIDI hardware sound module.

 

But Logic is different, unfortunately.

 

no it does the same thing, but the rewire bidule does not have a volume fader appearing in the cs in logic. you can still automate it. its not the end of the world./ you could put a single channel before it if you wanted to.

[Fred B]

I have just checked using ReWire in Cubase instead of Logic. And there it works as expected: The ReWired device is controlled by normal MIDI tracks and can be used just as if it was a stand-alone MIDI hardware sound module.

 

But Logic is different, unfortunately.

 

no it does the same thing, but the rewire bidule does not have a volume fader appearing in the cs in logic. you can still automate it. its not the end of the world./ you could put a single channel before it if you wanted to.

Ok, I see. Thanks for clarifying.

[inquiry]

- Users need to consider plucking the one sample they need out of 5 different sample instruments, and loading those in 1 sampler instance instead of using 5 sample instances ...

 

hmmm....i do that a bit or commit to audio, but its not always practical. the great thing about the new sample libraries are the range of articulations or velocity layers ...

I think you'll find (unless you're scoring 10-octave runs for every plug-in), that you could still select and delete all the samples above and below the note range you're playing in. You obviously have to be very careful and figure out the sample map layout, and test the sound after. But you can probably save alternate versions of the sample instruments with just one or a few octaves of range and use dramatically less RAM.

 

Then there's clicking on the Freeze button like the rest of us : )

 

(And don't these orchestral sample library companies even warn you to run their libraries separately in standalone mode — even on separate computers, piping the audio in and out of your DAW computer?)

 

Again, at this point in DAW history, the only thing I think is not normal about Logic is that it's crashing when reaching its RAM limit instead of error messaging gracefully.

 

Apple is definitely addressing the issue for the future — in fact it's the centerpiece of their entire 64-bit product development strategy. I seriously don't think they will be putting any more development resources into either of their legacy technologies — OS9 or 32-bit apps : )

[fattymatthews]

Im keen for the Plogue Bidule thread. Im going to be getting into this in the next few weeks as i'm also a victim of the memory hell. It seems this is the best solution for now, giving us access to another dirty 4GB of memory.

[ski]

Again, at this point in DAW history, the only thing I think is not normal about Logic is that it's crashing when reaching its RAM limit instead of error messaging gracefully.

 

IMO Logic's inability to prevent a crash is nothing compared to its inability to do very basic DAW things like correctly chase/play back notes and controllers. So forget about 64-bit this and Snow Leopard that. IMO the Logic boys have a lot of work to do just to get the program to work correctly on a very fundamental level. In light of how many such fundamental bugs currently exist, and keeping in mind that Snow Leopard + Logic (some future version) is guaranteed to suck for at least 1.5 years from the time it's released, I concur with the opinion stated above that running large orchestral libraries outside of Logic -- whether via plogue or even on other machines -- is probably the only workflow worth considering for the next two years.

 

But I hope I'm wrong.

[ski]

Speaking of ReWiring and IAC, does anyone know if Mainstage can be used as an EXS-24 playback "module" in conjunction with Logic?

[stevenson]

i checked it out and no you can't. also its not rewire compatible.

[ski]

Thanks. What about IAC access?

[stevenson]

there's no need. rewire has its own midi ports - the midi bidules.

 

i will create a new post on the subject of plogue and logic. i tried yesterday and the internet ate my post. swine internet.

[Fred B]

Speaking of ReWiring and IAC, does anyone know if Mainstage can be used as an EXS-24 playback "module" in conjunction with Logic?

I don't see any reason why it would not be possible with IAC. Of course, there is no audio returned to Logic.

[stevenson]

I don't see any reason why it would not be possible with IAC. Of course, there is no audio returned to Logic.

 

and that you won't get offline bouncing. you could use audio jack to get the audio back into logic but honestly why bother? at that rate it would be simpler to use au lab, and you should also bear in mind that there is a bug with IAC that causes a midi loop in certain situations in logic.

[Fred B]

I don't see any reason why it would not be possible with IAC. Of course, there is no audio returned to Logic.

 

and that you won't get offline bouncing. you could use audio jack to get the audio back into logic but honestly why bother? at that rate it would be simpler to use au lab, and you should also bear in mind that there is a bug with IAC that causes a midi loop in certain situations in logic.

 

But you can't use Au Lab for hosting the EXS24.

 

Regarding IAC, I don't think there is a bug with IAC itself.

The loop problem is caused by Logic as it listens by default to all ports.

 

Anyway, I agree that Bidule ReWired provides a reasonable solution.

I've just checked and can confirm that it works.

[KennaOkoye]

Thanks a million Stevenson and Dedmondson!

 

I have an Imac duo 2.4 with gigs. I run alot of of third party instrument plugs and libraries. When I had 1 gig.. it was weird... I felt like I could run longer when maxxed out because logic would go over the pass, and it seemed to memorize the pass, so the next time i went over it, I wouldn't get a crash. Now that I have 4 gigs, it doesn't do that anymore...maybe I am running more and I don't realize it lol... but the memorization feature is gone, I thought it would also come with having 4 gigs. When I freeze tracks, it doesn't really do much. Sidenote: I run ALOT of EXS24 instruments that I made which are on a paritioned drive.

 

What setting in logic will benefit me from buffer size to the way I run my sessions Stevenson. I had to take this thread with a grain of salt because everyone in here had a higher caliber than i had. Thanks!

 

Dedmondson I will try your ideas! i hated the fact that I have useless audio files

[/u][/i]

[stevenson]

I think you'll find (unless you're scoring 10-octave runs for every plug-in), that you could still select and delete all the samples above and below the note range you're playing in. You obviously have to be very careful and figure out the sample map layout, and test the sound after. But you can probably save alternate versions of the sample instruments with just one or a few octaves of range and use dramatically less RAM.

 

well you can do that in the exs editor but the exs has external memory mapping so it is not an issue. we are talking about 3rd partyu sample based plugs that take logic quickly to its limit. you certainly can't do anything about the patches in kontakt. i can't edit them at all. i wish i could - i have some other edits on some of my samples i would like to do.

 

but i know of people who do NOT use sample based plugs - just audio plugs that take logic to its limit. not much joy for them unless they offload some of those plugs to something external.

[KennaOkoye]

Im not sure if you some my post but it was directed towards you Stevenson