Find projects that use XYZ plugin

[Danny Wyatt]

So on Keyboard Maestro's forum someone who's also a Logic user mentioned something I would like to have as a tool, so I'm wondering if anyone here with a LOT more knowledge than I do knows if this is possible and if so, how?

Would it be possible to interpret Logic's session data to figure out which plugins each project is using and sort them on a list? Both stock plugins and third party. I mean, that information is stored somewhere, right?

[fuzzfilth]

It is highly unlikely that you'll find this information as plain text in a Project file, but if you don't try you'll never know. Open a Project File that contains Omnisphere with Text Edit and search for "Omnisphere"

Edited June 20, 2022 by fuzzfilth

[des99]

14 minutes ago, Danny Wyatt said:

Would it be possible to interpret Logic's session data to figure out which plugins each project is using and sort them on a list? Both stock plugins and third party. I mean, that information is stored somewhere, right?

It's not really possible to easily extract that information from Logic's projects. As someone who has on various occasions analysed and reversed parts of the project data, a lot of it is binary and encoded, and there's a lot of serialised objects, and without being able to properly interpret the project file (absolutely no trivial feat), you can't really get that data out.

It's not enough to search the file for text like "Omnisphere" because that's not how a reference to that plugin would be stored internally - it would be an object with some id's that refer to particular plugins on the system, and that object might be encoded and obfuscated anyway.

I do have a project in my list of possible things to do based on various reversing of the file format - like reporting the contents of projects, including audio files referenced, plugins and presets used and so on, possibly even fixing damaged or corrupt projects - but in order to do those things, you have to reverse engineer the file format first - or at least the bits of it you're most interested in - and that's a non-trivial task.

It would be useful, but there is no easy way of doing it, unfortunately. If there was an easy way of getting that data out, I'd have probably already made a tool to do it, as like I say it's been in my possible projects folder with notes for years, and I have done some work in this area already. It's not impossible, but would need proper programming and reverse engineering work, and is not something that KM would help with.

Edited June 20, 2022 by des99

[Danny Wyatt]

13 minutes ago, fuzzfilth said:

It is highly unlikely that you'll find this information as plain text in a Project file, but if you don't try you'll never know. Open a Project File that contains Omnisphere with Text Edit and search for "Omnisphere"

I was able to open the Project Data file to a test project with just the ES2 and a single channel strip and searching for the ES2 I saw this:

 

Then I tried with more instruments and even a SilverVerb and I was able to find those as well. Now converting that to some kind of tool... This other person on the KM forum is pretty good at working with KM, so maybe KM along with the Project Data file could be a potential workflow... who knows?

[Danny Wyatt]

9 minutes ago, des99 said:

It's not really possible to easily extract that information from Logic's projects. As someone who has on various occasions analysed and reversed parts of the project data, a lot of it is binary and encoded, and there's a lot of serialised objects, and without being able to properly interpret the project file (absolutely no trivial feat), you can't really get that data out.

It's not enough to search the file for text like "Omnisphere" because that's not how a reference to that plugin would be stored internally - it would be an object with some id's that refer to particular plugins on the system, and that object might be encoded and obfuscated anyway.

I do have a project in my list of possible things to do based on various reversing of the file format - like reporting the contents of projects, including audio files referenced, plugins and presets used and so on, possibly even fixing damaged or corrupt projects - but in order to do those things, you have to reverse engineer the file format first - or at least the bits of it you're most interested in - and that's a non-trivial task.

It would be useful, but there is no easy way of doing it, unfortunately. If there was an easy way of getting that data out, I'd have probably already made a tool to do it, as like I say it's been in my possible projects folder with notes for years, and I have done some work in this area already. It's not impossible, but would need proper programming and reverse engineering work, and is not something that KM would help with.

Read my previous reply. When I open the Project Data file I am able to find the plugin names. Maybe with some KM tricks we could extract those names into a list?

I mean even if we have to have a long list of "variables" associated with each plugin's name on our system along with stock plugins, we could filter that down?

Something not as "pretty" as we wish, but something that could eventually work?

[des99]

If it's something you can reliably find and extract, then by all means, give it a go, I'd be interested in what you find. It's possible that the "ES2" in your example is the text of a track name for example, or something else that isn't actually a plugin reference, but I have no idea as I haven't directly looked at this particular data or how it's stored.

Let us know what you find! 👍

[Danny Wyatt]

2 minutes ago, des99 said:

It's possible that the "ES2" in your example is the text of a track name for example, or something else that isn't actually a plugin reference

Not really, because I tried with 3 different tracks, they are all named Inst 1, inst 2, inst 3 by default. Also, one of the channel strips has the ES2 and then the SilverVerb and they both show up so I guess it's really finding the plugins.

Is there a tool or a way to type things where I can perform the search of more than 1 word? For example in Text Edit I can search for one or more words, but it will find that as a group so if I search for ES2 SilverVerb it will try to find that as a single "sentence". How to find them separately like ES2, SilverVerb? Is there an app that can do that? Or maybe TextEdit can do it if I just type the search in a different way? I have BBEdit installed if that is helpful, even though I'm not an expert.

[des99]

Yes, regex searches in BBEdit are your power search tools.

However, searching for known plugin names is not a viable approach for a proper working tool, because you'd need to know the names of every plugin ever made in order to search for them.

What a proper tool would need to do is identify the track data in the project and decode the plugin name field contained (if that's what it is) and report on the ones found per track, without needing to know in advance of all the plugins that could potentially exist.

(I know you're just investigating and prototyping at this stage, just mentioning it in case... And if this is just something you're developing for yourself, you can just add in all the plugins you own to the search list as a quick and dirty utility, which may work fine for your needs.)

Edited June 20, 2022 by des99

[Danny Wyatt]

1 minute ago, des99 said:

Yes, regex searches in BBEdit are your power search tools.

Ok that's another step. Let's see...

[des99]

BBEdit has a great regex utility called "Search Playgrounds" which will help you learn regexes - it's excellent. Honestly, I couldn't survive without BBEdit's regex searches, I use them constantly everyday...

[Danny Wyatt]

11 minutes ago, des99 said:

because you'd need to know the names of every plugin ever made in order to search for them

I'm aware of that... 😕 but maybe we can at least have a tool that creates a list of the stock plugins along with comparing it with the plugins already installed so we could see what plugins each project is using based on that list? For example, let's say you want to compare the list of plugins on an old computer to the list of the new computer and see if you are missing any? At the moment I'm going through that process so that would help me a lot!

I don't know, I will keep investigating this and see if I can find a workaround...

[des99]

I agree that such a tool would be nice to have (hence why it's in my possible projects to do list  ) and I'm willing to help or look into this if it can be done reliably (and if it's not too much of a headache!)

I'd probably prototype it as a PHP script or something but if it was going to be generalised it's probably best becoming a Mac app down the line - that way you could more easily determine what plugins are installed and therefore what plugins aren't available for a given project. There's lots of good use cases for this (I should probably look back over my previous notes too  )

[des99]

I had a quick look at a test project, and it does indeed contain the au plugin id's close by the instrument name, which would be a reasonably reliable way to extract this and check against installed plugins etc. All I'd need is a reliable way to identify the track data, and a reasonably dependable/readable track structure, and it might be doable.

If people are interested I can look at hacking something up to do this as a proof of concept that can potentially be developed futher...

Edited June 20, 2022 by des99

[Danny Wyatt]

44 minutes ago, des99 said:

I had a quick look at a test project, and it does indeed contain the au plugin id's close by the instrument name

Have you tried with both stock and third-party plugins, instruments and effects? I didn't really pay attention to that part of the IDs to be honest and I don't even know if I would be able to identify what an ID is...

One thing I noticed though, and I'm not sure how that would be of any use at all, is that most names are followed by the word GAME. The Sampler doesn't though.

 

44 minutes ago, des99 said:

If people are interested I can look at hacking something up to do this as a proof of concept that can potentially be developed futher...

I believe the answer would be yes

 

44 minutes ago, des99 said:

it does indeed contain the au plugin id's close by the instrument name

Can you show me what that looks like? And if you added a lot of different plugins, is this something that you see repeating on all of them, meaning, are they all showing the AU id?

A first step could be to just have a list of the plugins present in a project, which is already a great step. But what if in the future people could start contributing to an online database of AU IDs that we could then keep adding to our list? That way, maybe we can't have a list of ALL plugins that were ever created, but at least it would become more and more accurate. Just an idea

Edited June 20, 2022 by Danny Wyatt

[des99]

6 minutes ago, Danny Wyatt said:

names are followed by the word GAME

That's an chunk identifier, traditionally used in the file format header. The way to read it is "EMAG" (from it's obvious roots  )

I'm having a hack around at this now... making some progress...

BTW If you want to see the plugins installed on your system, with their au identifiers, open terminal, and type "auval -l". They are the identifiers I'm referring to.

I don't think an online database will be necessary.

Edited June 20, 2022 by des99

[Danny Wyatt]

1 minute ago, des99 said:

I'm having a hack around at this now... making some progress...

Something is telling me that someone got excited about the possibility of this becoming a real thing! Good to know!

The topic on the KM forum is this, if you by any chance want to check it as well. Don't know if there's any info there that could help somehow? Check the bottom of the topic, because that's where me and Neil talk about it: https://forum.keyboardmaestro.com/t/continue-macro-after-user-clicks-a-particular-image/28093/18

[noisyneil]

Hi all. I'm the other guy on the KM forum. 

Here's where I'm at with it.

The good news is that it works once you get the data into a variable, and if we can't get KM to read that file because of its encoding, we can still automate TextEdit to open the file and search. Nowhere near as quick, but it will work. If anyone knows a way to copy that data to a readable temp file, please chip in!

[Danny Wyatt]

@noisyneil

Just now, noisyneil said:

Hi all. I'm the other guy on the KM forum. 

Here's where I'm at with it.

The good news is that it works once you get the data into a variable, and if we can't get KM to read that file because of its encoding, we can still automate TextEdit to open the file and search. Nowhere near as quick, but it will work. If anyone knows a way to copy that data to a readable temp file, please chip in!

Hey, "the other guy"! haha

I guess you and @des99 are the ones with the knowledge to eventually come up with a solution for this. Glad to see that the door is now a little bit open!

[polanoid]

27 minutes ago, des99 said:

That's an chunk identifier, traditionally used in the file format header. The way to read it is "EMAG" (from it's obvious roots  )

It obviously is the manufacturer ID, as for e.g. Kontakt, it will be "-IN-" instead. It seems to be followed by two other four byte values which will be "umua" and "D$iN" for Kontakt 6 (remember to read all these backwards so it's "-NI-", "aumu" and "Ni$D", so I guess the key is to check for these groups of three four-byte values. So far I have found that emagic's Apple's plug-ins seem to be marked by "GAME", then 4 bytes of value zero, and then four bytes of which only the first one is non-zero, but unique for the plug-in (so it's always the same for e.g. a Channel EQ)

Anyone care to compile a list?  

BTW I recommend not to use TextEdit for this as it seems to not show zero bytes at all. 

Edited June 20, 2022 by polanoid

[Danny Wyatt]

5 minutes ago, polanoid said:

Apple's plug-ins seem to be marked by "GAME"

I noticed that the Sampler does not have "GAME" after the name. Not sure if that makes a difference...?

Glad to see another person who's knowledgeable joining this topic. I have no idea what you guys are saying, but you all sound way ahead of me so that's awesome!

 

I started with TextEdit on my tests, just because I didn't think about BBEdit and to be honest, I'm not that knowledgeable to even know what to find there... But when I opened the same file with BBEdit, I noticed at at least it had a lot of ??? in there so I'm assuming it shows more info that could be useful for you guys. I wish I could be more useful though...

Edited June 20, 2022 by Danny Wyatt

[des99]

14 minutes ago, polanoid said:

Anyone care to compile a list?

Yes, these are the plugin ids I was referring to. There's a lot of stuff encoded backwards, I guess because of object serialisation. If you look at the AU spec you'll see there's three main identifiers, one four-byte string for the manufacturer - "GAME" (it's backwards remember) for EMAGic stuff, one for the plugin name, and one for the plugin type (instrument, effect etc).

These identifiers are exactly the ones in your tag database folder, just encoded as numbers rather than UTF8, and it's how plugins are identified on macOS.

Like I say, "auval -l" will list all yours on your system, along with their identifiers.

Edited June 20, 2022 by des99

[Danny Wyatt]

20 minutes ago, des99 said:

Like I say, "auval -l" will list all yours on your system

It didn't work. It was auval -a instead

[polanoid]

26 minutes ago, des99 said:

There's a lot of stuff encoded backwards, I guess because of object serialisation

I wonder if this is due to endianness. Do you still have a project saved on a PowerPC machine to check if the IDs read forwards there?

[des99]

That's a different command.

See "auval -h" for help.

               '-a'       lists all available AudioUnits of any type
               '-al'      lists all available AudioUnits of any type and their location
               '-l'       lists all available AudioUnits of any type without instantiating them

The -l flag doesn't instantiate them, so the list is fast. It may be that if you're on an older macOS version, you're running an old auval utility that doesn't have this feature.

Edit: Tried on Mojave, that option was unrecognised - so yes, it was presumably implemented in Big Sur or Monterey...

Edited June 20, 2022 by des99

[des99]

3 minutes ago, polanoid said:

I wonder if this is due to endianness. Do you still have a project saved on a PowerPC machine to check if the IDs read forwards there?

I do, because I did a bunch of hacking on the LP7 and LP9 file formats back in the day. The headers in the project indicates the endianness of the contents (at least it did back then, I haven't done a huge amount of hacking on the LPX format, closely related though it is. But I'm not updating my knowledge about that right now , I'm reversing the Audio channel object format so I can reliably get the names etc from the project data.)

But note that only *some* of these text strings are encoded backwards, the rest are forwards as expected, so it's not a general "this whole file is in intel encodings" - hence my guess as to why. But it's anyone's guess, of course..!  I'm not claiming to be correct in that, just a guess...

Edited June 20, 2022 by des99

[des99]

Ok, it's going well so far. Summary is:

The Audio objects are indicated by an "OCuA" data block (I assume that means 'Audio Channel Object'), and there are blocks for all the object types - tracks, instruments, busses, aux, input, output etc.

If one of those audio objects of a valid type has a plugin on it, there will be a "UCuA" block following, and in there will be the plugin name and id's, and the complete AUpreset of the plugin state. I guess there will be multiple UCuA blocks if there are multiple plugins on that audio channel - haven't got that far yet.

I'm looking through the OCuA blocks, looking for UCuA blocks, and we can then extract the plugin data into an internal list for reporting/export purposes etc. I'm just doing the UCuA extraction bit now.

When I have a hacky but working script, I'll upload it for others to play with and see how well it works on your systems. It's not a final product, it will be just a hacky prototype - once it generally works, we can build on it from there to implement more useful functionality and make it solid etc. (We'll use the thread to list out the desired functionality and get suggestions etc.)

Later on, we might even make it a Mac app, but I'm just proof-of-concepting right now...

Edited June 20, 2022 by des99

[Danny Wyatt]

No one can stop @des99 now!!! haha

Really glad that this was unlocked today and seeing that progress is being made. As I said, at this point I really can't do much, but if/when a Mac app is a possibility, I would love to offer my time and design skills to create the GUI, if that's something you would like me to do

Keep us posted!

Edited June 20, 2022 by Danny Wyatt

[polanoid]

2 hours ago, des99 said:

But note that only *some* of these text strings are encoded backwards, the rest are forwards as expected, so it's not a general "this whole file is in intel encodings"

Note that endianness will only affect these four character "strings" put into a 32 bit integer (look for FourCharCode or OSType in the documentation at developer.apple.com), not regular (arbitrary length) strings.

Edited June 20, 2022 by polanoid

[des99]

No probs, like I said, it was something I already had an interest doing, your post prompted me to have a look at hacking something up and seeing where we are - so thanks for that.

I'm often most interested in hacking on something new when I'm supposed to be doing something else...  

[Danny Wyatt]

5 minutes ago, des99 said:

when I'm supposed to be doing something else

Don't we all...? haha