How to properly load async data into C++ generator plugin?

[Havremunken]

Hi all,
I made a prototype in Python, more or less a proof of concept, but the performance was horrible and I am now porting the functionality to a C++ project. Writing this for C4D 2024, and might port it back to older versions if there is a need for it.

The generator object has a filename field. My plugin reads some data from a file, and then visualizes it by drawing a number of polygon objects. This works fine, but I also want my users to be able to paste in a web URL to get a file that way. And of course, getting something from a web server is a potentially slow operation. There is a DNS lookup, a connection, fetching the data - then once I have the text file in memory I need to parse it, do a measure pass (this involves opening a "hidden" C4D document in the background, creating a number of objects (potentially hundreds, depending on the file the user loads), measuring their bounding boxes, using this in a layout algorithm etc. This is very much not optimal to do inside GetVirtualObjects() for my object plugin.

So I was thinking that the right way to do this is to have a "object cache" (not the C4D concept - C++ objects that I parse from the text file) in my plugin. If I could get notified by C4D when my filename field is changed (or perhaps when the user presses the "Refresh" button in my UI), I could initialize the async loading task either from local disk or the web. I could happily take my time doing the data parsing, opening new document for bounding box measurements etc. Then, once I am finished, I store the "object cache" in my object, alert Cinema 4D that it needs to refresh me, and GetVirtualObjects would simply operate on the in-memory object cache I built and not start doing async io in a method that should preferably be pretty fast.

I don't know what to search for in the docs in order to realize this, however. I know that my GetVirtualObjects is called whenever a parameter changes - imagine that I have a "size" parameter that is a Real, and that the user is sliding it around. Suddenly I'm making a bunch of web requests - not very performant!

I hope what I am asking for makes sense. Is there a particular message I am listening for? Or is there another way to be notified of a parameter change outside of GetVirtualObjects?

Thanks!

[CJtheTiger]

Good morning @Havremunken,

Is there a particular message I am listening for? Or is there another way to be notified of a parameter change outside of GetVirtualObjects?

In order to listen to parameter changes you can implement the Message function in your plugin and listen for the MSG_DESCRIPTION_POSTSETPARAMETER message. Cast the incoming data argument to DescriptionPostSetValue to extract the ID of the changed parameter and get the value through the node argument.

Sample code taken from here and adjusted slightly:

Bool OMyPlugin::Message(GeListNode* node, Int32 type, void* data) {
    BaseContainer* bc;
    ...
    if (type == MSG_DESCRIPTION_POSTSETPARAMETER)
    {
        // Get message data
        DescriptionPostSetValue* dparm = (DescriptionPostSetValue*)data;
        Int32 parameterId = (*(dparm->descid))[0].id;
        bc = static_cast<BaseObject*>(node)->GetDataInstance();

        switch (parameterId)
        {
        case FILE_PATH: // Check for the parameter ID of the file path parameter.
            // Load the file via bc->GetString(FILE_PATH).
            ApplicationOutput(bc->GetString(FILE_PATH));
            break;
        }
    }
    ...
}

This way you could also initiate the cancellation of the current loading process in case the file path is being changed while a file is already being loaded. Alternatively you could lock the parameter in the description using GetDEnabling while the file is being loaded.

From a UX point of view I recommend indicating the current processing status to the user so the user knows whether a file is loading or if the loading has finished/failed.

And a personal flavor hint: I'm used to providing control buttons (Load/Start/Import/... and Cancel) for potential long running operations. This way it is clear and intuitive to the user how to use your object.

Cheers,
Daniel

[Havremunken]

Good morning, Daniel, and thank you so much for your post!

I'm not near Visual Studio for a couple of hours, but I will try to implement your suggestions as soon as I am!

I am like you when it comes to the buttons - more control to the user is a good thing. And yes, I already have some visual feedback mechanisms to let the user know what is going on.

One question, perhaps a bit on the side - I plan to use C4Ds built in functionality to download over http(s). Should I do this on a separate thread, and then notify C4D of my new data once done, or should I just run on the UI thread (that I assume I am on when I get the message)?

[ferdinand]

Hello @Havremunken,

Thank you for reaching out to us. Your question would have benefitted from your code or mock code. Your topic is overly broad and although you pose a concrete question in the title the general implications of the topic are quite broad. This will reflect in my answer.

Moving your code from Python to C++ will yield little to no performance benefits regarding the problem you face. This is mostly an architecture problem.

• Expensive Operations in Scene Elements: In general, scene elements should avoid doing anything expensive. Cinema 4D will execute scene elements in parallel, e.g., call GetVirtualObjects(), and you spending there lots of time on an expensive operation will then result everyone having to wait for you before Cinema 4D can advance to the next stage of scene execution. There are three patterns to deal with expensive operations:
  • NodeData::Init: This is the primary place to offload expensive pre-computations which do not require user inputs. This could be either loading data from disk (JSON, FBX, etc) or some heavy computations (building/precomputing lookup tables for example).
  • NodeData::SetDParameter: With this method you can customize what happens when a parameter is being set. Primarily this means writing the data to the data container. But we can also use the method to pre- or postfix a parameter write event with other operations such as downloading data from a URL. Alternatively, we can listen to MSG_DESCRIPTION_POSTSETPARAMETER in NodeData::Message, which would have a similar effect, as it is broadcasted after ::SetParameter ran.
  • MSG_DESCRIPTION_COMMAND: Scene elements can also hold buttons. MSG_DESCRIPTION_COMMAND is broadcasted in NodeData::Message when a user clicks a button. Doing this is more or less the same as tieing the URL download event to the change of a parameter.
• Caching: When you do something expensive in a node, it should always be cached in some shape or form. The simplest form of this would be that we should not download our data each time ::GetVirtualObjects is called. Everything running in parallel should be highly optimized and use precomputed data as much as possible. The methods listed above are the first step to do this. But let us assume we have a scene with ten of your objects with identical settings. The scene is loaded, data from https://mydomain.foo, the initial value of the URL, is loaded. The user then switches all URLs to https://mydomain.bar, just to change his or her mind, and change it back to https://mydomain.foo. If implemented not with performance in mind, we would end up with 30 download operations where we could get away with two.

How to Deal with this

What to do depends a bit on how performant this should get and how expensive the operations (downloads) are.

1. The thing you absolutely must do is defer the downloading of data to SetDParameter or MSG_DESCRIPTION_POSTSETPARAMETER, so that the price is only paid once when the user is actually changing the URL and not every time your object-plugin is executed. You should not only do the downloading of data here, but also the loading of the downloaded data.
2. If you must then optimize things further, you will need a central hub that manages downloads for you. This could be a SceneHookData (C++) or a MessageData (Python) plugin. Doing this pattern in Python is possible, but you will have to get a bit creative here and there. The idea would be then:
   • The user changes the URL in an object.
   • We still overwrite NodeData::SetDParameter or use MSG_DESCRIPTION_POSTSETPARAMETER to catch the moment the user makes the change.
   • But we do not do any downloading ourselves anymore and instead just grab our scene hook and send a message to it: please make sure that the data for URL https://mydomain.foo are in store.
   • The scene hook receives the message and either downloads and loads the data when it hasn't encountered that URL yet, or does nothing when the URL is already known.
   • Let us assume the downloaded data is FBX. We load that data into a document and then grab the BaseObject hierarchies from it that we need. We could now store the data as a BaseArray<BaseObject*> attached to our scene hook. But such a state would not be stored when we store the scene. We could overwrite NodeData::Write for our scene hook to store that data, but it would be more elegant to overwrite NodeData::GetBranchInfo so that our scene hook establishes a part of the scene graph where the preloaded data is stored. Then we do not a BaseArray anymore, and instead can simply parent the downloaded templates to this specialized part of the scene graph.
   • When our object is being built, its GetVirtualObjects is being called, we now send again a message to the scene hook, but this time not to register the URL but to retrieve a copy of its data which we can then use right away.

Without concrete code and concrete usage examples, things tend to become a bit fuzzy as my wall of text above. Focus first on doing the downloading and loading of data when the user changes a parameter, you can then later make this more complex. When you are after a hyper-performant solution, C++ would be the language of choice; primarily because you have there access to implementing scene hooks and more liberties in sending and receiving data with the message system.

Cheers,
Ferdinand

PS: I deliberately ignored the async part of your question as this would have made things even more complicated. For now you should execute things in the thread you are in, which will be the main thread when you follow the footsteps I have lined out here.

[Havremunken]

Hi Ferdinand, and thank you for your thorough answer!

Sorry about the broad question; I guess it reflects some of my frustrations getting into this whole project. I don't know how much I don't know. The docs help a lot in some of the specifics, but not with others. I'll try to keep it more focused in the future.

There are several reasons for moving to C++ from Python, chief among them being that I am MUCH more at home in C++. I also like the idea of a compiled language much "closer to the metal". But yes, you are right, and the reason for my original post was indeed about ending up with the right architecture. I am happy to observe that your suggestions are very close to my ideas on how to do this - but I only had a vague feeling about the shape of them, not the map on how to go about it using your framework. So thanks a bunch for shedding some light on that for me!

Downloading inside GetVirtualObjects was just a "proof of concept Python thing" and of course not something I would do for the real plugin. And there are several operations happening after the download as well - file parsing into an object structure, the bounding box measurement mentioned etc. - before finally notifying Cinema 4D that we are ready to display something new.

Thank you also for bringing up storing the data with the scene file - that is not something that had crossed my mind yet, as I am still very new with the framework here.

I'm going to start with MSG_DESCRIPTION_POSTSETPARAMETER like CJtheTiger also mentioned, but move the code I have to use SetDParameter as the Message method will probably get crowded. Get that to trigger the loading process, notify C4D that a refresh is in order once it is done, and see where that takes me. You do mention that I should "grab" my scene hook plugin and ask it to do stuff for me - is there a way for me to do that? I am looking at the advanced painting example that registers a scene hook in paintundo.cpp - RegisterPaintUndoSystem() but can't find the "correct" way to talk to the actual instance of my future Scene Hook plugin.

Your post has left me a lot of food for thought, so once again - thank you very much!

[ferdinand]

Hey @Havremunken,

C++ or Python

Use C++ when you are more comfortable with it, it is certainly the API which gives you the most choices, so you could call it the 'best' API in that sense. But your problem is not tied to the inherent slowness of Python and I wanted to make that clear.

SetDParameter vs MSG_DESCRIPTION_POSTSETPARAMETER

SetDParameter is not a message (or part of the message logic) but the actual method that implements setting a parameter. When a user sets the parameter of an object in the Attribute Manager or via code, it will call the NodeData::SetDParameter method of your ObjectData hook. If there is none, the default implementation (C4DAtom::SetParameter) kicks in. But when you implement it, you can then customize how the parameter is written. You could for example ignore the passed in node and not store the data there but in a customized form on disk, or you could refuse writing certain values, and finally you could pre- and postfix the actual writing of data with other operations.

MSG_DESCRIPTION_POSTSETPARAMETER on the other hand is part of the node message stream (the types of messages send to a singular node) and will be received in NodeData::Message. It is broadcasted after C4DAtom::SetParameter ran.

'Crowed' code or not should not be a deciding factor here. You can factor out code into in its own function for a NodeData::Message case if you want to. SetDParameter is the slightly more complex approach as you also must handle parameter writing there. But you have more freedom regarding in which order things happen, and you also do not have to wait the small amount of time it takes before a message reaches you after the parameter has been set. The advantage of the message is that it is quite straight forward and when you only want to do things AFTER the parameter has been set, it is probably the best choice as the message delay will be quite small. We have this manual on SetDParameter.

What is a Scene Hook?

Cinema 4D uses the branching pattern to organize its scene graph. A BaseDocument has for example an object branch (to which all objects are attached). Each object in that object branch has a tag branch (holding all tags) and a tracks branch (holding all tracks). Each tag in the tags branch would then also have a tracks branch. This then forms the scene graph beyond the tangible hierarchical relations between things like objects and layers. The Active Object plugin from the C++ SDK is one way to explore and understand the scene graph. I have also written multiple postings about the subject, including Python demo code, you could for example start reading here.

A SceneHookData plugin is an element that is added to each and every scene a user loads. Scene hooks are spider-in-the-net scene elements that are invisible to users but control the scene they are in. Most dynamics systems in Cinema 4D have at least one scene hook which manages them, or things like snapping or the interactive render region from the classic renderer are also backed by a scene hook. A new 'empty' document already contains dozens of scene hooks.

Scene hooks are nodes (derived from NodeData) and therefore part of the branching/hierarchy structure of a document. Because they are nodes, they can also participate in inter-node message communication via C4DAtom/NodeData::Message. With BaseDocument::FindSceneHook you can grab a scene hook for a document. Find below an untested mock-code example.

The slightly advanced technique I was proposing then, was that instead of just attaching your data to your SceneHookData instance, e.g., this->myDownloadedObjectsArray, inserting it into the scene graph by implementing NodeData::GetBranchInfo for your hook. Your hook would then have a branch where it stores downloaded data (probably as BaseObject instances) and by that all downloaded data would be automatically saved with the document. That would be then the most advanced form of caching, as the user would truly only have to download things once.

Cheers,
Ferdinand

Untested Mock Code

const Int32 g_mySceneHookPluginID = 123456789; // Plugin ID of a scene hook you implemented.
const Int32 MSG_REGISTER_URL = 987654321; // A custom message type, must also be a plugin ID.

const Int32 ID_REGISTER_URL_VALUE = 0; // ID inside a message container for the URL value.

bool MyObjectDataPlugin::SomeMainThreadMethod(GeListNode* node, ...)
{
    // Retrieving a scene hook off main thread is fine, but we should only send messages on main 
    // thread. While sending messages off main thread won't crash Cinema 4D, a lot of code builds on
    // the assumption that #NodeData::Message runs on the main thread.
    if (!GeIsMainThread())
        return false;
    
    // Get the document from #node, the frontend entity representing this #MyObjectDataPlugin
    // instance. In this case #node would be a #BaseObject. How the representing entity is passed in
    // differs a bit from method to method. Sometimes we get something fairly low level like a
    // GeListNode, sometimes a BaseList2D, or sometimes a high level interface such as a BaseTag or
    // BaseObject. 
    BaseDocument* const doc = node->GetDocument();
    if (!doc)
        return false;

    // Find the #g_mySceneHookPluginID scene hook in the document #node is contained in.
    BaseSceneHook* const hook = doc->FindSceneHook(g_mySceneHookPluginID);
    if (!hook)
        return false;

    // #hook is now a node like any other, we can send for example messages to it. In this case we
    // want to send a message for registering #url which might be a new URL (to download) or not.
    const String url ("https:///foo.bar");

    // In Python we are bound to the message types that exist, in C++ we are not. We can either
    // use something existing like MSG_BASECONTAINER or make up our own message type.

    // BaseContainer case, here we set the container ID to our registered message ID so that a
    // recipient can verify that the message data is wellformed.
    BaseContainer msgData;
    msgData.SetId(MSG_REGISTER_URL)
    msgData.SetString(ID_REGISTER_URL_VALUE, url);
    hook->SendMessage(MSG_BASECONTAINER, &msgData);

    // The custom case, here we just cast our data to a void pointer. The data is usually something
    // more fancy like a struct or class in custom cases that can not be easily sent with a 
    // container. C-style casts are okay in this direction, but on the receiving end we MUST use 
    // safe casts to avoid crashes when something is sending malformed data.
    hook->SendMessage(MSG_REGISTER_URL, (void*)&url);

    // bool MySceneHookData::Message(GeListNode* node, Int32 type, void* data)	
    // {
    //     //...
    //     case MSG_REGISTER_URL:
    //     {
    //         const String* const url = static_cast<String*>(data);
    //         if (!url)
    //             return false;      
    //         // ...
    //     }
    // }
}

[Havremunken]

Hi Ferdinand,

Once again, thank you so much! I did read the docs for SetDParameter after posting last night and realized the scope of it, so my code is already using it instead of the message notification. Not just to avoid crowded code either. But I have some situations that require validation, adjusting one param based on another etc., so I will soon implement GetDParameter and add that as well.

Considering that less than 24 hours ago I had never heard of Scene Hooks before, I am now confident I will have your suggestion about using it to handle data loading, parsing, caching etc. implemented in an elegant way shortly. I was afraid I would have to find a contrived way to get the data back to the generator plugin, but using the SceneHookData I should be able to have everything nice and cached, and just use this in GetVirtualObjects to construct the actual geometry from my data structures.

THANK YOU a million once more, and if you see your boss, tell him hi from me and that you deserve a raise.

[Havremunken]

Hi @ferdinand !

I have a working scene hook now, it receives the message and starts the work assigned to it. However, there was one thing I wanted to make sure I do the right way;

My understanding is that per document, there is one instance of my scene hook, but of course there can be many instances of my object generator. So when the scene hook has finished the work it has to do (still working on this, this is the "meat" of my plugin, it will take some time but for now I have it return some dummy data), it needs to tell the correct plugin instance that it needs to "invalidate", set itself as dirty, request a refresh etc - basically trigger C4D to run GetVirtualObjects again.

Of course, I could include a reference to the instance of the plugin with the data message sent to the scene hook that contains the job to be done. Just include a "this", and let the scene hook use that reference to call a method on the plugin that starts the dirty process. But this sounds like a snake pit of worries about threading context etc. So to ensure that things are done properly, is there an "idiomatic" way for the scene hook to tell the plugin instance that your data is ready, go ahead and request a refresh now?

I think this should probably be simple, but I am still getting familiar with the SDK and C++ changed a lot since I last used it so I want to make sure I do things right.

Bonus question: I understand that it is EXTREMELY important that my scene hook be stable, or it could screw up C4D for all users of the plugin. My scene hook basically just listens for the single message I am sending from my generator object, it does not override any of the methods for interacting with the mouse or anything. Is there anything I absolutely need to do in the scene hook outside of what I am already doing (listening for the message)?

Thanks again!

[ferdinand]

Hey @Havremunken,

Thank you for your follow up questions. I think you misunderstood me a little bit, you will need at least two message types. It is only that I only exemplified one part in my last posting. You also overread a bit the part in my first answer (How to Deal with this 1.) that you should start with your ObjectData side, as that would be the less complex part upon you could then build in a second SceneHook step when you need even more performance.

Find below a more complete sketch how you could use a scene hook as a central download manager. Please understand that this again a sketch, pseudo code, and that there likely will be a few typos. When you get stuck, please share your compileable code, so that I can than build upon it.

Regarding your questions: The main thing you seem to be unaware of, is that you could use C4DAtom::Message to move data in both directions, from the sender to the recipient, and the other way around. So, just as we can push URLs to the hook (for it to process), we can also pull assets from the hook to us.

The warning/note about performance SceneHookData has to be taken with a grain of salt. All nodes are performance critical and should not do expensive things without care (this thread is an example for that fact). Scene hooks are node as any other node, their only difference is that they are always present in a scene, so doing stupid things really hurts compared to a node which only is active in a handful of scenes. Since we only use the C4DAtom::Message part of our SceneHookData, there is little which we can screw up here. The warning goes more towards scene hooks which draw things into viewports or poke arround in the scene graph.

Cheers,
Ferdinand

// THIS IS UNTESTED PSEUDO CODE - it will contain typos and mistakes, it is meant to demonstrate
// a principle in a tangible way and not to be compilable code.

// It would be better to have something like an AssetAccessData struct which is the message data type
// for your messages here, instead of what I am doing here, sending maxon::Url and BaseObject data.
// But I wanted to keep things shortish.

// EDIT: I made a little booboo, we really need this as we want to send and receive at the same time
// in MSG_GETA_ASSET.
struct AssetAccessData
{
    maxon::Url url;
    BaseObject* asset;
}

// -------------------------------------------------------------------------------------------------

// Realizes a centralized downloading and file loading entity.
//
// This is a central entity which manages the downloads and file loading for all your objects in a
// scene. Each object communicates with this central entity to access its data. We technically could
// do all these things also with a specialized tag or object (or any other node) with which all your
// objects communicate; this is in fact how we would do it in Python. The advantage of a scene hook
// is just that it is part of a scene by default, we do not have to manually hide it from the user,
// and it finally also provides a bit deeper access than other nodes in some cases. 
class AssetAccessManager : public SceneHookData
{
  // ...

private:
  // The pool of downloaded data. This is the 'bad' way to solve this, it would be better to store
  // the downloaded assets in the scene graph via NodeData::GetBranchInfo, but that would be
  // another support cases to flesh that out.
  maxon::HashMap<const maxon::Url, const BaseObject*> data;

  // Downloads content from #url into a file, loads that file, extracts the relevant BaseObject* 
  // from it and puts it into #data. This methods bears the most optimization potential. It will run
  // on the main thread (as we only call it from there), and ideally, it would defer the downloading
  // and loading of data to its own thread(s). But async logic brings its own problems - what to do
  // when something requests loading data which has not finalized yet?
  Bool LoadAssetFromUrl(const maxon::Url& url) { ... }

  // Checks if #url is currently in the process of being loaded. Only relevant when you decide to 
  // implement #LoadAssetFromUrl in a non-blocking manner.
  Bool AssetIsLoading(const maxon::Url& url) { ... }

  // Returns if the hook has already internalized data for #url, i.e., if data.Find(url) does 
  // return a BaseObject* or a nullptr.
  Bool ContainsAsset(const maxon::Url& url) { ... }

  // Returns the asset for the given #url. I designed this here so that the requester has to draw a 
  // copy as we give them access to the RO original. It would be better to use (COW) references but
  // I wanted to keep things simple here. Always making a copy ourselves and then sending that
  // copy would be not so ideal IMHO, as this could lead to a lot of copying where no copying is
  // required.
  const BaseObject* GetAsset(const maxon::Url& url) { ... }

public:
  Bool Message(GeListNode *node, Int32 type, void *data)
  {
    // ...

    // An entity in the scene asks us to internalize the data for the given URL.
    case MSG_ADD_ASSET:
    {
      // It might seem a bit nonsensical to put this main-thread check here, as we only
      // ourselves will be emitting this message type, but better safe than sorry.
      if (!GeIsMainThread())
        return false;

      const maxon::Url* const url = static_cast<maxon::Url*>(data);
      if (!url)
        return false;      
      
      // We have already data for this #url in store or are in the process of loading it.
      if (ContainsAsset(url) && !AssetIsLoading(url))
        return true;
      
      // When LoadAssetFromUrl is blocking, you should display a download bar in the status 
      // bar or in a popup as the AssetBrowser does. But since we only let user pay this price
      // once when he/she sets that URL for the first time, that would not be that bad. A
      // non-blocking LoadAssetFromUrl() would ofc be more elegant.
      LoadAssetFromUrl(url);
      return true;
    }
    // An entity in the scene requires data for a given #url. When everything went right, we
    // already should have that data in store. In a finalized version we should ofc implement a
    // fail safe. We will be usually here off-main-thread as we will be called from things like
    // GetVirtualObjects.
    case MSG_GET_ASSET:
    {
      // Edit: The incoming data must be a struct here so that we can receive the URL and send the
      // asset object.
      const AssetAccessData* const assetData = static_cast<AssetAccessData*>(data);
      if (!assetData || !assetData->url)
        return false;
      
      // The requester requested something that does not exist. I deal with it in a very brutish 
      // manner here and simply terminate things by setting the message data to the nullptr. We 
      // could also start a download instead but the question is then again: Do we actually want to 
      // pollute the cache building thread(s) in this manner?
      if (!ContainsAsset(assetData->url))
      {
        assetData->asset = nullptr; // technically not necessary, #data should already be the nullptr.
        return true
      }
      // The non-ideal case that we have implemented things async and the cache building 
      // occurred before we were done downloading. We can either wait as I do here (will only
      // happen for the first build event) or we could instead let the requester return some
      // dummy geom for such an GVO/build event.
      while (AssetIsLoading(assetData->url))
        continue;
      
      // Everything went well, we give the requester access to the desired data. The implicit
      assetData->asset = GetAsset(assetData->url);
      return true;
    }
  }
}

// -------------------------------------------------------------------------------------------------

// A scene element that makes use of #AssetAccessManager.
class FooObject : public ObjectData
{
public:

  // The part where we send data to our hook, so that it can process the data in a centralized 
  // manner to the benefit of every entity in the scene. I went here the easy route and just listen 
  // for MSG_DESCRIPTION_POSTSETPARAMETER to catch the moment the user has modified #ID_URL_PARAMETER.
  Bool Message(GeListNode *node, Int32 type, void *data)
  {
    // ...
    case MSG_DESCRIPTION_POSTSETPARAMETER:
    {
      // Get out when the message data is malformed or not a post-event for "our" parameter.
      DescriptionPostSetValue* param = static_cast<DescriptionPostSetValue*>(data);
			if (!param || ((*(param->descid))[0].id != ID_URL_PARAMETER))
        return true;
      
      BaseDocument* const doc = node->GetDocument();
      if (!doc)
        return false;

      BaseSceneHook* const hook = doc->FindSceneHook(g_mySceneHookPluginID);
      if (!hook)
        return true;

      // Get the string data from that parameter and convert it into an URL we can send. While 
      // GeData can carry maxon::Data, i.e., a maxon:Url, there is currently no GUI for handling
      // urls in this manner, so we must convert from String.
      GeData pdata;
      if (!node->GetParameter(ConstDescID(DescLevel(ID_URL_PARAMETER)), pdata, DESCFLAGS_GET::NONE))
        return true;

      const maxon::Url url (pdata.GetString());
      if (!hook->SendMessage(MSG_ADD_ASSET, (void*)&url))
        return true;
      
      return true;
    }

    // ...
  }

  // The receiving part of our communication with the hook. Here is the place where we need the 
  // processed data and ask our hook for it.
  BaseObject* GetVirtualObjects(BaseObject* op, const HierarchyHelp* hh) 
  {
    // Get the parameter values so that we start building our geometry cache. Bail when there is no
    // meaningful URL.
    const BaseContainer data = op->GetDataInstanceRef();
    const maxon::Url assetUrl (data.GetString(ID_URL_PARAMETER, ""_s));
    if (assetUrl.Compare("") == maxon.COMPARERESULT.EQUAL)
      return BaseObject::Alloc(Onull);

    // Ask our hook for our data.
    BaseDocument* const doc = op->GetDocument();
    if (!doc)
      return BaseObject::Alloc(Onull);

    BaseSceneHook* const hook = op->FindSceneHook(g_mySceneHookPluginID);
    if (!hook)
      return BaseObject::Alloc(Onull);

    // Edit: We must use a struct here :)
    AssetAccessData assetData;
    assetData.url = assetUrl;
    assetData.asset = nullptr;

    if (!hook->SendMessage(MSG_GET_ASSET, (void*)&assetData))
      return BaseObject::Alloc(Onull);
    if (!assetData.asset)
      return BaseObject::Alloc(Onull);
    
    // As explained above, I chose here the route that we do not have ownership of the sent data,
    // and must draw a copy when we want to use them in a non-read-only fashion, i.e., return them
    // in our cache.
    BaseObject* copy = assetData.asset->GetClone(COPYFLAGS::NONE, nullptr);

    // We return the copy of the asset owned and managed by the AssetAccessManager instance in our
    // scene as the cache of this object. So, when there would be 100 objects in the scene, only for
    // one we would pay the price of downloading (but we have to copy things 100 times, but that is
    // much cheaper),
    return copy;
  }
}

[Havremunken]

Ferdinand, I am once again in your debt! Thanks you very much for the very thorough response. This whole thread can in retrospect be seen as my "awakening from The Matrix".

Thanks for expanding my understanding. This just gives me one additional follow up question:

In your psuedo code when the objectdata plugin sends the MSG_ADD_ASSET message to the scene_hook, that process is kicked off. Then when Cinema 4D does a GetVirtualObjects pass, we send the MSG_GET_ASSET message to get the required data. This makes sense. However - as far as I can understand, the scene hook once done with its' work does not trigger an AddEvent() or anything like that.

Is that because this is, after all, unfinished psuedo code? Or will Cinema 4D do another scene recreation automatically for some reason (in that case, why would we need to EventAdd in other cases)? Or does this depend on whether or not the "work code" will be executed on the same thread that the message is received?

Sorry if this is another "Captain Obvious" question from me, I just want to make sure I am doing things in an optimal way that will remain responsive for the user.

Thanks!

[CJtheTiger]

Hi guys,

I might be completely off here but let me throw this into the room:

• These two lines in Message of the SceneHookData of @ferdinand 's pseudo code will block the current thread:

      while (AssetIsLoading(url))
        continue;

• This means that Message in the SceneHookData will only ever finish once the asset has in fact finished loading.
• This means that GVO of the calling ObjectData will also block until it has finished loading.

Let's just assume the worst case where a user creates a large number of ObjectDatas and sets all their URLs at once to different values, and loading and processing a single URL would take multiple seconds. Would this cause issues for C4D since so many threads would be blocking?

Please correct me if I misunderstood something.

Cheers,
Daniel

EDIT
Adding to @Havremunken 's response: Threads like this one really help a ton for building up that basic understanding of how C4D works, and @ferdinand provides in-depth answers explaining everything we need to know. I can't thank this man enough.

[ferdinand]

Hey @CJtheTiger,

yes, these two lines are intentionally blocking, that is why I have put them there They exist so that an asyncly gathered asset can finish loading. You therefore also need these two lines when you have designed LoadAssetFromUrl as non-blocking, i.e., that the user does not have to wait until an asset has been (down)loaded when her or she sets a before unseen asset URL in one of the scene elements. I made a comment in LoadAssetFromUrl that warns about this exact penalty here.

And, yes, the penalty is that we hold up the scene pipeline in its pass execution here, as we are coming from GVO, i.e., are in a GVO thread.

As stated in the code, another pattern could be to simply return a dummy/null while you are downloading. But then you would have to also track the requestee(s) for URLs, so that the hook can flag them dirty once it finished loading that URL, so that their GVO gets triggered again. My personal advice would be (as hinted at in the code): Just implement the downloading in a blocking manner as the asset browser does. The cases where a user wants download multiple things in parallel in quick succession are probably very rare.

Cheers,
Ferdinand

[ferdinand]

But know that I read my own code again, I discovered a bubu ^^. In MSG_GET_ASSET I use data both to push in a URL and to push out a BaseObject. And in GVO then I forgot to put in the URL when I send the message. You need a struct so that you can send and receive at the same time.

I have fixed my code above.

Cheers,
Ferdinand

[ferdinand]

Hello @Havremunken,

In your psuedo code when the objectdata plugin sends the MSG_ADD_ASSET message to the scene_hook, that process is kicked off. Then when Cinema 4D does a GetVirtualObjects pass, we send the MSG_GET_ASSET message to get the required data. This makes sense. However - as far as I can understand, the scene hook once done with its' work does not trigger an AddEvent() or anything like that.

A linearized event queue for a blocking download manager could look like this:

1. User modifies ID_URL_PARAMETER in object "FOO" to `stuff:///foo.bar`.
2. We intercept the event.
3. We call home to our AssetAccessManager.
4. The URL is new, we download and load it.
5. We return to the parameter setting event.
6. "FOO" is flagged as parameter dirty.
7. Cinema 4D adds an event because the user just modified an object.
8. In some cases Cinema 4D will not resolve the event queue immediately, but usually it will.
9. Cinema 4D executes the passes on the scene.  
10. It checks the scene elements for being dirty.
11. "FOO" is dirty, Cinema 4D attempts to build its cache.
12. FOO::GVO is being called.
13. We call home to our AssetManager.
14. It returns our asset.

In the case where we move (4.) out of its embedding main-thread, i.e., we implement an async download manager, we might run into the problem that our asset has not finished (down)loading once we reach (13). Then we have to finish the downloading in the cache building which is not so desirable. The other option in that case would be to return in such case a null object as the cache result. Then we would have to flag our objects as dirty and then push an event from the download manager once the URL has finished loading. But that all is not something I would recommend doing, just implement things in a blocking manner.

So, long story short: No, you do not have to call EventAdd, Cinema 4D will do it for you anyways, because the user changed a parameter.

Cheers,
Ferdinand

[Havremunken]

Thanks again, Ferdinand! I have been conditioned by other programming to fear and loathe anything that blocks the UI thread. However, the files that will be downloaded by users of my plugin would typically be in the 250-1000 bytes range, as long as the network isn't extremely slow I guess it should not be a huge problem.

I will do the blocking behavior for now, and then maybe start looking into Jobs or something if needed.