Cache, DependenceList, Touch and the likes

[Helper]

On 11/12/2015 at 04:46, xxxxxxxx wrote:

User Information:
Cinema 4D Version:   16.011 
Platform:   Windows  ;   
Language(s) :     C++  ;

---------
Hi there,

I'm working on an generator object, that potentially depends on lots of other objects in different ways, so cache handling is potentially tricky. So here is a bit of context and two three questions:

Historically the plugin uses BaseObject::NewDependenceList(), BaseObject::AddDependence() and BaseObject::CompareDependenceList() to check for changes of any child or linked input object. The idea of doing it this way originated here:

https://developers.maxon.net/forum/topic/4303/3871_prevent-recalculation-while-moving

There has been no confirmation yet (at least not in that thread), whether this is actually legal. The documentation doesn't explicitly state any more that it *has* to be a child object, but still isn't exactly verbose. It has been working for a while now, so I guess that's ok.

I also manually walk through the full hierarchy of all real child objects (not the linked objects) and call BaseObject::Touch(), to make sure they aren't marked as input objects and therefore not displayed any more. I think I can't use BaseObject::TouchDependenceList(), because that would touch objects that are merely linked as well. Again, a confirmation would be awesome.

Now on to the questions:

(1) Certain linked objects may be placed in the scene further down than my generator object. Since the evaluation of a scene seems to mostly conform to the order the objects are placed in the Object Manager, that means I'd potentially use an old state (at least when I'm using their cache geometry) and they would then get updated after I evaluate my own generator. Is there a way around that? I guess MoGraph has a similar problem when the Cloner is used to place objects on another object.

(2) I am also linking scene materials, not just other objects in the scene. What's the recommended way to track changes with those? I guess the dependence list doesn't work there, because BaseObject::AddDependence() takes BaseObject* as input and the documentation explicitly mentions objects. However, I haven't tried. Do I have to manually track the dirty counter in these cases?

(3) The most important question: For certain rendering optimizations to work, I would need to enforce that the caches of the child objects are built, even though I don't use them directly within ObjectData::GetVirtualObjects(). The only way I currently see to force evaulation of their cache is to call BaseObject::GetHierarchyClone() on the child objects. That seems wasteful, however, because it also generates a clone (which I'd then just discard). Also my impression is that this function messes with my dependence list. Is there a way to make sure the child caches are built without doing anything with the dependence list and/or creating a clone? Alternatively it would be good to know what BaseObject::GetHierarchyClone() does exactly and whether I can perhaps deal with it. Does it perhaps just call BaseObject::AddDependence()?

Would be really great to get some input on this. Furthermore, since topics like these keep coming up, it would perhaps be great to have a comprehensive page on working with the caching system (perhaps including some of the most common workarounds)? I think this is one of the trickier things to wrap ones head around.

Best
Timm

[Helper]

On 15/12/2015 at 02:50, xxxxxxxx wrote:

Hi Timm,

Sorry for replying with a delay of a few days. The questions you're asking aren't so easy to answer.

Originally posted by xxxxxxxx

I'm working on an generator object, that potentially depends on lots of other objects in different ways, so cache handling is potentially tricky. So here is a bit of context and two three questions:

Historically the plugin uses BaseObject::NewDependenceList(), BaseObject::AddDependence() and BaseObject::CompareDependenceList() to check for changes of any child or linked input object. The idea of doing it this way originated here: 
https://developers.maxon.net/forum/topic/4303/3871_prevent-recalculation-while-moving
There has been no confirmation yet (at least not in that thread), whether this is actually legal. The documentation doesn't explicitly state any more that it *has* to be a child object, but still isn't exactly verbose. It has been working for a while now, so I guess that's ok.

Internally AddDependence() has a Bool parameter to specify if the object dependence to add is for a child object.
The C++ API can't set this parameter and it's always passed as true.
But as Cinema uses a link to retrieve the dependence object it doesn't matter if it's actually a child...

Originally posted by xxxxxxxx

I also manually walk through the full hierarchy of all real child objects (not the linked objects) and call BaseObject::Touch(), to make sure they aren't marked as input objects and therefore not displayed any more. I think I can't use BaseObject::TouchDependenceList(), because that would touch objects that are merely linked as well. Again, a confirmation would be awesome.

Yes, TouchDependenceList() would touch the linked objects, not only the children. But I don't understand the logic here.
Touch() sets the BIT_CONTROLOBJECT for the object, empties the object's virtual/deform caches and resets its dirty state.
I don't think it's a good idea to call Touch() as this will invalidate the child input objects.

Originally posted by xxxxxxxx

(1) Certain linked objects may be placed in the scene further down than my generator object. Since the evaluation of a scene seems to mostly conform to the order the objects are placed in the Object Manager, that means I'd potentially use an old state (at least when I'm using their cache geometry) and they would then get updated after I evaluate my own generator. Is there a way around that? I guess MoGraph has a similar problem when the Cloner is used to place objects on another object.

This is where dependencies come in handy! Even if an object is placed down in the hierarchy and not as child, if it's a dependence the generator will be notified for any of its changes.

Originally posted by xxxxxxxx

(2) I am also linking scene materials, not just other objects in the scene. What's the recommended way to track changes with those? I guess the dependence list doesn't work there, because BaseObject::AddDependence() takes BaseObject* as input and the documentation explicitly mentions objects. However, I haven't tried. Do I have to manually track the dirty counter in these cases?

Yes, use links to scene materials and check their dirty status.

Originally posted by xxxxxxxx

(3) The most important question: For certain rendering optimizations to work, I would need to enforce that the caches of the child objects are built, even though I don't use them directly within ObjectData::GetVirtualObjects(). The only way I currently see to force evaulation of their cache is to call BaseObject::GetHierarchyClone() on the child objects. That seems wasteful, however, because it also generates a clone (which I'd then just discard). Also my impression is that this function messes with my dependence list. Is there a way to make sure the child caches are built without doing anything with the dependence list and/or creating a clone? Alternatively it would be good to know what BaseObject::GetHierarchyClone() does exactly and whether I can perhaps deal with it. Does it perhaps just call BaseObject::AddDependence()?

No GetHierarchyClone() doesn't only call AddDependence()
It also builds the children hierarchy. So you don't need to call AddDependence() for the child object, only for the linked objects.
I think you'd better use GetAndCheckHierarchyClone() (remember to call it on the first children only then all children will be processed). This should make sure to check and rebuild if necessary the state of the children hierarchy.
Note the code for GetAndCheckHierarchyClone() is included in the sources for the C++ API in source/c4d_baseobject.cpp

Originally posted by xxxxxxxx

Would be really great to get some input on this. Furthermore, since topics like these keep coming up, it would perhaps be great to have a comprehensive page on working with the caching system (perhaps including some of the most common workarounds)? I think this is one of the trickier things to wrap ones head around.

Thanks for the feedback. The information on object's cache and dependencies will definitely be updated and added to the documentation.

[Helper]

On 15/12/2015 at 05:37, xxxxxxxx wrote:

Hi Yannick,

thanks for the comprehensive reply!

Originally posted by xxxxxxxx

Yes, TouchDependenceList() would touch the linked objects, not only the children. But I don't understand the logic here. Touch() sets the BIT_CONTROLOBJECT for the object, empties the object's virtual/deform caches and resets its dirty state. I don't think it's a good idea to call Touch() as this will invalidate the child input objects.

Well the idea was in my original paragraph. I want to mark the object as input objects (which would make them invisible). It appeared to me, that TouchDependenceList() would be the way to go for this. The documentation says: "Marks all the objects in the dependence list to be replaced by the generator."

This is what I want. Furthermore my impression was, that Touch() does per object what TouchDependenceList() does for all objects in the dependency list.

So my questions here would be:

(1) Is Touch() indeed the per Object verion of TouchDependenceList() or not? If not, it might make sense to make that somewhat more explicit in the documentation. I'm assuming TouchDependenceList() does something else than Touch(), because the former is also use in BaseObject::GetAndCheckHierarchyClone().

(2) What do I want to do in this situation? Since I can't use TouchDependenceList() as long as I use the depencendy list to check for all objects I use (not just the children), what would be my alternative? Would I run through the objects and set BIT_CONTROLOBJECT? Is that really all TouchDependenceList() does?

Originally posted by xxxxxxxx

This is where dependencies come in handy! Even if an object is placed down in the hierarchy and not as child, if it's a dependence the generator will be notified for any of its changes.

I take this as a confirmation that using the depency list is the way to go for my linked objects.

What does "nofified" in this context mean exactly? Just consider the following case: I have one of my generator objects and a terrain object I'm distributing stuff on. Now the procedural terrain object sees a parameter change and the scene is reevaluated. So Cinema 4D runs through all objects, hits my generator, my generator grabs the terrain cache (which is still the old one) does its thing, then Cinema 4D hits the terrain object and only now the terrain objects' caches are rebuilt to the current state (which I would've needed already).

This is what I am seeing. My positioning on the terrain always seems to be one evaluation behind. As soon as something triggers another evaluation run, my plugin sees that the cache has changed and updates, but as long as that isn't triggered explicitly, I get an outdated cache. The only thing I currently see to reliably react to a current cache, is placing the terrain object further up with Object Manager, which obviously is a user experience nightmare. There might not be a solution, though, since even MoGraph has this problem. But if there is a way to do this (can I perhaps use a Modeling Command to bake the terrain manually?), it would be great to know.

Originally posted by xxxxxxxx

No GetHierarchyClone() doesn't only call AddDependence()
It also builds the children hierarchy. So you don't need to call AddDependence() for the child object, only for the linked objects. I think you'd better use GetAndCheckHierarchyClone() (remember to call it on the first children only then all children will be processed). This should make sure to check and rebuild if necessary the state of the children hierarchy. Note the code for GetAndCheckHierarchyClone() is included in the sources for the C++ API in source/c4d_baseobject.cpp

I think this is a misunderstanding. What I wanted to say was. I am aware of both GetHierarchyClone() and GetAndCheckHierarchyClone() (and also of the source of the latter). The problem is that their use appears not to play well with me also manually filling the depencence list. Consider the following block of code:

  
  
     // Use AddDependenceList() to add related objects to the depencence list   
  
#if 1   
     AddChildrenToDependencelist(op, op, hh, TRUE);   
#else   
     // this version does ensure that the caches are built, but it apparently messes with the   
     // depenncy list and forces the scene to be reevaluated. Only happens when   
     // HIERARCHYCLONEFLAGS_ASPOLY is used (as opposed to HIERARCHYCLONEFLAGS_ASIS). The latter   
     // doesn't force caches, though. Damn...   
     Bool tmpDirty;   
     for (BaseObject *obj = op->GetDown(); obj; obj = obj->GetNext())   
     {   
          AutoAlloc<AliasTrans> trans;   
          trans->Init(doc);   
          BaseObject *clone = op->GetHierarchyClone(hh, obj, HIERARCHYCLONEFLAGS_ASPOLY, &tmpDirty;, trans);   
          BaseObject::Free(clone);   
          trans->Translate(true);   
     }   
#endif   
  
     // Use AddDependenceList() to add more related objects to the depencence list   
  

That's the relevant original code I'm testing with. AddChildrenToDependencelist() runs through the hierarchy of child objects and calls AddDependence() on them. That will get us to update properly when parameters of the object change, but does *not* fore those child objects to have their caches ready. When I use the other branch of the #if/#else block, the caches are reliably built (when I use HIERARCHYCLONEFLAGS_ASPOLY), but I get an unnecessary clone which I directly discard, but also something now triggers a dirty flag to go off and triggers a reevaluation at *every* time GetVirtualObjects() is called. That made me think that GetHierarchyClone() somehow messes with my use of the dependence list (which otherwise you seem to approve of) in a way that either makes GetHierarchyClone() incompatible with my additional use of the depencence list or requires me to somehow take certain precautions that I'm not aware of (perhaps a different call order).

The other problem is that both GetHierarchyClone() and GetAndCheckHierarchyClone() create a copy of the input hierarchy (briefly mentioned already). Im almost all cases, all I need is just the caches of the input being built, but I don't actually need a copy of them. So creating a copy and then discarding it appears wasteful. The reason behind this is that I very often only create derivative geometry (like a bounding box hierarchy or a point cloud). For that derivative geometry, I need the cache to exist (since I otherwise wouldn't be able to create an exact bounding volume or point cloud), but I do not need that clone. This isn't nice, but not a showstopper, though.

Best
Timm

[Helper]

On 18/12/2015 at 04:10, xxxxxxxx wrote:

Originally posted by xxxxxxxx

(1) Is Touch() indeed the per Object verion of TouchDependenceList() or not? If not, it might make sense to make that somewhat more explicit in the documentation. I'm assuming TouchDependenceList() does something else than Touch(), because the former is also use in BaseObject::GetAndCheckHierarchyClone().

TouchDependenceList() simply calls Touch() for all of its dependent objects. So it shouldn't be used if you've dependencies that aren't child.

Originally posted by xxxxxxxx

(2) What do I want to do in this situation?

Yourself only know what you want to do! 

Originally posted by xxxxxxxx

(2) Since I can't use TouchDependenceList() as long as I use the depencendy list to check for all objects I use (not just the children), what would be my alternative? Would I run through the objects and set BIT_CONTROLOBJECT? Is that really all TouchDependenceList() does?

TouchDependenceList() calls Touch() for all dependencies. And as I alrady told you, Touch():
- Sets the BIT_CONTROLOBJECT for the object
- Empties the object's virtual/deform caches
- Resets the object's dirty state

To retrieve the updated cache for child and linked objects you can use a SendModelingCommand(MCOMMAND_CURRENTSTATETOOBJECT) (with MDATA_CURRENTSTATETOOBJECT_NOGENERATE set to true).

In the case of your generator you should better not use GetHierarchyClone() and GetAndCheckHierarchyClone() but call AddDependence() for all the related objects yourself (children plus linked) and check their dirty status. You can then retrieve their updated cache with a SendModelingCommand(MCOMMAND_CURRENTSTATETOOBJECT) if they've been changed.