how to get vertex color tag's data for a specific frame (different with current frame) without modifying current document?

BruceC · Feb 20, 2025, 8:12 AM

Hi,
could you please help me find the recommended way to read vertex color tag's color data at a specific frame (different with current frame) without modifying current document?
I think I should use BaseDocument::GetClone() and change the cloned document's time. However, I didn't find a good way to get the same vertex color tag and the object it belongs from the cloned document.
Could you please give me a help on this?
Thank you!

ferdinand · Feb 20, 2025, 9:28 AM

Hey @BruceC,

Thank you for reaching out to us. Let's split these effectively two questions into two parts.

Evaluating a BaseDocument at time #t

When you want to know the state of a scene at a time, #t, you can do this by setting the time of the document (BaseDocument::SetTime) and then executing its passes (BaseDocument::ExecutePasses), which is CInema 4D API slang for updating the scene state. This is configurable and can evaluate animations, build caches, and execute expressions, i.e., do the whole scene evaluation. But when you have a very heavy document, executing the passes is of course a not so cheap operation.

A more clever solution can be to manually interpolate animations - when that is all you need - via CCurve.GetValue. But that only works for animations that defined by keyframes and are decomposable into float values. The former is here not given for vertex colors, as you can only animate them procedurally.

This is all further complicated in a real life scenario, as being at frame t, then jumping to frame t + 10, and then just executing the passes might not be enough. The reason is because what you to query might be dependent on simulations; and your output will then be wrong when you just execute frame t and t + 10, instead of t, t + 1, ..., t + 10.

So, a preroll function could look like this:

using namespace cinema;

static maxon::Result<void> PrerollToTime(BaseDocument* const doc, const BaseTime& time)
{
  iferr_scope;
  CheckArgument(doc, "doc"_s, "Document is nullptr."_s);

  const Int32 fps = doc->GetFps();
  const Int32 firstFrame = doc->GetTime().GetFrame(fps) + 1;
  const Int32 lastFrame = time.GetFrame(fps);
  if (lastFrame < firstFrame - 1)
    return maxon::IllegalArgumentError(
      MAXON_SOURCE_LOCATION, "The target time is before the current time."_s);

  // Loop over all frames we need to reach #t and execute the passes. What to execute for each pass
  // (caches, animations, expressions) depends on what you want to achieve, and will of course have
  // an impact on the performance. Note that things likes cache building can have an impact on
  // animations and expressions, and vice versa. So, when one needs the 'true' scene state, one must
  // often calculate everything.
  for (Int32 frame = firstFrame; frame <= lastFrame; ++frame)
  {
    doc->SetTime(BaseTime(frame, fps));
    // Execute the passes for everything in the main thread (nullptr), use your thread instead when
    // you have one.
    if (!doc->ExecutePasses(nullptr, true, true, true, BUILDFLAGS::NONE))
      return maxon::UnexpectedError(MAXON_SOURCE_LOCATION, "Failed to execute pass."_s);
  }
  
  // For the last frame we should execute the passes twice, so that the scene can settle.
  if (!doc->ExecutePasses(nullptr, true, true, true, BUILDFLAGS::NONE))
      return maxon::UnexpectedError(MAXON_SOURCE_LOCATION, "Failed to execute pass."_s);

  // The document is now in the state of #time.
  return maxon::OK;
}

Finding a scene element #e in two scenes

I am a bit surprised that you ask that, as this is sort of the bread and butter for what you are doing. So, I might be misunderstanding you here. If that is so, please clarify what you mean.

The identity of scene elements is expressed in Cinema 4D by GeMarker. Such markers are given when objects are allocated (and by default also when copied) and are a hash of the creating machine's mac address, a timestamp and some secret sauce. An alternative way to retrieve the same data is BaseList2D::FindUniqueID(MAXON_CREATOR_ID) which just returns the raw data of the GeMarker obtainable via BaseList2D::GetMarker. When you copy scene elements or whole scenes, by default they are given new markers. You can prevent that with COPYFLAGS::PRIVATE_IDENTMARKER.

Never insert multiple nodes with the same marker into a scene.

An alternative could be to use links, i.e., a BaseLinks, for example in a BaseContainer stored with the document. But that is also just built on top of GeMarker.

Cheers,
Ferdinand

BruceC · Feb 21, 2025, 2:52 AM

Thank you for the reply, @ferdinand,

Thanks for the PrerollToTime() function! I didn't know that I need to ExecutePasses() for each frame until the lastFrame.

I'm writing a tag plugin that has a parameter which is a link to an existing vertex color tag, and a parameter to allow user specify a frame number. These two parameters work together, so the plugin will get the data from the vertex color tag at the specific frame, and use the data to do the rendering across all frames.
- 1.1 Because the tag plugin depends the linked vertex color tag, so whenever there is a change in the vertex color tag, the plugin needs to update its internal data based on the updated vertex color tag. Currently, I do this in the tag plugin's Execute() function. The tag plugin remembers the vertex color tag's pointer (initialized as nullptr) and dirty checksum (by C4DAtom::GetDirty(), and initialized as 0). So Execute() checks if the vertex color tag read from the link parameter changes or the vertex color tag's dirty checksum changes, if it changes, the vertex color tag's data at the specified frame will be read again.
  
  However, I found ExecutePasses() called in PrerollToTime() triggers the tag plugin's Execute() function (I think this is the cloned tag in the cloned document), and Execute() function tries to read the linked vertex color tag at the speficied frame again, so PrerollToTime() -> ExecutePasses() is called again, and triggers a new cloned tag in a new cloned document to run its Execute(), and so on.
  
  This makes me wonder if I'm not using the correct way to check if the linked vertex color tag's change. Could you please help find out what's C4D's recommended way to handle such case?
- 1.2 According to the sample code PrerollToTime(), the lastFrame (the target frame) must not be earlier than the current scene's time. This means if the scene's current time is later than the frame number (say, user slides the timeline to frame 10, but the frame number parameter is set to 5), there will be no way to get the desired data from the vertex color tag, right? How does C4d handle this kind of situation? Or there is no such situation at all in native C4d?
To get the same tag from the cloned scene, I could use code like below, but I am wondering if there is an existing API to do this, as I think this is a basic need.

BaseObject* getObjectByMarker(const BaseDocument& doc, const GeMarker& objMarker, BaseObject* start = nullptr)
{
    for (BaseObject* obj = start ? start : doc.GetFirstObject(); obj != nullptr; obj = obj->GetNext())
    {
        if (obj->GetMarker().IsEqual(objMarker))
        {
            return obj;
        }
        else
        {
            // check children
            BaseObject* childObj = obj->GetDown();
            if (childObj)
            {
                BaseObject* found = getObjectByMarker(doc, objMarker, childObj);
                if (found)
                {
                    return found;
                }
            }
        }
    }
    return nullptr;
}

BaseTag* getTagByMarker(const BaseDocument& doc, const GeMarker& objMarker, const GeMarker& tagMarker)
{
    BaseObject* obj = getObjectByMarker(doc, objMarker);
    if (obj)
    {
        for (BaseTag* tag = obj->GetFirstTag(); tag; tag = tag->GetNext())
        {
            if (tag->GetMarker().IsEqual(tagMarker))
            {
                return tag;
            }
        }
    }
    return nullptr;
}

I found I cannot read the cloned vertex color tag's data from the cloned document, because the owner of the cloned object is not an instance of Opoint.
According to the document, to read data from vertex color tag, I need to get the point count of the polyObject who owns the vertex color tag.

To do that, the BaseObject returned from BaseTag::GetObject() will need to be converted to PointObject, then PointObject::GetPointCount() will be used to to get the point count.
However, I found if the tag is from a cloned document, the BaseObject returned from BaseTag::GetObject() is not an instance of Opoint, so it cannot be converted to PointObject.

I can confirm that the BaseObject got from the original tag in the original document can be converted to PointObject.

I try to use the object from the cloned document, because I think the point count might change in a different frame. (The cloned document is at a different time)

BTW, the cloned document is created by doc->GetClone(COPYFLAGS::PRIVATE_IDENTMARKER, nullptr).

Sorry for the long list, please let me know if I need to split above questions to different topics.
Now, all these questions above make me wonder if the usage is designed appropriately.
Maybe from C4D design's point of view, it is not a proper design to let user specify the time (different with the current frame, and even can be a earlier frame) when the tag's data should be used to render all frames?

Thank you!

ferdinand · Feb 21, 2025, 6:06 PM

Hey @BruceC,

uff, that is a lot of points. I'll try to comb through most of it, but I do not have that much time today. Feel free to ask for clarifications where I left out things.

Currently, I do this in the tag plugin's Execute() function. The tag plugin remembers the vertex color tag's pointer (initialized as nullptr) and dirty checksum (by C4DAtom::GetDirty(), and initialized as 0).

Never user pointers to long term keep track of scene elements. Something not being nullptr does not mean you point to valid data. The node you point to could be long deleted, and then you point to garbage and access attempts will then result in access violations and a crash. Either use a BaseLink to store a reference to a scene element, or use a weak pointer. I.e., a pointer which becomes invalid when the pointed data is being deleted. You can for example see here how to use a maxon::WeakRawPtr.

However, I found ExecutePasses() called in PrerollToTime() triggers the tag plugin's Execute() function (I think this is the cloned tag in the cloned document), and Execute() function tries to read the linked vertex color tag at the speficied frame again, so PrerollToTime() -> ExecutePasses() is called again, and triggers a new cloned tag in a new cloned document to run its Execute(), and so on.

Yes, that is true and a common problem (not only in Cinema 4D). The superficial reason is that my preroll code example also passes True for the third argument expressions (effectively API slang for tags). When you would pass there false, your tag would not be executed again. But also all other tags would not be executed (which is likely not what you want). The solution to this can be registering a plugin ID, e.g., ID_FOO_PLUGIN_IS_COMPUTE_PASS. When you then clone your document, or execute the passes on an existing document, you write under that ID for example a bool into the data container of the document. Your tags Execute(tag, ...) then gets the document from tag and checks for that flag being present, to stop what it does that causes the infinite update loop. When operating on a document not owned by you, you would then delete that flag or set it to false after you manually invoked the passes, so that future updates made by Cinema 4D (which owns and operates the document) then take the 'normal' route.

The better solution is to generally design things in a manner that such infinite update loops cannot happen in the first place. I do not understand your problem well enough to give more concrete advice here. These scenarios are also not always avoidable.

According to the sample code PrerollToTime(), the lastFrame (the target frame) must not be earlier than the current scene's time.

The preroll function was just an example, not something you must follow to the letter. There is also a little bug I just see now, it should be of course if (lastFrame < firstFrame - 1). But the general idea is, that when you have a scene with simulations, i.e., stuff like Pyro or particles, where the state of the current frame depends on the last frame, you must basically cycle through all the correct scene states to get to that state. You can of course also jump to a previous point, but then you do not have to sim from NOW to FUTURE but from 0 to PREVIOUS.

Prerolling is not necessary when the scene does not contain any simulations, e.g., just key framed position animations, all simulations are cached, or you do not care about simulations and their impact on the rest of scene. Given what kind of plugins you do develop, you likely want to preroll.

Just to stress this again: Prerolling can be extremely expensive. When the user has a scene with 300 frames of Pyro simulations worth 30 minutes of computing, and you try to preroll half of it, your preroll function will run for 15 minutes.

What to do here, really depends on the exact case. Generally you try to avoid having to reach into the future or the past, as this is often a big no-no. The most common "pro" solution to that is caching. E.g., that you write yourself a vertex color cache in your case. When you only need to reach into the past, and this only applies in rendering, you could also effectively hide this from the user, as you must have passed and naturally calculated that value before (and therefore need no explicit cache recording).

Will answer the (2) portion on Monday, running a bit out of Friday time right now.

Cheers,
Ferdinand

BruceC · Feb 23, 2025, 11:00 PM

Thanks for your reply, @ferdinand

Never user pointers to long term keep track of scene elements. Something not being nullptr does not mean you point to valid data. The node you point to could be long deleted, and then you point to garbage and access attempts will then result in access violations and a crash. Either use a BaseLink to store a reference to a scene element, or use a weak pointer. I.e., a pointer which becomes invalid when the pointed data is being deleted. You can for example see here how to use a maxon::WeakRawPtr.

True, I didn't really use the pointer to long term keep track of the linked vertex color tag. Although the plugin remembers the vertex color tag's pointer, but it's just used to compare if the vertex tag link parameter changes. i.e. before reading data from the vertex color tag, the plugin always read the vertex color tag link parameter.

The solution to this can be registering a plugin ID, e.g., ID_FOO_PLUGIN_IS_COMPUTE_PASS. When you then clone your document, or execute the passes on an existing document, you write under that ID for example a bool into the data container of the document. Your tags Execute(tag, ...) then gets the document from tag and checks for that flag being present, to stop what it does that causes the infinite update loop.

Thank you very much, this works! Glad to learn this trick!

You can of course also jump to a previous point, but then you do not have to sim from NOW to FUTURE but from 0 to PREVIOUS

Thanks!

The most common "pro" solution to that is caching.

Yes, the linked vertex color tag's data is cached to avoid reading its data again and again as long as the vertex color tag doesn't change.

And for the question 3) I asked in the previous reply. I found after I use the trick to avoid infinite update loop, the problem is gone, owner of the cloned object is an instance of Opoint again. Do you know what the reason could be?

Thank you!

BruceC · Feb 24, 2025, 6:08 AM

The most common "pro" solution to that is caching. E.g., that you write yourself a vertex color cache in your case.

I'm wondering the example you mentioned here, do you mean create a hidden vertex color tag to cache the data?

ferdinand · Feb 24, 2025, 9:27 AM

Hey @BruceC,

since I am probably not going to find the time to answer (2) today (sorry), I'll at least answer this:

I'm wondering the example you mentioned here, do you mean create a hidden vertex color tag to cache the data?

No, I did not mean a hidden vertex tag, as a vertex tag could not hold all that data (regularly). In principle, the cache could be just some field on some data structure, e.g., HashMap<Int32, Block<ColorA>> _cacheForSomeVertexTag - a hashmap that associates frame numbers with arrays of colors, the colors of the vertex map at that frame. To fill that array, you either need a "bake vertex colors" command, similarly to how you can bake Pyro, particles, RBD, and SBD in Cinema 4D. Or when reaching into the past while rendering is enough, you could do the "en passant" solution I mentioned above. There is a lot of room for improvement here for doing this more cleverly than a brutish "bake vertex colors" command.

Such data would be of course volatile, i.e., would vanish when the user saves and closes a document. To change that, you would have to implement a node, e.g., a VertexColorCache tag, so that you can serialize that data with the scene. Technically speaking, you could also just inject your cache data into the data of a vertex color tag, but I would strongly advise against that. If you want to, you can then hide your VertexColorCache from the users view. Cinema 4D also hides quite a few caches from the users view, but I personally think that it would make more sense in this case to let the user have access. But I do not know as well as you do what you are trying to achieve, so I might be wrong.

Cheers,
Ferdinand

BruceC · Feb 25, 2025, 4:11 AM

Thank you, @ferdinand.
No worries at all, I know you must be very busy, just take your time. I appreciate all the help!
Cheers!

ferdinand · Feb 26, 2025, 1:10 PM

Hey @BruceC,

To get the same tag from the cloned scene, I could use code like below, but I am wondering if there is an existing API to do this, as I think this is a basic need.

Your code is correct as far as I can see that at a glance (the important thing is that you implemented traversal only semi-recursively; that you have a manual loop for GetNext) as you are otherwise will quite quickly run into stack overflows. When you want to make it nicer, you could implement fully interative traversal. Because semi-recursive traversal can still produce stack overflows for very deep scene graphs (depth of ~500 nodes or more). Iterative traversal is generally also more performant.

But this is all mostly splitting hairs; what you do is fine in 99.9% of the cases.

The main question seems to be here if we offer abstracted scene traversal in C++, as we for example do in Python with mxutils.IterateTree and mxutils.RecurseGraph. And the answer is unfortunately still "no". Maxime and I have talked about this with our core API colleagues more than once. But their stance has been always that doing this is trivial and everyone can do this themselves. In Python we are the system owners/architects; which is why we added there the functions in mxutils. Maxime and I recently talked about the subject again, and this is a good reminder to bring this up again with our core API team.

PS: There is IterateBaseList4D which the core team recently added with 2025.0.0, but it is highly specialized and probably not the right fit for you here. The impl. is in the frameworks, so you can check it out.

I found I cannot read the cloned vertex color tag's data from the cloned document, because the owner of the cloned object is not an instance of Opoint [...]

You can always get the data count of a variable tag with VariableTag::GetDataCount (and a vertex color tag inherits from that type). When you want to get the point count from the host object, including cases where the object is a generator (and not just a plain instance of Opolygon), you have to reach into the (deform) cache of the object. For something like an Ocube, this is trivial, as BaseObject::GetCache will there return directly an Opolygon instance. But caches can get very complex and deformations (BaseObject::GetDeformCache) can also play a role. I have once unpacked the subject of caches here for Python, but the same principles apply to C++. Vertex map data counts on generators which target complex cache hierarchies could be tricky to get right from this direction, as our simulation team tends to write pretty sophisticated solutions. I would try the VariableTag::GetDataCount route first when you only want to read things.

When you know that the node returned by BaseList2D::GetMain/BaseTag::GetObject is in instance of Opolygon (because you have for exampled tested for this with C4Atom::IsInstanceOf), you can then just static up-cast your node: static_cast<PolygonObject*>(node).

Cheers,
Ferdinand

Here is some example code I have written in Python. Things are bit diffrent there, since in Python there is no GeMarker but only MAXON_CREATOR_ID (which also exists in C++) and in Python we have abstracted scene traversal in the form of mxutils.RecurseGraph and co. But it demonstrates the core principle.

Result

When we clone the scene without COPYFLAGS_PRIVATE_IDENTMARKER, we are unable to find our object and its tag, when we do clone the scene with that flag, we find them both:

Code

"""Demonstrates how to find the 'same' scene element in a cloned document, when comparing nodes to
an original document.

In Python we can do this via BaseList2D.FindUniqueID() and MAXON_CREATOR_ID. In C++ we can do this
also via GeMarker for which MAXON_CREATOR_ID is just a quasi-alias as it returns the 16 bytes of
the marker of the node it is invoked on.

Must be run with a scene with at least one object with at least one tag.

Warning:

    Be EXTREMLY careful with PRIVATE_IDENTMARKER. It is private for a reason, as one can really 
    screw up things royally with it. Never insert multiple nodes with the same marker into one 
    document. Never insert multiple documents (a `BaseDocument` is also a node) with same 
    marker into Cinema 4D, e.g., via `InsertBaseDocument`. It is fine to have a temporary (non-
    inserted) document which is carbon copy of an inserted document (e.g., the active document).
"""

import c4d
import mxutils

from mxutils import CheckType

doc: c4d.documents.BaseDocument  # The currently active document.

def main() -> None:
    """Called by Cinema 4D when the script is being executed.
    """
    # Get the first object and its first tag in the scene or halt when there is none.
    obj: c4d.BaseObject = CheckType(doc.GetFirstObject())
    tag: c4d.BaseTag = CheckType(obj.GetFirstTag())

    # Get the MAXON creator IDs of the object and the tag, i.e., the 16 bytes stored by the 
    # respective GeMarkers of the object and the tag. In Python GeMarker is not exposed, but
    # in C++ we can take both the direct GeMarker route and the a bit more indirect route via
    # MAXON_CREATOR_ID.
    objUuid: bytes = bytes(CheckType(obj.FindUniqueID(c4d.MAXON_CREATOR_ID)))
    tagUuid: bytes = bytes(CheckType(tag.FindUniqueID(c4d.MAXON_CREATOR_ID)))

    print(f"{objUuid.hex() = }")
    print(f"{tagUuid.hex() = }")

    def SearchDocument(domain: c4d.documents.BaseDocument) -> None:
        """Searches the passed document for objects and tags that have the same #MAXON_CREATOR_ID
        as the first object and the first tag in the currently active document.
        """
        node: c4d.BaseList2D
        for node in mxutils.RecurseGraph(
            domain, False, True, True, branchFilter=[c4d.Obase, c4d.Tbase], stopBranchingEarly=True):
            nodeUuid: bytes = bytes(CheckType(node.FindUniqueID(c4d.MAXON_CREATOR_ID)))
            if (node.CheckType(c4d.Obase)): # the yielded node is an object
                print(f"\t{node.GetName() = } {nodeUuid.hex() == objUuid.hex() = }")
            elif (node.CheckType(c4d.Tbase)): # the yielded node is a tag
                print(f"\t{node.GetName() = } {nodeUuid.hex() == tagUuid.hex() = }")

    # Now clone the document without #PRIVATE_IDENTMARKER, i.e., cloning entities will be treated
    # as plainly instantiating them, and each node will get a new UUID. We will search the document
    # for matches but will find None.
    clone: c4d.documents.BaseDocument = CheckType(doc.GetClone(c4d.COPYFLAGS_0))
    print(f"Matches in: {clone = }")
    SearchDocument(clone)

    # Now clone the document with #PRIVATE_IDENTMARKER, i.e., cloning entities will be treated as
    # truly cloning them, and each node will get the same UUID. We will search the document for 
    # matches and will find them.
    #
    # NEVER INSERT DOCUMENTS cloned with #COPYFLAGS_PRIVATE_IDENTMARKER into
    # Cinema 4D. Never insert a node with a marker/MAXON_CREATOR_ID into  a document
    # which already holds a node with such marker: I.e., never clone a node from a document
    # with that flag and then insert the result. Crashes are the 'best' outcome here, you could
    # seriously courrupt user scenes like this.
    hardClone: c4d.documents.BaseDocument = CheckType(doc.GetClone(c4d.COPYFLAGS_PRIVATE_IDENTMARKER))
    print(f"Matches in: {hardClone = }")
    SearchDocument(hardClone)


if __name__ == '__main__':
    main()

BruceC

Thank you very much for all your detailed explanation, @ferdinand!