Browsing field layers causes dangling/not alive references

baca

@ferdinand said in Field Refusing Object:

    def getFieldLayers(node):
        """
        """
        visited = []
        end = node.GetUp() if isinstance(node, c4d.GeListNode) else None

        while node:
            # If the current node is a newly encountered one.
            if node not in visited:
                visited.append(node)
                # When it is a field layer, yield it and deal with group
                # fields ...
                if isinstance(node, c4d.modules.mograph.FieldLayer):
                    yield node
                    linkedNode = node.GetLinkedObject(doc)
                    if linkedNode and linkedNode.CheckType(c4d.Fgroup):
                        # Get the field list and iterate over it.
                        fieldList = linkedNode[c4d.FIELDGROUP_FIELDS]
                        root = fieldList.GetLayersRoot()
                        for nestedNode in getFieldLayers(root):
                            yield nestedNode

                # There are more special cases in the field list hierarchy,
                # e.g., folders, which also have to be treated specifically.
                # I did not do that here.

            # Normal depth-first traversal of a node tree
            if node.GetDown() and node.GetDown() not in visited:
                node = node.GetDown()
            elif node.GetNext():
                node = node.GetNext()
            else:
                node = node.GetUp()
            if node == end:
                return

If anyone else relies on this code (many thanks @ferdinand for this great example)
for some reason yield node in 2024.4 emits exception:

ReferenceError: the object 'c4d.modules.mograph.FieldLayer' is not alive

Currently I went with yield node.GetClone() — issue is gone and it works as expected.

But I'm not sure if it's proper way.

ferdinand

Hey @baca,

Well, "expected" is a difficult term in the context of 'not alive'. Let me first unpack things a bit:

What means 'not alive'?

Cinema 4D is a C++ based application. So, even when you write Python code, you are operating with C++ data in the backend.
When there is a scene element node: c4d.BaseList2D in your Python code, for example a field layer, then there is somewhere in the memory of Cinema 4D a block which is the original C++ data, let's call the block MEM_CPP. The block is a ::BaseList2D (the C++ type), not a c4d.BaseList2D (the Python type).
What the Python wrapper now does, it creates a Python object for that C++ object (MEM_CPP), a c4d.BaseList2D at MEM_PY (your node) which points to that MEM_CPP for all its operations so that you can effectively drive the C++ object from Python.

A 'not alive' node now means that the Python object lost its connection to the C++ object. I.e., it is what is commonly called a dangling pointer/reference. The Python object does not know anymore to whom to forward all the instructions its gets. There are cases where this makes sense or at least is obvious. When you have for example an async dialog and you get and store all objects in the scene in that dialog as a class attribute self._objects: list[c4d.BaseList2D] when it is opened. And then try to use that list over the lifetime of that dialog. It is obvious that this can go "out of whack" because they user could have simply deleted objects since he or she opened the dialog.

But the somewhat uncomfortable reality is that Cinema 4D also reallocates scene elements a lot. So when you have an object "Cube" in your scene, you might look at it one second and it sits at the memory location X and then you look a second later at it and it suddenly sits at Y. For the user nothing really has changed but in memory happend what is somewhat equal to the user manually deleting the object and then recreating it from scratch with all its settings. The details of when and why this happens are irrelevant at the end of the day. In C++ this means that you should be cautious with pointers (as the object might have been deleted) and in Python this means that object can go "dead".

You should always operate on fresh scene data. When you want to browse the field layers of some object in your scene, retrieve that object from the document for each time you want to browse the layers. In node plugins the retrieving part often becomes obsolete as we often get relevant nodes passed in for each call by Cinema 4D.

With that all being said, there is definitively also some weirdness in the Python wrapper where things tend to die without a obvious reason why, especially when passing them out of functions. This can happen when the node which is being passed out is not yet attached to a document (there is probably some bug or at least design flaw in the Python wrapper). I also have seen problems with yielding (as opposed to returning).

About your Questions

No, this happening is generally not expected. But it depends a bit on the context of what you are doing. When the scene element which you are browsing the field layers for is not attached to a document, because you for example cloned the element, then I would still say this is a bug but I would not be surprised as this problem is not new. Can you share a bit more code and context of what you are doing?
1. How are you calling the function?
2. Is owner of node, i.e., the thing which has the FIELDS parameter, attached to a document?
3. Do you clone stuff apart from your workaround?
4. Most importantly, is this indeed a regression, i.e., can you demonstrate that the same code runs in a pre 2024.4 version?
No, cloning is not the 'proper way', at least when we are somewhat strict. Cloning the node means that you create a copy. This not only costs performance as Cinema 4D has to duplicate all that memory, but more importantly any changes to such cloned field layer will not be propagated to your scene (because you work on a copy). When you just want to read data and do not mind some small performance penalty, this will technically work. But this is a hack and not "not nice code".
From the context it sounds a bit like that the node is still alive when being yielded in your function, but then ends up dead on the side of the recipient who called that function (i.e., the very oddness/bug I talked about above).

Without more context and code, it is hard to give a good answer here.

I have forked this question because while the follow up question was absolutely okay there, this could get lengthy or might end up being flagged as a bug. And we are then better served with our own topic.

Cheers,
Ferdinand

baca

Thanks @ferdinand ,

Appreciate your answer.
Since cloning might dramatically reduce performance — I'll pay attention to this information, and will try to find the issue.

Currently the thing is — my plugin worked for years (starting R25 if I remember correctly), and this part of the code worked as expected all this time without modifications.
It just iterates through fieldlist layers and calculate dirty value in order to redraw generator within GetVirtualObject() scope.

After updating to 2024.4 I get reported — my plugin stoped to work.
It was hard to debug the issue, since I printed layer or result of IsAlive() — issue didn't replicated. So the quickest resolution was to yield cloned layer.

It's not easy for me to share sample which reproduces the issue, I have complex structure of code. Maybe I'll spend sometime later if I'll not be able to get rid of the cloning.
But since I spend several hours to resolve that issue, I wanted to leave a comment for those who just uses your sample in their code, and it stop to work.

ferdinand

Hey @baca,

feel free to send us a mail to sdk_support(at)maxon(dot)net in case you cannot share code publicly. But we have to have a look at your code to see what is going on. When you say this worked before, we either have a soft regression (you did something which you should not but it somehow worked and we now broke that) or a strong one (we broke something that should work) on our hands. And for that we will need code to see what is going on.

And I would not say that the performance penalty is dramatic, it more likely will be neligable. But if everyone would do that everywhere, Cinema 4D would come to a crawl. I would say you can ship your plugin with that "fix" when it does not cause any issues for you. But we should work towards solving this properly.

Cheers,
Ferdinand