ReferenceError: the object 'c4d.Material' is not alive

[bentraje]

Hi,

Can't really find a forum post regarding the matter.
Anyhow, basically what I'm trying to do is repurpose the sample TreeView code and instead of the custom Texture object class. I just want to use the native c4d materials as the list items.

There are several attributes, which is manageable to revised, but the main bottleneck is this part of the code

    currentObjIndex = self.listOfTexture.index(obj)
ReferenceError: the object 'c4d.Material' is not alive

You can check the WIP code here:
https://pastebin.com/VFuN7MbR

[ferdinand]

Hello @bentraje,

Thank you for reaching out to us. A Python C4Datom, e.g., a BaseMaterial not being alive means that the pointer in the C++ layer is invalid. Which in turn implies that you cannot rely on pointers, which also impacts the Python layer, because when you retrieve a node, e.g., a a BaseMaterial, it is storing effectively a pointer.

What are pointers and why are they a problem in the Python API?

A pointer is similar to (a reference of) an instance, as it has a type, and it can be used to access members of the instance. Here is a simplified and idealized C++ pseudo code demonstrating the concept.

Thing item = new Thing();   // An instance of the type #Thing taking up the amount of memory
                            // whatever the memory footprint of #Thing is.
Thing* pItem;               // A pointer to a Thing instance, denoted by the little *, it only 
                            // stores the memory location of the pointed object, not the whole
                            // object. This pointer has not been yet initialized, i.e., is 
                            // currently pointing to nothing.

pItem = &item;              // #pItem now points to #item, it can be used to "pass around" #item.
pItem->value = 42           // Use the pointer to access the field #value in #item.

delete(item);               // Free the memory of #item. There is now no data structure with the 
                            // memory layout of #Thing at the memory location #pItem is pointing to.

pItem->value = 42;          // Depending on the used programming language, this either won't work
                            // anymore, crash the application, or even worse, change random data, as
                            // the former memory location of #item is now not occupied by a #Thing
                            // anymore and instead either just junk or (parts of) a new data 
                            // structure. 

And when you are doing this in your code,

class ListView(c4d.gui.TreeViewFunctions):
 
    def __init__(self):
        """
        """
        self.listOfTexture = doc.GetMaterials()

you are effectively telling Cinema 4D, 'Hey, please give me the pointers to all materials!'. Which it will do, wrapping them into BaseMaterial Python objects. There is no direct equivalent for GetMaterials in C++, but BaseDocument::GetFirstMaterial returns for example BaseMaterial*, i.e., a pointer to a material.

Now time goes on, and you then try to access self.listOfTexture[0] and Cinema 4D tells you, 'nope, this BaseMaterial pointer is not valid anymore' or in terms of our Python API, it is a 'dead' object. The reason for that is that is either that the object has been simply deleted or that Cinema 4d has reallocated it, i.e., moved it to a new memory location. Things are reallocated all the time and while you might be able to still see the 'same ' material in the Material Manger, it might has been long reallocated and a formerly established pointer to it is not valid anymore.

How do I deal with this?

This might be a very foreign concept to a Python user, as Python hides all this pointer madness away, but the general idea is that you cannot rely on pointers to data for any long-term access, unless you manage the lifetime of the pointed objects yourself (which you do not do in Cinema 4D in a sensible manner).

1. Copy things: Depending on what you want to do, you can simply copy the things you want to ensure to remain. In the most partial manner, this would mean copying the data containers of the materials: self._data = [item.GetData() for item in doc.GetMaterials()]. The problem with this is that you indeed get a copy. So, when a material is being changed after you retrieved the container, or when you want to change a material, this will not be reflected in/possible with your self._data
2. Identify things with other means than pointers: A pointer is just a way of saying, 'hey, I mean this thing there, yes, exactly, that thing!' and there are other ways to accomplish this, most notably GeMarker and its specialization of C4DAtom.FindUniqueID. With FindUniqueID and MAXON_CREATOR_ID you can uniquely identify objects, even over reallocation boundaries, i.e., a material which a user would consider 'the same' will always return the same UUID. This topic has been discussed multiple time on the forum, most recently here. There is also a thread where I asked some questions about the scope of the hashes for MAXON_CREATOR_ID when I was a user.

What you do in your case is up to you, and what you want to be done, I would write an abstraction layer for BaseMaterial (see end of posting).

Cheers,
Ferdinand

The code:

import c4d
import sys

doc: c4d.documents.BaseDocument # The active document.

class MaterialCollection:
    """Handles a collection of materials from a singular document for long-term referencing.
    
    Will reestablish material references when possible.
    """

    def __init__(self, collection: list[c4d.BaseMaterial]):
        """Initializes the collection with a list of materials. 
        """
        if not isinstance(collection, list):
            raise TypeError(f"{collection = }")

        # The list of materials, the list of markers for them, and their associated document.
        self._materials: list[c4d.BaseMaterial] = []
        self._markers: list[bytes] = []
        self._doc: c4d.documents.BaseDocument = None

        for material in collection:
            # Sort out illegal types, "dead" materials, and materials not attached to a document.
            if not isinstance(material, c4d.BaseMaterial) or not material.IsAlive():
                raise TypeError(f"{material} is not alive or not a BaseMaterial.")
            if material.GetDocument() is None:
                raise RuntimeError("Dangling material cannot be managed.")
            
            # Establish what the document of the materials is and that they all belong to the same
            # document. Technically, a BaseDocument can also be dead (when it has been closed),
            # but this is less likely to happen, so a stored reference to it can also become dead.
            if self._doc is None:
                self._doc = material.GetDocument()
            elif self._doc != material.GetDocument():
                raise RuntimeError("Can only manage materials attached to the same document.")

            # Append the material and its UUID marker to the internal lists.
            self._materials.append(material)
            self._markers.append(MaterialCollection.GetUUID(material))

    def __len__(self):
        """Returns the number of managed materials.
        """
        return len(self._materials)

    def __getitem__(self, index: int) -> c4d.BaseMaterial:
        """Returns the material at #index.
        """
        if len(self._materials) - 1 < index:
            raise IndexError(f"The index '{index}' is out of bounds.")
        
        # The item is still alive, we can return it directly.
        item: c4d.BaseMaterial = self._materials[index]
        if item.IsAlive():
            print ("Returning still valid material reference")
            return item

        # The item is not alive anymore, try to find its new instance.
        if not self._doc.IsAlive():
            raise RuntimeError(
                f"{self._doc} is not valid anymore, cannot reestablish material references.")

        # Get the marker identifying the material at #index.
        materialMarker: bytes = self._markers[index]

        # Iterate over all materials in #_doc to find one with a matching marker.
        for material in self._doc.GetMaterials():
            otherMarker: bytes = MaterialCollection.GetUUID(material)

            # We found a match, update the internal data, and return the material.
            if materialMarker == otherMarker:
                self._materials[index] = material
                print ("Returning re-established material reference")
                return material

        # There is no material with this marker anymore.
        raise RuntimeError(
            f"The material at {index} is not valid anymore and cannot be re-established.")

    def __iter__(self):
        """Iterates over all materials in the collection.
        """
        for index in range(len(self._materials)):
            yield self[index]

    @staticmethod
    def GetUUID(atom: c4d.C4DAtom) -> bytes:
        """Retrieves the UUID of an atom as a bytes object.

        Args:
            item (c4d.C4DAtom): The atom to get the UUID hash for.

        Raises:
            RuntimeError: On type assertion failure.

        Returns:
            bytes: The UUID of #atom.
        """
        if not isinstance(atom, c4d.C4DAtom):
            raise TypeError(f"{atom = }")

        # Get the MAXON_CREATOR_ID marker uniquely identifying the atom.
        uuid: memoryview = atom.FindUniqueID(c4d.MAXON_CREATOR_ID)
        if not isinstance(uuid, memoryview):
            raise RuntimeError(f"Illegal non-marked atom: {atom}")

        return bytes(uuid)

def main() -> None:
    """Runs the example and uses the handler type.
    """
    print ("-" * 80)
    # To make the example a bit more sensible, we are going to attach #MaterialCollection
    # instance to sys.
    collection: MaterialCollection = getattr(sys, "matCollection", None)
    if collection is None and doc.GetMaterials() != []:
        collection = MaterialCollection(doc.GetMaterials())
        setattr(sys, "matCollection", collection)
        print ("Stored material collection at sys.matCollection")
    elif collection is not None:
        print ("Retrieved stored material collection from sys.matCollection")
    else:
        print ("Document contains no materials.")

    # Iterate over the collection, when running the script multiple times while interacting with
    # the document, this will reestablish the references at some point.
    for item in collection:
        print (item)

    if len(collection) > 0:
        print (f"{collection[0] = }")
    else:
        print ("The collection has no items.")

if __name__ == "__main__":
    main()

[bentraje]

@ferdinand

Thanks for the comprehensive response.
I'm still figuring out the #2 solution presented which is more robust.

For the mean time, can I ask for the following questions

RE: Copy things
This is the most straightforward and probably the closest I can implement immediately in my current codebase. I am not dealing with interactive and huge amount of data (just a shy of 500 materials and 2-3k objects). Still manageable TBH.

Just wondering if there is any gotchas I should look out for on this route, so far, the only thing that I notice is if its a material, the preview shaderball doesn't update. Apart from that, it's the same object (the copy and the original one).

Here's the WIP code I'm using

orig_obj = # Some C4D object here. 
new_data = orig_obj.GetData()
# Modify the data/base container here 
orig_obj.SetData(new_data) 
c4d.EventAdd()

RE: matCollection

Where did you get the reference on this line specifically the matCollection?
getattr(sys, "matCollection", None)

I tried it on a standalone Python IDLE console, I get None. It only works on C4D.

It's not on the documentation but it seems like a reserve variable (i.e. sys.matCollection) that lives only on C4D (why not have it as c4d.matCollection?).

[ferdinand]

Hello @bentraje,

@bentraje said in ReferenceError: the object 'c4d.Material' is not alive:

Just wondering if there is any gotchas I should look out for on this route, so far, the only thing that I notice is if its a material, the preview shaderball doesn't update. Apart from that, it's the same object (the copy and the original one).

Here's the WIP code I'm using

orig_obj = # Some C4D object here. 
new_data = orig_obj.GetData()
# Modify the data/base container here 
orig_obj.SetData(new_data) 
c4d.EventAdd()

Unless I misunderstand you here, it will not work like this. With this approach you give up the ability to write data back or react to changed data automatically. Or you will also have to work with UUIDs. You cannot store your orig_obj as much as you cannot your listOfTexture. At some point the pointer will/can get out of whack. Depending on what you want to do, it might be easier to simply re-init your tree view more often, but I guess you do not want to lose your selection states or somthing like this. You could of course also solve this, but the solution would be again UUIDs

Where did you get the reference on this line specifically the matCollection?
getattr(sys, "matCollection", None)

I did not get this from anywhere, I made this one up. The example is meant to be run multiple times in a row where each execution has access to the data generated by the first execution. To do that, I just attached my data to sys, i.e., I added an attribute for my data to the module object. Hence the getattr and setattr. Think of it as sys.matCollection.

Cheers,
Ferdinand

[bentraje]

@ferdinand

re: straight copy of python object
thanks for the heads up.
yea, it doesn't work. haha.

re: uuid is better
I see what you mean now.
I'm trying to implement to the Treeview example and I'm getting an error.
RuntimeError: <dead c4d.documents.BaseDocument object at 0x000001E35AAAB900> is not valid anymore, cannot reestablish material references.

In your code, you mentioned:
# The item is not alive anymore, try to find its new instance.

I tried doing it when I'm instancing the ListView though this code but I still get the same problem

    collection: MaterialCollection = getattr(sys, "matCollection", None)
    if collection is None and doc.GetMaterials() != []:
        collection = MaterialCollection(doc.GetMaterials())
        setattr(sys, "matCollection", collection)
        print ("Stored material collection at sys.matCollection")
    elif collection is not None:
        print ("Retrieved stored material collection from sys.matCollection")
    else:
        print ("Document contains no materials.")
    _listView = ListView(collection)  # Our Instance of c4d.gui.TreeViewFunctions
    _listView._doc = c4d.documents.GetActiveDocument()

You can check the WIP code (with your material class) here:
https://pastebin.com/as0QHufY

[ferdinand]

Hey @bentraje,

I think you are still misunderstanding me. You do not have copy what I did in main there, i.e., the whole someNativeModule.someAttribute = something is not necessary. All you must do, is replace your:

self.listOfTexture = doc.GetMaterials()

with

self.listOfTexture = MaterialCollection(doc.GetMaterials())

you can then treat listOfTexture almost like a list of BaseMaterial, I at least implemented length, index access, and iteration for it as shown above, i.e., you can do this:

for item in collection:
    print (item)

if len(collection) > 0:
    print (f"{collection[0] = }")
else:
    print ("The collection has no items.")

The type MaterialCollection hides away re-establishing references by not only storing the references to BaseMaterial instances, but also their UUIDs. When you try to access myCollection[n] and the material is dead, i.e., the pointer has gone out of whack, it tries to reestablishes the material reference. The relevant work is done in MaterialCollection.__getitem__.

As always, I provided here an example and not an implementation, there are several things which could be improved about MaterialCollection; the document thing I mentioned in the code for example or implementing more operators for it, so that it behaves more like a real list of BaseMaterial. You could also inherit from list itself, but then you would have to overwrite __new__ which I deliberately did not do here, as it can confuse even more advanced Python users from time to time.

And last but not least; there is also LighLister by @m_adam. I haven't checked this version, but when I joined Maxon, I was actually given a different version of this plugin as a review task in the interview process, and he used there tags to solve the same problem. I.e., he attached a custom (hidden) tag to all things in his Light Lister dialog to store things like order and so on. With that he could then constantly refresh his version of self.listOfTexture and by that side-step the dangling references problem. This is also a way to solve it, but as I wrote then in my review, I at least personally think this is unnecessarily complicated. But it might feel more natural to others.

FYI: Maxime wrote this code before he joined Maxon, so please be aware that this is not reference code.

Cheers,
Ferdinand

[bentraje]

@ferdinand

Ah. My bad for not being explicit on my thought process

1. I actually did just implement this beforehand (i.e. that's the only change)
   self.listOfTexture = MaterialCollection(doc.GetMaterials())

but it still gets me.
RuntimeError: <dead c4d.documents.BaseDocument object at 0x0000025867F19C80> is not valid anymore,

2. That's why I tried to use the sys.matCollection you used.
   But I still get the error.

3. Then I tried to superimposed the existing document
   _listView._doc = c4d.documents.GetActiveDocument()
   since the problem is the document itself being dead not the material.
   But I still get the same error.

=====

I used the TreeView before but the items are not c4d objects. They are on a directory (i.e. simulating a file directory structure) so its always being referenced. So the thing about objects being reallocated constantly and thus being dead is somewhat new to me. (I don't do C++)

Anyhow, will dig further around see if I can find a culprit.

[ferdinand]

Hey @bentraje,

Tying scene data to a native module, e.g., sys, is a bad idea in a production environment, as it can cause things not to be freed which should be freed, and by that eat up precious RAM. This is a bit my fault, because I showed it here, without a fat warning 'Lazy Ferdinand-stuff, please do not repeat this in a production environment!'

RuntimeError: <dead c4d.documents.BaseDocument object at 0x0000025867F19C80> is not valid anymore

As lined out in my comments in the code, you could play the same game also for documents, as a document is also a C4DAtom, i.e., has an UUID. There are some natural boundaries here, e.g., when a user closes a document, it will be hard to reestablish a reference, as you would have then to search all volumes of a system for c4d files which have this UUID. But you could at least search in the opened documents. And as a rule, you should always retrieve documents from nodes, e.g., myMaterial.GetDocument() instead of relying on the active document or some stored document reference, you will sidestep a lot of problems doing this.

I considered this all obvious from the comments I wrote in my code, but I seem to have been overly assumptions here. My apologies, the lifetime of node references can be a tricky thing, as simple things as passing a node from one function to another can cause them to "die". I have updated my example to also include document handling, it should now do what you need. If you also want to handle objects, tags, shaders, etc., you will have to do this yourself.

Cheers,
Ferdinand

The output:

collection.Document = <c4d.documents.BaseDocument object called  with ID 110059 at 2216491009024>

collection.Materials = (<c4d.Material object called Mat.4/Mat with ID 5703 at 2216445076544>, <c4d.Material object called Mat.3/Mat with ID 5703 at 2216445076672>, <c4d.Material object called Mat.2/Mat with ID 5703 at 2216445076736>, <c4d.Material object called Mat.1/Mat with ID 5703 at 2216445076800>, <c4d.Material object called Mat/Mat with ID 5703 at 2216445076864>)

collection.Markers = (b',\xf0]w\xccG\x16\x0b\x14\x14\x114\xca\x1f\x00\x00', b',\xf0]w\xccG\x16\x0b*\x13\x114\x02\x1f\x00\x00', b',\xf0]w\xccG\x16\x0bO\x12\x114;\x1e\x00\x00', b',\xf0]w\xccG\x16\x0b\xd8\x10\x114s\x1d\x00\x00', b',\xf0]w\xccG\x16\x0b\x19\x0e\x114Y\x1b\x00\x00')

len(collection) = 5

collection[0] = <c4d.Material object called Mat.4/Mat with ID 5703 at 2216445076544>

item = <c4d.Material object called Mat.4/Mat with ID 5703 at 2216445076544>
item = <c4d.Material object called Mat.3/Mat with ID 5703 at 2216445076672>
item = <c4d.Material object called Mat.2/Mat with ID 5703 at 2216445076736>
item = <c4d.Material object called Mat.1/Mat with ID 5703 at 2216445076800>
item = <c4d.Material object called Mat/Mat with ID 5703 at 2216445076864>
>>> 

The code:

import c4d

doc: c4d.documents.BaseDocument # The active document.

class MaterialCollection:
    """Handles a collection of materials from a singular document for long-term referencing.
    
    Will reestablish material references when possible and also handles a document reference.
    """

    def __init__(self, doc: c4d.documents.BaseDocument, collection: list[c4d.BaseMaterial]):
        """Initializes the collection with a list of materials. 
        """
        if not isinstance(collection, list):
            raise TypeError(f"{collection = }")
        if not isinstance(doc, c4d.documents.BaseDocument):
            raise TypeError(f"{doc = }") 
        if not doc.IsAlive():
            raise RuntimeError(f"{doc = } is not alive.")

        # The list of materials, the list of markers for them, their associated document, and its
        # marker.
        self._materialCollection: list[c4d.BaseMaterial] = []
        self._materialMarkerCollection: list[bytes] = []
        self._doc: c4d.documents.BaseDocument = doc
        self._docMarker: bytes = MaterialCollection.GetUUID(doc)

        # Validate all materials and get their markers, we must do that before the reference is dead.
        for material in collection:
            if not isinstance(material, c4d.BaseMaterial) or not material.IsAlive():
                raise RuntimeError(f"{material} is not alive or not a BaseMaterial.")
            if material.GetDocument() is None:
                raise RuntimeError("Dangling material cannot be managed.")
            if material.GetDocument() != doc:
                raise RuntimeError("Material is not part of passed reference document.")
            
            # Append the material and its UUID marker to the internal lists.
            self._materialCollection.append(material)
            self._materialMarkerCollection.append(MaterialCollection.GetUUID(material))

    def __len__(self):
        """Returns the number of managed materials.
        """
        return len(self._materialCollection)

    def __getitem__(self, index: int) -> c4d.BaseMaterial:
        """Returns the material at #index.
        """
        if len(self._materialCollection) - 1 < index:
            raise IndexError(f"The index '{index}' is out of bounds.")
        
        # The item is still alive, we can return it directly.
        item: c4d.BaseMaterial = self._materialCollection[index]
        if item.IsAlive():
            return item

        # --- The item is not alive anymore, try to find its new instance. -------------------------
        
        # Get the marker identifying the material at #index.
        materialMarker: bytes = self._materialMarkerCollection[index]

        # Iterate over all materials in #_doc to find one with a matching marker. We now call here
        # the property #Document instead of _doc, to also allow for reestablishing the document
        # reference.
        for material in self.Document.GetMaterials():
            otherMarker: bytes = MaterialCollection.GetUUID(material)

            # We found a match, update the internal data, and return the material.
            if materialMarker == otherMarker:
                self._materialCollection[index] = material
                return material

        # There is no material with this marker anymore.
        raise RuntimeError(
            f"The material at {index} is not valid anymore and cannot be re-established.")

    def __iter__(self):
        """Iterates over all materials in the collection.
        """
        for index in range(len(self._materialCollection)):
            yield self[index]

    @property
    def Document (self) -> c4d.documents.BaseDocument:
        """Returns the managed document reference.

        Will try to reestablish a document reference within the open documents when necessary.
        """
        # Return the still valid reference.
        if self._doc.IsAlive():
            return self._doc

        # Search all open documents for one with a UUID that matches self._docMarker.
        doc: c4d.documents.BaseDocument = c4d.documents.GetFirstDocument()
        while doc:
            docMarker: bytes = MaterialCollection.GetUUID(doc)
            if docMarker == self._docMarker:
                self._doc = doc
                return doc
            
            doc = doc.GetNext()

        # We could also search and load documents from #c4d.documents.GetRecentDocumentsList() here
        # but I did not do that, as it seems unlikely that you want to do that.
        
        raise RuntimeError(
                f"A reference to the document UUID '{self._docMarker}' cannot be reestablished. "
                "The document has probably been closed.")
    
    @property
    def Materials(self) -> tuple[c4d.BaseMaterial]:
        """Returns an immutable collection of the internally managed materials.

        Might contain dangling material references, i.e., "dead" materials.
        """
        return tuple(self._materialCollection)

    @property
    def Markers(self) -> tuple[bytes]:
        """Returns an immutable collection of the internally managed material markers.
        """
        return tuple(self._materialMarkerCollection)

    @staticmethod
    def GetUUID(atom: c4d.C4DAtom) -> bytes:
        """Retrieves the UUID of an atom as a bytes object.

        Args:
            item (c4d.C4DAtom): The atom to get the UUID hash for.

        Raises:
            RuntimeError: On type assertion failure.

        Returns:
            bytes: The UUID of #atom.
        """
        if not isinstance(atom, c4d.C4DAtom):
            raise TypeError(f"{atom = }")

        # Get the MAXON_CREATOR_ID marker uniquely identifying the atom.
        uuid: memoryview = atom.FindUniqueID(c4d.MAXON_CREATOR_ID)
        if not isinstance(uuid, memoryview):
            raise RuntimeError(f"Illegal non-marked atom: {atom}")

        return bytes(uuid)

def main() -> None:
    """Showcases how to use the type.

    I removed my hacky module attribute injection stuff from the earlier example, since it seems to
    have lead to confusion.
    """
    # Get a document by either getting the active document or a document from some node and get the
    # contained materials.
    doc: c4d.documents.BaseDocument = c4d.documents.GetActiveDocument()
    # doc: c4d.documents.BaseDocument = someGeListNode.GetDocument()
    materials: list[c4d.BaseMaterial] = doc.GetMaterials()

    # Instantiate the handler
    collection: MaterialCollection = MaterialCollection(doc, materials)

    # This will return the document with the UUID of #doc, as long as the document is still open.
    print (f"{collection.Document = }")

    # Get the tuples, i.e., read-only access, for the internal data.
    print (f"\n{collection.Materials = }")
    print (f"\n{collection.Markers = }")

    # The number of managed materials.
    print (f"\n{len(collection) = }")
    
    # Access items with an index, this will reestablish references (when possible).
    if len(collection) > 1:
        print (f"\n{collection[0] = }") 

    # Iterate over the managed items, this will reestablish references (when possible).
    print("")
    for item in collection:
        print (f"{item = }")
   
if __name__ == "__main__":
    main()

[bentraje]

@ferdinand

Thanks for the response. I'll get back to you on this just having some stuff in the pipeline that are more immediate.

That said, I do want to confirm that your code is working from the get go.

It's just my implemention of it in the GUI dialog. I'll just have to iron out the logic on that one.