Problems with Reused Asset Resources in MGS_GETALLASSETS

[Peek]

@ferdinand : I've been working for a while with the solution of GetAllAssetsNew now but Today I noticed an issue:
When a texture is used in multiple shaders, in the "Project Asset Inspector" it has a "+" before "Asset". However when I pull that texture with GetAllAssetsNew it seems that is only displaying 1 "owner" shader, even though in reality it's multiple.

When I then loop over all texture assets to convert the paths from local to global, the extra owners keep their local paths and it's only changed for one of the owner shaders.

In my example I have 4 images assinged to 5 shaders. When I pull a len() on the list of assets I get 4. Then when I convert the paths to global, one of the shaders get's left behind and in the Project Asset Inspector the "+" vanishes and I now see 5 textures instead of 4. Same goes if I then pull a len() on the list again, it now says 5.

How can I make sure that it makes paths global on all owners of the texture?

Cheers,
Joep

Code:

def globalizeTextures():
    doc = c4d.documents.GetActiveDocument()
    textures = []
    c4d.documents.GetAllAssetsNew(doc, 0, "", flags=c4d.ASSETDATA_FLAG_TEXTURESONLY, 
                                  assetList=textures)

    print("The numder of textures is: ", len(textures))
    print("Textures are: ", textures)

    # Extract filename from all textures and put that name into the owner (shader) using the bitmap
    for texture in textures:
        print(texture)
        filePath = texture
        file = os.path.split(filePath["filename"])

        workPath = doc.GetDocumentPath()
        globalPath = str(os.path.join(workPath, "tex"))

         # Check whether the texture file is in the texture (sub)folder(s)
        for texture in os.walk(globalPath):
            i = 0             
            for filename in texture[2]:                
                # print("Texture in tex folder is: ", filename)
                # Check if the filenames match between the file in the shader and the file on the 
                # network
                if (file[1] == filename):
                    globalPath = str(os.path.join(str(globalPath),str(filename)))
                    owner = filePath["owner"]
                    print("Owner is: ",filePath["owner"])
                    returnShader(owner, globalPath)  
                else:
                    i += 1
                
                if (i == len(texture[2])):
                    if not filePath["exists"]:
                        print("This file ", file[1], " does not exist!")
                    else:
                        print("File ", file[1], " not found in the tex folder of the project, use "
                              "'Localize texture paths' first to copy the file to the tex folder "
                              "of the project")
        updateScene()
    return True

---

edit: forked form https://developers.maxon.net/forum/topic/14732/ by Ferdinand.

To have your code not "messed up", you should markdown formating.

```
def foo: return 3.14
```

which would render as

def foo: return 3.14

[ferdinand]

Hello @peek,

Thank you for reaching out to us. This question of yours did deviate too far from the subject of the topic you posted this under. Due to that, your posting has been forked. Note that our Forum Guidelines state:

• A topic must cover a singular subject; the initial posting must be a singular question.
  • Users can ask follow-up questions, asking for clarification or alternative approaches, but follow-up questions cannot change the subject.
  • When the subject of the topic is "How to create a cube object?", then a follow-up question cannot be "How to add a material to that cube?" as this would change the subject.
  • A valid follow-up question would be "Are there other ways to create a cube object, as the proposed solution has the drawback X for my use-case?" or "Could you please clarify the thing Y you did mention in your answer, as this is still unclear to me?".
  • There is some leverage-room for this rule, but it is rather small. Small changes in the subject are allowed, large changes are not.

Please try to follow these rules in the future. It is certainly no catastrophe when users violate these from time to time, as it can be tricky to decide when something is on-topic or not. But we must enforce this rule to a reasonable degree, so that the forum remains a searchable knowledge base.

About your Question

I wasn't aware of what you are reporting here: That an asset with the path P will only appear exactly once in the asset data yielded by the C++ method BaseDocument::GetAllAssets, no matter how often the path P is being used by scene elements.

When we look at the function description of BaseDocument::GetAllAssets, telling us that it "retrieves all assets from a document.", we can already suspect what I just confirmed by looking at the implementation; BaseDocument::GetAllAssets deliberatly filters its output to return all assets instead of all asset instances, i.e., it removes "duplicates".

In C++, users of the non-public API, could now do this, broadcast the message MSG_GETALLASSETS manually to gather all assets in a scene.

RootTextureString root {...};
AssetData assetData {doc, &root, assetFlags};
doc->MultiMessage(MULTIMSG_ROUTE::BROADCAST, MSG_GETALLASSETS, &assetData);

In the public C++ API, we have almost all tools to do this, only missing is the type RootTextureString which is a GeListHead which will hold all data returned by MSG_GETALLASSETS. I asked the developers if this was intentional decision, and if not, I will pull these types into the public C++ API, so that also 3-rd parties can use them.

In Python there are two more obstacles:

• MSG_GETALLASSETS is only supported as a received message in Python. So, you cannot MultiMessage broadcast that message in the first place.
• Completely unexposed is in Python also MSG_RENAMETEXTURES, the mechanism one is supposed to use for what you want to do.

You would have to reimplement asset discovery from scratch yourself, to do what you want to do. Which can be quite a task when you want to cover every little nook and cranny of Cinema 4D. Below I provided an example which works for the classic API scene graph, and some common data types. Everything else would have to be implemented by yourself.

Our Python dev @m_adam is currently on vacation. I will have to discuss with him if we want to consider any of this as a bug, but I have a hunch that he just like me will see this as odd but intended behavior. To wrap MSG_GETALLASSETS for Python, we would have to first wrap it for the public C++ API (technically not a necessity but how we do things). I have reached out to our Tech team to see if we intentionally hide things there. I have marked this topic as to_fix.

Cheers,
Ferdinand

Result:

Code:

"""Prints the node type name, node name, parameter ID, and value of each node-parameter pair in
#doc of type DTYPE_FILENAME, DTYPE_TEXTURE, DTYPE_STRING, or RSPath (the RS path type).

Only works for classic API scene elements and the explicitly wrapped data types. Other than the
functions GetAllAssets and GetAllAssetsNew, this approach will not filter out multiple owners of
the same asset.
"""

import c4d
import maxon

import os
import typing

doc: c4d.documents.BaseDocument # The active document

# --- Scene graph traversal code start -------------------------------------------------------------
# The graph traversal section has been taken from https://developers.maxon.net/forum/topic/14182/. I 
# removed here most comments for the sake of compactness. See original posting for details.
# --------------------------------------------------------------------------------------------------

IS_DEBUG: bool = False
G_TYPE_TESTERS_TABLE: dict[int: c4d.BaseList2D] = {}

def HasSymbolBase(t: int, other: int) -> bool:
    """Returns if the Cinema 4D type symbol #t is in an inheritance relation with the type symbol 
    #other.
    """
    if not isinstance(t, int) or not isinstance(other, int):
        raise TypeError(f"Illegal argument types: {t, other}")
    
    dummyNode: typing.Union[c4d.BaseList2D, int, None] = G_TYPE_TESTERS_TABLE.get(t, None)
    if dummyNode is None:
        try:
            dummyNode = c4d.BaseList2D(t)
            G_TYPE_TESTERS_TABLE[t] = dummyNode
        except BaseException:
            dummyNode = c4d.NOTOK
            G_TYPE_TESTERS_TABLE[t] = c4d.NOTOK

    if dummyNode == c4d.NOTOK:
        return t == other

    return dummyNode.IsInstanceOf(other)


def IterateGraph(node: c4d.BaseList2D, 
                 types: typing.Optional[list[int]] = None,
                 inspectCaches: bool = False, 
                 root: typing.Optional[c4d.BaseList2D] = None) -> typing.Iterator[c4d.BaseList2D]:
    """Iterates over the branching and hierarchical relations of #node.  

    Args:
        node: The starting node to inspect the contents for. Can be anything that is a BaseList2D,
         e.g., a document, an object, a material, a tag, a layer, etc.
        types (optional): The type of nodes which are in a relation with #node which should be 
         yielded. This will also respect inheritance, e.g., [c4d.Obase] will yield all objects,
         and [c4d.Ocube] will only yield cube objects. Will yield all node types when None. 
         Defaults to None.
        inspectCaches (optional): If the iteration should also branch into caches. Defaults to False.
        root: Private, do not override.

    Yields:
        Nodes which are in a relation with with #node and of a type or base type in #types.
    """
    if not isinstance(node, c4d.BaseList2D):
        return
    if IS_DEBUG:
        print (f"IterateGraph({node, types, root})")
    
    if root is None:
        root = node
    
    while isinstance(node, c4d.BaseList2D):
        if types is None or any(node.IsInstanceOf(t) for t in types):
            yield node

        if IS_DEBUG:
            print (f"\tBranches({node})")
        branchData: typing.Optional[list[dict]] = node.GetBranchInfo(c4d.GETBRANCHINFO_NONE)

        if branchData:
            for branchDict in branchData:
                geListHead: c4d.GeListHead = branchDict.get("head", None)
                branchName: str = branchDict.get("name", None)
                branchId: int = branchDict.get("id", None)

                if None in (geListHead, branchId):
                    continue
                if IS_DEBUG:
                    print (f"\t\t{branchName = }, {branchId = }")
                
                firstNodeInBranch: typing.Optional[c4d.BaseList2D] = geListHead.GetDown()
                shouldBranch: bool = (True 
                                    if types is None else 
                                    any(HasSymbolBase(t, branchId) for t in types))

                if firstNodeInBranch is None or not shouldBranch:
                    continue

                for branchNode in IterateGraph(firstNodeInBranch, types, inspectCaches, root):
                    yield branchNode
        
        # --- End of branching iteration.

        if inspectCaches and isinstance(node, c4d.BaseObject):
            for cache in (node.GetCache(), node.GetDeformCache()):
                for cacheNode in IterateGraph(cache, types, inspectCaches, root):
                    yield cacheNode

        for descendant in IterateGraph(node.GetDown(), types, inspectCaches, root):
            yield descendant

        node = node.GetNext() if node != root else None

# --- Scene graph traversal code end ---------------------------------------------------------------

def GetMostFileAssets(
        doc: c4d.documents.BaseDocument) -> typing.Iterator[tuple[c4d.BaseList2D, c4d.DescID, str]]:
    """Returns most file assets in a scene.

    Other than GetAllAssetsNew, this will not filter out duplicates of the same asset, nor does it
    rely on MSG_GETALLASSETS. This manually traverses a classic API scene graph which comes with
    a few drawbacks:

        * MSG_GETALLASSETS is backed by custom implementations of each type, i.e., it will 
          automatically expand into parts of a scene which are not covered by the classic API scene
          graph, e.g., material and scene nodes. Xpresso nodes and material systems based on it are
          covered, because Xpresso is part of the classic API scene graph. 
        * Support for material nodes could be added, but super custom stuff in 3rd party plugins 
          will likely be impossible to wrap in this manner.
        * We must implement data types manually. MSG_GETALLASSETS will just forward all relevant
          data to us, here we must pick the data types and the actual values we are interested in
          ourselves.
    
    Supporting material nodes, i.e., the "new" node system of Cinema 4D is possible but would be
    a bit extra work. One would have to look for NimbusRef's when iterating over the BaseList2D's 
    of the classic API scene graph and then traverse the nodes of the attached NodesGraphModelRef, 
    iterating over the attributes of each node, looking for attributes of type maxon.Url. 
    """
    # Iterate over all classic API scene element nodes in the document #doc (which itself is a node). 
    for node in IterateGraph(doc):
        # For each node iterate over its description. We only do this so that we get each parameter
        # #pid in node, we are actually not interested in the actual description (the _ parts). We
        # could also iterate over the data container, but the description is more convenient as it
        # handles recursion in the data container (containers within containers) to a certain degree
        # for us.
        for _, pid, _ in node.GetDescription(c4d.DESCFLAGS_DESC_NONE):
            # Step over all empty descid.
            if pid.GetDepth() < 1:
                continue
            
            # The data type of the parameter #pid is stored in its first desc level. 
            dtype: int = pid[0].dtype

            # The data types Cinema 4D is using to express file paths as parameters. Most things
            # are DTYPE_FILENAME ore DTYPE_TEXTURE, but some 'rogue' elements also use DTYPE_STRING,
            # e.g. the Relief Object. All three types are expressed as str in Python.
            if dtype in (c4d.DTYPE_FILENAME, c4d.DTYPE_TEXTURE, c4d.DTYPE_STRING):
                # Get the value and filter out anything that does not look like a file path.
                value: str = node[pid]
                if value != "" and os.path.isfile(value):
                    yield node, pid, value
            # Special handling for the data type Redshift is using to express file paths.
            if dtype == 1036765:
                pathid: c4d.DescID = c4d.DescID(
                    pid[0], c4d.DescLevel(c4d.REDSHIFT_FILE_PATH, c4d.DTYPE_STRING, 0))
                value: str = node.GetParameter(pathid, c4d.DESCFLAGS_GET_NONE)
                if isinstance(value, str) and value != "" and os.path.isfile(value):
                    yield node, pathid, value
            # Wrap other specialized data types here ...


def main() -> None:
    """Runs the example.
    """
    # Prints the node type name, node name, parameter ID, and value of each node-parameter pair in
    # #doc of type DTYPE_FILENAME, DTYPE_TEXTURE, DTYPE_STRING, or RSPath (the RS path type).
    for node, pid, path in GetMostFileAssets(doc):
        print(f"'{node.GetTypeName()}', '{node.GetName()}', {pid}, '{path}'")        

if __name__ == "__main__":
    main()

[Peek]

Hi @ferdinand , thanks a ton for the very extensive reply. I do understand where you are coming from decision/code wise and that it might indeed be intended behavior, but I also appreciate you flagging this behavior to be looked at!

I will look into the code you provided and see what I can make of it to work for our specific case. As with all the examples i receive on here it is also about learning itself, to become better.
I already found out that I can run my piece of code 2 times to catch the first set of instances, perhaps it is also possible to run that on a loop until the len() doesn't increase anymore It's a hack-job but it might be a good last resort.

To address the first comment about the post itself, my reasoning was that since it was about GetAllAssetsNew which we talked about last, I added it to my post to keep it all together.
But I can see how it deviated too much from the original question that started the topic, my apologies.

Cheers,