Path mapping in Cinema 4D using Redshift does not work

[karthikbp]

We're experiencing issues with path mapping in Cinema 4D scenes using Redshift when rendering on DeadlineCloud. Specifically, certain attachments, caches, and globalized path names aren't being correctly path mapped.

Currently, we're using "c4d.documents.GetAllAssetsNew()" to identify assets and then applying specialized handling for different asset types (shaders, objects, video posts, and materials). Our implementation can be viewed here: https://github.com/aws-deadline/deadline-cloud-for-cinema-4d/blob/mainline/src/deadline/cinema4d_adaptor/Cinema4DClient/cinema4d_handler.py#L55-L109

We have also tried using the environment variable : REDSHIFT_PATHOVERRIDE_FILE and that has also not worked for us. Here's how we tried to use it: https://github.com/joel-wong-aws/deadline-cloud-for-cinema-4d/blob/vdb_mapping/src/deadline/cinema4d_adaptor/Cinema4DAdaptor/adaptor.py#L105-L158

Despite our efforts, some assets aren't being properly remapped, particularly:
• Certain Redshift cache files
• Assets with globalized path names
• Some attachments that aren't captured by GetAllAssetsNew()

Could you advise on:

1. The recommended approach for comprehensive path mapping in Cinema 4D + Redshift on DeadlineCloud?
2. Any Redshift-specific considerations we should be aware of?

We appreciate any guidance you can provide.

[ferdinand]

Hello @karthikbp,

Welcome to the Maxon developers forum and its community, it is great to have you with us!

Getting Started

Before creating your next postings, we would recommend making yourself accustomed with our forum and support procedures. You did not do anything wrong, we point all new users to these rules.

• Forum Overview: Provides a broad overview of the fundamental structure and rules of this forum, such as the purpose of the different sub-forums or the fact that we will ban users who engage in hate speech or harassment.
• Support Procedures: Provides a more in detail overview of how we provide technical support for APIs here. This topic will tell you how to ask good questions and limits of our technical support.
• Forum Features: Provides an overview of the technical features of this forum, such as Markdown markup or file uploads.

It is strongly recommended to read the first two topics carefully, especially the section Support Procedures: How to Ask Questions.

About your Account

Please pick another username, it is obviously not okay to have "maxon" in your username. You can do this from Avatar in Menu Bar > Edit Settings > Change Username. Please do this within the next fourteen days (until 02/06/2025), otherwise we will rename it.

Please also note that we usually do not provide support to unverified accounts as yours (you never verified your mail). I also assume this comes from the same team as this support request did? You could also reach us via our contact form (when confidential information must be exchanged) but we should stop bothering end user support.

About your First Question

Your request is unfortunately very ambiguous, as you lack the reproducibility aspect lined out in Support Procedures: How to Ask Questions. , i.e., no executable code and data foundation with which we could reproduce your claims. I have seen your _remap_assetsfunction, but it ties into the rest of your file (it at least also calls _pathmap_recognized_types) and we cannot start poking around in your code/file to get it running in the first place.

The function GetAllAssetsNew is the Python frontend for the message MSG_GETALLASSETS which among others is broadcasted into a document by the Asset Inspector to gather all assets in a scene (via C4DAtom::MultiMessage). This means that each scene element that wants to support its path, font, etc, parameters being managed, must actively implement that message. It also means anything that is not a scene element (C4DAtom) cannot support it. Or to put it differently, if it does not show up in the Asset Inspector, it is not supported.

Your questions are all too ambiguous, I cannot really answer any of them.

• What does 'aren't being properly remapped' mean concretely?
• Which are the 'certain Redshift cache' files?
• What are 'globalized' path names? Are you talking about absolute paths?
• Which attachments are not captured by the function and what do you consider to be an 'attachment'? Do you mean assets? And when you do, which assets are not being captured by the function?

The recommended approach for comprehensive path mapping in Cinema 4D + Redshift on DeadlineCloud?

I am not really sure what is 'comprehensive path mapping' for you, but I assume you want to map/transform the paths referenced in a scene file. MSG_GETALLASSETS (GetAllAssetsNew in Python) is the tool for that. But there is no guarantee that a scene will expose everything with that message, only the things which cannot be recomputed are guaranteed to be exposed.

In general, you could also just abstractly iterate over a scene graph to for example find all scene elements that have a parameter of DTYPE_FILENAME:

"""Prints all DTYPE_FILENAME parameters to be found in the active scene.
"""
import c4d
import mxutils

doc: c4d.documents.BaseDocument  # The currently active document.
op: c4d.BaseObject | None  # The primary selected object in `doc`. Can be `None`.

def main() -> None:
    """Called by Cinema 4D when the script is being executed.
    """
    # We iterate over all scene elements in the document but ignore caches.
    element: c4d.BaseList2D
    for element in mxutils.RecurseGraph(doc, False, True, True):
        # Now we iterate over its description to find parameters of DTYPE_FILENAME.
        data: c4d.BaseContainer
        pid: c4d.DescID
        for data, pid, _ in element.GetDescription(c4d.DESCFLAGS_GET_NONE):
            # Get data type of the last component of the DescID and then check if it is a filename.
            dtype: int = pid[pid.GetDepth() - 1].dtype
            if dtype == c4d.DTYPE_FILENAME:
                print(f"Found path parameter with the label '{data[c4d.DESC_NAME]}' and value "
                      f"'{element[pid]}' in node '{element}'.")
                
    print ("Done.")
                
if __name__ == '__main__':
    # Execute main()
    main()

Here for example run on an empty scene:

What you will not be able to discover this way, are paths buried in GraphNode instances, e.g., nodes in Material Node graphs. You could also traverse them when you come across such graph, but that is not done automatically by RecurseGraph as it is a pure Cinema API graph traversal. Please note that when you test this locally, despite my warning, you might see your material nodes and their paths being reflected in this output. These are then Cinema API emulation (BaseList2D) nodes for the GraphNode nodes, to be able to display them in the Attribute Manager. There is no guarantee that they will be present in every scene, as they are constructed on demand, so you must traverse these graphs yourself to be sure.

But in general, I would not advise doing this manually, as Cinema 4D scenes are much more complex than you probably realize, and you will then have to deal with all that complexity. It is not out of the question that there is a bug in MSG_GETALLASSETS as we did recently some work there, but it is more likely that you look for things being exposed by that message which simply by design are not (e.g., some cache which can be recomputed easily). What we did not even touch is the whole cloud-overly-long path thing you are doing (your last support request), and how it could interfere with asset handling.

When you need more help, please make sure to provide for us executable examples, and please open separate requests for each of them (there are at least three different issues mentioned here in your bullet points).

Cheers,
Ferdinand

PS: What we did not even touch, is that your _remap_assets is incorrect, as you explicitly want to target Redshift but at the same time do not support node materials. Your code might work partially due to the mentioned emulation layer nodes, but when you let this code then run in a production environment with no GUI, or without interacting with the GUI first, this emulation layer won't exist. You can have a look at this thread to get a sense of what to do. You would have to implement each render engine you want to support.

[karthikbp]

Thanks for your response!

Please do this within the next fourteen days (until 02/06/2025), otherwise we will rename it.

Sorry about that. I'm not sure how the username got set to that and unfortunately I'm unable to change it. Can you guys change it for me to "karthikbp"?

Your request is unfortunately very ambiguous, as you lack the reproducibility aspect lined out in Support Procedures: How to Ask Questions.

I appreciate you sharing the article on how to ask questions properly.

To provide some context, at Deadline Cloud, we help customers submit their rendering workloads to AWS. Our solution (https://github.com/aws-deadline/deadline-cloud-for-cinema-4d) translates paths in a scene so that rendering machines can find all elements and render successfully. For instance, we map a local path like "D:\Path_to_local_my_file\tex\my_texture.bmp" to a remote path like "C:\path_to_my_remote_file\tex\my_texture.bmp" using a function called map_path. We developed this approach based on this forum post: https://developers.maxon.net/forum/topic/14388/change-textures-path-from-getallassetnew-in-the-new-node-editor/

We've encountered an issue with scenes like the Pyro Voxel scene in the Cinema 4D asset browser. When we cache the Pyro Output beforehand and submit, the renders come out completely black. This only happens with pre-cached scenes; uncached scenes render as expected. We suspect this is because the rendering machine can't locate the cache. I plan to create a separate forum post about this specific issue.

For the more general path mapping issue, in this forum post, you describe that:

Supporting all possible node spaces and all possible nodes for Redshift could be quite a bit of work

Given this context, I'm wondering if there's a simpler way to configure Cinema 4D for path mapping in remote rendering workloads. Ideally, we're looking for a generic solution that doesn't require mapping each path individually. We've explored using environment variables for Redshift (https://help.maxon.net/r3d/maya/en-us/Content/html/Redshift+Environment+Variables.html), but it didn't work for us. Is there an environment variable or configuration file where we can set general path mapping rules?

[ferdinand]

Hey @karthikbp,

Sorry about that. I'm not sure how the username got set to that and unfortunately I'm unable to change it. Can you guys change it for me to "karthikbp"?

I did rename your user to 'karthikbp'. You might have to adapt your login when you were logging in with your user name and not your mail handle.

We've encountered an issue with scenes like the Pyro Voxel scene in the Cinema 4D asset browser. When we cache the Pyro Output beforehand and submit, the renders come out completely black.

As lined out before, we need reproducible code examples and scenes. I cannot start debugging your code base for you. No code, means effectively no support here. When I cache some Pyro scene and then iterate over the 'assets' (i.e., scene dependencies, really unfortunate naming there), I see all my cache files being reflected. You said that you followed/copied an old thread of mine. Maybe you also copied the flag ASSETDATA_FLAG_TEXTURESONLY I used then (as that topic was about textures)?

Given this context, I'm wondering if there's a simpler way to configure Cinema 4D for path mapping in remote rendering workloads. Ideally, we're looking for a generic solution that doesn't require mapping each path individually.

Well, Cinema 4D supports relative paths and as you can see in my previous posting, the relative path tex is actually stored on a per document basis and not hard-coded. So you could modify that if you really wanted. I also showed you how you can abstractly iterate over a scene to find all paths, and then there is of course MSG_GETALLSSETS (i.e., GetAllAssetsNew in Python). I am not quite sure for which more generic solution you are looking here.

When you want to completely rewire all the path dependencies in a scene, you will have to do probably some manual work. You can also abstract away some manual labour in Material node graphs, as node spaces (when implemented by the owner) for example store what the texture node of that node space is (and which are the image ports on that texture node - but that is in accessible in Python atm).

I.e., with that you could prevent having to hard-code all texture node IDs for all render engines. But that is all future talk, when you do not have a working foundation.

What you want to be done here, is certainly possible, but I cannot debug your code for you, nor does it help much when you make claims of something not working without us being able to reproduce any of that. I personally would suggest using GetAllAssetsNew, but if you want to, you can also do it manually.

Cheers,
Ferdinand

[karthikbp]

Hi @ferdinand ,

Thanks for your response, I was not able to able to respond sooner as I was working on a different issue.

Here's the script that we used to print all the assets in the c4d scene file based off the scripts that you shared previously.

        import mxutils
        for element in mxutils.RecurseGraph(doc, True, True, True):
            # Now we iterate over its description to find parameters of DTYPE_FILENAME.
            data: c4d.BaseContainer
            pid: c4d.DescID
            for data, pid, _ in element.GetDescription(c4d.DESCFLAGS_GET_NONE):
                # Get data type of the last component of the DescID and then check if it is a filename.
                dtype: int = pid[pid.GetDepth() - 1].dtype
                if dtype == c4d.DTYPE_FILENAME:
                    print(f"Found path parameter with the label '{data[c4d.DESC_NAME]}' and value "
                        f"'{element[pid]}' in node '{element}'.")
        asset_data: list[dict] = []
        c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data)
        for item in asset_data:
            print(os.path.split(item.get("filename", ""))[-1])

And here's the output that we get while running it locally.

Found path parameter with the label 'Relative Path' and value 'tex' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 18149079616>'.
Found path parameter with the label 'OpenColorIO Config' and value '$(DEFAULT)' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 18149079616>'.
Found path parameter with the label 'Project Path' and value '/Users/my_user_name/Desktop/Pyro scene' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 18149079616>'.
Found path parameter with the label 'Project Name' and value 'Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 18149079616>'.
Found path parameter with the label 'File Path' and value '/Users/my_user_name/Desktop/Pyro scene/Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 18149079616>'.
Found path parameter with the label 'File' and value '' in node '<c4d.BaseObject object called RS Camera/RS Camera with ID 1057516 at 18444390464>'.
Found path parameter with the label 'Template Filename' and value '$prj_$frame.vdb' in node '<c4d.BaseObject object called Pyro Output/Pyro Output with ID 1059580 at 18448677568>'.
Found path parameter with the label 'Filename' and value '/Users/my_user_name/Desktop/Pyro scene/vol/Pyro_Voxel_v1_0001.vdb' in node '<c4d.BaseObject object called Pyro Output/Volume Loader with ID 1039866 at 18444442432>'.
Found path parameter with the label 'File' and value './$prj/$prj' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 18464012096>'.
Found path parameter with the label 'File' and value '' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 18464012096>'.
Found path parameter with the label 'OpenColorIO Configuration' and value '$OCIO' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'File' and value './Redshift/irradiance_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'File' and value './Redshift/irradiance_pointcloud_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'File' and value './Redshift/photon_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'Base Filename' and value '$prj_AOV' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'File' and value './Redshift/subsurface_scattering_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'.
Found path parameter with the label 'File' and value '' in node '<c4d.documents.BaseVideoPost object called RS Post-Effects/RS Post-Effects with ID 1040189 at 18440910144>'.
Pyro Voxel.c4d
file_80aa35ed1bf4db09~.hdr?name=ydr_minimalist_hdr.hdr&db=MaxonAssets.db (assets.maxon.net)
Pyro_Voxel_v1_0001.vdb
Pyro_Voxel_v1_0002.vdb
...
Pyro_Voxel_v1_0180.vdb

Here's the output that we get on a rendering machine on the same c4d scene file.

Found path parameter with the label 'Relative Path' and value 'tex' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 2363282316352>'.
Found path parameter with the label 'OpenColorIO Config' and value '$OCIO' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 2363282316352>'.
Found path parameter with the label 'Project Path' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 2363282316352>'.
Found path parameter with the label 'Project Name' and value 'Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 2363282316352>'.
Found path parameter with the label 'File Path' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene\Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called  with ID 110059 at 2363282316352>'.
Found path parameter with the label 'File' and value '' in node '<c4d.BaseObject object called RS Camera/RS Camera with ID 1057516 at 2363337280832>'.
Found path parameter with the label 'Template Filename' and value '$prj_$frame.vdb' in node '<c4d.BaseObject object called Pyro Output/Pyro Output with ID 1059580 at 2363336803648>'.
Found path parameter with the label 'File' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene\Pyro_Voxel\Pyro_Voxel' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 2363320759104>'.
Found path parameter with the label 'File' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 2363320759104>'.
Found path parameter with the label 'OpenColorIO Configuration' and value '$OCIO' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'File' and value './Redshift/irradiance_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'File' and value './Redshift/irradiance_pointcloud_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'File' and value './Redshift/photon_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'Base Filename' and value '$prj_AOV' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'File' and value './Redshift/subsurface_scattering_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'.
Found path parameter with the label 'File' and value '' in node '<c4d.documents.BaseVideoPost object called RS Post-Effects/RS Post-Effects with ID 1040189 at 2363328329536>'.
Pyro Voxel.c4d
file_80aa35ed1bf4db09~.hdr?name=ydr_minimalist_hdr.hdr&db=MaxonAssets.db (assets.maxon.net)
Pyro_Voxel_v1_0001.vdb

The main difference that we could notice is that somehow the rendering machine did not have the following line (pyro output/volume loader was missing)

Found path parameter with the label 'Filename' and value '/Users/my_user_name/Desktop/Pyro scene/vol/Pyro_Voxel_v1_0001.vdb' in node '<c4d.BaseObject object called Pyro Output/Volume Loader with ID 1039866 at 18444442432>'.

While submitting, we used relative paths and it still ended up giving us blank renders. Unfortunately, I'm not sure how I can share a reproducible for a rendering worker node (Let us know if we can setup a online video call so that we can share the issue that we are currently facing)

We're relatively new to Cinema 4D's internals, so any documentation or resources you could provide to help us better understand these issues would be greatly appreciated.

[ferdinand]

Hey @karthikbp,

No problem with the delay. But I must remind you of our support procedures:

• Please verify your account.
• Please provide reproducible data, e.g., a scene that shows the issue you are having. This was another perfect example for where I spent such much time on guessing what some devs might be doing, where a simple scene file would have made things easier. Print outs are nice but useless without context. I will not keep doing what I did here, meet new developers half way, it is simply not possible logistically. We are happy to help, but we need some data to work with. Something concrete we can run.

I would also recommend that you apply for Maxon Developer Program. I think your company had once MRD seats, but there are currently none as far as I can see.

About your Issue

Here's the script that we used to print all the assets in the c4d scene file based off the scripts that you shared previously.

You run in this script mxutils.RecurseGraph(doc, True, True, True), where my version used mxutils.RecurseGraph(doc, False, True, True). That was no accident, as this first parameter enables traversal of caches, which is not only expensive but usually irrelevant for most users, as caches are read only.

The main difference that we could notice is that somehow the rendering machine did not have the following line (pyro output/volume loader was missing)

The output you see there is part of a Python Output object cache. A VolumeLoader is the root element of all cached volumes caches¹, and then holds the individual volumes, usually data for each frame. The cache of the Output being empty means that it lacks the data to build it. Which likely means that something with your cache paths does not line up.

¹ - We have a bit conflicting terms here. So, there is once the notion of caches in a scene graph. That is also the kind of cache you enabled traversing when you set yieldCaches in RecurseGraph to True. Most BaseObjects have such cache, and there are two variant, but they all express the data generated by a parametric objects (a 'generator' in Cinema 4D slang). Read more about it in geometry_caches_s26.py. And then we have the Volume VDB caches on disk. For a VolumeLoader object to build its BaseObject:GetCache(), i.e., its virtual object hierarchy invisible to users, it requires access to files on disk - "the volume cache". And within this, the VolumeLoader is usually itself part of another objects cache, e.g., a Pyro Output object.

While submitting, we used relative paths and it still ended up giving us blank renders.

That sounds very unlikely, as thousands of people are using Pyro. But your claim got me a bit thinking. So we have this, the 'Cache Paths' parameter of a 'Pyro Output' object which is here the likely culprit for your issues.

and although it sort of implies that it uses relative paths when you press the 'Cache' button it very obviously does not.

So, my suspicion would be that you simply do not have relative paths there, and when you then just move the scene (on the same machine or onto another machine, the result should be same), the 'Pyro Output' cache building fails because it requires these VDB files on disk when "Use Cache" is enabled. If you would look into your scene graph, you would probably have a null object there in the cache.

---

But I still do not really understand what you are doing and why you are doing it. This strikes me a bit as one of the cases where someone asks us "What is 'a' + 1?". I can then either say "62" or say that this question does not make too much sense. And sometimes, especially more experienced developers then insist on storming that hill, although that is not really how you would tackle that problem in Cinema 4D.

In Cinema 4D, when you want to make a scene 'moveable', you just use 'Save Project with Assets ...'. In the API this would be then c4d.documents.SaveProject. This will handle everything for you, including this Pyro case. There could be of course cases where you have some render nodes and you want to save bandwidth and storage and therefore only want to store shared asset data once. But I would really contemplate twice if storming that hill is really worth it.

I have attached below a script which dissects a bit how to handle PYRO_OBJECT_CACHE_PATHS (there is a bit of an inconsistency how this parameter is handled in asset messages, I already talked with the simulation team about this).

Please provide tangible data for all further questions and follow-ups.

Cheers,
Ferdinand

"""Demos more ways how to traverse a scene and its data.

Can be run as a script manager script. You should have a Pyro Output object in the scene with a
cached simulation, and some node materials using some textures.

The script will traverse the scene graph abstractly, and among other things make all Pyro caches
relative to the scene document path.
"""

import re
import os
import typing

import c4d
import maxon
import mxutils

doc: c4d.documents.BaseDocument  # The currently active document.
op: c4d.BaseObject # The currently active object, can be None.

# An internal custom GUI that is not exposed in the public API, which is used to store multiple
# paths in a single string. Pyro leans into this GUI and extended it with some custom encoding.
CUSTOMGUI_PATHS: int = 300002000

# An expression to decompose a path element with in the list of 'paths' store in the parameter
# `PYRO_OBJECT_CACHE_PATHS` of a pyro object. A value of the parameter looks like this:
#
#   pyro_scene_v3_0001::.\vol\pyro_scene_v3_0001.vdb::0
#   pyro_scene_v4_0001::[0].\vol\pyro_scene_v4_0001.vdb::0
#   pyro_scene_v5_0001::[0].\vol\pyro_scene_v5_0001.vdb::0
#
# I.e., \n separated elements, and this expression captures the a line in the form:
#   <name>::[<prefix>]<path>::<postfix>
RE_PYRO_CACHE_PATHS: re.Pattern = re.compile(r"^(.*\:\:)([0])?(.*)(\:\:\d+)?$")

def ProcessCachePaths(doc: c4d.documents.BaseDocument, value: str) -> str:
    """Makes a (pyro) CUSTOMGUI_PATHS string path relative to a scene.

    This decomposes a DTYPE_STRING parameter using the PATHS custom GUI, used in the mode realized
    for Pyro. Note that there are other modes for this GUI, which will result in different encodings
    being used (so this function will not work for them).
    """
    docPath: str = os.path.normpath(doc.GetDocumentPath())
    if not docPath:
        raise RuntimeError("Cannot make paths relative to an unsaved document.")
    
    print(value, "\n")

    # Iterate over the lines in the value, and try to make them relative to the document path.
    lines: list[str] = []
    for line in value.split('\n'):
        if line.strip() == "":
            continue
        
        match: re.Match | None = RE_PYRO_CACHE_PATHS.match(line.strip())
        if not match:
            raise RuntimeError(f"Failed to match line '{line.strip()}' in '{value}'.")

        name: str = match.group(1).strip()
        prefix: str = match.group(2).strip() if isinstance(match.group(2), str) else ""
        cachePath: str = os.path.normpath(match.group(3).strip())
        postfix: str = match.group(4).strip() if isinstance(match.group(4), str) else ""

        if cachePath.startswith(docPath):
            cachePath = os.path.join(".", *cachePath[len(docPath)::].split(os.sep))

        lines.append(f"{name}{prefix}{cachePath}{postfix}")

    return "\n".join(lines)


def DemoAbstractTraversal() -> None:
    """Showcases more details of how to manually traverse a scene graph.
    """
    # So, as quite clearly stated in the last posting, this traversal was an example. It output you
    # all the parameters of type DTYPE_FILENAME in a scene. You can fashion this broader, but redoing
    # all the asset finding is not trivial as everything could be custom, hence the existence of
    # GetAllAssets().

    # As also stated, the last example also did not traverse the Maxon API scene graph, which is
    # where most of the modern material data is stored. The example below is a bit broader, as it
    # also traverse DTYPE_STRING parameters with the GUI PATHS, and checks out all Maxon API
    # node graphs. But it is still that - an example. You have to write your own code.

    # We again iterate over everything in the Cinema API scene graph, this is very much NOT a cheap
    # operation and should be avoided in production code when possible. In this example and my last
    # I deliberately disable cache traversal, and I have seen that you did enable it. Do not do that,
    # this makes it so much more expensive. Even a small scene with a few dozen objects will usually
    # have hundreds if not thousands of scene elements when you really go everywhere. When you then
    # traverse the description of each, you could easily end up with hundreds of thousands of 
    # operations, rendering this all very slow. 
    element: c4d.BaseList2D
    for element in mxutils.RecurseGraph(doc, False, True, True):
        print(f"Traversing Cinema API scene element '{element}")

        # --- Abstract parameter processing --------------------------------------------------------

        # Here we iterate over all parameters of the current scene element, and check if they are
        # of type DTYPE_FILENAME or DTYPE_STRING with the GUI PATHS. So, that we also catch Pyro
        # cache paths. But there are MANY more custom GUIs and some of them also store paths. Some
        # of them, such as CUSTOMGUI_PATHS, are not even exposed in the public API.

        for data, pid, _ in element.GetDescription(c4d.DESCFLAGS_GET_NONE):
            # Step over everything that is not a parameter, of type DTYPE_FILENAME or DTYPE_STRING.
            # For strings, only check string with the GUI 'PATHS'. This is mot even remotely every
            # way how paths are stored in a scene. Just have a look at all CUSTOMGUI_* symbols in
            # the C++ API, to het a sense. And these are only the exposed ones, there are more, as
            # demonstrated by the CUSTOMGUI_PATHS symbol below.
            dtype: int = pid[pid.GetDepth() - 1].dtype
            if dtype not in (c4d.DTYPE_FILENAME, c4d.DTYPE_STRING) or \
               dtype == c4d.DTYPE_STRING and data[c4d.DESC_CUSTOMGUI] != CUSTOMGUI_PATHS:
               continue
            
            # Make things a bit easier to read, since CUSTOMGUI_PATHS splits its values by newlines.
            value: str = str(element[pid]).replace("\n", "")
            print(f"\tFound parameter '{data[c4d.DESC_NAME]}' with value '{value}'"
                  f"({data[c4d.DESC_CUSTOMGUI] = }, {pid[0].dtype = }).")

        # --- Special Pyro handling ----------------------------------------------------------------

        # Here we specifically handle Pyro object when we come across them. We use the special
        # function ProcessCachePaths() to unpack a CUSTOMGUI_PATHS parameter, which is in Pyro
        # mode.

        if element.GetType() == c4d.Opyro:
            lines: list[str] = []
            cachePaths: str = element[c4d.PYRO_OBJECT_CACHE_PATHS]
            lines.append(f"\tOld pyro cache paths:")
            lines += [f"\t\t{line}" for line in cachePaths.split('\n') if line.strip()]
            cachePaths = ProcessCachePaths(doc, cachePaths)
            lines.append(f"\tRelative pyro cache paths:")
            lines += [f"\t\t{line}" for line in cachePaths.split('\n') if line.strip()]
            print("\n".join(lines))

        # --- Branching of into Maxon API scene graph traversal ------------------------------------

        # Here we branch of into the Maxon API scene graph traversal.
        if not element.IsNodeBased():
            continue

        # Each element in a Cinema API scene graph can have multiple NimbusBaseInterface references,
        # NimbusBaseRef's, associated with it. You can think of a nimbus reference as a set of 
        # purpose bound node graphs associated with a scene element. E.g., you have a node that has
        # nimbus reference for "node materials", and attached to that reference would then be one to
        # many node graphs that realize a material system for a node space (i.e., effectively a
        # render engine in this case).

        # We iterate over all nimbus reference, node space pairs of the current element ...
        nodeSpaceId: maxon.Id
        nimbusRef: maxon.NimbusBaseRef
        for nodeSpaceId, nimbusRef in element.GetAllNimbusRefs():
            # .. get the graph for the current nimbus reference/space pair, and then the root node
            # of the graph.
            graph: maxon.NodesGraphModelRef = nimbusRef.GetGraph(nodeSpaceId)
            root: maxon.GraphNode = graph.GetViewRoot()

            print(f"\tProcessing Maxon API node space '{nodeSpaceId}' with node handler '{nimbusRef}'.")

            # The Nodes API has the a bit unfortunate terminology that most entities in it are
            # a maxon.GraphNode, including things like input and output ports. That is because
            # #GraphNode refers to a node in a tree in which the Nodes API stores all entities
            # of graph. Not the graph the user sees.

            # Now we iterate over all nodes representing the input ports in the current graph. We 
            # could also search for what the Nodes API calls 'true nodes', i.e., #GraphNode entities
            # that represent what the use would consider a node, to search there its attributes,
            # i.e., things which cannot be wired together. But usually a node implementation
            # should not store there paths, but you never know what some node space implementation
            # does.
            inputPort: maxon.GraphNode
            for inputPort in root.GetInnerNodes(maxon.NODE_KIND.INPORT, True):
                value: maxon.Data = inputPort.GetPortValue()

                print (f"\tFound input port '{inputPort}' with value '{value}'.")
                if not isinstance(value, maxon.Url):
                    continue

                # We found some input port holding some form of URL. Get the owning true node
                # and print the information. We have to move up to levels, because a local true node
                # tree looks like this:
                #
                #   someNode@someHash             # The true node, a GraphNode.
                #       >                         # The root off all input ports, also a GraphNode.
                #           myInput               # An input port, also a GraphNode.
                #           anotherInput          # Another input port, also a GraphNode
                #           ...
                #       <                         # The root of all output ports, which is a GraphNode.
                #           myOutput              # An output port, also a GraphNode.
                #           anotherOutput         # Another output port, also a GraphNode
                #           ...
                #                                                        
                name: str = inputPort.GetValue("effectivename")
                inputPorts: maxon.GraphNode = inputPort.GetParent()
                trueNode: maxon.GraphNode = inputPorts.GetParent()
                if not trueNode or not trueNode.GetKind() == maxon.NODE_KIND.NODE:
                    print(f"Failed to find true node for input port '{inputPort}'.")
                    continue

                print(f"Found URL input port {inputPort} with value '{str(value)}', owned by the"
                      f"true node {trueNode}.")

    c4d.EventAdd()

def DemoAssetTraversal() -> None:
    """Demonstrates how to handle assets in a scene.
    """
    # We get all assets in the scene and start iterating over them. For what you want to do, you
    # porbably want to use the flag ASSETDATA_FLAG_MULTIPLEUSE, as will otherwise miss entities
    # that use the same asset multiple times, e.g., two materials using the same texture.
    container: list[dict[str, typing.Any]] = []
    if not c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_MULTIPLEUSE, container):
        raise RuntimeError("Failed to get all assets from the document.")
    
    # A little lookup table to avoid processing the same Pyro object multiple times.
    visitedPyro: set[bytes] = set()

    for asset in container:
        url: str = asset.get("filename", "")
        name: str = asset.get("assetname", "")
        pid: int = asset.get("paramId", 0)
        owner: c4d.BaseList2D | None = asset.get("owner", None)
        nodePath: str = asset.get("nodePath", "")
        nodeSpace: str = asset.get("nodeSpace", "")

        print(f"Processing asset '{url}' with the owner '{owner.GetName() if owner else ''}' and pid '{pid}'.")

        # Which will produce output for Pyro object like this, which is not very helpful for 
        # updating pyro caches, as it refers to internal parts of the Pyro object cache (i.e., the
        # cache in the scene graph, not the Pyro cache on disk), that are readonly to the user.

        # > Processing asset 'D:\temp\pyro\vol\pyro_scene_v7_0025.vdb' with the owner 'Pyro Output' 
        # > and pid '6000'.

        # We cannot even use the parameter ID, because that ID is used in the part of the volume 
        # loader cache inside the Pyro Output cache, that is responsible for rendering frame 25 in
        # this case.

        # So, to fix that, we have to ignore most of the asset data, and just manually fix Pyro 
        # objects (or do more complex traversal here as shown in #AbstractTraversal()).

        # The issue is that we get here yielded things inside a cache, (the scene graph cache
        # that of the Pyro object), which makes very little sense at least in the public API, as
        # these scene elements are effectively readonly to the user.

        # We could also dos this more generically without checking the cache element being a Volume
        # Loader, but I kept this safe. 
        if (isinstance(owner, c4d.BaseObject) and owner.GetType() == c4d.Ovolumeloader and 
            owner.GetCacheParent()):
            while owner.GetCacheParent():
                owner = owner.GetCacheParent()

            if owner.GetType() == c4d.Opyro:
                # We have already processed this Pyro object, so skip it.
                uuid: bytes = bytes(owner.FindUniqueID(c4d.MAXON_CREATOR_ID))
                if uuid in visitedPyro:
                    continue

                cachePaths: str = ProcessCachePaths(doc, owner[c4d.PYRO_OBJECT_CACHE_PATHS])
                owner[c4d.PYRO_OBJECT_CACHE_PATHS] = cachePaths
                cachePaths = cachePaths.replace("\n", "")
                print(f"Updated Pyro object cache paths to '{cachePaths}'.")

                # Now that we have done this, we should set a marker so that we do not process
                # this Pyro object again, because there is an asset for frame in its cache. 
                # Technically it would probably not hurt to run ProcessCachePaths() on already 
                # relative paths, but is cleaner like this.
                visitedPyro.add(uuid)

        # Last but not least, here is some abstracted node graph handling. This could be done better
        # and more complex, but that would be up to you.

        # This asset is for something Maxon API scene graph related, very likely a node material.
        if not nodePath or not nodeSpace or not isinstance(owner, c4d.BaseList2D):
            continue

        print(f"Processing node graph.")

        # As explained in the earlier example, we grab the graph referenced by the nodeSpace in the
        # asset data, by iterating over all nimbus references of the owner. We could also use
        # NodeMaterial, and then could directly get the graph, but would then not cover non-material
        # cases.
        sid: maxon.Id
        nimbusRef: maxon.NimbusBaseRef
        for sid, nimbusRef in owner.GetAllNimbusRefs():
            if sid != maxon.Id(nodeSpace):
                continue

            # Get the graph for the node space referenced by the asset data and the root node of it.
            graph: maxon.NodesGraphModelRef = nimbusRef.GetGraph(sid)
            root: maxon.GraphNode = graph.GetViewRoot()

            # A little helper function to untangle the different notions of what different render
            # engines pass as #nodePath in the asset data.
            def FindNode(node: maxon.GraphNode, path: str) -> maxon.GraphNode | None:
                """Due to the rather loose interpretation of node paths of some asset handlers,
                we have to write this helper function to find what is meant in a graph.

                A node path could look like this:

                    image@IPTcQlkjIRrgM33F_dJevQ<url

                Where "image@IPTcQlkjIRrgM33F_dJevQ" is the ID of a true node, the "Image" node in a
                Standard Render graph, "<" is its input ports GRaphNode root, and "url" the input
                port for the texture of the URL. We effectively walk the node tree here while 
                decomposing the node path.
                """
                child: maxon.GraphNode
                for child in node.GetChildren(mask=maxon.NODE_KIND.ALL_MASK):
                    cid: str = str(child.GetId())
                    if path.startswith(cid):
                        remainder: str = path[len(cid)::]
                        if remainder == "":
                            return child
                        return FindNode(child, remainder)
                    
                raise RuntimeError(f"Failed to find node with path '{path}' in {node}.")
            
            # Find the entity referenced by the nodePath in the asset data. This should by design be
            # a port, but not everyone is sticking to that rule.
            port: maxon.GraphNode = FindNode(root, nodePath)
            if port.IsNullValue():
                print(f"Failed to find asset data with path '{nodePath}' in {graph}.")
                continue
            
            # The value referenced in the asset data as a maxon.Url.
            assetValue: maxon.Url = maxon.Url(url)

            # As mentioned, #port, should be a port. But Redshift for example will give you a true 
            # node (the texture sample node that holds the URL port), but not the port. We could 
            # hardcode thing here (e.g., if nodeType == "TextureSampler": port = "url") but we
            # would have to do that then for every render engine which does what Redshift does (
            # never really checked what V_Ray, Arnold, and others do). A more generic way is to 
            # search the input ports of the node for the asset value.
            if port.GetKind() == maxon.NODE_KIND.NODE:
                for child in port.GetChildren(mask=maxon.NODE_KIND.INPORT):
                    if child.GetPortValue() == assetValue:
                        port = child
                        break

            # If we still do not have a port, then we cannot do anything with the asset data, as
            # we have no idea how to use it.
            if port.GetKind() != maxon.NODE_KIND.INPORT:
                print(f"Failed to find asset data with path '{nodePath}' in {graph}.")
                continue
            
            # Get the value of the port we found, and make sure it holds the value we expect. It 
            # would be a bit unusual that a node space does not use maxon.Url for URLs, but
            # not impossible.
            currentValue: maxon.Data = port.GetPortValue()
            if not isinstance(currentValue, maxon.Url) or currentValue != assetValue:
                print(f"Failed to find asset data with path '{nodePath}' in {graph}.")
                continue

            # This is then how we would write a new value to that "asset". Remember that URLs in
            # the Maxon API are true URLs, i.e., the require a scheme/protocol, such as "file:///".
            newUrl: maxon.Url = maxon.Url("file:///c:/bob/is/your/uncle.jpg")
            with graph.BeginTransaction() as transaction:
                port.SetPortValue(maxon.Url("file:///c:/bob/is/your/uncle.jpg"))
                transaction.Commit()

            print(f"Updated node asset reference on {port} with '{newUrl}'.")


def main() -> None:
    """Called by Cinema 4D when the script is being executed.
    """

    DemoAbstractTraversal()
    print("\n" + "-" * 80 + "\n")
    DemoAssetTraversal()

if __name__ == '__main__':
    main()

[karthikbp]

Hi @ferdinand ,

Thanks for your input so far. I think we are getting really close and I thank you for your patience.

Please verify your account.

I never got an email from Maxon to verify my email for some reason. Is it possible to send that email again? Or is there a way to request to send the email again?

Please provide reproducible data, e.g., a scene that shows the issue you are having.

Sorry, it took me a while to figure out a way to provide some reproducible script and data. Here are the steps that I take to reproduce the issue that I face.

1. I use the "Pyro Voxel" scene from the Cinema 4D asset browser.

2. I run the following script (the script that you shared for the Burning Pan scene)

import c4d
import os

doc: c4d.documents.BaseDocument # The active document

def main() -> None:
    """Called by Cinema 4D when the script is being executed.
    """
    print("===================")
    #scene_file = "/insert/path/to/scene/file/pyro_voxel.c4d"

    #doc = c4d.documents.LoadDocument(scene_file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS)
    asset_list = []

    asset_data: list[dict] = []
    val = c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data)
    for item in asset_data:
        print(os.path.split(item.get("filename", ""))[-1])

    print ("Done.")
    print("===========")


if __name__ == '__main__':
    main()

3. This gives me the output

===================
pyro_voxel.c4d
Pyro_Voxel_v2_0001.vdb
Pyro_Voxel_v2_0002.vdb
Pyro_Voxel_v2_0003.vdb
Pyro_Voxel_v2_0004.vdb
Pyro_Voxel_v2_0005.vdb
Pyro_Voxel_v2_0006.vdb
Pyro_Voxel_v2_0007.vdb
Pyro_Voxel_v2_0008.vdb
Pyro_Voxel_v2_0009.vdb
Pyro_Voxel_v2_0010.vdb
Pyro_Voxel_v2_0011.vdb
Pyro_Voxel_v2_0012.vdb
Pyro_Voxel_v2_0013.vdb
Pyro_Voxel_v2_0014.vdb
Pyro_Voxel_v2_0015.vdb
Pyro_Voxel_v2_0016.vdb
Pyro_Voxel_v2_0017.vdb
Pyro_Voxel_v2_0018.vdb
Pyro_Voxel_v2_0019.vdb
Pyro_Voxel_v2_0020.vdb
Pyro_Voxel_v2_0021.vdb
Pyro_Voxel_v2_0022.vdb
Pyro_Voxel_v2_0023.vdb
Pyro_Voxel_v2_0024.vdb
Pyro_Voxel_v2_0025.vdb
Pyro_Voxel_v2_0026.vdb
Pyro_Voxel_v2_0027.vdb
Pyro_Voxel_v2_0028.vdb
Pyro_Voxel_v2_0029.vdb
Pyro_Voxel_v2_0030.vdb
Pyro_Voxel_v2_0031.vdb
Pyro_Voxel_v2_0032.vdb
Pyro_Voxel_v2_0033.vdb
Pyro_Voxel_v2_0034.vdb
Pyro_Voxel_v2_0035.vdb
Pyro_Voxel_v2_0036.vdb
Pyro_Voxel_v2_0037.vdb
Pyro_Voxel_v2_0038.vdb
Pyro_Voxel_v2_0039.vdb
Pyro_Voxel_v2_0040.vdb
Pyro_Voxel_v2_0041.vdb
Pyro_Voxel_v2_0042.vdb
Pyro_Voxel_v2_0043.vdb
Pyro_Voxel_v2_0044.vdb
Pyro_Voxel_v2_0045.vdb
Pyro_Voxel_v2_0046.vdb
Pyro_Voxel_v2_0047.vdb
Pyro_Voxel_v2_0048.vdb
Pyro_Voxel_v2_0049.vdb
Pyro_Voxel_v2_0050.vdb
Pyro_Voxel_v2_0051.vdb
Pyro_Voxel_v2_0052.vdb
Pyro_Voxel_v2_0053.vdb
Pyro_Voxel_v2_0054.vdb
Pyro_Voxel_v2_0055.vdb
Pyro_Voxel_v2_0056.vdb
Pyro_Voxel_v2_0057.vdb
Pyro_Voxel_v2_0058.vdb
Pyro_Voxel_v2_0059.vdb
Pyro_Voxel_v2_0060.vdb
Pyro_Voxel_v2_0061.vdb
Pyro_Voxel_v2_0062.vdb
Pyro_Voxel_v2_0063.vdb
Pyro_Voxel_v2_0064.vdb
Pyro_Voxel_v2_0065.vdb
Pyro_Voxel_v2_0066.vdb
Pyro_Voxel_v2_0067.vdb
Pyro_Voxel_v2_0068.vdb
Pyro_Voxel_v2_0069.vdb
Pyro_Voxel_v2_0070.vdb
Pyro_Voxel_v2_0071.vdb
Pyro_Voxel_v2_0072.vdb
Pyro_Voxel_v2_0073.vdb
Pyro_Voxel_v2_0074.vdb
Pyro_Voxel_v2_0075.vdb
Pyro_Voxel_v2_0076.vdb
Pyro_Voxel_v2_0077.vdb
Pyro_Voxel_v2_0078.vdb
Pyro_Voxel_v2_0079.vdb
Pyro_Voxel_v2_0080.vdb
Pyro_Voxel_v2_0081.vdb
Pyro_Voxel_v2_0082.vdb
Pyro_Voxel_v2_0083.vdb
Pyro_Voxel_v2_0084.vdb
Pyro_Voxel_v2_0085.vdb
Pyro_Voxel_v2_0086.vdb
Pyro_Voxel_v2_0087.vdb
Pyro_Voxel_v2_0088.vdb
Pyro_Voxel_v2_0089.vdb
Pyro_Voxel_v2_0090.vdb
Pyro_Voxel_v2_0091.vdb
Pyro_Voxel_v2_0092.vdb
Pyro_Voxel_v2_0093.vdb
Pyro_Voxel_v2_0094.vdb
Pyro_Voxel_v2_0095.vdb
Pyro_Voxel_v2_0096.vdb
Pyro_Voxel_v2_0097.vdb
Pyro_Voxel_v2_0098.vdb
Pyro_Voxel_v2_0099.vdb
Pyro_Voxel_v2_0100.vdb
Pyro_Voxel_v2_0101.vdb
Pyro_Voxel_v2_0102.vdb
Pyro_Voxel_v2_0103.vdb
Pyro_Voxel_v2_0104.vdb
Pyro_Voxel_v2_0105.vdb
Pyro_Voxel_v2_0106.vdb
Pyro_Voxel_v2_0107.vdb
Pyro_Voxel_v2_0108.vdb
Pyro_Voxel_v2_0109.vdb
Pyro_Voxel_v2_0110.vdb
Pyro_Voxel_v2_0111.vdb
Pyro_Voxel_v2_0112.vdb
Pyro_Voxel_v2_0113.vdb
Pyro_Voxel_v2_0114.vdb
Pyro_Voxel_v2_0115.vdb
Pyro_Voxel_v2_0116.vdb
Pyro_Voxel_v2_0117.vdb
Pyro_Voxel_v2_0118.vdb
Pyro_Voxel_v2_0119.vdb
Pyro_Voxel_v2_0120.vdb
Pyro_Voxel_v2_0121.vdb
Pyro_Voxel_v2_0122.vdb
Pyro_Voxel_v2_0123.vdb
Pyro_Voxel_v2_0124.vdb
Pyro_Voxel_v2_0125.vdb
Pyro_Voxel_v2_0126.vdb
Pyro_Voxel_v2_0127.vdb
Pyro_Voxel_v2_0128.vdb
Pyro_Voxel_v2_0129.vdb
Pyro_Voxel_v2_0130.vdb
Pyro_Voxel_v2_0131.vdb
Pyro_Voxel_v2_0132.vdb
Pyro_Voxel_v2_0133.vdb
Pyro_Voxel_v2_0134.vdb
Pyro_Voxel_v2_0135.vdb
Pyro_Voxel_v2_0136.vdb
Pyro_Voxel_v2_0137.vdb
Pyro_Voxel_v2_0138.vdb
Pyro_Voxel_v2_0139.vdb
Pyro_Voxel_v2_0140.vdb
Pyro_Voxel_v2_0141.vdb
Pyro_Voxel_v2_0142.vdb
Pyro_Voxel_v2_0143.vdb
Pyro_Voxel_v2_0144.vdb
Pyro_Voxel_v2_0145.vdb
Pyro_Voxel_v2_0146.vdb
Pyro_Voxel_v2_0147.vdb
Pyro_Voxel_v2_0148.vdb
Pyro_Voxel_v2_0149.vdb
Pyro_Voxel_v2_0150.vdb
Pyro_Voxel_v2_0151.vdb
Pyro_Voxel_v2_0152.vdb
Pyro_Voxel_v2_0153.vdb
Pyro_Voxel_v2_0154.vdb
Pyro_Voxel_v2_0155.vdb
Pyro_Voxel_v2_0156.vdb
Pyro_Voxel_v2_0157.vdb
Pyro_Voxel_v2_0158.vdb
Pyro_Voxel_v2_0159.vdb
Pyro_Voxel_v2_0160.vdb
Pyro_Voxel_v2_0161.vdb
Pyro_Voxel_v2_0162.vdb
Pyro_Voxel_v2_0163.vdb
Pyro_Voxel_v2_0164.vdb
Pyro_Voxel_v2_0165.vdb
Pyro_Voxel_v2_0166.vdb
Pyro_Voxel_v2_0167.vdb
Pyro_Voxel_v2_0168.vdb
Pyro_Voxel_v2_0169.vdb
Pyro_Voxel_v2_0170.vdb
Pyro_Voxel_v2_0171.vdb
Pyro_Voxel_v2_0172.vdb
Pyro_Voxel_v2_0173.vdb
Pyro_Voxel_v2_0174.vdb
Pyro_Voxel_v2_0175.vdb
Pyro_Voxel_v2_0176.vdb
Pyro_Voxel_v2_0177.vdb
Pyro_Voxel_v2_0178.vdb
Pyro_Voxel_v2_0179.vdb
Pyro_Voxel_v2_0180.vdb
ydr_minimalist_hdr.hdr
Done.
===========

4. Now uncomment the scene_file and doc lines and then run the command again. I get something like

===================
pyro_voxel.c4d
Pyro_Voxel_v2_0001.vdb
ydr_minimalist_hdr.hdr
Done.
===========

Notice that it only has 1 cache file and I believe that the issue is that we are not loading the document with all its caches correctly which leads to the blank renders.
Maybe something to do with LoadDocument? Am I missing some flag that will get all the assets including the caches (I've tried multiple flags but haven't had much success)?

We use the LoadDocument function to set the scene file to render over here: https://github.com/aws-deadline/deadline-cloud-for-cinema-4d/blob/mainline/src/deadline/cinema4d_adaptor/Cinema4DClient/cinema4d_handler.py#L299-L319

Any help with fixing this would be really helpful.

Thanks,
Karthik

[ferdinand]

Hey @karthikbp,

Please read my previous posting, I very likely already explained there what your issue is and provided a solution. When you just move that file "/insert/path/to/scene/file/pyro_voxel.c4d" on disk, you will break its cache paths.

Otherwise, you would have to provide a file/project that you made relative and which is rendering black. But I am pretty sure that you have hardcoded paths in your PYRO_OBJECT_CACHE_PATHS parameters as explained above, which will break when you just move the scene, explaining the output. The reason your script is doing what it is doing is because that scene does not have its passes executed (among other things - caches built) when you loaded it from disk. There is also the oddity that Pyro considers frame zero not part of the animation, i.e., you must nudge Pyro to frame 1 for it to start building its caches and therefore in this case also asset messages.

What the simulation team did with the asset messages was not so good, as they report assets inside caches and do not report the actual parameter. But as lined out above, I already told them that.

As mentioned before, you should also use ASSETDATA_FLAG_MULTIPLEUSE and your node material handling is completely wrong (at least form what I saw on GitHub). Both can lead to incorrect renderings.

Cheers,
Ferdinand

PS: I have resent your validation mail to the e-mail handle your account is bound to.

When you write your script like this, it will also print the assets when you load the file from disk. But that is all more a technicality. Please read and follow what I wrote in my last post. This will only work when you do not move scene_file, as by default it will use absolute paths for the VDB files.

import c4d
import os

doc: c4d.documents.BaseDocument # The active document

def main() -> None:
    """Called by Cinema 4D when the script is being executed.
    """
    print("===================")
    scene_file = r"D:\temp\pyro\pyro_scene.c4d"
    doc = c4d.documents.LoadDocument(scene_file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS)
    doc.SetTime(c4d.BaseTime(1, 30)) # Frame 1 for a 30 FPS document.
    # Build the animations, chaches, and expressions for that 1st frame. Please see threads about prerolling for how
    # to preroll properly, the simple approach I use here, will not work for all complex simulation prerolling tasks.
    doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE)
    asset_list = []

    asset_data: list[dict] = []
    val = c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data)
    for item in asset_data:
        print(os.path.split(item.get("filename", ""))[-1])

    print ("Done.")
    print("===========")


if __name__ == '__main__':
    main()

[karthikbp]

Hi @ferdinand ,

Firstly, I would like to thank you for your help so far and we have now been able to get successful renders. Let me try replying to your previous response.

I am pretty sure that you have hardcoded paths in your PYRO_OBJECT_CACHE_PATHS parameters as explained above, which will break when you just move the scene, explaining the output

Yup. I think I forgot to mention this in my previous post but I've been using relative paths while submitting. To get relative paths, we have been using "Window" > "Project Asset Inspector" and then clicking on all the assets and "Localize Filenames". Also we have been saving the scenes with "Save Project with Assets"

The reason your script is doing what it is doing is because that scene does not have its passes executed (among other things - caches built) when you loaded it from disk.

Thanking you for sharing this. Adding doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE) seems to have fixed the issue for us and we are able to get successful renders.

But I've a question. Can we add ExecutePasses before rendering any Cinema 4D scene file? I've not experienced issues so far after adding this line but I'd like to know your thoughts as well if adding this line for even still images (i.e. single frame) has any drawbacks?

There is also the oddity that Pyro considers frame zero not part of the animation, i.e., you must nudge Pyro to frame 1 for it to start building its caches and therefore in this case also asset messages.

I've a question about this. What are the drawbacks of not adding doc.SetTime(c4d.BaseTime(1, 30)) ? I've seen that even without adding the line the renders have been successful.

As mentioned before, you should also use ASSETDATA_FLAG_MULTIPLEUSE and your node material handling is completely wrong (at least form what I saw on GitHub). Both can lead to incorrect renderings.

Yup. We will be reworking our code to just use relative paths for assets or using the "Save Project with Assets" as you suggested so that we don't need to apply specific path mapping.

PS: I have resent your validation mail to the e-mail handle your account is bound to.

Thanks, I've verified the email now.

[ferdinand]

Hey @karthikbp,

Yup. I think I forgot to mention this in my previous post but I've been using relative paths while submitting. To get relative paths, we have been using "Window" > "Project Asset Inspector" and then clicking on all the assets and "Localize Filenames". Also we have been saving the scenes with "Save Project with Assets"

First of all, good to hear that you made progress. But the verbosity of your approach should not be necessary, simply invoking 'Save Project with Assets' should be enough to collect all scene dependencies and make them relative to the document. You can also do this programmatically via c4d.documents.SaveProject .

Thanking you for sharing this. Adding doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE) seems to have fixed the issue for us and we are able to get successful renders. But I've a question. Can we add ExecutePasses before rendering any Cinema 4D scene file? I've not experienced issues so far after adding this line but I'd like to know your thoughts as well if adding this line for even still images (i.e. single frame) has any drawbacks?

Hm, that is genuinely concerning. Rendering a scene should entail its scene state being built on its own (in Cinema slang: executing the passes). When you really must do this before rendering a file on a node, something is very wrong there. I suspect that this is not absolutely true, and something more specific applies like "when for a pyro sim we render the first frame isolated on a node, then it is black unless we do this" or that you for example try to start rendering a simulation - which has not been cached - from a frame other than the first (then you will see nothing or incorrect output unless you preroll the simulation). But in any case, this would hint at either a substantial bug on our or your side.

I've a question about this. What are the drawbacks of not adding doc.SetTime(c4d.BaseTime(1, 30)) ? I've seen that even without adding the line the renders have been successful.

First of all, please do not copy my code. I wrote this in a quick and dirty fashion because I knew the parameters of my/your scene. In real life you would have to get the first frame of the animation (c4d.RDATA_FRAMEFROM ) from the render data and then its FPS, and then construct a time like c4d.BaseTime(fist + 1, fps).

Why I did this is due to the oddity of how Pyro builds its caches. You wanted to see there the assets reported, which will only work when the Pyro Output node has built its cache due to how irregular MSG_GETALLASSETS has been implemented for Pyro. This is tied to the fact that Pyro will not generate output for the first frame of its simulation (loosely speaking frame zero), so I advance the scene state to frame 1 and then execute the passes so that this document has its scene state built for that first frame.

I am on another machine today, and I think I already deleted my Pyro temp folder on my other machine, but I am pretty sure when you remove that line, the script will still print nothing despite building the caches, because on frame zero the Pyro sim will not build anything. For rendering things, you should not need any of this.

Cheers,
Ferdinand

[karthikbp]

Hi @ferdinand ,

when for a pyro sim we render the first frame isolated on a node, then it is black unless we do this

Yup. We only saw an improvement for Pyro scenes at our end. Other scenes work correctly even without adding the doc.ExecutePasses line.

[ferdinand]

Hey @karthikbp,

I cannot say much else without a scene file and example. What you are telling me sounds wrong, you should see at least the scene geometry on the 0st frame. For the pan scene we used above, it actually takes up to frame 2 (i.e., 0, 1, 2) before you see Pyro volumes. There could be a bug in Pyro for how it handles rendering a singular frame on a machine (which I think is what you are doing), or you are doing something wrong.

Executing the passes should neither generate output where naturally is none, nor should you have to execute them before rendering a frame. As lined out before, when you do not have a cached simulation scene, you must preroll it before rendering. I.e., you cannot jump directly to frame 20 and render it, you first must execute the passes up to that frame.

We are starting to run here in circles. Please consider applying for MRD as lined out here and via chat.

Cheers,
Ferdinand

[karthikbp]

Hi @ferdinand ,

We are starting to run here in circles. Please consider applying for MRD as lined out here and via chat.

I think that makes sense. I'll apply for MRD right away.