Path mapping in Cinema 4D using Redshift does not work
-
Hey @karthikbp,
Sorry about that. I'm not sure how the username got set to that and unfortunately I'm unable to change it. Can you guys change it for me to "karthikbp"?
I did rename your user to 'karthikbp'. You might have to adapt your login when you were logging in with your user name and not your mail handle.
We've encountered an issue with scenes like the Pyro Voxel scene in the Cinema 4D asset browser. When we cache the Pyro Output beforehand and submit, the renders come out completely black.
As lined out before, we need reproducible code examples and scenes. I cannot start debugging your code base for you. No code, means effectively no support here. When I cache some Pyro scene and then iterate over the 'assets' (i.e., scene dependencies, really unfortunate naming there), I see all my cache files being reflected. You said that you followed/copied an old thread of mine. Maybe you also copied the flag
ASSETDATA_FLAG_TEXTURESONLYI used then (as that topic was about textures)?
Given this context, I'm wondering if there's a simpler way to configure Cinema 4D for path mapping in remote rendering workloads. Ideally, we're looking for a generic solution that doesn't require mapping each path individually.
Well, Cinema 4D supports relative paths and as you can see in my previous posting, the relative path
texis actually stored on a per document basis and not hard-coded. So you could modify that if you really wanted. I also showed you how you can abstractly iterate over a scene to find all paths, and then there is of courseMSG_GETALLSSETS(i.e.,GetAllAssetsNewin Python). I am not quite sure for which more generic solution you are looking here.When you want to completely rewire all the path dependencies in a scene, you will have to do probably some manual work. You can also abstract away some manual labour in Material node graphs, as node spaces (when implemented by the owner) for example store what the texture node of that node space is (and which are the image ports on that texture node - but that is in accessible in Python atm).
I.e., with that you could prevent having to hard-code all texture node IDs for all render engines. But that is all future talk, when you do not have a working foundation.
What you want to be done here, is certainly possible, but I cannot debug your code for you, nor does it help much when you make claims of something not working without us being able to reproduce any of that. I personally would suggest using
GetAllAssetsNew, but if you want to, you can also do it manually.Cheers,
Ferdinand -
Hi @ferdinand ,
Thanks for your response, I was not able to able to respond sooner as I was working on a different issue.
Here's the script that we used to print all the assets in the c4d scene file based off the scripts that you shared previously.
import mxutils for element in mxutils.RecurseGraph(doc, True, True, True): # Now we iterate over its description to find parameters of DTYPE_FILENAME. data: c4d.BaseContainer pid: c4d.DescID for data, pid, _ in element.GetDescription(c4d.DESCFLAGS_GET_NONE): # Get data type of the last component of the DescID and then check if it is a filename. dtype: int = pid[pid.GetDepth() - 1].dtype if dtype == c4d.DTYPE_FILENAME: print(f"Found path parameter with the label '{data[c4d.DESC_NAME]}' and value " f"'{element[pid]}' in node '{element}'.") asset_data: list[dict] = [] c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data) for item in asset_data: print(os.path.split(item.get("filename", ""))[-1])And here's the output that we get while running it locally.
Found path parameter with the label 'Relative Path' and value 'tex' in node '<c4d.documents.BaseDocument object called with ID 110059 at 18149079616>'. Found path parameter with the label 'OpenColorIO Config' and value '$(DEFAULT)' in node '<c4d.documents.BaseDocument object called with ID 110059 at 18149079616>'. Found path parameter with the label 'Project Path' and value '/Users/my_user_name/Desktop/Pyro scene' in node '<c4d.documents.BaseDocument object called with ID 110059 at 18149079616>'. Found path parameter with the label 'Project Name' and value 'Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called with ID 110059 at 18149079616>'. Found path parameter with the label 'File Path' and value '/Users/my_user_name/Desktop/Pyro scene/Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called with ID 110059 at 18149079616>'. Found path parameter with the label 'File' and value '' in node '<c4d.BaseObject object called RS Camera/RS Camera with ID 1057516 at 18444390464>'. Found path parameter with the label 'Template Filename' and value '$prj_$frame.vdb' in node '<c4d.BaseObject object called Pyro Output/Pyro Output with ID 1059580 at 18448677568>'. Found path parameter with the label 'Filename' and value '/Users/my_user_name/Desktop/Pyro scene/vol/Pyro_Voxel_v1_0001.vdb' in node '<c4d.BaseObject object called Pyro Output/Volume Loader with ID 1039866 at 18444442432>'. Found path parameter with the label 'File' and value './$prj/$prj' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 18464012096>'. Found path parameter with the label 'File' and value '' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 18464012096>'. Found path parameter with the label 'OpenColorIO Configuration' and value '$OCIO' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'File' and value './Redshift/irradiance_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'File' and value './Redshift/irradiance_pointcloud_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'File' and value './Redshift/photon_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'Base Filename' and value '$prj_AOV' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'File' and value './Redshift/subsurface_scattering_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 18440927424>'. Found path parameter with the label 'File' and value '' in node '<c4d.documents.BaseVideoPost object called RS Post-Effects/RS Post-Effects with ID 1040189 at 18440910144>'. Pyro Voxel.c4d file_80aa35ed1bf4db09~.hdr?name=ydr_minimalist_hdr.hdr&db=MaxonAssets.db (assets.maxon.net) Pyro_Voxel_v1_0001.vdb Pyro_Voxel_v1_0002.vdb ... Pyro_Voxel_v1_0180.vdbHere's the output that we get on a rendering machine on the same c4d scene file.
Found path parameter with the label 'Relative Path' and value 'tex' in node '<c4d.documents.BaseDocument object called with ID 110059 at 2363282316352>'. Found path parameter with the label 'OpenColorIO Config' and value '$OCIO' in node '<c4d.documents.BaseDocument object called with ID 110059 at 2363282316352>'. Found path parameter with the label 'Project Path' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene' in node '<c4d.documents.BaseDocument object called with ID 110059 at 2363282316352>'. Found path parameter with the label 'Project Name' and value 'Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called with ID 110059 at 2363282316352>'. Found path parameter with the label 'File Path' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene\Pyro Voxel.c4d' in node '<c4d.documents.BaseDocument object called with ID 110059 at 2363282316352>'. Found path parameter with the label 'File' and value '' in node '<c4d.BaseObject object called RS Camera/RS Camera with ID 1057516 at 2363337280832>'. Found path parameter with the label 'Template Filename' and value '$prj_$frame.vdb' in node '<c4d.BaseObject object called Pyro Output/Pyro Output with ID 1059580 at 2363336803648>'. Found path parameter with the label 'File' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene\Pyro_Voxel\Pyro_Voxel' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 2363320759104>'. Found path parameter with the label 'File' and value 'C:\Sessions\session-e5c2\assetroot-3751\Users\my_user_id\Desktop\Pyro scene' in node '<c4d.documents.RenderData object called My Render Setting/My Render Setting with ID 110304 at 2363320759104>'. Found path parameter with the label 'OpenColorIO Configuration' and value '$OCIO' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'File' and value './Redshift/irradiance_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'File' and value './Redshift/irradiance_pointcloud_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'File' and value './Redshift/photon_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'Base Filename' and value '$prj_AOV' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'File' and value './Redshift/subsurface_scattering_map_auto.$frame.rsmap' in node '<c4d.documents.BaseVideoPost object called Redshift/Redshift with ID 1036219 at 2363328343872>'. Found path parameter with the label 'File' and value '' in node '<c4d.documents.BaseVideoPost object called RS Post-Effects/RS Post-Effects with ID 1040189 at 2363328329536>'. Pyro Voxel.c4d file_80aa35ed1bf4db09~.hdr?name=ydr_minimalist_hdr.hdr&db=MaxonAssets.db (assets.maxon.net) Pyro_Voxel_v1_0001.vdbThe main difference that we could notice is that somehow the rendering machine did not have the following line (pyro output/volume loader was missing)
Found path parameter with the label 'Filename' and value '/Users/my_user_name/Desktop/Pyro scene/vol/Pyro_Voxel_v1_0001.vdb' in node '<c4d.BaseObject object called Pyro Output/Volume Loader with ID 1039866 at 18444442432>'.While submitting, we used relative paths and it still ended up giving us blank renders. Unfortunately, I'm not sure how I can share a reproducible for a rendering worker node (Let us know if we can setup a online video call so that we can share the issue that we are currently facing)
We're relatively new to Cinema 4D's internals, so any documentation or resources you could provide to help us better understand these issues would be greatly appreciated.
-
Hey @karthikbp,
No problem with the delay. But I must remind you of our support procedures:
- Please verify your account.
- Please provide reproducible data, e.g., a scene that shows the issue you are having. This was another perfect example for where I spent such much time on guessing what some devs might be doing, where a simple scene file would have made things easier. Print outs are nice but useless without context. I will not keep doing what I did here, meet new developers half way, it is simply not possible logistically. We are happy to help, but we need some data to work with. Something concrete we can run.
I would also recommend that you apply for Maxon Developer Program. I think your company had once MRD seats, but there are currently none as far as I can see.
About your Issue
Here's the script that we used to print all the assets in the c4d scene file based off the scripts that you shared previously.
You run in this script
mxutils.RecurseGraph(doc, True, True, True), where my version usedmxutils.RecurseGraph(doc, False, True, True). That was no accident, as this first parameter enables traversal of caches, which is not only expensive but usually irrelevant for most users, as caches are read only.The main difference that we could notice is that somehow the rendering machine did not have the following line (pyro output/volume loader was missing)
The output you see there is part of a
Python Outputobject cache. AVolumeLoaderis the root element of all cached volumes caches¹, and then holds the individual volumes, usually data for each frame. The cache of the Output being empty means that it lacks the data to build it. Which likely means that something with your cache paths does not line up.¹ - We have a bit conflicting terms here. So, there is once the notion of caches in a scene graph. That is also the kind of cache you enabled traversing when you set
yieldCachesinRecurseGraphtoTrue. MostBaseObjectshave such cache, and there are two variant, but they all express the data generated by a parametric objects (a 'generator' in Cinema 4D slang). Read more about it in geometry_caches_s26.py. And then we have the Volume VDB caches on disk. For aVolumeLoaderobject to build itsBaseObject:GetCache(), i.e., its virtual object hierarchy invisible to users, it requires access to files on disk - "the volume cache". And within this, theVolumeLoaderis usually itself part of another objects cache, e.g., aPyro Outputobject.While submitting, we used relative paths and it still ended up giving us blank renders.
That sounds very unlikely, as thousands of people are using Pyro. But your claim got me a bit thinking. So we have this, the 'Cache Paths' parameter of a 'Pyro Output' object which is here the likely culprit for your issues.
and although it sort of implies that it uses relative paths when you press the 'Cache' button it very obviously does not.
So, my suspicion would be that you simply do not have relative paths there, and when you then just move the scene (on the same machine or onto another machine, the result should be same), the 'Pyro Output' cache building fails because it requires these VDB files on disk when "Use Cache" is enabled. If you would look into your scene graph, you would probably have a null object there in the cache.
But I still do not really understand what you are doing and why you are doing it. This strikes me a bit as one of the cases where someone asks us "What is 'a' + 1?". I can then either say "62" or say that this question does not make too much sense. And sometimes, especially more experienced developers then insist on storming that hill, although that is not really how you would tackle that problem in Cinema 4D.
In Cinema 4D, when you want to make a scene 'moveable', you just use 'Save Project with Assets ...'. In the API this would be then c4d.documents.SaveProject. This will handle everything for you, including this Pyro case. There could be of course cases where you have some render nodes and you want to save bandwidth and storage and therefore only want to store shared asset data once. But I would really contemplate twice if storming that hill is really worth it.
I have attached below a script which dissects a bit how to handle
PYRO_OBJECT_CACHE_PATHS(there is a bit of an inconsistency how this parameter is handled in asset messages, I already talked with the simulation team about this).Please provide tangible data for all further questions and follow-ups.
Cheers,
Ferdinand"""Demos more ways how to traverse a scene and its data. Can be run as a script manager script. You should have a Pyro Output object in the scene with a cached simulation, and some node materials using some textures. The script will traverse the scene graph abstractly, and among other things make all Pyro caches relative to the scene document path. """ import re import os import typing import c4d import maxon import mxutils doc: c4d.documents.BaseDocument # The currently active document. op: c4d.BaseObject # The currently active object, can be None. # An internal custom GUI that is not exposed in the public API, which is used to store multiple # paths in a single string. Pyro leans into this GUI and extended it with some custom encoding. CUSTOMGUI_PATHS: int = 300002000 # An expression to decompose a path element with in the list of 'paths' store in the parameter # `PYRO_OBJECT_CACHE_PATHS` of a pyro object. A value of the parameter looks like this: # # pyro_scene_v3_0001::.\vol\pyro_scene_v3_0001.vdb::0 # pyro_scene_v4_0001::[0].\vol\pyro_scene_v4_0001.vdb::0 # pyro_scene_v5_0001::[0].\vol\pyro_scene_v5_0001.vdb::0 # # I.e., \n separated elements, and this expression captures the a line in the form: # <name>::[<prefix>]<path>::<postfix> RE_PYRO_CACHE_PATHS: re.Pattern = re.compile(r"^(.*\:\:)([0])?(.*)(\:\:\d+)?$") def ProcessCachePaths(doc: c4d.documents.BaseDocument, value: str) -> str: """Makes a (pyro) CUSTOMGUI_PATHS string path relative to a scene. This decomposes a DTYPE_STRING parameter using the PATHS custom GUI, used in the mode realized for Pyro. Note that there are other modes for this GUI, which will result in different encodings being used (so this function will not work for them). """ docPath: str = os.path.normpath(doc.GetDocumentPath()) if not docPath: raise RuntimeError("Cannot make paths relative to an unsaved document.") print(value, "\n") # Iterate over the lines in the value, and try to make them relative to the document path. lines: list[str] = [] for line in value.split('\n'): if line.strip() == "": continue match: re.Match | None = RE_PYRO_CACHE_PATHS.match(line.strip()) if not match: raise RuntimeError(f"Failed to match line '{line.strip()}' in '{value}'.") name: str = match.group(1).strip() prefix: str = match.group(2).strip() if isinstance(match.group(2), str) else "" cachePath: str = os.path.normpath(match.group(3).strip()) postfix: str = match.group(4).strip() if isinstance(match.group(4), str) else "" if cachePath.startswith(docPath): cachePath = os.path.join(".", *cachePath[len(docPath)::].split(os.sep)) lines.append(f"{name}{prefix}{cachePath}{postfix}") return "\n".join(lines) def DemoAbstractTraversal() -> None: """Showcases more details of how to manually traverse a scene graph. """ # So, as quite clearly stated in the last posting, this traversal was an example. It output you # all the parameters of type DTYPE_FILENAME in a scene. You can fashion this broader, but redoing # all the asset finding is not trivial as everything could be custom, hence the existence of # GetAllAssets(). # As also stated, the last example also did not traverse the Maxon API scene graph, which is # where most of the modern material data is stored. The example below is a bit broader, as it # also traverse DTYPE_STRING parameters with the GUI PATHS, and checks out all Maxon API # node graphs. But it is still that - an example. You have to write your own code. # We again iterate over everything in the Cinema API scene graph, this is very much NOT a cheap # operation and should be avoided in production code when possible. In this example and my last # I deliberately disable cache traversal, and I have seen that you did enable it. Do not do that, # this makes it so much more expensive. Even a small scene with a few dozen objects will usually # have hundreds if not thousands of scene elements when you really go everywhere. When you then # traverse the description of each, you could easily end up with hundreds of thousands of # operations, rendering this all very slow. element: c4d.BaseList2D for element in mxutils.RecurseGraph(doc, False, True, True): print(f"Traversing Cinema API scene element '{element}") # --- Abstract parameter processing -------------------------------------------------------- # Here we iterate over all parameters of the current scene element, and check if they are # of type DTYPE_FILENAME or DTYPE_STRING with the GUI PATHS. So, that we also catch Pyro # cache paths. But there are MANY more custom GUIs and some of them also store paths. Some # of them, such as CUSTOMGUI_PATHS, are not even exposed in the public API. for data, pid, _ in element.GetDescription(c4d.DESCFLAGS_GET_NONE): # Step over everything that is not a parameter, of type DTYPE_FILENAME or DTYPE_STRING. # For strings, only check string with the GUI 'PATHS'. This is mot even remotely every # way how paths are stored in a scene. Just have a look at all CUSTOMGUI_* symbols in # the C++ API, to het a sense. And these are only the exposed ones, there are more, as # demonstrated by the CUSTOMGUI_PATHS symbol below. dtype: int = pid[pid.GetDepth() - 1].dtype if dtype not in (c4d.DTYPE_FILENAME, c4d.DTYPE_STRING) or \ dtype == c4d.DTYPE_STRING and data[c4d.DESC_CUSTOMGUI] != CUSTOMGUI_PATHS: continue # Make things a bit easier to read, since CUSTOMGUI_PATHS splits its values by newlines. value: str = str(element[pid]).replace("\n", "") print(f"\tFound parameter '{data[c4d.DESC_NAME]}' with value '{value}'" f"({data[c4d.DESC_CUSTOMGUI] = }, {pid[0].dtype = }).") # --- Special Pyro handling ---------------------------------------------------------------- # Here we specifically handle Pyro object when we come across them. We use the special # function ProcessCachePaths() to unpack a CUSTOMGUI_PATHS parameter, which is in Pyro # mode. if element.GetType() == c4d.Opyro: lines: list[str] = [] cachePaths: str = element[c4d.PYRO_OBJECT_CACHE_PATHS] lines.append(f"\tOld pyro cache paths:") lines += [f"\t\t{line}" for line in cachePaths.split('\n') if line.strip()] cachePaths = ProcessCachePaths(doc, cachePaths) lines.append(f"\tRelative pyro cache paths:") lines += [f"\t\t{line}" for line in cachePaths.split('\n') if line.strip()] print("\n".join(lines)) # --- Branching of into Maxon API scene graph traversal ------------------------------------ # Here we branch of into the Maxon API scene graph traversal. if not element.IsNodeBased(): continue # Each element in a Cinema API scene graph can have multiple NimbusBaseInterface references, # NimbusBaseRef's, associated with it. You can think of a nimbus reference as a set of # purpose bound node graphs associated with a scene element. E.g., you have a node that has # nimbus reference for "node materials", and attached to that reference would then be one to # many node graphs that realize a material system for a node space (i.e., effectively a # render engine in this case). # We iterate over all nimbus reference, node space pairs of the current element ... nodeSpaceId: maxon.Id nimbusRef: maxon.NimbusBaseRef for nodeSpaceId, nimbusRef in element.GetAllNimbusRefs(): # .. get the graph for the current nimbus reference/space pair, and then the root node # of the graph. graph: maxon.NodesGraphModelRef = nimbusRef.GetGraph(nodeSpaceId) root: maxon.GraphNode = graph.GetViewRoot() print(f"\tProcessing Maxon API node space '{nodeSpaceId}' with node handler '{nimbusRef}'.") # The Nodes API has the a bit unfortunate terminology that most entities in it are # a maxon.GraphNode, including things like input and output ports. That is because # #GraphNode refers to a node in a tree in which the Nodes API stores all entities # of graph. Not the graph the user sees. # Now we iterate over all nodes representing the input ports in the current graph. We # could also search for what the Nodes API calls 'true nodes', i.e., #GraphNode entities # that represent what the use would consider a node, to search there its attributes, # i.e., things which cannot be wired together. But usually a node implementation # should not store there paths, but you never know what some node space implementation # does. inputPort: maxon.GraphNode for inputPort in root.GetInnerNodes(maxon.NODE_KIND.INPORT, True): value: maxon.Data = inputPort.GetPortValue() print (f"\tFound input port '{inputPort}' with value '{value}'.") if not isinstance(value, maxon.Url): continue # We found some input port holding some form of URL. Get the owning true node # and print the information. We have to move up to levels, because a local true node # tree looks like this: # # someNode@someHash # The true node, a GraphNode. # > # The root off all input ports, also a GraphNode. # myInput # An input port, also a GraphNode. # anotherInput # Another input port, also a GraphNode # ... # < # The root of all output ports, which is a GraphNode. # myOutput # An output port, also a GraphNode. # anotherOutput # Another output port, also a GraphNode # ... # name: str = inputPort.GetValue("effectivename") inputPorts: maxon.GraphNode = inputPort.GetParent() trueNode: maxon.GraphNode = inputPorts.GetParent() if not trueNode or not trueNode.GetKind() == maxon.NODE_KIND.NODE: print(f"Failed to find true node for input port '{inputPort}'.") continue print(f"Found URL input port {inputPort} with value '{str(value)}', owned by the" f"true node {trueNode}.") c4d.EventAdd() def DemoAssetTraversal() -> None: """Demonstrates how to handle assets in a scene. """ # We get all assets in the scene and start iterating over them. For what you want to do, you # porbably want to use the flag ASSETDATA_FLAG_MULTIPLEUSE, as will otherwise miss entities # that use the same asset multiple times, e.g., two materials using the same texture. container: list[dict[str, typing.Any]] = [] if not c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_MULTIPLEUSE, container): raise RuntimeError("Failed to get all assets from the document.") # A little lookup table to avoid processing the same Pyro object multiple times. visitedPyro: set[bytes] = set() for asset in container: url: str = asset.get("filename", "") name: str = asset.get("assetname", "") pid: int = asset.get("paramId", 0) owner: c4d.BaseList2D | None = asset.get("owner", None) nodePath: str = asset.get("nodePath", "") nodeSpace: str = asset.get("nodeSpace", "") print(f"Processing asset '{url}' with the owner '{owner.GetName() if owner else ''}' and pid '{pid}'.") # Which will produce output for Pyro object like this, which is not very helpful for # updating pyro caches, as it refers to internal parts of the Pyro object cache (i.e., the # cache in the scene graph, not the Pyro cache on disk), that are readonly to the user. # > Processing asset 'D:\temp\pyro\vol\pyro_scene_v7_0025.vdb' with the owner 'Pyro Output' # > and pid '6000'. # We cannot even use the parameter ID, because that ID is used in the part of the volume # loader cache inside the Pyro Output cache, that is responsible for rendering frame 25 in # this case. # So, to fix that, we have to ignore most of the asset data, and just manually fix Pyro # objects (or do more complex traversal here as shown in #AbstractTraversal()). # The issue is that we get here yielded things inside a cache, (the scene graph cache # that of the Pyro object), which makes very little sense at least in the public API, as # these scene elements are effectively readonly to the user. # We could also dos this more generically without checking the cache element being a Volume # Loader, but I kept this safe. if (isinstance(owner, c4d.BaseObject) and owner.GetType() == c4d.Ovolumeloader and owner.GetCacheParent()): while owner.GetCacheParent(): owner = owner.GetCacheParent() if owner.GetType() == c4d.Opyro: # We have already processed this Pyro object, so skip it. uuid: bytes = bytes(owner.FindUniqueID(c4d.MAXON_CREATOR_ID)) if uuid in visitedPyro: continue cachePaths: str = ProcessCachePaths(doc, owner[c4d.PYRO_OBJECT_CACHE_PATHS]) owner[c4d.PYRO_OBJECT_CACHE_PATHS] = cachePaths cachePaths = cachePaths.replace("\n", "") print(f"Updated Pyro object cache paths to '{cachePaths}'.") # Now that we have done this, we should set a marker so that we do not process # this Pyro object again, because there is an asset for frame in its cache. # Technically it would probably not hurt to run ProcessCachePaths() on already # relative paths, but is cleaner like this. visitedPyro.add(uuid) # Last but not least, here is some abstracted node graph handling. This could be done better # and more complex, but that would be up to you. # This asset is for something Maxon API scene graph related, very likely a node material. if not nodePath or not nodeSpace or not isinstance(owner, c4d.BaseList2D): continue print(f"Processing node graph.") # As explained in the earlier example, we grab the graph referenced by the nodeSpace in the # asset data, by iterating over all nimbus references of the owner. We could also use # NodeMaterial, and then could directly get the graph, but would then not cover non-material # cases. sid: maxon.Id nimbusRef: maxon.NimbusBaseRef for sid, nimbusRef in owner.GetAllNimbusRefs(): if sid != maxon.Id(nodeSpace): continue # Get the graph for the node space referenced by the asset data and the root node of it. graph: maxon.NodesGraphModelRef = nimbusRef.GetGraph(sid) root: maxon.GraphNode = graph.GetViewRoot() # A little helper function to untangle the different notions of what different render # engines pass as #nodePath in the asset data. def FindNode(node: maxon.GraphNode, path: str) -> maxon.GraphNode | None: """Due to the rather loose interpretation of node paths of some asset handlers, we have to write this helper function to find what is meant in a graph. A node path could look like this: image@IPTcQlkjIRrgM33F_dJevQ<url Where "image@IPTcQlkjIRrgM33F_dJevQ" is the ID of a true node, the "Image" node in a Standard Render graph, "<" is its input ports GRaphNode root, and "url" the input port for the texture of the URL. We effectively walk the node tree here while decomposing the node path. """ child: maxon.GraphNode for child in node.GetChildren(mask=maxon.NODE_KIND.ALL_MASK): cid: str = str(child.GetId()) if path.startswith(cid): remainder: str = path[len(cid)::] if remainder == "": return child return FindNode(child, remainder) raise RuntimeError(f"Failed to find node with path '{path}' in {node}.") # Find the entity referenced by the nodePath in the asset data. This should by design be # a port, but not everyone is sticking to that rule. port: maxon.GraphNode = FindNode(root, nodePath) if port.IsNullValue(): print(f"Failed to find asset data with path '{nodePath}' in {graph}.") continue # The value referenced in the asset data as a maxon.Url. assetValue: maxon.Url = maxon.Url(url) # As mentioned, #port, should be a port. But Redshift for example will give you a true # node (the texture sample node that holds the URL port), but not the port. We could # hardcode thing here (e.g., if nodeType == "TextureSampler": port = "url") but we # would have to do that then for every render engine which does what Redshift does ( # never really checked what V_Ray, Arnold, and others do). A more generic way is to # search the input ports of the node for the asset value. if port.GetKind() == maxon.NODE_KIND.NODE: for child in port.GetChildren(mask=maxon.NODE_KIND.INPORT): if child.GetPortValue() == assetValue: port = child break # If we still do not have a port, then we cannot do anything with the asset data, as # we have no idea how to use it. if port.GetKind() != maxon.NODE_KIND.INPORT: print(f"Failed to find asset data with path '{nodePath}' in {graph}.") continue # Get the value of the port we found, and make sure it holds the value we expect. It # would be a bit unusual that a node space does not use maxon.Url for URLs, but # not impossible. currentValue: maxon.Data = port.GetPortValue() if not isinstance(currentValue, maxon.Url) or currentValue != assetValue: print(f"Failed to find asset data with path '{nodePath}' in {graph}.") continue # This is then how we would write a new value to that "asset". Remember that URLs in # the Maxon API are true URLs, i.e., the require a scheme/protocol, such as "file:///". newUrl: maxon.Url = maxon.Url("file:///c:/bob/is/your/uncle.jpg") with graph.BeginTransaction() as transaction: port.SetPortValue(maxon.Url("file:///c:/bob/is/your/uncle.jpg")) transaction.Commit() print(f"Updated node asset reference on {port} with '{newUrl}'.") def main() -> None: """Called by Cinema 4D when the script is being executed. """ DemoAbstractTraversal() print("\n" + "-" * 80 + "\n") DemoAssetTraversal() if __name__ == '__main__': main() -
Hi @ferdinand ,
Thanks for your input so far. I think we are getting really close and I thank you for your patience.
Please verify your account.
I never got an email from Maxon to verify my email for some reason. Is it possible to send that email again? Or is there a way to request to send the email again?
Please provide reproducible data, e.g., a scene that shows the issue you are having.
Sorry, it took me a while to figure out a way to provide some reproducible script and data. Here are the steps that I take to reproduce the issue that I face.
-
I use the "Pyro Voxel" scene from the Cinema 4D asset browser.
-
I run the following script (the script that you shared for the Burning Pan scene)
import c4d import os doc: c4d.documents.BaseDocument # The active document def main() -> None: """Called by Cinema 4D when the script is being executed. """ print("===================") #scene_file = "/insert/path/to/scene/file/pyro_voxel.c4d" #doc = c4d.documents.LoadDocument(scene_file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS) asset_list = [] asset_data: list[dict] = [] val = c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data) for item in asset_data: print(os.path.split(item.get("filename", ""))[-1]) print ("Done.") print("===========") if __name__ == '__main__': main()- This gives me the output
=================== pyro_voxel.c4d Pyro_Voxel_v2_0001.vdb Pyro_Voxel_v2_0002.vdb Pyro_Voxel_v2_0003.vdb Pyro_Voxel_v2_0004.vdb Pyro_Voxel_v2_0005.vdb Pyro_Voxel_v2_0006.vdb Pyro_Voxel_v2_0007.vdb Pyro_Voxel_v2_0008.vdb Pyro_Voxel_v2_0009.vdb Pyro_Voxel_v2_0010.vdb Pyro_Voxel_v2_0011.vdb Pyro_Voxel_v2_0012.vdb Pyro_Voxel_v2_0013.vdb Pyro_Voxel_v2_0014.vdb Pyro_Voxel_v2_0015.vdb Pyro_Voxel_v2_0016.vdb Pyro_Voxel_v2_0017.vdb Pyro_Voxel_v2_0018.vdb Pyro_Voxel_v2_0019.vdb Pyro_Voxel_v2_0020.vdb Pyro_Voxel_v2_0021.vdb Pyro_Voxel_v2_0022.vdb Pyro_Voxel_v2_0023.vdb Pyro_Voxel_v2_0024.vdb Pyro_Voxel_v2_0025.vdb Pyro_Voxel_v2_0026.vdb Pyro_Voxel_v2_0027.vdb Pyro_Voxel_v2_0028.vdb Pyro_Voxel_v2_0029.vdb Pyro_Voxel_v2_0030.vdb Pyro_Voxel_v2_0031.vdb Pyro_Voxel_v2_0032.vdb Pyro_Voxel_v2_0033.vdb Pyro_Voxel_v2_0034.vdb Pyro_Voxel_v2_0035.vdb Pyro_Voxel_v2_0036.vdb Pyro_Voxel_v2_0037.vdb Pyro_Voxel_v2_0038.vdb Pyro_Voxel_v2_0039.vdb Pyro_Voxel_v2_0040.vdb Pyro_Voxel_v2_0041.vdb Pyro_Voxel_v2_0042.vdb Pyro_Voxel_v2_0043.vdb Pyro_Voxel_v2_0044.vdb Pyro_Voxel_v2_0045.vdb Pyro_Voxel_v2_0046.vdb Pyro_Voxel_v2_0047.vdb Pyro_Voxel_v2_0048.vdb Pyro_Voxel_v2_0049.vdb Pyro_Voxel_v2_0050.vdb Pyro_Voxel_v2_0051.vdb Pyro_Voxel_v2_0052.vdb Pyro_Voxel_v2_0053.vdb Pyro_Voxel_v2_0054.vdb Pyro_Voxel_v2_0055.vdb Pyro_Voxel_v2_0056.vdb Pyro_Voxel_v2_0057.vdb Pyro_Voxel_v2_0058.vdb Pyro_Voxel_v2_0059.vdb Pyro_Voxel_v2_0060.vdb Pyro_Voxel_v2_0061.vdb Pyro_Voxel_v2_0062.vdb Pyro_Voxel_v2_0063.vdb Pyro_Voxel_v2_0064.vdb Pyro_Voxel_v2_0065.vdb Pyro_Voxel_v2_0066.vdb Pyro_Voxel_v2_0067.vdb Pyro_Voxel_v2_0068.vdb Pyro_Voxel_v2_0069.vdb Pyro_Voxel_v2_0070.vdb Pyro_Voxel_v2_0071.vdb Pyro_Voxel_v2_0072.vdb Pyro_Voxel_v2_0073.vdb Pyro_Voxel_v2_0074.vdb Pyro_Voxel_v2_0075.vdb Pyro_Voxel_v2_0076.vdb Pyro_Voxel_v2_0077.vdb Pyro_Voxel_v2_0078.vdb Pyro_Voxel_v2_0079.vdb Pyro_Voxel_v2_0080.vdb Pyro_Voxel_v2_0081.vdb Pyro_Voxel_v2_0082.vdb Pyro_Voxel_v2_0083.vdb Pyro_Voxel_v2_0084.vdb Pyro_Voxel_v2_0085.vdb Pyro_Voxel_v2_0086.vdb Pyro_Voxel_v2_0087.vdb Pyro_Voxel_v2_0088.vdb Pyro_Voxel_v2_0089.vdb Pyro_Voxel_v2_0090.vdb Pyro_Voxel_v2_0091.vdb Pyro_Voxel_v2_0092.vdb Pyro_Voxel_v2_0093.vdb Pyro_Voxel_v2_0094.vdb Pyro_Voxel_v2_0095.vdb Pyro_Voxel_v2_0096.vdb Pyro_Voxel_v2_0097.vdb Pyro_Voxel_v2_0098.vdb Pyro_Voxel_v2_0099.vdb Pyro_Voxel_v2_0100.vdb Pyro_Voxel_v2_0101.vdb Pyro_Voxel_v2_0102.vdb Pyro_Voxel_v2_0103.vdb Pyro_Voxel_v2_0104.vdb Pyro_Voxel_v2_0105.vdb Pyro_Voxel_v2_0106.vdb Pyro_Voxel_v2_0107.vdb Pyro_Voxel_v2_0108.vdb Pyro_Voxel_v2_0109.vdb Pyro_Voxel_v2_0110.vdb Pyro_Voxel_v2_0111.vdb Pyro_Voxel_v2_0112.vdb Pyro_Voxel_v2_0113.vdb Pyro_Voxel_v2_0114.vdb Pyro_Voxel_v2_0115.vdb Pyro_Voxel_v2_0116.vdb Pyro_Voxel_v2_0117.vdb Pyro_Voxel_v2_0118.vdb Pyro_Voxel_v2_0119.vdb Pyro_Voxel_v2_0120.vdb Pyro_Voxel_v2_0121.vdb Pyro_Voxel_v2_0122.vdb Pyro_Voxel_v2_0123.vdb Pyro_Voxel_v2_0124.vdb Pyro_Voxel_v2_0125.vdb Pyro_Voxel_v2_0126.vdb Pyro_Voxel_v2_0127.vdb Pyro_Voxel_v2_0128.vdb Pyro_Voxel_v2_0129.vdb Pyro_Voxel_v2_0130.vdb Pyro_Voxel_v2_0131.vdb Pyro_Voxel_v2_0132.vdb Pyro_Voxel_v2_0133.vdb Pyro_Voxel_v2_0134.vdb Pyro_Voxel_v2_0135.vdb Pyro_Voxel_v2_0136.vdb Pyro_Voxel_v2_0137.vdb Pyro_Voxel_v2_0138.vdb Pyro_Voxel_v2_0139.vdb Pyro_Voxel_v2_0140.vdb Pyro_Voxel_v2_0141.vdb Pyro_Voxel_v2_0142.vdb Pyro_Voxel_v2_0143.vdb Pyro_Voxel_v2_0144.vdb Pyro_Voxel_v2_0145.vdb Pyro_Voxel_v2_0146.vdb Pyro_Voxel_v2_0147.vdb Pyro_Voxel_v2_0148.vdb Pyro_Voxel_v2_0149.vdb Pyro_Voxel_v2_0150.vdb Pyro_Voxel_v2_0151.vdb Pyro_Voxel_v2_0152.vdb Pyro_Voxel_v2_0153.vdb Pyro_Voxel_v2_0154.vdb Pyro_Voxel_v2_0155.vdb Pyro_Voxel_v2_0156.vdb Pyro_Voxel_v2_0157.vdb Pyro_Voxel_v2_0158.vdb Pyro_Voxel_v2_0159.vdb Pyro_Voxel_v2_0160.vdb Pyro_Voxel_v2_0161.vdb Pyro_Voxel_v2_0162.vdb Pyro_Voxel_v2_0163.vdb Pyro_Voxel_v2_0164.vdb Pyro_Voxel_v2_0165.vdb Pyro_Voxel_v2_0166.vdb Pyro_Voxel_v2_0167.vdb Pyro_Voxel_v2_0168.vdb Pyro_Voxel_v2_0169.vdb Pyro_Voxel_v2_0170.vdb Pyro_Voxel_v2_0171.vdb Pyro_Voxel_v2_0172.vdb Pyro_Voxel_v2_0173.vdb Pyro_Voxel_v2_0174.vdb Pyro_Voxel_v2_0175.vdb Pyro_Voxel_v2_0176.vdb Pyro_Voxel_v2_0177.vdb Pyro_Voxel_v2_0178.vdb Pyro_Voxel_v2_0179.vdb Pyro_Voxel_v2_0180.vdb ydr_minimalist_hdr.hdr Done. ===========- Now uncomment the scene_file and doc lines and then run the command again. I get something like
=================== pyro_voxel.c4d Pyro_Voxel_v2_0001.vdb ydr_minimalist_hdr.hdr Done. ===========Notice that it only has 1 cache file and I believe that the issue is that we are not loading the document with all its caches correctly which leads to the blank renders.
Maybe something to do with LoadDocument? Am I missing some flag that will get all the assets including the caches (I've tried multiple flags but haven't had much success)?We use the LoadDocument function to set the scene file to render over here: https://github.com/aws-deadline/deadline-cloud-for-cinema-4d/blob/mainline/src/deadline/cinema4d_adaptor/Cinema4DClient/cinema4d_handler.py#L299-L319
Any help with fixing this would be really helpful.
Thanks,
Karthik -
-
Hey @karthikbp,
Please read my previous posting, I very likely already explained there what your issue is and provided a solution. When you just move that file
"/insert/path/to/scene/file/pyro_voxel.c4d"on disk, you will break its cache paths.Otherwise, you would have to provide a file/project that you made relative and which is rendering black. But I am pretty sure that you have hardcoded paths in your
PYRO_OBJECT_CACHE_PATHSparameters as explained above, which will break when you just move the scene, explaining the output. The reason your script is doing what it is doing is because that scene does not have its passes executed (among other things - caches built) when you loaded it from disk. There is also the oddity that Pyro considers frame zero not part of the animation, i.e., you must nudge Pyro to frame 1 for it to start building its caches and therefore in this case also asset messages.What the simulation team did with the asset messages was not so good, as they report assets inside caches and do not report the actual parameter. But as lined out above, I already told them that.
As mentioned before, you should also use
ASSETDATA_FLAG_MULTIPLEUSEand your node material handling is completely wrong (at least form what I saw on GitHub). Both can lead to incorrect renderings.Cheers,
FerdinandPS: I have resent your validation mail to the e-mail handle your account is bound to.
When you write your script like this, it will also print the assets when you load the file from disk. But that is all more a technicality. Please read and follow what I wrote in my last post. This will only work when you do not move
scene_file, as by default it will use absolute paths for the VDB files.import c4d import os doc: c4d.documents.BaseDocument # The active document def main() -> None: """Called by Cinema 4D when the script is being executed. """ print("===================") scene_file = r"D:\temp\pyro\pyro_scene.c4d" doc = c4d.documents.LoadDocument(scene_file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS) doc.SetTime(c4d.BaseTime(1, 30)) # Frame 1 for a 30 FPS document. # Build the animations, chaches, and expressions for that 1st frame. Please see threads about prerolling for how # to preroll properly, the simple approach I use here, will not work for all complex simulation prerolling tasks. doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE) asset_list = [] asset_data: list[dict] = [] val = c4d.documents.GetAllAssetsNew(doc, False, "", c4d.ASSETDATA_FLAG_NONE, asset_data) for item in asset_data: print(os.path.split(item.get("filename", ""))[-1]) print ("Done.") print("===========") if __name__ == '__main__': main() -
Hi @ferdinand ,
Firstly, I would like to thank you for your help so far and we have now been able to get successful renders. Let me try replying to your previous response.
I am pretty sure that you have hardcoded paths in your PYRO_OBJECT_CACHE_PATHS parameters as explained above, which will break when you just move the scene, explaining the output
Yup. I think I forgot to mention this in my previous post but I've been using relative paths while submitting. To get relative paths, we have been using "Window" > "Project Asset Inspector" and then clicking on all the assets and "Localize Filenames". Also we have been saving the scenes with "Save Project with Assets"
The reason your script is doing what it is doing is because that scene does not have its passes executed (among other things - caches built) when you loaded it from disk.
Thanking you for sharing this. Adding
doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE)seems to have fixed the issue for us and we are able to get successful renders.But I've a question. Can we add
ExecutePassesbefore rendering any Cinema 4D scene file? I've not experienced issues so far after adding this line but I'd like to know your thoughts as well if adding this line for even still images (i.e. single frame) has any drawbacks?There is also the oddity that Pyro considers frame zero not part of the animation, i.e., you must nudge Pyro to frame 1 for it to start building its caches and therefore in this case also asset messages.
I've a question about this. What are the drawbacks of not adding
doc.SetTime(c4d.BaseTime(1, 30))? I've seen that even without adding the line the renders have been successful.As mentioned before, you should also use ASSETDATA_FLAG_MULTIPLEUSE and your node material handling is completely wrong (at least form what I saw on GitHub). Both can lead to incorrect renderings.
Yup. We will be reworking our code to just use relative paths for assets or using the "Save Project with Assets" as you suggested so that we don't need to apply specific path mapping.
PS: I have resent your validation mail to the e-mail handle your account is bound to.
Thanks, I've verified the email now.
-
Hey @karthikbp,
Yup. I think I forgot to mention this in my previous post but I've been using relative paths while submitting. To get relative paths, we have been using "Window" > "Project Asset Inspector" and then clicking on all the assets and "Localize Filenames". Also we have been saving the scenes with "Save Project with Assets"
First of all, good to hear that you made progress. But the verbosity of your approach should not be necessary, simply invoking 'Save Project with Assets' should be enough to collect all scene dependencies and make them relative to the document. You can also do this programmatically via c4d.documents.SaveProject .
Thanking you for sharing this. Adding
doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE)seems to have fixed the issue for us and we are able to get successful renders. But I've a question. Can we addExecutePassesbefore rendering any Cinema 4D scene file? I've not experienced issues so far after adding this line but I'd like to know your thoughts as well if adding this line for even still images (i.e. single frame) has any drawbacks?Hm, that is genuinely concerning. Rendering a scene should entail its scene state being built on its own (in Cinema slang: executing the passes). When you really must do this before rendering a file on a node, something is very wrong there. I suspect that this is not absolutely true, and something more specific applies like "when for a pyro sim we render the first frame isolated on a node, then it is black unless we do this" or that you for example try to start rendering a simulation - which has not been cached - from a frame other than the first (then you will see nothing or incorrect output unless you preroll the simulation). But in any case, this would hint at either a substantial bug on our or your side.
I've a question about this. What are the drawbacks of not adding
doc.SetTime(c4d.BaseTime(1, 30))? I've seen that even without adding the line the renders have been successful.First of all, please do not copy my code. I wrote this in a quick and dirty fashion because I knew the parameters of my/your scene. In real life you would have to get the first frame of the animation (
c4d.RDATA_FRAMEFROM) from the render data and then its FPS, and then construct a time likec4d.BaseTime(fist + 1, fps).Why I did this is due to the oddity of how Pyro builds its caches. You wanted to see there the assets reported, which will only work when the Pyro Output node has built its cache due to how irregular MSG_GETALLASSETS has been implemented for Pyro. This is tied to the fact that Pyro will not generate output for the first frame of its simulation (loosely speaking frame zero), so I advance the scene state to frame 1 and then execute the passes so that this document has its scene state built for that first frame.
I am on another machine today, and I think I already deleted my Pyro temp folder on my other machine, but I am pretty sure when you remove that line, the script will still print nothing despite building the caches, because on frame zero the Pyro sim will not build anything. For rendering things, you should not need any of this.
Cheers,
Ferdinand -
Hi @ferdinand ,
when for a pyro sim we render the first frame isolated on a node, then it is black unless we do this
Yup. We only saw an improvement for Pyro scenes at our end. Other scenes work correctly even without adding the
doc.ExecutePassesline. -
Hey @karthikbp,
I cannot say much else without a scene file and example. What you are telling me sounds wrong, you should see at least the scene geometry on the 0st frame. For the pan scene we used above, it actually takes up to frame 2 (i.e., 0, 1, 2) before you see Pyro volumes. There could be a bug in Pyro for how it handles rendering a singular frame on a machine (which I think is what you are doing), or you are doing something wrong.
Executing the passes should neither generate output where naturally is none, nor should you have to execute them before rendering a frame. As lined out before, when you do not have a cached simulation scene, you must preroll it before rendering. I.e., you cannot jump directly to frame 20 and render it, you first must execute the passes up to that frame.
We are starting to run here in circles. Please consider applying for MRD as lined out here and via chat.
Cheers,
Ferdinand -
Hi @ferdinand ,
We are starting to run here in circles. Please consider applying for MRD as lined out here and via chat.
I think that makes sense. I'll apply for MRD right away.
