ExecutePasses performance keeps slowing down per-frame and per-run — possible cache issue?

[Aprecigout]

Hello,

After many search in the and in this forum—especially this topic: https://developers.maxon.net/forum/topic/14675/executepasses-to-slow/3?_=1751270994697&lang=en-GB—I’m running into a performance problem with ExecutePasses. The execution time grows from frame to frame and that slowdown is carried over between frames.

To give you some context: I work for a client who produces large-scale drone shows. Their artists use Cinema 4D to architect the whole drone scenography. We have developed several internal plug-ins and tools that help those artists design inside Cinema 4D and then validate the scene against real-world constraints so everything works safely on-site.

My issue arises during the export of these scenes to the drones’ proprietary format. Because the artists rely heavily on MoGraph to manage entire drone fleets, I call ExecutePasses at export time to extract the data I need. Export speed is critical in the field; we often have to tweak the scenography at the last minute.

After profiling, the overwhelming bottleneck is ExecutePasses.

To dig further, I wrote a minimal script that does nothing but run ExecutePasses over the first 10 000 frames:

import c4d
import time
import timeit

def main():
    # Get the active document
    doc = c4d.documents.GetActiveDocument()
    if doc is None:
        print("No active document")
        return
    
    # Get start and end frames
    start_frame = doc[c4d.DOCUMENT_MINTIME].GetFrame(doc.GetFps())
    end_frame = 10000
    #end_frame = doc[c4d.DOCUMENT_MAXTIME].GetFrame(doc.GetFps())
    fps = doc.GetFps()
    
    print(f"ExecutePasses Test - Cinema 4D")
    print(f"Document: {doc.GetDocumentName()}")
    print(f"Frames: {start_frame} to {end_frame} ({end_frame - start_frame} frames)")
    print(f"FPS: {fps}")
    print("-" * 50)
    
    # Measure total time
    start_time_total = timeit.default_timer()
    
    # Store frame times
    frame_times = []
    
    # Loop over each frame
    total_frames = end_frame - start_frame
    cumulative_time = 0
    for i, frame in enumerate(range(start_frame, end_frame)):
        # Update status bar
        progress = (i * 100) // total_frames
        c4d.StatusSetBar(progress)
        
        # Measure time for this frame
        start_time_frame = timeit.default_timer()
        
        # Set frame time
        time_val = c4d.BaseTime(frame, fps)
        doc.SetTime(time_val)
        
        # Execute passes
        doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_EXPORTONLY)
        
        # Calculate elapsed time for this frame
        frame_time = timeit.default_timer() - start_time_frame
        frame_times.append(frame_time)
        
        # Calculate cumulative time
        cumulative_time += frame_time
        
        # Display progress
        print(f"{frame};{frame_time:.4f};{cumulative_time:.4f}")
    
    # Calculate total time
    total_time = timeit.default_timer() - start_time_total
    
    # Display results
    print("-" * 50)
    print("RESULTS:")
    print(f"Total time: {total_time:.4f}s")
    print(f"Average time per frame: {sum(frame_times) / len(frame_times):.4f}s")
    print(f"Min time per frame: {min(frame_times):.4f}s")
    print(f"Max time per frame: {max(frame_times):.4f}s")
    print(f"Number of frames processed: {len(frame_times)}")
    
    # Display slowest frames
    print("\nSlowest frames:")
    indexed_times = [(i + start_frame, t) for i, t in enumerate(frame_times)]
    indexed_times.sort(key=lambda x: x[1], reverse=True)
    for i, (frame_num, frame_time) in enumerate(indexed_times[:5]):
        print(f"  Frame {frame_num}: {frame_time:.4f}s")
    
    print(f"\nTest completed - Cinema 4D {c4d.GetC4DVersion()}")
    
    # Clear status bar
    c4d.StatusClear()

if __name__ == '__main__':
    main() 

The test procedure is simply:

• First run
• Second run
• Third run
• Restart C4D and reload the scene
• Fourth run

For each run I log the ExecutePasses execution time for every frame, then plot the result:

Note : I don’t have an in-depth understanding of how Cinema 4D’s internal caches work, so please forgive any misconceptions below.

This behaviour feels like some kind of cache that never gets cleared: each run takes longer than the previous one, and the per-frame time at the start of a run matches the time at which the previous run ended.

Following i have some questions :

1. Is there a recommended way to flush or bypass the cache/memory buildup that seems to happen inside ExecutePasses, so execution time stays consistent?
2. If such mechanisms exist, could someone outline the usual workflow or API calls to use—or point me to the relevant documentation?

Because of the artists’ scenes, I unfortunately need to enable all ExecutePasses flags (cache, animation and `expression) so turning any of them off isn’t an option.

Finally, could C4DThread help in this context, or am I barking up the wrong tree? My experiments based on the thread linked above haven’t produced conclusive results.

Another Note : I haven’t attached the test .c4d file here because it contains a full scenography created for my client, but I can provide it privately in accordance with your procedure described here: https://plugincafe.maxon.net/guidlines_cp#support-data.

Thanks in advance for any insights!

[ferdinand]

Hey @Aprecigout,

Welcome to the Maxon developers forum and its community, it is great to have you with us!

Getting Started

Before creating your next postings, we would recommend making yourself accustomed with our forum and support procedures. You did not do anything wrong, we point all new users to these rules.

• Forum Overview: Provides a broad overview of the fundamental structure and rules of this forum, such as the purpose of the different sub-forums or the fact that we will ban users who engage in hate speech or harassment.
• Support Procedures: Provides a more in detail overview of how we provide technical support for APIs here. This topic will tell you how to ask good questions and limits of our technical support.
• Forum Features: Provides an overview of the technical features of this forum, such as Markdown markup or file uploads.

It is strongly recommended to read the first two topics carefully, especially the section Support Procedures: How to Ask Questions.

About your First Question

Please have a look at Support Procedures: How to Ask Questions, there are too many question here. Which often derails threads, but since you are new, let's try it like this.

Is there a recommended way to flush or bypass the cache/memory buildup that seems to happen inside ExecutePasses, so execution time stays consistent?

Caches cannot be flushed, as they are a mandatory part of the scene graph. A scene always has exactly one cache per scene element which requires a cache. Without it is not operational. And while the term is effectively correct, the scene graph cache is not what you usually picture when talking about caches. Caches are the discrete embodiment of parametric objects and deformers. You can think of them as an invisible part of the scene hierarchy. With the 'Active Object' plugin from the C++ SDK we can have a peek at what cache means. Here we unfold the first cube object clone of the cloner in the cache.

In the next release of Cinema 4D there will also be mxutils.GetSceneGraphString which has a similar purpose as the the plugin from the C++ SDK. For the scene shown above it will print what is shown in [1]. Everything below the [Cache] of Cloner is the cache of that scene element; a rather complex hidden hierarchy which itself contains caches which must be unpacked.

This also hints at the non-linear nature of caches. When you have a scene with 100 frames, where frame 0 is literally the empty scene and on frame 100 you have thousands of high resolution parametric objects which have to be rebuilt for this frame, and on top of that multiple complex simulations (pyro, particles, liquids), then executing the passes for frame 0 will be very quick as there is literally nothing to do, while executing the passes for frame 100 could cost seconds or even minutes (when none of the simulations are cached).

If such mechanisms exist, could someone outline the usual workflow or API calls to use—or point me to the relevant documentation?

Without knowing what you want to do, that is impossible to answer. Your code there could be slightly incorrect, as for 'pre-rolling' you usually want to execute the last pass twice, so that simulations can settle. When you just want to step through a scene, what you are doing is okay. It also should not make a big difference if your execute the passes for a scene state once ot twice, as all scene elements should only rebuild its caches when necessary when asked to do so. So, when you for example excute the passes and it takes 60 seconds, and then do it right again, the second run should only take a fraction of the first execution, as most scene elments should see that they are not dirty anymore, and just return their already existing cache.

But in general it is a bit odd that you execute the passes on all frames. It is very rare that you have to do that from Python. Maybe you could explain why you are doing that?

Finally, could C4DThread help in this context, or am I barking up the wrong tree? My experiments based on the thread linked above haven’t produced conclusive results.

The first argument of ExecutePasses is the thread to which the call shall be bound. You can use this to make the pass execution non-blocking for the main thread.

This also hints at the broader answer. The pass execution is of course already heavily optimized and runs in as many threads as the machine can muster and things such as ObjectData::GetVirtualObjects which are the backbone of cache building are run massively in parallel. The only thing you can decide is if you want to make your call non-blocking for the main thread or not (where the GUI code runs).

Not explicitly asked but sort of the elephant in the room: Executing the passes for all frames of a document varies drastically.

Just like many modern DCCs, Cinema 4D has a pretty sophisticated backed. Cinema 4D has for example on top of the "caching" of the scene graph a memoization core, which records the results of previous computations and reuses them when the same data is requested again. There is naturally some variance in such complex systems, where tiny changes in the input conditions can lead to significant differences in the execution time.

But what you show us there, that executing all passes of a scene takes twice or three times as long as the first time, is not normal. But I would at first be a bit doubtful that your findings are correct, as this would hint at a massive bug in the pass execution system. There could be an issue with the Python VM. I would recommend to unload the document in between the runs, to ensure that all possible memory is really freed.

Executing the passes for a single frame is already a not cheap operation, executing the passes for all frames of a document can be extensively expensive, since scene initialization makes up a good chunk of the render time of a document. So, doing this more than once in a row, is not the most clever thing. When you want to do this on multiple documents in a row, you should of course unload documents you are done with, so that you can free the memory.

Cheers,
Ferdinand

[1] Using print(mxutils.GetSceneGraphString(doc)) to visualize the content of a scene. Since we pass the whole document and not just the cloner, really everything gets unpacked here. Everything below the [Cache] child of 'Cloner' (BaseObject: Omgcloner) is the cache of the cloner object.

'' (BaseDocument: Tbasedocument)
├── [Branch] 'Objects' (Obase)
│   └── 'Cloner' (BaseObject: Omgcloner)
│       ├── [Cache]
│       │   └── 'Null' (BaseObject: Onull)
│       │       ├── 'Cube 0' (BaseObject: Ocube)
│       │       │   ├── [Cache]
│       │       │   │   └── 'Cube 0' (PolygonObject: Opolygon)
│       │       │   │       ├── [Deform Cache]
│       │       │   │       │   └── 'Cube 0' (PolygonObject: Opolygon)
│       │       │   │       │       └── [Branch] 'Tags' (Tbase)
│       │       │   │       │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │       │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │       │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │       │           └── '' (PointTag: Tpoint)
│       │       │   │       └── [Branch] 'Tags' (Tbase)
│       │       │   │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │           └── '' (PointTag: Tpoint)
│       │       │   ├── [Branch] 'Tags' (Tbase)
│       │       │   │   ├── 'Motion Graphics Color Tag' (BaseTag: Tmgcolor)
│       │       │   │   └── 'Phong' (BaseTag: Tphong)
│       │       │   └── 'Bend' (BaseObject: Obend)
│       │       ├── 'Sphere 1' (BaseObject: Osphere)
│       │       │   ├── [Cache]
│       │       │   │   └── 'Sphere 1' (PolygonObject: Opolygon)
│       │       │   │       ├── [Deform Cache]
│       │       │   │       │   └── 'Sphere 1' (PolygonObject: Opolygon)
│       │       │   │       │       └── [Branch] 'Tags' (Tbase)
│       │       │   │       │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │       │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │       │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │       │           └── '' (PointTag: Tpoint)
│       │       │   │       └── [Branch] 'Tags' (Tbase)
│       │       │   │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │           └── '' (PointTag: Tpoint)
│       │       │   ├── [Branch] 'Tags' (Tbase)
│       │       │   │   ├── 'Motion Graphics Color Tag' (BaseTag: Tmgcolor)
│       │       │   │   └── 'Phong' (BaseTag: Tphong)
│       │       │   └── 'Bend' (BaseObject: Obend)
│       │       ├── 'Cube 2' (BaseObject: Ocube)
│       │       │   ├── [Cache]
│       │       │   │   └── 'Cube 2' (PolygonObject: Opolygon)
│       │       │   │       ├── [Deform Cache]
│       │       │   │       │   └── 'Cube 2' (PolygonObject: Opolygon)
│       │       │   │       │       └── [Branch] 'Tags' (Tbase)
│       │       │   │       │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │       │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │       │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │       │           └── '' (PointTag: Tpoint)
│       │       │   │       └── [Branch] 'Tags' (Tbase)
│       │       │   │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │           └── '' (PointTag: Tpoint)
│       │       │   ├── [Branch] 'Tags' (Tbase)
│       │       │   │   ├── 'Motion Graphics Color Tag' (BaseTag: Tmgcolor)
│       │       │   │   └── 'Phong' (BaseTag: Tphong)
│       │       │   └── 'Bend' (BaseObject: Obend)
│       │       ├── 'Sphere 3' (BaseObject: Osphere)
│       │       │   ├── [Cache]
│       │       │   │   └── 'Sphere 3' (PolygonObject: Opolygon)
│       │       │   │       ├── [Deform Cache]
│       │       │   │       │   └── 'Sphere 3' (PolygonObject: Opolygon)
│       │       │   │       │       └── [Branch] 'Tags' (Tbase)
│       │       │   │       │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │       │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │       │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │       │           └── '' (PointTag: Tpoint)
│       │       │   │       └── [Branch] 'Tags' (Tbase)
│       │       │   │           ├── 'Phong' (BaseTag: Tphong)
│       │       │   │           ├── 'UVW' (UVWTag: Tuvw)
│       │       │   │           ├── '' (PolygonTag: Tpolygon)
│       │       │   │           └── '' (PointTag: Tpoint)
│       │       │   ├── [Branch] 'Tags' (Tbase)
│       │       │   │   ├── 'Motion Graphics Color Tag' (BaseTag: Tmgcolor)
│       │       │   │   └── 'Phong' (BaseTag: Tphong)
│       │       │   └── 'Bend' (BaseObject: Obend)
│       │       └── 'Cube 4' (BaseObject: Ocube)
│       │           ├── [Cache]
│       │           │   └── 'Cube 4' (PolygonObject: Opolygon)
│       │           │       ├── [Deform Cache]
│       │           │       │   └── 'Cube 4' (PolygonObject: Opolygon)
│       │           │       │       └── [Branch] 'Tags' (Tbase)
│       │           │       │           ├── 'Phong' (BaseTag: Tphong)
│       │           │       │           ├── 'UVW' (UVWTag: Tuvw)
│       │           │       │           ├── '' (PolygonTag: Tpolygon)
│       │           │       │           └── '' (PointTag: Tpoint)
│       │           │       └── [Branch] 'Tags' (Tbase)
│       │           │           ├── 'Phong' (BaseTag: Tphong)
│       │           │           ├── 'UVW' (UVWTag: Tuvw)
│       │           │           ├── '' (PolygonTag: Tpolygon)
│       │           │           └── '' (PointTag: Tpoint)
│       │           ├── [Branch] 'Tags' (Tbase)
│       │           │   ├── 'Motion Graphics Color Tag' (BaseTag: Tmgcolor)
│       │           │   └── 'Phong' (BaseTag: Tphong)
│       │           └── 'Bend' (BaseObject: Obend)
│       ├── [Branch] 'Tags' (Tbase)
│       │   └── 'Info' (BaseTag: ID_MOTAGDATA)
│       ├── 'Cube' (BaseObject: Ocube)
│       │   ├── [Branch] 'Tags' (Tbase)
│       │   │   └── 'Phong' (BaseTag: Tphong)
│       │   └── 'Bend' (BaseObject: Obend)
│       └── 'Sphere' (BaseObject: Osphere)
│           ├── [Branch] 'Tags' (Tbase)
│           │   └── 'Phong' (BaseTag: Tphong)
│           └── 'Bend' (BaseObject: Obend)
├── [Branch] 'Render Settings' (Rbase)
│   └── 'My Render Setting' (RenderData: Rbase)
│       ├── [Branch] 'Post Effects' (VPbase)
│       │   ├── 'Magic Bullet Looks' (BaseVideoPost: VPMagicBulletLooks)
│       │   └── 'Redshift' (BaseVideoPost: VPrsrenderer)
│       └── [Branch] 'Multi-Pass' (Zmultipass)
│           └── 'Post Effects' (BaseList2D: Zmultipass)
├── [Branch] 'Scene Hooks' (SHplugin)
│   ├── 'STHOOK' (BaseList2D: 1012061)
│   ├── 'RSCameraObjectTargetDistancePicker' (BaseList2D: 31028063)
│   ├── 'Python Embedded Change Monitor' (BaseList2D: 1058422)
│   ├── 'SceneHook' (BaseList2D: 1028481)
│   ├── 'CmSceneHook' (BaseList2D: 1026839)
│   ├── 'CameraMorphDrawSceneHook' (BaseList2D: 1029281)
│   ├── 'MotionCameraDrawSceneHook' (BaseList2D: 1029338)
│   ├── 'USD Scene Hook' (BaseList2D: 1055307)
│   ├── 'Substance Assets' (BaseList2D: 1032107)
│   ├── 'Alembic Archive Hook' (BaseList2D: 1028458)
│   ├── 'UpdateMerge Hook' (BaseList2D: 465001602)
│   ├── 'ArchiExchangeCADHook' (BaseList2D: 200000216)
│   ├── 'SLA wave scene hook' (BaseList2D: REG_EXP_PARSER)
│   ├── 'Thinking Particles' (TP_MasterSystem: ID_THINKINGPARTICLES)
│   ├── '' (BaseList2D: 1035577)
│   ├── 'Bullet' (BaseList2D: 180000100)
│   ├── 'XRefs' (BaseList2D: 1025807)
│   ├── 'CAManagerHook' (BaseList2D: 1019636)
│   │   └── [Branch] 'Weights Handler Head' (Tbaselist2d)
│   │       └── 'Weights Handler' (BaseList2D: 1037891)
│   ├── 'Volume Save Manager Hook' (BaseList2D: 1040459)
│   ├── 'UV Display 3D SceneHook' (BaseList2D: 1054166)
│   ├── 'uvhook' (BaseList2D: 1053309)
│   ├── 'ScatterPlacementHook' (BaseList2D: 1058060)
│   ├── 'Tool System Hook' (BaseList2D: ID_TOOL_SYSTEM_HOOK)
│   │   └── [Branch] 'SBM' (431000215)
│   │       └── 'Symmetry node' (BaseList2D: 431000215)
│   │           └── [Branch] 'C4DCoreWrapper' (200001044)
│   │               └── 'Symmetry node - net.maxon.symmetry.context.modeling' (BaseList2D: 300001078)
│   ├── 'MoGraphSceneHook' (BaseList2D: 1019525)
│   ├── 'gozScenehook' (BaseList2D: 1059748)
│   ├── 'Simulation' (BaseList2D: ID_SIMULATIONSCENE_HOOK)
│   │   └── [Branch] 'Simulation World' (Obase)
│   │       └── 'Default Simulation Scene' (BaseObject: Osimulationscene)
│   ├── 'PersistentHook' (BaseList2D: 180420202)
│   ├── 'Scene Nodes' (BaseList2D: SCENENODES_IDS_SCENEHOOK_ID)
│   ├── 'NE_SceneHook' (BaseList2D: 465002367)
│   ├── 'Take Hook' (BaseList2D: 431000055)
│   │   └── [Branch] 'Take System Branch' (TakeBase)
│   │       └── 'Main' (BaseTake: TakeBase)
│   │           └── [Branch] 'Override Folders' (431000073)
│   │               └── 'Overrides' (BaseList2D: 431000073)
│   │                   ├── 'Others' (BaseList2D: 431000073)
│   │                   ├── 'Layers' (BaseList2D: 431000073)
│   │                   ├── 'Materials' (BaseList2D: 431000073)
│   │                   ├── 'Shaders' (BaseList2D: 431000073)
│   │                   ├── 'Tags' (BaseList2D: 431000073)
│   │                   └── 'Objects' (BaseList2D: 431000073)
│   ├── 'CombineAc18_AutoCombine_SceneHook' (BaseList2D: 1032178)
│   ├── 'PLKHUD' (BaseList2D: 1020132)
│   │   └── [Branch] 'PSUNDOHEAD' (Obase)
│   │       └── 'PKHOP' (BaseObject: 1020120)
│   ├── 'RenderManager Hook' (BaseList2D: 465003509)
│   ├── 'Sound Scrubbing Hook' (BaseList2D: 100004815)
│   ├── 'To Do' (BaseList2D: 465001536)
│   ├── 'Animation' (BaseList2D: 465001535)
│   ├── 'BaseSettings Hook' (BaseList2D: ID_BS_HOOK)
│   ├── '' (BaseList2D: 1060457)
│   ├── 'SculptBrushModifierSceneHook' (BaseList2D: 1030499)
│   ├── 'Sculpt Objects' (BaseList2D: 1024182)
│   ├── 'HairHighlightHook' (BaseList2D: 1018870)
│   ├── 'MeshObject Scene Hook' (BaseList2D: 1037041)
│   ├── 'Lod Hook' (BaseList2D: 431000182)
│   ├── 'Annotation Tag SceneHook' (BaseList2D: 1030679)
│   ├── 'Sniper' (BaseList2D: 430000000)
│   ├── 'Mesh Check Hook' (BaseList2D: 431000027)
│   ├── 'Modeling Objects Hook' (BaseList2D: 431000032)
│   │   └── [Branch] 'Modeling Objects Branch' (431000031)
│   │       ├── 'Pattern Direction Manipulator' (BaseObject: Opatternmanipulator)
│   │       ├── 'Plane Manipulator' (BaseObject: Oplanemanipulator)
│   │       ├── 'Pivot Manipulator' (BaseObject: Opivotmanipulator)
│   │       ├── 'Knife Line Manipulator' (BaseObject: 431000168)
│   │       ├── 'Subdivision Manipulator' (BaseObject: 431000172)
│   │       └── 'PolyPenObject' (BaseObject: 431000031)
│   ├── 'Snap Scenehook' (BaseList2D: 440000111)
│   │   ├── [Branch] 'WpSH' (440000111)
│   │   │   └── 'WorkPlane' (BaseObject: Oworkplane)
│   │   └── [Branch] 'MdSH' (Tbase)
│   │       └── 'Modeling Settings' (BaseList2D: 440000140)
│   ├── 'Doodle Hook' (BaseList2D: 1022212)
│   ├── 'Stereoscopic' (BaseList2D: 450000226)
│   ├── 'ViewportExtHookHUD' (BaseList2D: ID_VIEW_SCENEHOOKHUD)
│   ├── 'ViewportExtHookhighlight' (BaseList2D: ID_VIEW_SCENEHOOKHIGHLIGHT)
│   ├── 'MeasureSceneHook' (BaseList2D: ID_MEASURE_SCENEHOOK)
│   ├── 'Redshift' (BaseList2D: 1036748)
│   ├── 'GvHook' (BaseList2D: ID_SCENEHOOK_PLUGIN)
│   ├── 'Material Scene Hook' (BaseList2D: 300001077)
│   ├── 'TargetDistancePicker' (BaseList2D: 1028063)
│   └── 'BodyPaint SceneHook' (BaseList2D: 1036428)
└── [Branch] '' (Tbasedraw)
    └── '' (BaseList2D: 110306)

[Aprecigout]

Thank you for the very detailed reply—your explanation of the internal workings really helps me avoid saying more “silly things.”

To give you even more context on why we lean so heavily on ExecutePasses: our artists make massive use of MoGraph, cloners, clones and theirs effectors to modify many aspects of the scene:

• Clone colours (which map directly to the LEDs on the real drones)
• Clone positions in 3-D space (hence, drone positions)
• ...

When it’s time to export the scenography to the drone-friendly format, we currently rely on a mix of ExecutePasses and direct MoGraph matrix access (GeGetMoData(), GetArray(c4d.MODATA_COLOR))

Small wrinkle: For certain type of scenography we even run the scene twice—first with some effectors on, then with them off—to harvest an extra colour value we need.

Right now, ExecutePasses is the only way I know to retrieve per-frame information for every clone. If there is a lighter-weight or more direct approach, I would be thrilled to hear about it, because I understand from your message how expensive ExecutePasses can be.

Following your advice, I also rewrote the minimal test in C++ (so, no Python VM involved) and it looks like i had the same behaviour there—so the slowdown doesn’t appear to be a Python-specific issue ?!.

Thanks again for such a comprehensive answer; it’s always a pleasure reading this forum!

[ferdinand]

Hey @Aprecigout,

helps me avoid saying more “silly things.”

You cannot make an omelette without breaking eggs, so no worries. There are no silly questions.

To give you even more context on why we lean so heavily on ExecutePasses: our artists make massive use of MoGraph ...

Thank you for the details. This gives a bit more insight. But without a concrete scene, it is still very hard to evaluate for me where this comes from (if there is a bug somewhere). As hinted at in my last posting, the Python VM is a rather unlikely suspect for a cause. But the alternative would be that our scene evaluation is massively bugged which is even more unlikely. But it is good to know that C++ is an option for you. In general, I am still not convinced that your findings, that Cinema 4D irregularly slows down on scene execution, are correct. Are you sure that you unload the documents between runs? I.e., something like this shown in [1]? Because if you do not, you will of course occupy more and more memory with each document loaded.

Right now, ExecutePasses is the only way I know to retrieve per-frame information for every clone.

This is also a bit ambiguous, but this might be wrong. Executing the passes does not necessarily mean a document will build discrete data for each clone or everything in general. It will just build what it deems necessary. When a MoGraph cloner is in 'Multi-Instance' mode it will actually only build the first clone concretely, the rest of the clones is still being described non-discretely (i.e., sparely) via the MoData tag (so that it realizes the memory saving aspect this mode promises). You can read this thread, it might contain relevant information for you.

When you want a super flattened document, you could invoke Save for Cineware from the app, or SaveDocument with the SAVEDOCUMENTFLAGS flag SAVECACHES from the API to save a document with exhaustively build caches. But once you load such document into any form of Cinema 4D (e.g., Cinema 4D, Commandline, c4dpy, etc. ) it will throw away all these caches and switch back to an optimized scene model. To faithfully read such c4d export document, you must use the Cineware AP. Which is C++ only and not always trivial in its details. But when you are comfortable with C++, and you need a truly discretely serialized document, this is the way to go. Just to be verbose: With the Cineware API you can ONLY read discrete data. Anything dynamic/computed is not available here. Because you can use the Cineware API without a Cinema 4D installation. So, executing the passes is for example not possible here. And not all aspects of a scene can be exported into this format. But MoGraph systems will be exported.

Cheers,
Ferdinand

[1]

"""A simple example for how to unload documents and execute passes in a thread.

This is pseudo code I wrote blindly and did not run.
"""


import c4d
import time

doc: c4d.documents.BaseDocument # A preloaded document, could be None.

class PassesThread (c4d.threading.C4DThread):
    """Executes the passes on a document in a thread.

    Using this inside a plain Script Manager makes no sense, since it is itself blocking. You need
    something asynchronous such as an async dialog for this to be useful.
    """
    def __init__(self, doc: c4d.documents.BaseDocument) -> None:
        self._doc = doc
        self._result: bool = False 
        self.Start()

    def Main(self) -> None:
        self._result = self._doc.ExecutePasses(
            self.Get(), True, True, True, c4d.BUILDFLAGS_NONE)

def main() -> None:
    """Called by Cinema 4D to execute the script.
    """
    # The file paths of the documents to load and execute, could also be the same file over and over again.
    for fPath in (...):
        # Kill the current document to free up resources and load the new one.
        if isinstance(doc, c4d.documents.BaseDocument):
            c4d.documents.KillDocument(doc)

        doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(fPath, ...)
        frames: list[c4d.BaseTime] = [...]
        for f in frames:
            doc.SetTime(f)

            # Create a thread to execute the passes, which makes no sense here since we will wait for
            # the outcome anyway, but it shows the principle. But you can in any case always only run
            # of these threads at a time.
            thread: PassesThread = PassesThread(doc)
            while thread.IsRunning():
                time.sleep(1.0) # Avoid blasting the thread with finish checks.

        # For the final frame, we should execute the passes again, so that simulations can settle.
        thread: PassesThread = PassesThread(doc)
        while thread.IsRunning():
            time. Sleep(1.0)

[Aprecigout]

You cannot make an omelette without breaking eggs, so no worries. There are no silly questions.

Thank you for your patience and clear explanations.

I may have wandered a little off topic, but the artists’ main objective is still to export to the drone proprietary format as fast as possible (when their is last minute change on the field), whatever the method. Our current ExecutePasses‐based implementation is definitely not carved in stone.

But it is good to know that C++ is an option for you.

Yes, moving more logic to C++ is on the table. For context, the tool-set we give the artists is already split between C++ plugins and Python scripts/plugins. Python is the historic layer that lets us prototype and satisfy quick requests; C++ lets us dig deeper into the SDK, but obviously development time is higher.

But without a concrete scene, it is still very hard to evaluate for me where this comes from

I have just sent you an example document containing a drone scenography via the e-mail address listed in https://plugincafe.maxon.net/guidlines_cp#support-data.
Hope it helps.

Are you sure that you unload the documents between runs? I.e., something like this shown in [1]? Because if you do not, you will of course occupy more and more memory with each document loaded.

You are absolutely right—I simply ported my small test to C++ and didn’t unload/reload the document between runs. I understand the issue and I’m working on implementing proper unloading on our actual scripts.

When a MoGraph cloner is in ‘Multi-Instance’ mode it will actually only build the first clone concretely …

That’s an angle I had never considered. Our artists never touch that setting—their cloners stay in Instance mode all the time. If i understand correctly the other thread, that’s actually the only reason our current ExecutePasses workflow still works: in Instance mode Cinema 4D materialises every clone, so on our call we can see full per-clone colours
Thanks again for highlighting this!

To faithfully read such c4d export document, you must use the Cineware AP

I will also investigate Cineware; if I can extract the required information directly from a flattened file, that could drastically speed up the export to our proprietary format.

Thanks again—and please don’t hesitate to tell me if, after looking at the sample scene I sent, another method for extracting the data comes to mind!

[ferdinand]

Hey @Aprecigout,

so, I had a look at your scene. For me it runs in under 0.4 seconds on each run. I did not use your code though. You do some things which are not fast when you look at the context of your document (which has 30k frames), such as, for example, printing a line for all 30k frames. But your timings do not include these calls, so they should not be reflected and certainly also not cause an increase over multiple runs (as Python simply starts to truncate its console at some point). Also, the BUILDFLAGS_EXPORTONLY flag is a bit questionable in your context, but it should make things faster and not slower. My tests are with the more common case of no build flags (but I also tried your flag, and things were slightly faster).

Not unloading the document and operating in the active document are the likely causes for the difference, except for your measuring being faulty somehow. But even ignoring this does not really explain your numbers. I used one of our builtin profiling tools, the mxutils.TIMEIT decorator.

There is of course also some difference between the timings of my calls, but with 0.355 sec, 0.360 sec, 0359 sec, ..., it is well within a to be expected variance of 5% for something as complex as scene execution.

Cheers,
Ferdinand

edit: And for good measure, I also set the count to 50, and executing the passes for the 48th, 49th, and 50th time took 0.362 sec, 0.355 sec, and 0.350 sec for me. So, even over much more repetitions nothing significant happens for me.

Result

--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 6327756096>, 30, 0, 34200), {})' with 68406 function calls in 0.355 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.013    0.013    0.355    0.355 .../untitled 19.py:6(ExecutePasses)
    34201    0.338    0.000    0.338    0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34201    0.004    0.000    0.004    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 .../python311/mxutils/mxu_base.py:107(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 22211370816>, 30, 0, 34200), {})' with 68406 function calls in 0.360 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.013    0.013    0.360    0.360 .../untitled 19.py:6(ExecutePasses)
    34201    0.340    0.000    0.340    0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34201    0.006    0.000    0.006    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 .../python311/mxutils/mxu_base.py:107(CheckType)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 22211397440>, 30, 0, 34200), {})' with 68406 function calls in 0.359 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.012    0.012    0.359    0.359 .../untitled 19.py:6(ExecutePasses)
    34201    0.342    0.000    0.342    0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34201    0.005    0.000    0.005    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 .../python311/mxutils/mxu_base.py:107(CheckType)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 22211328064>, 30, 0, 34200), {})' with 68406 function calls in 0.373 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.012    0.012    0.373    0.373 .../untitled 19.py:6(ExecutePasses)
    34201    0.356    0.000    0.356    0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34201    0.005    0.000    0.005    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 .../python311/mxutils/mxu_base.py:107(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 22211401792>, 30, 0, 34200), {})' with 68406 function calls in 0.366 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.013    0.013    0.366    0.366 .../untitled 19.py:6(ExecutePasses)
    34201    0.347    0.000    0.347    0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34201    0.005    0.000    0.005    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 .../python311/mxutils/mxu_base.py:107(CheckType)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Code

"""Provides a short example for how to execute the passes on all frames of a document, multiple times
in a row, while measuring the time it takes to execute the passes.
"""

import c4d
import mxutils

# A debug variable to control whether we want to see debug output or not and the file we are going
# to load and profile over and over again.
# 
# Always use a global constant such as IS_DEBUG when using the mxutils.TIMEIT decorator, as the 
# overhead especially for profiling (compared to pure timing) can be quite high. Profiling should 
# NEVER leak into production code! With a global constant, this is much more unlikely to happen.
IS_DEBUG: bool = True
FILE_PATH: str = "/Users/f_hoppe/Downloads/user_scene.c4d"

@mxutils.TIMEIT(enabled=IS_DEBUG, useProfiler=True, showArgs=True)
def ExecutePasses(doc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None:
   """Executes the passes for the given document in the given frame range.
   """
   mxutils.CheckType(doc, c4d.documents.BaseDocument)
   for f in range(minFrame, maxFrame + 1):
       doc.SetTime(c4d.BaseTime(f, fps))
       doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_EXPORTONLY)

def main() -> None:
   """Called by Cinema 4D to execute the script.
   """
   doc: c4d.documents.BaseDocument | None = None
   # We just iterate five times over the same document, to see if there are any slowdowns. 
   for i, p in enumerate([FILE_PATH] * 5):
       # Unload the current document if it exists and then load the new one.
       if isinstance(doc, c4d.documents.BaseDocument):
           c4d.documents.KillDocument(doc)
       doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(p, c4d.SCENEFILTER_NONE)

       c4d.gui.StatusSetText(f"Building passes for document {i + 1}/5.")

       # Execute the passes for the loaded document with our custom function which uses the 
       # mxutils.TIMEIT decorator to profile the event.
       fps: int = doc.GetFps()
       minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps)
       maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps)
       ExecutePasses(doc, fps, minFrame, maxFrame)

   c4d.gui.StatusClear()
   c4d.EventAdd()
   

if __name__ == '__main__':
   main() 

[Aprecigout]

Hi @ferdinand ,

Thank you for analysing our file, for your reply here, and for the e‑mail.
Sorry for the delay—we were completely absorbed in preparing a large show this weekend and yesterday.

I tested your code and, indeed, there is a huge difference between your execution times and ours.
Getting sub‑second performance for this operation would be a dream!

However, when I tried to apply your approach I hit reality.
Unless I’m mistaken, the flag c4d.SCENEFILTER_NONE skews the test on this scenography because no objects or materials are loaded (as a reminder, we need to extract clone data over the full duration of the scenography).
So I replaced the import flag in your script with SCENEFILTER_OBJECTS | SCENEFILTER_MATERIALS and tested with an incrementing frame count (see source code at [1]).

After that change I get results much closer to what I showed in my first post [2].

Yesterday’s feedback
During the tests we ran for yesterday’s show, we ran onto another performance problem (which is completely normal behavior for me) : once the machine starts swapping, export performance drops sharply.
We also noticed—quite logically again—that disabling all effectors, fields, and other objects via keyframes when they are not used helps ExecutePasses a lot. We’re exploring every possible avenue to speed up our exports.

We’ve observed that the longer the scenography runs, the higher the per‑frame time becomes.
We’re therefore experimenting with splitting ExecutePasses into chunks of 5 000 frames and using KillDocument() / LoadDocument() between chunks, to see if that lowers the total export time. Do you think this approach can help us ?

[1]

"""Provides a short example for how to execute the passes on all frames of a document, multiple times
in a row, while measuring the time it takes to execute the passes.
"""

import c4d
import mxutils

# A debug variable to control whether we want to see debug output or not and the file we are going
# to load and profile over and over again.
# 
# Always use a global constant such as IS_DEBUG when using the mxutils.TIMEIT decorator, as the 
# overhead especially for profiling (compared to pure timing) can be quite high. Profiling should 
# NEVER leak into production code! With a global constant, this is much more unlikely to happen.
IS_DEBUG: bool = True
FILE_PATH: str = "/Users/f_hoppe/Downloads/user_scene.c4d"

@mxutils.TIMEIT(enabled=IS_DEBUG, useProfiler=True, showArgs=True)
def ExecutePasses(doc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None:
    """Executes the passes for the given document in the given frame range.
    """
    mxutils.CheckType(doc, c4d.documents.BaseDocument)
    for f in range(minFrame, maxFrame + 1):
        doc.SetTime(c4d.BaseTime(f, fps))
        doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_EXPORTONLY)

def main() -> None:
    """Called by Cinema 4D to execute the script.
    """
    doc: c4d.documents.BaseDocument | None = None
    # Liste des maxFrame pour chaque tour
    max_frames_per_tour = [5000, 10000, 20000, 30000, None]  # None = Max Frame du document
    
    # We just iterate five times over the same document, to see if there are any slowdowns. 
    for i, p in enumerate([FILE_PATH] * 5):
        # Unload the current document if it exists and then load the new one.
        if isinstance(doc, c4d.documents.BaseDocument):
            c4d.documents.KillDocument(doc)
        doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(p, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS)

        # Déterminer le maxFrame pour ce tour
        fps: int = doc.GetFps()
        minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps)
       
        if max_frames_per_tour[i] is None:
            # Tour 5 : utiliser le Max Frame du document
            maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps)
        else:
            # Tours 1-4 : utiliser la valeur spécifiée
            maxFrame: int = max_frames_per_tour[i]
        c4d.gui.StatusSetText(f"Building passes for document {i + 1}/5 (frames: {minFrame}-{maxFrame}).")

        # Execute the passes for the loaded document with our custom function which uses the 
        # mxutils.TIMEIT decorator to profile the event.
        ExecutePasses(doc, fps, minFrame, maxFrame)

    c4d.gui.StatusClear()
    c4d.EventAdd()
   

if __name__ == '__main__':
    main() 

[2]

TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 4487911744>, 30, 0, 5000), {})' with 10006 function calls in 43.606 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.015    0.015   43.606   43.606 scriptmanager:17(ExecutePasses)
     5001   43.587    0.009   43.587    0.009 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
     5001    0.004    0.000    0.004    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 21114098496>, 30, 0, 10000), {})' with 20006 function calls in 151.821 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.038    0.038  151.821  151.821 scriptmanager:17(ExecutePasses)
    10001  151.774    0.015  151.774    0.015 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    10001    0.010    0.000    0.010    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 19602281536>, 30, 0, 20000), {})' with 40006 function calls in 429.998 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.102    0.102  429.998  429.998 scriptmanager:17(ExecutePasses)
    20001  429.872    0.021  429.872    0.021 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    20001    0.024    0.000    0.024    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 21296656704>, 30, 0, 30000), {})' with 60006 function calls in 868.329 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.165    0.165  868.329  868.329 scriptmanager:17(ExecutePasses)
    30001  868.126    0.029  868.126    0.029 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    30001    0.039    0.000    0.039    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}
--------------------------------------------------------------------------------
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called  with ID 110059 at 15425544000>, 30, 0, 34650), {})' with 69306 function calls in 1131.141 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.196    0.196 1131.141 1131.141 scriptmanager:17(ExecutePasses)
    34651 1130.895    0.033 1130.895    0.033 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects}
    34651    0.050    0.000    0.050    0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects}
        1    0.000    0.000    0.000    0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.isinstance}

[ferdinand]

Hey, you are absolutely right. I will have a look, something is fishy there, not just with your scene.

[Aprecigout]

Thanks @ferdinand ! I hope you will find something interesting to our problem

[ferdinand]

Yes, I am having a look with a profiler at why your scene behaves like it behaves. I first tried with the Python script, but that is just all too slow to get anything meaningful out of it (every seconds you profile costs you hundreds of megabytes). I today rewrote your example in C++, we should have an answer latest by next week.

But what I already can say, it that my initial statement that this concerns all scenes it not really true. When I take for example Sporty - Backflip.c4d, one of the scene assets in Example Scenes/Disciplines/Character Animation which IMHO makes a good benchmarking scene, this is not directly reflected. It has by default only roughly 90 frames, but even why I blow this up to 1000 frames, I do not see the same catastrophic exponential development as for your scene. There is some variance but it not nearly as bad. But your scene is 30 times longer and super heavy on MoGraph. Putting 34650 frames of animation (roughly 19 minutes at the FPS of the document) into a single shot, could of course be called a bit optimistic. Evaluating the scene in chunks could be an option. The thing I have not tried yet, is if this also applies to plain playback in the app. As that is nothing other than executing the passes. I.e., if letting the animation play, would slow down more and more over time. Because the animation is 1155 seconds long. When we follow your timings, with 1131.141 seconds for the 5th iteration, and the nature how the numbers develop, Cinema 4D should have issues the latest on the 6th or 7th cycle to keep up with real time playback.

[ferdinand]

Hey,

Since this is a bug, I have moved this into the bugs forum.

What is the problem?

I rewrote your example in C++ (see [1]), just to be sure, and the outcome is there the same as it was in Python (as it was expected). For your scene, I did end up with a log like this:

Loaded file '...\2025Creteil_LUX01_017.c4d\2025Creteil_LUX01_017.c4d' 1 of 5 times.
Reached frame 0 of 35000 after 0.000 seconds.
Reached frame 2500 of 35000 after 15.033 seconds.
Reached frame 5000 of 35000 after 87.554 seconds.
Reached frame 7500 of 35000 after 188.842 seconds.
Reached frame 10000 of 35000 after 324.395 seconds.
Reached frame 12500 of 35000 after 447.492 seconds.
Reached frame 15000 of 35000 after 594.437 seconds.
Reached frame 17500 of 35000 after 760.503 seconds.
Reached frame 20000 of 35000 after 950.866 seconds.

As you can see, it gets slower and slower with each chunk of 2500 frames I processed. And since our scene execution is not much more than calling ExecutePasses, I then also tried our Calculate FPS > Anim Test to check out how fast it can build the scene (Anim Test will build as fast as it can, press Esc to stop a test early, you can find the tool with the Commander (SHIFT + C) by searching for 'Calculate FPS'). As you can see, executing the first ~1000 frames took ~10sec with an FPS of 105, while doing the same for ~2000 frames took ~25sec at an FPS of ~85. And this trend continues.

After attaching a profiler, we found out that there is a bug specific to Mograph dependency handling, which hits your scene really hard. It is worsened by several factors of your scene as such as its total length and some of the specific scene elements you use.

What I have not yet figured out, is where your slowdowns about scene reloading boundaries came from. I could not really reproduce them.

We created a fix for this MoGraph bug and it will likely be contained in a future release of Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it will be delivered, as things can always change last minute. The intended release is the next major release of Cinema 4D.

For artists: The critical thing was the double-use of effectors in different group effectors. While this is a totally valid move scene-wise, it was what pushed this bug really into overdrive. So, if you can, you should avoid doing this (or just wait for 2026 which is not that far off anymore).

Solution

Your initial hunch of doing things in batches was spot on. There is however no cache which is being filled, but it is this Mograph bug which slows things down. When I profiled your scene, I also noticed that it only utilized about 15% to 20% of my CPU time. My initial statement that you cannot speed up ExecutePasses still holds true, as Cinema 4D already parallelizes as much as possible.

But what we can do in your case is parallelize the execution on a higher level. We cannot speed up a singular frame being built, but we can speed up the frames in total being built by building multiple frames in parallel. Find my code in [2].

This kind of parallelization does not work on every scene. It requires the scene to only hold frames that are individually evaluatable. I.e., it cannot contain for example simulations, such as cloth or pyro, where the cache result for a frame N depends on frame N-1, i.e., where you can only build the frames in consecutive order in a singular document. But such things can usually be cached into files which then fixes this issue.

In fact the whole workaround only works for such scenes.

Each TIMEIT event for Main() is for the payload of a thread, and run() is here the function which holds all the threads. I.e., all 10,000 frames have been built in ~ 55 seconds.

PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 0 to 999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 1000 to 1999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 2000 to 2999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 3000 to 3999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 4000 to 4999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 5000 to 5999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 6000 to 6999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 7000 to 7999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 8000 to 8999.
PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 9000 to 9999.
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 43.71924 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 44.71118 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 46.1065 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 47.58107 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 47.94522 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 48.24915 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 48.48529 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 48.70043 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 48.90281 sec
--------------------------------------------------------------------------------
TIMEIT: Ran 'Main()' in 48.97808 sec                 // This is the runtime for one of the chunks.
--------------------------------------------------------------------------------
TIMEIT: Ran 'run()' in 54.33163 sec                  // This is the total runtime.

Cheers,
Ferdinand

Code

[1] The Cpp code:

cinema::Bool Execute(cinema::BaseDocument* doc, cinema::GeDialog* parentManager)
{
	using namespace cinema;

	Filename file;
	if (!file.FileSelect(FILESELECTTYPE::SCENES, FILESELECT::LOAD, "Select file to execute passes on"_s))
	{
		ApplicationOutput("Could not load file."_s);
		return true;
	}

	const String fileName = file.GetString();
	BaseDocument* temp = nullptr;

	for (Int32 i = 0; i < 5; i++)
	{
		if (temp)
			KillDocument(temp);

		temp = LoadDocument(file, SCENEFILTER::OBJECTS, nullptr);
		if (!temp)
		{
			ApplicationOutput("Failed to load file: @.", fileName);
			break;
		}
		ApplicationOutput("Loaded file '@' @ of @ times.", fileName, i + 1, 5);

		const Int32 fps = temp->GetFps();
		const Int32 minFrame = temp->GetLoopMinTime().GetFrame(fps);
		const Int32 maxFrame = temp->GetLoopMaxTime().GetFrame(fps);
		const maxon::TimeValue startTime = maxon::TimeValue::GetTime();

		for (Int32 frame = minFrame; frame <= maxFrame; frame++)
		{
			if (frame % 2500 == 0)
				ApplicationOutput("Reached frame @ of @ after @ seconds.", frame, maxFrame, (maxon::TimeValue::GetTime() - startTime).GetSeconds());

			temp->SetTime(BaseTime(frame, fps));
			temp->ExecutePasses(nullptr, true, true, true, BUILDFLAGS::NONE);
		}

		const maxon::TimeValue elapsedTime = maxon::TimeValue::GetTime() - startTime;
		ApplicationOutput("Executed passes for @ frames in @ seconds.", (maxFrame - minFrame), elapsedTime.GetSeconds());
	}

	if (temp)
		KillDocument(temp);
	
	return true;
}

[2] The Python code:

"""Builds the passes for all frames of a document in chunks in parallel.
"""

import c4d
import mxutils
import time

IS_DEBUG: bool = True

class PassesThread (c4d.threading.C4DThread):
    """Executes the passes for a given document in a given frame range.
    """
    def __init__(self, file: str, fps: int, minFrame: int, maxFrame: int) -> None:
        """Initializes the thread with the document and frame range to process.
        """
        self._file: str = mxutils.CheckType(file, str)
        self._fps: int = mxutils.CheckType(fps, int)
        self._minFrame: int = mxutils.CheckType(minFrame, int)
        self._maxFrame: int = mxutils.CheckType(maxFrame, int)
        self._result: bool = False

        # Start the thread once it has been instantiated. We could also do this from the outside, 
        # and sometimes that is preferable, but I went with this option here.
        self.Start()
        print(f"PassesThread initialized for file {self._file} from frame {self._minFrame} to {self._maxFrame}.")

    @mxutils.TIMEIT(enabled=IS_DEBUG)
    def Main(self) -> None:
        """Called by Cinema 4D as the payload-method of the thread when it is started.

        We execute the passes for the given document in the given frame range and also load and
        unload the document for the chunk we process. This is necessary to avoid MoGraph issue and
        also to parallelize the execution of the passes (each pass needs its own document so that
        we can work on different frame ranges in parallel).
        """
        doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(
            self._file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS)
        
        for f in range(self._minFrame, self._maxFrame + 1):
            doc.SetTime(c4d.BaseTime(f, self._fps))
            if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY):
                break

        # I did not trust my own code here, :D, as the speed ups seemed to be too good to be true.
        # So, I checked if at least the last frame built everything correctly (to some extent). With 
        # 10 threads my CPU was at ~90% all the time and begging for mercy :D

        # didBuildCaches: bool = True
        # for obj in mxutils.IterateTree(doc.GetFirstObject()):
        #     # We are ignoring deform caches as it would be more complicated to figure out if 
        #     # something should have a deform cache or not. But we can check all generators for
        #     # having a cache. This of course does not check caches in caches (we would need 
        #     # mxutils.RecurseGraph for that), but this is good enough as a sanity check.
        #     if obj.GetInfo() & c4d.OBJECT_GENERATOR and not obj.GetCache():
        #         didBuildCaches = False

        # print(f"Did build caches: {didBuildCaches}")

        c4d.documents.KillDocument(doc)
        self._result = True

    @property
    def DidSucceed(self) -> bool:
        """Returns whether the passes were executed successfully.
        """
        return self._result

@mxutils.TIMEIT(enabled=IS_DEBUG)
def run() -> None: # Named run so that we can more easily distinguish it PassesThread.Main.
    """
    """
    file: str = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_SCENES, "Select a scene file", c4d.FILESELECT_LOAD)
    if not file:
        print("No file selected.")
        return
    
    # Get the documents animation data to determine the frame range.
    doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(file, c4d.SCENEFILTER_NONE)
    fps: int = doc.GetFps()
    minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps)
    maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps)

    # Split up frames into chunks of X frames to avoid memory issues with very large scenes. There 
    # is currently a bug in MoGraph scenes that causes slow-downs the longer they get. In the usual
    # frame range of a few hundred frames, this is not very noticeable, but it becomes noticeable
    # for thousands or even tens of thousands of frames. So, we split things into chunk and unload
    # the document in between. The chunk size can be adjusted to taste, from the very brief tests
    # I did, 1000 frames seems to be a good value, after that things start to slow down.
    #
    # We created a fix for this MoGraph bug and it will likely be contained in a future release of
    # Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it 
    # will be delivered, as things can always change last minute.
    chunkSize: int = 1000
    chunks: list[tuple[int, int]] = [
        (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1))
        for i in range((maxFrame - minFrame + chunkSize) // chunkSize)
    ]

    # Limit to 10 chunks for demonstration purposes, as your scene is very long. In production code,
    # you would also have to do some load balancing here, as you usually do not want to start too
    # many threads at once. Ten workers is probably already at the upper limit of what is reasonable.
    chunks = chunks[:10]

    # I did also sketch out the threading subject we already talked about. As mentioned, Cinema 4D 
    # will parallelize scene execution on its own. We cannot make one pass go faster. But your scene 
    # is relatively lightweight per frame, so I end up only using about 20% of my CPU when I run 
    # this frame by frame (as there is nothing left to optimize for Cinema 4D in that one frame).
    #
    # What is special about your scene, is that each frame is independent of the other frames. I.e.,
    # you can just jump to a frame X and execute its passes and get the correct result. That would 
    # for example not work for a scene which contains a (non-cached) simulation, e.g., a cloth or
    # pyro simulation. In that case, you would have to run the passes for all frames up to
    # frame X to get the correct result for frame X. 


    # So, what we are doing is parallelize the execution in its width, we are working on multiple 
    # chunks of frame ranges at the same time. This is not a speed-up for one frame, but it is a 
    # speed-up for the whole scene.
    
    # Create threads for each chunk and directly start them. Then loop over all the threads, 
    # removing them one by one as they finish, until all threads are done. Each frame chunk will
    # be processed in its own thread in parallel to the other chunks.
    threads: list[PassesThread] = [
        PassesThread(file, fps, startFrame, endFrame) for startFrame, endFrame in chunks
    ]

    while threads:
        threads = [thread for thread in threads if thread.IsRunning()]
        time.sleep(2.0) # Avoid spamming the threads with too many checks.

if __name__ == '__main__':
    run()

[Aprecigout]

Hey @ferdinand

Thank you so much for another extremely thorough reply!

As you pointed out in your previous post, I had never really tested with normal playback. In real‑world usage I guess the artists never noticed the issue because—on a 20‑minute+ scenography—they eventually stop playing the timeline in real time and just jump to key moments or scrub quickly.

I’m aware that our MoGraph usage is a bit “extreme”, but I’m relieved a bug has been identified; it matches the artists’ feeling that “it didn’t behave like this before.”
That said, drone shows have grown in duration, drone count, and choreography complexity in recent years, so that is probably a big factor too.

Thanks for the practical tips for the artists; I have a meeting with them later today and will brief them on the “double use of effectors in different groups” pitfall you mentioned.

Regarding the solution: we had looked into that approach before, but apparently not correctly. Retesting with your Python code gave us significant performance gains—many thanks!

Note: I’m aware of the limits of parallelisation, but for now that’s not an issue: every scenography is processed the same way and we don’t have any simulations. Still, I’ll keep a single‑threaded fallback in mind in case we need it later. (Our artists always have new ideas )

Besides that, I have a small question, this morning I ran more tests and tried using a clone of the document inside each thread [1] instead of re‑loading the document [3]. On a heavy scenography the load/destroy step takes a noticeable chunk of time, whereas cloning gives even better performance [2] vs. the reload approach [4].
Is there any downside to the usage of document cloning—e.g. hidden memory cost or something else I should watch out for?

Thanks again for your time and the deep analysis; I’m always blown away by the quality of support here.
Looking forward for the 2026!

Attachments

[1] Code using LoadDocuments (little update of your script to handle a pool of thread in whole scene duration)

"""Builds the passes for all frames of a document in chunks in parallel.
"""

import c4d
import mxutils
import time

IS_DEBUG: bool = True

class PassesThread (c4d.threading.C4DThread):
    """Executes the passes for a given document in a given frame range.
    """
    def __init__(self, file: str, fps: int, minFrame: int, maxFrame: int) -> None:
        """Initializes the thread with the document and frame range to process.
        """
        self._file: str = mxutils.CheckType(file, str)
        self._fps: int = mxutils.CheckType(fps, int)
        self._minFrame: int = mxutils.CheckType(minFrame, int)
        self._maxFrame: int = mxutils.CheckType(maxFrame, int)
        self._result: bool = False

        # Start the thread once it has been instantiated. We could also do this from the outside, 
        # and sometimes that is preferable, but I went with this option here.
        self.Start()
        print(f"PassesThread initialized for file {self._file} from frame {self._minFrame} to {self._maxFrame}.")

    @mxutils.TIMEIT(enabled=IS_DEBUG)
    def Main(self) -> None:
        """Called by Cinema 4D as the payload-method of the thread when it is started.

        We execute the passes for the given document in the given frame range and also load and
        unload the document for the chunk we process. This is necessary to avoid MoGraph issue and
        also to parallelize the execution of the passes (each pass needs its own document so that
        we can work on different frame ranges in parallel).
        """
        doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(
            self._file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS)
        
        for f in range(self._minFrame, self._maxFrame + 1):
            doc.SetTime(c4d.BaseTime(f, self._fps))
            if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY):
                break

        # I did not trust my own code here, :D, as the speed ups seemed to be too good to be true.
        # So, I checked if at least the last frame built everything correctly (to some extent). With 
        # 10 threads my CPU was at ~90% all the time and begging for mercy :D

        # didBuildCaches: bool = True
        # for obj in mxutils.IterateTree(doc.GetFirstObject()):
        #     # We are ignoring deform caches as it would be more complicated to figure out if 
        #     # something should have a deform cache or not. But we can check all generators for
        #     # having a cache. This of course does not check caches in caches (we would need 
        #     # mxutils.RecurseGraph for that), but this is good enough as a sanity check.
        #     if obj.GetInfo() & c4d.OBJECT_GENERATOR and not obj.GetCache():
        #         didBuildCaches = False

        # print(f"Did build caches: {didBuildCaches}")

        c4d.documents.KillDocument(doc)
        self._result = True

    @property
    def DidSucceed(self) -> bool:
        """Returns whether the passes were executed successfully.
        """
        return self._result

@mxutils.TIMEIT(enabled=IS_DEBUG)
def run() -> None: # Named run so that we can more easily distinguish it PassesThread.Main.
    """
    """
    file: str = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_SCENES, "Select a scene file", c4d.FILESELECT_LOAD)
    if not file:
        print("No file selected.")
        return
    
    # Get the documents animation data to determine the frame range.
    doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(file, c4d.SCENEFILTER_NONE)
    fps: int = doc.GetFps()
    minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps)
    maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps)

    # Split up frames into chunks of X frames to avoid memory issues with very large scenes. There 
    # is currently a bug in MoGraph scenes that causes slow-downs the longer they get. In the usual
    # frame range of a few hundred frames, this is not very noticeable, but it becomes noticeable
    # for thousands or even tens of thousands of frames. So, we split things into chunk and unload
    # the document in between. The chunk size can be adjusted to taste, from the very brief tests
    # I did, 1000 frames seems to be a good value, after that things start to slow down.
    #
    # We created a fix for this MoGraph bug and it will likely be contained in a future release of
    # Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it 
    # will be delivered, as things can always change last minute.
    chunkSize: int = 1000
    chunks: list[tuple[int, int]] = [
        (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1))
        for i in range((maxFrame - minFrame + chunkSize) // chunkSize)
    ]

    # Thread pool of 10 threads
    maxConcurrentThreads: int = 10
    totalChunks: int = len(chunks)
    completedChunks: int = 0
    chunksToProcess: list[tuple[int, int]] = chunks.copy()
    
    # I did also sketch out the threading subject we already talked about. As mentioned, Cinema 4D 
    # will parallelize scene execution on its own. We cannot make one pass go faster. But your scene 
    # is relatively lightweight per frame, so I end up only using about 20% of my CPU when I run 
    # this frame by frame (as there is nothing left to optimize for Cinema 4D in that one frame).
    #
    # What is special about your scene, is that each frame is independent of the other frames. I.e.,
    # you can just jump to a frame X and execute its passes and get the correct result. That would 
    # for example not work for a scene which contains a (non-cached) simulation, e.g., a cloth or
    # pyro simulation. In that case, you would have to run the passes for all frames up to
    # frame X to get the correct result for frame X. 


    # So, what we are doing is parallelize the execution in its width, we are working on multiple 
    # chunks of frame ranges at the same time. This is not a speed-up for one frame, but it is a 
    # speed-up for the whole scene.
    
    # Create threads for each chunk and directly start them. Then loop over all the threads, 
    # removing them one by one as they finish, until all threads are done. Each frame chunk will
    # be processed in its own thread in parallel to the other chunks.
    activeThreads: list[PassesThread] = []
    
    print(f"Total chunks to process: {totalChunks}")
    
    for i in range(min(maxConcurrentThreads, len(chunksToProcess))):
        startFrame, endFrame = chunksToProcess.pop(0)
        thread = PassesThread(file, fps, startFrame, endFrame)
        activeThreads.append(thread)
    
    while activeThreads or chunksToProcess:
        # Update status
        totalFrames = maxFrame - minFrame + 1
        processedFrames = completedChunks * chunkSize
        c4d.gui.StatusSetText(f"Handling scene chunks ({totalFrames} Frames) ({completedChunks}/{totalChunks})")
        
        # Check if thread finished
        finishedThreads = []
        for thread in activeThreads:
            if not thread.IsRunning():
                if thread.DidSucceed:
                    completedChunks += 1
                    print(f"Chunk completed successfully. Progress: {completedChunks}/{totalChunks}")
                else:
                    print(f"Chunk failed. Progress: {completedChunks}/{totalChunks}")
                finishedThreads.append(thread)
        
        # Remove finished threads
        for thread in finishedThreads:
            activeThreads.remove(thread)
        
        # Starts new thread to fill pool if scene is not ended
        while len(activeThreads) < maxConcurrentThreads and chunksToProcess:
            startFrame, endFrame = chunksToProcess.pop(0)
            thread = PassesThread(file, fps, startFrame, endFrame)
            activeThreads.append(thread)
            print(f"Started new thread for chunk {startFrame}-{endFrame}. Active threads: {len(activeThreads)}")
        
        time.sleep(1.0)
    
    c4d.gui.StatusSetText(f"Scene processing completed ({totalFrames} Frames) ({totalChunks}/{totalChunks})")
    print(f"All chunks processed. Total: {completedChunks}/{totalChunks}")

if __name__ == '__main__':
    run()

[2] Performance with Load/Destroy (i stripped some prints)

TIMEIT: Ran 'Main()' in 17.60864 sec
TIMEIT: Ran 'Main()' in 18.19359 sec
TIMEIT: Ran 'Main()' in 19.2093 sec
TIMEIT: Ran 'Main()' in 19.47807 sec
TIMEIT: Ran 'Main()' in 19.86996 sec
TIMEIT: Ran 'Main()' in 20.04131 sec
TIMEIT: Ran 'Main()' in 20.18938 sec
TIMEIT: Ran 'Main()' in 20.5281 sec
TIMEIT: Ran 'Main()' in 20.80194 sec
TIMEIT: Ran 'Main()' in 21.14046 sec
TIMEIT: Ran 'Main()' in 5.43413 sec
TIMEIT: Ran 'Main()' in 6.53501 sec
TIMEIT: Ran 'Main()' in 5.20132 sec
TIMEIT: Ran 'Main()' in 5.41878 sec
TIMEIT: Ran 'Main()' in 5.54765 sec
TIMEIT: Ran 'Main()' in 4.66633 sec
TIMEIT: Ran 'Main()' in 4.71172 sec
TIMEIT: Ran 'Main()' in 4.75155 sec
TIMEIT: Ran 'Main()' in 4.82119 sec
TIMEIT: Ran 'Main()' in 5.14475 sec
TIMEIT: Ran 'Main()' in 5.01931 sec
TIMEIT: Ran 'Main()' in 5.38826 sec
TIMEIT: Ran 'Main()' in 4.73412 sec
TIMEIT: Ran 'Main()' in 4.9887 sec
TIMEIT: Ran 'Main()' in 5.12512 sec
TIMEIT: Ran 'Main()' in 5.30819 sec
TIMEIT: Ran 'Main()' in 5.43618 sec
TIMEIT: Ran 'Main()' in 5.52055 sec
TIMEIT: Ran 'Main()' in 5.71583 sec
TIMEIT: Ran 'Main()' in 4.44371 sec
TIMEIT: Ran 'Main()' in 3.422 sec
TIMEIT: Ran 'Main()' in 2.4513 sec
TIMEIT: Ran 'Main()' in 3.48958 sec
TIMEIT: Ran 'Main()' in 3.50926 sec
TIMEIT: Ran 'Main()' in 3.54951 sec
All chunks processed. Total: 35/35
--------------------------------------------------------------------------------
TIMEIT: Ran 'run()' in 47.51687 sec

[3] Code using doc.GetClone()

"""Builds the passes for all frames of a document in chunks in parallel.
"""

import c4d
import mxutils
import time

IS_DEBUG: bool = True

class PassesThread (c4d.threading.C4DThread):
    """Executes the passes for a given document in a given frame range.
    """
    def __init__(self, originalDoc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None:
        """Initializes the thread with the document and frame range to process.
        """
        self._originalDoc = mxutils.CheckType(originalDoc, c4d.documents.BaseDocument)
        self._fps: int = mxutils.CheckType(fps, int)
        self._minFrame: int = mxutils.CheckType(minFrame, int)
        self._maxFrame: int = mxutils.CheckType(maxFrame, int)
        self._result: bool = False

        # Start the thread once it has been instantiated. We could also do this from the outside, 
        # and sometimes that is preferable, but I went with this option here.
        self.Start()
        print(f"PassesThread initialized from frame {self._minFrame} to {self._maxFrame}.")

    @mxutils.TIMEIT(enabled=IS_DEBUG)
    def Main(self) -> None:
        doc = self._originalDoc.GetClone(c4d.COPYFLAGS_DOCUMENT)
        
        for f in range(self._minFrame, self._maxFrame + 1):
            doc.SetTime(c4d.BaseTime(f, self._fps))
            if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY):
                break

    @property
    def DidSucceed(self) -> bool:
        """Returns whether the passes were executed successfully.
        """
        return self._result

@mxutils.TIMEIT(enabled=IS_DEBUG)
def run() -> None:
    doc = c4d.documents.GetActiveDocument()
    fps: int = doc.GetFps()
    minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps)
    maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps)
    
    chunkSize: int = 1000
    chunks: list[tuple[int, int]] = [
        (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1))
        for i in range((maxFrame - minFrame + chunkSize) // chunkSize)
    ]

    # Thread pool with maximum 10 concurrent threads
    maxConcurrentThreads: int = 10
    totalChunks: int = len(chunks)
    completedChunks: int = 0
    chunksToProcess: list[tuple[int, int]] = chunks.copy()
    activeThreads: list[PassesThread] = []
    
    print(f"Total chunks to process: {totalChunks}")
    
    # Start the first threads (up to maxConcurrentThreads)
    for i in range(min(maxConcurrentThreads, len(chunksToProcess))):
        startFrame, endFrame = chunksToProcess.pop(0)
        thread = PassesThread(doc, fps, startFrame, endFrame)
        activeThreads.append(thread)
    
    # Main thread pool loop
    while activeThreads or chunksToProcess:
        # Update status
        totalFrames = maxFrame - minFrame + 1
        processedFrames = completedChunks * chunkSize
        c4d.gui.StatusSetText(f"Handling scene chunks ({totalFrames} Frames) ({completedChunks}/{totalChunks})")
        
        # Check finished threads
        finishedThreads = []
        for thread in activeThreads:
            if not thread.IsRunning():
                if thread.DidSucceed:
                    completedChunks += 1
                    print(f"Chunk completed successfully. Progress: {completedChunks}/{totalChunks}")
                else:
                    print(f"Chunk failed. Progress: {completedChunks}/{totalChunks}")
                finishedThreads.append(thread)
        
        # Remove finished threads from the active list
        for thread in finishedThreads:
            activeThreads.remove(thread)
        
        # Start new threads if chunks are pending and we have available slots
        while len(activeThreads) < maxConcurrentThreads and chunksToProcess:
            startFrame, endFrame = chunksToProcess.pop(0)
            thread = PassesThread(doc, fps, startFrame, endFrame)
            activeThreads.append(thread)
            print(f"Started new thread for chunk {startFrame}-{endFrame}. Active threads: {len(activeThreads)}")
        
        # Wait a bit before checking again
        time.sleep(1.0)
    
    # Update final status
    c4d.gui.StatusSetText(f"Scene processing completed ({totalFrames} Frames) ({totalChunks}/{totalChunks})")
    print(f"All chunks processed. Total: {completedChunks}/{totalChunks}")

if __name__ == '__main__':
    run()

[4] Performance (Document Cloning) (I stripped many useless print ;))

Total chunks to process: 35
TIMEIT: Ran 'Main()' in 4.73206 sec
TIMEIT: Ran 'Main()' in 4.75516 sec
TIMEIT: Ran 'Main()' in 5.0718 sec
TIMEIT: Ran 'Main()' in 5.36385 sec
TIMEIT: Ran 'Main()' in 5.32922 sec
TIMEIT: Ran 'Main()' in 5.61483 sec
TIMEIT: Ran 'Main()' in 6.64787 sec
TIMEIT: Ran 'Main()' in 6.50201 sec
TIMEIT: Ran 'Main()' in 6.47022 sec
TIMEIT: Ran 'Main()' in 6.47917 sec
TIMEIT: Ran 'Main()' in 4.06786 sec
TIMEIT: Ran 'Main()' in 4.09047 sec
TIMEIT: Ran 'Main()' in 4.2562 sec
TIMEIT: Ran 'Main()' in 3.88825 sec
TIMEIT: Ran 'Main()' in 4.46758 sec
TIMEIT: Ran 'Main()' in 4.5405 sec
TIMEIT: Ran 'Main()' in 4.29954 sec
TIMEIT: Ran 'Main()' in 4.25477 sec
TIMEIT: Ran 'Main()' in 4.371 sec
TIMEIT: Ran 'Main()' in 4.40597 sec
TIMEIT: Ran 'Main()' in 4.34581 sec
TIMEIT: Ran 'Main()' in 4.07137 sec
TIMEIT: Ran 'Main()' in 5.32505 sec
TIMEIT: Ran 'Main()' in 4.6659 sec
TIMEIT: Ran 'Main()' in 5.9585 sec
TIMEIT: Ran 'Main()' in 4.23031 sec
TIMEIT: Ran 'Main()' in 4.31894 sec
TIMEIT: Ran 'Main()' in 6.03186 sec
TIMEIT: Ran 'Main()' in 5.28033 sec
TIMEIT: Ran 'Main()' in 5.33497 sec
TIMEIT: Ran 'Main()' in 3.66333 sec
TIMEIT: Ran 'Main()' in 3.74228 sec
TIMEIT: Ran 'Main()' in 1.93532 sec
TIMEIT: Ran 'Main()' in 2.65944 sec
TIMEIT: Ran 'Main()' in 2.6315 sec
All chunks processed. Total: 35/35
--------------------------------------------------------------------------------
TIMEIT: Ran 'run()' in 20.56978 sec

[ferdinand]

Hey,

your second code snippet never sets self._result.

Regarding loading vs. cloning documents: That I did it how I did it was no accident, but that does not necessarily mean that cloning things is bad. C4DAtom::GetClone is implemented in the different scene elements independently; i.e., BaseObject::GetClone, BaseDocument::GetClone, BaseTag::GetClone, etc. and C4DAtom::GetClone is just the empty implementation. The different implementations then rely on C4DAtom::CopyTo to copy data from the reference into a new instance of that scene element type. On top of that comes that Cinema 4D is largely Copy-on-Write these days, with data actually only being copied when you try to modify it. This all explains why BaseDocument::GetClone is so inexplicably performant, it just copies the metadata and only copies the real payload of the scene when it has to. On top of that comes that cloning from the same document from multiple threads in parallel could entail read-access violations (although somewhat unlikely).

On the other hand, our rendering pipeline does exactly the same, it clones documents for rendering (the cloning is however done on the main thread).

I personally would say what you are doing is probably okay, but I would not write such code as example code. I would have to spend more time on this to figure out if this is absolutely safe. Doing the actual cloning off-main-thread seems risky though (but is probably also fine in 99,9% of the cases).

Cheers,
Ferdinand