ExecutePasses performance keeps slowing down per-frame and per-run — possible cache issue?
-
Hey @Aprecigout,
helps me avoid saying more “silly things.”
You cannot make an omelette without breaking eggs, so no worries. There are no silly questions.
To give you even more context on why we lean so heavily on ExecutePasses: our artists make massive use of MoGraph ...
Thank you for the details. This gives a bit more insight. But without a concrete scene, it is still very hard to evaluate for me where this comes from (if there is a bug somewhere). As hinted at in my last posting, the Python VM is a rather unlikely suspect for a cause. But the alternative would be that our scene evaluation is massively bugged which is even more unlikely. But it is good to know that C++ is an option for you. In general, I am still not convinced that your findings, that Cinema 4D irregularly slows down on scene execution, are correct. Are you sure that you unload the documents between runs? I.e., something like this shown in [1]? Because if you do not, you will of course occupy more and more memory with each document loaded.
Right now, ExecutePasses is the only way I know to retrieve per-frame information for every clone.
This is also a bit ambiguous, but this might be wrong. Executing the passes does not necessarily mean a document will build discrete data for each clone or everything in general. It will just build what it deems necessary. When a MoGraph cloner is in 'Multi-Instance' mode it will actually only build the first clone concretely, the rest of the clones is still being described non-discretely (i.e., sparely) via the
MoDatatag (so that it realizes the memory saving aspect this mode promises). You can read this thread, it might contain relevant information for you.When you want a super flattened document, you could invoke Save for Cineware from the app, or
SaveDocumentwith theSAVEDOCUMENTFLAGSflagSAVECACHESfrom the API to save a document with exhaustively build caches. But once you load such document into any form of Cinema 4D (e.g.,Cinema 4D,Commandline,c4dpy, etc. ) it will throw away all these caches and switch back to an optimized scene model. To faithfully read suchc4dexport document, you must use the Cineware AP. Which is C++ only and not always trivial in its details. But when you are comfortable with C++, and you need a truly discretely serialized document, this is the way to go. Just to be verbose: With the Cineware API you can ONLY read discrete data. Anything dynamic/computed is not available here. Because you can use the Cineware API without a Cinema 4D installation. So, executing the passes is for example not possible here. And not all aspects of a scene can be exported into this format. But MoGraph systems will be exported.Cheers,
Ferdinand[1]
"""A simple example for how to unload documents and execute passes in a thread. This is pseudo code I wrote blindly and did not run. """ import c4d import time doc: c4d.documents.BaseDocument # A preloaded document, could be None. class PassesThread (c4d.threading.C4DThread): """Executes the passes on a document in a thread. Using this inside a plain Script Manager makes no sense, since it is itself blocking. You need something asynchronous such as an async dialog for this to be useful. """ def __init__(self, doc: c4d.documents.BaseDocument) -> None: self._doc = doc self._result: bool = False self.Start() def Main(self) -> None: self._result = self._doc.ExecutePasses( self.Get(), True, True, True, c4d.BUILDFLAGS_NONE) def main() -> None: """Called by Cinema 4D to execute the script. """ # The file paths of the documents to load and execute, could also be the same file over and over again. for fPath in (...): # Kill the current document to free up resources and load the new one. if isinstance(doc, c4d.documents.BaseDocument): c4d.documents.KillDocument(doc) doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(fPath, ...) frames: list[c4d.BaseTime] = [...] for f in frames: doc.SetTime(f) # Create a thread to execute the passes, which makes no sense here since we will wait for # the outcome anyway, but it shows the principle. But you can in any case always only run # of these threads at a time. thread: PassesThread = PassesThread(doc) while thread.IsRunning(): time.sleep(1.0) # Avoid blasting the thread with finish checks. # For the final frame, we should execute the passes again, so that simulations can settle. thread: PassesThread = PassesThread(doc) while thread.IsRunning(): time. Sleep(1.0) -
You cannot make an omelette without breaking eggs, so no worries. There are no silly questions.
Thank you for your patience and clear explanations.
I may have wandered a little off topic, but the artists’ main objective is still to export to the drone proprietary format as fast as possible (when their is last minute change on the field), whatever the method. Our current ExecutePasses‐based implementation is definitely not carved in stone.
But it is good to know that C++ is an option for you.
Yes, moving more logic to C++ is on the table. For context, the tool-set we give the artists is already split between C++ plugins and Python scripts/plugins. Python is the historic layer that lets us prototype and satisfy quick requests; C++ lets us dig deeper into the SDK, but obviously development time is higher.
But without a concrete scene, it is still very hard to evaluate for me where this comes from
I have just sent you an example document containing a drone scenography via the e-mail address listed in https://plugincafe.maxon.net/guidlines_cp#support-data.
Hope it helps.Are you sure that you unload the documents between runs? I.e., something like this shown in [1]? Because if you do not, you will of course occupy more and more memory with each document loaded.
You are absolutely right—I simply ported my small test to C++ and didn’t unload/reload the document between runs. I understand the issue and I’m working on implementing proper unloading on our actual scripts.
When a MoGraph cloner is in ‘Multi-Instance’ mode it will actually only build the first clone concretely …
That’s an angle I had never considered. Our artists never touch that setting—their cloners stay in Instance mode all the time. If i understand correctly the other thread, that’s actually the only reason our current ExecutePasses workflow still works: in Instance mode Cinema 4D materialises every clone, so on our call we can see full per-clone colours
Thanks again for highlighting this!To faithfully read such c4d export document, you must use the Cineware AP
I will also investigate Cineware; if I can extract the required information directly from a flattened file, that could drastically speed up the export to our proprietary format.
Thanks again—and please don’t hesitate to tell me if, after looking at the sample scene I sent, another method for extracting the data comes to mind!
-
Hey @Aprecigout,
so, I had a look at your scene. For me it runs in under 0.4 seconds on each run. I did not use your code though. You do some things which are not fast when you look at the context of your document (which has 30k frames), such as, for example, printing a line for all 30k frames. But your timings do not include these calls, so they should not be reflected and certainly also not cause an increase over multiple runs (as Python simply starts to truncate its console at some point). Also, the
BUILDFLAGS_EXPORTONLYflag is a bit questionable in your context, but it should make things faster and not slower. My tests are with the more common case of no build flags (but I also tried your flag, and things were slightly faster).Not unloading the document and operating in the active document are the likely causes for the difference, except for your measuring being faulty somehow. But even ignoring this does not really explain your numbers. I used one of our builtin profiling tools, the
mxutils.TIMEITdecorator.There is of course also some difference between the timings of my calls, but with 0.355 sec, 0.360 sec, 0359 sec, ..., it is well within a to be expected variance of 5% for something as complex as scene execution.
Cheers,
Ferdinandedit: And for good measure, I also set the count to
50, and executing the passes for the 48th, 49th, and 50th time took 0.362 sec, 0.355 sec, and 0.350 sec for me. So, even over much more repetitions nothing significant happens for me.Result
-------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 6327756096>, 30, 0, 34200), {})' with 68406 function calls in 0.355 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.013 0.013 0.355 0.355 .../untitled 19.py:6(ExecutePasses) 34201 0.338 0.000 0.338 0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34201 0.004 0.000 0.004 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 .../python311/mxutils/mxu_base.py:107(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 22211370816>, 30, 0, 34200), {})' with 68406 function calls in 0.360 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.013 0.013 0.360 0.360 .../untitled 19.py:6(ExecutePasses) 34201 0.340 0.000 0.340 0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34201 0.006 0.000 0.006 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 .../python311/mxutils/mxu_base.py:107(CheckType) 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 22211397440>, 30, 0, 34200), {})' with 68406 function calls in 0.359 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.012 0.012 0.359 0.359 .../untitled 19.py:6(ExecutePasses) 34201 0.342 0.000 0.342 0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34201 0.005 0.000 0.005 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 .../python311/mxutils/mxu_base.py:107(CheckType) 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 22211328064>, 30, 0, 34200), {})' with 68406 function calls in 0.373 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.012 0.012 0.373 0.373 .../untitled 19.py:6(ExecutePasses) 34201 0.356 0.000 0.356 0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34201 0.005 0.000 0.005 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 .../python311/mxutils/mxu_base.py:107(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 22211401792>, 30, 0, 34200), {})' with 68406 function calls in 0.366 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.013 0.013 0.366 0.366 .../untitled 19.py:6(ExecutePasses) 34201 0.347 0.000 0.347 0.000 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34201 0.005 0.000 0.005 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 .../python311/mxutils/mxu_base.py:107(CheckType) 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}Code
"""Provides a short example for how to execute the passes on all frames of a document, multiple times in a row, while measuring the time it takes to execute the passes. """ import c4d import mxutils # A debug variable to control whether we want to see debug output or not and the file we are going # to load and profile over and over again. # # Always use a global constant such as IS_DEBUG when using the mxutils.TIMEIT decorator, as the # overhead especially for profiling (compared to pure timing) can be quite high. Profiling should # NEVER leak into production code! With a global constant, this is much more unlikely to happen. IS_DEBUG: bool = True FILE_PATH: str = "/Users/f_hoppe/Downloads/user_scene.c4d" @mxutils.TIMEIT(enabled=IS_DEBUG, useProfiler=True, showArgs=True) def ExecutePasses(doc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None: """Executes the passes for the given document in the given frame range. """ mxutils.CheckType(doc, c4d.documents.BaseDocument) for f in range(minFrame, maxFrame + 1): doc.SetTime(c4d.BaseTime(f, fps)) doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_EXPORTONLY) def main() -> None: """Called by Cinema 4D to execute the script. """ doc: c4d.documents.BaseDocument | None = None # We just iterate five times over the same document, to see if there are any slowdowns. for i, p in enumerate([FILE_PATH] * 5): # Unload the current document if it exists and then load the new one. if isinstance(doc, c4d.documents.BaseDocument): c4d.documents.KillDocument(doc) doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(p, c4d.SCENEFILTER_NONE) c4d.gui.StatusSetText(f"Building passes for document {i + 1}/5.") # Execute the passes for the loaded document with our custom function which uses the # mxutils.TIMEIT decorator to profile the event. fps: int = doc.GetFps() minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps) maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps) ExecutePasses(doc, fps, minFrame, maxFrame) c4d.gui.StatusClear() c4d.EventAdd() if __name__ == '__main__': main() -
Hi @ferdinand ,
Thank you for analysing our file, for your reply here, and for the e‑mail.
Sorry for the delay—we were completely absorbed in preparing a large show this weekend and yesterday.
I tested your code and, indeed, there is a huge difference between your execution times and ours.
Getting sub‑second performance for this operation would be a dream!
However, when I tried to apply your approach I hit reality.
Unless I’m mistaken, the flagc4d.SCENEFILTER_NONEskews the test on this scenography because no objects or materials are loaded (as a reminder, we need to extract clone data over the full duration of the scenography).
So I replaced the import flag in your script withSCENEFILTER_OBJECTS | SCENEFILTER_MATERIALSand tested with an incrementing frame count (see source code at [1]).After that change I get results much closer to what I showed in my first post [2].
Yesterday’s feedback
During the tests we ran for yesterday’s show, we ran onto another performance problem (which is completely normal behavior for me) : once the machine starts swapping, export performance drops sharply.
We also noticed—quite logically again—that disabling all effectors, fields, and other objects via keyframes when they are not used helpsExecutePassesa lot. We’re exploring every possible avenue to speed up our exports.We’ve observed that the longer the scenography runs, the higher the per‑frame time becomes.
We’re therefore experimenting with splittingExecutePassesinto chunks of 5 000 frames and usingKillDocument()/LoadDocument()between chunks, to see if that lowers the total export time. Do you think this approach can help us ?[1]
"""Provides a short example for how to execute the passes on all frames of a document, multiple times in a row, while measuring the time it takes to execute the passes. """ import c4d import mxutils # A debug variable to control whether we want to see debug output or not and the file we are going # to load and profile over and over again. # # Always use a global constant such as IS_DEBUG when using the mxutils.TIMEIT decorator, as the # overhead especially for profiling (compared to pure timing) can be quite high. Profiling should # NEVER leak into production code! With a global constant, this is much more unlikely to happen. IS_DEBUG: bool = True FILE_PATH: str = "/Users/f_hoppe/Downloads/user_scene.c4d" @mxutils.TIMEIT(enabled=IS_DEBUG, useProfiler=True, showArgs=True) def ExecutePasses(doc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None: """Executes the passes for the given document in the given frame range. """ mxutils.CheckType(doc, c4d.documents.BaseDocument) for f in range(minFrame, maxFrame + 1): doc.SetTime(c4d.BaseTime(f, fps)) doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_EXPORTONLY) def main() -> None: """Called by Cinema 4D to execute the script. """ doc: c4d.documents.BaseDocument | None = None # Liste des maxFrame pour chaque tour max_frames_per_tour = [5000, 10000, 20000, 30000, None] # None = Max Frame du document # We just iterate five times over the same document, to see if there are any slowdowns. for i, p in enumerate([FILE_PATH] * 5): # Unload the current document if it exists and then load the new one. if isinstance(doc, c4d.documents.BaseDocument): c4d.documents.KillDocument(doc) doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(p, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS) # Déterminer le maxFrame pour ce tour fps: int = doc.GetFps() minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps) if max_frames_per_tour[i] is None: # Tour 5 : utiliser le Max Frame du document maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps) else: # Tours 1-4 : utiliser la valeur spécifiée maxFrame: int = max_frames_per_tour[i] c4d.gui.StatusSetText(f"Building passes for document {i + 1}/5 (frames: {minFrame}-{maxFrame}).") # Execute the passes for the loaded document with our custom function which uses the # mxutils.TIMEIT decorator to profile the event. ExecutePasses(doc, fps, minFrame, maxFrame) c4d.gui.StatusClear() c4d.EventAdd() if __name__ == '__main__': main()[2]
TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 4487911744>, 30, 0, 5000), {})' with 10006 function calls in 43.606 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.015 0.015 43.606 43.606 scriptmanager:17(ExecutePasses) 5001 43.587 0.009 43.587 0.009 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 5001 0.004 0.000 0.004 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 21114098496>, 30, 0, 10000), {})' with 20006 function calls in 151.821 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.038 0.038 151.821 151.821 scriptmanager:17(ExecutePasses) 10001 151.774 0.015 151.774 0.015 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 10001 0.010 0.000 0.010 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 19602281536>, 30, 0, 20000), {})' with 40006 function calls in 429.998 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.102 0.102 429.998 429.998 scriptmanager:17(ExecutePasses) 20001 429.872 0.021 429.872 0.021 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 20001 0.024 0.000 0.024 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 21296656704>, 30, 0, 30000), {})' with 60006 function calls in 868.329 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.165 0.165 868.329 868.329 scriptmanager:17(ExecutePasses) 30001 868.126 0.029 868.126 0.029 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 30001 0.039 0.000 0.039 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -------------------------------------------------------------------------------- TIMEIT: Ran 'ExecutePasses((<c4d.documents.BaseDocument object called with ID 110059 at 15425544000>, 30, 0, 34650), {})' with 69306 function calls in 1131.141 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.196 0.196 1131.141 1131.141 scriptmanager:17(ExecutePasses) 34651 1130.895 0.033 1130.895 0.033 {method 'ExecutePasses' of 'c4d.documents.BaseDocument' objects} 34651 0.050 0.000 0.050 0.000 {method 'SetTime' of 'c4d.documents.BaseDocument' objects} 1 0.000 0.000 0.000 0.000 /Applications/Maxon Cinema 4D 2025/resource/modules/python/libs/python311/mxutils/mxu_base.py:101(CheckType) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance} -
Hey, you are absolutely right. I will have a look, something is fishy there, not just with your scene.
-
Thanks @ferdinand ! I hope you will find something interesting to our problem
-
Yes, I am having a look with a profiler at why your scene behaves like it behaves. I first tried with the Python script, but that is just all too slow to get anything meaningful out of it (every seconds you profile costs you hundreds of megabytes). I today rewrote your example in C++, we should have an answer latest by next week.
But what I already can say, it that my initial statement that this concerns all scenes it not really true. When I take for example
Sporty - Backflip.c4d, one of the scene assets inExample Scenes/Disciplines/Character Animationwhich IMHO makes a good benchmarking scene, this is not directly reflected. It has by default only roughly 90 frames, but even why I blow this up to 1000 frames, I do not see the same catastrophic exponential development as for your scene. There is some variance but it not nearly as bad. But your scene is 30 times longer and super heavy on MoGraph. Putting 34650 frames of animation (roughly 19 minutes at the FPS of the document) into a single shot, could of course be called a bit optimistic. Evaluating the scene in chunks could be an option. The thing I have not tried yet, is if this also applies to plain playback in the app. As that is nothing other than executing the passes. I.e., if letting the animation play, would slow down more and more over time. Because the animation is 1155 seconds long. When we follow your timings, with 1131.141 seconds for the 5th iteration, and the nature how the numbers develop, Cinema 4D should have issues the latest on the 6th or 7th cycle to keep up with real time playback.MAXON SDK Specialist
developers.maxon.net -
Hey,
Since this is a bug, I have moved this into the bugs forum.What is the problem?
I rewrote your example in C++ (see [1]), just to be sure, and the outcome is there the same as it was in Python (as it was expected). For your scene, I did end up with a log like this:
Loaded file '...\2025Creteil_LUX01_017.c4d\2025Creteil_LUX01_017.c4d' 1 of 5 times. Reached frame 0 of 35000 after 0.000 seconds. Reached frame 2500 of 35000 after 15.033 seconds. Reached frame 5000 of 35000 after 87.554 seconds. Reached frame 7500 of 35000 after 188.842 seconds. Reached frame 10000 of 35000 after 324.395 seconds. Reached frame 12500 of 35000 after 447.492 seconds. Reached frame 15000 of 35000 after 594.437 seconds. Reached frame 17500 of 35000 after 760.503 seconds. Reached frame 20000 of 35000 after 950.866 seconds.As you can see, it gets slower and slower with each chunk of 2500 frames I processed. And since our scene execution is not much more than calling
ExecutePasses, I then also tried our Calculate FPS > Anim Test to check out how fast it can build the scene (Anim Test will build as fast as it can, pressEscto stop a test early, you can find the tool with the Commander (SHIFT + C) by searching for 'Calculate FPS'). As you can see, executing the first ~1000 frames took ~10sec with an FPS of 105, while doing the same for ~2000 frames took ~25sec at an FPS of ~85. And this trend continues.
After attaching a profiler, we found out that there is a bug specific to Mograph dependency handling, which hits your scene really hard. It is worsened by several factors of your scene as such as its total length and some of the specific scene elements you use.
What I have not yet figured out, is where your slowdowns about scene reloading boundaries came from. I could not really reproduce them. We created a fix for this MoGraph bug and it will likely be contained in a future release of Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it will be delivered, as things can always change last minute. The intended release is the next major release of Cinema 4D. For artists: The critical thing was the double-use of effectors in different group effectors. While this is a totally valid move scene-wise, it was what pushed this bug really into overdrive. So, if you can, you should avoid doing this (or just wait for 2026 which is not that far off anymore).Solution
Your initial hunch of doing things in batches was spot on. There is however no cache which is being filled, but it is this Mograph bug which slows things down. When I profiled your scene, I also noticed that it only utilized about 15% to 20% of my CPU time. My initial statement that you cannot speed up
ExecutePassesstill holds true, as Cinema 4D already parallelizes as much as possible.But what we can do in your case is parallelize the execution on a higher level. We cannot speed up a singular frame being built, but we can speed up the frames in total being built by building multiple frames in parallel. Find my code in [2].
This kind of parallelization does not work on every scene. It requires the scene to only hold frames that are individually evaluatable. I.e., it cannot contain for example simulations, such as cloth or pyro, where the cache result for a frame N depends on frame N-1, i.e., where you can only build the frames in consecutive order in a singular document. But such things can usually be cached into files which then fixes this issue.In fact the whole workaround only works for such scenes.
Each
TIMEITevent forMain()is for the payload of a thread, andrun()is here the function which holds all the threads. I.e., all 10,000 frames have been built in ~ 55 seconds.PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 0 to 999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 1000 to 1999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 2000 to 2999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 3000 to 3999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 4000 to 4999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 5000 to 5999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 6000 to 6999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 7000 to 7999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 8000 to 8999. PassesThread initialized for file ...\2025Creteil_LUX01_017.c4d from frame 9000 to 9999. -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 43.71924 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 44.71118 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 46.1065 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 47.58107 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 47.94522 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 48.24915 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 48.48529 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 48.70043 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 48.90281 sec -------------------------------------------------------------------------------- TIMEIT: Ran 'Main()' in 48.97808 sec // This is the runtime for one of the chunks. -------------------------------------------------------------------------------- TIMEIT: Ran 'run()' in 54.33163 sec // This is the total runtime.Cheers,
FerdinandCode
[1] The Cpp code:
cinema::Bool Execute(cinema::BaseDocument* doc, cinema::GeDialog* parentManager) { using namespace cinema; Filename file; if (!file.FileSelect(FILESELECTTYPE::SCENES, FILESELECT::LOAD, "Select file to execute passes on"_s)) { ApplicationOutput("Could not load file."_s); return true; } const String fileName = file.GetString(); BaseDocument* temp = nullptr; for (Int32 i = 0; i < 5; i++) { if (temp) KillDocument(temp); temp = LoadDocument(file, SCENEFILTER::OBJECTS, nullptr); if (!temp) { ApplicationOutput("Failed to load file: @.", fileName); break; } ApplicationOutput("Loaded file '@' @ of @ times.", fileName, i + 1, 5); const Int32 fps = temp->GetFps(); const Int32 minFrame = temp->GetLoopMinTime().GetFrame(fps); const Int32 maxFrame = temp->GetLoopMaxTime().GetFrame(fps); const maxon::TimeValue startTime = maxon::TimeValue::GetTime(); for (Int32 frame = minFrame; frame <= maxFrame; frame++) { if (frame % 2500 == 0) ApplicationOutput("Reached frame @ of @ after @ seconds.", frame, maxFrame, (maxon::TimeValue::GetTime() - startTime).GetSeconds()); temp->SetTime(BaseTime(frame, fps)); temp->ExecutePasses(nullptr, true, true, true, BUILDFLAGS::NONE); } const maxon::TimeValue elapsedTime = maxon::TimeValue::GetTime() - startTime; ApplicationOutput("Executed passes for @ frames in @ seconds.", (maxFrame - minFrame), elapsedTime.GetSeconds()); } if (temp) KillDocument(temp); return true; }[2] The Python code:
"""Builds the passes for all frames of a document in chunks in parallel. """ import c4d import mxutils import time IS_DEBUG: bool = True class PassesThread (c4d.threading.C4DThread): """Executes the passes for a given document in a given frame range. """ def __init__(self, file: str, fps: int, minFrame: int, maxFrame: int) -> None: """Initializes the thread with the document and frame range to process. """ self._file: str = mxutils.CheckType(file, str) self._fps: int = mxutils.CheckType(fps, int) self._minFrame: int = mxutils.CheckType(minFrame, int) self._maxFrame: int = mxutils.CheckType(maxFrame, int) self._result: bool = False # Start the thread once it has been instantiated. We could also do this from the outside, # and sometimes that is preferable, but I went with this option here. self.Start() print(f"PassesThread initialized for file {self._file} from frame {self._minFrame} to {self._maxFrame}.") @mxutils.TIMEIT(enabled=IS_DEBUG) def Main(self) -> None: """Called by Cinema 4D as the payload-method of the thread when it is started. We execute the passes for the given document in the given frame range and also load and unload the document for the chunk we process. This is necessary to avoid MoGraph issue and also to parallelize the execution of the passes (each pass needs its own document so that we can work on different frame ranges in parallel). """ doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument( self._file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS) for f in range(self._minFrame, self._maxFrame + 1): doc.SetTime(c4d.BaseTime(f, self._fps)) if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY): break # I did not trust my own code here, :D, as the speed ups seemed to be too good to be true. # So, I checked if at least the last frame built everything correctly (to some extent). With # 10 threads my CPU was at ~90% all the time and begging for mercy :D # didBuildCaches: bool = True # for obj in mxutils.IterateTree(doc.GetFirstObject()): # # We are ignoring deform caches as it would be more complicated to figure out if # # something should have a deform cache or not. But we can check all generators for # # having a cache. This of course does not check caches in caches (we would need # # mxutils.RecurseGraph for that), but this is good enough as a sanity check. # if obj.GetInfo() & c4d.OBJECT_GENERATOR and not obj.GetCache(): # didBuildCaches = False # print(f"Did build caches: {didBuildCaches}") c4d.documents.KillDocument(doc) self._result = True @property def DidSucceed(self) -> bool: """Returns whether the passes were executed successfully. """ return self._result @mxutils.TIMEIT(enabled=IS_DEBUG) def run() -> None: # Named run so that we can more easily distinguish it PassesThread.Main. """ """ file: str = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_SCENES, "Select a scene file", c4d.FILESELECT_LOAD) if not file: print("No file selected.") return # Get the documents animation data to determine the frame range. doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(file, c4d.SCENEFILTER_NONE) fps: int = doc.GetFps() minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps) maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps) # Split up frames into chunks of X frames to avoid memory issues with very large scenes. There # is currently a bug in MoGraph scenes that causes slow-downs the longer they get. In the usual # frame range of a few hundred frames, this is not very noticeable, but it becomes noticeable # for thousands or even tens of thousands of frames. So, we split things into chunk and unload # the document in between. The chunk size can be adjusted to taste, from the very brief tests # I did, 1000 frames seems to be a good value, after that things start to slow down. # # We created a fix for this MoGraph bug and it will likely be contained in a future release of # Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it # will be delivered, as things can always change last minute. chunkSize: int = 1000 chunks: list[tuple[int, int]] = [ (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1)) for i in range((maxFrame - minFrame + chunkSize) // chunkSize) ] # Limit to 10 chunks for demonstration purposes, as your scene is very long. In production code, # you would also have to do some load balancing here, as you usually do not want to start too # many threads at once. Ten workers is probably already at the upper limit of what is reasonable. chunks = chunks[:10] # I did also sketch out the threading subject we already talked about. As mentioned, Cinema 4D # will parallelize scene execution on its own. We cannot make one pass go faster. But your scene # is relatively lightweight per frame, so I end up only using about 20% of my CPU when I run # this frame by frame (as there is nothing left to optimize for Cinema 4D in that one frame). # # What is special about your scene, is that each frame is independent of the other frames. I.e., # you can just jump to a frame X and execute its passes and get the correct result. That would # for example not work for a scene which contains a (non-cached) simulation, e.g., a cloth or # pyro simulation. In that case, you would have to run the passes for all frames up to # frame X to get the correct result for frame X. # So, what we are doing is parallelize the execution in its width, we are working on multiple # chunks of frame ranges at the same time. This is not a speed-up for one frame, but it is a # speed-up for the whole scene. # Create threads for each chunk and directly start them. Then loop over all the threads, # removing them one by one as they finish, until all threads are done. Each frame chunk will # be processed in its own thread in parallel to the other chunks. threads: list[PassesThread] = [ PassesThread(file, fps, startFrame, endFrame) for startFrame, endFrame in chunks ] while threads: threads = [thread for thread in threads if thread.IsRunning()] time.sleep(2.0) # Avoid spamming the threads with too many checks. if __name__ == '__main__': run() -
F ferdinand moved this topic from Cinema 4D SDK on
-
Hey @ferdinand
Thank you so much for another extremely thorough reply!
As you pointed out in your previous post, I had never really tested with normal playback. In real‑world usage I guess the artists never noticed the issue because—on a 20‑minute+ scenography—they eventually stop playing the timeline in real time and just jump to key moments or scrub quickly.
I’m aware that our MoGraph usage is a bit “extreme”, but I’m relieved a bug has been identified; it matches the artists’ feeling that “it didn’t behave like this before.”
That said, drone shows have grown in duration, drone count, and choreography complexity in recent years, so that is probably a big factor too.
Thanks for the practical tips for the artists; I have a meeting with them later today and will brief them on the “double use of effectors in different groups” pitfall you mentioned.
Regarding the solution: we had looked into that approach before, but apparently not correctly. Retesting with your Python code gave us significant performance gains—many thanks!
Note: I’m aware of the limits of parallelisation, but for now that’s not an issue: every scenography is processed the same way and we don’t have any simulations. Still, I’ll keep a single‑threaded fallback in mind in case we need it later. (Our artists always have new ideas
)Besides that, I have a small question, this morning I ran more tests and tried using a clone of the document inside each thread [1] instead of re‑loading the document [3]. On a heavy scenography the load/destroy step takes a noticeable chunk of time, whereas cloning gives even better performance [2] vs. the reload approach [4].
Is there any downside to the usage of document cloning—e.g. hidden memory cost or something else I should watch out for?Thanks again for your time and the deep analysis; I’m always blown away by the quality of support here.
Looking forward for the 2026!Attachments
[1] Code using LoadDocuments (little update of your script to handle a pool of thread in whole scene duration)
"""Builds the passes for all frames of a document in chunks in parallel. """ import c4d import mxutils import time IS_DEBUG: bool = True class PassesThread (c4d.threading.C4DThread): """Executes the passes for a given document in a given frame range. """ def __init__(self, file: str, fps: int, minFrame: int, maxFrame: int) -> None: """Initializes the thread with the document and frame range to process. """ self._file: str = mxutils.CheckType(file, str) self._fps: int = mxutils.CheckType(fps, int) self._minFrame: int = mxutils.CheckType(minFrame, int) self._maxFrame: int = mxutils.CheckType(maxFrame, int) self._result: bool = False # Start the thread once it has been instantiated. We could also do this from the outside, # and sometimes that is preferable, but I went with this option here. self.Start() print(f"PassesThread initialized for file {self._file} from frame {self._minFrame} to {self._maxFrame}.") @mxutils.TIMEIT(enabled=IS_DEBUG) def Main(self) -> None: """Called by Cinema 4D as the payload-method of the thread when it is started. We execute the passes for the given document in the given frame range and also load and unload the document for the chunk we process. This is necessary to avoid MoGraph issue and also to parallelize the execution of the passes (each pass needs its own document so that we can work on different frame ranges in parallel). """ doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument( self._file, c4d.SCENEFILTER_OBJECTS | c4d.SCENEFILTER_MATERIALS) for f in range(self._minFrame, self._maxFrame + 1): doc.SetTime(c4d.BaseTime(f, self._fps)) if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY): break # I did not trust my own code here, :D, as the speed ups seemed to be too good to be true. # So, I checked if at least the last frame built everything correctly (to some extent). With # 10 threads my CPU was at ~90% all the time and begging for mercy :D # didBuildCaches: bool = True # for obj in mxutils.IterateTree(doc.GetFirstObject()): # # We are ignoring deform caches as it would be more complicated to figure out if # # something should have a deform cache or not. But we can check all generators for # # having a cache. This of course does not check caches in caches (we would need # # mxutils.RecurseGraph for that), but this is good enough as a sanity check. # if obj.GetInfo() & c4d.OBJECT_GENERATOR and not obj.GetCache(): # didBuildCaches = False # print(f"Did build caches: {didBuildCaches}") c4d.documents.KillDocument(doc) self._result = True @property def DidSucceed(self) -> bool: """Returns whether the passes were executed successfully. """ return self._result @mxutils.TIMEIT(enabled=IS_DEBUG) def run() -> None: # Named run so that we can more easily distinguish it PassesThread.Main. """ """ file: str = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_SCENES, "Select a scene file", c4d.FILESELECT_LOAD) if not file: print("No file selected.") return # Get the documents animation data to determine the frame range. doc: c4d.documents.BaseDocument = c4d.documents.LoadDocument(file, c4d.SCENEFILTER_NONE) fps: int = doc.GetFps() minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps) maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps) # Split up frames into chunks of X frames to avoid memory issues with very large scenes. There # is currently a bug in MoGraph scenes that causes slow-downs the longer they get. In the usual # frame range of a few hundred frames, this is not very noticeable, but it becomes noticeable # for thousands or even tens of thousands of frames. So, we split things into chunk and unload # the document in between. The chunk size can be adjusted to taste, from the very brief tests # I did, 1000 frames seems to be a good value, after that things start to slow down. # # We created a fix for this MoGraph bug and it will likely be contained in a future release of # Cinema 4D. But as always, we cannot guarantee that this will be delivered at all or when it # will be delivered, as things can always change last minute. chunkSize: int = 1000 chunks: list[tuple[int, int]] = [ (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1)) for i in range((maxFrame - minFrame + chunkSize) // chunkSize) ] # Thread pool of 10 threads maxConcurrentThreads: int = 10 totalChunks: int = len(chunks) completedChunks: int = 0 chunksToProcess: list[tuple[int, int]] = chunks.copy() # I did also sketch out the threading subject we already talked about. As mentioned, Cinema 4D # will parallelize scene execution on its own. We cannot make one pass go faster. But your scene # is relatively lightweight per frame, so I end up only using about 20% of my CPU when I run # this frame by frame (as there is nothing left to optimize for Cinema 4D in that one frame). # # What is special about your scene, is that each frame is independent of the other frames. I.e., # you can just jump to a frame X and execute its passes and get the correct result. That would # for example not work for a scene which contains a (non-cached) simulation, e.g., a cloth or # pyro simulation. In that case, you would have to run the passes for all frames up to # frame X to get the correct result for frame X. # So, what we are doing is parallelize the execution in its width, we are working on multiple # chunks of frame ranges at the same time. This is not a speed-up for one frame, but it is a # speed-up for the whole scene. # Create threads for each chunk and directly start them. Then loop over all the threads, # removing them one by one as they finish, until all threads are done. Each frame chunk will # be processed in its own thread in parallel to the other chunks. activeThreads: list[PassesThread] = [] print(f"Total chunks to process: {totalChunks}") for i in range(min(maxConcurrentThreads, len(chunksToProcess))): startFrame, endFrame = chunksToProcess.pop(0) thread = PassesThread(file, fps, startFrame, endFrame) activeThreads.append(thread) while activeThreads or chunksToProcess: # Update status totalFrames = maxFrame - minFrame + 1 processedFrames = completedChunks * chunkSize c4d.gui.StatusSetText(f"Handling scene chunks ({totalFrames} Frames) ({completedChunks}/{totalChunks})") # Check if thread finished finishedThreads = [] for thread in activeThreads: if not thread.IsRunning(): if thread.DidSucceed: completedChunks += 1 print(f"Chunk completed successfully. Progress: {completedChunks}/{totalChunks}") else: print(f"Chunk failed. Progress: {completedChunks}/{totalChunks}") finishedThreads.append(thread) # Remove finished threads for thread in finishedThreads: activeThreads.remove(thread) # Starts new thread to fill pool if scene is not ended while len(activeThreads) < maxConcurrentThreads and chunksToProcess: startFrame, endFrame = chunksToProcess.pop(0) thread = PassesThread(file, fps, startFrame, endFrame) activeThreads.append(thread) print(f"Started new thread for chunk {startFrame}-{endFrame}. Active threads: {len(activeThreads)}") time.sleep(1.0) c4d.gui.StatusSetText(f"Scene processing completed ({totalFrames} Frames) ({totalChunks}/{totalChunks})") print(f"All chunks processed. Total: {completedChunks}/{totalChunks}") if __name__ == '__main__': run()[2] Performance with Load/Destroy (i stripped some prints)
TIMEIT: Ran 'Main()' in 17.60864 sec TIMEIT: Ran 'Main()' in 18.19359 sec TIMEIT: Ran 'Main()' in 19.2093 sec TIMEIT: Ran 'Main()' in 19.47807 sec TIMEIT: Ran 'Main()' in 19.86996 sec TIMEIT: Ran 'Main()' in 20.04131 sec TIMEIT: Ran 'Main()' in 20.18938 sec TIMEIT: Ran 'Main()' in 20.5281 sec TIMEIT: Ran 'Main()' in 20.80194 sec TIMEIT: Ran 'Main()' in 21.14046 sec TIMEIT: Ran 'Main()' in 5.43413 sec TIMEIT: Ran 'Main()' in 6.53501 sec TIMEIT: Ran 'Main()' in 5.20132 sec TIMEIT: Ran 'Main()' in 5.41878 sec TIMEIT: Ran 'Main()' in 5.54765 sec TIMEIT: Ran 'Main()' in 4.66633 sec TIMEIT: Ran 'Main()' in 4.71172 sec TIMEIT: Ran 'Main()' in 4.75155 sec TIMEIT: Ran 'Main()' in 4.82119 sec TIMEIT: Ran 'Main()' in 5.14475 sec TIMEIT: Ran 'Main()' in 5.01931 sec TIMEIT: Ran 'Main()' in 5.38826 sec TIMEIT: Ran 'Main()' in 4.73412 sec TIMEIT: Ran 'Main()' in 4.9887 sec TIMEIT: Ran 'Main()' in 5.12512 sec TIMEIT: Ran 'Main()' in 5.30819 sec TIMEIT: Ran 'Main()' in 5.43618 sec TIMEIT: Ran 'Main()' in 5.52055 sec TIMEIT: Ran 'Main()' in 5.71583 sec TIMEIT: Ran 'Main()' in 4.44371 sec TIMEIT: Ran 'Main()' in 3.422 sec TIMEIT: Ran 'Main()' in 2.4513 sec TIMEIT: Ran 'Main()' in 3.48958 sec TIMEIT: Ran 'Main()' in 3.50926 sec TIMEIT: Ran 'Main()' in 3.54951 sec All chunks processed. Total: 35/35 -------------------------------------------------------------------------------- TIMEIT: Ran 'run()' in 47.51687 sec[3] Code using
doc.GetClone()"""Builds the passes for all frames of a document in chunks in parallel. """ import c4d import mxutils import time IS_DEBUG: bool = True class PassesThread (c4d.threading.C4DThread): """Executes the passes for a given document in a given frame range. """ def __init__(self, originalDoc: c4d.documents.BaseDocument, fps: int, minFrame: int, maxFrame: int) -> None: """Initializes the thread with the document and frame range to process. """ self._originalDoc = mxutils.CheckType(originalDoc, c4d.documents.BaseDocument) self._fps: int = mxutils.CheckType(fps, int) self._minFrame: int = mxutils.CheckType(minFrame, int) self._maxFrame: int = mxutils.CheckType(maxFrame, int) self._result: bool = False # Start the thread once it has been instantiated. We could also do this from the outside, # and sometimes that is preferable, but I went with this option here. self.Start() print(f"PassesThread initialized from frame {self._minFrame} to {self._maxFrame}.") @mxutils.TIMEIT(enabled=IS_DEBUG) def Main(self) -> None: doc = self._originalDoc.GetClone(c4d.COPYFLAGS_DOCUMENT) for f in range(self._minFrame, self._maxFrame + 1): doc.SetTime(c4d.BaseTime(f, self._fps)) if not doc.ExecutePasses(self.Get(), True, True, True, c4d.BUILDFLAGS_EXPORTONLY): break @property def DidSucceed(self) -> bool: """Returns whether the passes were executed successfully. """ return self._result @mxutils.TIMEIT(enabled=IS_DEBUG) def run() -> None: doc = c4d.documents.GetActiveDocument() fps: int = doc.GetFps() minFrame: int = doc[c4d.DOCUMENT_MINTIME].GetFrame(fps) maxFrame: int = doc[c4d.DOCUMENT_MAXTIME].GetFrame(fps) chunkSize: int = 1000 chunks: list[tuple[int, int]] = [ (minFrame + i * chunkSize, min(maxFrame, minFrame + (i + 1) * chunkSize - 1)) for i in range((maxFrame - minFrame + chunkSize) // chunkSize) ] # Thread pool with maximum 10 concurrent threads maxConcurrentThreads: int = 10 totalChunks: int = len(chunks) completedChunks: int = 0 chunksToProcess: list[tuple[int, int]] = chunks.copy() activeThreads: list[PassesThread] = [] print(f"Total chunks to process: {totalChunks}") # Start the first threads (up to maxConcurrentThreads) for i in range(min(maxConcurrentThreads, len(chunksToProcess))): startFrame, endFrame = chunksToProcess.pop(0) thread = PassesThread(doc, fps, startFrame, endFrame) activeThreads.append(thread) # Main thread pool loop while activeThreads or chunksToProcess: # Update status totalFrames = maxFrame - minFrame + 1 processedFrames = completedChunks * chunkSize c4d.gui.StatusSetText(f"Handling scene chunks ({totalFrames} Frames) ({completedChunks}/{totalChunks})") # Check finished threads finishedThreads = [] for thread in activeThreads: if not thread.IsRunning(): if thread.DidSucceed: completedChunks += 1 print(f"Chunk completed successfully. Progress: {completedChunks}/{totalChunks}") else: print(f"Chunk failed. Progress: {completedChunks}/{totalChunks}") finishedThreads.append(thread) # Remove finished threads from the active list for thread in finishedThreads: activeThreads.remove(thread) # Start new threads if chunks are pending and we have available slots while len(activeThreads) < maxConcurrentThreads and chunksToProcess: startFrame, endFrame = chunksToProcess.pop(0) thread = PassesThread(doc, fps, startFrame, endFrame) activeThreads.append(thread) print(f"Started new thread for chunk {startFrame}-{endFrame}. Active threads: {len(activeThreads)}") # Wait a bit before checking again time.sleep(1.0) # Update final status c4d.gui.StatusSetText(f"Scene processing completed ({totalFrames} Frames) ({totalChunks}/{totalChunks})") print(f"All chunks processed. Total: {completedChunks}/{totalChunks}") if __name__ == '__main__': run()[4] Performance (Document Cloning) (I stripped many useless print ;))
Total chunks to process: 35 TIMEIT: Ran 'Main()' in 4.73206 sec TIMEIT: Ran 'Main()' in 4.75516 sec TIMEIT: Ran 'Main()' in 5.0718 sec TIMEIT: Ran 'Main()' in 5.36385 sec TIMEIT: Ran 'Main()' in 5.32922 sec TIMEIT: Ran 'Main()' in 5.61483 sec TIMEIT: Ran 'Main()' in 6.64787 sec TIMEIT: Ran 'Main()' in 6.50201 sec TIMEIT: Ran 'Main()' in 6.47022 sec TIMEIT: Ran 'Main()' in 6.47917 sec TIMEIT: Ran 'Main()' in 4.06786 sec TIMEIT: Ran 'Main()' in 4.09047 sec TIMEIT: Ran 'Main()' in 4.2562 sec TIMEIT: Ran 'Main()' in 3.88825 sec TIMEIT: Ran 'Main()' in 4.46758 sec TIMEIT: Ran 'Main()' in 4.5405 sec TIMEIT: Ran 'Main()' in 4.29954 sec TIMEIT: Ran 'Main()' in 4.25477 sec TIMEIT: Ran 'Main()' in 4.371 sec TIMEIT: Ran 'Main()' in 4.40597 sec TIMEIT: Ran 'Main()' in 4.34581 sec TIMEIT: Ran 'Main()' in 4.07137 sec TIMEIT: Ran 'Main()' in 5.32505 sec TIMEIT: Ran 'Main()' in 4.6659 sec TIMEIT: Ran 'Main()' in 5.9585 sec TIMEIT: Ran 'Main()' in 4.23031 sec TIMEIT: Ran 'Main()' in 4.31894 sec TIMEIT: Ran 'Main()' in 6.03186 sec TIMEIT: Ran 'Main()' in 5.28033 sec TIMEIT: Ran 'Main()' in 5.33497 sec TIMEIT: Ran 'Main()' in 3.66333 sec TIMEIT: Ran 'Main()' in 3.74228 sec TIMEIT: Ran 'Main()' in 1.93532 sec TIMEIT: Ran 'Main()' in 2.65944 sec TIMEIT: Ran 'Main()' in 2.6315 sec All chunks processed. Total: 35/35 -------------------------------------------------------------------------------- TIMEIT: Ran 'run()' in 20.56978 sec -
Hey,
your second code snippet never sets
self._result.Regarding loading vs. cloning documents: That I did it how I did it was no accident, but that does not necessarily mean that cloning things is bad.
C4DAtom::GetCloneis implemented in the different scene elements independently; i.e.,BaseObject::GetClone,BaseDocument::GetClone,BaseTag::GetClone, etc. andC4DAtom::GetCloneis just the empty implementation. The different implementations then rely onC4DAtom::CopyToto copy data from the reference into a new instance of that scene element type. On top of that comes that Cinema 4D is largely Copy-on-Write these days, with data actually only being copied when you try to modify it. This all explains whyBaseDocument::GetCloneis so inexplicably performant, it just copies the metadata and only copies the real payload of the scene when it has to. On top of that comes that cloning from the same document from multiple threads in parallel could entail read-access violations (although somewhat unlikely).On the other hand, our rendering pipeline does exactly the same, it clones documents for rendering (the cloning is however done on the main thread).
I personally would say what you are doing is probably okay, but I would not write such code as example code. I would have to spend more time on this to figure out if this is absolutely safe. Doing the actual cloning off-main-thread seems risky though (but is probably also fine in 99,9% of the cases).
Cheers,
Ferdinand