ExecutePasses to slow

[Joel]

Hi,

I have a project where the Python scripts must gather information, like the position and color, of certain objects at all frames.
As of right now, I am using SetTime and ExecutePasses. This action of SetTime ExecutePasses takes 0.13 seconds however my projects have 12000 frames therefore just the information gathering takes 26 minutes.
Is there a way to speed up this prosses?

Thank you for your time.
Regards,
Joel

[fwilleke80]

ExecutePasses() runs on all objects in the document. Depending on what those objects need to do in order to update, ExecutePasses() can take a while.

[ferdinand]

Hello @Joel,

Thank you for reaching out to us. Your question depends on context:

• Which passes are you executing? Cache, expressions, animation, all?
• What is the content of your scene? Executing the cache pass for 12,000 frames on a scene with 50 nodes is not even the same universe as 12,000 frames for all passes for a document with 50,000 nodes.

You should also share code, as we are otherwise only guessing what you do. For 12,000 frames and 1.560 sec execution time, the average is 0.12 f/sec. If that value is reasonable depends on the scene. But in general, that value seems okay, especially for Python.

As of right now, I am using SetTime and ExecutePasses. [...] This action of SetTime ExecutePasses takes 0.13

I assume you are talking here about BaseDocument.SetTime. It seems unlikely that calling BaseDocument.SetTime alone has such high runtime costs. Please share code if this is indeed the case. It seems more likely that EventAdd and ExecutePasses are the culprits.

Is there a way to speed up this prosses?

Well, there is no magic sauce with which you can make things faster.

• You can be more selective in which passes you execute. For your case, it seems unnecessary to execute the cache pass for example, animation and expressions should be enough.
• You could parallelize the task. Doing this in a manner that is actually faster is tricky because you have to clone documents. It will also not work for documents which are not 'scrubable', i.e., where the state of the frame n + 1 relies on the state of the frame n. Find a sketch below.
• Do not execute the passes at all. For something like a keyframed position animation, you do not need passes at all. You can directly interpolate the value with CCurve.GetValue. This will work for float and integer parameters, and indirectly for vectors and colors, as you can then interpolate each channel. For parameters which do not have a track you would then simply collect the static value. This will not work for data that is neither static nor a track, e.g., a position driven by Xpresso, the Align on Spline tag, etc.

Cheers,
Ferdinand

Result:

We had a brief internal discussion about the feasibility of speeding up BaseDocument.ExecutePasses in this manner. I am still doubtful that this will ever be faster. As you can see, I was unsuccessful. This was run on a document with 500 frames containing 100 objects with a position track animation. In more extreme scenes, parallelism might outrun sequentialism, but I am doubtful.

Executing SequentialPasses() took 0.0908 seconds.
Executing ParallelPasses() took 1.4128 seconds.

Code:

import c4d
import time
import math
import functools

def TimeIt(func):
    """Provides a makeshift decorator for timing functions executions.
    """
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        """Wraps and measures the execution time of the wrapped function.
        """
        t0: int = time.perf_counter()
        result: typing.Any = func(*args, **kwargs)
        print(f"Executing {func.__name__}() took {(time.perf_counter() - t0):.4f} seconds.")
        return result
    return wrapper

class PassesWrapper:
    """Wraps executing the passes on multiple document frames in parallel.
    """
    class PassesThread (c4d.threading.C4DThread):
        """Executes the passes on a document in a thread.
        """
        def __init__(self, doc: c4d.documents.BaseDocument) -> None:
            self._doc = doc
            self._result: bool = False 
            self.Start()

        def Main(self) -> None:
            self._result = self._doc.ExecutePasses(
                self.Get(), True, True, True, c4d.BUILDFLAGS_NONE)

    def __init__(self, doc: c4d.documents.BaseDocument, threadCount: int = 20) -> None:
        """
        """
        self._doc: c4d.documents.BaseDocument = doc
        self._threadCount: int = max(4, min(threadCount, 64))

    def Run(self) -> None:
        """Runs the wrapper, automatically balancing the threads.

        This has very much been written hastily, it is more an illustration than an implementation.
        """
        fps: int = self._doc.GetFps()
        tMin: float = self._doc.GetMinTime().Get()
        tMax: float = self._doc.GetMaxTime().Get()
        delta: float = tMax - tMin
        binTime: float = delta / self._threadCount
        binFrames: int = math.ceil(binTime * fps) # clamping problems here I haven't dealt with.

        # We clone #threadCount documents and then use these documents to iteratively consume all
        # frames of the document.
        clones = [self._doc.GetClone(0) for _ in range(self._threadCount)]

        # Iterate over the frame range one bin must cover per iteration. E.g. a document has
        # 100 frames which is evaluated by 20 threads means a binFrames value of 5.
        for f in range(binFrames):
            # Iterate over the documents and set their time to their offset plus the current 
            # relative frame in the bin, e.g., document 4 and the third frame: 4 * 5 + 3 == 23
            for i, doc in enumerate(clones):
                doc.SetTime(c4d.BaseTime(i * binFrames + f, fps))

            # Start the threads and let them run.
            threads: list[PassesWrapper.PassesThread] = [
                PassesWrapper.PassesThread(doc) for doc in clones]

            while threads:
                temp: list[PassesWrapper.PassesThread] = []
                for t in threads:
                    if t.IsRunning():
                        temp.append(t)
                    else:
                        t.End()
                threads = temp


@TimeIt
def SequentialPasses(doc: c4d.documents.BaseDocument) -> None:
    """Sequentially executes all passes for each frame in #doc.
    """
    fps: int = doc.GetFps()
    fMin: int = doc.GetMinTime().GetFrame(fps)
    fMax: int = doc.GetMaxTime().GetFrame(fps)

    for f in range(fMin, fMax + 1):
        t: c4d.BaseTime = c4d.BaseTime(f, fps)
        doc.SetTime(t)

        if not doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE):
            raise RuntimeError("Could not execute passes.")

@TimeIt
def ParallelPasses(docs: list[c4d.documents.BaseDocument]) -> None:
    """Executes all passes for each frame in #doc in parallel with 20 threads.
    """
    wrapper: PassesWrapper = PassesWrapper(doc, threadCount=20)
    wrapper.Run()

doc: c4d.documents.BaseDocument

def main() -> None:
    """
    """
    SequentialPasses(doc)
    ParallelPasses(doc)

if __name__ == "__main__":
    main()

[Joel]

@ferdinand
Thank you for your reply.
As of right now, I am executing all passes as I do not know for sure which passes I need.
As I said I need the position and the base object color of sphere objects that follow a cloner. And this cloner has movements and colors set by inheritances.

I have these spheres in a null so my code works in this manner (note I have two arrays for frames one for position and another for color as sometimes are different):

null_spheres = doc.SearchObject('Spheres')

spheres_names = []
spheres_loc = []
spheres_col = []

frame_start = 0
frame_end = 12000
project_fps = 24

for i in null_spheres.GetChildren():
  spheres_names.append(i.GetName())
  spheres_loc.append([])
  spheres_col.append([])

frames_loc = list(range(frame_start, frame_end+1, project_fps))
frames_col = list(range(frame_start, frame_end+1, project_fps))

doc.SetTime(c4d.BaseTime(frame_start, project_fps))
doc.ExecutePasses(None, True,  True, True, c4d.BUILDFLAGS_NONE)

for frame_col in frames_col:
   t = c4d.BaseTime(frame_col, project_fps)
   doc.SetTime(t)
   doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE)
   for index, sphere in enumerate(spheres_names):
      op = doc.SearchObject(sphere)
      col = op[c4d.ID_BASEOBJECT_COLOR]
      spheres_col[index].append([col.x, col.y, col.z])
      if frame_col in frames_loc:
         pos = op.GetMg().off
         spheres_loc[index].append([pos.x, pos.y, pos.z])

As of right now, the number of spheres is around 200 but it will increase to 500 so I want to try and speed up this process as much as possible.

[ferdinand]

Hey @Joel,

Without a scene and executable code, the code snippet does not say much.

But you do some odd things in your script which drive up its runtime costs, e. g., op = doc.SearchObject(sphere) or a lot of unnecessary data transformations. Which will ofc sum up if you do them 12,000 * 200 times. What in your 27 minutes execution time is caused by non-optimized code I cannot tell you without executable code and the scene.

The code I provided below runs in 96.3 seconds on the also provided scene file with 15,000 frames and 1,000 sphere objects with a differently seeded Vibrate tag each. This value goes down to 80.1 seconds when disabling the cache pass, and down to 63.2 seconds when also disabling the animation pass (only the expression pass is contributing to my scene, since the animation is Vibrate tag driven).

When a scene contains also changing data in the cache and animation pass, the relative gains will be larger.

My system: Cinema 4D 2023.2, Win 11Pro, i7-10700K, 32 GB ram

Cheers,
Ferdinand

File: execute_passes.c4d
Code:

"""Executes all passes for each frame in the active document and collects position and color
information for all objects below the object #ROOT_OBJECT_NAME.

Runs in  96.3 seconds on 1,000 objects and 15,000 frames on my machine. This value goes down to
80.1 seconds when disabling the cache pass, and down to 63.2 seconds when also disabling the 
animation pass (only the expression pass is contributing in my scene, since the animation is
vibrate tag driven).
"""

import c4d
import time
import json
import typing
import functools

doc: c4d.documents.BaseDocument

def TimeIt(func):
    """Provides a makeshift decorator for timing functions executions.
    """
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        """Wraps and measures the execution time of the wrapped function.
        """
        t0: int = time.perf_counter()
        result: typing.Any = func(*args, **kwargs)
        print(f"Executing {func.__name__}() took {(time.perf_counter() - t0):.4f} seconds.")
        return result
    return wrapper

# --------------------------------------------------------------------------------------------------
ROOT_OBJECT_NAME: str = "sphere_root" # The object below which the sphere objects lie.
SERIALIZE_TO_JSON: bool = False       # If to serialize data to JSON, probably not a good idea for
                                      # larger amounts of frames as JSON does suck for big data.
JSON_FILE: str = "e:\\dump.json"      # The JSON file to serialize to.

@TimeIt
def main() -> None:
    """Runs the example.
    """
    # Find the root object.
    root: c4d.BaseObject = doc.SearchObject(ROOT_OBJECT_NAME)
    if not root:
        raise RuntimeError("Could not find root object.")

    # Build a list of data containers for all children of #root.
    results: list[dict[str, any]] = [
        {
            "op": child, 
            "name": child.GetName(), 
            "col": [], 
            "pos": [] 
        }
        for child in root.GetChildren()
    ]

    # The object #root has no children.
    if len(results) < 1:
        raise RuntimeError("Root object does not have any children.")

    # Get the animation span of the document.
    fps: int = doc.GetFps()
    fMin: int = doc.GetMinTime().GetFrame(fps)
    fMax: int = doc.GetMaxTime().GetFrame(fps)

    # Little helper to convert c4d.Vector instances into tuples as you did. I would avoid
    # doing this when possible, as this is quite a bit of copying of data.
    def Vec2Tuple(vec: c4d.Vector) -> tuple[float]: return (vec.x, vec.y, vec.z)

    # Execute all passes between fMin and fMax and get the position and color data of all spheres
    # for each execution.
    for i, f in enumerate(range(fMin, fMax + 1)):
        c4d.StatusSetText(f"Dumping frame: {i}/{fMax - fMin}")
        doc.SetTime(c4d.BaseTime(f, fps))
        doc.ExecutePasses(None, True, True, True, c4d.BUILDFLAGS_NONE)
        for item in results:
            item["col"].append(Vec2Tuple(item["op"][c4d.ID_BASEOBJECT_COLOR]))
            item["pos"].append(Vec2Tuple(item["op"].GetMg().off))

    # Exit when we do not want to serialize data to disk.
    if not SERIALIZE_TO_JSON:
        c4d.StatusClear()
        return

    # When we want to serialize #results, we must replace the fields "op" holding BaseObject
    # references with something which can be serialized to a JSON file. I went here for the UUID
    # of the node as a `str`.
    for item in results:
        item["op"] = str(bytes(item["op"].FindUniqueID(c4d.MAXON_CREATOR_ID)))

    # Write data to disk, will probably be the costliest part of this script.
    c4d.StatusSetText(f"Serializing data to '{JSON_FILE}'.")
    with open(JSON_FILE, "w") as f:
        json.dump(results, f, indent=2)
    c4d.StatusClear()

if __name__ == "__main__":
    main()

[Joel]

@ferdinand
May I send you the project file and executable via a private channel?

[ferdinand]

Hello @Joel,

please follow the Support Procedures: Confidential Data section in our Forum Guidelines for sharing confidential data with us.

Cheers,
Ferdinand

[jana]

Hello @Joel,

without further questions or postings, we will consider this topic as solved by Friday, the 11th of august 2023 and flag it accordingly.

Thank you for your understanding,
Maxon SDK Group