Crash from processing too many xrefs.

[Visualride 0]

My plugin searches for and replaces strings of text in xrefs. It first gathers into a list all xref c4d files from a designated directory tree. If the directory contains more than a hundred or so legacy xrefs, which have several embedded xrefs, the plugin crashes.
Two questions:

1. Based on my supplied code, is there a way to optimize what I have to avoid crashes?
2. Is there a way to flush the cache after every directory of files processed? I've tried cache_clear() with no luck.
   Here are, hopefully, the relevant parts of my code:

def GetVirtualObjects(self, op, hierarchyhelp):
        dirty = op.CheckCache(hierarchyhelp) or op.IsDirty(c4d.DIRTY_DATA)
        if dirty is False:
            return op.GetCache(hierarchyhelp)

        baseNull = c4d.BaseObject(c4d.Onull)
        pluginNode = c4d.BaseObject(c4d.Onull)
        pluginNode.InsertUnder(baseNull)

        c4d.EventAdd()
        return baseNull

def Process_Process(self, op, rootDir, findTextList, replaceTextList, simulateFolderOnOff, simulateFolder, logFile_OnOff, logFileFolder):
        filesList = []
        now = datetime.now()
        logReport = ("Date: " + now.strftime("%m-%d-%Y") + '\n' + "Time: " + now.strftime("%H:%M:%S") + '\n' + '\n')

        for root, subFolders, files in os.walk(rootDir):
            for file in files:
                if file[-4:] == ".c4d":
                    pathWithFile = root + "/" + file
                    filesList.append(pathWithFile)

        changed = False
        for fixFile in filesList:
            c4d.documents.LoadFile(fixFile)
            doc = c4d.documents.GetActiveDocument()
            c4d.documents.InsertBaseDocument(doc)
            c4d.documents.SetActiveDocument(doc)
            logFile = True

            xList = []
            Selected = self.get_all_objects(doc.GetFirstObject(), [])
            for ListOb in Selected:
                theType = ListOb.GetTypeName()
                if theType == "Legacy XRef" or theType == "Simple XRef":
                    if ListOb[c4d.SCENEINSTANCE_FILENAME] != None:
                        xList.append(ListOb) #Build xref list
            for y in xList:
                xrefNameAndPath = y[c4d.SCENEINSTANCE_FILENAME]
                originalXrefNameAndPath = xrefNameAndPath
                for x in range(10):
                    if findTextList[x] != "":
                        finalXrefNameAndPath = self.ireplace(findTextList[x], replaceTextList[x], xrefNameAndPath)
                        if xrefNameAndPath != finalXrefNameAndPath:
                            y[c4d.SCENEINSTANCE_FILENAME] = finalXrefNameAndPath
                            if logFile == True:
                                logReport = (logReport + "File: " + fixFile + '\n')
                            logFile = False
                            xrefNameAndPath = finalXrefNameAndPath
                            changed = True
                if changed == True:
                    logReport = (logReport + "Original Xref Name: " + originalXrefNameAndPath + '\n' + "New Xref Name: " + finalXrefNameAndPath + '\n')

            if changed == True:
                logReport = (logReport + '\n')
                if simulateFolderOnOff == False:
                    c4d.documents.SaveDocument(doc, fixFile, c4d.SAVEDOCUMENTFLAGS_0, c4d.FORMAT_C4DEXPORT)
                if simulateFolderOnOff == True:
                    theFileName = self.get_name(fixFile)
                    simFolder_name = (simulateFolder + '/' + theFileName)
                    c4d.documents.SaveDocument(doc, simFolder_name, c4d.SAVEDOCUMENTFLAGS_0, c4d.FORMAT_C4DEXPORT)

            changed = False
            c4d.documents.KillDocument(doc)

        return logReport

[ferdinand]

Hello @Visualride-0,

thank you for reaching out to us. I doubt that a high number of Xrefs are the cause of your problem,it is more likely just a symptom of your problems that they manifest strongly with many objects present.

Situation

The code you show is here unfortunately does not provide much insight in what you are doing and you forgot to mention quite a bit of key information. Please have a look at Support Procedures: Reporting Bugs and Crashes for how to do this better in the future.

1. I am assuming you are writing some kind of NodeData plugin, likely an ObjectData plugin, because you show us here GetVirtualObjects (GVO).
2. You do not tell or show us in which context your method Process_Process is called, I am going to assume that you are doing this off-main-thread, e.g., in GVO.

Problems

1. The main problem seems to be that you are violating threading restrictions, as you seem to call your Process_Process off-main-thread (OMT). And while the docs say that changing parameters is allowed but not recommended OMT, I would heavily advise against changing SCENEINSTANCE_FILENAME in Xrefs OMT, as this implies loading a new file which entails a lot of not thread safe things. Inserting documents, setting the active document, and saving a scene are also extreme violations of thread safety.
2. If you want a node (an object, a tag, a shader, a scene hook, etc.) which modifies the active scene, you will need a button (click) or a message in whose context you will then run your non-thread-safe code (both will result in a message being sent to your node). And while NodeMessage.Message usually does run on the main thread, you will still have to check with GeIsMainThread before you are doing anything non-threadsafe. Methods like ObjectData.GetVirtualObjects or TagData.Execute will never run on the main thread.
3. You also seem to reinvent a bit the wheel with finding your Xrefs, the natural way to find scene assets would be c4d.documents.GetAllAssetsNew. In Python we do not have the bits to replace assets, as we do not have handling for the actual message MSG_GETALLASSETS as we do in C++. But you could at least streamline the getting Xrefs part.
4. Please excuse me being blunt here, but your code in Process_Process is quite a bit convoluted which makes it hard to read and debug.
   • Remove clutter like logging until you have working code. Python's file IO is also not thread safe.
   • Try to split your code into logical blocks , there is at least the scene traversal/Xrefs gathering part, there is the parameter writing part, and there is the scene saving part. It will help you to identify problems with your code.
   • Give variables, attributes, and functions a meaningful name, the list xList, the loop variables x and y, and many more are violations of that.
   • pathWithFile = root + "/" + file is not a valid way to join paths. Use os.path.join(a, b, c, d, ...) to join two or many path elements in a safe manner for all OS.

Is there a way to flush the cache after every directory of files processed? I've tried cache_clear() with no luck.

I do not understand what you are talking about. Are you talking about the cache of an geometry object in a scene? We also do not have any method called cache_clear, as we do not use underscores in function names because we use C++ naming conventions in Python.

You cannot flush object geometry caches in Cinema 4D (you can but you really should not unless you really know what you are doing). What you can do is mark an object as dirty with C4DAtom.SetDirty and manually invoke the caches for all dirty elements in a scene being rebuilt with BaseDocument.ExecutePasses. You should be VERY careful with nodes flagging themselves as dirty, as this can easily lead to crashes. Manually having to rebuild caches in the context of normal execution of a loaded scene is usually not necessary and a sign of an incorrectly designed plugin.

Cheers,
Ferdinand

[Visualride 0]

@ferdinand
Thank you for your response and suggestions.

I created a script version of my plugin, but it still crashed after running out of memory. Here it is in the bare bones version paired down to simply loading all the files into a list, opening them and closing the file. My included screenshot shows the memory load increasing till I run out of memory. How do I avoid this?

import c4d, math, string
from c4d import gui, plugins, utils, bitmaps
import os


# Main function
def main():
    
    filesList = []
    initialPath = "/Volumes/Argo3D/"
    rootDir = c4d.storage.LoadDialog(c4d.FILESELECTTYPE_ANYTHING, "***** -PLEASE CHOOSE THE DIRECTORY ROOT- *****", c4d.FILESELECT_DIRECTORY, "", initialPath) + "/"

    findText = c4d.gui.InputDialog("Enter the text to find (To replace)", "")
    if findText == "":
        c4d.gui.MessageDialog("No FIND text entered")
        return
    replaceText = c4d.gui.InputDialog("Enter the text to replace the found text", "")
    if replaceText == "":
        c4d.gui.MessageDialog("No REPLACE text entered")
        return

    for root, subFolders, files in os.walk(rootDir):
        for file in files:
            if file[-4:] == ".c4d":
                pathWithFile = root + "/" + file
                filesList.append(pathWithFile)

    for fixFile in filesList:
        changed = False
        c4d.documents.LoadFile(fixFile)
        doc = c4d.documents.GetActiveDocument()
        c4d.documents.InsertBaseDocument(doc)
        c4d.documents.SetActiveDocument(doc)
        c4d.documents.SaveDocument(doc, fixFile, c4d.SAVEDOCUMENTFLAGS_0, c4d.FORMAT_C4DEXPORT)
        c4d.documents.KillDocument(doc)
    c4d.gui.MessageDialog("Done!")


    c4d.EventAdd()

# Execute main()
if __name__=='__main__':
    main()

[ferdinand]

Hey @Visualride-0,

how many files are in your file list and how heavy is the average file in that list? Very rough ballpark numbers are enough.

Cheers,
Ferdinand

[Visualride 0]

8000 files. Average about 4mb. The total is about 29gb.

[ferdinand]

Hey @Visualride-0,

you seem to run into some form of memory leak. That you open that many files in a row on one Cinema 4D instance is of course not really intended. This could either be a l smaller leak in our API that simply manifests more or only at all when you load that many files.

Generally, you could separate out the file iteration into an external script, a shell script or a vanilla Python script if you want, and then simply run c4dpy with your script per file or for batches of 10 or 100 files. Starting and shutting down c4dpy 8000 times in a row will of course take its toll in time, so one would be incentivized to bundle up more files to bring down that number of starts and shutdowns.

I had a look at your code, and apart from your redundant document insertion, I do not see much wrong with it, as it is basically just the loop on fileList. Cinema 4D should have caught your mistake there and prevented you from opening the same file twice, but I have not tested it.

for f in filesList:
    # This line will load #f into a BaseDocument, insert it into the opened files, and make it the
    # active document. Unlike for BaseDocument.LoadDocument, the following lines are therefore 
    # redundant.
    c4d.documents.LoadFile(f)
    # doc = c4d.documents.GetActiveDocument()
    # c4d.documents.InsertBaseDocument(doc)
    # c4d.documents.SetActiveDocument(doc)
    c4d.documents.SaveDocument(c4d.documents.GetActiveDocument(), f, c4d.SAVEDOCUMENTFLAGS_0, c4d.FORMAT_C4DEXPORT)
    # This does what it should do, close the active document, and create a new one. But to be
    # ultra safe, you could also call c4d.documents.CloseAllDocuments().
    c4d.documents.KillDocument(doc)

What escapes mea bit too, is the purpose of what we are doing here, since we just overwrite files with itself. But I guess this is minimized code, so it is only meant to demonstrate the problem. For my own understanding, the files in fileList are all *.c4d files, right?

Fixing your code as shown above will likely not fix the problem, it seems more likely to me that there is a memory leak somewhere. For an intermediate solution, you will probably have to do what I proposed above, split this into batches of Cinema 4D or c4dpy instances to flush your memory by shutting down Cinema 4D.

But we would like to have a look at the problem. Could you provide a sample of the documents, something between 10 or 100 files, so that we can test it with your data? You can send me a cloud storage link via a PM on the forum or via a mail to sdk_support(at)maxon(dot)net. Mail attachments will not work here due to the 50MB limitation.

We strive for Cinema 4D being memory leak free, and if there is a severe leak we are not aware of, we should fix it. But this might take time.

Cheers,
Ferdinand