Python NODES - Shouldn't really they can change the scene?

SolarPH

Hi there!

I was seeking some answer about having AddUndo() event to Undo a MergeDocument() event on Cinema4D R14 (I haven't tested it on R19 yet because I want my rigs to be still accessible to 32-bit users), but it seems like there was no way adding an AddUndo() event to anything that came through MergeDocument().

Although I decided to scrap the Undo Part and made the flow to be:
-- Import Rig Parts > Arrange Parts > Modify UserData > Delete Importer --

I mean, there was no stability issues on Cinema4D for me, but..

Do Python NODES(correction from it being Tags before) cannot be the way to modify the scene in a dedicated rig? I mean I have ways to secure the objects being loaded by these Scene Modifiers inside Python Tags without breaking the scene, including Mesh Swapping and Object Auto-Arrangement, and the Rig Importer (when the undo part was scrapped on this one and made it a one-pass use)

Can it really affect C4D stability?

Except infinite looping because when a code was probably gonna loop, there will be always a one-cycle switch and a verifier before the code will run.

m_adam

Hi @SolarPH when you say -- Import Rig Parts > Arrange Parts > Modify UserData > Delete Importer --
it looks like these steps can come from a series of buttons pressed.
If its the case then you react to the message method and in this case, since the initial caller is a GUI interaction this comes from the main thread. And if you are in the main thread you are safe to do whatever you want (make sure to call StopAllThread, but you can even call EventAdd)

If you are doing all of that in the Execute method, then it's not ok and it could affect the c4d stability (we do have crash reports about that).
The main method of the python tag is just the method called during the Execute method of the C++ Python TagData. So that means this execution is done in a threaded environment, see Threading Information.
The whole scene execution is multithreaded, so if you modify this scene while you are in this scene execution the best case it will work because you modify something that will be evaluated after, in the wrong case you will make the change but nothing happens (because the object cache is already evaluated and you do your stuff afterward) or in the worst case, it will crash due to the threading environment you modify and object that is currently accessed in another thread.

PluginStudent

The purpose of a Python Tag is to define the behavior of existing objects in the scene, similar to an XPRESSO tag. The Python Tag is part of the scene execution pipeline, and you must not edit scene from withing the scene execution.

If you want to edit the scene (insert stuff etc.) you must do this solely from the main thread.

SolarPH

@m_adam said in Python Tags - Shouldn't really they can change the scene?:

Hi @SolarPH when you say -- Import Rig Parts > Arrange Parts > Modify UserData > Delete Importer --
it looks like these steps can come from a series of buttons pressed.

It can be, but I planned it to be all automated using a series of codes.

If you are doing all of that in the Execute method, then it's not ok and it could affect the c4d stability (we do have crash reports about that).

I've set all of those outside Execute method to ensure that it wouldn't loop infinitely. I often connect it to an if/else statement (connected to a boolean/integer switch that switches itself off after the first cycle) to ensure that it wouldn't make Cinema4D crash.

make sure to call StopAllThread, but you can even call EventAdd

I actually haven't tested StopAllThread line yet, but I can already use EventAdd() after any commands that changes the scene in terms of heirarchy and inserted objects. I haven't really tested doing a delete command, but I have some ways to let the code know what to delete after the cycle. Utilizng ports as a way to actually pass the correct object data on the code makes it easier for me to execute this kinds of codes on my python tags.

if you are in the main thread you are safe to do whatever you want

Also, According to the online manual about python codes for cinema4d (official documentation), for example, I do

c4d.documents.MergeDocument(doc,Path,c4d.SCENEFILTER_OBJECTS|c4d.SCENEFILTER_MATERIALS,None)

will make it run on Main C4D Thread (i don't actually know if not putting the None at the end of the code makes it run on another thread, but when I tried, it doesn't actually crash C4D since the line only runs once because of the nature of the switch I used)

@PluginStudent said in Python Tags - Shouldn't really they can change the scene?:

The purpose of a Python Tag is to define the behavior of existing objects in the scene, similar to an XPRESSO tag. The Python Tag is part of the scene execution pipeline, and you must not edit scene from withing the scene execution.

If you want to edit the scene (insert stuff etc.) you must do this solely from the main thread.

ALSO I just noticed... I meant Python Node, not Python Tag. lemme correct that one....

m_adam

An express tag is a tag, and it evaluation takes place as any other tag, meaning it has the same limitation as a Python Scripting Tag.

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

I've set all of those outside Execute method to ensure that it wouldn't loop infinitely. I often connect it to an if/else statement (connected to a boolean/integer switch that switches itself off after the first cycle) to ensure that it wouldn't make Cinema4D crash.

I'm not sure to follow you here but since you modify the scene content while the scene execution (aka xpresso execution) then it can affect the stability, and it does not matter if it's executed once or 100 times, so you may run into a conflict with another part of the scene while this other part is accessed by something else, and you have a crash.

make sure to call StopAllThread, but you can even call EventAdd

I actually haven't tested StopAllThread line yet, but I can already use EventAdd() after any commands that changes the scene in terms of heirarchy and inserted objects. I haven't really tested doing a delete command, but I have some ways to let the code know what to delete after the cycle. Utilizng ports as a way to actually pass the correct object data on the code makes it easier for me to execute this kinds of codes on my python tags.

I was speaking if you are in the main thread, which is not the case here because you are in a tag execution, so if you call StopAllThread within your thread, your thread will be canceled (due to the nature of python, your python code may execute completely but not the other part of the xpresso evaluation)

Also, According to the online manual about python codes for cinema4d (official documentation), for example, I do
c4d.documents.MergeDocument(doc,Path,c4d.SCENEFILTER_OBJECTS|c4d.SCENEFILTER_MATERIALS,None)
will make it run on Main C4D Thread (i don't actually know if not putting the None at the end of the code makes it run on another thread, but when I tried, it doesn't actually crash C4D since the line only runs once because of the nature of the switch I used)

Passing None in the MergeDocument will not make it run into the main thread, you just pass the thread you are currently in, and it will be this thread that will be tested if you should leave or not, but it does not modify where the code is executed. So calling MergeDocument is not fine because, in the end, you modify the current document especially if there are some materials. And this could make Cinema 4D Crash since a lot of stuff happens during a merge.

Cheers,
Maxime.

SolarPH

@m_adam said in Python NODES - Shouldn't really they can change the scene?:

Passing None in the MergeDocument will not make it run into the main thread, you just pass the thread you are currently in, and it will be this thread that will be tested if you should leave or not, but it does not modify where the code is executed. So calling MergeDocument is not fine because, in the end, you modify the current document especially if there are some materials. And this could make Cinema 4D Crash since a lot of stuff happens during a merge.

Is there any way for the execution to be placed from a python node to anywhere it's safer to run into?

So far, there's no issues about crash or lag during a merge (the merge happens on scene to preset folder method)

Also, I will send the sample files which I work into on the Rig Importer and see what can be done for making sure it was to run safer (ofcourse not letting it crash Cinema4D during merge)

PluginStudent

A Python node should do even less. It is only supposed to read data from its in-ports and write the result of some operation to its out-ports.

If you want to automate your process, why not simply write a Python Manager Script?

ferdinand

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

Is there any way for the execution to be placed from a python node to anywhere it's safer to run into?

Hi,

I have admittedly not red the whole thread here, but one approach to modifying the scene graph from a non-main-thread context could be registering a plugin id and using that id as a message id.

From your threaded context (i.e. your Xpresso Python node) you can send the id as a core message. In a separately implemented MessageData plugin you then listen for that message and upon reception set a flag in the instance of your MessageData plugin that you want to do something.

When you are sure, that you are back in the main thread in your ```MessageData`` plugin (you might have to use a timer message for that), you can execute whatever the message told you to execute.

Cheers,
zipit

SolarPH

@zipit said in Python NODES - Shouldn't really they can change the scene?:

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

Is there any way for the execution to be placed from a python node to anywhere it's safer to run into?

Hi,

I have admittedly not red the whole thread here, but one approach to modifying the scene graph from a non-main-thread context could be registering a plugin id and using that id as a message id.

From your threaded context (i.e. your Xpresso Python node) you can send the id as a core message. In a separately implemented MessageData plugin you then listen for that message and upon reception set a flag in the instance of your MessageData plugin that you want to do something.

When you are sure, that you are back in the main thread in your ```MessageData`` plugin (you might have to use a timer message for that), you can execute whatever the message told you to execute.

Cheers,
zipit

Is there any way to not make it dependent on a plugin? My actual main goal is to modify the scene (not in the level of me calling any GUI around c4d). It seems to be successful and I was just about to post the scene with the specific importer and the specific preset which my importer was made for. You can look at it later. For now, it was being tested on R19 from R14

SolarPH

@m_adam @zipit @PluginStudent
Can you check the file I have on this folder (google drive)
https://drive.google.com/open?id=1IZ76o55libdd8d_DUtljETEt7v6FU6MY
The time of the rig files being available are only limited, since this rig is sold for purchase for animators.

Upon observation, there was no delays about the objects being imported having any delay before it shows up on my Viewport, and I also noticed that in both versions (R14 and R19), obj.Remove() adds an Undo event, and I can retrieve back the Rig Importer without deleting the imported character.

On R19 observation though, it seems like c4d.EventAdd() doesn't force all updates to show up on viewport, as if it cannot update the viewport from execution source being a Tag. Since the object is a one-time use, and cannot be keyframed, it doesn't pose too much problem on the scene.

Other than that, nothing much. Just looking up for your recommendations about making a behavior like this python node, and how to make it really on the Main Thread without any need of any plugin (because I actually need this to be compatible with all versions from R14 to R21+ without updating any plugins).

ferdinand

@SolarPH said in [Python NODES - Shouldn't really they can change the scene?]

Is there any way to not make it dependent on a plugin?

I am not quite sure what you exactly do mean by that. Technically you can do all sorts of stuff, but the only things - that I could think of - that would allow you to escape into the main thread from within an expression, would clearly fall into the domain off hacks. And hacks are rarely a good idea and very likely to cause you more work than just a MessageData plugin.

I did not look at your data, but a general thought on asynchronous data access. While crashes are a certainly a possibility, the point you should be more worried about is malformed data and document states. Other parts of Cinema or other plugins might rely on a certain scene state the moment you are poking around in the scene graph (without the knowledge of Cinema) and your changes will cause them to produce malformed data. Which could lead in extreme cases to a corrupted scene. So not having Cinema crash does not really prove anything.

Cheers,
zipit

SolarPH

Also, Anyone know how to extract Thread Name? Or if possible, a list of Thread ID with their thread name would be useful.

I have put a piece of code on my node and this was what it prints out:

Also, I wonder if procesisng the threads can have a delay using a timer thing...

m_adam

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

Also, Anyone know how to extract Thread Name? Or if possible, a list of Thread ID with their thread name would be useful.

The Cinema 4d SDK offers no way to retrieve a list of all running threads, and thread doesn't have a name.

Also, I wonder if procesisng the threads can have a delay using a timer thing...

I'm not sure to understand what do you mean. In Python and in general, the SDK offers no means to interact with other threads.

In the end, the only proper way is to create a MessageData plugin to react to a Core Event and do the stuff within this MessageData plugin.

Cheers,
Maxime.

PluginStudent

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

without any need of any plugin

If you want to have interaction with a scene without plugins you could do this:

Add a Python Scripting Tag to your scene
Add user-data buttons and other UI to that tag
implement the message() function of that tag
in that message() function, you can react to interaction with the tag's user-data UI
you can write code that is executed from the main thread to edit the scene safely
and leave the poor Python Node and all that thread stuff alone...

SolarPH

@PluginStudent said in Python NODES - Shouldn't really they can change the scene?:

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

without any need of any plugin

If you want to have interaction with a scene without plugins you could do this:

Add a Python Scripting Tag to your scene

Add user-data buttons and other UI to that tag

implement the message() function of that tag

in that message() function, you can react to interaction with the tag's user-data UI

you can write code that is executed from the main thread to edit the scene safely

and leave the poor Python Node and all that thread stuff alone...

I checked with the c4d.threading.GeIsMainThread() on Python Scripting Tag and Python Generator. Seems both runs on Main Thread. I can use them, but...

Using those ways have no way to lock the code using a password, which I always do with my code to protect them against unwanted changes.

Is there any way to seal the code until it executes all?

Also, is there a way to look if a relative directory exists like "preset://mylib.lib4d/" without the other stuff inside it? That's what I need right now to do verification on the files before my python node passes the code to python generator or python scripting tag before they execute anything, just to still secure the code.

Edit: LoadFile() and ShowInFinder() doesn't work as I expected. I hope there was a specific one that works from R14 to the latest one (just for the sake of people who still use 32-bit)

EDIT(2):

After testing Python Scripting Tag with the same code but transfered using the Python Node protected inside XGroup, it has a delay until you deselect the owner, then it executes the code, but it doesn't run on the main thread as printed by c4d.threading.GeIsMainThread()

Test in a Python Generator also not want to run on Main Thread during merge...

also I can't get what message() was used

SolarPH

I got these results, later on, I encountered the said crash:

v1 (Python Node Direct) = Secured Code, Draw Delay, No Crash
v2 (Python Node Code Transfer to Python Tag) = Secured Code, Function Delay, Draw Delay, No Crash
v3 (Python Generator) = Exposed Code, No Delay, No Crash
v3.1 (Python Node Code Transfer to Python Generator) = Secured Code, Crashes before Draw

ferdinand

@SolarPH said in [Python NODES - Shouldn't really they can change the scene?]

I checked with the c4d.threading.GeIsMainThread() on Python Scripting Tag and Python Generator. Seems both runs on Main Thread. I can use them, but...

The main functions of both these objects are not guaranteed to be executed from the main thread. Like all node execution they normally run in another thread. message will run most of the time in the main thread, but you have no guarantee. The only method you can be sure of running from the main thread is draw in a Python scripting tag.

Using those ways have no way to lock the code using a password, which I always do with my code to protect them against unwanted changes.

If you only want to protect yourself against unwanted changes (not quite sure what would classify as that and what not), there are plenty of Python minifiers and obfuscators out there that will work with any Python interpreter. PyArmor is pretty popular.

Also, is there a way to look if a relative directory exists like "preset://mylib.lib4d/" without the other stuff inside it?

This statement is a bit nit-picky, but this is not really a relative path, but a url with a unusual scheme (the Maxon preset scheme that is). You can get the preset folders with c4d.storage.GeGetC4DPath() and the respective path symbols and then check with these and os.path.exists. Something like that:

import c4d
import os

# The preset library paths of Cinema 4D
PRESET_PATHS = [
    c4d.storage.GeGetC4DPath(c4d.C4D_PATH_LIBRARY),
    c4d.storage.GeGetC4DPath(c4d.C4D_PATH_LIBRARY_USER)
]

def check_preset_url_exists(preset_url):
    """Checks if a MAXON preset url does resolve.

    Args:
        preset_url (str): A url in the MAXON 'preset' scheme.
    
    Returns:
        str or False: The resolved preset path in the file-path scheme of the OS or False if the preset does not exist.
    
    Raises:
        ValueError: When preset_url is not a valid preset url.
    """
    # Half-heartedly sort out some malformed preset urls.
    if (not isinstance(preset_url, str) or 
        not preset_url.startswith("preset://") or
        not preset_url.endswith(".lib4d")):
        msg = r"Not a preset path: {}."
        raise ValueError(msg.format(preset_url))

    # Chop of the scheme
    schemeless_preset_url = preset_url[9:]
    # Join it with the the preset paths and test if such file does exist.
    for path in PRESET_PATHS:
        full_path = os.path.join(path, schemeless_preset_url)
        if os.path.exists(full_path):
            return full_path
    # When we reach this point no such preset does exist.
    return False

def main():
    """
    """
    # Should print a path on any Cinema installation.
    preset = "preset://browser/default.lib4d"
    print check_preset_url_exists(preset)
    # Should print "False" unless you have a lib called "fake_lib" in 
    # the root of any of the preset paths.
    preset = "preset://fake_lib.lib4d"
    print check_preset_url_exists(preset)

if __name__ == "__main__":
    main()

Cheers,
zipit

SolarPH

@zipit said in Python NODES - Shouldn't really they can change the scene?:

This statement is a bit nit-picky, but this is not really a relative path, but a url with a unusual scheme (the Maxon preset scheme that is). You can get the preset folders with c4d.storage.GeGetC4DPath() and the respective path symbols and then check with these and os.path.exists.

Thanks for mentioning that, and also telling the correct usage. I've beem havong a bad rime using os.path.relpath() and os.path.abspath() with os.path.exists()

Edit: One thing though, this code kinda has it's own holes, because I encountered a problem once more. I think what os.path.exists() looks up is the OS Absolute Path, which in this case, not very helpful because of two points:

1: The preset:// followed by file name do not follow the OS path of the file, like preset://thispreset.lib4d was the path, but preset://browser/thispreset.lib4d is the one which it was looking for.

The preset name on C4D do not follow the actual filename, for example, I have ThisRig by Someone.lib4d for OS path, and c4d reads it on the program as ThisRig v# - someone and have the path preset://ThisRig v# - someone.lib4d insted of the one being looked up by os.path.exists

Results on a test run using your code:
preset://browser/default.lib4d prints out the full OS path

preset://testrig.lib4d prints False

preset://broswer/testrig.lib4d prints out the full OS path if file name was testrig.lib4d

preset://broswer/testrig.lib4d prints False if file name was testrig0.lib4d

So far, I want to detect if this path exists on Cinema4D preset list, even if it doesn't exist in any of the directories being used by Cinema4D itself, as long as it's listed in the Preset list. This was due to the nature of some users just drags and drops a .lib4d file from some folder.

However, I think using a combination of detecting if a Bitmap is valid by looking at the file resolution if it matches, since it also brings the correct and always updated preset:// path (Update Textures are still a need since this isn't automatic when either Preset Folder name was changed, or it was moved to a new preset. I just thought of other ways to do that other than the Bitmap way.

ferdinand

@SolarPH said in Python NODES - Shouldn't really they can change the scene?:

Edit: One thing though, this code kinda has it's own holes, because I encountered a problem ...

I never said that it is perfect, I just implemented, what I thought you were after. If you want to find all file paths where the file name matches the file name part of a given preset URL, you could use os.walk. Something like this:

def find_preset_url_candidates(url):
    """Find all OS file paths for which the file name matches the file name
     of a given MAXON preset url.

    Args:
        url (str): The url in the MAXON 'preset' scheme to resolve.

    Returns:
        list[str]: The resolved preset paths.

    Raises:
        ValueError: When the argument preset_url is not a valid preset url.
    """
    # Half-heartedly sort out some malformed preset urls.
    url = str(url)
    if (not url.startswith("preset://") or
            not url.endswith(".lib4d")):
        msg = r"Not a preset url: {}."
        raise ValueError(msg.format(url))

    # Split the url into the scheme-path and filename-extension parts
    _, preset_file = os.path.split(url)

    candidates = []
    # For each preset path walk that path and test if any file matches the
    # file name of our preset file name.
    for preset_path in PRESET_PATHS:
        for root, folders, files in os.walk(preset_path):
            if preset_file in files:
                candidates.append((os.path.join(root, preset_file)))
    return candidates

My example does not deal with any fuzzy path searches / does not sort its output by the edit-distance (you would have to do that yourself). To test successfully against something like testrig0.lib4d you would have to implement a Hamming distance or Levenshtein distance to sort out what is close enough. For your example you would need the latter. Or you could just use a regular expression if you do not care about the edit distance.

Cheers,
zipit