Ways around the Python GIL

ElementDan

Hey all!

So I'm currently trying to do something processor intensive and would very much like to utilise more cores. Multithreading isn't much use because of the GIL. And multiprocessing...if it could work it wouldn't be able to call any C4D functions so that's not useful. So I came up with another idea: Launching several instances of Cinema in parallel and splitting the workload across them via commandline arguments intercepted by a message plugin. But, for some reason that no longer works:
"Warning: Unknown arguments: -parallel"
Has this function been removed in recent versions?

I tried commandline also but it closes instantly because I'm not rendering anything.
Any help here would be greatly appreciated.

Manuel

Hi,

the -parallel option was removed with the R21 and the new licence system. You cannot launch several times the same Cinema4D executable. You need to duplicate the directory and have several licences. Even if it seems to work on mac, it's not safe and guaranty.
The only solution is to move to c++. What are you trying to do?

Cheers,
Manuel

ElementDan

Was trying to avoid that, I'm building a plugin which natively generates minecraft terrain from minecraft save files. It works pretty well, just could be much faster with multithreading or multiprocessing. I have no experience with C++ but I guess its time to bite the bullet.

Manuel

Interesting. Well not only with c++ you will be able to use parallelFor but just c++ alone is a lot faster.

orestiskon

@m_magalhaes Hi Manuel, would it be viable to make a python library using C++ and call that when intense calculations are necessary?

m_adam

Hi @orestiskon, all the python c4d module is just a Python C++ library wrapped around the Cinema C++ SDK.
So of course you can compile a particular part of your code in C++ (using the C++ SDK) and then call this with Python.

Unfortually wrapping only parrallelFor will make 0 sense (and pretty hard to do to synchronize Python object together, but lets say that technical wise it is easy as porting GetActiveDocument) since Python is only working in 1 thread, 1 CPU, there is no logic of porting parallelFor since in any case due to the nature of Python, each loop will need to wait for other to call your python script.

So the only good way would be to have a C++ function wrapping the ParrallelFor and all duties done in C++ and then expose this C++ function to Python and then call this in Python.

Cheers,
Maxime.

orestiskon

@m_adam Hi Maxime,
sounds good, do you know of any resources or examples on getting started with something like this?

m_adam

This is using the old API, Add functions to python via C++ SDK [R20]

This is also possible to do it with the new python framework, unfortually we don't have any public example for that, I will see what I can come up within the next days, but if you are dealing with Classic API types, it's most likely that the old API is more suited for your needed

Cheers,
Maxime.

orestiskon

@m_adam thank you, I'll try that out and return with questions

ferdinand

Hello @orestiskon,

without any further questions or replies, we will consider this thread as solved by Monday the 20th and flag it accordingly.

Thank you for your understanding,
Ferdinand