Importing pythonapi from ctypes freezes C4D

lasselauch

Hi there,
I've just noticed something slightly odd.. While doing the following:

from ctypes import pythonapi, c_void_p, py_object

either in a script or plugin, Cinema4D freezes instantly.

I came across this issue, because I wanted to implement a way to react to my SpecialEventAdd() messages. I stumbled upon this example:

https://github.com/Rokoko/rokoko-studio-live-cinema4d/blob/ba4bd7a3c7eb814b13296a7ef564b0024efb0c4d/rokoko_utils.py#L37

Is this still the way to go in 2024 to receive the SpecialEventAdd() p1 or p2 messages? And why is this instantly freezing C4D on my MacBook Air M2?

Cheers,
Lasse

m_adam

Hi @lasselauch it depends of the version you use, if its less than R23 which is bound to Python 2.7, you should use the method GetCoreMessageParamOld but if you use a newer version (which I hope) you should use GetCoreMessageParam23. You can find this information in the Python 3 Migration - Manual.

Cheers,
Maxime.

lasselauch

Hi @m_adam,
yes, that is correct. However, I experience a freeze everytime I do:
from ctypes import pythonapi
which I need either way.

ferdinand

Hey @lasselauch,

Maxime is now on vacation (for quite some time), you will have to wait for his reply.

We talked this morning about your thread, and my take was that this is all out of scope of support because at least I would neither support ctypes nor third party code (the example you linked to). But Maxime wanted to support this, so it would be up to him. SpecialEventAdd relies in C++ on casting your data, a concept that naturally does not translate well to Python. You can make stuff work, as the code demonstrates you linked to, but it is unnecessarily complicated and there are also quite some pitfalls in ctypes, most importantly threading. In Python you can send two integers with SpecialEventAdd (not very useful), or anything you can fit into an int using struct (also not that useful). You should explain what you are trying to do, then we maybe can tell you how to achieve that.

With all that being said, running from ctypes import pythonapi does not freeze my MacBook with 2024.4.0. You should also clarify which version you are using, you have tagged this posting as S24 but you are talking about 2024. But for everything ctypes related you will have to wait for Maxime.

Cheers,
Ferdinand

lasselauch

Hi Ferdinand, thanks for the reply and I totally understand the out of scope of support part!
Thanks!

Regarding the freeze, my system specs are:
15", Apple M2 2023
macOS 13.5.2 (22G91)
C4D 2024.4.0
Just trying to import ctypes freezes C4D instantly.

Regarding the SpecialEventAdd() case:
I essentially want to act after a threaded operation is finished. So the last line of my threaded function calls:
c4d.SpecialEventAdd(ids.PLUGIN_ID_EXPORT)

To distinguish between different types of export I thought I could use the p1 (x) e.g.: c4d.SpecialEventAdd(ids.PLUGIN_ID_EXPORT, x)

However when using this inside my dialog, I'm not sure how to receive the different p1 or p2 integers inside the CoreMessage():

def CoreMessage(self, id, msg):
    if id == ids.PLUGIN_ID_EXPORT:
        logger.info("PLUGIN_ID_EXPORT Message")
        if msg == x:
            logger.info("Finished thread Message")
            do_some_logic_here()
            return True
    return False

So what I'm asking is: I think different int are sufficient enough for my case, but I'm not sure how to receive those correctly within the CoreMessage or Message of a commanddata plugin.

Thanks again,
Lasse

ferdinand

Hey @lasselauch,

we will have a look at the crash, I will move the thread here into bugs when we can confirm it.

About your problem at hand: I have written a code example which should get you going. In short, the message data is stored under BFM_CORE_PAR1 for p1 and _PAR2 for p2. For the rest, please read the example. But I personally would still advise against taking such route unless one really has to.

Make sure to set USE_CTYPES to FALSE in the script when you still encounter issues importing it.

Cheers,
Ferdinand

Result

Code

"""Demonstrates sending messages using both SpecialEventAdd() and a custom message stream
 implementation.

The general idea of the MessageStream type is to have centralized place to store data as key-value
pairs. In most cases, we can just define and expose such data container in a module which is imported
by all plugins which have to share the data. I went here the slightly more complex route of the case
where plugins are not able to do this, and all come with their own implementation and the data then
being share via a "standard" module like sys, os, or what I picked here, c4d.

This also demonstrates the most simple use case of SpeicalEventAdd(), CoreMessage() and ctypes, 
where we send an integer as the message data. As expressed in the thread above, I am not a big fan
of this approach, this is all very un-pythonic and error prone.

We technically COULD do all what MessageStream does and more here with ctypes alone, but the more 
complex our data gets, the more we will have to do to unpack our data from the wrapping capsule, and
when we want to send anything else than ints, we will also have to start packing up data when
sending them. This is all just a huge pain in the *** unless you are someone who is very comfortable
with the Python C API and the memory layout of your sent data anyway.

Note:
    What I have not done here, is made this thread safe. If we have multiple threads sending messages
    to the message stream, we will have to make sure that we do not run into access conflicts. This
    could either be done via calling c4d.threading.GeThreadLock() before accessing the message stream
    and .GeThreadUnlock() after, but that will really lock up everything which is not so great. An
    alternative would be to use a Python's threading.Lock(), threading.RLock() or threading.Semaphore()
    to lock the access to a message stream. The problem is there that our threads, c4d.threading.
    C4DThread, are not fully compatible with Python's threading, so be careful when using them 
    together. But I have used threading.Semaphore() in the past without encountering any issues.
"""
USE_CTYPES: bool = True # Toggle for using ctypes and with it SpecialEventAdd() or not. Disable this
                        # when you experience crashes when importing ctypes.
IS_DEBUG: bool = True   # Toggle for debug behavior.
PID_MY_CORE_MESSAGE: int = 1063958 # A registered plugin ID to uniquely identify our core message.

import c4d
import time
import inspect

if USE_CTYPES:
    import ctypes

from mxutils import CheckType

class MessageStream:
    """Realizes a message stream that can be observed by others.

    An observer is just a delegate, i.e., a function that is called when 'something' happens in the
    message stream. Or in other words, more or less what the classic API message functions do in 
    Cinema 4D. The main difference between the observer/delegate pattern and old school messages is 
    that one has to manually register with the observable, here via AddObserver().

    Can also be used as a plain message list without any observers, i.e., in conjunction with
    SpecialEventAdd() and CoreMessage().

    I used here ints as message IDs, but we could also use strings or any other hashable type. I 
    have done this solely so that the type works better in conjunction with SpecialEventAdd() which
    is int focused.

    See also:
        https://en.wikipedia.org/wiki/Observer_pattern

    Example:
        # Define a function that will be called when something happens in the message stream,
        # the observable in Maxon terms, we are interested in.

        def OnMessageStreamEvent(event: str, mid: int, data: any) -> bool:
            if event == MessageStream.EVENT_ADD:
                print (f"An item with ID {mid} was added to the message stream.")
                return True # Consume the event.

            return False # Otherwise do not consume the event.

        # Get the message stream with the handle 'main' and add our observer to it.
        stream: MessageStream = MessageStream.GetStream("main")
        stream.AddObserver(OnMessageStreamEvent)

        # Somewhere else and some time later we can now send messages to the stream.
        MessageStream.GetStream("main").Put(1, "Hello world!")

        # Will print:
        # An item with ID 1 was added to the message stream.
    """
    EVENT_ADD: str = "ADD" # And item was added to the message stream.
    EVENT_REMOVE: str = "REMOVE" # An item was removed from the message stream.
    EVENT_FLUSH: str = "FLUSH" # The message stream was flushed.
    EVENT_UPDATE: str = "UPDATE" # An item in the message stream was updated.

    ATTACHMENT_POINT: object = c4d # The object/module to attach the global message streams to.
    STREAM_ATTRIBUTE: str = "__MESSAGE_STREAM_" # The prefix for the global message stream attributes.

    # --- Object model -----------------------------------------------------------------------------

    def __init__(self, handle: str = ""):
        """Initializes the message stream with an optional handle.
        """
        self._handle: str = CheckType(handle, str)
        self._data: dict[int, any] = {}
        self._observers: list[callable] = []

    def __str__(self) -> str:
        """Returns a string representation of the message stream.
        """
        return (f"<MessageStream('{self._handle}') at {hex(id(self)).upper()} with "
                f"{len(self._observers)} observers>")
    
    def __repr__(self) -> str:
        """Returns a string representation of the message stream.
        """
        return str(self)
    
    def __len__(self) -> int:
        """Returns the number of messages in the message stream.
        """
        return len(self._data)
    
    def __contains__(self, mid: int) -> bool:
        """Checks if a message ID is in the message stream.
        """
        return mid in self._data
    
    def __iter__(self) -> iter:
        """Returns an iterator over the message stream.
        """
        return iter(self._data)
    
    def __notify__(self, event: str, mid: int, data: any) -> None:
        """Notifies all observers of the message stream about an event until it is being consumed
        or until all observers have been notified.
        """
        for observer in self._observers:
            if observer(event, mid, data):
                break
    
    def __getitem__(self, mid: int) -> any:
        """Returns the data of a message in the message stream.
        """
        if mid not in self._data:
            raise KeyError(f"The message ID {mid} is not in the message stream.")
        return self._data[mid]
    
    def __setitem__(self, mid: int, data: any) -> None:
        """Puts a message into the message stream.
        """
        isUpdate: bool = mid in self._data
        self._data[mid] = data
        self.__notify__(self.EVENT_UPDATE if isUpdate else self.EVENT_ADD, mid, data)

    def __delitem__(self, mid: int) -> None:
        """Removes a message from the message stream.
        """
        if mid not in self._data:
            raise KeyError(f"The message ID {mid} is not in the message stream.")
        data: any = self._data[mid]
        del self._data[mid]
        self.__notify__(self.EVENT_REMOVE, mid, data)

    def __flush__(self) -> None:
        """Flushes the message stream.
        """
        self._data.clear()
        self.__notify__(self.EVENT_FLUSH, 0, None)

    # --- Properties --------------------------------------------------------------------------------

    @property
    def Handle(self) -> str:
        """Returns the handle of the message stream.
        """
        return self._handle
    
    @property
    def Observers(self) -> tuple[callable]:
        """Returns a shallow copy of the observer list of the message stream.
        """
        return tuple(self._observers)
    
    @property
    def Messages(self) -> dict[int, any]:
        """Returns a shallow copy of the messages of the message stream.
        """
        return dict(self._data)
    
    # --- Observer pattern -------------------------------------------------------------------------
    
    def AddObserver(self, observer: callable) -> None:
        """Adds a delegate to the message stream that is called when the message stream is modified.

        The delegate must have the signature observer(event: str, mid: int, data: any) -> bool. 
        Returning true will consume the event, returning false will propagate it to the next
        observer. The event can be one of the EVENT_* symbols. The mid is the message ID and the data
        is the message data.
        """
        if not callable(observer):
            raise TypeError("The observer must be a callable.")
        if observer in self._observers:
            raise ValueError("The observer is already registered.")
        
        # Make sure that the observer can in general deal with the required signature.
        try:
            sig: inspect.Signature = inspect.signature(observer)
            sig.bind(self.EVENT_ADD, 0, 0)
        except Exception as e:
            raise ValueError("The observer must have the signature observer(event: str, mid: int, "
                             "data: any) -> bool.") from e

        self._observers.append(observer)

    def RemoveObserver(self, observer: callable) -> None:
        """Removes a delegate from the message stream.
        """
        if observer not in self._observers:
            raise ValueError(f"The observer {observer} is not registered in the message "
                             f"stream {self}.")
        
        self._observers.remove(observer)

    # --- Message stream ---------------------------------------------------------------------------

    def Flush(self) -> None:
        """Flushes the message stream.
        """
        self.__flush__()

    def Put(self, mid: int, data: any) -> None:
        """Puts a message into the message stream.
        """
        self[mid] = data
    
    def Get(self, mid: int, default: any = None) -> any:
        """Gets a message from the message stream.
        """
        return self[mid] if mid in self else default
    
    def Remove(self, mid: int) -> None:
        """Removes a message from the message stream.
        """
        del self[mid]

    # --- Static methods ---------------------------------------------------------------------------

    @staticmethod
    def GetStream(handle: str = "main") -> "MessageStream":
        """Returns the globally accessible message stream which has been attached to 
        #ATTACHMENT_POINT under the given #handle.

        Doing it in this way is only necessary when we want to have a globally accessible message
        stream with multiple users which cannot share a module, otherwise we can just put this
        class in a file `stream.py`, declare there module level instance of this class and import
        this module in all plugins/modules which need it.

            ```stream.py
            class MessageStream:
                ...
            
            MY_STREAM: MessageStream = MessageStream("my_stream")
            ```

            ```plugin.py
            from stream import MY_STREAM

            class MyPlugin (...):
                def __init__(self):
                    MY_STREAM.AddObserver(self.OnMessageStreamEvent)
                    ...
            ```
            ```other_plugin.py
            from stream import MY_STREAM

            class OtherPlugin (...):
                    def __init__(self):
                        MY_STREAM.AddObserver(self.OnMessageStreamEvent)
                        ...
            ```
        """
        attribute: str = f"{MessageStream.STREAM_ATTRIBUTE}_{handle.upper()}__"
        if not hasattr(MessageStream.ATTACHMENT_POINT, attribute):
            setattr(MessageStream.ATTACHMENT_POINT, attribute, MessageStream(handle))

        stream: MessageStream = getattr(MessageStream.ATTACHMENT_POINT, attribute, None)
        if ((stream.__class__.__qualname__ != MessageStream.__qualname__) or 
            (stream.Handle != handle)):
            raise ValueError(f"Could not get the message stream for the handle '{handle}'.")

        return stream
    
    @staticmethod
    def RemoveAllStreams() -> None:
        """Removes all globally accessible streams which have been attached to #ATTACHMENT_POINT.

        Doing this can become necessary in development, as we otherwise cannot reload/update the 
        type when it is still in memory.
        """
        for attr in dir(MessageStream.ATTACHMENT_POINT):
            if attr.startswith(MessageStream.STREAM_ATTRIBUTE):
                delattr(MessageStream.ATTACHMENT_POINT, attr)

if IS_DEBUG:
    MessageStream.RemoveAllStreams()

# --- Implementation of a message dialog which makes use of all this -------------------------------

class MessageDialog(c4d.gui.GeDialog):
    """Realizes a dialog that sends messages using both SpecialEventAdd() and the custom message
    stream implementation.
    """
    # The IDs of the dialog elements.
    ID_BUTTON: int = 1000
    ID_MESSAGE: int = 1001

    def __init__(self):
        """Initializes the dialog by attaching an observer to the default custom message stream.
        """
        MessageStream.GetStream().AddObserver(self.OnMessageStreamEvent)
    
    def CreateLayout(self) -> bool:
        """Sets up a UI to send a string message via the custom message stream.
        """
        self.SetTitle("Message Dialog")
        self.GroupBorderSpace(10, 10, 10, 10)
        self.GroupSpace(5, 5)
        self.AddEditText(self.ID_MESSAGE, c4d.BFH_SCALEFIT, initw=200)
        self.AddButton(self.ID_BUTTON, c4d.BFH_CENTER, name="Send Message")
        self.SetString(self.ID_MESSAGE, "Hello world!")

        return True
    
    def Command(self, cid: int, msg: c4d.BaseContainer) -> bool:
        """Sets of both a message via the custom message stream and a core message when the user
        presses the "Send Message" button.
        """
        if cid == self.ID_BUTTON:
            # Define the message ID as the current time in seconds and the message as the text in
            # the edit field.
            mid: int = int(time.time())
            msg: str = self.GetString(self.ID_MESSAGE)

            # Send a message via the our custom message stream implementation.
            stream: MessageStream = MessageStream.GetStream()
            stream.Put(mid, msg)
            print (f"Sent message with ID {mid} to {stream}.")

            # And also invoke a core message using SpecialEventAdd() and ctypes, we just send here
            # the message ID as p1 while we rely on storing the actual message in the message stream.
            # We could also send more complex data here, but this can all get quite complex quite 
            # quickly.
            if USE_CTYPES:
                c4d.SpecialEventAdd(PID_MY_CORE_MESSAGE, p1=mid, p2=0)
                print (f"Sent core message with the ID {PID_MY_CORE_MESSAGE} and data {mid}.")
        
        return True
    
    def OnMessageStreamEvent(self, event: str, mid: int, data: any) -> bool:
        """Called when a message is sent to the custom message stream implementation.
        """
        print (f"An '{event}' event occurred for message with ID '{mid}' and data '{data}'.")
        return False # We do not consume the event.
    
    def CoreMessage(self, cid: int, msg: c4d.BaseContainer) -> bool:
        """Called by Cinema 4D when a core message is sent.
        """
        # We receive a core message which notifies us that something has put something into our 
        # custom message stream, the data which is being sent to us via the core message is just the
        # identifier of the message in our custom message stream. The core message data contains the
        # p1 and p2 values of the SpecialEventAdd() call which we must unpack here (just p1 in our
        # case).
        if cid == PID_MY_CORE_MESSAGE and USE_CTYPES:
            capsule: any = msg.GetVoid(c4d.BFM_CORE_PAR1)
            ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_int
            ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object]
            p1: int = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None)

            # Do something with the message.
            print (f"Received core message with packed p1 data: {p1}")
            data: str | None = MessageStream.GetStream().Get(p1, None)
            print (f"Resolved to actual payload '{data}' for code message p1 payload '{p1}'.")
        
        return True

if __name__ == "__main__":
    dlg: MessageDialog = MessageDialog()
    dlg.Open(c4d.DLG_TYPE_MODAL_RESIZEABLE)

ferdinand

Hey @lasselauch,

We are not able to reproduce this crash on an Intel, M1, or M3 MacBook with 2024.4.0. Please provide and submit a crash report when this is still a problem for you. I would also recommend reinstalling Cinema 4D to rule out that your installation was damaged.

Cheers,
Ferdinand