External dependencies: The right way to do

Helper

On 11/03/2013 at 14:17, xxxxxxxx wrote:

Update 2016-01-22: Read this post instead. The information in this thread is old!

Hello fellow developers. Today I want to talk about third-party modules in
a Python plugin. It is often necessary to use modules that are not delivered
with the CPython distribution in your plugin because of certain functionality
they provide. There are two common ways to import external libraries in your
plugin, and they both have their advantages and disadvantages.

What we, as developers, might like the most, is to just import the module and
don't care about the rest. For this, we have to put the module in a place where
the Python interpreter in Cinema can find it.

1. Insert the module in:

{Cinema Preferences Folder}/library/python/packages/{OS Name}

The Python interpreter embedded in Cinema will search for modules
there additionally to the standart paths. However, the module
has to be copied twice for Windows since there is a win32 and win64
folder and the Python distribution will search in the respective folder
depending on the Cinema build you run.

2. Modify the PYTHONPATH environment variable: We can add a path to this
environment variable where we can put all external modules we ever
want to use. The advantage of this method is that *any* Python
installation and not only the Cinema 4D installation will check the
paths defined in this environment variable. This way, we only need one
copy of the external module on our whole computer.

Either way, the user has to perform this step, too. This isn't actuall a great
deal and can be done quickly, but keep in mind that a normal user might not be
familar with folder-structures, environment variables and all the stuff. Not
kidding, this is not a rare case.
The user will also be responsible for updating the external modules when your
new version of your plugin requires a newer version, etc. All in all, we end
up giving support to the users of our plugins on how to correctly fullfil all
dependencies.

How can we make it better?

We can distribute the external dependencies with our plugin. The user will
just unpack the plugin archive and it will run. Sounds good, eh? But it's
tricky !! First of all, how do we deliver an external module with our plugin?

We will put it into a folder and import it from there, like:

lib\n    module.py
res\n    <resource files>
plugin.pyp

And the code to load the module:

import os
import sys
  
dirname = os.path.dirname(__file__)
lib_path = os.path.join(dirname, 'lib')
sys.path.insert(0, lib_path)
  
try:
    import module
finally:
    # Remove the path we've just inserted.
    sys.path.pop(0)

Notice the try/finally clause. Even when an error occures while loading one of the
modules within the try-block, the finally block garuantees that the path we have added
will be removed again!

So where's the tricky part?

Let's make this more fancy and say that we have two plugins relieng on the same module,
for the sake of example we will call this module "module". Therefor, we got wo plugins,
one dependency, but both distribute the module with them.

# plugin1.pyp
import module
  
# plugin2.pyp
import module

Yet, there is not problem. But it comes to a problem when the plugins rely on
different versions of the module. Eg. plugin2 relies on version 2 of the module while
plugin1 is already satisfied with version 1. Version 2 added new functions to the module.

# plugin1.pyp
import module
module.some_func()
  
# plugin2.pyp
import module
module.some_func()
module.new_func()

The call to module.new_func() can easily fail when plugin1 is loaded before
plugin2. After plugin1 imported the module in version 1, the module resides
in the sys.modules dictionary, and this is the place where the import
mechanism of Python looks in the first place before searching for the module
on the HD.

Once again: When plugin1 is loaded before plugin2, import module will load
"module.py" from plugin1 into plugin2 and not "module.py" from the plugin2!
The module loaded by plugin1 might even be a completely different one than
plugin2 but have the same name. In this case, one of the plugins is condenced
to fail!

How to fix?

You should always remove modules imported from your plugin-local folder from
sys.modules (not to mention to remove the path added to sys.path to actually
import the module). Best practice is to restore the old module configuration
after importing local libraries.

# Store the old module configuration.
old_modules = sys.modules.copy()
  
# Import your stuff, for example:
lib_path = os.path.join(os.path.dirname(__file__), 'lib')
sys.path.insert(0, lib_path)
try:
    import module
finally:
    sys.path.pop(0)
  
# Restore the previous module configuration making sure to not
# remove modules not loaded from the local libraries folder.
for k, v in sys.modules.items() :
    if k not in old_modules and hasattr(v, '__file__') or not v:
        if not v or v.__file__.startswith(lib_path) :
            sys.modules.pop(k)
    else:
        sys.modules[k] = v

Edit: I have hit strange errors when removing modules loaded by a module from the
local plugin distribution that are from the Python standard library or at least not
from the plugin's lib dir. With hasattr(v, '__file__') we're checking if the module
is a built-in module or not. If it isn't, we check if the filename of the module
starts with the lib_path of our plugin. If so, we can safely remove it!

But note that you should remove built-in modules when they are distributed with your
plugin (native modules referring to C/C++ modules with *.pyd suffix)! Imagine you are
delivering the _ssl module for Windows with your plugin, then do it like this:

# .. add lib path to sys.path and store old module configuration
  
import _ssl
del sys.modules['_ssl']
  
# ... remove modules like above!

The code above would be too complex to put it into a one-line expression, which is
why I striked through the following block:
~~If you're feeling nasty, you can also put the last five lines into a one-liner (just~~
~~split up for readability) :~~
~~
~~
~~[sys.modules.__setitem__(k, v) if k in old_modules else sys.modules.pop(k)~~
~~ for k, v in sys.modules.items()]~~

Conclusion : Do always ensure that you do not influence other plugins by
importing external modules.

edit: 2013/03/13
Reloading modules at runtime

Cinema 4D allows us to reload Python plugins at runtime so we do not have to restart the
application after making changes to a plugin. However, loaded modules not be reloaded
with this command. There's a reload() function built-in to Python so we can use it in
PluginMessage() and react on the C4DPL_RELOADPYTHONPLUGINS message.

def PluginMessage(msg, data) :
    if msg == c4d.C4DPL_RELOADPYTHONPLUGINS:
        reload(module)

But it's more tricky than that. The reload() function only works when the module can
be found in sys.modules (which we have removed it from) and if the module can be found
via sys.path (we removed the path the module is located at from this list). So, we
again have to find a work-around to this. For this workaround, we have to dive into
some black magic. We could reload a single module simply by

import sys
import imp
import types
  
def load_module(module=None, filename=None, name=None) :
    if module:
        filename = module.__file__
        name = module.__name__
    elif not all(filename, name) :
        raise TypeError('expected a module or both filename and name.')
  
    # Obtain the module that was previously stored.
    prev_mod = sys.modules.get(name, None)
  
    # Load the module.
    new_mod = imp.load_source(name, filename)
  
    # The new module has been inserted into sys.modules, we have to
    # fix that again.
    if prev_mod:
        sys.modules[name] = prev_mod
    else:
        sys.modules.pop(name)
  
    return new_mod
  
def reload_recursive(module) :
    mod = load_module(module)
    for name, submodule in vars(module).iteritems() :
        if isinstance(submodule, types.ModuleType) :
            new_mod = reload_recursive(submodule)
            setattr(module, name, new_mod)
    return mod

But then again, it comes to issues when the module imports modules from your other
dependencies, because the modules are not on sys.path. As hacky as it sounds, we
have to insert the required paths to sys.path again temporarily. So here's the solution:

import os
import sys
import imp
  
def get_root_module(modname, suffixes='pyc pyo py'.split()) :
    r"""
    Returns the root-file or folder of a module filename. The return-value
    is a tuple of ``(root_path, is_file)``.
    """
  
    dirname, basename = os.path.split(modname)
  
    # Check if the module-filename is part of a Python package.
    in_package =  False
    for sufx in suffixes:
        init_mod = os.path.join(dirname, '__init__.%s' % sufx)
        if os.path.exists(init_mod) :
            in_package = True
            break
  
    # Go on recursively if the module is in a package or return the
    # module path and if it is a file.
    if in_package:
        return get_root_module(dirname)
    else:
        return os.path.normpath(modname), os.path.isfile(modname)
  
def reload_modules(*modules) :
    r"""
    Reload the passed module objects. Restores the previous module
    configuration after reloading.
    """
  
    # Find all root-directories of the passed modules.
    paths = set()
    for mod in modules:
        mod_name = mod.__name__
        if imp.is_builtin(mod_name) != 0 or not hasattr(mod, '__file__') :
            raise RuntimeError('cannot reload built-in module %s.' % mod_name)
  
        mod_root, is_file = get_root_module(mod.__file__)
        dirname = os.path.dirname(mod_root)
        dirname = os.path.normpath(dirname)
        paths.add(dirname)
  
    # Change the system path and store a copy of the current module
    # configuration.
    old_path = sys.path
    old_mods = sys.modules.copy()
    sys.path = list(paths) + sys.path
  
    try:
        # Reload the modules.
        new_mods = []
        for mod in modules:
            if mod.__name__ in sys.modules:
                new_mod = reload(mod)
            else:
                new_mod = __import__(mod.__name__)
            new_mods.append(new_mod)
    finally:
        # Restore the import search path and module configuration.
        sys.path = old_path
        for k, v in sys.modules.items() :
            if k not in old_mods:
                sys.modules.pop(k)
  
    return new_mods
  
def PluginMessage(msg, data) :
    if msg == c4d.C4DPL_RELOADPYTHONPLUGINS:
        global module
        module, = reload_modules(module)

Welcome back after reading through this monster for such a "simple task". The
reload_modules() function will use the reload() function if the module is available
in sys.modules or use the standart import mechanism to import it if the reload()
function can not be used.

Holy crap, why should I stick to these rules?

Seriously, just do your fellow developers a favor. And also yourself. And the users of
your plguins. I've hit this problem just two days before and that is why I wanted to
share the solution with you. Yes, I've hit it myself, with two of my own plugins!

All of the things I've mentioned above to not fuzz other plugins by importing the
dependencies would not be a necessity if the users install the dependencies somewhere
all plugins can find them (and updates them accordingly to new requirements of other
plugins etc.). But as already mentioned, you will either end up giving very much user
support or loose users because of this "inconvenience" (it actually is just a 5 minute
task, if even).

I admit that I usually tell the users to install the dependency as described above for
free plugins. There are also cases were you are actually forced to deliver the
dependencies with your plugin. For example, both of my current clients explicitly stated
that they want the dependencies to be distribute with the plugins. Ouch, I've spend a lot
of time figuring the above out, but since they pay me to do what they want, I'll have
to do it.

If you have any questions regarding this topic, please don't hesitate to ask here.

All the best,
-Niklas

Helper

On 12/03/2013 at 08:08, xxxxxxxx wrote:

hey, good tutorial. something like this should be in in the documentation. as this is
targeted at beginners you might want to add a part about relaoding modules, so that
you can dabug your modules without having to restart c4d for each change.

Helper

On 12/03/2013 at 09:28, xxxxxxxx wrote:

Hi Ferdinand,

thanks for the feedback. Good idea, I'll add this asap.

-Niklas

Helper

On 13/03/2013 at 06:19, xxxxxxxx wrote:

I've added "Reloading modules at runtime" and a final word to the entry.

Best,
-Niklas

Helper

On 14/03/2013 at 07:18, xxxxxxxx wrote:

~~I have updated the article on built-in modules. I've hit strange errors when a built-in module~~
~~was removed from sys.modules.~~

I have hit strange problems when removing non-local modules (eg modules imported by the
local distributed modules from the Python STL) from sys.modules. Not removing those fixes the
problems and is also more resource efficient!

The post above is updated respectively.