Distributing Python Plugins that have Dependencies

dskeithbuck

I'm working on a python plugin that will rely on HTTP communication in order to ingest/export data. My research has led me to the requests python library. Unfortunately, it's not part of the standard Python library so I'll need to import it.

I've found my way to Niklas Rosenstein's localimport module and the minified version I can drop at the top of a *.pyp file in order to import a library without polluting the global sys.modules.

This works great if I'm importing 1 or 2 simple python modules, however requests has a number of dependencies on other python modules, each of which is dependent on yet another set of python modules. This would be easy enough if I was the only person that needed this plugin. I'd just use pip to automatically download and install of the dependencies in a requirements.txt. However, I can't easily do that with C4D's embedded python installation, and I'm certain I can't trust most end users to do the same.

I assume not, but is there a simple way to setup something like a virtualenv with C4D's python installation that automatically downloads and installs all modules my plugin needs without polluting the global sys.modules? I've tried installing all of the dependencies on my machine's python installation and then copying them into ./res/modules and then using localimport but I run into a recursion depth error.

Any suggestions or workarounds would be greatly appreciated.
Thanks,

Donovan

ferdinand

Hi,

the requests dependency chain doesn't seem to bad:

requests==2.22.0
  - certifi [required: >=2017.4.17, installed: 2019.6.16]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.9, installed: 2.8]
  - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]

Apart from shenanigans like using the pip of Cinema's Python installation (it is there, it is just not deployed) I would just go with the dedicated Python libraries folders that c4d does provide (and pip would ignore) and install the modules/packages there manually via a little script for the user.

Cheers
zipit

m_adam

Hi @dskeithbuck, unfortunately, we have nothing to offer there. I know some user have an installer that runs some pre-build script. But it's still a bit tricky as c4dpy need to be launched at least once before to be used in order to login from MyMaxon.
Another solution may be to directly install into the python located in the resource/modules/python/libs/... folder.
So this way it's also possible to create a CommandData that will download and install pip and all others needed module not sure if it's a good solution for you but at least it should work. Note that once pip is installed you can use pip at runtime on your script How to install and import python modules at runtime.

In any case please feel free to give us some feedback or any idea on this topic, or if you have an idea of how it could work, please let us know.

Cheers,
Maxime.

ferdinand

@m_adam said in Distributing Python Plugins that have Dependencies:

Note that once pip is installed you can use pip at runtime on your script How to install and import python modules at runtime.

Hi,

I might be misunderstanding something here, but I still want to point out that espically pip2 is not thread safe and it is explicitly disencouraged to use pip as a module.

Cheers
zipit

m_adam

Thanks for pointing to theses limitations I was not aware of.

However, if you execute pip into a running Cinema 4D as CommandData in the Execute function it shouldn't be an issue:

The pip code assumes that is in sole control of the global state of the program. It's the case in theCommandData::Execute you are in charge of the global state of the program.
Pip’s code is not thread safe. You are in the Main Thread.
Pip assumes that once it has finished its work, the process will terminate. A Python script in Cinema 4D can't close Cinema 4D or the python interpreter.

So while I never tested, looking at the limitation it should work without any issue. As you can have you plugin checking if import can be done, if not simply leave. Then have another CommandData plugin to install your stuffs. Restart Cinema 4D and it should work.

Cheers,
Maxime.

dskeithbuck

@zipit Request's dependency chain isn't too bad, but a couple of those libraries also have dependencies which is where it starts to get a little ugly. I'll give it another go.

dskeithbuck

@m_adam Thanks, I'll have a look at pip and/or an installer script. I'm a bit concerned about adding anything directly to the resource/modules/python/libs folder as there's a chance some other plugin or script will have installed/need a different version of the same library. But, I suppose that I can report this as an installation error to the user and let them sort out which is more important for them to have installed.

I'll be sure to let you know if I come to a resolution.

m_adam

While I know is not the perfect solution, and is a really quick and dirty I would say find bellow a script to setup a build with some python module.

Just call c4dpy file.py.

import subprocess
import urllib
import sys
import c4d
import os
import ssl
import shutil
import importlib
import maxon

currentDir = os.path.dirname(__file__)
c4dpyPath = sys.executable
c4dTempFolder = c4d.storage.GeGetC4DPath(c4d.C4D_PATH_STARTUPWRITE)

c4dDir = maxon.Application.GetUrl(maxon.APPLICATION_URLTYPE.STARTUP_DIR).GetPath()
if os.name != "nt" and not c4dDir.startswith(os.sep):
    c4dDir = os.sep + c4dDir

try:
    import pip
    print "pip already installed"
except ImportError:
    print('start downloading get-pip.py')

    url = 'https://bootstrap.pypa.io/get-pip.py'
    pipPath = os.path.join(c4dTempFolder, "get-pip.py")

    # Quick hack for MAC until https://developers.maxon.net/forum/topic/11370/urllib2-urlopen-fails-on-c4d-for-mac/8 is fixed
    f = os.path.join(c4dDir, "resource", "ssl", "cacert.pem")
    context = ssl.create_default_context(cafile=f)

    # Downloads pip
    urllib.urlretrieve(url, pipPath, context=context)

    print('start installing pip')

    if os.name == "nt":
        subprocess.call([c4dpyPath, pipPath, "--no-warn-script-location"], shell=True)
    else:
        os.system("{0} {1} {2}".format(c4dpyPath, pipPath, "--no-warn-script-location"))

    shutil.rmtree(pipPath, ignore_errors=True)

def installModule(moduleName):
    try:
        importlib.import_module(moduleName)
        print "{0} already installed".format(moduleName)
    except ImportError:

        print('start installing {0}'.format(moduleName))

        if os.name == "nt":
            subprocess.call([c4dpyPath, "-m", "pip", "install", moduleName, "--no-warn-script-location"], shell=True)
        else:
            os.system("{0} {1} {2} {3}".format(c4dpyPath, "-m pip install", moduleName,"--no-warn-script-location"))

        print('{0} installation is done'.format(moduleName))

installModule("numpy")

Cheers,
Maxime

dskeithbuck

@m_adam said in Distributing Python Plugins that have Dependencies:

Just call c4dpy file.py.

I'm not having much luck running or calling c4dpy in R21.

Microsoft Windows [Version 10.0.17763.737]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\donovan>cd "C:\Program Files\Maxon Cinema 4D R21"

C:\Program Files\Maxon Cinema 4D R21>c4dpy
Error running authentication: invalid http response 400.  (https://id.maxon.net/oauth2/access?scope=openid+profile+email&grant_type=refresh_token&refresh_token=[[REDACTED]]&client_secret=[[REDACTED]]) [http_file.cpp(313)]
----
Enter Maxon Account Settings (ENTER to keep input):
License Check error: invalid http response 400.  (https://id.maxon.net/oauth2/access?scope=openid+profile+email&grant_type=refresh_token&refresh_token=[[REDACTED]]&&client_id=[[REDACTED]]&client_secret=[[REDACTED]]) [http_file.cpp(313)]
Error: Invalid License

I'm able to open Cinema 4D and don't have any licensing issues there. I also did a complete uninstall/reinstall of C4D and had the same issues.

Thank you for writing that script, it seems like it will do the trick if I can get c4dpy to run.

m_adam

Try to open Cinema 4D.
Go to the preference opens the preference folder.
Go one level before. There is normally another cinéma 4d pref folder called like the previous one with the prefix _p (or _c Im on my phone writting by memory i will confirme tomorrow). Delete thid folder. Try again.

Cheers,
Maxime.

dskeithbuck

@m_adam It's a _p suffix. Deleting it eliminated the error messages and allowed me to log in. Thank you!

Is there any chance that c4dpy.exe can get its license from the C4D installation without having to manually enter it in the command prompt? It's not too much of an issue for me, but I imagine seeing the terminal will be a bit intimidating for end users.

dskeithbuck

@m_adam said in Distributing Python Plugins that have Dependencies:

While I know is not the perfect solution, and is a really quick and dirty I would say find bellow a script to setup a build with some python module.

Hi Maxime,

Your script got me most of the way there, but I ran into some issues.

It installs libraries to the default location for system python libraries rather than C4D's lib folder. So, I started modifying it to install to userprefs/python27/lib using the --target flag. But I ran into issues with spaces in the pathname. Which led me to switch to calling with a list of commands/parameters rather than a string.

After some tweaking, I arrived at this:

"""Install C4D Python Modules
Installs python modules in C4D's embedded python installation

## Authors

Maxime of Maxon Computer
Donovan Keith of Buck Design

Reference:
https://developers.maxon.net/forum/topic/11775/distributing-python-plugins-that-have-dependencies/8

## Requirements

Cinema 4D R21+

## Usage Instructions

1. Copy this file onto your desktop.
2. Open `terminal` (Mac OS) or `cmd` (Windows) and navigate to your Cinema 4D installation.
3. `$ c4dpy`
    1. This will run a Cinema 4D specific version of python.
    2. If prompted to login, do so.
    3. If you get an HTTP/authentication error:
        1. Open Cinema 4D.
        2. Edit > Preferences and click on "Open Preferences Folder"
        3. Go up one level in finder/explorer.
        4. Look for a folder with the same name and a `_p` suffix.
        5. Delete this folder.
        6. Try running `c4dpy` again from the console.
4. Exit the interactive python session by typing `exit()`
5. Run this script.  
`>>> c4dpy "/path/to/this/script/install_c4dpy_modules.py"`

## Known Limitations

1. Hasn't been tested on Mac OS
2. Is checking the `c4dpy` installation for whether a python module has been installed, but is installing in the `c4d` installation.

"""

print "C4D Py Requirements Installer"

import subprocess
import urllib
import sys
import c4d
import os
import ssl
import shutil
import importlib
import maxon

currentDir = os.path.dirname(__file__)
c4dpyPath = sys.executable
c4dpyTempFolder = c4d.storage.GeGetC4DPath(c4d.C4D_PATH_STARTUPWRITE)

# Because we're using `c4dpy`, it will pull a different prefs folder from the user's c4d installation.
# `Maxon Cinema 4D R21_64C2B3BD_p` becomes `Maxon Cinema 4D R21_64C2B3BD`
c4dTempFolder = c4dpyTempFolder[:-2] if c4dpyTempFolder.endswith("_p") else c4dpyTempFolder

c4dDir = maxon.Application.GetUrl(maxon.APPLICATION_URLTYPE.STARTUP_DIR).GetPath()
if os.name != "nt" and not c4dDir.startswith(os.sep):
    c4dDir = os.sep + c4dDir

# Just using the `--user` flag with `pip` will install to the System python dir, rather than the
# C4D embedded python's dir. So we specify exactly where we want to install.
c4dUserPythonLibPath = os.path.join(c4dTempFolder, "python27", "libs")

try:
    import pip
    print "pip already installed"
except ImportError:
    print('start downloading get-pip.py')

    url = 'https://bootstrap.pypa.io/get-pip.py'
    pipPath = os.path.join(c4dTempFolder, "get-pip.py")

    # Quick hack for MAC until https://developers.maxon.net/forum/topic/11370/urllib2-urlopen-fails-on-c4d-for-mac/8 is fixed
    f = os.path.join(c4dDir, "resource", "ssl", "cacert.pem")
    context = ssl.create_default_context(cafile=f)

    # Downloads pip
    urllib.urlretrieve(url, pipPath, context=context)

    print('start installing pip')

    if os.name == "nt":
        subprocess.call([c4dpyPath, pipPath, "--no-warn-script-location", "--user"], shell=True)
    else:
        os.system("{0} {1} {2} {3}".format(c4dpyPath, pipPath, "--no-warn-script-location", "--user"))

    shutil.rmtree(pipPath, ignore_errors=True)

def installModule(moduleName):
    try:
        importlib.import_module(moduleName)
        print "{0} already installed".format(moduleName)
    except ImportError:

        try:
            print('start installing {0}'.format(moduleName))

            # We build up the command as list rather than a string so that Windows will properly handle paths that include spaces
            cmd = [c4dpyPath, '-m', 'pip', 'install', moduleName, '--target', c4dUserPythonLibPath]
            subprocess.call(cmd)
        except subprocess.CalledProcessError as e:
            print e.output
        else:
            print('{0} installation is done'.format(moduleName))

# Add any modules you want installed to this list.
required_modules = ["requests"]

for module in required_modules:
    installModule(module)

m_adam

Hi, @dskeithbuck thanks for sharing yours though. And I'm glad if it works for you. But I still understand that an interpreter can be quiet intimidating for the user. And if you found a nicer solution, don't hesitate to share.

As an Idea but didn't try and will not have the time to do it today (so if you try, do it at your own risks), but you could try to install things using directly the python executable from the resource folder (as previously when c4dpy didn't exist). Keep in mind this python is a standard python so that means it does not come with any c4d related stuff (so no Maxon/c4d module). This way you don't need the user to log in and can execute the whole thing without the need to pop up the console, and maybe a check if Cinema 4D is running is safer to avoid any issue. You can retrieve a tempfile using the tempfile module in python which comes by default with python.

With that's said regarding the previous error 400, did you already used this particular c4dpy previously, or it was the first time? Did you already "setupped" this particular c4dpy in any IDE? Any more information to reproduce is welcome.

Cheers,
Maxime.

dskeithbuck

@m_adam said in Distributing Python Plugins that have Dependencies:

As an Idea but didn't try and will not have the time to do it today (so if you try, do it at your own risks), but you could try to install things using directly the python executable from the resource folder

I'm moving onto different parts of development, but I'll likely want to investigate this later and will post an update if I do. Thanks!