Fast conversion of numpy array to list of c4d.Vect

[Helper]

On 11/11/2015 at 05:50, xxxxxxxx wrote:

Hello all,
I am using numpy for array calculations on a c4d.pointobject. The manipulations within numpy are super fast, but it turns out that converting to and from c4d.Vectors is a real bottleneck. Since this is an essential step when using numpy, I was hoping that someone else has already found a good solution for it?

The fastest code I could come up with is attached below. Having a look at the execution times is interesting, because it shows that the internal getting and setting of vectors is much faster than the conversion to and from numpy of those vector objects.

c4d.GetAllPoints: 0.073354
list2np              : 0.455179
np manipulation: 0.067916
np2list              : 0.439967
c4d.SetAllPoints : 0.030023

Does anybody have suggestions as to further speed up this code? Especially the np2list function is critical, since it has to be called every frame (when used in a generator). I am open to exotic solutions such as cython, numba or whatever, as long as I can get it working on my machine.
Your help is really appreciated!
regards,

Hermen

code to be run from a script with a point object selected

import c4d
from c4d import Vector
import numpy as np
#import c4d2numpy as npc
import time

now = time.clock
Vec = Vector

y & z are reversed because of left-handed system in cinema 4d

def ipts(pts) :
    for p in pts:
        yield p.x
        yield p.z
        yield p.y
        
def tprint(tl) :
    t0 = [t[0] for t in tl]
    s0 = [t[1] for t in tl]
    print ' '
    for i,t in enumerate(t0) :
        if i==0: continue
        print s0 _, t-t0[i-1]

def list2np(lst) :
    A = np.fromiter(ipts(lst), dtype=np.float, count=3*len(lst))
    return A.reshape((len(lst), 3))

def np2list(A) :
    return map(Vec, A[:,0], A[:,2], A[:,1])

def main() :
    op.ResizeObject(1000000)
    t = []
    t.append( (now(), 'start' ) )
    pts = op.GetAllPoints()
    t.append( (now(), 'c4d.GetAllPoints:' ) )
    A = list2np(pts)
    t.append( (now(), 'list2np               :' ) )
    #print A.shape
    B = A + 10. * np.random.random((A.shape))
    t.append( (now(), 'np manipulation:' ) )
    pts = np2list(B)
    #print len(pts)
    t.append( (now(), 'np2list               :' ) )
    op.SetAllPoints(pts)
    op.Message(c4d.MSG_UPDATE)
    t.append( (now(), 'c4d.SetAllPoints:' ) )
    op.SetAllPoints(pts)
    op.Message(c4d.MSG_UPDATE)
    tprint(t)

if __name__=='__main__':
    main()

[Helper]

On 11/11/2015 at 14:09, xxxxxxxx wrote:

mmm, quiet here…

After some further investigation I found the numpy function "np.frompyfunc" and it does what it says pretty fast. For those of you interested, see an example of the code below. It executed in about half the time as the previous np2list, in 0.288461 sec.

Meanwhile, I've played around with multiprocessing a bit, but the overhead for a function with return value seems just too big, it only takes longer on more processes. So I am fairly happy with his latest result, it's still about three times faster than an ordinary for loop.

But if anybody has a better suggestion, I would love to hear it!
regards,
Hermen

code to replace np2list

vec_array = np.frompyfunc(Vector, 3, 1)

def np2list_2(A) :
    """
    y & z are reversed because of left-handed
    system in cinema 4d
    """
    pts = (vec_array(A[:,0], A[:,2], A[:,1])).tolist()
    return pts

[Helper]

On 12/11/2015 at 02:08, xxxxxxxx wrote:

Hello
not exacly for conversation (as name of topic), but i made several tests by wrapping into pyCuda or cythonize script-body(with numpy code too). And found it was faster. I test with pyVoro(python extesion for Voro++) and pyOpenVDB.

[Helper]

On 12/11/2015 at 04:56, xxxxxxxx wrote:

Hello Ilya,
What do you mean by 'not exactly for conversion'? It is meant to be working from within c4d. 
PyVoro, PyCuda, PyOpenVDB sound a little too exciting for me. As far as I can see, they require building extra packages, and I am on mac osx, so it'd be different than on windows.

Cython might be an option though, but I am having trouble typing stuff right. For example, do you know how to cdef/ctypedef the Vector object from c4d? Without it, I only got a marginal increase in performance. How much (%) did you get, btw?
regards,

[Helper]

On 12/11/2015 at 06:18, xxxxxxxx wrote:

several colleagues and i had tasks for scripting environment to speed up conversation of mesh/procedural data to numpy arrays and then to pyVoro/OpenVDB. Yes, i compiled these py-extesions. also a few processing was boosted in cuda/gpu compute environment or slightly modified cython.

I forgot to point at such moment - Hermen, try to create topic somewhere else, forum, social network in your region, ... from my experince and search over support forum - teamers do not support 3d  party's plugins and extensions.
For example french4d has topic about speed algos in python
i use network between several math/programming russian universities

[Helper]

On 12/11/2015 at 15:16, xxxxxxxx wrote:

OK, I've got cython running as well, but the performance increase is very marginal compared to the function I posted previously. That's good enough for me, I guess, and if I need more speed, I'll check elsewhere. Thanks for your suggestions!