if I am not overlooking something here, the function recurse_hierarchy is neither recursive nor will it retrieve the layers for all objects in the scene. It will retrieve the layers for all siblings of the passed node, that were born after that node.
You probably made a mistake copying your code or misunderstood what is meant by recursive.
To retrieve the active object you have to iteration trough the hierarchy. (even GetActiveObject)
In Python, GetActiveObjects is calling a c++ function so with lots of object it will be faster than iterating the hierarchy with python functions.
We don't know what the limit is. To mesure that you have to make a bunch of tests with different scenarios and get the mean.
Yes your code could be optimized but we agree with you here this is not really the point, a R20 code is expected to have the same performance with R21 and here this is not the case.
And @C4DS I agree using GeIsMainThread() == false I would say partially fix the issue (sometimes it works as R20, sometimes not) We keep investigating it.
The broken forward compatibility between R20 and R21 is reported on Changes in R21.
With regard to backward binary compatibility, it has never been possible to run a plugin built against a more recent API and load on a previous Cinema 4D executable as reported on Portability and Compatibility.