SubContainers and Symbol ranges

ECHekman

I am looking into mapping symbols from my plugin application to symbols in c4d.
This is for ShaderData/TagData/ObjectData derived plugins.

However im running into two issues. Some of these symbols clash with existing symbols in c4d.
1 I have read that the first 1000 are reserved. And that I shouldnt go above 1 million?
2. My other related issue is that I would like to reuse symbols in the same NodeData plugin.

My solution idea was to use sub-containers to avoid both of these symbol clashes.
However I cannot find in the sdk if the same symbol restrictions apply for subcontainers.
Can I just put in any ID in a subcontainer? Or are there limitations for those aswel?

Additionally, do links work the same as with normal BaseContainers?

ferdinand

Hey @ECHekman,

Thank you for reaching out to us. As you know, you should avoid topics with multiple questions, as they have a tendency to get out of hand.

However im running into two issues. Some of these symbols clash with existing symbols in c4d.

Yes, many symbols in Cinema 4D collide, dialog message symbols (BFM_SOMETHING) collide for example with atom message (MSG_SOMETHING) which again collide with core and event messages (EVMSG_SOMETHING). The symbols for values inside the data containers of scene elements also collide with each other. This is for once because the address space of Int32 might not be big enough to address everything Cinema 4D has to address, and the curation of that would be very cumbersome or even impossible in the case of third parties.

The larger problem than value collision (both ID_FOO and ID_BAR resolve to 1000) is actual symbol collision (you have the plugins A and B which both define an ID_FOO which then resolves to different numeric values). In C++ this is offset by the fact that you can selectively include header files. In Python you cannot do this and all symbol labels must fit here collision free into the c4d package namespace. One should therefore always prefix node symbols with the node they are for. E.g., for an ObjectData plugin omyobject.res, the symbols should look like this: ID_MYOBJECT_FOO, ID_MYOBJECT_BAR and not ID_FOO or ID_BAR.

I have read that the first 1000 are reserved. And that I shouldn't go above 1 million?

That is true, the parameter slots 0 to 1000 are reserved for the Basic tab each node has (with different content based on the node type) and some internal things. The name of a node is for example stored under ID_BASELIST_NAME which is an alias for 900.

I know that there is this information floating around that one should not use high numbers like a million. For stuff like plugin IDs (which you should not invent in the first place) it is true that we use the end of the Int32 address space for special purposes, so you REALLY should not invent a plugin ID that is 2^32 - 5. But that would not be millions but billions. It could be that we in the past did occupy slots at the end of the Int32 space for scene elements. But I am currently not aware of any cases.

But for readability and conformance reasons I would recommend to stick to the Cinema 4D pattern of staying in the low hundreds of thousands. Due to how BaseContainer works, there could also be a performance argument for small key values, but I am (a) not 100% sure and (b) it can probably be ignored on modern hardware.

What should not be ignored is that BaseContainer is neither a hash map not a fixed size array but a list of key value pairs at its core. Storing 100, 1000, or 10 000 values is fine, and you can also store 100 000 values if you really have to. But you should be aware that access times are not constant (at the advantage that for the common case of small container sizes of ~1000 items, this will outperform hashing).

My other related issue is that I would like to reuse symbols in the same NodeData plugin.

You can reuse symbols between nodes as much as you want. What I would personally consider formally not so nice, is when you would just define a symbol in the resource of a node A and then also use it to store things in the data container of node B. But it is technically totally fine, I would just consider it poor code hygiene.

You can always just define a symbol in the resource header file of your plugin. I.e., not every smybol in omyobject.h must appear in omyobject.res and omyobject.str. When you want to define a 'global' symbol, you should do that in your c4d_symbols.h.
The built-in mechanism for symbol sharing are includes, e.g., INCLUDE Obase; is the line which includes the file obase.res and its symbols into your object plugin and makes sure that it has the common UI elements and parameters all object nodes have. You can define your own base descriptions and include them. I would invite you to have a look at {C4D_FOLDER}/resource.zip/modules/c4d_base/description to check out how Cinema 4D composes complex descriptions. All node types have a base, e.g., Obase, Tbase, Xbase, etc., which are all based on the atomic description Obaselist. But there is more modularization, things like spline interpolations, field handling, and more are all encapsulated in reusable description elements.

You can also build dynamically descriptions or allocate data. In a node, we then usually define a base symbol ID_MYOBJECT_ITEMS_BASE = 50000 at a safe offset tp the rest of the data (so that we have room for adding other static parameters in the future), a stride ID_MYOBJECT_ITEMS_STRIDE = 100 (it does not matter that we define here the stride as 100 as we are not going to write with it into the data container at that value) and optionally a few named offsets (ID_MYOBJECT_ITEM_NAME = 0, _COLOR = 2, _LAYER = 3, ..) and then write with these values at runtime into the data container of the node; e.g.,

const Int32 i = ID_MYOBJECT_ITEMS_BASE  + ID_MYOBJECT_ITEMS_STRIDE * layer;
bc.SetString(i + ID_MYOBJECT_ITEM_NAME, "MyName"_s);
bc.SetVector(i + ID_MYOBJECT_ITEM_COLOR, myColor);
...

My solution idea was to use sub-containers to avoid both of these symbol clashes. However I cannot find in the sdk if the same symbol restrictions apply for subcontainers.

You can do whatever you want to in sub-containers. But I would avoid overly large containers (holding millions of items) for performance reasons. Only the data container of a scene element (BaseList2D::GetDataInstance) comes with the restriction that you should not write into the 0 to 1000 range, as there it is where scene elements store their atomic data. A BaseContainer itself is free of restrictions.

Additionally, do links work the same as with normal BaseContainers?

Are you talking about BaseLink fields of a container? And what is here the implicitly referred counter part of 'normal' containers? A sub container? Links are stored document agnostic, i.e., you have to pass a document within which you want to resolve the link. It does not matter if you store that link in the top level of a nodes data container, in a sub-container of that data container, a bare BaseContainer instance in memory, or a hyper file on disk. They all will be able to resolve their link when passed the correct document which contains a node with the matching marker stored in the link.

Cheers,
Ferdinand

ECHekman

Thank you for the detailed answer.
I think I have settled on how to store data then that makes it the most future proof for us.
A quick mockup of our setup would be something like this then:

// OurShaderNode.h
enum OurShaderNode
{
    OUR_NODE_TYPE = 1001,
    OUR_NODE_PINS,
    OUR_NODE_PIN_LINKS,
    OUR_NODE_ATTRIBUTES,


    OUR_NODE_OFFSET = 10000,
// Pin Ids. These map to IDs from our application
    OUR_NODE_PIN_SCALE = OUR_NODE_OFFSET + 1; // Maps to our internal ID PIN_SCALE which is 1, same for the others
    OUR_NODE_PIN_ALBEDO = OUR_NODE_OFFSET + 2;
    OUR_NODE_PIN_SPECULAR = OUR_NODE_OFFSET + 3;
    OUR_NODE_PIN_NORMAL = OUR_NODE_OFFSET + 4;
    OUR_NODE_PIN_FRAME = OUR_NODE_OFFSET + 245;

// Attribute. These map to IDs from our application
    OUR_NODE_ATTRIBUTE_VALUE  = OUR_NODE_OFFSET + 1;
    OUR_NODE_ATTRIBUTE_COUNT  = OUR_NODE_OFFSET + 2;
    OUR_NODE_ATTRIBUTE_SCALE  = OUR_NODE_OFFSET + 3;
    OUR_NODE_ATTRIBUTE_FRAME  = OUR_NODE_OFFSET + 4;
    OUR_NODE_ATTRIBUTE_FILENAME  = OUR_NODE_OFFSET + 3456;


}

// pluginOurShaderNode.cpp
// These two functions will map c4d IDs to our internal Ids
int ConvertToOurID(OurShaderNode c4dId) { return c4dId - OUR_NODE_OFFSET; }
int ConvertToC4D(int ourId) { return ourId + OUR_NODE_OFFSET;  }

// pluginOurShaderNode is derived from the base class ShaderData
Bool pluginOurShaderNode::Init(GeListNode* node, Bool isCloneInit)
{
    BaseContainer *data = ((BaseShader*)node)->GetDataInstance();

    // The subcontainers use overlapping numbers for their IDs, but this setup gives us the biggest range of id numbers
    BaseContainer PinContainer; // Contains all pin data
    AttributeContainer.SetVector(OUR_NODE_PIN_ALBEDO, Vector(1,0,0));
    BaseContainer PinLinkContainer; // Containes links to other shadernodes. 
    AttributeContainer.SetLink(OUR_NODE_PIN_ALBEDO, nullptr);

    BaseContainer AttributeContainer; // Contains all attribute data
    AttributeContainer.SetInt32(OUR_NODE_ATTRIBUTE_VALUE);

    data->SetContainer(OUR_NODE_PINS, PinContainer);
    data->SetContainer(OUR_NODE_PIN_LINKS, PinLinkContainer); 
    data->SetContainer(OUR_NODE_ATTRIBUTES, AttributeContainer);
}

ECHekman

Ofcourse in the code example i mean to set the pin data and links in their correct subcontainer, and not in the AttributeContainer

ferdinand

Hey @ECHekman,

I assume you are looking here for some kind of approval from my side? That looks fine, but it is hard to give a concrete assessment with mock code. But as lined out above, there is little which can go wrong here. Splitting up your data into sub-containers could help with access times when you plan to add lots of 'attributes' and 'pins'. So, it is good that you do that. And shifting your application IDs into lower ranges for Cinema 4D is probably not technically necessary but will lead to more 'natural' IDs. I cannot really see much going wrong here.

The only thing which is not so nice, is the unsafe C-style cast in BaseContainer *data = ((BaseShader*)node)->GetDataInstance();. We recommend using static_cast for safe casts and reinterpret_cast for casts which are explicitly meant to be unsafe.

Cheers,
Ferdinand