Efficient way of saving array to BaseContainer

[BruceC]

Hi,
could you please help me to find out the efficient way of saving/reading an array with a large size of vectors to BaseContainer?

I could store/read each Vector in the array one by one as below, but when the size of the array is large, this method will cause performance issue.

save()
{
    maxon::BaseArray<Vector> arrayData;
    BaseContainer dataBC;
    Int dataCount = arrayData.GetCount();
    dataBC.SetInt32(XX_COUNT, dataCount);
    for (Int idx = 0; idx < dataCount; ++idx)
    {
        dataBC.SetVector(XX_ITEM + idx, positions[idx]);
    }
}
load()
{
    maxon::BaseArray<Vector> arrayData;
    BaseContainer dataBC;

    Int dataCount = dataBC->GetInt32(XX_COUNT, 0);
    arrayData.resize(dataCount);

    for (Int32 idx = 0; idx < dataCount; ++idx)
    {
        arrayData[idx] = dataBC->GetVector(XX_ITEM + idx, Vector(0, 0, 0));
    }
}

Then I remember vertex color tag has a special way of reading the color data stored. So I think C4D supports an efficient way of save/loading large array data.
I tried the below way by storing a raw memory chunk, but it doesn't work. It seems it only stores the memory address rather than the content of the memory address, and I suspect it will have serialize/deserialize problem when the scene file is opened in another platform (operating system).

save()
{
    maxon::BaseArray<Vector> arrayData;
    BaseContainer dataBC;

    Int dataCount = dataBC->GetInt32(XX_COUNT, 0);

    Vector* rawData = NewMem(Vector, arrayData.GetCount()) iferr_return;
    MemCopy((void*)rawData, (void*)arrayData.GetFirst(), Int(sizeof(Vector) * arrayData.GetCount()));
    GeData data(rawData, VOIDVALUE);

    dataBC.SetData(XX_ITEM, data);

    DeleteMem(rawData);
}

load()
{
    maxon::BaseArray<Vector> arrayData;
    Int32 dataCount = dataBC->GetInt32(XX_COUNT, 0);
    arrayData.resize(dataCount);

    const GeData& data = dataBC->GetData(XX_ITEM);
    Vector* rawData = (Vector *)data.GetVoid();
    for (Int32 idx = 0; idx < dataCount; ++idx)
    {
        arrayData[idx] = rawData[idx];
    }
}

So could you please help me to find out the right efficient way of saving/reading an array with a large size of vectors to BaseContainer?

Thank you very much!

[ferdinand]

Hey @BruceC,

Thank you for reaching out to us. I assume this is about your vertex color cache? VariableTag derived types such as VertexMapTag or VertexColorTag do not store their primary data, i.e., the "variables", in their data container. Instead, they overwrite NodeData::Read, ::Write, and ::CopyTo. We have many examples for that in the docs and code examples, I recently added this example which would be the most modern example which does that (although the case there is a bit different).

Here is for example what the vertex map does when it is asked to serialize itself:

void VertexMapTag::Write(HyperFile *hf) const
{
  SUPER::Write(hf);

  hf->WriteChunkStart(GetType(), VERTEXMAPTAGFILELEVEL);

  Int32 cnt = GetDataCount();
  const Float32 *padr = GetDataAddressR();
  hf->WriteInt32(cnt);
  if (cnt > 0)
  {
    hf->WriteArray(padr, HYPERFILEARRAY::SREAL, sizeof(Float32), cnt);
  }
  hf->WriteInt32(_samplingFields == TAG_VARIABLEFIELD ? TAG_EXPRESSION : 0);

  hf->WriteChunkEnd();
}

I.e., it uses HyperFile::WriteArray. However, BaseContainer and HyperFile are internally very closely related, and I do not think that their pure read/write performance is that different. Writing all your vectors one by one is of course not going to be very fast, but you can write other containers into a container (implying that you have put them before one by one into a container), you can write maxon data, and last but not least you can write raw memory. There is an example for your exact case of storing a BaseArray<Vector>.

What to do, depends a bit on your goals. Writing and reading data fast into/from a BaseContainer should not be a problem. The issue is that when you store heavy data in the data container of a node, it could end up being 'in the way' for the rest of the scene evaluation. I.e., you blow up the data container of a node to hundreds of megabytes; which usually is meant to be the size of a few kilobytes. Every time something is traversing the data container of that node or is trying to copying it, all that data would have to be carried around.

I am personally a big fan of writing things into the data container of a node, as it is usually the smarter option IMHO. But heavy data should be stored outside of the data container of a node. When you just want to use a BaseContainer on its own, without storing it in a node, you are fine. But I would then not understand why you are doing that.

Cheers,
Ferdinand

[BruceC]

Thank you @ferdinand for the help!

Write raw memory does help a lot, appreciate!