BaseContainer::SetMemory() question

rsodre

Hi!

I have a custom data type that need to store some raw data blocks, at CustomDataTypeClass::WriteData(). Each memory block has an integer id/index and may be read in a different order as saved, so instead of using HyperFile::WriteMemory(), I'm using HyperFile::WriteContainer(), with BaseContainer::SetMemory() setting each id/block pair. The API reference for BaseContainer::SetMemory() says this about the memory pointer:

mem: The memory buffer. The container takes the ownership over the memory buffer.

I usually assumes that the BaseContainer creates it's own copy of the data it stores, and owns it. If we store an Int32, it will internally create a GeData with the value. Doesn't it apply to memory too? If I use SetMemory(), will it alloc a copy of the buffer internally in a GeData ank keep it until it's destroyed? That ownership statement got me worried, sounds like the BaseContainer just saves the pointer and takes ownership of the buffer I'm passing to it, instead of allocating new buffer to store. If that's the case, after storing the BaseContainer to the HyperFile, when it's destroyed it will free my original buffer, with bad consequences. Should I allocate a new buffer before storing? In that scenario, will it be stored in the project file?

Same question applies for GeData, it's not clear if GeData::SetMemory() will allocate a new block or just take ownership of the buffer.

r_gigante

Hi Roger thanks for reaching us and sorry for coming late back.
It took a bit of research to find the "way" things are supposed to work and this slowed the whole discussion a bit.

With regard to your request, I need to spend a few word on clarifying how BaseContainer::SetMemory() and GeData::SetMemory() work:

both methods expect only memory chunks being allocated on heap: attempting to hand-over those in the stack will simply cause a Cinema crash.
in BaseContainer::SetMemory() a temporary GeData is created on the fly and both the original pointer and size of the memory chunk are passed to instantiate the GeData. As this temporary GeData is copied in the BaseContainer the temporary GeData goes out of scope and gets destroyed with the side effect that also source memory chunk values are lost.
in GeData::SetMemory() a temporary ByteArray is created on the fly and the original pointer and size of the memory chunk are passed to instantiate the ByteArray. As this temporary ByteArray is used to fill the GeData the temporary ByteArray goes out of scope and gets destroyed with the side effect that also source memory chunk values are lost.

Considering this scenario, what's written in the documentation is, to a certain extent, misleading, because it's not actually taking the ownership, but rather it's copying the data of the memory chunk in theBaseContainer(or the GeData) and destroying the memory chunk used as source.

If this is not intended to happen, you can eventually consider, after using the SetMemory in the GeData or in the BaseContainer to use the corresponding GetMemory() (BaseContainer / GeData) and/or GetMemoryAndRelease()(BaseContainer / GeData) to access the new pointer being stored and use it for any further reason.

Last but not least, when using the GetMemoryAndRelease() beware of properly freeing the memory being referenced by the pointer returned otherwise you'll end up in memory leak issues.

Best, Riccardo

rsodre

Thanks @r_gigante .

So I think I'm doing it right. Passing my original pointer was resulting in a crash, but allocating a copy works fine. This is how I'm inserting a memory block into a BaseContainer that is saved to the HyperFile.

GeData data( DA_BYTEARRAY, DEFAULTVALUE );
if( bc->InsData( dataId, data ) == nullptr
   || bc->FindIndex( dataId ) == NOTOK )
{
	return false;
}

// alloc new memory block
auto bufferCopy = TNewMemClear<void*>( Int64( length ) );
CopyMem( buffer, bufferCopy, Int( length ) );
		
// fill memory block, BaseContainer takes ownership
bc->SetMemory( dataId, bufferCopy, Int( length ) );
if( bc->FindIndex( dataId ) == NOTOK  )
{
	return false;
}

For reading is simpler, I'm copying the contents from the BaseContainer:

Int bytesRead = 0;
auto dataPointer = bc->GetMemory( dataId, bytesRead, nullptr );
CopyMem( dataPointer, buffer, bytesRead );

a_block

Hi Roger,

what type is bufferand how is it allocated? I'm just wondering, if it may be possible to avoid this additional copy to bufferCopy.

Cheers,
Andreas

rsodre

I'm saving several data chunks, that my main plugin engine makes care of packing and unpacking. So my buffer is a void*, it can be anything.

If I don't make this copy to pass to SetMemory, C4D crashes. I think correctly, since the plugin engine owns it and will later deallocate.

a_block

Hello Roger,

my thought was, if you allocated buffer properly via NewMem (or one of its siblings), then you should be able to use it directly. But as Cinema 4D will do an internal copy and destroy the passed buffer, you'd need to do something like this (pseudocode):

SetMemory(buffer)
buffer = GetMemory() // to get the pointer to the buffer owned by Cinema

// ... and then in the end, if you need to free the memory yourself and can't rely on the memory being destroyed on destruction of the BaseContainer
buffer = GetMemoryAndRelease()
Free(buffer)

Just thinking and probably you have thought of this option anyway.

Cheers,
Andreas