UDP Receive AioBuffer Size limited to 1024 bytes, how to increase it? [C4D SDK]

[felixc4d]

I am developing a plugin that connect to an external APP via TCP/UDP connection. I managed using the TCP/UDP interface provided in the C4D SDK [TCP ref] [UDP ref].

The TCP connection has worked fine but the UDP connection has encountered the following problem:

1. the received message buffer is limited to 1024 bytes while the entire message is over 5000 bytes (we cannot reduce the buffer size or split the buffer to chunks as it was set in the external APP)

2. I've dived deep into the sdk source files and cannot find where to change the AioBuffer size, I know that you can specify the receive buffer size with the setsockopt, but how do I pass in the mSocket parameter from NetworkUdpInterface?

	setsockopt(mSocket, SOL_SOCKET, SO_RCVBUF, (char*)&bufSize, sizeof(bufSize));

Here is my example code

maxon::Result<void> NetworkManager::CreateUDPConnection() {
	// create AioServiceRef
	iferr_scope;
	g_ioService = maxon::AioServiceRef::Create() iferr_return;
	g_ioService.Start() iferr_return;

	// create and start server
	auto localAddr = maxon::NetworkIpAddrPort(maxon::WILDCARD_IPV4_ADDRESS, mLocalPort);
	
// set the callback
	UdpRecvHandler = UdpRecvCompHandler::CreateByReference(this,
		static_cast<maxon::Result<void>(NetworkManager::*)(maxon::Result<void>, maxon::AioBuffer, maxon::NetworkIpAddrPort)>(&NetworkManager::UDPCallback));
	
	g_udpServer = maxon::NetworkUdpInterface::CreateUdpServer(localAddr, std::move(UdpRecvHandler), g_ioService) iferr_return;

	g_udpServer.Start() iferr_return;
	return maxon::OK;
}

maxon::Result<void> NetworkManager::UDPCallback(maxon::Result<void> result, maxon::AioBuffer receivedData, maxon::NetworkIpAddrPort sender)
{
	iferr_scope;
	const maxon::String message{ receivedData };
DiagnosticOutput("UDPCallback Message: msg Len @, MemSize @, CapCount @ |  @", message.GetLength(), receivedData.GetMemorySize(), receivedData.GetCapacityCount(),message);
// ...codes that handles the message
	return maxon::OK;
}

Here is the debug log shows that the message is only 1024 bytes

UDPCallback Message: msg Len 1024, MemSize 1048, CapCount 1024 |

Please help~ Any kind of hint is welcome~

[ferdinand]

Hey @felixc4d,

Thank you for reaching out to us. This UDP backend once has been written for our net.render service. I just had a quick look at the source code, and the culprit seems to be this:

Result<Bool> ReadToBuffer(SocketT sockfd, AioBuffer& bufOut, NetworkIpAddrPort& sourceAddrOut)
{
	iferr_scope;

	bufOut.Resize(1024) iferr_return; // The buffer is being forcibly resized.
	Block<Char> bufBlock(bufOut.GetFirst(), bufOut.GetCount());

	Int res = Socket::TryRecvFrom(sockfd, bufBlock, 0, sourceAddrOut) iferr_return;
  // ...
}

Where then TryRecvFrom then more or less is just calling recvfrom. When you look at Microsofts implementation of recvfrom, you can find this statement:

If the datagram or message is larger than the buffer specified, the buffer is filled with the first part of the datagram, and recvfrom generates the error WSAEMSGSIZE. For unreliable protocols (for example, UDP) the excess data is lost.

So, to summarize, we read with a fixed buffer length of 1024 bytes. And you also cannot just read multiple times in succession, as that data is being lost when you read with a buffer of insufficient length. I will update the UDP manual to reflect this limitation.

We cannot just change that value, because this might have unforeseen consequences for internal systems. We technically could make this a variable, definable in some constructor or so, but that would also have to be done and supported. Given the fringe nature of the subject, I am inclined to declare this an accepted limitation. You would either have to conform to communicating in chunks of 1kb, or simply include and use the native OS functions as we do internally.

Cheers,
Ferdinand

[felixc4d]

Hi @ferdinand,

Thank you for the detailed explanation. I will try to implement the UDP server using the basic SOCKET from winsock library.

While digging into the SDK, I found that the AioServiceRef do have a function called RegisterReader, but I could not manage to successfully use it and there is no example code that I can learn from.

SO I wonder:

1. Is it a good way to using RegisterReader function combined with the customized low level SOCKET? So that I can bypass the limitation of 1024 bytes?
2. If so, can you please provide some example code of how to use the RegisterReader ? Suppose that I have the following code:

// I am not sure if I should create a new Reader Class to override the NotifyForRead function
class AioReaderRefNew : public maxon::AioReaderRef
{
public:
    AioReaderRefNew() = default;

    maxon::Result<void> NotifyForRead(maxon::SocketT socket, maxon::Result<void> result) {
        iferr_scope;
        DiagnosticOutput("NotifyForRead");
        return maxon::OK;
    }
};

AioReaderRefNew udpServerReader;

mSocket  = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

g_ioService = maxon::AioServiceRef::Create() iferr_return;

if (g_ioService.RegisterReader(mSocket, udpServerReader) == maxon::OK);

g_ioService.Start() iferr_return;

Apparaently the above did not successfully start the aioservice, so may be you could fix it?

Thank you

[ferdinand]

Hey @felixc4d,

You inheriting from AioReaderRef does not make too much sense. The Maxon API uses interfaces and references. A reference just points to an interface object and increases its reference count, so that the object can be garbage collected once its ref-count reaches zero. References are almost never manually implemented but implemented automatically by the source processor. So, the thing you declare and inherit from is the interface. But that as its name implies, it is actually only the interface and usually is not the actual implementation.

The Maxon API mostly uses components to implement relevant functionalities. Components are elements that are loaded dynamically into interfaces at runtime and effectively realize composition over inheritance. So, you can have a FooRef a and a FooRef b for two FooInterface objects which act completely differently at runtime, because the interfaces have different components loaded which realize them.

I guess you want to override the NotifyForRead function of AioReaderInterface. For that you would have to write a component for AioReaderInterface and then load it at runtime into an object of one of its concrete forms such as NetworkUdpServerInterface. When I look at for example NetworkUdpServerImpl (the component for NetworkUdpServerInterface), you are sort of in luck, as that component has not been marked as final, so you at least technically have the ability to load another component in there without the interface refusing that.

But when you realize a component, you must realize all the MAXON_METHOD methods of the interface the component is for, so you cannot just overwrite that one method. There are way and techniques around this, but then we really reach internal/non-public territory.

I could here into more detail but that is all a bit pointless, because when you do this for NetworkUdpServerInterface and its client counter part, you will just end up again at ReadToBuffer shown above. Because what you have to customize in the end, is the call to the native OS library functions and these are burried deep within our non-public code, so you cannot change that from the outside.

I am afraid you must either use another protocol, you said that TCP worked fine for you, or really implement things yourself. I am also not really convinced that you cannot communicate in chunks of 1kb, as you can split up and reassemble things. Finally, the last option would be to just use a third party library which wraps at least Windows and macOS for you (not sure if you want to support Linux).

Cheers,
Ferdinand