Why are GUIDs not globally unique?

[Helper]

On 18/08/2017 at 05:30, xxxxxxxx wrote:

Wow, thanks Riccardo! Please pin this topic somewhere.

[Helper]

On 18/08/2017 at 06:23, xxxxxxxx wrote:

Actually the plan is to add the info to the SDK docs.

[Helper]

On 18/08/2017 at 06:49, xxxxxxxx wrote:

Hi Riccardo,

thank you for your extensive research.

Unfortunately, your findings confirm my own research:

• GUIDs are not unique as duplicated of the same GUID can appear in caches

• GeMarker change when the generator updates objects. Thus loosing tracking between different render calls

Objects that are repesented in the scene work fine. Unfortunately, a typical Cinema 4D scene's geometry that needs to be rendered is mostly found in caches and deform caches. Generators and deformers cannot be rendered directly.

The generated geometry cannot be reliably tracked/identified. Identical GUIDs are used for different objects in caches. GeMarker change when the geometry is changed.

Generators should make use of unique IPs so that objects can be reliably tracked across different frames. Unfortunately, for example the Subdivsion Object does not generate IPs for its generated objects in the Cache. Putting a hierarchy of objects into a Subdivision object has two results: all objects are marked as control object. Control objects themselves indicate that they are not to be used directly but that they generate the data that should be used. Typically, this data is found in the cache of the object itself. The Subdivision Object exhibits different behaviour. All the generated geometry is put in it's own cache, regardless where the control object for that geometry. This way, there is no reliable way to deduct the control object that originally was responsible for creating the data.

The problem I have is not getting to the actual geometry to render but to keep track of which object generated what and to only update the changes. With rendering times of a few milliseconds it ridiculous to spend dozens of secoonds or even a few minutes to prepare the frame.

In conclusion: Ultimately, GUIDs and GeMarker can be used to differentiate between objects. Identification is not possible though. Also, UniqueIPs - a proposed method to track/identify genearated objects during animation - is not consistenly used by generators.

• The same GUID is used for different objects in caches. Visible geometry for many scenes can be found exclusively in caches.

• GeMarker changes when the geometry for generated objects change, making it impossible to identify the object between two consecutive renderings.

• Cloning documents changes GeMarkers. Cloning is happening frequently when rendering.

• The proposed mmethod to track generated object by using their UniqueIP fails as generators like the Subdivision Object does not assign UniqueIPs

BTW: I need to identify objects not only between consecutive renderings but I also track changes "LIVE" to immediately update the realtime rendering. I am not using VolumeData in the VideoPost Plugin for this very reason.

I am using Hierarchy class derived classes to traverse the scene and also handle instances myself.

Here is a scene that is showing some of the duplicate GUID problems and also shows how Subdivision Object is handling caches and UniqueIPs differently: https://drive.google.com/file/d/0BwFHFNK-2yFAMGpIVkRodDlxbzg/view?usp=sharing

Image of resulting test-rendering here: https://drive.google.com/file/d/0BwFHFNK-2yFAcjFlaGJ1NG5wb28/view?usp=sharing

Edit: Here is my log of the scene: https://drive.google.com/file/d/0BwFHFNK-2yFAUVhtU2ZOT21LV1E/view?usp=sharing
For IP I print the cache parent ID and the IP. ID in general is the assigned ID I created. (cached) has a CacheParent is not part of the scene the user sees in the object manager.

I am having troubles with identification in this scene mostly due to the nesting of instances and putting everything into a Subdivision Object. One Holder_01 "instance of an instance of an instance" and a couple of Holder_02 objects are eluding my efforts to identify them.

Thanks,
Martin

BTW: I am out of the office next week. So I will not be able to respond in a timely fashion.

[Helper]

On 18/08/2017 at 14:06, xxxxxxxx wrote:

Martin, attach few expression tags and you will get always changing addresses/pointers of objects. Same object remains untouched, but you make a mouse click and it changes and it will report to dirty flags, and most likely to ID assigning staff that you must remove/add things (not just update them slightly). Welcome to Cinema 4D >:E :))
Let's ask Maxon to solve this ID issue and update the API for recent couple of major Cinema versions.

[Helper]

On 18/08/2017 at 15:56, xxxxxxxx wrote:

i also would like is to be able to identify objects by using an unique object id.
as stated here BaseObject.GetGUID() is not unique (?)
and even if it is, there is no method to get the BaseObject if you just have the GUID.
(is there one?)

btw it would be nice if it wouldnt be just an in-memory id,
but would stay the same when quitting the app and reopening the file.

ps if you str() an object you get
<c4d.BaseObject object called 'Null/Null' with ID 5140 at 0x1412bd6d0>
is this hex address usable for indentification?

[Helper]

On 28/08/2017 at 01:31, xxxxxxxx wrote:

I have found that the best way to identify an object that is not in a cache is to use AddUniqueID(). This should be serialized to the file as well. It has been working fine for me.

If you don't care about changed IDs with cloned scenes or re-created objects, you can always use GeMarker().

My problem is that I actually have scenes where ALL visible geometry is exclusively coming from object caches. This is where my approaches start to fall appart. Most of the time I can rely on FindUniqueID() of the CacheParent() and the UniqueIP() to reconstruct a unique identification. Unfortunately, this breaks with ojects like the subdivision surface object who do not assign uniqueIPs and also do not make the generator source objects identifiable.

@zeorge: having the GUID, traversing the document to get the BaseObject should not be that time consuming. Even with several thousand objects.

[Helper]

On 28/08/2017 at 03:56, xxxxxxxx wrote:

If I may drop some insight here on why GUIDs might not always be unique: It's likely due to the use of crc32. In scenes with many objects, especially many generators used, there will be collisions.

I ended up calculating GUIDs myself, only using UniqueIPs, hasing them with xxHash (https://github.com/Cyan4973/xxHash), fixing some of the obvious shortcomings by adding dummy IPs for generators - or something like that, it's been a while.

Works pretty well for uniqely identifying objects across frames of an animation - unfortunately it's useless for identifying objects across insertions/deletions in the scene.

Originally posted by xxxxxxxx

Actually the plan is to add the info to the SDK docs.

Please add "not using crc32" to said plan.

[Helper]

On 28/08/2017 at 04:29, xxxxxxxx wrote:

From what I've experienced, the identical GUIDs were not due to hash clashing but actually objects that where generated from the same source object, thus being "identical" to them. Unfortunately, that makes it impossible to identify the generated objects

I also use hashing throughout my parsing and use 64-bit SipHash (https://131002.net/siphash/).

[Helper]

On 01/09/2017 at 09:44, xxxxxxxx wrote:

Thanks both Yves and Martin for keeping your insights to come.

@fused: actually we internally use CRC64 which make the chance to get GUID collisions really unlikely; the reason because there might be equal GUIDs in the scene is mostly related to how GUIDs are currently generated.
@assoc: to reduce (not to exclude) the chance to avoid GUIDs duplication you can enumerate you objects in the scene by setting their IP (BaseObject::SetUniqueIP()) using something like:

  
const Bool SetObjectsIP(BaseObject *op)  
{  
  Int32 ipnum = 1;  
  while (op)  
  {  
      if (!SetObjectsIP(op->GetDown()))  
          return false;  
  
      op->SetUniqueIP(ipnum++);  
  
      op = op->GetNext();  
  }  
  
  return true;  
}  

As I've described in my previous post, we are aware of the fact that caches might end up in sharing the same UID, and we are also aware that unique identification is a pretty relevant topic for interactive rendering purposes. That said I'll try to rise up in the future an internal discussion on the topic to see if and how the desired behavior could be addressed.

Best, Riccardo.

[Helper]

On 04/09/2017 at 00:13, xxxxxxxx wrote:

Thanks Ricardo.

Yes, using SetUniqueIP() is what I would expect to be used by generators that have several objects in their caches. As I am parsing the scene - not generating objects myself - I have no influence how other generators are handling UniqueIPs.

In this regard, I noticed that the Subdivision object does not use UniqueIPs. So I am having a hard time to identify the generated objects between scene changes because they don't have UniqueIPs. So I was trying to use their GUID as fallback but then noticed that the Subdivision object had objects with the same GUID in its cache as well. The GeMarker unfortunately does not help either as it changes frequently.

So I have this case where all methods of the SDK to identify generated objects in a cache fails.

Thanks,
Martin