BaseLinkArray looses link? (was: Refreshing a nested BaseShader)

[fwilleke80]

Hello,

this is related to: https://developers.maxon.net/forum/topic/12823/shader-that-gets-data-from-an-object-refresh/6

My ShaderData uses data from an ObjectData to generate its output. It requests the data from the object during InitRender(). Using the pattern shown in the above linked thread, the ObjectData sets the Shader dirty (DIRTYFLAGS::DATA) and also sends an MSG_UPDATE to it, once the ObjectData has finished computing its data. So, after, every change to the ObjectData's attributes, the ShaderData should be dirty and is told to update.

This works flawlessly, if the shader sits directly e.g. in a material's Color channel (even without the MSG_UPDATE). However, if I put it into another shader, e.g. a Colorizer, the shader's result is not updated until I change any of the shader's attributes. What can do I have to do with the shader to make sure it is updated, and the material preview is re-rendered, even if it's located somewhere deep down in other shaders in the material?

EDIT: I found out what the reason is! Just don't know the solution.

As shown in the above linked thread, I am using a BaseLinkArray to keep a list of observers. Initially, this works fine. The ObjectData holds an Observable object, and the ShaderData nicely subscribes to it. Calling const Int32 observerCount = _observers.GetRealCount(doc); on the BaseLinkArray returns 1. Setting the ShaderData dirty from within the ObjectData works.

But after I put the shader into a Colorizer, _observers.GetRealCount(doc) returns 0, while _observers.GetCount() still returns 1. Therefore, using _observers.GetIndex() to get the link to the shader from the array returns nullptr. It seems the BaseLink in the array is broken after putting the linked shader into another one.

Why? Well, probably because the BaseShader that now sits inside the Colorizer is not the same object as the one that was there before, right? But shouldn't that be updated automatically by Cinema, when the shader is placed inside the Colorizer? Isn't that what BaseLinks are meant for?

Thanks & greetings,
Frank

[fwilleke80]

My first idea would be, to catch a specific message that is maybe sent to the shader when it is being put into the Colorizer.

However, the only message that comes in when this happens, is 1001071 (MSG_MULTI_RENDERNOTIFICATION). I guess I could use it to update the BaseLink in the observable. But it seems that message comes in very often, simply when clicking through the material editor. Does not sound like a good idea.

[Manuel]

hi,

one question / suggestion :

is there any difference if you change the order of creating your shader ?
for example
add your shader -> add colorizer (that will move your shader inside the colorizer)
add colorizer-> go inside and add your shader.

If there's any change, that could mean that your shader is "copied" and not only moved.
I would check if the CopyTo function is used in the process. That would allow you to update the object's array.

I would need a bit of time to recreate an example and see if i can find a solution.
(you can probably test it before me)

Cheers,
Manuel

[fwilleke80]

Hi Manuel,

thanks for the hint! This does in deed sound promising.

And yes, the shader is copied when put into the Colorizer. As far as I remember, almost nothing in Cinema is ever moved, it's all copied.
I added CopyTo() to my shader class, and implemented it like this:

Bool MyShader::CopyTo(NodeData *dest, GeListNode *snode, GeListNode *dnode, COPYFLAGS flags, AliasTrans *trn)
{
	iferr_scope_handler
	{
		GePrint(err.GetMessage());
		return false;
	};

	// Get document of destination node
	// Note: When CopyTo() is called because the shader is moved into another shader (e.g. Colorizer), this is always nullptr.
	BaseDocument *destDoc = dnode->GetDocument();

	// Get container of source node
	const BaseContainer &dataRef = static_cast<BaseShader*>(snode)->GetDataInstanceRef();

	// Get linked object from source node's container
	BaseObject *linkedObject = dataRef.GetObjectLink(XTERRAINOPERATORSHADER_OPERATORLINK, snode->GetDocument());

	// If we have a linked object, and if the destination node has a document
	if (linkedObject && destDoc)
	{
		// Get the node's ObjectData
		MyObjectData *linkedNodeData = linkedObject->GetNodeData<MyObjectData>();
		if (! linkedNodeData)
			return false;

		// Get ref to the Observable object
		MyObservable &observable = linkedNodeData->GetObservableRef();

		// Subscribe to shader to observable
		observable.Subscribe(dnode, destDoc) iferr_return;
	}

	return SUPER::CopyTo(dest, snode, dnode, flags, trn);
}

It changes nothing. Still doesn't work. _observers.GetIndex() still returns nullptr, even though _observers.GetCount() returns 1.

What I noticed:

• MyShader::CopyTo() is called numerous times, even if I just navigate through the material manager. When I put my shader into a Colorizer, CopyTo() is called 10 times!
• When CopyTo()is called because I move the shader into a Colorizer, dnode->GetDocument() returns nullptr. I tried using snode->GetDocument() instead. Now, after creating my shader, then moving it into a Colorizer, and then changing the Colorizer's gradient once, I have 25(!) subscribers in the observable. And all the BaseLinks are broken!! Not a single one can be retrieved with GetIndex().

I still ask myself why I even have to do all this. Isn't the purpose of BaseLinks to have stable links to nodes even if they're moved and change pointer address? All in all, I'm starting to wonder if a hidden InExcludeList in MyObject's description wouldn't be easier. Incredibly clumsy, yes, but it might just work...

Cheers,
Frank

[Manuel]

hi,

About the hidden include list, yes it should work. Note that even if you store the baselink inside the basecontainer of your object, they should be updated by an aliastrans.
That would be a bit better than an hidden in/exclude list.

Cheers,
Manuel

[fwilleke80]

Aha, that's interesting!

So, if I got this right, the problem is that my BaseLinkArray is hidden inside my NodeData where it can't be found by Cinema, hence can't be updated by the AliasTrans?

If it was just one BaseLink it would be easy. But I have an arbitrary number of BaseLinks in a BaseLinkArray. Can that somehow be stored in the BaseContainer? Or is there any other way to make my BaseLinkArray known to Cinema, so it can be updated? Or a way to update it myself?

In the SDK docs, it says about AliasTrans::Translate(): "Translates the links in all objects that the translator has come across.". Can I make it "come across" my BaseLinkArray?

@kbar: In the other thread you mentioned that you also implemented an observer pattern. Did you run into this problem, too?

Thanks & greetings,
Frank

[Manuel]

hi,
when you Init your aliastrans, it trying to find all it can everywhere. So every BaseContainer are scanned.
That also include the document's BaseContainer.

As you can't say aliastrans where to look, you have to place your data where it can find them.

(but the hidden in/exclude should work)

[fwilleke80]

Thank you

Cheers,
Frank

[fwilleke80]

Just a little follow-up on this:

It would be very much appreciated if there was like a Message() coming in, or a callback, or some other way to simply present any BaseLink members of the NodeData to an AliasTrans.

Cheers,
Frank

[fwilleke80]

It's a never-ending story...
InExcludeList does not seem to be the answer.

I have changed my Observable code to use an InExcludeList in the description of the Observable (the BaseObject that is to be observed). When that object is linked in the Observer (the BaseShader that links to the BaseObject via a LINK description element), I catch SetDParameter(), remove the Shader from the previously linked Observable's InExcludeList, and add it to the newly linked Observable's InExcludeList.

So far, that works just fine.

However, when nesting the BaseShader in another shader, e.g. a Colorizer, the shader's CopyTo() is called. Here, I am trying to add the destination shader to the Observable's InExcludeList. It does not work.

Bool MyShaderData::CopyTo(NodeData *dest, GeListNode *snode, GeListNode *dnode, COPYFLAGS flags, AliasTrans *trn)
{
	iferr_scope_handler
	{
		GePrint(err.GetMessage());
		return false;
	};

	// Get everything we need
	BaseDocument *sourceDoc = snode->GetDocument();
	BaseDocument *destDoc = dnode->GetDocument(); // This is always NULL!
	BaseShader *sourceShader = static_cast<BaseShader*>(snode);
	BaseShader *destShader = static_cast<BaseShader*>(dnode);

	// Get linked object from the shader's LINK
	const BaseContainer &dataRef = sourceShader->GetDataInstanceRef();
	BaseObject *linkedObject = dataRef.GetObjectLink(MYSHADER_OPERATORLINK, snode->GetDocument());

	if (linkedObject && sourceDoc/*destDoc*/) // Can't check destDoc, as it's always NULL
	{
		// Get linked object's Observable instance
		MyObjectData *linkedObjectData = linkedObject->GetNodeData<MyObjectData>();
		if (!linkedObjectData)
			return false;
		MyObservable &observable = linkedObjectData->GetObservableRef();
		// Add destShader as new observer to the linked object's Observable
		observable.Subscribe(destShader, linkedObject, MYOBJECT_OBSERVERLIST, sourceDoc) iferr_return;
	}

	// Translate (doesn't seem to make a difference)
	if (trn)
		trn->Translate(true);

	return SUPER::CopyTo(dest, snode, dnode, flags, trn);
}

I am looking at the InExcludeList in the linked object. After MyShader::CopyTo()was called, the list always appears empty. observable.Subscribe() is called, but the new copy of the shader does not show up in it.

So, my question remains:
How can I maintain a BaseLink to my shader in an object, after the shader has been copied?

Cheers,
Frank

ADDITION:
Interesting. If I add an InExcludeList to any object, just for a test, and not actually handle it anywhere in my code, it also looses the link to any BaseShader I add to the list, when that BaseShader is then nested in another shader. I just added a useless InExcludeList to some object, linked a Noise shader in it, and then nested the Noise Shader in a Colorizer. Whoops, the InEcludeList is empty

So, it seems, the reason for this not working isn't even in my code. Is this a bug in Cinema, or a general problem, or does the object that holds the InExcludeList have to do something in order to make it work? Catch some message, or something like that?

[fwilleke80]

Any news on this one?

A heads-up about how to maintain a reliable list of BaseLinks to BaseShaders in the scene would be most welcome!

Greetings,
Frank

[ferdinand]

Hi,

@fwilleke80 said in BaseLinkArray looses link? (was: Refreshing a nested BaseShader):

Interesting. If I add an InExcludeList to any object, just for a test, and not actually handle it anywhere in my code, it also looses the link to any BaseShader I add to the list, when that BaseShader is then nested in another shader. I just added a useless InExcludeList to some object, linked a Noise shader in it, and then nested the Noise Shader in a Colorizer. Whoops, the InEcludeList is empty

I tried this in Python and was not able to reproduce it. Which does not mean that it odes not exist, but you might want to provide more details on what leads to this described behaviour. Below you will find a Python script which does what I did read into the quoted paragraph of yours above.

Cheers,
zipit

"""Will mimic the steps you described to invoke a faulty InEx behavior.

You can run this without any prerequisites in the Script Manager, it will
create a null object and a material with three shaders which are linked in
the null as described by you.

I was not able to reproduce the behavior described by you in the posting,
specifically this paragraph:

    "I just added a useless InExcludeList to some object, linked a Noise
    shader in it, and then nested the Noise Shader in a Colorizer. Whoops,
    the InEcludeList is empty [...]"

I interpreted your statement as such, as you want to link the shaders BEFORE
they are actually a part of the scene graph. At least this is the only way
your sentence does make sense to me. But as far as I can tell, everything 
just seems to work fine? 

Are you sure, that your shader graph is actually valid? Because if it isn't,
obviously also the InEx will fail.
"""


import c4d

def main():
    """Will create a setup as described by you.
    """
    # A null with an InEx user data element.
    null = c4d.BaseList2D(c4d.Onull)
    bc = c4d.GetCustomDataTypeDefault(c4d.CUSTOMDATATYPE_INEXCLUDE_LIST)
    bc[c4d.DESC_NAME] = "shaders"
    null.AddUserData(bc)
    inex = null[c4d.ID_USERDATA, 1]

    # A material and three shaders as described by you.
    material = c4d.BaseList2D(c4d.Mmaterial)
    shader_colorizer = c4d.BaseList2D(c4d.Xcolorizer)
    shader_noise = c4d.BaseList2D(c4d.Xnoise)
    shader_noise.SetName("noise")
    shader_noise_nested = c4d.BaseList2D(c4d.Xnoise)
    shader_noise_nested.SetName("noise_nested")

    # Trying to fulfill your condition of linking the shaders BEFORE they
    # are bing nested.
    inex.InsertObject(shader_noise, 0)
    inex.InsertObject(shader_noise_nested, 0)
    null[c4d.ID_USERDATA, 1] = inex

    # Compose the material, i.e. do the nesting.
    material[c4d.MATERIAL_USE_DIFFUSION] = True
    material[c4d.MATERIAL_USE_REFLECTION] = False
    material[c4d.MATERIAL_COLOR_SHADER] = shader_noise
    material[c4d.MATERIAL_DIFFUSION_SHADER] = shader_colorizer
    shader_colorizer[c4d.SLA_COLORIZER_TEXTURE] = shader_noise_nested

    # Insert the nodes into the scene graph.
    material.InsertShader(shader_colorizer)
    material.InsertShader(shader_noise)
    shader_colorizer.InsertShader(shader_noise_nested)
    doc.InsertObject(null)
    doc.InsertMaterial(material)

    # Update stuff
    material.Message(c4d.MSG_UPDATE)
    material.Update(True, True)
    c4d.EventAdd()

if __name__=='__main__':
    main()

[fwilleke80]

Hi @zipit,

thank you for investing the time to try this out!

I don't know why it works for you, but it certainly does not for me.

Take a look at this video, where I reproduce the problem completely without writing any custom plugin code. Loosing the BaseLink to a BaseShader when that shader is nested into another shader seems to be default behaviour in Cinema 4D, and I find that highly unsatisfying

I wish the SDK Team would say something about this, or even present a workaround.

Video:
https://frankwilleke.de/misc/C4D_looses_link_to_nested_shader.mov

In this video, you can see that, for a test, I added an IN_EXCLUDE and a LINK element to the description of the standard Cube object. Then I create a material, and in that material I create a Noise shader. That shader I link in the IN_EXCLUDE and the LINK elements. Works as expected. Then I nest the Noise shader in a Colorizer shader, and both links are gone. Since none of my plugin code is contributing to this behaviour, I don't think that the issue is anywhere in my code.

Since both, the IN_EXCLUDE and the LINK element, are using BaseLinks, I think the BaseLink itself is the culprit here.

Cheers,
Frank

Edit:
Found a way to demonstrate this without even touching any .h/.res files to add elements:

Video:
https://frankwilleke.de/misc/C4D_looses_link_to_nested_shader_2.mov

When linking a BaseShader inside an XPresso Object Node, the link is also lost when the linked shader is being nested.

[Manuel]

hi,

sorry for the delay (i was on vacation last week), I'm on it.

Cheers,
Manuel

[fwilleke80]

No worries

[a_block]

Hi,

I only briefly scanned this topic, but I have a few thoughts and questions. Simply ignore me, if I talk rubbish as usual...

Does this problem occur with every shader that supports subshaders or only a few? Does it happen with the layer shader as well, for example?
If I remember correctly BaseLinks work only correctly (especially in AliasTrans context), if the linked object is a valid member of a document (e.g. in a NodeData for example via branch. Now, if my mind doesn't trick me, there are certain shaders, which do not properly hold there subshaders in the shader tree. Which caused me all kinds of issues, when trying to initiate shader updates from a NodeData and I ended up with special code for certain shaders (I think, it were mainly those stemming from good ol' Smells Like Almonds shaders), basically extracting the subshaders from their parameters.

And another thought, maybe you already tried and I overlooked it in this thread: Does using ForceGetLink() change anything for you, when trying to resolve those links? If I'm not mistaken, C4D also resolves links beyond a document's context with this.

Cheers,
Andreas

[fwilleke80]

Hi Andreas,

Well, I didn't try ALL the shaders there are, but it does happen with shaders that are not my plugin shader. So far, with all shaders I have tried.

In my example videos, I do have the LINK and IN_EXCLUDE elements on an object in the scene, and in an XPresso node, both loose the link.

ForceGetLink() does not change anything, unfortunately. It just returns nullptr.

Cheers,
Frank

[fwilleke80]

I suspect I won't get an answer here...

I have figured out a way on my own now. It's pretty cumbersome, requires too much code in too many places to post it here, and seems to work flawlessly though it looks fishy.

It is based on keeping BaseLinks to the BaseMaterials that own the BaseShaders. Then, when I need to notify the Observers, I go through all the materials in the list, then iterate all their child BaseShaders until I find the ones that have a link to my Observable object, and then I notify these. Plus, some extra measures to make sure the Observer re-subscribes when specific things happen to it.

It is, like we say in German, from behind through the chest into the eye, but it actually works. Hooray.

Cheers,
Frank

[Manuel]

hi,

never say never. Sorry for the delay, this is a "unusual" case ^^

I did try several options (observable, notification, message etc)

I finished using the function AddEventNotification witch is private. (so i had to figure out how it was working)
The shader and the viewport is updated but with a small latency.

We can try to improve that solution if you want or you can go with your own.

Below the code for both the ObjectData and the Shader.

The shader need to register to be notified. (this is automatic, nothing to add on the ObjectData's side)

Than you need to check the message in Message
I feel that the important part is to remove the notification and re/add it. (not sure what it can do if you don't do it ^^)

class pc12019_object : public ObjectData
{
	INSTANCEOF(pc12019_object, ObjectData);
public:
	static NodeData* Alloc() { return NewObjClear(pc12019_object); }



	BaseObject* GetVirtualObjects(BaseObject* op, HierarchyHelp* hh) override
	{
		return BaseObject::Alloc(Ocube);
	}


	Bool GetDDescription(GeListNode* node, Description* description, DESCFLAGS_DESC& flags) override
	{
		if (!description->LoadDescription(Obase))
			return false;

		const DescID* singleid = description->GetSingleDescID();


		const DescID cid = DescLevel(OBJ_COLOR, DTYPE_COLOR, 0);

		if (!singleid || cid.IsPartOf(*singleid, nullptr))
		{
			BaseContainer bc = GetCustomDataTypeDefault(DTYPE_COLOR);

			bc.SetString(DESC_NAME, "Color"_s);
			bc.SetData(DESC_GUIOPEN, true);

			description->SetParameter(cid, bc, DescLevel(ID_OBJECTPROPERTIES));
		}

		flags |= DESCFLAGS_DESC::LOADED;

		return SUPER::GetDDescription(node, description, flags);
	}




};

class pc12019_shader : public ShaderData
{
	INSTANCEOF(pc12019_shader, ShaderData);
public:
	static NodeData* Alloc() { return NewObjClear(pc12019_shader); }


	Vector Output(BaseShader* sh, ChannelData* cd) override
	{
		return color;
	}


	Bool Read(GeListNode* node, HyperFile* hf, Int32 level) override
	{
		// need to read the color

		return true;
	}


	Bool Write(GeListNode* node, HyperFile* hf) override
	{
		// need to write the color
		return true;
	}


	Bool CopyTo(NodeData* dest, GeListNode* snode, GeListNode* dnode, COPYFLAGS flags, AliasTrans* trn) override
	{
		pc12019_shader* destShader = static_cast<pc12019_shader*>(dest);
		destShader->color = color;

		return true;
	}


	Bool GetDDescription(GeListNode* node, Description* description, DESCFLAGS_DESC& flags) override
	{
		if (!description->LoadDescription(Xbase))
			return false;

		const DescID* singleid = description->GetSingleDescID();


		DescID cid = DescLevel(SH_OBJECTLINK, DTYPE_BASELISTLINK, 0);

		if (!singleid || cid.IsPartOf(*singleid, nullptr))
		{
			BaseContainer bc = GetCustomDataTypeDefault(DTYPE_BASELISTLINK);

			bc.SetString(DESC_NAME, "Link"_s);
			bc.SetInt32(DESC_CUSTOMGUI, CUSTOMGUI_LINKBOX);

			BaseContainer ac;
			ac.SetInt32(Obase, 1);
			bc.SetContainer(DESC_ACCEPT, ac);

			description->SetParameter(cid, bc, DescLevel(ID_SHADERPROPERTIES));
		}



		flags |= DESCFLAGS_DESC::LOADED;

		return SUPER::GetDDescription(node, description, flags);
	}

	INITRENDERRESULT InitRender(BaseShader* sh, const InitRenderStruct& irs) override
	{
		iferr_scope_handler
		{
			err.DiagOutput();
			return INITRENDERRESULT::UNKNOWNERROR;
		};
		

		if (irs.doc == nullptr)
			return INITRENDERRESULT::OK;
		
		BaseContainer* bcShader = sh->GetDataInstance();
		if (bcShader == nullptr)
			return INITRENDERRESULT::UNKNOWNERROR;
		
		BaseObject * linkedObject = bcShader->GetObjectLink(SH_OBJECTLINK, irs.doc);

		if (linkedObject == nullptr)
			return INITRENDERRESULT::OK;

		RemoveNotification(sh, linkedObject);
		AddNotification(sh, linkedObject);
		
		GeData data;
		linkedObject->GetParameter(OBJ_COLOR, data, DESCFLAGS_GET::NONE);
		color = data.GetVector();
	
		return INITRENDERRESULT::OK;
	}


		Bool Init(GeListNode* node) override
		{
			BaseMaterial* mat = static_cast<BaseMaterial*>(node);
			BaseContainer* bc = mat->GetDataInstance();
			bc->SetLink(SH_OBJECTLINK, nullptr);
			
			return true;



		}

		void Free(GeListNode* node) override
		{
			GeData data;
			BaseObject* op = static_cast<BaseObject*>(node);
			if (op == nullptr)
				return;
			BaseDocument* doc = node->GetDocument();
			if (doc == nullptr)
				return;
			
			op->GetParameter(SH_OBJECTLINK, data, DESCFLAGS_GET::NONE);
			BaseList2D* linkedObject = data.GetLink(doc);
			RemoveNotification(node, linkedObject);
			
		}

		
		
		Bool RemoveNotification(GeListNode* node, BaseList2D* linkedObject)
		{
			if ((node == nullptr) || (linkedObject == nullptr))
				return false;

			BaseObject* op = static_cast<BaseObject*>(node);
			if (op == nullptr)
				return false;

			BaseDocument* doc = op->GetDocument();
			if (doc == nullptr)
				return false;

			if (linkedObject->FindEventNotification(doc, op, NOTIFY_EVENT::CACHE))
			{
				return linkedObject->RemoveEventNotification(doc, op, NOTIFY_EVENT::CACHE);
			}

			return true;
		}

		Bool AddNotification(GeListNode* node, BaseList2D* linkedObject)
		{
			BaseObject* op = static_cast<BaseObject*>(node);
			if (op == nullptr)
				return false;

			BaseDocument* doc = op->GetDocument();
			if (doc == nullptr)
				return false;

			if (!linkedObject->FindEventNotification(doc, op, NOTIFY_EVENT::MESSAGE))
			{
				return linkedObject->AddEventNotification(op, NOTIFY_EVENT::MESSAGE, NOTIFY_EVENT_FLAG::NONE, nullptr);
			}

			return true;
		}

		Bool Message(GeListNode* node, Int32 type, void* data) override
		{
			if (type == MSG_NOTIFY_EVENT)
			{
				NotifyEventData *eventData = static_cast<NotifyEventData*>(data);
				// as we can add several notification we check if we received the right one
				if (eventData->eventid == NOTIFY_EVENT::MESSAGE)
				{
					NotifyEventMsg* eventMsg = static_cast<NotifyEventMsg*>(eventData->event_data);
					// Checks if the message that the object is fowarding us if the right one. (MSG_DESCRIPTION_POSTSETPARAMETER)
					if (eventMsg->msg_id == MSG_DESCRIPTION_POSTSETPARAMETER)
					{
						DescriptionPostSetValue* dpsv = static_cast<DescriptionPostSetValue*>(eventMsg->msg_data);
						
						const Int32 id  = dpsv->descid[0][0].id;
						// Checks if the parameter that have been change is part of the message.
						if (id == OBJ_COLOR)
						{
							// ask the shader to update itself.
							node->SetDirty(DIRTYFLAGS::ALL);
							node->Message(MSG_DESCRIPTION_POSTSETPARAMETER); // this will trigger the mat update and the viewport update. (with a small delay)
						}
					}
				}
					return true;
			}
			return SUPER::Message(node, type, data);
		}

private:
		Vector color = Vector(1);
};

Cheers,
Manuel