Execution Delay When Reading Data from JSON file

bentraje

Hi,

For some reason, I get an execution delay when reading data from JSON file compared to hardcoded data. I understand the latter is faster. But I expect the former to be also instantaneous since I'm not computing numbers (i.e. just reading them)

You can see the problem here (15 sec). First part is executing with JSON data while the latter is executing with hardcoded values
https://www.dropbox.com/s/5yzmbsil9u2xa7m/c4d192_execution_delay_from_json_data.mp4?dl=0

Is there a way around this?
Thank you for looking at my problem

ferdinand

@bentraje said in Execution Delay When Reading Data from JSON file:

For some reason, I get an execution delay when reading data from JSON file compared to hardcoded delay.

Hi,

I am not sure what you are expecting, but:

Opening files will take some time.
Parsing JSON will take some time, Python's native JSON parser is especially slow. You loose most of your time probably here.

What you can do:

More a minor point in the given example, but your code currently loads a file for each scene object, so for 100 objects you will have to open 100 files, introducing considerable overhead.
Choose a faster JSON parser (see for a comparison here)
Choose a more appropriate method of serialisation. Text based serialisation is a great thing and JSON has many advantages. Speed isn't one of them. Binary serialisation or more simple text based format like CSV will speed up things considerably.

Cheers
zipit

bentraje

@zipit

Thanks for the response. Happy New Year.

RE: Opening files and Python's native JSON parser is especially slow.
I understand this but whenever I print the data from the JSON, it outputs to the console instantaneously. Same performance with printing hardcoded data to the console.

Also, the sample file contains only 1 object (and I selected only one object). It should be instantaneous.

So I was thinking I'm missing something in using JSON data with regard to specific C4D API.

RE: 100 objects
Point taken. I should probably revise that.

RE:Choose a faster JSON parser
Unfortunately, I'm married to the built-in JSON parser shipped with C4D since I'll be sending it to other users.

RE: Binary serialisation or more simple text based format like CSV will speed up things considerably.

Yea, I agree but I have more or less the same code for Blender (i.e. reading JSON data and applying it to an object) for at least 40 objects and it is instantaneous.

So I was thinking I'm missing something in using JSON data with regard to specific C4D API.

ferdinand

@bentraje said in Execution Delay When Reading Data from JSON file:

I understand this but whenever I print the data from the JSON, it outputs to the console instantaneously. Same performance with printing hardcoded data to the console.

Hi,

a Happy New Year to you too. This all is rather hard to judge from a video, some code, example data and benchmarks would be probably more helpful.

The only thing I can spot in your video, is that you access your values three times for each component of the vector, effectively making the hash map access not an O(1), but an O(3) operation. But this should not have that much of an impact, since it is a constant (O(1)) operation in the first place.

As I said, a testable example would be helpful. Out of my head I cannot think of a reason for the described behaviour.

Cheers
zipit

bentraje

You can ignore this thread for now.
After restart, I started to create a sample file to be uploaded. For some reason, it now works as expected and I didn't modify the code at all.

Maybe some weird bug?

Anyhow, thanks for the help!

m_adam

I don't have a lot to add here, @zipit almost pointed everything moreover doing proper profiling for hard drive can be a bit tricky to set up for couple of reasons:

Another process may randomly use hard drive (e.g Antivirus checking).
Cached file optimization made by either the OS, the processor or even the hard drive itself.

Cheers,
Maxime.