RecordKey() & Memory usage

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 03/12/2008 at 11:24, xxxxxxxx wrote:

Something just has to be wrong with this picture...
For  38,010 total keys (skipping 4/5ths of them), I'm using a gigabyte of memory.
For 740,880 total keys, the .FBX import is using ~75 mb (in the ball-park of 10,000 keys per mb)
...I must be doing something wrong.  Any help?

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 03/12/2008 at 11:34, xxxxxxxx wrote:

..also, if I just comment out the RecordKey() lines, I only use ~4mb (for the tracks and mesh objects, etc., but no keys recorded), so this is something specific to RecordKey(), or the way I'm using it.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 00:30, xxxxxxxx wrote:

I added some tracking code and here are some additional stats:

• average memory allocated per RecordKey() call - 27KB (as much as half a megabyte for just 3 keys , but there appears to be a memory pooling mechanism in place, since somtimes it's zero)
• 3,954 translation keys uses 104.57MB
• 34,056 rotation keys uses 917.32MB
• 38,010 total keys uses 104MB + 917MB = 1.02GB

...so, I guess my question boils down to: Should RecordKey() be using an average of 27KB per call?
I'm still baffled how the .FBX import, which is creating far more keys than I am is using far less memory.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 01:10, xxxxxxxx wrote:

Interesting... I'm playing with my skip count (skipping more frames, so fewer total keys get generated) and getting different results...

• 24 trans @ 0bytes avg + 201 rot @ 2.62KB avg = 526.39KB total
• **57  trans @0bytes avg **** + **  453 rot @ 3.10KB avg = 1.37MB total
• 171  trans @ 5.13KB avg +   1515 rot @ 3.24KB avg =   5.65MB total
• 249  trans @ 2.11KB avg +   2238 rot @ 4.31KB avg =   9.94MB total
• 381 trans @ 3.68KB avg + 3177 rot @ 4.91KB avg =   16.62MB total
• 765  trans @ 8.02KB avg +   6555 rot @ 7.14KB avg =   51.72MB total
• 2004  trans @ 14.09KB avg +   16989 rot @ 15.06KB avg =   277.43MB total
• 3954  trans @ 26.99KB avg +   34056 rot @ 27.59KB avg =   1.02GB total

...note that my tracking code is wrapped soley around sets of 3 calls to RecordKey(), so the averages and totals ONLY reflect memory actually used by calling that routine.   Also note the progression in average memory used per call as the number of total keys generated gets higher - from 0- 2.6KB for a couple hundred keys, up to ten times that number for 38,000+ keys.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 01:49, xxxxxxxx wrote:

Additional data-points...
Here are numbers for "max amount allocated (pool?) to generate 3 keys" (ie. either a TransX/TransY/TransZ or RotX/RotY/RotZ set of keys)...

• @225 keys - 175.47KB
• @510 keys - 175.47KB
• @1686 keys - 350.83KB
• @2487 keys - 175.48KB (odd)
• @3558 keys - 350.81KB
• @7320 keys - 350.83KB
• @18993 keys - 350.83KB
• @20655 keys - 525.95KB
• @23466 keys - 350.84KB (odd)
• @26766 keys - 526.28KB
• @31398 keys - 526.28KB
• @38010 keys - 526.16KB
• @47019 keys - 526.28KB (per key averages are up to ~ 33KB )
• @62667 keys - 689.83KB (per key averages  are up to ~ 43KB )
• @94491 keys - 868.80KB (per key averages  are up to ~ 65KB )

...I couldn't go any higher, because I was using 5.84GB of memory for that last reading.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 02:01, xxxxxxxx wrote:

One thought (which I'll go check out)... I'm compiling this with the R10.1 SDK, but am using R10.5 to test with, which also means that the (built-in) FBX importer is compiled for R10.5.  Maybe this bug got fixed between R10 and R10.5?  I'll brb
EDIT: D'oh! if anything, the problem is slightly worse in R11 (compiled with R11 SDK and run in R11).

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 02:37, xxxxxxxx wrote:

Finally, in case anyone is trying to reproduce my results, here's my updated code snippet...

    
    
      
    DescID idPosX = DescID(DescLevel(ID_BASEOBJECT_POSITION,DTYPE_VECTOR,0),DescLevel(VECTOR_X,DTYPE_REAL,0));  
    DescID idPosY = DescID(DescLevel(ID_BASEOBJECT_POSITION,DTYPE_VECTOR,0),DescLevel(VECTOR_Y,DTYPE_REAL,0));  
    DescID idPosZ = DescID(DescLevel(ID_BASEOBJECT_POSITION,DTYPE_VECTOR,0),DescLevel(VECTOR_Z,DTYPE_REAL,0));  
    DescID idRotX = DescID(DescLevel(ID_BASEOBJECT_ROTATION,DTYPE_VECTOR,0),DescLevel(VECTOR_X,DTYPE_REAL,0));  
    DescID idRotY = DescID(DescLevel(ID_BASEOBJECT_ROTATION,DTYPE_VECTOR,0),DescLevel(VECTOR_Y,DTYPE_REAL,0));  
    DescID idRotZ = DescID(DescLevel(ID_BASEOBJECT_ROTATION,DTYPE_VECTOR,0),DescLevel(VECTOR_Z,DTYPE_REAL,0));
    
    
    
    
    Matrix am;  
    DWORD totalkeys = 0;  
    BaseContainer stat;  
    LLONG inuse1, inuse2, memdiff, maxmem = 0;  
    LLONG tmemavg, tmemcnt = 0, tmemtotal = 0;  
    LLONG rmemavg, rmemcnt = 0, rmemtotal = 0; 
    
    
    
    
    //---------...S N I P...-----------------------------------------------
    
    
    
    
    if( (DWORD)(pKey->time) == i )  
    {  
        // convert to Cinema4D Matrix  
        am = Matrix( Vector(pJ->matLocal[0][3], pJ->matLocal[1][3], pJ->matLocal[2][3]),  
                     Vector(pJ->matLocal[0][0], pJ->matLocal[1][0], pJ->matLocal[2][0]),  
                     Vector(pJ->matLocal[0][1], pJ->matLocal[1][1], pJ->matLocal[2][1]),  
                     Vector(pJ->matLocal[0][2], pJ->matLocal[1][2], pJ->matLocal[2][2]));
    
    
    
    
        BaseObject *pJointObj = pJ->pJointObj;  
        pJointObj->SetMl(am);  
        pJointObj->Message(MSG_UPDATE);  
        stime = BaseTime(pKey->time, m_animationFps);
    
    
    
    
        GeGetMemoryStat(stat); // check memeory usage  
        inuse1 = stat.GetLLong(C4D_MEMORY_STAT_MEMORY_INUSE);
    
    
    
    
        m_pBaseDoc->RecordKey(pJointObj, stime, idPosX, NULL, false, false, false);  
        m_pBaseDoc->RecordKey(pJointObj, stime, idPosY, NULL, false, false, false);  
        m_pBaseDoc->RecordKey(pJointObj, stime, idPosZ, NULL, false, false, false);
    
    
    
    
        GeGetMemoryStat(stat); // check memeory usage  
        inuse2 = stat.GetLLong(C4D_MEMORY_STAT_MEMORY_INUSE);  
        memdiff = inuse2 - inuse1;  
        if( memdiff > maxmem )  
            maxmem = memdiff; // update max allocation tracking  
        tmemtotal += memdiff; // update total memory used for trans keys and count  
        tmemcnt +=3;          // used to compute average-memory-per-trans-key later
    
    
    
    
        totalkeys += 3;  
        break;  
    }  
    

...the above also shows where and how my memory tracking was done (again, there's a similar block of code for the rotation keys, with separate tracking variables).

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 09:34, xxxxxxxx wrote:

In my case, I would just create the track and keys the more low-level way ala: Ctrack::Alloc() and CCurve->AddKey(). It's more work (adding, adding undos, and setting values) but maybe this would reduce the memory consumption you are experiencing with RecordKey().

The consumptive results of RecordKey() could be something to do with undos. Maybe a test here would be to reduce your Undo Level setting drastically and see if there is any change in memory consumption overall.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 09:48, xxxxxxxx wrote:

Yeah, I've been suspecting that RecordKey() is doing some sort of AddUndo() internally, based on the symptoms.  Where can I change the Undo Level setting?  nm.. I found it in preferences... testing now...

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 10:08, xxxxxxxx wrote:

Well, changing the Undo Depth doesn't seem to help, but I'll start looking into the lower level methods - thanks.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 10:10, xxxxxxxx wrote:

Automatic undos in the SDK can be either a blessing or a curse. I've no experience with RecordKey() so am unfamiliar with its behavior. It may be storing a full change undo (UNDO_CHANGE) which always includes the object, its children, and it branches (including tags, materials, animation) which can eat memory for breakfast as I was forced to deal with unavoidably.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 10:32, xxxxxxxx wrote:

Hmm... just to clarify...
Say I have 50 joints, with 6 tracks each (posx, posy, posz, rotx, roty, rotz). So that's 300 total tracks.  To add a new key, I would then:

• look up or otherwise retrieve my ptr to the CTrack (if I ultimately don't need CTrack pointers, I might just stash the CCurve pointers to start with)
• get the CCurve from the CTrack ( pCurve = pTrack->GetCurve() )
• either:
  • Allocate a CKey ( pKey = CKey::Alloc() ), fill it in and pCurve->InsertKey(pKey) it.
  • or:   pKey = pCurve->AddKey( time )  then just fill the key in ( pCurve->SetKeyDefault() + pKey->SetValue() )

...is that all?  What's this undo stuff you're talking about?  In this case, I'm creating a new document, with enirely new data / keys.  Is there any need for me to create undos?

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 11:10, xxxxxxxx wrote:

Bingo!  Well, I need to figure out/fix the rotation values again (as soon as I finish my happy-dance), but I went from 277MB being used down to 3MB by just doing the CCurve/CKey thing (it's a lot faster as well).

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 11:15, xxxxxxxx wrote:

When you create the new CTrack and CKeys, you could be adding undos (UNDO_NEW) - but it depends. In my case (IPP), the content could be being added to an existing document in which the user wants to be able to undo the addition. But if you are always creating a new document, the undos for these is probably unnecessary as there is nothing to 'undo' in a newly created document.

I should add, if you aren't doing this already, after filling in your document, do doc->SetChanged() so that it is marked as unsaved (you only need to do this if you don't add undos).

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 11:24, xxxxxxxx wrote:

...and I went from _not being able to load the full animation_ to 33MB for the entire ~190,000 keys - yay.
It looks like I just need to fix some gimble-lock issue with the rotations now.  Once I get it working, I'll post the updated code.

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 04/12/2008 at 12:18, xxxxxxxx wrote:

After taking time out for another happy-dance, here's the final working code (this time I'll post the rotation key section instead of the position key section.. and have removed all the memory tracking stuff for clarity) :

    
    
      
    //---------...S N I P...--- some variables setup earlier -----------------  
    Matrix am;  
    CKey *pCKey;  
    LONG keyNdx;  
    Vector hpbBase = MatrixToHPB(pJ->ml);  // 'rest' position, per joint  
    BaseTime stime = BaseTime(i, m_animationFps);  
    //---------...S N I P...-----------------------------------------------
    
    
    
    
    if( (DWORD)(pMyKey->time) == i ) // pMyKey is not a CKey, it's a different structure where my data is stored  
    {  
        // convert to Cinema4D Matrix  
        am = Matrix( Vector(pJ->matLocal[0][3], pJ->matLocal[1][3], pJ->matLocal[2][3]),  
                     Vector(pJ->matLocal[0][0], pJ->matLocal[1][0], pJ->matLocal[2][0]),  
                     Vector(pJ->matLocal[0][1], pJ->matLocal[1][1], pJ->matLocal[2][1]),  
                     Vector(pJ->matLocal[0][2], pJ->matLocal[1][2], pJ->matLocal[2][2]));
    
    
    
    
        Vector hpb = MatrixToHPB(am);        // get HPB angles from joint-local animation matrix  
        hpb = GetOptimalAngle(hpbBase, hpb); // correct for gimble-lock
    
    
    
    
        pCKey= pJ->pCurves[3]->AddKey(stime, &keyNdx); // RotX key  
        if( pCKey) { pJ->pCurves[3]->SetKeyDefault(m_pBaseDoc, keyNdx); pCKey->SetValue(pJ->pCurves[3], hpb.x); }
    
    
    
    
        pCKey= pJ->pCurves[4]->AddKey(stime, &keyNdx); // RotY key  
        if( pCKey) { pJ->pCurves[4]->SetKeyDefault(m_pBaseDoc, keyNdx); pCKey->SetValue(pJ->pCurves[4], hpb.y); }
    
    
    
    
        pCKey= pJ->pCurves[5]->AddKey(stime, &keyNdx); // RotZ key  
        if( pCKey) { pJ->pCurves[5]->SetKeyDefault(m_pBaseDoc, keyNdx); pCKey->SetValue(pJ->pCurves[5], hpb.z); }
    
    
    
    
    
    
    
        break;  
    }  
    

...the position-key code is basically the same, but just uses am.off.x/y/z values.  Also, I just cached the CCurve pointers as an array in my joint structure (0,1,2 are the position curve pointers).
As a final comment for the curious, each key generated now uses ~200 bytes :).

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 08/12/2008 at 02:02, xxxxxxxx wrote:

I gather you resolved your problem then?

cheers,
Matthias

[Helper]

THE POST BELOW IS MORE THAN 5 YEARS OLD. RELATED SUPPORT INFORMATION MIGHT BE OUTDATED OR DEPRECATED

On 08/12/2008 at 02:21, xxxxxxxx wrote:

Yes - thanks.  It's apparently not a good idea to use RecordKey() :).
I could sure use some help with CAWeight Tags now though: https://developers.maxon.net/forum/topic/4233