The Falcon Scavenge operation needs several improvements;

1) There is a possibility that some age groups may be very large in comparison
the the full size of the record cache.  This can happen when record changes are
handled by the engine in bursts.  In order to prevent scavenging too many
records when one of these large age groups occur, we could either try to
scavenge partial age groups or prevent large age groups.  There are too many
problems associated with scavenging partial age groups, so the best approach is
to prevent them.  A new age group should be started whenever the record cache
size increases by minAgeGroupSize.  minAgeGroupSize can be RecordCacheSize /
NumberOfAgeGroups.

1) Scavenging needs to be started on a load based event.  A prototype of this
included a call to trigger this event from Table::allocRecord() and
Table::allocRecordVersion().  It would make a call to
Table::checkRecordScavenge() approximately every 100 records allocated.  Then
whenever a new age group is created and the current record cache active size has
passed the scavenge threshold, it would call a scavenge cycle.

2) The scavenger thread should be a separate thread from the scheduler so that
it can be signaled as soon as the record cache hits the scavenge threshold. 
This will wake it up sooner than the scheduled event.

4) Once load-based scavenging can be triggered, the scavenge threshold can be
adjusted up from its current level at 66% of cache.  Also, the scavenge floor
can be adjusted upwards as well.  However, the scavenge cycle should not be too
short since there is some overhead involved in preparing for a scavenge.   The
higher the floor, the less you scavenge per cycle.  So it should not be done too
often.  But it is currently not being done often enough.

5)The forceRecordScavenge() process tries to save time by not calling
inventoryRecords and just scavenging as much as it can.  So it currently
scavenges too much and makes the client thread that need memory wait too long. 
This should be a rare event once the load-based scavenge is implemented and
tuned.  So to simplify the code, forceRecordScavenge() should also just do a
load-based scavenge.

• WL#4477: Falcon Engine - Scavenge: load based trigger
• WL#4478: Falcon Engine - Scavenge: Prevent Large Age Groups

You must be logged in to tag this worklog

No Comments yet