outliers in bkmult cause persistent over-allocation

From e-mail discussion PZ<->ET on 13 Dec 2019 11:02:53 +0100

rsa250 parameters say: tasks.sieve.bkmult = 1,1l:1.25975,1s:1.5,2s:1.1

We observe, for two distinct work units:

# Average J=61369 for 234 special-q's, max bucket fill -bkmult 1,1l:1.25975,1s:1.83129,2s:1.1
# Average J=60763 for 228 special-q's, max bucket fill -bkmult 1,1l:1.25975,1s:1.88459,2s:1.1

There is some (very limited) wasted time due to the fact that some special-q's must be restarted. More concerning is the fact that because of these special-qs, memory allocation grows globally, which may cause a performance decrease in some scenarios.

An idea could be to store the history of the bucket fill statistics, define the "persistent" bkmult as being (say) mean + 4 standard deviations, and allow temporary over-allocations in exceptional cases ? (outliers may be caused by many reasons, maybe transient, maybe structural)

We should probably investigate first whether we can find a use case where this effect plays an important role.