PPoossttffiixx QQuueeuuee SScchheedduulleerr
-------------------------------------------------------------------------------
OOvveerrvviieeww
The queue manager is by far the most complex part of the Postfix mail system.
It schedules delivery of new mail, retries failed deliveries at specific times,
and removes mail from the queue after the last delivery attempt. There are two
major classes of mechanisms that control the operation of the queue manager.
Topics covered by this document:
* Concurrency scheduling, concerned with the number of concurrent deliveries
to a specific destination, including decisions on when to suspend
deliveries after persistent failures.
* Preemptive scheduling, concerned with the selection of email messages and
recipients for a given destination.
* Credits, something this document would not be complete without.
CCoonnccuurrrreennccyy sscchheedduulliinngg
The following sections document the Postfix 2.5 concurrency scheduler, after a
discussion of the limitations of the existing concurrency scheduler. This is
followed by results of medium-concurrency experiments, and a discussion of
trade-offs between performance and robustness.
The material is organized as follows:
* Drawbacks of the existing concurrency scheduler
* Summary of the Postfix 2.5 concurrency feedback algorithm
* Summary of the Postfix 2.5 "dead destination" detection algorithm
* Pseudocode for the Postfix 2.5 concurrency scheduler
* Results for delivery to concurrency limited servers
* Discussion of concurrency limited server results
* Limitations of less-than-1 per delivery feedback
* Concurrency configuration parameters
DDrraawwbbaacckkss ooff tthhee eexxiissttiinngg ccoonnccuurrrreennccyy sscchheedduulleerr
From the start, Postfix has used a simple but robust algorithm where the per-
destination delivery concurrency is decremented by 1 after delivery failed due
to connection or handshake failure, and incremented by 1 otherwise. Of course
the concurrency is never allowed to exceed the maximum per-destination
concurrency limit. And when a destination's concurrency level drops to zero,
the destination is declared "dead" and delivery is suspended.
Drawbacks of +/-1 concurrency feedback per delivery are:
* Overshoot due to exponential delivery concurrency growth with each pseudo-
cohort(*). This can be an issue with high-concurrency channels. For
example, with the default initial concurrency of 5, concurrency would
proceed over time as (5-10-20).
* Throttling down to zero concurrency after a single pseudo-cohort(*)
failure. This was especially an issue with low-concurrency channels where a
single failure could be sufficient to mark a destination as "dead", causing
the suspension of further deliveries to the affected destination.
(*) A pseudo-cohort is a number of delivery requests equal to a destination's
delivery concurrency.
The revised concurrency scheduler has a highly modular structure. It uses
separate mechanisms for per-destination concurrency control and for "dead
destination" detection. The concurrency control in turn is built from two
separate mechanisms: it supports less-than-1 feedback per delivery to allow for
more gradual concurrency adjustments, and it uses feedback hysteresis to
suppress concurrency oscillations. And instead of waiting for delivery
concurrency to throttle down to zero, a destination is declared "dead" after a
configurable number of pseudo-cohorts reports connection or handshake failure.
SSuummmmaarryy ooff tthhee PPoossttffiixx 22..55 ccoonnccuurrrreennccyy ffeeeeddbbaacckk aallggoorriitthhmm
We want to increment a destination's delivery concurrency when some (not
necessarily consecutive) number of deliveries complete without connection or
handshake failure. This is implemented with positive feedback g(N) where N is
the destination's delivery concurrency. With g(N)=1 feedback per delivery,
concurrency increases by 1 after each positive feedback event; this gives us
the old scheduler's exponential growth in time. With g(N)=1/N feedback per
delivery, concurrency increases by 1 after an entire pseudo-cohort N of
positive feedback reports; this gives us linear growth in time. Less-than-
1 feedback per delivery and integer truncation naturally give us hysteresis, so
that transitions to larger concurrency happen every 1/g(N) positive feedback
events.
We want to decrement a destination's delivery concurrency when some (not
necessarily consecutive) number of deliveries complete after connection or
handshake failure. This is implemented with negative feedback f(N) where N is
the destination's delivery concurrency. With f(N)=1 feedback per delivery,
concurrency decreases by 1 after each negative feedback event; this gives us
the old scheduler's behavior where concurrency is throttled down dramatically
after a single pseudo-cohort failure. With f(N)=1/N feedback per delivery,
concurrency backs off more gently. Again, less-than-1 feedback per delivery and
integer truncation naturally give us hysteresis, so that transitions to lower
concurrency happen every 1/f(N) negative feedback events.
However, with negative feedback we introduce a subtle twist. We "reverse" the
negative hysteresis cycle so that the transition to lower concurrency happens
at the bbeeggiinnnniinngg of a sequence of 1/f(N) negative feedback events. Otherwise, a
correction for overload would be made too late. This makes the choice of f(N)
relatively unimportant, as borne out by measurements later in this document.
In summary, the main ingredients for the Postfix 2.5 concurrency feedback
algorithm are a) the option of less-than-1 positive feedback per delivery to
avoid overwhelming servers, b) the option of less-than-1 negative feedback per
delivery to avoid giving up too fast, c) feedback hysteresis to avoid rapid
oscillation, and c) a "reverse" hysteresis cycle for negative feedback, so that
it can correct for overload quickly.
SSuummmmaarryy ooff tthhee PPoossttffiixx 22..55 ""ddeeaadd ddeessttiinnaattiioonn"" ddeetteeccttiioonn aallggoorriitthhmm
We want to suspend deliveries to a specific destination after some number of
deliveries suffers connection or handshake failure. The old scheduler declares
a destination "dead" when negative (-1) feedback throttles the delivery
concurrency down to zero. With less-than-1 feedback per delivery, this
throttling down would obviously take too long. We therefore have to separate
"dead destination" detection from concurrency feedback. This is implemented by
introducing the concept of pseudo-cohort failure. The Postfix 2.5 concurrency
scheduler declares a destination "dead" after a configurable number of pseudo-
cohorts suffers from connection or handshake failures. The old scheduler
corresponds to the special case where the pseudo-cohort failure limit is equal
to 1.
PPsseeuuddooccooddee ffoorr tthhee PPoossttffiixx 22..55 ccoonnccuurrrreennccyy sscchheedduulleerr
The pseudo code shows how the ideas behind new concurrency scheduler are
implemented as of November 2007. The actual code can be found in the module
qmgr/qmgr_queue.c.
Types:
Each destination has one set of the following variables
int concurrency
double success
double failure
double fail_cohorts
Feedback functions:
N is concurrency; x, y are arbitrary numbers in [0..1] inclusive
positive feedback: g(N) = x/N | x/sqrt(N) | x
negative feedback: f(N) = y/N | y/sqrt(N) | y
Initialization:
concurrency = initial_concurrency
success = 0
failure = 0
fail_cohorts = 0
After success:
fail_cohorts = 0
Be prepared for feedback > hysteresis, or rounding error
success += g(concurrency)
while (success >= 1) Hysteresis 1
concurrency += 1 Hysteresis 1
failure = 0
success -= 1 Hysteresis 1
Be prepared for overshoot
if (concurrency > concurrency limit)
concurrency = concurrency limit
Safety:
Don't apply positive feedback unless
concurrency < busy_refcount + init_dest_concurrency
otherwise negative feedback effect could be delayed
After failure:
if (concurrency > 0)
fail_cohorts += 1.0 / concurrency
if (fail_cohorts > cohort_failure_limit)
concurrency = 0
if (concurrency > 0)
Be prepared for feedback > hysteresis, rounding errors
failure -= f(concurrency)
while (failure < 0)
concurrency -= 1 Hysteresis 1
failure += 1 Hysteresis 1
success = 0
Be prepared for overshoot
if (concurrency < 1)
concurrency = 1
RReessuullttss ffoorr ddeelliivveerryy ttoo ccoonnccuurrrreennccyy lliimmiitteedd sseerrvveerrss
Discussions about the concurrency scheduler redesign started early 2004, when
the primary goal was to find alternatives that did not exhibit exponential
growth or rapid concurrency throttling. No code was implemented until late
2007, when the primary concern had shifted towards better handling of server
concurrency limits. For this reason we measure how well the new scheduler does
this job. The table below compares mail delivery performance of the old +/-
1 feedback per delivery with several less-than-1 feedback functions, for
different limited-concurrency server scenarios. Measurements were done with a
FreeBSD 6.2 client and with FreeBSD 6.2 and various Linux servers.
Server configuration:
* The mail flow was slowed down with 1 second latency per recipient
("smtpd_client_restrictions = sleep 1"). The purpose was to make results
less dependent on hardware details, by avoiding slow-downs by queue file I/
O, logging I/O, and network I/O.
* Concurrency was limited by the server process limit ("default_process_limit
= 5" and "smtpd_client_event_limit_exceptions = static:all"). Postfix was
stopped and started after changing the process limit, because the same
number is also used as the backlog argument to the listen(2) system call,
and "postfix reload" does not re-issue this call.
* Mail was discarded with "local_recipient_maps = static:all" and
"local_transport = discard". The discard action in access maps or header/
body checks could not be used as it fails to update the in_flow_delay
counters.
Client configuration:
* Queue file overhead was minimized by sending one message to a virtual alias
that expanded into 2000 different remote recipients. All recipients were
accounted for according to the maillog file. The
virtual_alias_expansion_limit setting was increased to avoid complaints
from the cleanup(8) server.
* The number of deliveries was maximized with
"smtp_destination_recipient_limit = 2". A smaller limit would cause Postfix
to schedule the concurrency per recipient instead of domain, which is not
what we want.
* Maximum concurrency was limited with "smtp_destination_concurrency_limit =
20", and initial_destination_concurrency was set to the same value.
* The positive and negative concurrency feedback hysteresis was 1.
Concurrency was incremented by 1 at the END of 1/feedback steps of positive
feedback, and was decremented by 1 at the START of 1/feedback steps of
negative feedback.
* The SMTP client used the default 30s SMTP connect timeout and 300s SMTP
greeting timeout.
IImmppaacctt ooff tthhee 3300ss SSMMTTPP ccoonnnneecctt ttiimmeeoouutt
The first results are for a FreeBSD 6.2 server, where our artificially low
listen(2) backlog results in a very short kernel queue for established
connections. The table shows that all deferred deliveries failed due to a 30s
connection timeout, and none failed due to a server greeting timeout. This
measurement simulates what happens when the server's connection queue is
completely full under load, and the TCP engine drops new connections.
cclliieenntt sseerrvveerr ffeeeeddbbaacckk ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt ttiimmeedd--oouutt iinn
lliimmiitt lliimmiitt ssttyyllee ccaacchhiinngg ddeeffeerrrreedd ccoonnccuurrrreennccyy ccoonnnneecctt//
aavveerraaggee//ssttddddeevv ggrreeeettiinngg
-------------------------------------------------------------------------
20 5 1/N no 9.9 19.4 0.49 198 -
20 5 1/N yes 10.3 19.4 0.49 206 -
20 5 1/sqrt(N) no 10.4 19.6 0.59 208 -
20 5 1/sqrt(N) yes 10.6 19.6 0.61 212 -
20 5 1 no 10.1 19.5 1.29 202 -
20 5 1 yes 10.8 19.3 1.57 216 -
-------------------------------------------------------------------------
A busy server with a completely full connection queue. N is the client
delivery concurrency. Failed deliveries time out after 30s without
completing the TCP handshake. See text for a discussion of results.
IImmppaacctt ooff tthhee 330000ss SSMMTTPP ggrreeeettiinngg ttiimmeeoouutt
The next table shows results for a Fedora Core 8 server (results for RedHat 7.3
are identical). In this case, the artificially small listen(2) backlog argument
does not impact our measurement. The table shows that practically all deferred
deliveries fail after the 300s SMTP greeting timeout. As these timeouts were
10x longer than with the first measurement, we increased the recipient count
(and thus the running time) by a factor of 10 to keep the results comparable.
The deferred mail percentages are a factor 10 lower than with the first
measurement, because the 1s per-recipient delay was 1/300th of the greeting
timeout instead of 1/30th of the connection timeout.
cclliieenntt sseerrvveerr ffeeeeddbbaacckk ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt ttiimmeedd--oouutt iinn
lliimmiitt lliimmiitt ssttyyllee ccaacchhiinngg ddeeffeerrrreedd ccoonnccuurrrreennccyy ccoonnnneecctt//
aavveerraaggee//ssttddddeevv ggrreeeettiinngg
-------------------------------------------------------------------------
20 5 1/N no 1.16 19.8 0.37 - 230
20 5 1/N yes 1.36 19.8 0.36 - 272
20 5 1/sqrt(N) no 1.21 19.9 0.23 4 238
20 5 1/sqrt(N) yes 1.36 20.0 0.23 - 272
20 5 1 no 1.18 20.0 0.16 - 236
20 5 1 yes 1.39 20.0 0.16 - 278
-------------------------------------------------------------------------
A busy server with a non-full connection queue. N is the client delivery
concurrency. Failed deliveries complete at the TCP level, but time out
after 300s while waiting for the SMTP greeting. See text for a discussion
of results.
IImmppaacctt ooff aaccttiivvee sseerrvveerr ccoonnccuurrrreennccyy lliimmiitteerr
The final concurrency limited result shows what happens when SMTP connections
don't time out, but are rejected immediately with the Postfix server's
smtpd_client_connection_count_limit feature (the server replies with a 421
status and disconnects immediately). Similar results can be expected with
concurrency limiting features built into other MTAs or firewalls. For this
measurement we specified a server concurrency limit and a client initial
destination concurrency of 5, and a server process limit of 10; all other
conditions were the same as with the first measurement. The same result would
be obtained with a FreeBSD or Linux server, because the "pushing back" is done
entirely by the receiving side.
cclliieenntt sseerrvveerr ffeeeeddbbaacckk ccoonnnneeccttiioonn ppeerrcceennttaaggee cclliieenntt tthheeoorreettiiccaall
lliimmiitt lliimmiitt ssttyyllee ccaacchhiinngg ddeeffeerrrreedd ccoonnccuurrrreennccyy ddeeffeerr rraattee
aavveerraaggee//ssttddddeevv
-------------------------------------------------------------------------
20 5 1/N no 16.5 5.17 0.38 1/6
20 5 1/N yes 16.5 5.17 0.38 1/6
20 5 1/sqrt(N) no 24.5 5.28 0.45 1/4
20 5 1/sqrt(N) yes 24.3 5.28 0.46 1/4
20 5 1 no 49.7 5.63 0.67 1/2
20 5 1 yes 49.7 5.68 0.70 1/2
-------------------------------------------------------------------------
A server with active per-client concurrency limiter that replies with 421
and disconnects. N is the client delivery concurrency. The theoretical
defer rate is 1/(1+roundup(1/feedback)). This is always 1/2 with the fixed
+/-1 feedback per delivery; with the concurrency-dependent feedback
variants, the defer rate decreases with increasing concurrency. See text
for a discussion of results.
DDiissccuussssiioonn ooff ccoonnccuurrrreennccyy lliimmiitteedd sseerrvveerr rreessuullttss
All results in the previous sections are based on the first delivery runs only;
they do not include any second etc. delivery attempts. It's also worth noting
that the measurements look at steady-state behavior only. They don't show what
happens when the client starts sending at a much higher or lower concurrency.
The first two examples show that the effect of feedback is negligible when
concurrency is limited due to congestion. This is because the initial
concurrency is already at the client's concurrency maximum, and because there
is 10-100 times more positive than negative feedback. Under these conditions,
it is no surprise that the contribution from SMTP connection caching is also
negligible.
In the last example, the old +/-1 feedback per delivery will defer 50% of the
mail when confronted with an active (anvil-style) server concurrency limit,
where the server hangs up immediately with a 421 status (a TCP-level RST would
have the same result). Less aggressive feedback mechanisms fare better than
more aggressive ones. Concurrency-dependent feedback fares even better at
higher concurrencies than shown here, but has limitations as discussed in the
next section.
LLiimmiittaattiioonnss ooff lleessss--tthhaann--11 ppeerr ddeelliivveerryy ffeeeeddbbaacckk
Less-than-1 feedback is of interest primarily when sending large amounts of
mail to destinations with active concurrency limiters (servers that reply with
421, or firewalls that send RST). When sending small amounts of mail per
destination, less-than-1 per-delivery feedback won't have a noticeable effect
on the per-destination concurrency, because the number of deliveries to the
same destination is too small. You might just as well use zero per-delivery
feedback and stay with the initial per-destination concurrency. And when mail
deliveries fail due to congestion instead of active concurrency limiters, the
measurements above show that per-delivery feedback has no effect. With large
amounts of mail you might just as well use zero per-delivery feedback and start
with the maximal per-destination concurrency.
The scheduler with less-than-1 concurrency feedback per delivery solves a
problem with servers that have active concurrency limiters. This works only
because feedback is handled in a peculiar manner: positive feedback will
increment the concurrency by 1 at the eenndd of a sequence of events of length 1/
feedback, while negative feedback will decrement concurrency by 1 at the
bbeeggiinnnniinngg of such a sequence. This is how Postfix adjusts quickly for overshoot
without causing lots of mail to be deferred. Without this difference in
feedback treatment, less-than-1 feedback per delivery would defer 50% of the
mail, and would be no better in this respect than the old +/-1 feedback per
delivery.
Unfortunately, the same feature that corrects quickly for concurrency overshoot
also makes the scheduler more sensitive for noisy negative feedback. The reason
is that one lonely negative feedback event has the same effect as a complete
sequence of length 1/feedback: in both cases delivery concurrency is dropped by
1 immediately. As a worst-case scenario, consider multiple servers behind a
load balancer on a single IP address, and no backup MX address. When 1 out of K
servers fails to complete the SMTP handshake or drops the connection, a
scheduler with 1/N (N = concurrency) feedback stops increasing its concurrency
once it reaches a concurrency level of about K, even though the good servers
behind the load balancer are perfectly capable of handling more traffic.
This noise problem gets worse as the amount of positive feedback per delivery
gets smaller. A compromise is to use fixed less-than-1 positive feedback values
instead of concurrency-dependent positive feedback. For example, to tolerate 1
of 4 bad servers in the above load balancer scenario, use positive feedback of
1/4 per "good" delivery (no connect or handshake error), and use an equal or
smaller amount of negative feedback per "bad" delivery. The downside of using
concurrency-independent feedback is that some of the old +/-1 feedback problems
will return at large concurrencies. Sites that must deliver mail at non-trivial
per-destination concurrencies will require special configuration.
CCoonnccuurrrreennccyy ccoonnffiigguurraattiioonn ppaarraammeetteerrss
The Postfix 2.5 concurrency scheduler is controlled with the following
configuration parameters, where "transport_foo" provides a transport-specific
parameter override. All parameter default settings are compatible with earlier
Postfix versions.
PPaarraammeetteerr nnaammee PPoossttffiixx DDeessccrriippttiioonn
vveerrssiioonn
---------------------------------------------------------------------------
Initial per-
initial_destination_concurrency all destination
transport_initial_destination_concurrency 2.5 delivery
concurrency
Maximum per-
default_destination_concurrency_limit all destination
transport_destination_concurrency_limit all delivery
concurrency
Per-
destination
positive
feedback
default_destination_concurrency_positive_feedback 2.5 amount, per
transport_destination_concurrency_positive_feedback 2.5 delivery that
does not fail
with
connection or
handshake
failure
Per-
destination
negative
feedback
default_destination_concurrency_negative_feedback 2.5 amount, per
transport_destination_concurrency_negative_feedback 2.5 delivery that
fails with
connection or
handshake
failure
Number of
failed
pseudo-
cohorts after
default_destination_concurrency_failed_cohort_limit 2.5 which a
transport_destination_concurrency_failed_cohort_limit 2.5 destination
is declared
"dead" and
delivery is
suspended
Enable
verbose
destination_concurrency_feedback_debug 2.5 logging of
concurrency
scheduler
activity
---------------------------------------------------------------------------
PPrreeeemmppttiivvee sscchheedduulliinngg
The following sections describe the new queue manager and its preemptive
scheduler algorithm. Note that the document was originally written to describe
the changes between the new queue manager (in this text referred to as nqmgr,
the name it was known by before it became the default queue manager) and the
old queue manager (referred to as oqmgr). This is why it refers to oqmgr every
so often.
This document is divided into sections as follows:
* The structures used by nqmgr
* What happens when nqmgr picks up the message - how it is assigned to
transports, jobs, peers, entries
* How does the entry selection work
* How does the preemption work - what messages may be preempted and how and
what messages are chosen to preempt them
* How destination concurrency limits affect the scheduling algorithm
* Dealing with memory resource limits
TThhee ssttrruuccttuurreess uusseedd bbyy nnqqmmggrr
Let's start by recapitulating the structures and terms used when referring to
queue manager and how it operates. Many of these are partially described
elsewhere, but it is nice to have a coherent overview in one place:
* Each message structure represents one mail message which Postfix is to
deliver. The message recipients specify to what destinations is the message
to be delivered and what transports are going to be used for the delivery.
* Each recipient entry groups a batch of recipients of one message which are
all going to be delivered to the same destination.
* Each transport structure groups everything what is going to be delivered by
delivery agents dedicated for that transport. Each transport maintains a
set of queues (describing the destinations it shall talk to) and jobs
(referencing the messages it shall deliver).
* Each transport queue (not to be confused with the on-disk active queue or
incoming queue) groups everything what is going be delivered to given
destination (aka nexthop) by its transport. Each queue belongs to one
transport, so each destination may be referred to by several queues, one
for each transport. Each queue maintains a list of all recipient entries
(batches of message recipients) which shall be delivered to given
destination (the todo list), and a list of recipient entries already being
delivered by the delivery agents (the busy list).
* Each queue corresponds to multiple peer structures. Each peer structure is
like the queue structure, belonging to one transport and referencing one
destination. The difference is that it lists only the recipient entries
which all originate from the same message, unlike the queue structure,
whose entries may originate from various messages. For messages with few
recipients, there is usually just one recipient entry for each destination,
resulting in one recipient entry per peer. But for large mailing list
messages the recipients may need to be split to multiple recipient entries,
in which case the peer structure may list many entries for single
destination.
* Each transport job groups everything it takes to deliver one message via
its transport. Each job represents one message within the context of the
transport. The job belongs to one transport and message, so each message
may have multiple jobs, one for each transport. The job groups all the peer
structures, which describe the destinations the job's message has to be
delivered to.
The first four structures are common to both nqmgr and oqmgr, the latter two
were introduced by nqmgr.
These terms are used extensively in the text below, feel free to look up the
description above anytime you'll feel you have lost a sense what is what.
WWhhaatt hhaappppeennss wwhheenn nnqqmmggrr ppiicckkss uupp tthhee mmeessssaaggee
Whenever nqmgr moves a queue file into the active queue, the following happens:
It reads all necessary information from the queue file as oqmgr does, and also
reads as many recipients as possible - more on that later, for now let's just
pretend it always reads all recipients.
Then it resolves the recipients as oqmgr does, which means obtaining (address,
nexthop, transport) triple for each recipient. For each triple, it finds the
transport; if it does not exist yet, it instantiates it (unless it's dead).
Within the transport, it finds the destination queue for given nexthop; if it
does not exist yet, it instantiates it (unless it's dead). The triple is then
bound to given destination queue. This happens in qmgr_resolve() and is
basically the same as in oqmgr.
Then for each triple which was bound to some queue (and thus transport), the
program finds the job which represents the message within that transport's
context; if it does not exist yet, it instantiates it. Within the job, it finds
the peer which represents the bound destination queue within this jobs context;
if it does not exist yet, it instantiates it. Finally, it stores the address
from the resolved triple to the recipient entry which is appended to both the
queue entry list and the peer entry list. The addresses for same nexthop are
batched in the entries up to recipient_concurrency limit for that transport.
This happens in qmgr_assign() and apart from that it operates with job and peer
structures it is basically the same as in oqmgr.
When the job is instantiated, it is enqueued on the transport's job list based
on the time its message was picked up by nqmgr. For first batch of recipients
this means it is appended to the end of the job list, but the ordering of the
job list by the enqueue time is important as we will see shortly.
[Now you should have pretty good idea what is the state of the nqmgr after
couple of messages was picked up, what is the relation between all those job,
peer, queue and entry structures.]
HHooww ddooeess tthhee eennttrryy sseelleeccttiioonn wwoorrkk
Having prepared all those above mentioned structures, the task of the nqmgr's
scheduler is to choose the recipient entries one at a time and pass them to the
delivery agent for corresponding transport. Now how does this work?
The first approximation of the new scheduling algorithm is like this:
foreach transport (round-robin-by-transport)
do
if transport busy continue
if transport process limit reached continue
foreach transport's job (in the order of the transport's job list)
do
foreach job's peer (round-robin-by-destination)
if peer->queue->concurrency < peer->queue->window
return next peer entry.
done
done
done
Now what is the "order of the transport's job list"? As we know already, the
job list is by default kept in the order the message was picked up by the
nqmgr. So by default we get the top-level round-robin transport, and within
each transport we get the FIFO message delivery. The round-robin of the peers
by the destination is perhaps of little importance in most real-life cases
(unless the recipient_concurrency limit is reached, in one job there is only
one peer structure for each destination), but theoretically it makes sure that
even within single jobs, destinations are treated fairly.
[By now you should have a feeling you really know how the scheduler works,
except for the preemption, under ideal conditions - that is, no recipient
resource limits and no destination concurrency problems.]
HHooww ddooeess tthhee pprreeeemmppttiioonn wwoorrkk
As you might perhaps expect by now, the transport's job list does not remain
sorted by the job's message enqueue time all the time. The most cool thing
about nqmgr is not the simple FIFO delivery, but that it is able to slip mail
with little recipients past the mailing-list bulk mail. This is what the job
preemption is about - shuffling the jobs on the transport's job list to get the
best message delivery rates. Now how is it achieved?
First I have to tell you that there are in fact two job lists in each
transport. One is the scheduler's job list, which the scheduler is free to play
with, while the other one keeps the jobs always listed in the order of the
enqueue time and is used for recipient pool management we will discuss later.
For now, we will deal with the scheduler's job list only.
So, we have the job list, which is first ordered by the time the jobs' messages
were enqueued, oldest messages first, the most recently picked one at the end.
For now, let's assume that there are no destination concurrency problems.
Without preemption, we pick some entry of the first (oldest) job on the queue,
assign it to delivery agent, pick another one from the same job, assign it
again, and so on, until all the entries are used and the job is delivered. We
would then move onto the next job and so on and on. Now how do we manage to
sneak in some entries from the recently added jobs when the first job on the
job list belongs to a message going to the mailing-list and has thousands of
recipient entries?
The nqmgr's answer is that we can artificially "inflate" the delivery time of
that first job by some constant for free - it is basically the same trick you
might remember as "accumulation of potential" from the amortized complexity
lessons. For example, instead of delivering the entries of the first job on the
job list every time a delivery agent becomes available, we can do it only every
second time. If you view the moments the delivery agent becomes available on a
timeline as "delivery slots", then instead of using every delivery slot for the
first job, we can use only every other slot, and still the overall delivery
efficiency of the first job remains the same. So the delivery 11112222 becomes
1.1.1.1.2.2.2.2 (1 and 2 are the imaginary job numbers, . denotes the free
slot). Now what do we do with free slots?
As you might have guessed, we will use them for sneaking the mail with little
recipients in. For example, if we have one four-recipient mail followed by four
one recipients mail, the delivery sequence (that is, the sequence in which the
jobs are assigned to the delivery slots) might look like this: 12131415. Hmm,
fine for sneaking in the single recipient mail, but how do we sneak in the mail
with more than one recipient? Say if we have one four-recipient mail followed
by two two-recipient mails?
The simple answer would be to use delivery sequence 12121313. But the problem
is that this does not scale well. Imagine you have mail with thousand
recipients followed by mail with hundred recipients. It is tempting to suggest
the delivery sequence like 121212...., but alas! Imagine there arrives another
mail with say ten recipients. But there are no free slots anymore, so it can't
slip by, not even if it had just only one recipients. It will be stuck until
the hundred-recipient mail is delivered, which really sucks.
So, it becomes obvious that while inflating the message to get free slots is
great idea, one has to be really careful of how the free slots are assigned,
otherwise one might corner himself. So, how does nqmgr really use the free
slots?
The key idea is that one does not have to generate the free slots in a uniform
way. The delivery sequence 111...1 is no worse than 1.1.1.1, in fact, it is
even better as some entries are in the first case selected earlier than in the
second case, and none is selected later! So it is possible to first
"accumulate" the free delivery slots and then use them all at once. It is even
possible to accumulate some, then use them, then accumulate some more and use
them again, as in 11..1.1 .
Let's get back to the one hundred recipient example. We now know that we could
first accumulate one hundred free slots, and only after then to preempt the
first job and sneak the one hundred recipient mail in. Applying the algorithm
recursively, we see the hundred recipient job can accumulate ten free delivery
slots, and then we could preempt it and sneak in the ten-recipient mail... Wait
wait wait! Could we? Aren't we overinflating the original one thousand
recipient mail?
Well, despite it looks so at the first glance, another trick will allow us to
answer "no, we are not!". If we had said that we will inflate the delivery time
twice at maximum, and then we consider every other slot as a free slot, then we
would overinflate in case of the recursive preemption. BUT! The trick is that
if we use only every n-th slot as a free slot for n>2, there is always some
worst inflation factor which we can guarantee not to be breached, even if we
apply the algorithm recursively. To be precise, if for every k>1 normally used
slots we accumulate one free delivery slot, than the inflation factor is not
worse than k/(k-1) no matter how many recursive preemptions happen. And it's
not worse than (k+1)/k if only non-recursive preemption happens. Now, having
got through the theory and the related math, let's see how nqmgr implements
this.
Each job has so called "available delivery slot" counter. Each transport has a
transport_delivery_slot_cost parameter, which defaults to
default_delivery_slot_cost parameter which is set to 5 by default. This is the
k from the paragraph above. Each time k entries of the job are selected for
delivery, this counter is incremented by one. Once there are some slots
accumulated, job which requires no more than that amount of slots to be fully
delivered can preempt this job.
[Well, the truth is, the counter is incremented every time an entry is selected
and it is divided by k when it is used. Or even more true, there is no
division, the other side of the equation is multiplied by k. But for the
understanding it's good enough to use the above approximation of the truth.]
OK, so now we know the conditions which must be satisfied so one job can
preempt another one. But what job gets preempted, how do we choose what job
preempts it if there are several valid candidates, and when does all this
exactly happen?
The answer for the first part is simple. The job whose entry was selected the
last time is so called current job. Normally, it is the first job on the
scheduler's job list, but destination concurrency limits may change this as we
will see later. It is always only the current job which may get preempted.
Now for the second part. The current job has certain amount of recipient
entries, and as such may accumulate at maximum some amount of available
delivery slots. It might have already accumulated some, and perhaps even
already used some when it was preempted before (remember a job can be preempted
several times). In either case, we know how many are accumulated and how many
are left to deliver, so we know how many it may yet accumulate at maximum.
Every other job which may be delivered by less than that amount of slots is a
valid candidate for preemption. How do we choose among them?
The answer is - the one with maximum enqueue_time/recipient_entry_count. That
is, the older the job is, the more we should try to deliver it in order to get
best message delivery rates. These rates are of course subject to how many
recipients the message has, therefore the division by the recipient (entry)
count. No one shall be surprised that message with n recipients takes n times
longer to deliver than message with one recipient.
Now let's recap the previous two paragraphs. Isn't it too complicated? Why
don't the candidates come only among the jobs which can be delivered within the
amount of slots the current job already accumulated? Why do we need to estimate
how much it has yet to accumulate? If you found out the answer, congratulate
yourself. If we did it this simple way, we would always choose the candidate
with least recipient entries. If there were enough single recipient mails
coming in, they would always slip by the bulk mail as soon as possible, and the
two and more recipients mail would never get a chance, no matter how long they
have been sitting around in the job list.
This candidate selection has interesting implication - that when we choose the
best candidate for preemption (this is done in qmgr_choose_candidate()), it may
happen that we may not use it for preemption immediately. This leads to an
answer to the last part of the original question - when does the preemption
happen?
The preemption attempt happens every time next transport's recipient entry is
to be chosen for delivery. To avoid needless overhead, the preemption is not
attempted if the current job could never accumulate more than
transport_minimum_delivery_slots (defaults to default_minimum_delivery_slots
which defaults to 3). If there is already enough accumulated slots to preempt
the current job by the chosen best candidate, it is done immediately. This
basically means that the candidate is moved in front of the current job on the
scheduler's job list and decreasing the accumulated slot counter by the amount
used by the candidate. If there is not enough slots... well, I could say that
nothing happens and the another preemption is attempted the next time. But
that's not the complete truth.
The truth is that it turns out that it is not really necessary to wait until
the jobs counter accumulates all the delivery slots in advance. Say we have
ten-recipient mail followed by two two-recipient mails. If the preemption
happened when enough delivery slot accumulate (assuming slot cost 2), the
delivery sequence becomes 11112211113311. Now what would we get if we would
wait only for 50% of the necessary slots to accumulate and we promise we would
wait for the remaining 50% later, after we get back to the preempted job? If we
use such slot loan, the delivery sequence becomes 11221111331111. As we can
see, it makes it no considerably worse for the delivery of the ten-recipient
mail, but it allows the small messages to be delivered sooner.
The concept of these slot loans is where the transport_delivery_slot_discount
and transport_delivery_slot_loan come from (they default to
default_delivery_slot_discount and default_delivery_slot_loan, whose values are
by default 50 and 3, respectively). The discount (resp. loan) specifies how
many percent (resp. how many slots) one "gets in advance", when the amount of
slots required to deliver the best candidate is compared with the amount of
slots the current slot had accumulated so far.
And it pretty much concludes this chapter.
[Now you should have a feeling that you pretty much understand the scheduler
and the preemption, or at least that you will have it after you read the last
chapter couple more times. You shall clearly see the job list and the
preemption happening at its head, in ideal delivery conditions. The feeling of
understanding shall last until you start wondering what happens if some of the
jobs are blocked, which you might eventually figure out correctly from what had
been said already. But I would be surprised if your mental image of the
scheduler's functionality is not completely shattered once you start wondering
how it works when not all recipients may be read in-core. More on that later.]
HHooww ddeessttiinnaattiioonn ccoonnccuurrrreennccyy lliimmiittss aaffffeecctt tthhee sscchheedduulliinngg aallggoorriitthhmm
The nqmgr uses the same algorithm for destination concurrency control as oqmgr.
Now what happens when the destination limits are reached and no more entries
for that destination may be selected by the scheduler?
From user's point of view it is all simple. If some of the peers of a job can't
be selected, those peers are simply skipped by the entry selection algorithm
(the pseudo-code described before) and only the selectable ones are used. If
none of the peers may be selected, the job is declared a "blocker job". Blocker
jobs are skipped by the entry selection algorithm and they are also excluded
from the candidates for preemption of current job. Thus the scheduler
effectively behaves as if the blocker jobs didn't exist on the job list at all.
As soon as at least one of the peers of a blocker job becomes unblocked (that
is, the delivery agent handling the delivery of the recipient entry for given
destination successfully finishes), the job's blocker status is removed and the
job again participates in all further scheduler actions normally.
So the summary is that the users don't really have to be concerned about the
interaction of the destination limits and scheduling algorithm. It works well
on its own and there are no knobs they would need to control it.
From a programmer's point of view, the blocker jobs complicate the scheduler
quite a lot. Without them, the jobs on the job list would be normally delivered
in strict FIFO order. If the current job is preempted, the job preempting it is
completely delivered unless it is preempted itself. Without blockers, the
current job is thus always either the first job on the job list, or the top of
the stack of jobs preempting the first job on the job list.
The visualization of the job list and the preemption stack without blockers
would be like this:
first job-> 1--2--3--5--6--8--... <- job list
on job list |
4 <- preemption stack
|
current job-> 7
In the example above we see that job 1 was preempted by job 4 and then job 4
was preempted by job 7. After job 7 is completed, remaining entries of job 4
are selected, and once they are all selected, job 1 continues.
As we see, it's all very clean and straightforward. Now how does this change
because of blockers?
The answer is: a lot. Any job may become blocker job at any time, and also
become normal job again at any time. This has several important implications:
1. The jobs may be completed in arbitrary order. For example, in the example
above, if the current job 7 becomes blocked, the next job 4 may complete
before the job 7 becomes unblocked again. Or if both 7 and 4 are blocked,
then 1 is completed, then 7 becomes unblocked and is completed, then 2 is
completed and only after that 4 becomes unblocked and is completed... You
get the idea.
[Interesting side note: even when jobs are delivered out of order, from
single destination's point of view the jobs are still delivered in the
expected order (that is, FIFO unless there was some preemption involved).
This is because whenever a destination queue becomes unblocked (the
destination limit allows selection of more recipient entries for that
destination), all jobs which have peers for that destination are unblocked
at once.]
2. The idea of the preemption stack at the head of the job list is gone. That
is, it must be possible to preempt any job on the job list. For example, if
the jobs 7, 4, 1 and 2 in the example above become all blocked, job 3
becomes the current job. And of course we do not want the preemption to be
affected by the fact that there are some blocked jobs or not. Therefore, if
it turns out that job 3 might be preempted by job 6, the implementation
shall make it possible.
3. The idea of the linear preemption stack itself is gone. It's no longer true
that one job is always preempted by only one job at one time (that is
directly preempted, not counting the recursively nested jobs). For example,
in the example above, job 1 is directly preempted by only job 4, and job 4
by job 7. Now assume job 7 becomes blocked, and job 4 is being delivered.
If it accumulates enough delivery slots, it is natural that it might be
preempted for example by job 8. Now job 4 is preempted by both job 7 AND
job 8 at the same time.
Now combine the points 2) and 3) with point 1) again and you realize that the
relations on the once linear job list became pretty complicated. If we extend
the point 3) example: jobs 7 and 8 preempt job 4, now job 8 becomes blocked
too, then job 4 completes. Tricky, huh?
If I illustrate the relations after the above mentioned examples (but those in
point 1)), the situation would look like this:
v- parent
adoptive parent -> 1--2--3--5--... <- "stack" level 0
| |
parent gone -> ? 6 <- "stack" level 1
/ \
children -> 7 8 ^- child <- "stack" level 2
^- siblings
Now how does nqmgr deal with all these complicated relations?
Well, it maintains them all as described, but fortunately, all these relations
are necessary only for purposes of proper counting of available delivery slots.
For purposes of ordering the jobs for entry selection, the original rule still
applies: "the job preempting the current job is moved in front of the current
job on the job list". So for entry selection purposes, the job relations remain
as simple as this:
7--8--1--2--6--3--5--.. <- scheduler's job list order
The job list order and the preemption parent/child/siblings relations are
maintained separately. And because the selection works only with the job list,
you can happily forget about those complicated relations unless you want to
study the nqmgr sources. In that case the text above might provide some helpful
introduction to the problem domain. Otherwise I suggest you just forget about
all this and stick with the user's point of view: the blocker jobs are simply
ignored.
[By now, you should have a feeling that there is more things going under the
hood than you ever wanted to know. You decide that forgetting about this
chapter is the best you can do for the sake of your mind's health and you
basically stick with the idea how the scheduler works in ideal conditions, when
there are no blockers, which is good enough.]
DDeeaalliinngg wwiitthh mmeemmoorryy rreessoouurrccee lliimmiittss
When discussing the nqmgr scheduler, we have so far assumed that all recipients
of all messages in the active queue are completely read into the memory. This
is simply not true. There is an upper bound on the amount of memory the nqmgr
may use, and therefore it must impose some limits on the information it may
store in the memory at any given time.
First of all, not all messages may be read in-core at once. At any time, only
qmgr_message_active_limit messages may be read in-core at maximum. When read
into memory, the messages are picked from the incoming and deferred message
queues and moved to the active queue (incoming having priority), so if there is
more than qmgr_message_active_limit messages queued in the active queue, the
rest will have to wait until (some of) the messages in the active queue are
completely delivered (or deferred).
Even with the limited amount of in-core messages, there is another limit which
must be imposed in order to avoid memory exhaustion. Each message may contain
huge amount of recipients (tens or hundreds of thousands are not uncommon), so
if nqmgr read all recipients of all messages in the active queue, it may easily
run out of memory. Therefore there must be some upper bound on the amount of
message recipients which are read into the memory at the same time.
Before discussing how exactly nqmgr implements the recipient limits, let's see
how the sole existence of the limits themselves affects the nqmgr and its
scheduler.
The message limit is straightforward - it just limits the size of the lookahead
the nqmgr's scheduler has when choosing which message can preempt the current
one. Messages not in the active queue simply are not considered at all.
The recipient limit complicates more things. First of all, the message reading
code must support reading the recipients in batches, which among other things
means accessing the queue file several times and continuing where the last
recipient batch ended. This is invoked by the scheduler whenever the current
job has space for more recipients, subject to transport's refill_limit and
refill_delay parameters. It is also done any time when all in-core recipients
of the message are dealt with (which may also mean they were deferred) but
there are still more in the queue file.
The second complication is that with some recipients left unread in the queue
file, the scheduler can't operate with exact counts of recipient entries. With
unread recipients, it is not clear how many recipient entries there will be, as
they are subject to per-destination grouping. It is not even clear to what
transports (and thus jobs) the recipients will be assigned. And with messages
coming from the deferred queue, it is not even clear how many unread recipients
are still to be delivered. This all means that the scheduler must use only
estimates of how many recipients entries there will be. Fortunately, it is
possible to estimate the minimum and maximum correctly, so the scheduler can
always err on the safe side. Obviously, the better the estimates, the better
results, so it is best when we are able to read all recipients in-core and turn
the estimates into exact counts, or at least try to read as many as possible to
make the estimates as accurate as possible.
The third complication is that it is no longer true that the scheduler is done
with a job once all of its in-core recipients are delivered. It is possible
that the job will be revived later, when another batch of recipients is read in
core. It is also possible that some jobs will be created for the first time
long after the first batch of recipients was read in core. The nqmgr code must
be ready to handle all such situations.
And finally, the fourth complication is that the nqmgr code must somehow impose
the recipient limit itself. Now how does it achieve it?
Perhaps the easiest solution would be to say that each message may have at
maximum X recipients stored in-core, but such solution would be poor for
several reasons. With reasonable qmgr_message_active_limit values, the X would
have to be quite low to maintain reasonable memory footprint. And with low X
lots of things would not work well. The nqmgr would have problems to use the
transport_destination_recipient_limit efficiently. The scheduler's preemption
would be suboptimal as the recipient count estimates would be inaccurate. The
message queue file would have to be accessed many times to read in more
recipients again and again.
Therefore it seems reasonable to have a solution which does not use a limit
imposed on per-message basis, but which maintains a pool of available recipient
slots, which can be shared among all messages in the most efficient manner. And
as we do not want separate transports to compete for resources whenever
possible, it seems appropriate to maintain such recipient pool for each
transport separately. This is the general idea, now how does it work in
practice?
First we have to solve little chicken-and-egg problem. If we want to use the
per-transport recipient pools, we first need to know to what transport(s) is
the message assigned. But we will find that out only after we read in the
recipients first. So it is obvious that we first have to read in some
recipients, use them to find out to what transports is the message to be
assigned, and only after that we can use the per-transport recipient pools.
Now how many recipients shall we read for the first time? This is what
qmgr_message_recipient_minimum and qmgr_message_recipient_limit values control.
The qmgr_message_recipient_minimum value specifies how many recipients of each
message we will read for the first time, no matter what. It is necessary to
read at least one recipient before we can assign the message to a transport and
create the first job. However, reading only qmgr_message_recipient_minimum
recipients even if there are only few messages with few recipients in-core
would be wasteful. Therefore if there is less than qmgr_message_recipient_limit
recipients in-core so far, the first batch of recipients may be larger than
qmgr_message_recipient_minimum - as large as is required to reach the
qmgr_message_recipient_limit limit.
Once the first batch of recipients was read in core and the message jobs were
created, the size of the subsequent recipient batches (if any - of course it's
best when all recipients are read in one batch) is based solely on the position
of the message jobs on their corresponding transports' job lists. Each
transport has a pool of transport_recipient_limit recipient slots which it can
distribute among its jobs (how this is done is described later). The subsequent
recipient batch may be as large as the sum of all recipient slots of all jobs
of the message permits (plus the qmgr_message_recipient_minimum amount which
always applies).
For example, if a message has three jobs, first with 1 recipient still in-core
and 4 recipient slots, second with 5 recipient in-core and 5 recipient slots,
and third with 2 recipients in-core and 0 recipient slots, it has 1+5+2=7
recipients in-core and 4+5+0=9 jobs' recipients slots in total. This means that
we could immediately read 2+qmgr_message_recipient_minimum more recipients of
that message in core.
The above example illustrates several things which might be worth mentioning
explicitly: first, note that although the per-transport slots are assigned to
particular jobs, we can't guarantee that once the next batch of recipients is
read in core, that the corresponding amounts of recipients will be assigned to
those jobs. The jobs lend its slots to the message as a whole, so it is
possible that some jobs end up sponsoring other jobs of their message. For
example, if in the example above the 2 newly read recipients were assigned to
the second job, the first job sponsored the second job with 2 slots. The second
notable thing is the third job, which has more recipients in-core than it has
slots. Apart from sponsoring by other job we just saw it can be result of the
first recipient batch, which is sponsored from global recipient pool of
qmgr_message_recipient_limit recipients. It can be also sponsored from the
message recipient pool of qmgr_message_recipient_minimum recipients.
Now how does each transport distribute the recipient slots among its jobs? The
strategy is quite simple. As most scheduler activity happens on the head of the
job list, it is our intention to make sure that the scheduler has the best
estimates of the recipient counts for those jobs. As we mentioned above, this
means that we want to try to make sure that the messages of those jobs have all
recipients read in-core. Therefore the transport distributes the slots "along"
the job list from start to end. In this case the job list sorted by message
enqueue time is used, because it doesn't change over time as the scheduler's
job list does.
More specifically, each time a job is created and appended to the job list, it
gets all unused recipient slots from its transport's pool. It keeps them until
all recipients of its message are read. When this happens, all unused recipient
slots are transferred to the next job (which is now in fact now first such job)
on the job list which still has some recipients unread, or eventually back to
the transport pool if there is no such job. Such transfer then also happens
whenever a recipient entry of that job is delivered.
There is also a scenario when a job is not appended to the end of the job list
(for example it was created as a result of second or later recipient batch).
Then it works exactly as above, except that if it was put in front of the first
unread job (that is, the job of a message which still has some unread
recipients in queue file), that job is first forced to return all of its unused
recipient slots to the transport pool.
The algorithm just described leads to the following state: The first unread job
on the job list always gets all the remaining recipient slots of that transport
(if there are any). The jobs queued before this job are completely read (that
is, all recipients of their message were already read in core) and have at
maximum as many slots as they still have recipients in-core (the maximum is
there because of the sponsoring mentioned before) and the jobs after this job
get nothing from the transport recipient pool (unless they got something before
and then the first unread job was created and enqueued in front of them later -
in such case the also get at maximum as many slots as they have recipients in-
core).
Things work fine in such state for most of the time, because the current job is
either completely read in-core or has as much recipient slots as there are, but
there is one situation which we still have to take care of specially. Imagine
if the current job is preempted by some unread job from the job list and there
are no more recipient slots available, so this new current job could read only
batches of qmgr_message_recipient_minimum recipients at a time. This would
really degrade performance. For this reason, each transport has extra pool of
transport_extra_recipient_limit recipient slots, dedicated exactly for this
situation. Each time an unread job preempts the current job, it gets half of
the remaining recipient slots from the normal pool and this extra pool.
And that's it. It sure does sound pretty complicated, but fortunately most
people don't really have to care how exactly it works as long as it works.
Perhaps the only important things to know for most people are the following
upper bound formulas:
Each transport has at maximum
max(
qmgr_message_recipient_minimum * qmgr_message_active_limit
+ *_recipient_limit + *_extra_recipient_limit,
qmgr_message_recipient_limit
)
recipients in core.
The total amount of recipients in core is
max(
qmgr_message_recipient_minimum * qmgr_message_active_limit
+ sum( *_recipient_limit + *_extra_recipient_limit ),
qmgr_message_recipient_limit
)
where the sum is over all used transports.
And this terribly complicated chapter concludes the documentation of nqmgr
scheduler.
[By now you should theoretically know the nqmgr scheduler inside out. In
practice, you still hope that you will never have to really understand the last
or last two chapters completely, and fortunately most people really won't.
Understanding how the scheduler works in ideal conditions is more than good
enough for vast majority of users.]
CCrreeddiittss
* Wietse Venema designed and implemented the initial queue manager with per-
domain FIFO scheduling, and per-delivery +/-1 concurrency feedback.
* Patrik Rak designed and implemented preemption where mail with fewer
recipients can slip past mail with more recipients in a controlled manner,
and wrote up its documentation.
* Wietse Venema initiated a discussion with Patrik Rak and Victor Duchovni on
alternatives for the +/-1 feedback scheduler's aggressive behavior. This is
when K/N feedback was reviewed (N = concurrency). The discussion ended
without a good solution for both negative feedback and dead site detection.
* Victor Duchovni resumed work on concurrency feedback in the context of
concurrency-limited servers.
* Wietse Venema then re-designed the concurrency scheduler in terms of the
simplest possible concepts: less-than-1 concurrency feedback per delivery,
forward and reverse concurrency feedback hysteresis, and pseudo-cohort
failure. At this same time, concurrency feedback was separated from dead
site detection.
* These simplifications, and their modular implementation, helped to develop
further insights into the different roles that positive and negative
concurrency feedback play, and helped to identify some worst-case
scenarios.