MesgApi Details
Contents |
[edit] Implementation Details: performance tuning and miscellany
Introduction | Spread Toolkit | Examples | Download | Install | API Reference | Comparison | Details
[edit] Connection Pools
[edit] The send pool
It would be possible for all send_mesg() calls to share a single connection to the local Spread daemon. However, that connection would become a performance bottleneck, as all sending threads would have to serialize behind a mutex that protects the socket descriptor in the underlying send() system call. Instead, the message API opens up a small pool of Spread connections, the send pool, and attempts to distribute send_mesg() calls evenly across the pool.
The send pool uses 8 Spread connections by default. Its size can be configured at compile-time. The Message API is designed around the goal that its concurrency should be tied to the size of the send pool. Whenever you are not able to send messages fast enough, it should be possible to send more messages in the same period of time after increasing the size of the send pool.
[edit] The receive pool
Each call to join_mesg_group() or track_memberships() (e.g., each message handle visible to a user) uses a dedicated Spread connection. The total number of these connections that may be open at one time is limited by the size of the receive pool. The default size of the receive pool is 56, and may be configured at compile-time.
[edit] Guaranteed Messages
Many of the data structures inside the Message API deal with "guaranteed" messages -- messages sent with a particular recipient specified in the third argument of a send_mesg() call to a group that is being watched by track_memberships().
[edit] The memberships thread
A call to track_memberships() launches a background thread which joins a group, keeps a record of the group's membership over time, and communicates with senders whenever guaranteed messages are received. This memberships thread tracks a group's member list, so it is able to inform the sender whether the intended recipient is or is not a group member at the time the message is received.
[edit] Outboxes
When a MySQL query sends a guaranteed message, it uses a connection from the send pool, where a small data structure called an outbox helps to coordinate delivery confirmation with the memberships thread. If a sender runs out of available outboxes, send_mesg() calls will begin to fail and return NULL. By default, each sender has 150 outboxes. The maximum number of outboxes ever in use appears in the MaxQ column of mesg_status(). Information about outboxes currently in use (if any) appears in the "Outboxes" section of the status report. Under normal conditions, MaxQ is likely to be zero or very small, and the Outboxes section of the report is likely to be empty, but during a network transition or outage the outboxes may fill up. The number of outboxes per sender can be configured at compile-time.
[edit] Network Transitions
Suppose that a network link goes down, or a machine crashes, and disrupts the topology of the Spread network. The ensuing action will probably unfold along these lines:
- The memberships thread will receive a TRANSITIONAL MEMBERSHIP message from Spread, indicating that group membership is in flux.
- Normal (two-argument) send_mesg() calls will continue to send messages; recv_mesg() calls will continue to receive messages; but three-argument guaranteed send_mesg() calls will start to hang. Since the memberships thread no longer has a confirmed member list, it cannot "sign off" on guaranteed message delivery.
- The remaining Spread daemons go into "high gear," trying to determine whether the lost members are truly gone and to confirm a new membership list. You can often observe increased CPU utilization during this period, which may last for 5 to 30 seconds (depending on the complexity of the network.)
- Once Spread has identified the fault and recovered the network, it will send the membersips thread a new member list that is retroactive to the time of the TRANSITIONAL message. Using this list, the guaranteed send_mesg() calls will at last be able to return. Messages that were sent to group members who were on the machine or network segment that crashed will return 0, and those sent to members who remained present will return 1.
[edit] Compile-Time Configuration
All of the values mentioned here that are configurable at compile-time can be edited in the udf_spread.h file.
Spread itself is also has some possible compile-time configuration. The MAX_SESSION_MESSAGES constant defined in spread.h, for example, determines how many messages Spread will queue up for a slow message receiver before it gives up on the receiver and disconnects it. (If you edit Spread.h and recompile Spread, don't forget to also recompile and reinstall the Message API).
For more information, contact John David Duncan, jdd@mysql.com
This product uses software developed by Spread Concepts LLC for use in the Spread toolkit. For more information about Spread see http://www.spread.org