MySQL Internals Test Synchronization
← Back to MySQL Internals overview page
Contents |
[edit] Test Synchronization
There is a class of problems that require two or more cooperating threads to reproduce them.
A subclass of these problems is known as "race conditions". They require one thread to execute a certain piece of code while another thread executes another certain piece of code.
The vast majority of race conditions cannot be repeated reliably without some sort of synchronization of the involved threads. In most cases it is unlikely that the threads run through these code pieces at the right time. In this context 'synchronization' means to force the threads to meet at the critical code places.
In this chapter I'll describe some synchronization mechanisms:
- Sleep
- Wait Condition
- Dbug Sleep
- Error Injection
- User-Level Locks
- Debug Sync Point
- Backup Breakpoint
- Debug Sync Facility
[edit] Sleep
In some cases race conditions can be repeated when all but one thread are blocked (for example waiting for an SQL lock). Then the remaining thread has plenty of time to go through the critical piece of code.
The problem here is to assure that the blocking threads run until they reach their blocking point before the remaining thread reaches the critical code.
One solution is to use the 'sleep' command of 'mysqltest' in front of the SQL statement that drives the remaining thread into the critical code.
Example:
--connection conn1
LOCK TABLE t1 WRITE;
--connection conn2
# This will block in wait_for_lock().
send INSERT INTO t1 VALUES (1);
--connection conn1
# Sleep until we can be sure that conn2 reached wait_for_lock().
--sleep 2
# Run through the critical code.
FLUSH TABLE t1;
The BIG, BIG problem with 'sleep' is that you need to specify a fixed time. It must be big enough so that the test works as intended even on a very slow machine that is under heavy load. Hence it is much too big for the average machine. A major waste of time.
The bottom line is: AVOID 'SLEEP' WHEREVER POSSIBLE.
[edit] Wait Condition
Like 'sleep', this method can also be used, when all but one thread reach a blocked state.
If you are able to detect that the threads are in their blocked state by using SQL statements, then you can use this method. The remaining thread runs the statement(s) until the expected result is returned. Then it continues with the test.
Example:
--connection conn1
LOCK TABLE t1 WRITE;
--connection conn2
# Get the id of this thread.
let $conn2_id= `SELECT CONNECTION_ID()`;
# This will block in wait_for_lock().
send INSERT INTO t1 VALUES (1);
--connection conn1
# Specify the condition that shows if conn2 reached wait_for_lock().
let $wait_condition= SELECT 1 FROM INFORMATION_SCHEMA.PROCESSLIST
WHERE ID = $conn2_id AND STATE = 'Locked';
# Run the condition in a loop until it becomes true.
--source include/wait_condition.inc
# Run through the critical code.
FLUSH TABLE t1;
In conn2 we get the thread ID first. In conn1 we use a SELECT statement that returns '1' when the processlist shows that conn2 reached the 'Locked' state. With this setup we call the wait_condition method. It runs the statement and checks the result. If the condition is not met, it sleeps for 0.1 second and retries.
The maximum waste of time is 0.1 seconds. This is much better than the 'sleep' method, but could still waste a little time.
Another problem is that the condition could be "fuzzy" in some situations. In the example above, the thread state (proc_info) is set to "Locked" right before the locking function is called. In theory it could happen that conn1 continues before conn2 did acquire the lock. The test would then fail to repeat what it was intended to do.
The "Debug Sync Facility" should be able to replace most of the "wait condition" uses.
[edit] Dbug Sleep
In cases where the normal server code does not have a block point at the critical place, one can insert an artificial synchronization point.
open_tables(...)
DBUG_EXECUTE_IF("sleep_open_and_lock_after_open", {
const char *old_proc_info= thd->proc_info;
thd->proc_info= "DBUG sleep";
my_sleep(6000000);
thd->proc_info= old_proc_info;});
lock_tables(...)
In this case, if the 'debug' keyword 'sleep_open_and_lock_after_open' is set, a thread sleeps for 6 seconds after open_tables() and before lock_tables(). Before sleeping, it sets the thread state (proc_info) to 'DBUG sleep'. The test file that uses this synchronization point looks like so:
--connection conn1
let $conn1_id= `SELECT CONNECTION_ID()`;
# System variable 'debug' exists only in debug servers
--error 0, ER_UNKNOWN_SYSTEM_VARIABLE
SET SESSION debug="+d,sleep_open_and_lock_after_open";
send INSERT INTO t1 VALUES (1);
--connection conn2
# Specify the condition that shows if conn1 reached the sync point.
let $wait_condition= SELECT 1 FROM INFORMATION_SCHEMA.PROCESSLIST
WHERE ID = $conn1_id AND STATE = 'DBUG sleep';
# Run the condition in a loop until it becomes true.
--source include/wait_condition.inc
# Run through the critical code.
FLUSH TABLE t1;
So one can add synchronization points almost everywhere. But only at the cost of the wasted time of a sleep + a wait condition.
This method requires that you modify and recompile the server code. Another problem is that the synchronization point does not exist in non-debug servers. Not even the system variable 'debug' exists in a non-debug server. Each test must be written so that it works on a debug server as well as on a non-debug server. If this is not possible, the test must be moved into a test file that includes 'have_debug.inc'. Setting the possibly not existing variable can be protected by the --error 0, ER_UNKNOWN_SYSTEM_VARIABLE command. It says that the next statement can either succeed (0) or fail (ER_UNKNOWN_SYSTEM_VARIABLE).
Finally the method is bad when the execution should be traced with the DBUG facility. Setting one (or more) 'debug' keywords disables all other keywords. One would need to add a pretty long list for a meaningful trace.
The bottom line is: Use the "Dbug Sleep" method when there is no other way to repeat a problem. However, the "Debug Sync Facility" should be able to replace all "Dbug Sleep" synchronization points.
[edit] Error Injection
Note: The ERROR_INJECT framework has been removed in an early 6.0 version. It may be added back later.
The error injection method is based on the DBUG framework just like the Dbug Sleep method. In the code you can use the following macros:
ERROR_INJECT_ACTION(keyword,action) ERROR_INJECT_CRASH(keyword) ERROR_INJECT(keyword) SET_ERROR_INJECT_VALUE(value) ERROR_INJECT_VALUE_ACTION(value,action) ERROR_INJECT_VALUE_CRASH(value) ERROR_INJECT_VALUE(value)
'keyword' is the debug keyword that you set in the test file with:
SET SESSION debug='+d,keyword1,keyword2,keyword3';
'value' is an unsigned long integer value. It is stored in THD::error_inject_value by SET_ERROR_INJECT_VALUE(value) and examined by the other *_VALUE* macros.
All of the ERROR_INJECT_* macros can/must be used in an expression. Their value is 0 (zero) in most cases. Exceptions are mentioned below.
Most of the ERROR_INJECT_* macros remove the keyword from the debug keyword list or clear THD::error_inject_value respectively before they executes their action. This means each of them will never execute twice within one SQL statement. But if multiple non-VALUE macros are run through in a statement, each can execute once if they use distinct keywords. There is just one THD::error_inject_value, not a list. So when any *_VALUE* macro clears it, all other *_VALUE* macros are disabled. Unless a new value is set by SET_ERROR_INJECT_VALUE somewhere. Obvious exceptions of keyword/value removal are SET_ERROR_INJECT_VALUE and the CRASH macros.
The ERROR_INJECT_ACTION macro is very similar to the DBUG_EXECUTE_IF macro (see the "Dbug Sleep section). But remember the removal of the keyword/value.
The ERROR_INJECT_VALUE_ACTION is similar to ERROR_INJECT_ACTION. But it is controlled by the thread local value set by SET_ERROR_INJECT_VALUE. Also the action must be written as an expression. You can call a function that returns a value valid in the expression in which ERROR_INJECT_VALUE_ACTION appears. But if you want to open a block "{...}" you need to make an expression from it: "({...}, 0)". Also, if ERROR_INJECT_VALUE_ACTION is executed, it returns the value that the 'action' expression returns, not just 0 (zero) like ERROR_INJECT_ACTION does.
ERROR_INJECT_CRASH and ERROR_INJECT_VALUE_CRASH are pretty self-explanatory.
ERROR_INJECT and ERROR_INJECT_VALUE are for expression evaluation. They return 1 if the keyword is set or the value matches THD::error_inject_value respectively. Otherwise 0.
SET_ERROR_INJECT_VALUE copies the argument to THD::error_inject_value.
Downsides: The error injection method is NOT enabled in the server by default. You need to ./configure --with-error-inject
The method is currently not used in the standard test suite anywhere. So you cannot copy and modify an example, but have to learn it the hard way.
When controlling error injection from the test files, explicit debug keywords are required, which has the same downsides as mentioned under Dbug Sleep.
If not using the ERROR_INJECT macros in an expression, expect the compiler warning "statement has no effect".
[edit] User-Level Locks
User-level locks are controlled with the SQL functions
GET_LOCK(str,timeout) IS_FREE_LOCK(str) IS_USED_LOCK(str) RELEASE_LOCK(str)
They can be used at places where SQL statements accept SQL functions. Depending on their appearance in the select list, the where clause, the group by clause, etc, of select, update or other statements, these statements can be blocked at different code points. The set of blockable places is limited. Nevertheless, a couple of synchronization problems can be solved with user-level locks.
Example:
# Using InnoDB table with innodb_lock_wait_timeout=1 second.
--connection conn1
# Take an share lock on t1.
LOCK TABLE t1 IN SHARE MODE;
--connection conn2
# Acquire the user level lock "mysqltest1".
SELECT GET_LOCK("mysqltest1", 10);
# INSERT must wait in background for the SQL lock on t1 to go away.
send INSERT INTO t1 VALUES (1);
--connection conn1
# Wait in background until the insert times out and releases the
# user level lock. conn1 will then own the lock.
send SELECT GET_LOCK("mysqltest1", 10);
--connection conn2
# Wait for INSERT to timeout.
--error ER_LOCK_WAIT_TIMEOUT
reap;
# Now let conn1 get the lock and continue.
SELECT RELEASE_LOCK("mysqltest1");
COMMIT;
--connection conn1
reap;
# We do not need the lock any more.
SELECT RELEASE_LOCK("mysqltest1");
# Commit releases the share lock on t1.
COMMIT;
A good article about possible uses of user-level locks is from Martin Friebe. MySQL Internals mailing list, 10 Dec 2007: http://lists.mysql.com/internals/35220
One limitation of user-level locks is that a thread can have one lock at a time only. This limits the method to relatively simple cases.
[edit] Debug Sync Point
Note: Debug Sync Points were based on user-level locks. They were part of the MySQL code until the 6.0.5 versions. Debug Sync Points have been removed from the code in favor of the Debug Sync Facility.
Debug Sync Points give user-level locks the ability to synchronize at arbitrary points in code.
open_tables(...)
DBUG_SYNC_POINT("debug_lock.after_open_tables", 10);
lock_tables(...)
The synchronization points behave similar to
RELEASE_LOCK(<whatever the thread has>); IS_FREE_LOCK(str) OR (GET_LOCK(str,timeout) AND RELEASE_LOCK(str))
This means that the synchronization point releases any lock that the thread may have, waits to acquire the lock if another thread has it, and releases it immediately. If the lock is free (not used by any thread), the synchronization point does nothing but release any user-level lock of the current thread.
So the idea of DBUG_SYNC_POINT is that it does nothing when the user-level lock is not in use by any thread, and does wait for it to become free when it is in use. That way you can block a thread at a synchronization point by acquiring the user-level lock and let it continue by releasing the lock.
This can be used as a "signal". The thread acquires a lock (the "signal" lock) and releases it implicitly when reaching the synchronization point. The other thread, which tried to get the "signal" lock after this thread, gets the lock at the same moment and can continue.
It can be used as a "wait". The other thread has the "synchronization point" lock ("debug_lock.after_open_tables" in this example) and this thread blocks on it in the synchronization point.
Unfortunately I was not able to figure out, how to use it for "signal" _plus_ "wait". While the other thread could have the "synchronization point" lock and this thread have the "signal" lock, and hence reaching the synchronization point would release the "signal" lock and wait on the "synchronization point" lock, the other thread would not be able to wait on the "signal" lock, because it has the "synchronization point" lock. A thread can have one user lock only. When the other thread tries to wait for the "signal" lock, it implicitly releases the "synchronization point" lock. This would be okay if one could be sure that this thread reached the synchronization point before the other thread releases the "synchronization point" lock. Otherwise no wait would happen at the synchronization point. The test would not test what it should test.
A possible workaround might be a third thread, which takes the "synchronization point" lock in the beginning and releases it at the right moment. But this could easily lead to a big number of threads for more complex situations. Tests using this method are likely to become ununderstandable.
It is probably a bug in the implementation that DBUG_SYNC_POINT releases any lock unconditionally. The method is not widely used. I found just one single use in sql_repl.cc. I guess lock releasing was added to prevent that a synchronization point could wait on the threads own lock. The behavior could be fixed easily if the method should find more use.
The DBUG_SYNC_POINT method is available in debug servers only. If it is used in the test suite, similar precautions for writing tests have to be taken as mentioned in the "Dbug Sleep" section.
[edit] Backup Breakpoint
Note: Backup Breakpoints were based on DBUG_SYNC_POINT. They were part of the MySQL code in some early 6.0 versions. Backup Breakpoints have been removed from the code in favor of the Debug Sync Facility.
open_tables(...)
BACKUP_BREAKPOINT("bp_after_open_tables");
lock_tables(...)
The BACKUP_BREAKPOINT macro consists basically of:
DBUG_EXECUTE_IF("backup_debug", DBUG_SYNC_POINT((S), 300))
Opportunities and downsides of the DBUG_SYNC_POINT method apply here too.
In addition we had the downside that DBUG tracing was hampered as explained in the "Dbug Sleep" section.
[edit] Debug Sync Facility
The Debug Sync Facility is available as of MySQL 5.1.41, 5.5.0, and 6.0.6. With a properly configured server (see #Debug Sync Activation/Deactivation), this facility allows placement of synchronization points in the server code by using the DEBUG_SYNC macro:
open_tables(...) DEBUG_SYNC(thd, "after_open_tables"); lock_tables(...)
When activated, a synchronization point can
- Emit a signal and/or
- Wait for a signal
Nomenclature:
- signal
- A value of a global variable that persists until overwritten by a new signal. The global variable can also be seen as a "signal post" or "flag mast". Then the signal is what is attached to the "signal post" or "flag mast".
- emit a signal
- Assign the value (the signal) to the global variable ("set a flag") and broadcast a global condition to wake those waiting for a signal.
- wait for a signal
- Loop over waiting for the global condition until the global value matches the wait-for signal.
By default, all synchronization points are inactive. They do nothing (except burn a couple of CPU cycles for checking if they are active).
A synchronization point becomes active when an action is requested for it. To do so, assign a value to the DEBUG_SYNC system variable:
SET DEBUG_SYNC= 'after_open_tables SIGNAL opened WAIT_FOR flushed';
This activates the synchronization point named 'after_open_tables'. The activation requests the synchronization point to emit the signal 'opened' and wait for another thread to emit the signal 'flushed' when the thread's execution runs through the synchronization point.
For every synchronization point there can be one action per thread only. Every thread can request multiple actions, but only one per synchronization point. In other words, a thread can activate multiple synchronization points.
Here is an example how to activate and use the synchronization points:
--connection conn1
SET DEBUG_SYNC= 'after_open_tables SIGNAL opened WAIT_FOR flushed';
send INSERT INTO t1 VALUES(1);
--connection conn2
SET DEBUG_SYNC= 'now WAIT_FOR opened';
SET DEBUG_SYNC= 'after_abort_locks SIGNAL flushed';
FLUSH TABLE t1;
When conn1 runs through the INSERT statement, it hits the synchronization point 'after_open_tables'. It notices that it is active and executes its action. It emits the signal 'opened' and waits for another thread to emit the signal 'flushed'.
conn2 waits immediately at the special synchronization point 'now' for another thread to emit the 'opened' signal.
A signal remains in effect until it is overwritten. If conn1 signals 'opened' before conn2 reaches 'now', conn2 will still find the 'opened' signal. It does not wait in this case.
When conn2 reaches 'after_abort_locks', it signals 'flushed', which lets conn1 awake.
Normally the activation of a synchronization point is cleared when it has been executed. Sometimes it is necessary to keep the synchronization point active for another execution. You can add an execute count to the action:
SET DEBUG_SYNC= 'name SIGNAL sig EXECUTE 3';
This sets the synchronization point's activation counter to 3. Each execution decrements the counter. After the third execution the synchronization point becomes inactive.
One of the primary goals of this facility is to eliminate sleeps from the test suite. In most cases it should be possible to rewrite test cases so that they do not need to sleep. (Note that Debug Sync can synchronize only multiple threads within a single process. It cannot synchronize multiple processes.) However, to support test development, and as a last resort, synchronization point waiting times out. There is a default timeout, but it can be overridden:
SET DEBUG_SYNC= 'name WAIT_FOR sig TIMEOUT 10 EXECUTE 2';
TIMEOUT 0 is special: If the signal is not present, the wait times out immediately.
If a wait timeout occurs (even on TIMEOUT 0), a warning is generated so that it shows up in the test result.
You can throw an error message and kill the query when a synchronization point is hit a certain number of times:
SET DEBUG_SYNC= 'name HIT_LIMIT 3';
Or combine it with signal and/or wait:
SET DEBUG_SYNC= 'name SIGNAL sig EXECUTE 2 HIT_LIMIT 3';
Here the first two hits emit the signal, the third hit returns the error message and kills the query.
For cases where you are not sure that an action is taken and thus cleared in any case, you can forcibly clear (deactivate) a synchronization point:
SET DEBUG_SYNC= 'name CLEAR';
If you want to clear all actions and clear the global signal, use:
SET DEBUG_SYNC= 'RESET';
This is the only way to reset the global signal to an empty string.
For testing of the facility itself you can execute a synchronization point just as if it had been hit:
SET DEBUG_SYNC= 'name TEST';
[edit] Formal Syntax for DEBUG_SYNC Values
The string to "assign" to the DEBUG_SYNC variable can contain:
{RESET |
<sync point name> TEST |
<sync point name> CLEAR |
<sync point name> {{SIGNAL <signal name> |
WAIT_FOR <signal name> [TIMEOUT <seconds>]}
[EXECUTE <count>] &| HIT_LIMIT <count>}
Here '&|' means 'and/or'. This means that one of the sections separated by '&|' must be present or both of them.
[edit] Debug Sync Activation/Deactivation
The Debug Sync facility is an optional part of the MySQL server. To cause Debug Sync to be compiled into the server, use the --enable-debug-sync option:
./configure --enable-debug-sync
Debug Sync is also compiled in if you configure with the --with-debug option (which implies --enable-debug-sync), unless you also use the --disable-debug-sync option.
The Debug Sync Facility, when compiled in, is disabled by default. To enable it, start mysqld with the --debug-sync-timeout[=N] option, where N is a timeout value greater than 0. N becomes the default timeout for the WAIT_FOR action of individual synchronization points. If N is 0, Debug Sync stays disabled. If the option is given without a value, the timeout is set to 300 seconds.
The DEBUG_SYNC system variable is the user interface to the Debug Sync facility. If Debug Sync is not compiled in, this variable is not available. If compiled in, the global DEBUG_SYNC value is read only and indicates whether the facility is enabled. By default, Debug Sync is disabled and the value of DEBUG_SYNC is "OFF". If the server is started with --debug-sync-timeout=N, where N is a timeout value greater than 0, Debug Sync is enabled and the value of DEBUG_SYNC is "ON - current signal" followed by the signal name. Also, N becomes the default timeout for individual synchronization points.
The session value can be read by any user and will have the same value as the global variable. The session value can be set by users that have the SUPER privilege to control synchronization points.
Setting the DEBUG_SYNC system variable requires the 'SUPER' privilege. You cannot read back the string that you assigned to the variable, unless you assign the value that the variable does already have. But that would give a parse error. A syntactically correct string is parsed into a debug synchronization point action and stored apart from the variable value.
The Debug Sync facility is enabled by default in the test suite, but can be disabled with:
mysql-test-run.pl ... --debug-sync-timeout=0 ...
Likewise, the default wait timeout can be set:
mysql-test-run.pl ... --debug-sync-timeout=10 ...
For test cases that require the Debug Sync facility, include the following line in the test case file:
--source include/have_debug_sync.inc
[edit] Debug Sync Implementation
Pseudo code for a synchronization point:
#define DEBUG_SYNC(thd, sync_point_name)
if (unlikely(opt_debug_sync_timeout))
debug_sync(thd, STRING_WITH_LEN(sync_point_name))
The synchronization point performs a binary search in a sorted array of actions for this thread.
The SET DEBUG_SYNC statement adds a requested action to the array or overwrites an existing action for the same synchronization point. When it adds a new action, the array is sorted again.
[edit] A typical synchronization pattern
There are quite a few places in MySQL, where we use a synchronization pattern like this:
pthread_mutex_lock(&mutex);
thd->enter_cond(&condition_variable, &mutex, new_message);
#if defined(ENABLE_DEBUG_SYNC)
if (!thd->killed && !end_of_wait_condition)
DEBUG_SYNC(thd, "sync_point_name");
#endif
while (!thd->killed && !end_of_wait_condition)
pthread_cond_wait(&condition_variable, &mutex);
thd->exit_cond(old_message);
Here some explanations:
thd->enter_cond() is used to register the condition variable and the mutex in thd->mysys_var. This is done to allow the thread to be interrupted (killed) from its sleep. Another thread can find the condition variable to signal and mutex to use for synchronization in this thread's THD::mysys_var.
thd->enter_cond() requires the mutex to be acquired in advance.
thd->exit_cond() unregisters the condition variable and mutex and releases the mutex.
If you want to have a Debug Sync point with the wait, please place it behind enter_cond(). Only then you can safely decide, if the wait will be taken. Also you will have THD::proc_info correct when the sync point emits a signal. DEBUG_SYNC sets its own proc_info, but restores the previous one before releasing its internal mutex. As soon as another thread sees the signal, it does also see the proc_info from before entering the sync point. In this case it will be "new_message", which is associated with the wait that is to be synchronized.
In the example above, the wait condition is repeated before the sync point. This is done to skip the sync point, if no wait takes place. The sync point is before the loop (not inside the loop) to have it hit once only. It is possible that the condition variable is signaled multiple times without the wait condition to be true.
A bit off-topic: At some places, the loop is taken around the whole synchronization pattern:
while (!thd->killed && !end_of_wait_condition)
{
pthread_mutex_lock(&mutex);
thd->enter_cond(&condition_variable, &mutex, new_message);
if (!thd->killed [&& !end_of_wait_condition])
{
[DEBUG_SYNC(thd, "sync_point_name");]
pthread_cond_wait(&condition_variable, &mutex);
}
thd->exit_cond(old_message);
}
Note that it is important to repeat the test for thd->killed after enter_cond(). Otherwise the killing thread may kill this thread after it tested thd->killed in the loop condition and before it registered the condition variable and mutex in enter_cond(). In this case, the killing thread does not know that this thread is going to wait on a condition variable. It would just set THD::killed. But if we would not test it again, we would go asleep though we are killed. If the killing thread would kill us when we are after the second test, but still before sleeping, we hold the mutex, which is registered in mysys_var. The killing thread would try to acquire the mutex before signaling the condition variable. Since the mutex is only released implicitly in pthread_cond_wait(), the signaling happens at the right place. We have a safe synchronization.
[edit] Co-work with the DBUG facility
When running the MySQL test suite with the --debug command line option, the Debug Sync Facility writes trace messages to the DBUG trace. The following shell commands proved very useful in extracting relevant information:
egrep 'query:|debug_sync_exec:' mysql-test/var/log/mysqld.1.trace
It shows all executed SQL statements and all actions executed by synchronization points.
Sometimes it is also useful to see, which synchronization points have been run through (hit) with or without executing actions. Then add "|debug_sync_point:" to the egrep pattern.
[edit] Debug Sync Further reading
For complete syntax tests, functional tests, and examples see the test case debug_sync.test.
See also worklog entry WL#4259 - Debug Sync Facility
Reference manual 5.1
- 2.3.2 Typical configure Options (--enable-debug-sync)
- 5.1.2 Command Options (--debug-sync-timeout)
- 5.1.4 System Variables (debug_sync)
Test framework manual
- 4.14 Thread Synchronization in Test Cases (have_debug_sync.inc)
- 5.3 mysql-test-run.pl (--debug-sync-timeout)