MySQL Error Handling and Reporting
Presenter: Tim Smith Time: Thursday ... 2007, at 6am PST = 9am EST = 15 CET = 16 EET Estimated length of Session: One hour Scribe: Peter Lavin Attendees: To register for this Session - Please enter your name here: * Alexander Nozdrin * AntonyCurtis * Hartmut Holzgraefe * Oleksandr Byelkin
[edit] Topic
The following notes are rough, and I apologize for their incompleteness. I hope that, out of today's session, a more complete and useful description can be created regarding error handling.
Infrastructure:
===================================================
my_errmsgs_list:
- Starts with a list of global errors, from mysys/errors.c: globerrs, ER_*
- Errors are added by my_error_register():
- init_errmessage(), called from mysqld's main()
- Replaces the default global errors list
- Reads language-specific errmsg.sys file and stuffs the error messages
into an array, and then registers them
- Macro EE() peeks into globerrs array to get error text
- ha_init_errors(), from ha_init(), from mysqld's main()
- Adds HA_ERR_* error messages for handler-related errors
- Many error texts are copies of EE(ER_*)
- init_client_errors(), called from mysql_server_init() (thus, from
mysql_init())
- adds CR_* error messages
- Macro ER() peeks into client_errors array to get error text
net.report_error / last_errno / last_error / sqlstate
===================================================
The net.report_error flag is often set when a problem condition occurs. It is
Used by / Fed to:
- mysql_error(), in client.c (important part of MySQL C API)
- slave_print_msg(), and others, in log_event.cc, slave.cc
- my_message(), in sql_insert.cc, sql_prepare.cc
- push_warning(), in sql_show.cc
- sql_print_warning(), in libmysqld/sql_connect.cc
- sql_print_error(), in rpl_rli.cc
my_error()
===================================================
my_error():
- Looks up error number message string in my_errmsgs_list
- Formats message into buffer, and calls error_handler_hook():
- For mysqld, it is my_message_sql()
- For all other programs, it is my_message_no_curses()
my_printf_error():
- Like my_error(), but uses the supplied message string
- Ignores the "error" argument
my_message():
- Like my_printf_error(), but without my_vsnprintf() call
my_message_no_curses():
- Beep if ME_BELL flag
- Print message to stderr
- Ignores the first "error" argument
sql_print_error(), _warning(), _information():
===================================================
Write messages to error log via vprint_msg_to_log (buried under a few layers of
C++ classes, see log.h and log.cc).
my_message_sql():
===================================================
mysqld sets error_handler_hook = my_message_sql; so every time my_error() or
similar function is called, my_message_sql() handles it.
thd->fatal_error():
===================================================
During query execution, a fatal error is marked with thd->fatal_error(). This
sets net->report_error, and also thd->is_fatal_error. That flag is checked in
many places as a sign of a failed operation, and is usually being passed up the
call chain to a my_error() call.
Examples:
my_read():
- Set my_errno= errno
- Call my_error(), if asked for (MyFlags & (MY_WME | MY_FAE | MY_FNABP))
- Return -1
Some interesting special cases:
my_malloc():
- Calls exit(1) if allocation fails and MY_FAE flag is set
Side notes:
MYSQL_DEBUG environment variable, can create debug trace log from application
which doesn't have a --debug option
Idioms:
A common pattern is:
my_bool foo()
{
my_bool error= TRUE; /* Error */
Resource r;
if (!(r= get_resource()))
goto end;
if (use_resource(r))
goto end;
error= FALSE; /* Success */
end:
if (r)
free_resource(r);
return error;
}
Things which need more work:
- Exactly how report_error, last_errno, etc. are used, and how they interact with is_fatal_error
- Mention of Internal_error_handler class?
- While client code sets last_errno and last_error, server code sets just last_errno, but also report_error; what are the general rules for how this used?
- When must my_errno be set, and when can it be ignored?
And, probably there are some major things that I've left out, due to ignorance or lack of time.
[edit] Session notes
IRC Log of MySQL Error Handling and Reporting
Presenter: Tim Smith
[edit] =============================
<timothy> now, a while back stefan solicited ideas for new mysql university topics
<timothy> i suggested error handling and reporing
<timothy> because i wanted to learn more about it, from kostja or someone
<timothy> stefan said, "ok, when would you like to present on that?"
<timothy> :-/ <joro> :-) <timothy> so, I've collected together some of what I can understand about this topic, but I'm sure there can be many improvements to what I've got
<timothy> so please feel free to correct or embellish; we can set aside some time for that at the end, too <timothy> i'll briefly describe the error messages themselves
<timothy> * Infrastructure
<timothy> while many messages that the user sees are just hardcoded somewhere, there is a setup for standard error messages
<timothy> there are: global errors, client-specific errors, and handler-specific errors
<timothy> mysql has a structure for storing these, a way of mapping from a number to a message string, and several functions which use it
<timothy> global errors are defined in mysys/errors.c, and use the ER_* constants for error numbers
<timothy> a client program adds to these (inside mysql_server_init()) the CR_* errors
<timothy> and mysqld also adds handler-specific ones, in ha_init()
<timothy> those are HA_ERR_*
<timothy> so if you see a CR_FOOBAR error, it's a client error message and is defined in the client_errors array
<timothy> also, you'll see some macros, like: EE(ER_FOO)
<timothy> those just map ER_FOO (a number) to an index into the array
<timothy> I mean, EE(ER_FOO) is the error string for ER_FOO; it is globerrs[ER_FOO - ER_ERROR_FIRST]
<timothy> same thing can be done with ER(CR_FOO) for client errors
<timothy> alright, that's stuff just to help read the code, and understand where those strings are defined
<sanja> there is also mysql system library errors EE_* and they even should be maped to ER_* but code looks brocken whan I looked on it year ago
<timothy> most of them are printf-style formats
<timothy> it's important to not add more %* escapes to an existing error format
<andrey_> Why?
<andrey_> ah, ok, because it will read from non-passed memory
<timothy> because an old server binary which uses a new error message file will pass too few arguments
<timothy> to vsnprintf
<timothy> oh, i forgot to mention that the global errors which are hard-coded in mysys/errors.c get removed, and replaced with the errmsg.sys language-specific errors
<timothy> in mysqld
<andrey_> new server binary complains if there are less messages, the old binaries don't complain if there are more than expected?
<sanja> andrey_: where you saw it complaining AFAIK it just crashes or print garbage
<andrey_> I have seen sth like "errmsg.sys has only xyz errors, but the server expects abc errors"
<timothy> andrey_: you're asking a different question; about a different number of error messages, vs. the format string for a single existing error, right?
<andrey_> kinda
<andrey_> anyway
<timothy> ok
<andrey_> good point anyway
<timothy> yes, and i think you're correct but forget the detail there
<sanja> andrey_: it is about number of errors not about arguments of one error
<sanja> i.e. if number of errors changed it will complain
<timothy> ok. so, the array(s) of errors get used by my_error()
<timothy> and its friends, my_printf_error() and my_message()
<timothy> well, those last two don't really care about the error number, sorry
<timothy> my_error() does look up the error number in the errors list, and it uses the associated error string in a my_vsnprint() call
<timothy> so this is a great place to set a break point if you want to see where an error happens
<sanja> it should be used only when error array is not initialized
<timothy> set it in my_error(), and it will stop at the right place
<timothy> sanja: which should be used only when error array isn't initialized?
<sanja> my_message AFAIK
<timothy> ok
<timothy> another layer on top of this is the error_handler_hook global
<timothy> my_error() and friends call error_handler_hook to actually handle the error
<timothy> (print it, whatever)
<sanja> BTW I'd prefer to break on my_message_sql (it is usual error handler)
<timothy> by default, it points at my_message_no_curses
<timothy> which just beeps if requested and prints the message to stderr
<timothy> but mysqld sets it to my_message_sql()
<timothy> which is a bit more complicated
<timothy> there is the structure in place to add some error handling callbacks; one in thd (that is, thd->handle_error()), and one for stored procedures
andrey_ recalls something like this in mysys/
<timothy> assuming that those don't handle the error, then the error is pushed onto the warnings list
<timothy> this is actually a good function to look at, i think
<timothy> the thd keeps a list of warnings, which persist beyond a single query
<timothy> thd->warn_list
<timothy> now, here's someplace that i'm fuzzy
<timothy> where does the thd->warn_list get sent to the client?
<sanja> timothy: by request
<timothy> i didn't quite figure that out
<sanja> usualy only number of warnings sent
<sanja> then user issue SHOW WARNINGS
<andrey_> .oO(or SHOW ERRORS)
<timothy> ok
<timothy> erm, also not seeing where sql_print_error() is called in my_message_sql(), but that's alright
<timothy> i'll run it in a debugger later :-)
<timothy> ok
<timothy> how's everyone doing so far?
<jbalint> ok
<sanja> my_message_sql only set net.report_error which mean the we have errors and should send at least one at athe end of statement execution
<timothy> yes, report_error (and thd->fatal_error()) need to be examined next
<sanja> BTW there is thd-> clear_error() which corerectly clear this state if one want remove error or replace it with other one
<sanja> (one should not change net.report_error / last_errno / last_error directly
<sanja> ) <timothy> i'll just mention that sql_print_error() (and _warning() and _information()) simply drill down through some C++ logging classes (log.cc) to finally call vprint_msg_to_log(), which writes a timestamped message to stderr
<timothy> ok, so let's walk on to an area that i've probably butchered
<timothy> i confess i couldn't get a high-level view of what is going on with net.report_error, last_errno and last_error
<timothy> the report_error flag should indicate that the server should report an error to the client
<timothy> however, it is also used to test for failure in a number of operations
<sanja> timothy: you are right
<sanja> if server should report an error then error accured :)
<timothy> yes
<timothy> :-)
<timothy> but now, sanja, you mention that one should not set this directly
<sanja> but there is also fatal_error which set report error too but require more fast abort
<timothy> one should use my_error(), thd->fatal_error(), or thd->clear_error() only?
<timothy> in normal situations?
<sanja> yes and ths->clear_error()
<timothy> ok, good.
<sanja> s/ths/thd/
<timothy> one slightly confusing thing is that stored procedures do use net.report_error in a slightly different way, to coordinate error handling for nested procedures (as far as i can tell)
<timothy> so a lot of the thd->net.report_error= 1 code is due to that
<timothy> ok <timothy> one last thing on my list to talk about is my_errno
<timothy> it is used in mysys, to keep track of what caused an error
<timothy> typically a function will return non-zero for failure, and will store the reason for failure in my_errno
<timothy> however, this isn't done consistently, so it seems you really do have to look at the code before you can use my_errno
<timothy> and not just assume it will be set after a failed mysys call
<timothy> of course, the style mysql prefers is: for functions that return a pointer, NULL indicates failure, and otherwise success
<sanja> as I understood it set the call which substitute usual system cals which set errno (but I never check it sorely)
<timothy> for other functions, 0 (sometimes called FALSE) indicates success, and otherwise (sometimes called TRUE) is failure
<sanja> server functions should return non-zero on failure other made historically as I understand
<sanja> exclusion made for functions return pointers
<timothy> for me, i couldn't find a clear category for which functions set my_errno
<timothy> so....
<timothy> if a function returns failure, how is it handled?
<timothy> do you: just return failure yourself, up to your caller?
<sanja> in the server it used usually like this
<timothy> check my_errno, and print it using my_message()?
<sanja> if (func1(...) || func2(...)) {
<sanja> errro processing
<timothy> check net.last_errno?
<sanja> }
<timothy> check thd->is_fatal_error?
<timothy> check thd->net.report_error?
<sanja> it depends if you want abort with aborting connection (i.e. asap) you check thd->is_fatal_error
<timothy> this is where I don't have clear guidelines (i just say, copy from code around you)
<sanja> if it is place to check any error then check thd->net.report_error
<sanja> thd->is_fatal_error ments unrecoverable error (connection will be lost), thd->net.report_error mens just an error
<timothy> ok!
<sanja> also I think should be mentioned that memory allocators in server set all need error states so you need only check NULL and abort in safe way if NULL returned
<timothy> now, i heard rumours of runtime team having some ideas (or plans?) for some improvements to the error handling
<timothy> i asked kostja but he is very busy at the moment
<timothy> so perhaps the plans are only related to how stored procedures handle errors
<timothy> anyways, i think this is all somewhat confusing, and harder to keep track of then needed, but it does work pretty well in practice it seems
<timothy> sanja, good point about memory allocators
<timothy> that is another good topic - Sql_alloc and placement new :-)
<timothy> ok. this is all i have at this point
<timothy> anyone have any questions or other suggestions?
<timothy> any stories to tell about bugs which were related to misusing the error handling features of mysql?
<sanja> timothy: thank you, you made huge amount of work collecting this in some system! :)
<sanja> (I am impressed)
<ingo> Thanks timothy.
<timothy> sanja: thank you. i'm still wanting a bigger picture of some parts of it
<jbalint> thanks, good info
<timothy> i feel i have tunnel vision
<joro_uni> timothy: it would be great to add a text to the internals doc to summarize what to do on error (if you're wring a function).
<timothy> joro_uni: definitely, i hope that can come out of this
<serg> about "stories to tell about bugs"
<timothy> my intention is to talk to a few other people regarding this presentation and flesh out the details, and create a new wiki page in the internals doc
<timothy> serg: yes?
<serg> one of the typical bugs - abort a statement with an error, without sending an error packet to the client
serg: can you show what that would look like in pseudo-code?
<serg> a server waits for a new command from the client, the client waits for a reply from the server <serg> well
<sanja> serg: it should be handled automagically if thd->net.report_error set
<serg> sanja: yes
<serg> but sometimes we reset an error flag
<serg> or do another magic-fighting trick
<serg> we had quite a few of those bugs
<serg> a recent one was with INSERT ... ON DUPLICATE KEY, iirc
<serg> timothy: normally, as sanja said, my_error sets a flag, and at the end of a statement processing
<serg> an error packet is sent to the client
<timothy> yes
<timothy> and "abort a statement with an error" is properly done with my_error(), so in the normal case the programmer doesn't have to worry about this
<serg> yes. but the fact is, there were quite a few bugs with this exactly problem
<timothy> but if we're playing tricks with .report_error, then it can go wrong
<serg> yes
<timothy> alright, is it time to wrap this up?
<timothy> going once?
<timothy> going twice?
<timothy> thank you all for being here! especially sanja :-)