(This is a dump of the wiki where I kept notes on this; it's not

necessarily up to date, or correct at this point, but this is the basic

gist.)

"Creativity is hiding your sources."

I stole this idea verbatim from a chapter of Joel on Software. He stole

it from Microsoft, Apple, Netscape Talkback, etc.

http://www.fogcreek.com/FogBugz/docs/30/UsingFogBUGZtoGetCrashRep.html

Rationale

* Bad end-users don't report bugs.

* Really bad end-users report bugs badly.

* Most crashes happen at the same 3-5 points in a program.

* Steve Ballmer claims 20% of the bugs are 80% of the crashes, and 1% of

the bugs are 50% of the crashes.

* Keeping track of incidents individually takes an inordinate amount of time.

Basic Overview

MojoCrash is statically linked to an app, activates when there's a

crash, and eventually reports the bug. MojoCrash takes the user (mostly)

out of bug reporting on the client side. MojoCrash takes the developer

(mostly) out of bug management on the server side.

Basic Vocabulary

Strings: All strings are UTF-8 encoded unless otherwise specified, but

for all intents and purposes, this mostly means ASCII in this project.

All newlines are ASCII 10 (0x0A...'\n'), like Unix, regardless of client

or server platform, unless otherwise specified.

Client: the application on the end-user's side that reports crashes.

Server: the application on the developer's side that receives crash

reports.

Hook: A function pointer supplied by the app. A callback that may or may

not be implemented.

Trigger: Something that activates MojoCrash; an exception, GPF,

segfault, assertion failure, etc.

Catcher: Function that handles a trigger.

Reporter App: External program on end-user's machine that actually deals

with user input and transmission of crash info to server.

High-level Goals

* Catch crashes and bugs in the field.

* Feed them to developers with minimal user interaction.

* Sort them into meaningful data with minimal developer interaction.

High-level implementation

Server-side: Linux, Apache, PHP/Perl, Bugzilla. This must use these

components, for ease of migration between platforms and avoidance of

vendor lock-in. All the usual good Open Source mojo.

Client-side: C-callable library separate from the application, although

probably statically linked, and also a separate application that does

the heavy lifting.

Client catches a crash, logs it to disk. A new process runs (the

"reporter app") to handle the information. This is necessary since a

crashing app is in an unknown state and shouldn't do more in the process

than absolutely necessary; it also helps separate the functionality and

remove assumptions about how most of the client code needs to behave.

While it will always be a separate process, please note that the

crashing program may also be the reporter app; optionally, it may handle

the reporting on the next run of the program.

Reporter app sees log, confers with user, sends data as HTTP POST to

server, server tosses it to right place for developers to examine.

Client Library

There is only one entry point into MojoCrash for the client to use:

extern "C" int MOJOCRASH_install(const char *appname, const char *version, const MOJOCRASH_hooks *hooks);

The client-side code is initialized via MOJOCRASH_install(), and

interacts with the program through various hooks, all of which are

optional (you may specify NULL for any hook you don't want to implement,

or pass a NULL pointer for the struct if you don't need any of them).

Once MOJOCRASH_install() is called, the library will do its work if

there is a segmentation fault, assertion failure, etc. In a normal,

error-free run of the application, MojoCrash does nothing at all after

returning from this call. There is no uninstall: once MojoCrash is

ready, it stays ready until the process terminates.

The application hooks are fed to MojoCrash through the MOJOCRASH_hooks

structure. This struct is a collection of function pointers. Any pointer

can be NULL, including the pointer you supply for the entire struct, and

MojoCrash will use a reasonable default.

It's worth noting that you can manipulate MojoCrash through its hooks to

various ends that have nothing to do with the normal concept of a

"crash" ... for example, you can hook it up to only respond to assertion

failures or other special conditions instead of the normal segfault/GPF

type of crash, you can wire it to manage a scripting language or virtual

machine instead of the process itself, and other uses we haven't yet

conceived. Override individual pieces, or all of them, to taste.

Also note that if you're handling real crashes, you might be operating

with a corrupted memory heap, or with random portions of your process's

memory overwritten. This is an inexact science that demands a degree of

hope, but you shouldn't aggravate the situation:

Try to avoid malloc(), free(), etc in any hook other than

install_crash_catcher(), as your heap may be damaged during a legitimate

crash. Use static memory, and preallocate what you reasonably can.

Try to avoid touching any application-specific data; you should assume

it is unreliable, and the pointer you are dereferencing may be invalid,

causing a crash.

Try to precalculate data you might want to report during a crash,

instead of processing it when the system is in an uncertain state.

Try to avoid grabbing mutexes; a different thread could have crashed

while holding it, and you will deadlock.

Try to avoid unnecessary i/o; file handles could be corrupted or

blocking.

Try to avoid blocking. Go down quickly.

Try to avoid doing ANYTHING except the bare minimum. All you really want

to do here is get the minimal amount of useful data from the dying

process and stash it somewhere. All the heavy lifting, including a user

interface to apologize for the crash, will be done by a different

process in a known state.

Obviously, if you are "crashing" a virtual machine, scripting language

interpreter, or something else that's separate from the native process,

you can do whatever you like in the hooks, as your process is not in an

uncertain state.

It can not be stressed enough that, for all the complexity and following

discussion of the hooking system, most applications can supply no hooks

at all; the default, internal behaviour is precisely what most

applications need.

The hooks structure looks like this:

typedef struct MOJOCRASH_hooks

{

int (*install_crash_catcher)(MOJOCRASH_catcher catcher);

int (*preflight)(int sig, int crash_count);

int (*start_crashlog)(void);

int (*new_crashlog_line)(const char *str);

int (*get_callstack)(MOJOCRASH_get_callstack_callback callback);

int (*get_objects)(MOJOCRASH_get_objects_callback callback);

int (*get_etc)(MOJOCRASH_get_etc_callback callback);

int (*end_crashlog)(void);

int (*postflight)(void);

void (*die)(void);

} MOJOCRASH_hooks;

The install_crash_catcher hook lets the app manually install the

catcher, in case they need to maintain a chain they've previously set

up, or trigger MojoCrash in unusual ways, such as only when assert()

fails, or whatnot. Most applications will leave this hook set to NULL,

in which case MojoCrash will trigger in the normal ways for a given

platform (SIGSEGV and other signals on Unix, GPF handler on Windows,

etc). Returns zero on error, non-zero if trigger was

installed...MOJOCRASH_install() will fail if this returns zero, but

results are undefined if you really installed a catcher and reported

failure. To be clear, if you specify this hook, MojoCrash will not

install any sort of catchers on its own, and you must make sure that

"catcher" will be called when there is a crash condition. Most other

hooks are called in response to "catcher" being called (that is, when

MojoCrash is "triggered"). This hook is called during

MOJOCRASH_install().

The preflight hook allows an application to perform some act when

MojoCrash is triggered. This is called first inside the internal

MojoCrash catcher. There are two arguments to this hook: the signal

number related to the crash (this would be something like 11 (SIGSEGV)

on Unix, or 13 (GPF) on Windows, or some app-specific value, etc), and

the number of recursions into the catcher (first crash is 1, if there's

a crash in the crash handler, it's 2, etc...this number decrements each

time the catcher finishes, so unless you have a double-fault or whatnot,

this should be 1, even if you run the handler multiple times). Return

non-zero to signify that you don't want MojoCrash to continue handling

this trigger. If you return zero, the internal catcher will run, which

calls other hooks, etc. In a game, this might be a good place to release

the video display if you've got a fullscreen context...please consider

that the system is in an uncertain state when there's a crash, so it

would be wise to do as little work in this function as possible. If this

is left as NULL, MojoCrash does the following, depending on the value of

crash_count:

* returns 1 (just go forward to the usual handler).

* returns 0 (just try to stop crash handler immediately).

* Calls the die() hook. If that returns, returns 0.

* Tries to gently terminate the process (like exit(); static destructors

may run). If that returns, returns 0.

* Tries to forcefully terminate the process (like _exit(); let the OS sort

it all out). If that returns, returns 0.

* Just returns zero from there out. Nothing else you can do here, really.

The start_crashlog hook lets you create a crashlog manually. If this is

NULL, MojoCrash will write a standard file to a reasonable location for

future examination. If you want to stash the data somewhere else, or

modify it en route, you should override this hook. Please note that

you'll also need to override the end_crashlog() and new_crashlog_line()

hooks, as the default hooks operate on an internal file handle created

in the default start_crashlog(). The default behaviour also chooses a

unique, unused filename, and you will probably have to do likewise.

Otherwise, you'll get weird results, if not a double-fault. Also, please

note that the reporter app will expect several details of the default

behaviour, but you can override that as necessary, too. This hook does

not return the file handle, if there's a file handle at all; please keep

it in a static variable that your hooks can access. This hook returns

zero if there's an error (which will cause the crash handler to call the

die() hook and then return if the process still exists). If the hook

returns non-zero, the crash handler continues. Please note that if the

crash handler fails at any point, either through a hook reporting

failure or any other cause, we may NOT call end_crashlog(). Be prepared

for the process to terminate at any time, even outside of your control,

with the crashlog in any state. The reporter app will reject and delete

incomplete crashlogs.

The new_crashlog_line hook is called for each line of text going into

the crashlog. It may be called at any time after the start_crashlog()

hook sucessfully returns. The parameter to this hook is a

null-terminated C string in UTF-8 encoding. If this is NULL, MojoCrash

will write the line to the file it created in the start_crashlog() hook,

so if you implement this hook, you will need to implement the

start_crashlog() hook, too, or strange things will happen! This hook

returns zero if there's an error (which will cause the crash handler to

call the die() hook and then return if the process still exists). If the

hook returns non-zero, the crash handler continues.

The get_callstack hook allows the application to specify the crashing

process's callstack. This is called after the start_crashlog() hook, and

supplies a callback that will eventually call the new_crashlog_line()

hook. If this is NULL, MojoCrash will use the platform's facilities to

determine the current callstack, which is usually what you want.

However, some apps maintain their own callstack information that has

more clarity than the junk the OS supplies, and others might want to

supply a callstack relating to a scripting language or virtual machine

and not the application process itself. This hook has a callback

function as its only parameter; this is to prevent the need to allocate

a buffer of arbitrary size for the callstack. There is no limit on the

size of callstacks, but it's not unreasonable to put an upper cap on

them, in case the system ends up in a loop (a corrupted stack is not

unheard of during crashes, after all). Each call to the callback

function reports a void pointer that represents the next frame pointer

on the callstack, starting at the top (so the crashing function, or

perhaps the crash handler, should be the first thing passed to the

callback, and something like main() should be the last). This hook does

not concern itself with the actual function names, source files or line

numbers of the callstack, just the addresses. This hook returns zero if

there's an error (which will cause the crash handler to call the die()

hook and then return if the process still exists). If the hook returns

non-zero, the crash handler continues.

The get_etc hook allows the application to specify app-specific

information. This is called sometime after the start_crashlog() hook,

and supplies a callback that will eventually call the

new_crashlog_line() hook. If this is NULL, MojoCrash will do nothing

here (any "etc" information reported is optional and specific to the

app), which is usually what you want. The crash handler will report

other things about the system (OS version, processor type, etc) separate

from any of the hooks, so you might not need to dig up other information

here anyhow. This hook has a callback function as its only parameter;

this is to prevent the need to allocate a buffer of arbitrary size for

the list of objects. There is no limit on the number of "etc" items.

Each call to the callback function reports key and value pairs, each a

string. For example, you might pass the key "GL_VERSION" and the value

"2.1.1 NVIDIA 100.14.19" if you wanted to know what OpenGL's

glGetString(GL_VERSION) call reports on the crashing machine (please

preallocate this string, though, since you ARE crashing at this point,

and the GL isn't necessarily reliable!). There are no conventions on

what format the key and value must conform to, except that both need to

be null-terminated UTF-8 strings. This hook returns zero if there's an

error (which will cause the crash handler to call the die() hook and

then return if the process still exists). If the hook returns non-zero,

the crash handler continues.

The end_crashlog hook is called if (and only if!) the crash handler has

successfully run completely through all other hooks involved in

producing data for the crashlog, and should be used to finalize what was

started in start_crashlog(). If this is NULL, MojoCrash will close the

internal file handle it created and ensure the crashlog data is flushed

to disk. You'll need to implement all of start_crashlog(),

new_crashlog_line() and end_crashlog(), or none of them. In most cases,

the default hooks are fine (and the reporter app expects the default

behaviour unless you override that, too). Please note that if there's a

failure in the crash handler (one of the hooks returns zero, etc), this

hook is NOT called. If this is going to be a serious problem, be

prepared to clean up in the die() hook, but you should assume that the

process could terminate without calling either (which will mean the OS

will clean up open files, etc, instead of you). This hook returns zero

if there's an error (which will cause the crash handler to call the

die() hook and then return if the process still exists). If the hook

returns non-zero, the crash handler continues.

The postflight hook runs after the end_crashlog() hook returns

successfully. This allows you to do something after MojoCrash has logged

the error to disk for transmission, such as your own cleanup (but

remember, don't do more than necessary: you are crashing, after all!).

If this hook is NULL, it's a no-op internally. You should return

non-zero on success from this hook, and zero if there's an error, but

this is merely for future expansion: in either case, MojoCrash will call

the die() hook and then return if the process still exists.

The die hook can be called at any time. This is called by the crash

handler before returning, either when all other hooks have returned

successfully or when one of the hooks reports failure. If this returns,

the crash handler will return, too, so you want to make sure it does not

return for legitimate program crashes; in these cases, you may want to

terminate the process: throw a C++ exception, use longjmp(), or

otherwise alter control flow. If your "crashes" are cleanly separated

from the OS process, such as an assertion in a scripting language or a

problem with a virtual machine, etc, you may want to override this hook

to do nothing, and then regain control normally when the crash handler

returns. If this hook is NULL, the crash handler will try to gently kill

the process (this is usually equivalent to the C runtime's exit()

function, not the _exit() one, so static destructors might now run).

To summarize the chain of events for an unexceptional case:

* Application calls MOJOCRASH_install().

* MojoCrash stores hooks, replaces NULL hooks with internal functions.

* install_crash_catcher hook runs.

* MOJOCRASH_install() returns.

* Application later crashes for whatever reason, triggering the crash catcher.

* Control jumps to MojoCrash.

* preflight hook runs.

* start_crashlog hook runs.

* Some internal data gets added to the log with new_crashlog_line hook.

* get_objects hook runs, calling callback which calls new_crashlog_line hook.

* get_callstack hook runs, calling callback which calls new_crashlog_line hook.

* get_etc hook runs, calling callback which calls new_crashlog_line hook.

* end_crashlog hook runs.

* postflight hook runs.

* die hook runs.

* Application terminates.

Crash log format

(This information is still subject to change.)

The crashlog is a plain text file in UTF-8 encoding, using Unix newlines

(ASCII 0x0D or '\n') for endlines. Here's an example log:

# This is a log of a program crash. It is meant to help the

# developers debug the problem that caused the crash.

#

# This log was generated by software called "MojoCrash" but

# please note that MojoCrash did NOT cause the crash in the first

# place. It only helped report the problem. Please do not contact the

# MojoCrash developers about this problem, we didn't cause it and

# probably haven't heard of the crashing program.

#

# If you need to contact someone about a bug, please contact the

# original vendor of your program instead. Thanks!

MOJOCRASH 0.0.1

CRASHLOG_VERSION 1

APPLICATION_NAME TestApp

APPLICATION_VERSION 1.0

PLATFORM_NAME linux

CPUARCH_NAME x86-64

PLATFORM_VERSION 2.6.22.10

CRASH_SIGNAL 11

APPLICATION_UPTIME 0

CRASH_TIME 1199350192

OBJECT test-bin/4194304/12124

OBJECT libc.so.6/47641679454208/1381596

OBJECT ld-linux-x86-64.so.2/47641677234176/117128

CALLSTACK 4201103

CALLSTACK 4200063

CALLSTACK 4199576

CALLSTACK 4199829

CALLSTACK 47641679656912

CALLSTACK 4202709

CALLSTACK 4202726

CALLSTACK 4202737

CALLSTACK 4202904

CALLSTACK 47641679575876

CALLSTACK 4196729

ETC_KEY ThisIsMyKey

ETC_VALUE ThisIsMyValue

ETC_KEY ThisIsAnotherKey

ETC_VALUE ThisIsAnotherValue

END

Please note that blank lines are ignored. Lines starting with '#' chars

are treated as comments and are thus ignored.

The first not-ignored line is always "MOJOCRASH <version>" and

represents the version of MojoCrash that produced this crashlog. The

second line is the crashlog format version. Currently this is always

"CRASHLOG_VERSION 1", but the value increments whenever new fields are

added. Servers should understand all fields up to the version they

handle (and in most cases, should pass other fields through unmolested,

as most are meant to be human-readable anyhow). The rest of the lines

can appear in any order and frequency (although they tend to have a

static pattern, as that's how the source code dumps them out). Here they

are:

"APPLICATION_NAME <string>" is the name of the application. This should

be an identifier, not an English title, so "UT2004" is better than

"Unreal Tournament 2004" here. Aim for not having spaces or "strange"

characters.

"APPLICATION_VERSION <string>" is the version of the crashing app. The

string can be any format: 1.0, svn-revision-452, 5.2.1.0, "december 25,

2:03 am", whatever makes this build uniquely identifiable.

"PLATFORM_NAME <string>" is the platform that the crashing app was run

upon: "linux", "macosx", "windows", etc.

"PLATFORM_VERSION <string>" attempts to identify the platform's

revision. This might be the kernel version on Linux, "5.0" on Windows

XP, "10.4.1" on Mac OS X Tiger, etc.

"CRASH_SIGNAL <int>" is the signal number for the crash. This might be

11 (SIGSEGV) on Unix, 13 (GPF) on Windows, etc. This is the int passed

to the preflight() hook.

"APPLICATION_UPTIME <int>" is the amount of time the application ran, in

seconds, between MOJOCRASH_install() being called and the crash handler

starting, so in reasonable cases, this is roughly the amount of time

that the program ran before crashing.

"CRASH_TIME <int>" is the time when the app crashed, in seconds since

the Unix epoch (midnight, Jan 1st, 1970). This can be translated to a

specific date and time when the crash occurred.

"OBJECT <string>/<int>/<int>" specify code objects: that's the filename,

base address, and size in bytes, respectively. These repeat for each

object.

"CALLSTACK <int>" specify the callstack. Each line is a frame pointer,

starting with the top of the stack (the crash or crash handler, usually)

and ending with main() or whatnot. This repeats for each stack frame and

collectively represents the entire callstack.

"ETC_KEY <string>" specifies a key reported in the get_etc() hook. It

must be followed by an "ETC_VALUE" line. There may be any number of

these repeated in the log. Please limit keys to printable characters

(alphanumeric and underscores).

"ETC_VALUE <string>" specifies a value reported in the get_etc() hook.

It follows an "ETC_KEY" line. There may be any number of these repeated

in the log. If you need a newline, please encode it as a \n sequence, as

a real endline char would end the ETC_VALUE data.

"END" must be the last line in the log, demonstrating that the log is

complete and not truncated.

Reporter App

(This section is out of date, please revise!)

The reporter app is a seperate process because it allows us to not care

about the main application's requirements for threading, UI, etc. It

also allows us to do the complex work from outside a process that might

have any sort of strange memory corruption and otherwise mangled state.

Basic chain of events for the reporter app:

* Reporter app reads pending bug reports and deletes them from disk.

* If pending reports look like garbage, reporter app terminates immediately.

* Reporter app shows user the various info it plans to send, lets them

optionally enter an email address.

* User declines sending? Delete crash log and terminate immediately.

* User approves sending? Connect to server and HTTP POST the data.

* Read response from server, don't care about the result, hang up.

Posting to the web server

Connect to the server ("crashcatcher.mojogames.com" or whatever), TCP

port 80, and send something like this:

POST / HTTP/1.1

Host: crashcatcher.mojogames.com

User-Agent: mojocrash/0.0.1

Accept: text/plain

Accept-Charset: utf-8

Connection: close

Content-Type: multipart/form-data; boundary="9234j239fj2gjklsdu82jfl0234"

Content-Length: 1687

--9234j239fj2gjklsdu82jfl0234

Content-Disposition: form-data; name="report1"; filename="file1.txt"

Content-Type: text/plain; charset=utf8

(crash report log file goes here.)

--9234j239fj2gjklsdu82jfl0234--

Going through this line-by-line: !!! FIXME: this is out of date.

There's a POST command to http://crashcatcher.mojogames.com/ ... you

determine the POST and the "Host:" lines from the url variable passed to

MOJOCRASH_report().

The post is done by the MojoCrash client (User-Agent). The server will

reject all posts from clients with a different User-Agent.

We are sending utf-8 encoded, plain text, and expect the reply to be as

such, too.

Content-Type line is required to be application/x-www-form-urlencoded so

the server doesn't mangle it.

Content-Length is just the strlen() of the data.

There must be a blank line after that so the server knows the real data

is now coming.

The actual POST data then follows.

Client reads response and hangs up.

Please note that there's a hook that allows transformation of the data, which could, say, gzip-compress it and add an HTTP header that says as such.

To be clear, we're POSTing a few variables here.

The first is "app" ("UT2004") and that's the appname variable passed to

MOJOCRASH_install().

The second is "version" ("3355") and that's the version variable passed

to MOJOCRASH_install().

Next is "stack" which is the stack dump. Above it is url-encoded, but in

human-readable terms, it looks like this:

printf (stdio.c:801) [0xabcdef01]

main (myprog.c:23) [0x10000000]

...that is, the program crashed in the printf() function, on line 801 of

the source code stdio.c. The memory address of the crash point in memory

was 0xabcdef01. printf(), in this case, was called on line 23 of

myprog.c, in the function main(). Programs without debug symbols will

just leave out the (stdio.c:801) part.

Next is "email" ("a@b.com"). This is what the user supplied, and it may

be an empty string if they chose to ignore it.

Finally is "other" ("kernel 2.4.2") and that's whatever was supplied by

the get_other_info callback.

Server

The server side is an Apache process that reads the request, parses the

data, and passes it to a PHP or mod_perl script. This means the tedious

parts, like logging, basic string parsing, or deciding if this was a

valid virtual host, are already done by the webserver.

The script, when it receives control, will work as glue to Bugzilla.

This can be done as a perl CGI script to take advantage of the existing

bugzilla code, or can exist outside of bugzilla and use their example

bug entry scripts. The bugzilla itself may or may not be available to

the public, so this script may be the only way they can interact with

it.

Once the script is running, it will need to decide if the given stack

already exists in a bug entry, and if not, create a new entry. If it

already exists, it should just append a new comment to the existing bug

report with the pertinent details (email address, the "other" field,

etc) and add a vote to the entry...the vote is just a convenient way to

sort the bug list by most popular crashes...the one with the most votes

is the one people are reporting the most.

If the bug was successfully added/updated in bugzilla, it will return a

HTTP 200 result code ("success") and disconnect from the client. If

there was a temporary problem, it should report 503 ("Service

unavailable") and the client may optionally include logic to try

resending sometime later. If there was a permanent problem or the

client's POST was malformed, it should report 400 ("Bad Request") and

the client should never try to resend this bug report.

To be considered...

Callstacks for multiple threads?

Checksums/validation for posted logs?