HOW TO FIGURE OUT WHAT IS CAUSING AN APACHE SEGMENTATION FAULT

Ever seen this in your Apache error log?

1	[notice] child pid 10024 exit signal Segmentation fault (11)

If you have, read on. At my day job I deal with both Apache and PHP a lot. If you have ever tried to figure out why your PHP code seems to cause Apache segmentation faults, you probably experienced the same as I did. Lots of pain, frustration and headaches. However all is not lost. There is a way to figure out whatPHP code is making Apache act crazy. I’ll try to explain how to debug Apache using gdb to locate that nasty bug that is causing you to lose your precious beauty sleep.

BEFORE WE BEGIN

Now, Apache does not dump core by default. We need to do some work before that happens. If you don’t like to get your hands dirty compiling stuff manually, you should leave now, this is not for you.

Still here? Good! Before we start, make sure you have:

1. Root access to your web server
2. Apache source code and everything needed to (re-)compile Apache
3. PHP source code (or just download the .gbdinit file)

MAKE APACHE DUMP CORE

The first step is to make Apache not change user when it starts up and forks, so we will make it run as root the whole time. To accomplish this we need to compile Apache with -DBIG_SECURITY_HOLE. For obvious reasons, this is not recommended for production.

1 2 3

make clean export EXTRA_CFLAGS="-DBIG_SECURITY_HOLE" ./configure && make && make install

Now specify the “root” as the “User” in httpd.conf. While we’re editing httpd.conf we’ll add a setting to specify where Apache should put our core dump.

1 2 3

User nobody [...] CoreDumpDirectory /tmp/apache

Make sure /tmp/apache exists

1	mkdir -p /tmp/apache

On most systems the core file size is set to zero by default. We’ll go ahead and change it to unlimited.

1	ulimit -c 0

You can check your current core dump file size limit by running

1

ulimit -a

Restart Apache

1	apachectl restart

Next time Apache crashes with a Segmentation fault it should make us a core dump. If you did everything correctly you should see something like this in the apache error_log.

1 2	[notice] child pid 16430 exit signal Segmentation fault (11), possible coredump in /tmp/apache

Note: For reasons unclear to me my dump file ended up on the root (/) and not in the directory we specified with CoreDumpDirectory. If anyone knows why please drop me a comment.

MAKING SENSE OF THE CORE DUMP

At this point we could run gdb and get a backtrace, however that will only show us the function called inside php itself and not pinpoint where in our php code the problem is. To get a backtrace of our PHPcode we need to use the “dump_bt” function inside the .gbdinit file. Copy the .gbdinit file to your home directory

1	cp <php_source>/.gdbinit ~/

Start gdb with the path to httpd from the source as first parameter and the path to the core dump file as second

1 2	cd ~ gdb <apache_source>/src/httpd /tmp/apache/<core_file>

Run bt_dump from .gdbinit:

1	(gdb) dump_bt executor_globals.current_execute_data

Instead of a internal PHP backtrace we should get a nice backtrace of our PHP code.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

[...] [0xbf75f6fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf76249c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf7627bc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf76555c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf76587c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf76861c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf76893c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf76b6dc] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf76b9fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf76e79c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf76eabc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [0xbf77185c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484 [0xbf771b7c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767 [...]

In this example we can clearly see there is a recursive problem with query() and unlock() on line 767 and 484 in ezmysqldb.php. With the help of this backtrace in only took a quick look in ezmysqldb.php before I had a bug report written and posted: http://issues.ez.no/11038.

Thanks to my good friend and co-worker Derick Rethans for telling me about the “dump_bt” function.