apache 2.2 - Why are my httpd mpm_prefork processes being reaped so quickly?

[EDIT]

Bad question, mostly
- see my answer

[/EDIT]

We've
got a system running RHEL6, x64. We are using a local installation of apache 2.2.22 from
source. we serve
primarily:

1. mod_perl
   applications (with a local installation of perl
   5.16.0)


2. tomcat applications proxied with
   mod_jk

Here
is some context; the main question is
below.

All of this talks to an
Oracle backend.

We are having issues with
Oracle becoming unresponsive. We think this is because we're hitting the maximum process
limit in oracle. We've upped the process limit, but now we are hitting memory pressure
on the oracle server. We have tons of oracle sessions sitting idle. I can trace a bunch
of them back to the httpd processes.

We have
mod_perl's Apache::DBI start up a new connection to the database with each httpd child
that's spawned. We are concerned that these are not always getting closed out properly
when the httpd's exit...and the httpd's are exiting very
frequently. I know that it would be good to modify the mod_perl applications to use some
better form of db connection pooling; we plan to pursue that but would like to solve our
immediate problem sooner.

So
here's the main question.

We are using the prefork
MPM.

The apache child processes are
lasting at most a few minutes. Log analysis shows that each one is serving
fewer than 50 clients before exiting; the last request each child serves is
OPTIONS * HTTP/1.0 on some sort of internal connection; I'm
under the impression that this is a "ping" from the master process.

I've adjusted the MPM config as follows. I
didn't want to raise MinSpareServers too high, because, after all, i'm trying to
minimize the number of sessions to oracle.

MinSpareServers
            5
MaxSpareServers 30
MaxClients
            150

MaxRequestsPerChild
            10000

Right now we're
serving 250-300 requests per minute.

We've got
21 httpd's running, the eldest (other than the master, owned by root) being 3 minutes
old.

This rate of reaping of the apache
children really seems excessive. What could be causing
it?

Apache was built with:

 $ ./configure
            --prefix=/opt/apache --with-ssl=/usr/lib --enable-expires --enable-ext-filter
            --enable-info --enable-mime-magic --enable-rewrite --enable-so --enable-speling
            --enable-ssl --enable-usertrack --enable-proxy --enable-headers
            --enable-log-forensic

Apache
config info:

%
            /opt/apache/bin/httpd -V
Server version: Apache/2.2.22 (Unix)
Server
            built: Jul 23 2012 22:30:13
Server's Module Magic Number:
            20051115:30

Server loaded: APR 1.4.5, APR-Util
            1.4.1
Compiled using: APR 1.4.5, APR-Util 1.4.1
Architecture:
            64-bit
Server MPM: Prefork
 threaded: no
 forked: yes
            (variable process count)
Server compiled with....
 -D
            APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D
            APR_HAS_MMAP

 -D APR_HAVE_IPV6 (IPv4-mapped addresses
            enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D
            APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D
            APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D
            DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/apache"
 -D
            SUEXEC_BIN="/opt/apache/bin/suexec"
 -D
            DEFAULT_PIDLOG="logs/httpd.pid"

 -D
            DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D
            DEFAULT_LOCKFILE="logs/accept.lock"
 -D
            DEFAULT_ERRORLOG="logs/error_log"
 -D
            AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D
            SERVER_CONFIG_FILE="conf/httpd.conf"

modules
are compiled into apache rather than shared
libs:

% /opt/apache/bin/httpd
            -l

Compiled in modules:
 core.c

            mod_authn_file.c
 mod_authn_default.c
 mod_authz_host.c

            mod_authz_groupfile.c
 mod_authz_user.c

            mod_authz_default.c
 mod_auth_basic.c

            mod_ext_filter.c

 mod_include.c
 mod_filter.c

            mod_log_config.c
 mod_log_forensic.c
 mod_env.c

            mod_mime_magic.c
 mod_expires.c
 mod_headers.c

            mod_usertrack.c
 mod_setenvif.c

 mod_version.c

            mod_proxy.c
 mod_proxy_connect.c
 mod_proxy_ftp.c

            mod_proxy_http.c
 mod_proxy_scgi.c
 mod_proxy_ajp.c

            mod_proxy_balancer.c
 mod_ssl.c
 prefork.c


            http_core.c
 mod_mime.c
 mod_status.c

            mod_autoindex.c
 mod_asis.c
 mod_info.c

            mod_cgi.c
 mod_negotiation.c
 mod_dir.c

            mod_actions.c

 mod_speling.c
 mod_userdir.c

            mod_alias.c
 mod_rewrite.c

            mod_so.c

One final
note - the red hat httpd, apr, and perl packages are all installed, but ldd shows that
none of those libraries are linked with the running httpd.