apache 2.2 - How Would I Restrict a Linux Binary to a Limited Amount of RAM?

I would like to be able to limit an installed binary to
only be able to use up to a certain amount of RAM. I don't want it to get
killed if it exceeds it, only that that would be the max amount that it could
use. I would like the process to die once it reaches a certain amount of
RAM, preferably before the server starts to swap
heavily.

The problem I am facing is that I am
running an Apache 2.2 server with PHP and some custom code that a developer is writing
for us. The problem is that somewhere in there code they launch a PHP exec call that
launches ImageMagick's 'convert' to create a resized image
file.

I'm not privy to a lot of details to the
project or the code, but need to find a solution to keep them from killing the server
until they can find a way to optimize the
code.

I had thought that I could do this with
/etc/security/limits.conf and setting a limit on the apache user, but it seems to have
no effect. This is what I used:

www-data hard as
500

If I understand it correctly,
this should have limited any apache user process to a maximum to 500kb, however, when I
ran a test script that would chew up a lot of RAM, this actually got up to 1.5GB before
I killed it. Here is the output of 'ps auxf' after the setting change and a system
reboot:

USER PID %CPU %MEM VSZ RSS
            TTY STAT START TIME COMMAND
root 5268 0.0 0.0 401072 10264 ? Ss 15:28 0:00
            /usr/sbin/apache2 -k start
www-data 5274 0.0 0.0 402468 9484 ? S 15:28 0:00 \_
            /usr/sbin/apache2 -k start
www-data 5285 102 9.4 1633500 1503452 ? Rl 15:29
            0:58 | \_ /usr/bin/convert ../tours/28786/.….
www-data 5275 0.0 0.0 401072
            5812 ? S 15:28 0:00 \_ /usr/sbin/apache2 -k
            start

Next I thought I
could do it with Apache's RlimitMEM setting, but get the same result of it not getting
limited. Here is what I have in my apache.conf
file:

RLimitMEM 500000
512000

It wasn't until many hours later that I
figured out that if the process actually reached that amount that it would die with an
OOM error.

Would love any ideas on how to set
this limit so other things could function on the server, and all of them could play
together nicely.