apache 2.2 - How Would I Restrict a Linux Binary to a Limited Amount of RAM?

I would like to be able to limit an installed binary to only be able to use up to a certain amount of RAM. I don't want it to get killed if it exceeds it, only that that would be the max amount that it could use. I would like the process to die once it reaches a certain amount of RAM, preferably before the server starts to swap heavily.

The problem I am facing is that I am running an Apache 2.2 server with PHP and some custom code that a developer is writing for us. The problem is that somewhere in there code they launch a PHP exec call that launches ImageMagick's 'convert' to create a resized image file.

I'm not privy to a lot of details to the project or the code, but need to find a solution to keep them from killing the server until they can find a way to optimize the code.

I had thought that I could do this with /etc/security/limits.conf and setting a limit on the apache user, but it seems to have no effect. This is what I used:

www-data hard as 500

If I understand it correctly, this should have limited any apache user process to a maximum to 500kb, however, when I ran a test script that would chew up a lot of RAM, this actually got up to 1.5GB before I killed it. Here is the output of 'ps auxf' after the setting change and a system reboot:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      5268  0.0  0.0 401072 10264 ?        Ss   15:28   0:00 /usr/sbin/apache2 -k start
www-data  5274  0.0  0.0 402468  9484 ?        S    15:28   0:00  \_ /usr/sbin/apache2 -k start
www-data  5285  102  9.4 1633500 1503452 ?     Rl   15:29   0:58  |   \_ /usr/bin/convert ../tours/28786/.….
www-data  5275  0.0  0.0 401072  5812 ?        S    15:28   0:00  \_ /usr/sbin/apache2 -k start

Next I thought I could do it with Apache's RlimitMEM setting, but get the same result of it not getting limited. Here is what I have in my apache.conf file:

RLimitMEM 500000 512000

It wasn't until many hours later that I figured out that if the process actually reached that amount that it would die with an OOM error.

Would love any ideas on how to set this limit so other things could function on the server, and all of them could play together nicely.