monitoring - Script to monitor specific processes' memory utilization

itemprop="text">

I have a client I do some remote
server administration work for who recently started a 2-week trial period with New
Relic. They have recently had some performance issues, and also asked me to take a look
at things.

They saw that their memory on one of
their servers was only hitting 50% according to New Relic, and became suspicious that
the memory metrics / graphs were off (and emailed New Relic about
it).

I spent some time this afternoon reviewing
configuration files, CPU / Memory / IO graphs on New Relic, and making some
recommendations.

I'm not convinced that the
memory metrics are wrong. For example, on my client's DB server, New Relic said MySQL
was using (at times) 3,481Mb of RAM. I ran PS to obtain the pid for MySQL, and ran the
following command:

cat
            /proc/29313/status

Which
output the following for
VmRSS:

VmRSS: 3566268
            kB

(which is 3,482Mb
of RAM)

my.cnf has some settings I
recommended that they tweak (key_buffer_size,
etc...).

[Edit: If
you're curious, this is irrelevant to my question below, but this particular server is
running cPanel and has 32 GB of RAM. It's a dedicated MySQL server and has a number of
databases running on it.]

My question is
regarding whether or not VmRSS is the best metric for monitoring how much memory a
certain process is using. If it is, I'm thinking about writing a script that would run
on cron every 30 seconds or so for about 1-2 weeks, and then review the data. The script
would look something like
this:

$PID = pgrep
            mysql
cat /proc/$PID/status | grep VmRSS: >
            /var/log/my-custom-process-memory-monitor.log

Is
this a good idea? Can you think of anything better? Why or why not?

Answer

RSS (resident set size) can be unreliable, because processes such as fork() do
copy-on-write paging when allocating memory for the new
process.

I.E if you spawn a process, give it 50M
of memory to play with then fork() the loop 10 times - the RSS size will report a 500MB
memory usage when in actual fact your more likely to be using just over 50M
size.

For non-forking processes RSS should be
generally reliable as far as I could tell.