monitoring - Script to monitor specific processes' memory utilization

I have a client I do some remote server administration work for who recently started a 2-week trial period with New Relic. They have recently had some performance issues, and also asked me to take a look at things.

They saw that their memory on one of their servers was only hitting 50% according to New Relic, and became suspicious that the memory metrics / graphs were off (and emailed New Relic about it).

I spent some time this afternoon reviewing configuration files, CPU / Memory / IO graphs on New Relic, and making some recommendations.

I'm not convinced that the memory metrics are wrong. For example, on my client's DB server, New Relic said MySQL was using (at times) 3,481Mb of RAM. I ran PS to obtain the pid for MySQL, and ran the following command:

cat /proc/29313/status

Which output the following for VmRSS:

VmRSS:   3566268 kB

(which is 3,482Mb of RAM)

my.cnf has some settings I recommended that they tweak (key_buffer_size, etc...).

[Edit: If you're curious, this is irrelevant to my question below, but this particular server is running cPanel and has 32 GB of RAM. It's a dedicated MySQL server and has a number of databases running on it.]

My question is regarding whether or not VmRSS is the best metric for monitoring how much memory a certain process is using. If it is, I'm thinking about writing a script that would run on cron every 30 seconds or so for about 1-2 weeks, and then review the data. The script would look something like this:

$PID = pgrep mysql
cat /proc/$PID/status | grep VmRSS: > /var/log/my-custom-process-memory-monitor.log

Is this a good idea? Can you think of anything better? Why or why not?

Answer

RSS (resident set size) can be unreliable, because processes such as fork() do copy-on-write paging when allocating memory for the new process.

I.E if you spawn a process, give it 50M of memory to play with then fork() the loop 10 times - the RSS size will report a 500MB memory usage when in actual fact your more likely to be using just over 50M size.

For non-forking processes RSS should be generally reliable as far as I could tell.