Skip to main content

linux - Amazon Micro Instance Crashed - Help Me Figure Out Why?




I am running an Amazon AWS Micro Linux Instance and it crashed during some "light" usage a few days ago. I am running an app that uploads photos to the server. We had maybe 10 users uploading multiple photos during a 1 hour period.



At some point, the server stopped responding. I logged into the AWS Console and found that the "Intstance Reachability Check" had failed. I rebooted the server, restarted PHP and MySQL, and then had to repair a few MySQL tables that had been corrupted.



I had the monitoring tools turned on and the CPU Usage indicates that we topped out at 28% CPU Usage - After reading some more documentation about Micro Instances, I do not think we maxed out on the CPU, but I could be wrong.



I don't know a enough to understand what the logs mean. I have found what I believe to be the logs from the server from the time where the problem occurred, I am hoping that someone can help me decipher what happened:




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157082] httpd

invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157100] httpd
cpuset=/ mems_allowed=0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157108] Pid: 1824,
comm: httpd Not tainted 2.6.35.14-97.44.amzn1.x86_64 #1



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157117] Call Trace:




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157131]
[] ? cpuset_print_task_mems_allowed+0x98/0xa0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157143]
[] dump_header.clone.1+0x77/0x1a0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157153]
[] ? _raw_spin_unlock_irqrestore+0x19/0x20



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157164]

[] ? ___ratelimit+0x9f/0x120



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157172]
[] oom_kill_process.clone.0+0x76/0x140



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157185]
[] __out_of_memory+0x118/0x190



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157196]
[] out_of_memory+0x82/0x1c0




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157204]
[] __alloc_pages_nodemask+0x689/0x6a0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157215]
[] alloc_pages_current+0x94/0xf0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157223]
[] __page_cache_alloc+0x7f/0x90




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157232]
[] __do_page_cache_readahead+0xc0/0x200



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157240]
[] ra_submit+0x1c/0x20



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157248]
[] filemap_fault+0x3e3/0x430



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157257]

[] __do_fault+0x4f/0x4b0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157265]
[] handle_mm_fault+0x1b4/0xb40



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157275]
[] ? xen_clocksource_read+0x21/0x30



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157283]
[] ? xen_clocksource_get_cycles+0x9/0x10




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157293]
[] ? ktime_get_ts+0xa3/0xd0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157302]
[] do_page_fault+0x112/0x310



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157310]
[] page_fault+0x25/0x30




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157316] Mem-Info:



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157320] Node 0 DMA
per-cpu:



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157327] CPU 0:
hi: 0, btch: 1 usd: 0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157332] Node 0
DMA32 per-cpu:




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157339] CPU 0:
hi: 186, btch: 31 usd: 30Jul 23 00:19:07 ip-10-117-66-219 kernel:
[19699669.157347] active_anon:70901 inactive_anon:71000
isolated_anon:0Jul 23 00:19:07 ip-10-117-66-219 kernel:
[19699669.157348] active_file:13 inactive_file:28 isolated_file:0Jul
23 00:19:07 ip-10-117-66-219 kernel: [19699669.157349] unevictable:0
dirty:4 writeback:0 unstable:0Jul 23 00:19:07 ip-10-117-66-219 kernel:
[19699669.157350] free:1374 slab_reclaimable:613
slab_unreclaimable:1843




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157351] mapped:25
shmem:34 pagetables:4842 bounce:0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157430] Node 0 DMA
free:2436kB min:76kB low:92kB high:112kB active_anon:6104kB
inactive_anon:6272kB active_file:0kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:20kB unstable:0kB

bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157459]
lowmem_reserve[]: 0 590 590 590



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157475] Node 0
DMA32 free:3060kB min:3068kB low:3832kB high:4600kB
active_anon:277500kB inactive_anon:277728kB active_file:52kB
inactive_file:112kB unevictable:0kB isolated(anon):0k B
isolated(file):0kB present:604988kB mlocked:0kB dirty:16kB

writeback:0kB mapped:100kB shmem:136kB slab_reclaimable:2452kB
slab_unreclaimable:7372kB kernel_stack:752kB pagetables:19348kB
unstable:0kB bounce:0kB writeback_tmp:0kB pa ges_scanned:256
all_unreclaimable? yes



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157503]
lowmem_reserve[]: 0 0 0 0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157518] Node 0 DMA:
1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB

1*2048kB 0*4096kB = 2436kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157553] Node 0
DMA32: 729*4kB 4*8kB 5*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 3060kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157587] 80 total
pagecache pages



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157592] 0 pages in

swap cache



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157597] Swap cache
stats: add 0, delete 0, find 0/0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157602] Free swap
= 0kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.157607] Total swap
= 0kB




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] 157439
pages RAM



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] 4626 pages
reserved



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] 38855 pages
shared




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] 149663
pages non-shared



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] Out of
memory: kill process 2135 (httpd) score 87500 or a child



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.158050] Killed
process 2135 (httpd) vsz:350000kB, anon-rss:49376kB, file-rss:0kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.164960] httpd

invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.164990] httpd
cpuset=/ mems_allowed=0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.164998] Pid: 1824,
comm: httpd Not tainted 2.6.35.14-97.44.amzn1.x86_64 #1



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165006] Call Trace:




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165020]
[] ? cpuset_print_task_mems_allowed+0x98/0xa0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165032]
[] dump_header.clone.1+0x77/0x1a0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165042]
[] ? _raw_spin_unlock_irqrestore+0x19/0x20



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165053]

[] ? ___ratelimit+0x9f/0x120



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165061]
[] oom_kill_process.clone.0+0x76/0x140



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165069]
[] __out_of_memory+0x118/0x190



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165077]
[] out_of_memory+0x82/0x1c0




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165086]
[] __alloc_pages_nodemask+0x689/0x6a0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165096]
[] alloc_pages_current+0x94/0xf0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165105]
[] __page_cache_alloc+0x7f/0x90




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165114]
[] __do_page_cache_readahead+0xc0/0x200



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165122]
[] ra_submit+0x1c/0x20



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165129]
[] filemap_fault+0x3e3/0x430



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165139]

[] __do_fault+0x4f/0x4b0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165147]
[] handle_mm_fault+0x1b4/0xb40



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165156]
[] ? xen_clocksource_read+0x21/0x30



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165164]
[] ? xen_clocksource_get_cycles+0x9/0x10




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165178]
[] ? ktime_get_ts+0xa3/0xd0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165187]
[] do_page_fault+0x112/0x310



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165195]
[] page_fault+0x25/0x30




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165201] Mem-Info:



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165205] Node 0 DMA
per-cpu:



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165212] CPU 0:
hi: 0, btch: 1 usd: 0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165217] Node 0
DMA32 per-cpu:




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165224] CPU 0:
hi: 186, btch: 31 usd: 130



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165232]
active_anon:70836 inactive_anon:71000 isolated_anon:0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165233]
active_file:13 inactive_file:28 isolated_file:0




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165234]
unevictable:0 dirty:4 writeback:0 unstable:0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165235] free:1343
slab_reclaimable:613 slab_unreclaimable:1843



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165236] mapped:25
shmem:34 pagetables:4842 bounce:0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165262] Node 0 DMA

free:2436kB min:76kB low:92kB high:112kB active_anon:6104kB
inactive_anon:6272kB active_file:0kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15772kB mlocked:0kB
dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:20kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165290]
lowmem_reserve[]: 0 590 590 590




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165305] Node 0
DMA32 free:2936kB min:3068kB low:3832kB high:4600kB
active_anon:277240kB inactive_anon:277728kB active_file:52kB
inactive_file:112kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:604988kB mlocked:0kB dirty:16kB
writeback:0kB mapped:100kB shmem:136kB slab_reclaimable:2452kB
slab_unreclaimable:7372kB kernel_stack:752kB pagetables:19348kB
unstable:0kB bounce:0kB writeback_tmp:0kB pa ges_scanned:256
all_unreclaimable? yes




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165334]
lowmem_reserve[]: 0 0 0 0Jul 23 00:19:07 ip-10-117-66-219 kernel:
[19699669.165348] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB
1*25 6kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2436kBJul 23 00:19:07
ip-10-117-66-219 kernel: [19699669.165420] Node 0 DMA32: 710*4kB 4*8kB
4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB
0*4096kB = 2936kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 80 total
pagecache pages




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 0 pages in
swap cache



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] Swap cache
stats: add 0, delete 0, find 0/0



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] Free swap
= 0kB




Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] Total swap
= 0kB



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 157439
pages RAM



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 4626 pages
reserved



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 37922 pages

shared



Jul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648] 149594
pages non-sharedJul 23 00:19:07 ip-10-117-66-219 kernel:
[19699669.165648] Out of memory: kill process 1859 (httpd) score 87366
or a childJul 23 00:19:07 ip-10-117-66-219 kernel: [19699669.165648]
Killed process 1859 (httpd) vsz:349464kB, anon-rss:49312k B,
file-rss:0kB





Please let me know if there is anything additional that I can provide.


Answer



The Linux out-of-memory killer felt obliged to free some RAM. Which it then attempted to do (httpd being the logged target). However, it looks like it was not successful enough and ended up locking up hard, which would not get logged since the server was out of RAM and thus didn't have memory for log-buffers.


Comments

Post a Comment

Popular posts from this blog

linux - iDRAC6 Virtual Media native library cannot be loaded

When attempting to mount Virtual Media on a iDRAC6 IP KVM session I get the following error: I'm using Ubuntu 9.04 and: $ javaws -version Java(TM) Web Start 1.6.0_16 $ uname -a Linux aud22419-linux 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:39:06 UTC 2009 x86_64 GNU/Linux $ firefox -version Mozilla Firefox 3.0.14, Copyright (c) 1998 - 2009 mozilla.org On Windows + IE it (unsurprisingly) works. I've just gotten off the phone with the Dell tech support and I was told it is known to work on Linux + Firefox, albeit Ubuntu is not supported (by Dell, that is). Has anyone out there managed to mount virtual media in the same scenario?
Post a Comment
Read more

hp proliant - Smart Array P822 with HBA Mode?

We get an HP DL360 G8 with an Smart Array P822 controller. On that controller will come a HP StorageWorks D2700 . Does anybody know, that it is possible to run the Smart Array P822 in HBA mode? I found only information about the P410i, who can run HBA. If this is not supported, what you think about the LSI 9207-8e controller? Will this fit good in that setup? The Hardware we get is used but all original from HP. The StorageWorks has 25 x 900 GB SAS 10K disks. Because the disks are not new I would like to use only 22 for raid6, and the rest for spare (I need to see if the disk count is optimal or not for zfs). It would be nice if I'm not stick to SAS in future. As OS I would like to install debian stretch with zfs 0.71 as file system and software raid. I have see that hp has an page for debian to. I would like to use hba mode because it is recommend, that zfs know at most as possible about the disk, and I'm independent from the raid controller. For us zfs have many benefits,
Post a Comment
Read more

apache 2.2 - Server Potentially Compromised -- c99madshell

So, low and behold, a legacy site we've been hosting for a client had a version of FCKEditor that allowed someone to upload the dreaded c99madshell exploit onto our web host. I'm not a big security buff -- frankly I'm just a dev currently responsible for S/A duties due to a loss of personnel. Accordingly, I'd love any help you server-faulters could provide in assessing the damage from the exploit. To give you a bit of information: The file was uploaded into a directory within the webroot, "/_img/fck_uploads/File/". The Apache user and group are restricted such that they can't log in and don't have permissions outside of the directory from which we serve sites. All the files had 770 permissions (user rwx, group rwx, other none) -- something I wanted to fix but was told to hold off on as it wasn't "high priority" (hopefully this changes that). So it seems the hackers could've easily executed the script. Now I wasn't able
Post a Comment
Read more