Skip to main content

linux - MySQL Tables crashing randomly



This is one of the many random Tables that get corrupted. Any ideas why and what would be causing this?



How do I keep MySQL tables from crashing and MySQL from crashing?



Repairing USR_wp537

USR_wp537.rev_commentmeta OK
USR_wp537.rev_comments OK

USR_wp537.rev_links OK
USR_wp537.rev_options OK
USR_wp537.rev_postmeta OK
USR_wp537.rev_posts
Error : Incorrect key file for table './USR_wp537/rev_posts'; try to repair it
Error : Incorrect key file for table 'rev_posts'; try to repair it
error : Corrupt
USR_wp537.rev_term_relationships OK
USR_wp537.rev_term_taxonomy OK
USR_wp537.rev_terms OK

USR_wp537.rev_usermeta OK
USR_wp537.rev_users


Eventually the only way to fix is to do a mysql> REPAIR TABLE USE_FRM;



This is also mysql 5.5




top - 20:17:11 up 4 days, 8:57, 1 user, load average: 0.41, 0.37,

0.36 Tasks: 204 total, 1 running, 203 sleeping, 0 stopped, 0 zombie Cpu0 : 11.0%us, 0.3%sy, 0.0%ni, 88.7%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st Cpu1 : 5.0%us, 1.0%sy, 0.0%ni, 93.4%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st Cpu2 : 10.4%us, 0.7%sy, 0.0%ni, 89.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 1.3%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 16288440k total, 15292940k used, 995500k free, 1398928k buffers Swap: 8191992k
total, 0k used, 8191992k free, 9351404k cached




my.cnf



[mysqld]
core-file
default-storage-engine=MyISAM

local-infile=0
symbolic-links=0
skip-networking
max_connections = 500
max_user_connections = 40
key_buffer = 500M
myisam_sort_buffer_size = 32M
join_buffer_size = 256M
read_buffer_size = 2M
sort_buffer_size = 2M

read_rnd_buffer_size = 2M
table_cache = 1024
thread_cache_size = 16K
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 256M
max_heap_table_size = 256M
max_allowed_packet = 160M
max_connect_errors = 10
query_cache_limit = 1M

query_cache_size = 64M
query_cache_type = 1
low_priority_updates=1
concurrent_insert=ALWAYS
log-error=/var/log/mysql/error.log
myisam_repair_threads=4
[mysqld_safe]
core-file-size = unlimited
open_files_limit = 8192
log-error=/var/log/mysql/error.log


[mysqldump]
quick
max_allowed_packet = 512M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M



EDIT:
Looks like MySQL for some reason did a restart?



130131 20:30:59 [ERROR] Got an error from thread_id=817995, /builddir/build/BUILD/mysql-5.5.28/mysql-5.5.28/storage/myisam/mi_write.c:223
130131 20:30:59 [ERROR] MySQL thread id 817998, OS thread handle 0x7f8589e2a700, query id 24842525 localhost ahmad_wp Waiting for table level lock
SELECT option_value FROM wp_options WHERE option_name = 'widget_momizat-posts-images' LIMIT 1
130131 20:30:59 [ERROR] MySQL thread id 817995, OS thread handle 0x7f858692f700, query id 24842524 localhost ahmad_wp update
INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('rewrite_rules', 'a:91:{s:27:\"typename/([0-9]{4})/(.+)/?$\";s:30:\"index.php?typename=$matches[2]\";s:47:\"category/(.+?)/feed/(feed|rdf|rss|rss2|atom)/?$\";s:52:\"index.php?category_name=$matches[1]&feed=$matches[2]\";s:42:\"category/(.+?)/(feed|rdf|rss|rss2|atom)/?$\";s:52:\"index.php?category_name=$matches[1]&feed=$matches[2]\";s:35:\"category/(.+?)/page/?([0-9]{1,})/?$\";s:53:\"index.php?category_name=$matches[1]&paged=$matches[2]\";s:17:\"category/(.+?)/?$\";s:35:\"index.php?category_name=$matches[1]\";s:44:\"tag/([^/]+)/feed/(feed|rdf|rss|rss2|atom)/?$\";s:42:\"index.php?tag=$matches[1]&feed=$matches[2]\";s:39:\"tag/([^/]+)/(feed|rdf|rss|rss2|atom)/?$\";s:42:\"index.php?tag=$matches[1]&feed=$matches[2]\";s:32:\"tag/([^/]+)/page/?([0-9]{1,})/?$\";s:43:\"index.php?tag=$matches[1]&paged=$matches[2]\";s:14:\"tag/([^/]+)/?$\";s:25:

130131 20:33:42 mysqld_safe Number of processes running now: 0
130131 20:33:42 mysqld_safe mysqld restarted
130131 20:33:42 [Note] libgovernor.so found
130131 20:33:42 [Note] All governors functions found too
130131 20:33:42 [ERROR] Governor not connected
130131 20:33:42 [Note] All governors lve functions found too
130131 20:33:42 [Note] Plugin 'FEDERATED' is disabled.
130131 20:33:42 InnoDB: The InnoDB memory heap is disabled
130131 20:33:42 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130131 20:33:42 InnoDB: Compressed tables use zlib 1.2.3

130131 20:33:42 InnoDB: Using Linux native AIO
130131 20:33:42 InnoDB: Initializing buffer pool, size = 128.0M
130131 20:33:42 InnoDB: Completed initialization of buffer pool
130131 20:33:42 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130131 20:33:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite

InnoDB: buffer...
130131 20:33:42 InnoDB: Waiting for the background threads to start
130131 20:33:43 InnoDB: 1.1.8 started; log sequence number 5726524860
130131 20:33:43 [Note] Event Scheduler: Loaded 1 event
130131 20:33:43 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.28-cll-lve' socket: '/var/lib/mysql/mysql.sock' port: 0 MySQL Community Server (GPL)


EDIT: hard drives are clean




EDIT: using mcelog to check RAM



EDIT: the following was used to check for issues in messages



  902  fgrep -i seg /var/log/messages
903 fgrep -i mce /var/log/messages
904 tail /var/log/mysql/error.log
906 smartctl --all /dev/sda
907 smartctl -t short /dev/sda
908 tail /var/log/mysql/error.log

910 smartctl --all /dev/sda
911 fgrep -i i/o /var/log/messages
912 fgrep -i i/o /var/log/messages*
913 fgrep -i sda /var/log/messages*
914 fgrep -i sense /var/log/messages*
915 egrep 'sda.*Error' sense /var/log/messages*
916 egrep -h 'sda.*Error' sense /var/log/messages*
917 egrep -h 'sda.*Error' /var/log/messages*

df /tmp

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 4031680 97304 3729576 3% /tmp

Answer



Contrary to popular belief, MySQL tables don't just crash for no reason. Crashes are often attributed to high load or too many rows. That's crap.



If you have crashed tables then there is an underlying problem with the server, such as bad memory or a bad hard drive. It could also be a bug in the MySQL server but that is very unlikely. If you are a customer of a web host there is nothing you can do except get your host to fix the underlying problem.


Comments

Popular posts from this blog

linux - iDRAC6 Virtual Media native library cannot be loaded

When attempting to mount Virtual Media on a iDRAC6 IP KVM session I get the following error: I'm using Ubuntu 9.04 and: $ javaws -version Java(TM) Web Start 1.6.0_16 $ uname -a Linux aud22419-linux 2.6.28-15-generic #51-Ubuntu SMP Mon Aug 31 13:39:06 UTC 2009 x86_64 GNU/Linux $ firefox -version Mozilla Firefox 3.0.14, Copyright (c) 1998 - 2009 mozilla.org On Windows + IE it (unsurprisingly) works. I've just gotten off the phone with the Dell tech support and I was told it is known to work on Linux + Firefox, albeit Ubuntu is not supported (by Dell, that is). Has anyone out there managed to mount virtual media in the same scenario?

hp proliant - Smart Array P822 with HBA Mode?

We get an HP DL360 G8 with an Smart Array P822 controller. On that controller will come a HP StorageWorks D2700 . Does anybody know, that it is possible to run the Smart Array P822 in HBA mode? I found only information about the P410i, who can run HBA. If this is not supported, what you think about the LSI 9207-8e controller? Will this fit good in that setup? The Hardware we get is used but all original from HP. The StorageWorks has 25 x 900 GB SAS 10K disks. Because the disks are not new I would like to use only 22 for raid6, and the rest for spare (I need to see if the disk count is optimal or not for zfs). It would be nice if I'm not stick to SAS in future. As OS I would like to install debian stretch with zfs 0.71 as file system and software raid. I have see that hp has an page for debian to. I would like to use hba mode because it is recommend, that zfs know at most as possible about the disk, and I'm independent from the raid controller. For us zfs have many benefits,

apache 2.2 - Server Potentially Compromised -- c99madshell

So, low and behold, a legacy site we've been hosting for a client had a version of FCKEditor that allowed someone to upload the dreaded c99madshell exploit onto our web host. I'm not a big security buff -- frankly I'm just a dev currently responsible for S/A duties due to a loss of personnel. Accordingly, I'd love any help you server-faulters could provide in assessing the damage from the exploit. To give you a bit of information: The file was uploaded into a directory within the webroot, "/_img/fck_uploads/File/". The Apache user and group are restricted such that they can't log in and don't have permissions outside of the directory from which we serve sites. All the files had 770 permissions (user rwx, group rwx, other none) -- something I wanted to fix but was told to hold off on as it wasn't "high priority" (hopefully this changes that). So it seems the hackers could've easily executed the script. Now I wasn't able