User:Brett Wuth/Running Notes/2016-03

From PCSAR

Jump to: navigation, search

Contents

[edit] 2016-03-01

documents:


[edit] 2016-03-05

documents:


[edit] 2016-03-07

documents:


[edit] 2016-03-14

documents:


[edit] 2016-03-16

documents:


[edit] 2016-03-17

documents:


[edit] 2016-03-18

documents:


[edit] 2016-03-19

documents:


[edit] 2016-03-23

documents:

[edit] 2016-03-26

Susanna reports website not working.

wiki pages show 

PCSAR has a problem

Sorry! This site is experiencing technical difficulties.

Try waiting a few minutes and reloading.

(Can't contact the database server: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (localhost))

pcsar:/var/log/apache2# ps ax | grep mysql
 4395 pts/0    S<+    0:00 grep mysql
pcsar:/var/log/apache2# /etc/init.d/mysql stop
Stopping MySQL database server: mysqld.
pcsar:/var/log/apache2# /etc/init.d/mysql start
Starting MySQL database server: mysqld ..
Checking for corrupt, not cleanly closed and upgrade needing tables..
pcsar:/var/log/apache2# ps ax | grep mysql
 4440 pts/0    S<     0:00 /bin/sh /usr/bin/mysqld_safe
 4477 pts/0    S<l    0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
 4478 pts/0    S<     0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
 4566 pts/0    S<+    0:00 grep mysql
pcsar:/var/log/apache2#

Mysql is not running. Starting it fixes the problem.

About once every 20 to 60 minutes log shows: 

Mar 25 14:35:58 pcsar mysqld_safe[3004]: Number of processes running now: 0
Mar 25 14:35:59 pcsar mysqld_safe[3006]: restarted
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 14:36:04 pcsar mysqld[3009]: 160325 14:36:04  InnoDB: Database was not shut down normally!
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Starting crash recovery.
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: buffer...
Mar 25 14:36:13 pcsar mysqld[3009]: 160325 14:36:13  InnoDB: Started; log sequence number 02469806575
Mar 25 14:36:19 pcsar mysqld[3009]: 160325 14:36:19 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 14:36:19 pcsar mysqld[3009]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysql/mysqld.sock'  port: 3306  (Debian)

last sequence before hang: 

Mar 25 15:40:42 pcsar mysqld_safe[3131]: Number of processes running now: 0
Mar 25 15:40:43 pcsar mysqld_safe[3133]: restarted
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 15:40:50 pcsar mysqld[3136]: 160325 15:40:50  InnoDB: Database was not shut down normally!
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Starting crash recovery.
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: buffer...
Mar 25 15:40:56 pcsar mysqld[3136]: 160325 15:40:56  InnoDB: Started; log sequence number 0 2469847120
Mar 25 15:41:03 pcsar mysqld[3136]: 160325 15:41:03 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 15:41:03 pcsar mysqld[3136]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Mar 25 15:42:22 pcsar mysqld_safe[3154]: Number of processes running now: 0
Mar 25 15:42:27 pcsar mysqld_safe[3156]: restarted
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 15:42:39 pcsar mysqld[3159]: 160325 15:42:38  InnoDB: Database was not shut down normally!
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Starting crash recovery.
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: buffer...
Mar 25 15:42:44 pcsar mysqld[3159]: 160325 15:42:44  InnoDB: Started; log sequence number 0 2469847120
Mar 25 15:42:47 pcsar mysqld[3159]: 160325 15:42:47 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 15:42:47 pcsar mysqld[3159]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Mar 25 15:45:26 pcsar mysqld_safe[3176]: Number of processes running now: 0
Mar 25 15:45:26 pcsar mysqld_safe[3178]: restarted
Mar 25 15:45:51 pcsar mysqld_safe[3183]: ended

Suspect same issue as 2016-02-27 Too much memory used. Processes are being killed because run out of memory. Eventually that's mysqld_safe, so no further recovery.

[edit] 2016-03-27

documents:

[edit] 2016-03-28

documents:

[edit] 2016-03-30

documents:

Retrieved from "http://pcsar.dyndns.org:8080/mediawiki/index.php/User:Brett_Wuth/Running_Notes/2016-03"
Personal tools