User:Brett Wuth/Running Notes/2016-03

From PCSAR

< User:Brett Wuth | Running Notes(Difference between revisions)
Jump to: navigation, search
(Add Members:2016-03-17_15_43_00u-scan.pdf)
Current revision (06:38, 25 August 2016) (edit) (undo)
(Add Members:2016-03-18 phone.com statement.pdf)
 
(23 intermediate revisions not shown.)
Line 19: Line 19:
* {{link|Image:Members:2016-03-07_12_07_30u-scan.pdf}}
* {{link|Image:Members:2016-03-07_12_07_30u-scan.pdf}}
* {{link|Image:Members:2016-03-08_05_36_04u-scan.pdf}}
* {{link|Image:Members:2016-03-08_05_36_04u-scan.pdf}}
 +
 +
 +
== 2016-03-14 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-14 Delorme InvoiceIR08218441.pdf}}
Line 31: Line 37:
documents:
documents:
* {{link|Image:Members:2016-03-17_15_43_00u-scan.pdf}}
* {{link|Image:Members:2016-03-17_15_43_00u-scan.pdf}}
 +
 +
 +
== 2016-03-18 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-18 phone.com statement.pdf}}
 +
 +
 +
== 2016-03-19 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-19_12_39_41u-scan.pdf}}
== 2016-03-23 ==
== 2016-03-23 ==
* {{subpage|2016-03-23 Administrator|Phone meeting with Administrator}}
* {{subpage|2016-03-23 Administrator|Phone meeting with Administrator}}
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-23_10_45_08u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-23_10_48_58u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-23_10_49_55u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-23_10_54_24u-scan0013.pdf}}
 +
* {{link|Image:Members:2016-03-23_11_01_05u-scan.pdf}}
 +
== 2016-03-26 ==
 +
Susanna reports website not working.
 +
 +
wiki pages show
 +
<pre>
 +
PCSAR has a problem
 +
 +
Sorry! This site is experiencing technical difficulties.
 +
 +
Try waiting a few minutes and reloading.
 +
 +
(Can't contact the database server: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (localhost))
 +
</pre>
 +
 +
<pre>
 +
pcsar:/var/log/apache2# ps ax | grep mysql
 +
4395 pts/0 S<+ 0:00 grep mysql
 +
pcsar:/var/log/apache2# /etc/init.d/mysql stop
 +
Stopping MySQL database server: mysqld.
 +
pcsar:/var/log/apache2# /etc/init.d/mysql start
 +
Starting MySQL database server: mysqld ..
 +
Checking for corrupt, not cleanly closed and upgrade needing tables..
 +
pcsar:/var/log/apache2# ps ax | grep mysql
 +
4440 pts/0 S< 0:00 /bin/sh /usr/bin/mysqld_safe
 +
4477 pts/0 S<l 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
 +
4478 pts/0 S< 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
 +
4566 pts/0 S<+ 0:00 grep mysql
 +
pcsar:/var/log/apache2#
 +
</pre>
 +
 +
Mysql is not running. Starting it fixes the problem.
 +
 +
About once every 20 to 60 minutes
 +
log shows:
 +
<pre>
 +
Mar 25 14:35:58 pcsar mysqld_safe[3004]: Number of processes running now: 0
 +
Mar 25 14:35:59 pcsar mysqld_safe[3006]: restarted
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: The log sequence number in ibdata files does not match
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: the log sequence number in the ib_logfiles!
 +
Mar 25 14:36:04 pcsar mysqld[3009]: 160325 14:36:04 InnoDB: Database was not shut down normally!
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Starting crash recovery.
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Reading tablespace information from the .ibd files...
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Restoring possible half-written data pages from the doublewrite
 +
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: buffer...
 +
Mar 25 14:36:13 pcsar mysqld[3009]: 160325 14:36:13 InnoDB: Started; log sequence number 02469806575
 +
Mar 25 14:36:19 pcsar mysqld[3009]: 160325 14:36:19 [Note] /usr/sbin/mysqld: ready for connections.
 +
Mar 25 14:36:19 pcsar mysqld[3009]: Version: '5.0.51a-24+lenny5' socket: '/var/run/mysql/mysqld.sock' port: 3306 (Debian)
 +
</pre>
 +
last sequence before hang:
 +
<pre>
 +
Mar 25 15:40:42 pcsar mysqld_safe[3131]: Number of processes running now: 0
 +
Mar 25 15:40:43 pcsar mysqld_safe[3133]: restarted
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: The log sequence number in ibdata files does not match
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: the log sequence number in the ib_logfiles!
 +
Mar 25 15:40:50 pcsar mysqld[3136]: 160325 15:40:50 InnoDB: Database was not shut down normally!
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Starting crash recovery.
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Reading tablespace information from the .ibd files...
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Restoring possible half-written data pages from the doublewrite
 +
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: buffer...
 +
Mar 25 15:40:56 pcsar mysqld[3136]: 160325 15:40:56 InnoDB: Started; log sequence number 0 2469847120
 +
Mar 25 15:41:03 pcsar mysqld[3136]: 160325 15:41:03 [Note] /usr/sbin/mysqld: ready for connections.
 +
Mar 25 15:41:03 pcsar mysqld[3136]: Version: '5.0.51a-24+lenny5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian)
 +
Mar 25 15:42:22 pcsar mysqld_safe[3154]: Number of processes running now: 0
 +
Mar 25 15:42:27 pcsar mysqld_safe[3156]: restarted
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: The log sequence number in ibdata files does not match
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: the log sequence number in the ib_logfiles!
 +
Mar 25 15:42:39 pcsar mysqld[3159]: 160325 15:42:38 InnoDB: Database was not shut down normally!
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Starting crash recovery.
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Reading tablespace information from the .ibd files...
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Restoring possible half-written data pages from the doublewrite
 +
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: buffer...
 +
Mar 25 15:42:44 pcsar mysqld[3159]: 160325 15:42:44 InnoDB: Started; log sequence number 0 2469847120
 +
Mar 25 15:42:47 pcsar mysqld[3159]: 160325 15:42:47 [Note] /usr/sbin/mysqld: ready for connections.
 +
Mar 25 15:42:47 pcsar mysqld[3159]: Version: '5.0.51a-24+lenny5' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian)
 +
Mar 25 15:45:26 pcsar mysqld_safe[3176]: Number of processes running now: 0
 +
Mar 25 15:45:26 pcsar mysqld_safe[3178]: restarted
 +
Mar 25 15:45:51 pcsar mysqld_safe[3183]: ended
 +
</pre>
 +
Suspect same issue as {{link|Members:Server/Web server/Issue/2016-02-27 Too much memory used}}.
 +
Processes are being killed because run out of memory. Eventually that's mysqld_safe, so no further recovery.
 +
== 2016-03-27 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-27_19_14_57u-scan.pdf}}
 +
== 2016-03-28 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-28_21_24_48u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_25_52u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_27_24u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_28_22u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_29_31u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_31_24u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_33_13u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_35_41u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_38_27u-scan0015.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_40_59u-scan0030.pdf}}
 +
* {{link|Image:Members:2016-03-28_21_43_31u-scan0045.pdf}}
 +
* {{link|Image:Members:2016-03-29_03_48_38u-scan.pdf}}
 +
* {{link|Image:Members:2016-03-29_04_10_24u-scan.pdf}}
 +
== 2016-03-30 ==
 +
 +
documents:
 +
* {{link|Image:Members:2016-03-30_14_16_46u-scan.pdf}}

Current revision

Contents

[edit] 2016-03-01

documents:


[edit] 2016-03-05

documents:


[edit] 2016-03-07

documents:


[edit] 2016-03-14

documents:


[edit] 2016-03-16

documents:


[edit] 2016-03-17

documents:


[edit] 2016-03-18

documents:


[edit] 2016-03-19

documents:


[edit] 2016-03-23

documents:

[edit] 2016-03-26

Susanna reports website not working.

wiki pages show 

PCSAR has a problem

Sorry! This site is experiencing technical difficulties.

Try waiting a few minutes and reloading.

(Can't contact the database server: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (localhost))

pcsar:/var/log/apache2# ps ax | grep mysql
 4395 pts/0    S<+    0:00 grep mysql
pcsar:/var/log/apache2# /etc/init.d/mysql stop
Stopping MySQL database server: mysqld.
pcsar:/var/log/apache2# /etc/init.d/mysql start
Starting MySQL database server: mysqld ..
Checking for corrupt, not cleanly closed and upgrade needing tables..
pcsar:/var/log/apache2# ps ax | grep mysql
 4440 pts/0    S<     0:00 /bin/sh /usr/bin/mysqld_safe
 4477 pts/0    S<l    0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
 4478 pts/0    S<     0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
 4566 pts/0    S<+    0:00 grep mysql
pcsar:/var/log/apache2#

Mysql is not running. Starting it fixes the problem.

About once every 20 to 60 minutes log shows: 

Mar 25 14:35:58 pcsar mysqld_safe[3004]: Number of processes running now: 0
Mar 25 14:35:59 pcsar mysqld_safe[3006]: restarted
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 14:36:04 pcsar mysqld[3009]: 160325 14:36:04  InnoDB: Database was not shut down normally!
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Starting crash recovery.
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 14:36:04 pcsar mysqld[3009]: InnoDB: buffer...
Mar 25 14:36:13 pcsar mysqld[3009]: 160325 14:36:13  InnoDB: Started; log sequence number 02469806575
Mar 25 14:36:19 pcsar mysqld[3009]: 160325 14:36:19 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 14:36:19 pcsar mysqld[3009]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysql/mysqld.sock'  port: 3306  (Debian)

last sequence before hang: 

Mar 25 15:40:42 pcsar mysqld_safe[3131]: Number of processes running now: 0
Mar 25 15:40:43 pcsar mysqld_safe[3133]: restarted
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 15:40:50 pcsar mysqld[3136]: 160325 15:40:50  InnoDB: Database was not shut down normally!
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Starting crash recovery.
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 15:40:50 pcsar mysqld[3136]: InnoDB: buffer...
Mar 25 15:40:56 pcsar mysqld[3136]: 160325 15:40:56  InnoDB: Started; log sequence number 0 2469847120
Mar 25 15:41:03 pcsar mysqld[3136]: 160325 15:41:03 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 15:41:03 pcsar mysqld[3136]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Mar 25 15:42:22 pcsar mysqld_safe[3154]: Number of processes running now: 0
Mar 25 15:42:27 pcsar mysqld_safe[3156]: restarted
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: The log sequence number in ibdata files does not match
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: the log sequence number in the ib_logfiles!
Mar 25 15:42:39 pcsar mysqld[3159]: 160325 15:42:38  InnoDB: Database was not shut down normally!
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Starting crash recovery.
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Reading tablespace information from the .ibd files...
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 25 15:42:39 pcsar mysqld[3159]: InnoDB: buffer...
Mar 25 15:42:44 pcsar mysqld[3159]: 160325 15:42:44  InnoDB: Started; log sequence number 0 2469847120
Mar 25 15:42:47 pcsar mysqld[3159]: 160325 15:42:47 [Note] /usr/sbin/mysqld: ready for connections.
Mar 25 15:42:47 pcsar mysqld[3159]: Version: '5.0.51a-24+lenny5'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Mar 25 15:45:26 pcsar mysqld_safe[3176]: Number of processes running now: 0
Mar 25 15:45:26 pcsar mysqld_safe[3178]: restarted
Mar 25 15:45:51 pcsar mysqld_safe[3183]: ended

Suspect same issue as 2016-02-27 Too much memory used. Processes are being killed because run out of memory. Eventually that's mysqld_safe, so no further recovery.

[edit] 2016-03-27

documents:

[edit] 2016-03-28

documents:

[edit] 2016-03-30

documents:

Retrieved from "http://pcsar.dyndns.org/mediawiki/index.php/User:Brett_Wuth/Running_Notes/2016-03"
Personal tools