The Ultimate Guide to High Availability with MariaDB
This is a read-only copy of the MariaDB Knowledgebase generated on 2024-11-16. For the latest, interactive version please visit https://mariadb.com/kb/.

MariaDB crashes on force shutdown of server (cant open gtid_slave_pos)

Hi, we have an enterprise server where we run MariaDB 10.1.14. We are upgrading the OS on our server and in order to do that, we are taking backup of MariaDB through dump as well taking binary backup.

Once OS installation is done, we restore the database using dump and once database is restarted we get error messages such as

Logs:

[Warning] InnoDB: Cannot open table mysql/gtid_slave_pos from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.

2017-05-23 2:36:19 140199378811648 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine

2017-05-23 2:36:19 140199022545664 [Warning] InnoDB: Cannot open table mysql/gtid_slave_pos from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.

2017-05-23 2:36:19 140199022545664 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine

2017-05-23 02:36:20 7f82a0efcb00 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

It generally doesnt seems to cause much problem. But if we do force shutdown of our servers to replicate power failure scenario, after start up MariaDB refuses to come up with below errors

Crash Logs

2017-05-24 2:24:18 140306666678400 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace MyDB/My_backup uses space ID: 3 at filepath: ./MyDB/My_backup.ibd. Cannot open tablespace mysql/gtid_slave_pos which uses space ID: 3 at filepath: ./mysql/gtid_slave_pos.ibd

2017-05-24 02:24:18 7f9bb106e880 InnoDB: Operating system error number 2 in a file operation.

InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./mysql/gtid_slave_pos.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here.

I have following question in this regard

1. Is it really gtid_slave_pos which is causing the crash

2. I can recover if I use innodb_force_recovery, take dump, deleted ib files and restart database. However it will be good if at first place we do not come across this situation at all.

3. Is there a way where in after restore I can make sure these errors do not appear at all?

4. Is it normal for innodb to crash in case of power failure and not recover? I know innodb tries its best to do crash recovery.

Any kind of help in this reagrad will be appreciated.

Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.