The Ultimate Guide to High Availability with MariaDB
This is a read-only copy of the MariaDB Knowledgebase generated on 2024-11-16. For the latest, interactive version please visit https://mariadb.com/kb/.

Maria 10.2.18: error with Galera joining the second node

We are trying to create a Galera cluster using "mariabackup". The first node is running. The second node started and got the error:

WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20181005 15:13:33.199)
find: `/u01/mydata/mariadb/e3logintp1/undotbs': No such file or directory
removed `/u01/mydata/mariadb/e3logintp1/ib_logfile3'
removed .......
removed `/u01/mydata/mariadb/e3logintp1/aria_log_control'
removed `/u01/mydata/mariadb/e3logintp1/ibdata1'
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20181005 15:13:33.246)
2018-10-05 15:13:45 140677281212160 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.88.227.147' --datadir '/u01/mydata/mariadb/e3logintp1/'  --defaults-file '/u01/mydata/mariadb/e3logintp1/my.cnf'  --parent '11236' --binlog '/u01/myarch/mariadb/e3logintp1/mysql-bin'  '': 1 (Operation not permitted)
2018-10-05 15:13:45 140677281212160 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2018-10-05 15:13:45 140677875656480 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2018-10-05 15:13:45 140677875656480 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

Here is the my.cnf:

[mysqld]
datadir                         = /u01/mydata/mariadb/e3logintp1
socket                          = /u01/mydata/mariadb/e3logintp1/mysql.sock
pid-file                        = /u01/mydata/mariadb/e3logintp1/mysqld.pid
log-error                       = /u01/mydata/mariadb/e3logintp1/mysqld.err
slow_query_log_file             = /u01/mydata/mariadb/e3logintp1/mysqld-slow.log
general_log_file                = /u01/mydata/mariadb/e3logintp1/mysqld.log
tmpdir                          = /u01/mydata/mariadb/mytmp
autocommit                      = 0
skip-external-locking
key_buffer_size                 = 256M
max_allowed_packet              = 64M
table_open_cache                = 256
sort_buffer_size                = 1M
read_buffer_size                = 1M
read_rnd_buffer_size            = 4M
log_bin_trust_function_creators = TRUE
myisam_sort_buffer_size         = 64M
thread_cache_size               = 8
character-set-server            = utf8mb4
character-set-client-handshake  = false
table_open_cache_instances      = 2
explicit_defaults_for_timestamp = true
lower_case_table_names          = 1
transaction-isolation           = read-committed
expire_logs_days                = 3
log-bin                         = /u01/myarch/mariadb/e3logintp1/mysql-bin
log_slave_updates
max_binlog_size                 = 104857600
innodb_data_home_dir            = /u01/mydata/mariadb/e3logintp1
#innodb_data_file_path           = systemts1:1G:autoextend:max:4G
innodb_log_group_home_dir       = /u01/mydata/mariadb/e3logintp1
innodb_log_file_size            = 100M
innodb_log_files_in_group       = 10
innodb_file_per_table           = 1
innodb_undo_directory           = /u01/mydata/mariadb/e3logintp1/undotbs
innodb_undo_tablespaces         = 1
innodb_undo_logs=1
#innodb_large_prefix             = ON

user                            = e3logint
binlog_format                   = ROW
bind-address                    = 10.88.227.147
port                            = 20306
sql_mode                        = no_engine_substitution

language = /u01/app/mariadb/current/share/english/
plugin_dir = /u01/app/mariadb/current/lib/plugin

innodb_autoinc_lock_mode        = 2
innodb_flush_log_at_trx_commit  = 0
innodb_buffer_pool_size         = 122M
server_id                       = 2

wsrep_sst_method                = mariabackup
wsrep_sst_auth                  = xxxxx:yyyyy

#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on                        = ON
wsrep_provider                  = /u01/app/mariadb/current/lib/libgalera_smm.so
wsrep_provider_options          = "gcache.size=300M; gcache.page_size=300M"
wsrep_cluster_name              = "e3logint"
wsrep_cluster_address           = "gcomm://10.88.227.147,10.88.99.147"
wsrep_node_address              = "10.88.227.147"
wsrep_node_name                 = "e3logintp1"
default_storage_engine          = InnoDB

Thanks in advanced.

Answer Answered by Geoff Montee in this comment.

You'll have to check the SST logs. See mariabackup SST Method: Logs.

The mariabackup SST method has its own logging outside of the MariaDB Server logging.

By default, on the donor node, it logs to innobackup.backup.log. This log file is located in the datadir.

By default, on the joiner node, it logs to innobackup.prepare.log and innobackup.move.log. These log files are located in the .sst directory, which is a hidden directory inside the datadir.

These log files are overwritten by each subsequent SST, so if an SST fails, it is best to copy them somewhere safe before starting another SST, so that the log files can be analyzed. See MDEV-17973 about that.

You can redirect the SST logs to the syslog instead by setting the following in your MariaDB configuration file:

[sst]
sst-syslog=1
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.