Galera Cluster large differences in databases sizes on each server
I'm not sure if this is intended or not (seems unlikley but I'm still new at Galera setup)
Background: We have had a 3 node cluster in production for a few months with no issues, today one of our websites went offline and looking at the database dumps on Server1 (done nightly on each server as a safety backup using mysqldump) it went from 28mb to 428kb overnight.
So I restored the database to Server1 assuming it would replicate out to the cluster however it did not, when I did a dump on Server 2 and 3 they both showed a size of 428kb and not the 28mb. I tried then to do a restore to Server2 only to then notice that Server1 then reduced its database size back to 428kb.
What I found is whatever server I do the database restore on which ever server I am on, the other two then shrink down to the 428kb size and become unusable for this one database (other databases seem unaffected)?
I figured the servers must be out of sync or a split brain since however all servers are showing they are in sync (the commands run on the database server below commands are identical on each server). In the mean time I've pointed the website away from load-balancing and pointed directly to the working node (with the 28mb database)
Any help or advice would be amazing, if you need any logs please let me know what I need to supply as I'm still new to this.
Sync tests:
MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_state_uuid'; +--------------------------+--------------------------------------+ | Variable_name | Value | +--------------------------+--------------------------------------+ | wsrep_cluster_state_uuid | e26d5130-76a1-11e9-ad39-bf3cedba50ad | +--------------------------+--------------------------------------+ 1 row in set (0.00 sec) MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_conf_id'; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | wsrep_cluster_conf_id | 41 | +-----------------------+-------+ 1 row in set (0.00 sec) MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 3 | +--------------------+-------+ 1 row in set (0.01 sec) MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; +----------------------+---------+ | Variable_name | Value | +----------------------+---------+ | wsrep_cluster_status | Primary | +----------------------+---------+ 1 row in set (0.00 sec) MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_ready'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | wsrep_ready | ON | +---------------+-------+ 1 row in set (0.00 sec) MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_connected'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | wsrep_connected | ON | +-----------------+-------+ 1 row in set (0.00 sec)