When 2nd node attempts to add to galera, donor gets: --innobackupex: unknown option '--wsrep-new-cluster'

Hi community,

I am standing up a galera cluster like others have set up, so I am confident in the basic galera.cnf, access, users, etc. I start the first node with "galera_new_cluster" as usual and it stays up and show a cluster size of 1. I add the 2nd node, 172.20.7.212, with "systemctl start mariadb". The first node, being the donor, responds with this in the log:

WSREP_SST: [INFO] Streaming the backup to joiner at 172.20.7.212 4444 (20210428 15:02:25.997)

WSREP_SST: [INFO] Evaluating /usr/bin/mariabackup --innobackupex --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> /var/lib/mysqlmariabackup.backup.log | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 15:02:25.999)

WSREP_SST: [ERROR] /usr/bin/mariabackup finished with error: 2. Check syslog or /var/lib/mysql/mariabackup.backup.log for details (20210428 15:02:26.013)

WSREP_SST: [ERROR] Cleanup after exit with status:22 (20210428 15:02:26.014)

And that 2nd node fails to start. So I look at /var/lib/mysql/mariabackup.backup.log on the donor. That file is strange, it appears to be getting created and deleted very frequently. And it is called/var/lib/mysqlmariabackup.backup.log with two forward slashes as shown. It contains:

--innobackupex: unknown option '--wsrep-new-cluster'

Similar errors show up in the log:

2021-04-28 16:07:58 0 [ERROR] WSREP: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809

2021-04-28 16:07:58 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809: 1 (Operation not permitted)

If I run those commands on the command line I get the same error in the mariabackup.backup.log file

wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809

WSREP_SST: [INFO] Streaming with xbstream (20210428 16:57:09.815)

WSREP_SST: [INFO] Using socat as streamer (20210428 16:57:09.817)

WSREP_SST: [INFO] Using /tmp/tmp.e4YEwI3vbA as xtrabackup temporary directory (20210428 16:57:09.833)

WSREP_SST: [INFO] Using /tmp/tmp.adYYlhKsqj as innobackupex temporary directory (20210428 16:57:09.836)

WSREP_SST: [INFO] Streaming GTID file before SST (20210428 16:57:09.839)

WSREP_SST: [INFO] Evaluating mbstream -c ${INFO_FILE} | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 16:57:09.842)

WSREP_SST: [INFO] Sleeping before data transfer for SST (20210428 16:57:09.848)

WSREP_SST: [INFO] Streaming the backup to joiner at 172.20.7.212 4444 (20210428 16:57:19.850)

WSREP_SST: [INFO] Evaluating /bin/mariabackup --innobackupex --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> /var/lib/mysqlmariabackup.backup.log | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 16:57:19.852) 2021/04/28 16:57:19 socat[25372] E connect(5, AF=2 172.20.7.212:4444, 16): Connection refused

WSREP_SST: [ERROR] /bin/mariabackup finished with error: 2. Check syslog or /var/lib/mysqlmariabackup.backup.log for details (20210428 16:57:19.865)

WSREP_SST: [ERROR] Cleanup after exit with status:22 (20210428 16:57:19.867)

WSREP_SST: [INFO] Cleaning up temporary directories (20210428 16:57:19.869)

Indeed, there is no --wsrep-new-cluster option for mariabackup. Maybe it could be in the $tmpopts or $INNOEXTRA vars?

What could result in this situation and how can a fix or work around it?