MaxScale 21.06 ColumnStore Monitor
ColumnStore Monitor
The ColumnStore monitor, csmon
, is a monitor module for MariaDB ColumnStore
servers. The monitor supports ColumnStore version 1.5.
Required Grants
The credentials defined with the user
and password
parameters must have all
grants on the infinidb_vtable
database.
For example, to create a user for this monitor with the required grants execute the following SQL.
CREATE USER 'maxscale'@'maxscalehost' IDENTIFIED BY 'maxscale-password'; GRANT ALL ON infinidb_vtable.* TO 'maxscale'@'maxscalehost';
Configuration
Read the Monitor Common document for a list of supported common monitor parameters.
version
With this deprecated optional parameter the used ColumnStore version is
specified. The only allowed value is 1.5
.
admin_port
This optional parameter specifies the port of the ColumnStore administrative
daemon. The default value is 8640
. Note that the daemons of all nodes must
be listening on the same port.
admin_base_path
This optional parameter specifies the base path of the ColumnStore
administrative daemon. The default value is /cmapi/0.4.0
.
api_key
This optional parameter specifies the API key to be used in the
communication with the ColumnStore administrative daemon. If no
key is specified, then a key will be generated and stored to the
file api_key.txt
in the directory with the same name as the
monitor in data directory of MaxScale. Typically that will
be /var/lib/maxscale/<monitor-section>/api_key.txt
.
Note that ColumnStore will store the first key provided and thereafter require it, so changing the key requires the resetting of the key on the ColumnStore nodes as well.
local_address
With this parameter it is specified what IP MaxScale should
tell the ColumnStore nodes it resides at. Either it or
local_address
at the global level in the MaxScale
configuration file must be specified. If both have been
specified, then the one specified for the monitor overrides.
dynamic_node_detection
This optional boolean parameter specifies whether the monitor should
autonomously figure out the ColumnStore cluster configuration or whether
it should solely rely upon the monitor configuration in the configuration
file. Please see Dynamic Node Detection for a
thorough discussion on the meaning of the parameter. The default value
is false
.
cluster_monitor_interval
This optional parameter, meaningful only if dynamic_node_detection
is
true
specifies how often the monitor should probe the ColumnStore
cluster and adapt to any changes that have occurred in the number of
nodes of the cluster. The default value is 10s
, that is, the
cluster configuration is probed every 10 seconds.
Note that as the probing is performed at the regular monitor round,
the value should be some multiple of monitor_interval
.
Dynamic Node Detection
NOTE If dynamic node detection is used, the network setup must be such that the hostname/IP-address of a ColumnStore node is the same when viewed both from MaxScale and from another node.
By default, the ColumnStore monitor behaves like the regular MariaDB monitor. That is, it only monitors the servers it has been configured with.
If dynamic_node_detection
has been enabled, the behaviour of the monitor
changes significantly. Instead of being explicitly told which servers it
should monitor, the monitor is only told how to get into contact with the
cluster whereafter it autonomously figures out the cluster configuration
and creates dynamic server entries accordingly.
When dynamic node detection is enabled, the servers the monitor has been configured with are only used for "bootstrapping" the monitor, because at the initial startup the monitor does not otherwise know how to get into contact with the cluster.
In the following is shown a configuration using dynamic node detection.
[CsBootstrap1] type=server address=mcs1 port=3306 protocol=mariadbbackend [CsBootstrap2] type=server address=mcs2 port=3306 protocol=mariadbbackend [CsMonitor] type=monitor module=csmon servers=CsBootstrap1, CsBootstrap2 dynamic_node_detection=true ...
As can be seen, the server entries look just like any other server entries, but to make them stand out and to indicate what they are used for, they have the word bootstrap in their name.
In principle, it is sufficient with a single entry, but to cater for the case that a node happens to be down, it is adviseable to have more than one. Once the monitor has been able to connect to a node, it will fetch the configuration and store information about the nodes locally. On subsequent startups, the monitor will use the bootstrap information only if it cannot connect using the persisted information. Also, if there has been any change in the bootstrap servers, the persisted information is not used.
Based on the information obtained from the cluster itself, the monitor
will create dynamic server instances that are named as @@
followed by
the monitor name, followed by a :
, followed by the hostname.
If the cluster in fact consists of three nodes, then the output of
maxctrl list servers
may look like
┌──────────────────┬─────────┬──────┬─────────────┬─────────────────┬──────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├──────────────────┼─────────┼──────┼─────────────┼─────────────────┼──────┤ │ @@CSMonitor:mcs2 │ mcs2 │ 3306 │ 0 │ Slave, Running │ │ ├──────────────────┼─────────┼──────┼─────────────┼─────────────────┼──────┤ │ @@CSMonitor:mcs3 │ mcs3 │ 3306 │ 0 │ Master, Running │ │ ├──────────────────┼─────────┼──────┼─────────────┼─────────────────┼──────┤ │ @@CSMonitor:mcs1 │ mcs1 │ 3306 │ 0 │ Slave, Running │ │ ├──────────────────┼─────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsBootstrap1 │ mcs1 │ 3306 │ 0 │ Slave, Running │ │ ├──────────────────┼─────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsBootstrap2 │ mcs2 │ 3306 │ 0 │ Slave, Running │ │ └──────────────────┴─────────┴──────┴─────────────┴─────────────────┴──────┘
Note that there will be dynamic server entries also for the nodes for which there is a bootstrap entry.
When the service is defined, it is imperative that it does not explicitly
refer to either the bootstrap or the dynamic entries. Instead, it should
refer to the monitor using the cluster
parameter.
[RWS] type=service router=readwritesplit cluster=CsMonitor ...
With this configuration the RWS service will automatically adapt to any changes made to the ColumnStore cluster.
Commands
The ColumnStore monitor provides module commands using which the ColumnStore cluster can be managed. The commands can be invoked using the REST-API with a client such as curl or using maxctrl.
All commands require the monitor instance name as the first parameters. Additional parameters must be provided depending on the command.
Note that as maxctrl itself has a timeout of 10 seconds, if a timeout larger than that is provided to any command, the timeout of maxctrl must also be increased. For instance:
maxctrl --timeout 30s call command csmon shutdown CsMonitor 20s
Here a 30 second timeout is specified for maxctrl to ensure that it does not expire before the timeout of 20s provided for the shutdown command possibly does.
The output is always a JSON object.
In the following, assume a configuration like this:
[CsNode1] type=server ... [CsNode2] type=server ... [CsMonitor] type=monitor module=csmon servers=CsNode1,CsNode2 ...
start
Starts the ColumnStore cluster.
call command csmon start <monitor-name> <timeout>
Example
call command csmon start CsMonitor 20s
shutdown
Shuts down the ColumnStore cluster.
call command csmon shutdown <monitor-name> <timeout>
Example
call command csmon shutdown CsMonitor 20s
status
Get the status of the ColumnStore cluster.
call command csmon status <monitor-name> [<server>]
Returns the status of the cluster or the status of a specific server.
Example
call command csmon status CsMonitor call command csmon status CsMonitor CsNode1
mode-set
Sets the mode of the cluster.
call command csmon mode-set <monitor-name> (readonly|readwrite) <timeout>
Example
call command csmon mode-set CsMonitor readonly 20s
config-get
Returns the cluster configuration.
call command csmon config-get <monitor-name> [<server-name>]
If no server is specified, the configuration is fetched from the first server in the monitor configuration, otherwise from the specified server.
Note that if everything is in order, the returned configuration should be identical regardless of the server it is fetched from.
Example
call command csmon config-get CsMonitor CsNode2
add-node
Adds a new node located on the server at the hostname or IP host to the ColumnStore cluster.
call command csmon add-node <monitor-name> <host> <timeout>
Example
call command csmon add-node CsMonitor mcs2 20s
For a more complete example, please refer to adding a node.
remove-node
Remove the node located on the server at the hostname or IP host from the ColumnStore cluster.
call command csmon remove-node <monitor-name> <host> <timeout>
Example
call command csmon remove-node CsMonitor mcs2 20s
For a more complete example, please refer to removing a node.
Example
The following is an example of a csmon
configuration.
[CSMonitor] type=monitor module=csmon version=1.5 servers=CsNode1,CsNode2 user=myuser password=mypwd monitor_interval=5000 api_key=somekey1234
Adding a Node
Note that in the following dynamic_node_detection
is not used, but
the monitor is configured in the traditional way. The impact of
dynamic_node_detection
is described here.
Adding a new node to a ColumnStore cluster can be performed dynamically at runtime, but it must be done in two steps. First, the node is added to ColumnStore and then, the corresponding server object (that possibly has to be created) in the MaxScale configuration is added to the ColumnStore monitor.
In the following, assume a two node ColumnStore cluster and an initial MaxScale configuration like.
[CsNode1] type=server ... [CsNode2] type=server ... [CsMonitor] type=monitor module=csmon servers=CsNode1,CsNode2 ...
Invoking maxctrl list servers
will now show:
┌─────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode1 │ 10.10.10.10 │ 3306 │ 0 │ Master, Running │ │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode2 │ 10.10.10.11 │ 3306 │ 0 │ Slave, Running │ │ └─────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
If we now want to add a new ColumnStore node, located at mcs3/10.10.10.12
to the cluster, the steps are as follows.
First the node is added
maxctrl --timeout 30s call command csmon add-node CsMonitor mcs3 20s
After a while the following is output:
{ "links": { "self": "http://localhost:8989/v1/maxscale/modules/csmon/add-node" }, "meta": { "message": "Node mcs3 successfully added to cluster.", "result": { "node_id": "mcs3", "timestamp": "2020-08-07 10:03:49.474539" }, "success": true } }
At this point, the ColumnStore cluster consists of three nodes. However, the ColumnStore monitor is not yet aware of the new node.
First we need to create the corresponding server object.
maxctrl create server CsNode3 10.10.10.12
Invoking maxctrl list servers
will now show:
┌─────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode3 │ 10.10.10.12 │ 3306 │ 0 │ Down │ │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode1 │ 10.10.10.10 │ 3306 │ 0 │ Master, Running │ │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode2 │ 10.10.10.11 │ 3306 │ 0 │ Slave, Running │ │ └─────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
The server CsNode3
has been created, but its state is Down
since
it is not yet being monitored.
┌───────────┬─────────┬──────────────────┐ │ Monitor │ State │ Servers │ ├───────────┼─────────┼──────────────────┤ │ CsMonitor │ Running │ CsNode1, CsNode2 │ └───────────┴─────────┴──────────────────┘
It must now be added to the monitor.
maxctrl link monitor CsMonitor CsNode3
Now the server is monitored and maxctrl list monitors
shows:
┌───────────┬─────────┬───────────────────────────┐ │ Monitor │ State │ Servers │ ├───────────┼─────────┼───────────────────────────┤ │ CsMonitor │ Running │ CsNode1, CsNode2, CsNode3 │ └───────────┴─────────┴───────────────────────────┘
The state of the new node is now also set correctly, as shown by
maxctrl list servers
.
┌─────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode3 │ 10.10.10.12 │ 3306 │ 0 │ Slave, Running │ │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode1 │ 10.10.10.10 │ 3306 │ 0 │ Master, Running │ │ ├─────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤ │ CsNode2 │ 10.10.10.11 │ 3306 │ 0 │ Slave, Running │ │ └─────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
Note that the MaxScale server object can be created at any point, but
it must not be added to the monitor before the node has been added to
the ColumnStore cluster using call command csmon add-node
.
Impact of dynamic_node_detection
If dynamic_node_detection
is enabled, there is no need to create any
explicit server entries. All that needs to be done, is to add the node
and the monitor will adapt automatically. Note that it does not matter
whether the node is added indirectly via maxscale or directly using the
REST-API of ColumnStore. The only difference is that in the former case,
MaxScale may detect the new situation slightly faster.
Removing a Node
Note that in the following dynamic_node_detection
is not used, but
the monitor is configured in the traditional way. The impact of
dynamic_node_detection
is described here.
Removing a node should be performed in the reverse order of how a node was added. First, the MaxScale server should be removed from the monitor. Then, the node should be removed from the ColumnStore cluster.
Suppose we want to remove the ColumnStore node at mcs2/10.10.10.12
and the current situation is as:
┌───────────┬─────────┬───────────────────────────┐ │ Monitor │ State │ Servers │ ├───────────┼─────────┼───────────────────────────┤ │ CsMonitor │ Running │ CsNode1, CsNode2, CsNode3 │ └───────────┴─────────┴───────────────────────────┘
First, the server is removed from the monitor.
maxctrl unlink monitor CsMonitor CsNode3
Checking with maxctrl list monitors
we see that the server has
indeed been removed.
┌───────────┬─────────┬──────────────────┐ │ Monitor │ State │ Servers │ ├───────────┼─────────┼──────────────────┤ │ CsMonitor │ Running │ CsNode1, CsNode2 │ └───────────┴─────────┴──────────────────┘
Now the node can be removed from the cluster itself.
maxctrl --timeout 30s call command csmon remove-node CsMonitor mcs3 20s { "links": { "self": "http://localhost:8989/v1/maxscale/modules/csmon/remove-node" }, "meta": { "message": "Node mcs3 removed from the cluster.", "result": { "node_id": "mcs3", "timestamp": "2020-08-07 11:41:36.573425" }, "success": true } }
Impact of dynamic_node_detection
If dynamic_node_detection
is enabled, there is in general no need
to explicitly remove a static server entry (as there never was one in
the first place). The only exception is if the removed node happened
to be a bootstrap server. In that case, the server entry should be
removed from the monitor's list of servers (used as bootstrap nodes).
If that is not done, then the monitor will log a warning at each startup.