The Library

Related Documents

Testing a Cluster

Length: 499 words; Published: October 20, 2014; Topic: General; Level: Beginner

When you have a cluster running, you may want to test certain features, to ensure that they are working properly or to prepare yourself for handling actual problems that may arise.

Replication Testing

With these steps, you can test that your Galera Cluster is working as expected. First, using the database client, verify that all nodes have connected to each other. To do this, execute the SHOW STATUS statement as shown below:

SHOW STATUS LIKE 'wsrep_%';

+---------------------------+------------+
| Variable_name             | Value      |
+---------------------------+------------+
...
| wsrep_local_state_comment | Synced (6) |
| wsrep_cluster_size        | 3          |
| wsrep_ready               | ON         |
+---------------------------+------------+

Because of the LIKE operator, only variables beginning with wsrep_ are returned. The three variables pertinent here are the ones shown in the example above.

  • wsrep_local_state_comment: The Synced value indicates that the node is connected to the cluster and operational.
  • wsrep_cluster_size: The numeric value returned indicates the number of nodes in the cluster.
  • wsrep_ready: A value of ON indicates that this node, in which the SQL statement was executed, is connected to the cluster and able to handle transactions.

For the next test, try creating a table and inserting data into it. Use a database client on node1 to enter these SQL statements:

CREATE DATABASE galeratest;

USE galeratest;

CREATE TABLE test_table (
   id INT PRIMARY KEY AUTO_INCREMENT,
   msg TEXT ) ENGINE=InnoDB;

INSERT INTO test_table (msg)
   VALUES ("Hello my dear cluster.");

INSERT INTO test_table (msg)
   VALUES ("Hello, again, cluster dear.");

These statements will create a galeratest database and a test_table table within it. The last two SQL statements insert data into that table. After doing this, log into node2 and check that the data was correctly replicated. You would do this with by executing the following SQL statement on node2:

SELECT * FROM galeratest.test_table;

+----+-----------------------------+
| id | msg                         |
+----+-----------------------------+
|  1 | Hello my dear cluster.     |
|  2 | Hello, again, cluster dear. |
+----+-----------------------------+

The results returned from the SELECT statement indicate that the data entered on node1 was replicated on node2.

Split-Brain Testing

There are a few steps to test Galera Cluster for split-brain situations on a two-node cluster. First, disconnect the network connection between the two nodes. At this point, the Quorum will be lost and the nodes will not serve requests.

Now, reconnect the network connection. The quorum will remain lost and the nodes still will not serve requests.

To reset the quorum, execute the following SQL statement on one of the database clients:

SET GLOBAL wsrep_provider_options='pc.bootstrap=1';

At this point, the quorum should be reset and the cluster recovered.

Failure Simulation

You can also test Galera Cluster by simulating various failure situations on three nodes. To simulate a crash of a single mysqld process, execute the following command from the command-line on one of the nodes:

$ killall -9 mysqld

To simulate a network disconnection, use iptables or netem to block all TCP/IP traffic to a node.

To simulate an entire server crash, run each mysqld in a virtualized guest, and abruptly terminate the entire virtual instance.

If you have three or more Galera Cluster nodes, the cluster should be able to survive the simulations.

Related Documents