MENU
» Unlabelled » Install Galera Cluster Install the Atomic & EPEL repositories Create ssh-keys (on each server)

Install Galera Cluster Install the Atomic & EPEL repositories Create ssh-keys (on each server)

alt
arabic_logo_set__2_by_mystafa-d5g1hy7
This document describes the installation of the Galera/Interworx cluster. This cluster provides the default Load Balancing that is available within Interworx, and additionally MySQL loadbalancing through MySQL Galera Clustering.
This guide assumes that you have installed your server with CentOS 6.4 (or later), and that you have double NIC’s with one External connection (192.168.120.x), and the other in a private VLAN (172.20.0.x).


Install the Atomic & EPEL repositories

Since we need some additional packages, we have to add some repositories to the server installation.
yum -y install wget wget http://ftp.nluug.nl/pub/os/Linux/distr/fedora-epel/6/i386/epel-release-6-8.noarch.rpm rpm -Uvh epel-release-6-8.noarch.rpm wget -q -O – http://www.atomicorp.com/installers/atomic | sh
Add a line to the atomic repo (since we don’t want to use the atomic repo for mysql)
nano /etc/yum.repos.d/atomic.repo
[atomic]
exclude=mysql*

Monitoring tools

In this part we will setup, the local monitoring tools
yum -y install nano git iftop ntop htop mytop lynx screen gcc mutt innotop iotop mtr man perl-DBD-MySQL

Other packages

Ok, now we are able to monitor the server we want to install the real software:
Webserver programs
yum -y install httpd clamav mysql mysql-server mysql-devel php-common php-dom php-pear php-soap php-pdo php-mysql php-devel php-gd php-ldap php-mbstring php-intl php-mcrypt phpmyadmin php-xmlrpc php-cli php-iconv php-ctype php-tokenizer aspell php-xcache xcache-admin
Common programs (especially needed if you use iscsi for Interworx later on)
yum -y install iscsi-initiator-utils lsscsi device-mapper-multipath dstat nfs-utils nfs-utils-lib

Set hostname

On all of the servers we will add the hostnames to the /etc/hosts file
echo 192.168.120.1 master.hosting.local master >> /etc/hosts
echo 192.168.120.2 slave1.hosting.local slave1 >> /etc/hosts
echo 192.168.120.3 slave2.hosting.local slave2 >> /etc/hosts
echo 192.168.120.4 slave3.hosting.local slave3 >> /etc/hosts
echo 172.20.0.1 master >> /etc/hosts
echo 172.20.0.2 slave1 >> /etc/hosts
echo 172.20.0.3 slave2 >> /etc/hosts
echo 172.20.0.4 slave3 >> /etc/hosts
If this is done, make sure that hostname and hostname -f returns the same value!!
and hostname -i   doesn’t give you 127.0.0.1 or 127.0.1.1.
hostname && hostname -f

SElinux & IPTables

Disable these services by running:
service iptables stop
setenforce 0
Edit the file /etc/sysconfig/selinux so it reads:
SELINUX=disabled

Configure ntpd

Since cluster services need the correct time we have to install the ntpd (timeserver deamon).
yum -y install ntp && chkconfig ntpd on
Let’s put in some timeserver as well:
nano /etc/ntp.conf
server pool.ntp.org

Create ssh-keys (on each server)

Because all the servers should be able to communicate with each other we have to create on each server a ssh-key.
ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Now add the created ssh public keys (/root/.ssh/id_rsa.pub) to all the servers in the file:
nano ~/.ssh/authorized_keys
If this step is completed try to logon to each server, and accept storage of the host key.
ssh master  Yes
exit
ssh slave1 Yes
exit
ssh slave2 Yes
exit
ssh slave3  Yes
exit
Now restart the server:
shutdown -r now
We are ready to go a step further and start with the configuration of MySQL.
First inititate MySQL for the first run:
mkdir /var/log/mysql
chown -R mysql:mysql /var/log/mysql
Now we can start mysql for the first time:
chkconfig mysqld on && service mysqld start

Create a MySQL restore set

To make it later on possible to install configure Interworx without messing up the Galera installation we have to copy some file to a secure location (on all nodes):
service mysqld stop
mkdir /tmp/interworx /tmp/interworx/etc/
cp /etc/my.cnf /tmp/interworx/etc/my.cnf
mkdir /tmp/interworx/etc/init.d
cp /etc/init.d/mysqld /tmp/interworx/etc/init.d/mysqld
mkdir /tmp/interworx/usr /tmp/interworx/usr/bin
cp /usr/bin/mysqld_safe /tmp/interworx/usr/bin/mysqld_safe
mkdir /tmp/interworx/usr/libexec
cp /usr/libexec/mysqld /tmp/interworx/usr/libexec/mysqld
mkdir /tmp/interworx/var/ /tmp/interworx/var/lib
cp -r /var/lib/mysql /tmp/interworx/var/lib

Galera-Configuration

To configure the Galera Cluster we go to the http://www.severalnines.com/galera-configurator/ website. We picked the following settings, but change according to your requirements:
Vendor:                      Codership (based on MySQL 5.5)
Infrastructure:              Other
Operating  System:           RHEL6 - Redhat 6.3/Fedora/Centos 6.3/OLN 6.3/Amazon AMI Platform:Linux  64-bit (x86_64)
Number of Galera  Servers:   3+1
MySQL PortNumber:            3333    (haproxy, interworx on 3306)
Galera PortNumber:           4567
Galera SST  PortNumber:      4444
SSH PortNumber:              22
OS User:                     root
MySQL Server  password:      pw4mydatabase
CMON DB password:            pw4myCMON
Firewall  (IPTables):        Disable
System Memory:               16 Gb
WAN:                         no
Skip DNS Resolve:            no
Database Size:               < 8GB
MySQL Usage:                 High write/High read
Number of cores:             16
Max connections per  server: 1500
Innodb_buffer_pool_size:     11319
Innodb_file_per_tabel:    yes
ClusterControl  Server:      172.20.0.1
Apache user:                 apache
WWWROOT:                     /var/www/html/
Config  directory:           /etc/
Server-ID 1:                 172.20.0.2
Datadir:                     /var/lib/mysql 
Server-ID 2:                 172.20.0.3
Server-ID  3:                172.20.0.4
Email address:               your@email.com
Click on Generate Deployment Scripts and retrieve them from your mailbox shortly after.

Install Galera Cluster

Now we are ready to install the Galera cluster software
cd /usr/local/src/
wget http://www.severalnines.com/galera-configurator/tmp/bhhrgonbsft2e3o2vhlt0ip9v0/s9s-galera-2.2.0-rpm.tar.gz
tar xvfz s9s-galera-2.2.0-rpm.tar.gz
cd s9s-galera-2.2.0-rpm/mysql/scripts/install
Before running the installer change the following:
echo “local_mysql_port=3333″ >> ../../config/cmon.cnf.agent
echo “local_mysql_port=3333″ >> ../../config/cmon.cnf.controller
echo “local_mysql_port=3333″ >> ../../config/cmon_rrd.cnf
and add the bold line to the files ../../config/my.cnf and ../../config/my.cnf.cmon
[MYSQLD]
old-passwords=1
Now run the installer:
./deploy.sh  2>&1  | tee cc.log
First the setup will try to ping all nodes. If ok, setup will start, since we disabled SELinux and Configured the SSH keys we can answer the following:
do you want to set SELinux to Permissive mode. n
Can you SSH from this host to all other hosts without password? y
After successfully configuring the Galera cluster, there should be a ClusterMonitor (cmon) page where you can connect to on the master node at:
http://192.168.120.1/cmon

JPGraph

Some pages require JPGraph, install this on the master:
cd /usr/local/src
wget http://jpgraph.net/download/download.php?p=5
mv download.php\?p\=5 jpgraph.tar.gz
tar xvfz jpgraph.tar.gz -C /var/www/html/cmon
cd /var/www/html/cmon
ln -s jpgraph-3.5.0b1 jpgraph
chown apache:apache -R jpgraph*

Change MySQL.cnf (configure UTF-8 etc.)

Since we want to use the UTF-8 character set in our database we add this option to the [MYSQLD] section of the file:
#Add some interworx  settings
 #symbolic-links=0
# define default  character sets
   collation-server = utf8_unicode_ci
   init-connect='SET NAMES utf8'
   character-set-server  = utf8
#log
   slow_query_log  = 1
   slow-query-log-file=slow-queries.log
   long_query_time  = 10
#OTHER THINGS,  BUFFERS ETC
   myisam-sort-buffer-size  = 8M
   skip-name-resolve
   memlock=0
   event_scheduler=1
Furthermore we change the environment variables to UTF8:
echo “LANG=en_US.utf-8″ >> /etc/environment
echo “LC_ALL=en_US.utf-8″ >> /etc/environment
Now we can restart MySQL to make this setting active for the databases that will be created in the future.
service mysql restart
Now we have to correct some installer files. Since mysql-libs is removed by the Galera setup. We have to use the –excludepath option to get around the following error:
file /usr/share/mysql/russian/errmsg.sys from install of MySQL-server-5.5.28_wsrep_23.7-1.rhel5.x86_64 conflicts with file from package mysql-libs-5.1.67-1.el6_3.x86_64
Download the mysql-libs rpm file and run it with the –excludepath option (which isn’t available when using yum ):
cd /usr/local/src
wget ftp://mirror.switch.ch/pool/1/mirror/scientificlinux/6rolling/x86_64/os/Packages/mysql-libs-5.1.67-1.el6_3.x86_64.rpm
rpm -Uvh mysql-libs-5.1.67-1.el6_3.x86_64.rpm –excludepath=/usr/share/mysql/
Reinstall the packages that where removed because of the mysql-libs dependencies.
yum -y install nagios-plugins-all perl-DBD-MySQL innotop mytop

HAProxy setup

Next step is installing the HAProxy setup that uses cmon of the Galera cluster, to show its status information. At the master server we have to run the following commands:
# the installer needs to be extracted inside the galera installation folder!!!
cd /usr/local/src/s9s-galera-2.2.0-rpm/mysql/scripts/install
wget http://severalnines.com/downloads/s9s-haproxy.tar.gz
tar zxvf s9s-haproxy.tar.gz
cd haproxy
Before installing lets change some settings inside the following files:
install-haproxy.sh:           HAPROXY_MYSQL_LISTEN_PORT="3306"
makecfg.sh:                  "\tserver ${SPLIT[0]}  ${SPLIT[0]}:3333 check $STAT"
mysqlchk.sh.galera           MYSQL_PORT="3333"
mysqlchk.sh.mysqlcluster     MYSQL_PORT="3333"
Now run the installer:
./install-haproxy.sh 172.20.0.1 rhel galera
When the setup is completed without errors you have to grant the installer host rights to mysql instances on each server.
mysql -uroot -ppw4mydatabase
GRANT ALL ON *.* TO ‘root’@’172.20.0.1′ IDENTIFIED BY ‘pw4mydatabase‘;
FLUSH PRIVILEGES;
exit
There is something what probably will go wrong tho, which is related to the defaults requiretty setting. It is possible that you get an error message like:
Pseudo-terminal will not be allocated because stdin is not a terminal.
sudo: sorry, you must have a tty to run sudo stdin is not a terminal.
If this happens edit the sudoers file (on all servers) by using:
visudo
and change the line
 Defaults    requirettys
into
#Defaults    requirettys
Now it should be possible to connect to the loadbalanced cluster on the following address:
mysql -h192.168.120.1 -P3306 -uroot -ppw4mydatabase
Check this from the slave nodes to make sure.
You can connect to the status page of the HAProxy, http://192.168.120.1:9600
(default admin/admin). You can change the username/password by editing the file:
nano /etc/haproxy/haproxy.cfg
userlist STATSUSERS
group admin users admin
user admin insecure-password pw4adminHA
user stats insecure-password pw4userHA
Now we are ready to install Interworx.

InterWorx-ControlPanel

Before installing stop haproxy & mysql & cmon on the nodes
service mysql stop && service cmon stop && service haproxy stop
Now we start moving around some file to restore the default MySQL environment (we have /var/lib/mysql on different disks, so if you dont, make a copy of the directories and restore these after installing interworx):
umount /var/lib/mysql
cp -r /tmp/interworx/var/lib/mysql /var/lib
chown -R mysql:mysql /var/lib/mysql
cp /tmp/interworx/etc/init.d/mysqld /etc/init.d/mysqld
chmod 755 /etc/init.d/mysqld
cp /tmp/interworx/usr/libexec/mysqld /usr/libexec/mysqld
chmod 755 /usr/libexec/mysqld
mkdir /var/run/mysqld
chown mysql:mysql /var/run/mysqld
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.galera
cp /tmp/interworx/usr/bin/mysqld_safe /usr/bin/mysqld_safe
chmod 755 /usr/bin/mysqld_safe
mv /etc/my.cnf /etc/galera.cnf
cp /tmp/interworx/etc/my.cnf /etc/my.cnf
mv /etc/init.d/mysql /etc/init.d/mysql.org
Edit the file
nano /etc/my.cnf
[mysqld]
old-passwords=1
Since Interworx and Galera remove and install lots of packages, we have to take some precautions.  Default it isn’t possible to combine those to packages. The steps before (create a backup set of the default mysql) can resolve this problem.
We first have to download the installer file and modify it:
cd /usr/local/src && wget -q http://updates.interworx.com/iworx/scripts/iworx-cp-install.sh
Now we have to comment out line 321, to prevent Interworx to mess up the mysql install.
nano iworx-cp-install.sh
# mysqlinstall
Then we can save the file and start the installer.
sh ./iworx-cp-install.sh
Press <enter> to begin the install…
After a while, you will get:
-=-=-=-=-= ALL DONE! THANK YOU! FOR USING InterWorx-CP =-=-=-=-=-
It’s good to check after installation the logs at  /home/interworx/var/log/error.log.
If ok, you’ll want to activate your InterWorx CP License.
Register now Interworx by using the website, since the command line doesn’t seem to work properly. Go to the website http://public IP Address:2080/nodeworx and enter the registration information:

After a while , you will probably get a blank page. To verify that the setup was completed you could test it with the command line method.

Scripted Activation Procedure

Run the goiworx.pex script with the following parameters (replace the bold text with the licensekeys below):
# Note: Make sure  the below is all one line. Replace LICENSE_KEY, EMAIL, and PASSWORD
 #~iworx/bin/goiworx.pex  --key=INTERWORX_YOURKEY --email=your@email.com --password=pw4yourIworx --ignorechecks
Now go to the website http://public IP Address/nodeworx and enter your login credentials. Accept the license agreement (some time you have to do this twice). If logged on you will be redirected to the page to configure your DNS servers:

Enter the requested primary/secondary DNS servers and click on update.  Click on System Services on the left side, followed by MySQL Server and Overview. Here we want to configure our Galera MySQL root password.

Click save again, logout and empty the IE cache incl. cookies etc.
If you have external nameservers, restore the original /etc/resolv.conf
echo “nameserver mynameserverip1” > /etc/resolv.conf
echo “nameserver mynameserverip2” >> /etc/resolv.conf
We now have configured the basic part of Interworx, ready to move those accounts into the Galera environment.   Repeat these steps for the other servers.
Ok, we now should be able to connect to the page http://Public IP Address Master/nodeworx and logon to interworx.  Of course we also have to logon to the slave servers later on.
After logging in to Interworx on the master server we want to configure the slave nodes:

Click on the setup button underneath Interworx-CP Cluster Manager.

Choose the Quorum IP Address (HeartBeat LAN), and click setup. We now going to add the slave nodes.
Since we need a API-Key of each slave server we logon tot the slave servers, click on Nodeworx in the left menu, click on API-key and then on Generate on the right side panel.

After you click on generate you will see an API-key being generated.

Copy this key and go back to the master server.

Add the slave by using their IP-Address since the Heartbeat node isn’t separate available in the DNS.  Past the API-key, run test first, and then add the API-key. If all of this is working, and cmon also is green we are ready to go further. We start with rolling back the MySQL environment to the Galera environment. First stop on all servers the services!
service iworx stop && service mysqld stop
Now we rollback the Galera MySQL environment (on all servers). Again: we use separate disks for mysql, if you dont use the copies you made before.
mv /etc/my.cnf /etc/interworx.cnf
mv /etc/galera.cnf /etc/my.cnf
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.interworx
mv /usr/bin/mysqld_safe.galera /usr/bin/mysqld_safe
rm -rf /var/run/mysqld
rm -rf /usr/libexec/mysqld
rm -rf /etc/init.d/mysqld
mv /var/lib/mysql /var/lib/mysql.iworx
mkdir /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql/
mount -a
mv /etc/init.d/mysql.org /etc/init.d/mysql
Now start on all servers the services:
service mysql start && service cmon start && service iworx start && service haproxy start
Wait until cmon recovers again!
It is possible that MySQL want start the first time stop and start it until it works normal. There could be a lost pid file or subsys file, that you have to remove first. Anyway if it works normal we can remove some extra files
rm -rf /var/lib/mysql.iworx /usr/bin/mysqld_safe.interworx /tmp/interworx
If al servers are done, we have to insert the Iworx users into the Galera MSQL databases. On the master server and galera cluster we run the following mysql statements:
These values are available from Interworx if you install without the modified installer.
mysql --user=root  --password=pw4mydatabase
INSERT INTO  `mysql`.`user` VALUES ('localhost','iworx','[your iworx password in old password format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('127.0.0.1','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.2','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.3','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.4','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.1','iworx','[your iworx master password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL);
GRANT ALL ON *.* TO  'iworx'@'172.20.0.1' IDENTIFIED BY ‘pw4mydatabase’;
FLUSH PRIVILEGES;
exit
Wait until its up and running and in sync on the webpage http://master IP Address/cmon (use FireFox since this one updated the page better than IE).
Let’s start Iworx again:
on Thursday, August 1, 2013

Post a Comment

CodeNirvana
Newer Posts Older Posts
© Copyright Softwares Zone
Back To Top