Installing Ganglia 3.6.0 on Red Hat 6.4

Jul - 29 2014 | By

book_cover

 

 

 

 

Okay, so after having some issues with my Spark Cluster i decided to download and install Ganglia 3.6.0. But was shocked to find very little documentation and dead links on their site. So after some digging i found some blog’s that helped but I was only able to get it working by reading many different blog’s and fixing issues as i went

 

So below is a summary of what I used to get it working for me, it should work for most and due to so many “flavours” of linux out there – it may not work for everyone.

 

So what is Ganglia? Its a resource monitoring tool for clusters of machines – well this is what it is for me. – Cloud resource monitoring.

 

There are 3 area’s of Ganglia.

 

1) Ganglia Monitoring Daemon (gmond) 
It is a lightweight service that is installed on every machine you’d like to monitor.
Gmond has four main responsibilities:

  • Monitor changes in host state.
  • Announce relevant changes.
  • Listen to the state of all other ganglia nodes via a unicast or multicast channel.
  • Answer requests for an XML description of the cluster state.

Each gmond transmits in information in two different ways:

  • Unicasting or Multicasting host state in external data representation (XDR) format using UDP messages.
  • Sending XML over a TCP connection.

 

2) Ganglia Meta Daemon (gmetad)
The ganglia meta daemon (gmetad) is a service that collects data from other gmetad and gmond sources and stores their state to disk in indexed round-robin databases. Gmetad provides a simple query mechanism for collecting specific information about groups of machines.

 

 

3) Ganglia PHP Web Front-end 
The Ganglia web front-end provides a view of the gathered information via real-time dynamic web pages. Most importantly, it displays Ganglia data in a meaningful way for system administrators and computer users using PHP.

 

 

So lets get started …..

 

 

1) Prerequisite

  • Following dependent package needs to be installed first, before installing Ganglia.
       yum -y install apr-devel apr-util check-devel cairo-devel pango-devel libxml2-devel rpmbuild glib2-devel \
       dbus-devel freetype-devel fontconfig-devel gcc-c++ expat-devel python-devel libXrender-devel

  • Other dependent packages that need to be Installed are rrdtool and confuse that are covered in installation steps.

 

 

2) Installation
2.1) Create user ganglia by which we will be running ganglia services:

   useradd ganglia -p ganglia



2.2) Create a directory where you would download ganglia source:

   mkdir /usr/local/src
   cd /usr/local/src



2.3) First of all we need to Download/Untar/Compile/Install all necessary files for confuse:

   wget http://savannah.nongnu.org/download/confuse/confuse-2.7.tar.gz
   tar -xzvf confuse-2.7.tar.gz
   cd confuse-2.7
   ./configure ./configure CFLAGS=-fPIC –disable-nls
   make
   make install



2.4) Secondly we need to Download/Untar/Compile/Install all necessary files for rrdtool:

   cd /usr/local/src
   wget http://oss.oetiker.ch/rrdtool/pub/rrdtool.tar.gz
   tar -xzvf rrdtool.tar.gz
   cd rrdtool-1.4.8/
   ./configure --prefix=/usr
   make -j8
   make install
   which rrdtool



2.5) Secondly we need to Download/Untar/Compile/Install prc

   wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.35.tar.gz
   tar -xzvf pcre-8.35.tar.gz
   cd pcre-8.35
   ./configure
   make
   make install

 

 

2.6) Make sure all the new installed package have libraries linked using *ldconfig* command:

   vi /etc/ld.so.conf
      /usr/local/lib
   Then execute the following command:
   ldconfig


2.7) Now all the dependency are installed now is the time to Download/Untar/Compile/Install all necessary files for ganglia-core package:

   cd /usr/local/src
   wget http://sourceforge.net/projects/ganglia/files/ganglia%20monitoring%20core/3.6.0/ganglia-3.6.0.tar.gz
   tar -xzvf ganglia-3.6.0.tar.gz
   cd ganglia-3.6.0
   ./configure --with-gmetad

open config.h and change  the variable "GANGLIA_MODULE_DIR" to your module directory install this installed fodler

   make -j8
   make install

NOTE: You should exit without errors. If you see errors, then you may want to check for missing libraries.

 

 

3) Configuring Ganglia
3.1) Create the config directory for Ganglia:

   mkdir /etc/ganglia




3.2) Copy the sample gmetad configuration file: 
   cp gmetad/gmetad.conf /etc/ganglia/



3.3) Generate the initial gmond configuration file: 

   gmond -t | tee /etc/ganglia/gmond.conf



3.4) Copy the initial startup script, change the binary and config path and enable it on boot: 

   cp gmetad/gmetad.init /etc/rc.d/init.d/gmetad
   cp gmond/gmond.init /etc/rc.d/init.d/gmond
   cd /etc/init.d
   find and update the following lines
   vi gmetad
       GMETAD=/usr/local/sbin/gmetad     
       daemon $GMETAD -c /etc/ganglia/gmetad.conf

   vi gmond
       GMOND=/usr/local/sbin/gmond
       daemon $GMOND -c /etc/ganglia/gmond.conf

   chkconfig --add gmetad
   chkconfig --add gmond


3.4) Now create a storage directory for RRDTool and make sure rrdtool can write to it: 

   mkdir -p /var/lib/ganglia/rrds
   chown ganglia:ganglia /var/lib/ganglia/rrds



3.5) Modify the following parameter in gmetad config file: 

   vi /etc/ganglia/gmetad.conf
      data_source "my cluster" FQDN Name of Ganglia Server
      setuid_username "ganglia"
      case_sensitive_hostnames 0

Note: Replace “FQDN Name of Ganglia Server” with your DNS server name. E.G
data_source “my cluster” 127.0.0.1

 

 

 

3.6) Lastly modify the following parameter in gmond config file:

   vi /etc/ganglia/gmond.conf
  
   globals {
   ...
   user = ganglia
   ...
   }

   cluster {
      name = "my cluster"
      owner = "Ops"
      latlong = "unspecified"
      url = "unspecified"
    }

   udp_send_channel {
      host = FQDN Name of Ganglia Server
      port = 8649
      ttl = 1
    }

   udp_recv_channel {
      #mcast_join = 239.2.11.71
      port = 8649
      #bind = 239.2.11.71
      #retry_bind = true
      # Size of the UDP buffer. If you are handling lots of metrics you really
      # should bump it up to e.g. 10MB or even higher.
      # buffer = 10485760
    }

   tcp_accept_channel {
      port = 8649
   }


   update the "path" value in all modules with the pre-fix
   path = "/usr/local/lib64/ganglia/"

E.G = path = “/usr/local/lib64/ganglia/modcpu.so”

 

 

 

4) Validation and Testing
4.1) First try to run gmetad daemon in debug mode and see everything is fine on one of the terminal window:

   gmetad -d 5 -c /etc/ganglia/gmetad.conf

   Going to run as user nobody
   Sources are ...
   Source: [my cluster, step 15] has 1 sources
   	127.0.0.1
   xml listening on port 8651
   interactive xml listening on port 8652
   Data thread 140442277627648 is monitoring [my cluster] data source
      	127.0.0.1
   cleanup thread has been started
   data_thread() for [Ganglia Test Setup] failed to contact node 127.0.0.1
   data_thread() got no answer from any [Ganglia Test Setup] datasource



4.2) First try to run gmond daemon in debug mode and see everything is fine on second terminal window:

   gmond -d 5 -c /etc/ganglia/gmond.conf

   saving metadata for metric: disk_free host: localhost
   Processing a metric value message from localhost
   ***Allocating value packet for host--server001.gauri.com-- and metric --disk_free-- ****

   Processing a metric metadata message from localhost
   ***Allocating metadata packet for host--localhost-- and metric --part_max_used-- ****

NOTE : you can ignore the error for mod_phyton

 

 

 

4.3) Open another terminal window and see rrd file are created or not:

   ls -lh /var/lib/ganglia/rrds/
   total 8.0K
   drwxr-xr-x. 4 ganglia ganglia 4.0K Aug 16 16:28 Ganglia Test Setup
   drwxr-xr-x. 2 ganglia ganglia 4.0K Aug 16 16:28 __SummaryInfo__



4.4) Once you are convinced that everything is fine, stop these process in daemon mode (by pressing CTRL + C) and start there individual service that we have created earlier:

   service gmetad start
   service gmond start
 



4.5) Verify process are running and respective ports are opened:

   ps -ef | grep -v grep | grep gm
   ganglia   6226     1  0 16:59 ?        00:00:00 /usr/local/sbin/gmetad -c /etc/ganglia/gmetad.conf
   ganglia   6267     1  0 17:01 ?        00:00:00 /usr/local/sbin/gmond -c /etc/ganglia/gmond.conf
   netstat -plane | egrep 'gmon|gme'
   tcp        0      0 0.0.0.0:8649                0.0.0.0:*     LISTEN      502        1067310    6267/gmond  
   tcp        0      0 0.0.0.0:8651                0.0.0.0:*     LISTEN      502        1047072    6226/gmetad 
   tcp        0      0 0.0.0.0:8652                0.0.0.0:*     LISTEN      502        1047073    6226/gmetad 
   udp        0      0 0.0.0.0:8649                0.0.0.0:*                 502        1067309    6267/gmond 
 


 

 

5) Deploying Ganglia Web
5.1) Download the package and Untar it:

   cd /usr/local/src/
   wget http://sourceforge.net/projects/ganglia/files/ganglia-web/3.5.10/ganglia-web-3.5.10.tar.gz
   tar -xzvf ganglia-web-3.5.10.tar.gz
   cd ganglia-web-3.5.10

 

 

5.2) Now install Ganglia Web, once we have done the config changes:

   make install



5.3) Disable firewall:

  /etc/init.d/iptables save
  /etc/init.d/iptables stop
  chkconfig iptables off




5.4) Temporary disabling SELinux rules:

   echo 0 >/selinux/enforce
 

5.5) Start Apache:

   apachectl start
 

5.6) Try to open Ganglia Web UI in your favourite Web Browser:

   http://localhost/ganglia

   OR  

http://Server-IP-Address/ganglia

Once we are fully convinced that Ganglia Server is running successfully, now is time to do nodes (i.e. Client) Setup.

 


6) Preparing Client package for ganglia Build up the Ganglia Client Package that will be deployed on client machine:

   tar -czvf /tmp/ganglia-client.tar.gz /usr/local/sbin/gmond /etc/ganglia/gmond.conf /etc/init.d/gmond \
   /usr/local/lib64/libganglia-3.6.0.so.0* /lib64/libexpat.so.1* /usr/lib64/libconfuse.so* \
   /usr/lib64/libapr-1.so* /usr/local/lib64/ganglia



7) Deploying Client package
7.1) SCP the tar package on one of the Client box from Ganglia Server:

   scp /tmp/ganglia-client.tar.gz root@CLIENT-MACHINE-IP-OR-NAME:/tmp/


7.2) Untar the package using following command:

   tar -C / -xzvf /tmp/ganglia-client.tar.gz

 

7.3) we need to Download/Untar/Compile/Install prc

   wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.35.tar.gz
   tar -xzvf pcre-8.35.tar.gz
   cd pcre-8.35
   ./configure
   make
   make install



7.4) Make sure all the new installed package have libraries linked using *ldconfig* command:

   vi /etc/ld.so.conf
      /usr/local/lib
   Then execute the following command:
   ldconfig
 


7.5) Create user ganglia as well and start the gmond service:

   useradd ganglia -p ganglia
   service gmond start


7.6) Verify the gmond process is running:

   ps -ef | grep -v grep | grep gmond


7.7) Also check the rrd file is created for this machine on the Ganglia Server or checking Ganglia Web UI:

   ls -lh /var/lib/ganglia/rrds/Ganglia\ Test\ Setup/
   total 8.0K
   drwxr-xr-x. 2 ganglia ganglia 4.0K Aug 16 16:28 localhost
   drwxr-xr-x. 2 ganglia ganglia 4.0K Aug 16 16:28 web001
   drwxr-xr-x. 2 ganglia ganglia 4.0K Aug 16 16:28 __SummaryInfo__


7.8) Add the other nodes (Client) to Ganglia Server by repeating the steps mentioned from 7.1 to 7.8:

Congratulations! You have successfully deployed Ganglia Setup,

 

ganglia snap

 

Comments are closed. Please see front page on how to contact me