About CloumonELK

CloumonELK is a monitoring solution package based on the popular ELK (ElasticSearch, Logstash and Kibana) stack. It helps you to build your own monitoring stack easily, without hassles to download and install ELK from the scratch. CloumonELK provides predefined configuration files to collect metrics from bigdata components such as Apache Tajo, Apache Hadoop, Apache HBase, Apache ZooKeeper and Apache Flume. CloumonELK can be a good start point to monitor the metrics from bigdata components.

ELK Versions


Getting started

Download CloumonELK source codes

git clone https://github.com/gruter/cloumon-elk.git

Run the download-modules script to get the binaries of Elasticsearch, Logstash and Kibana.


Then go to your download directory and edit the env.sh file.

$ vi env.sh

Install Elasticsearch

Install elasticsearch with the following command. Change hostname and heapsize according to your environment.

./install.sh elasticsearch -s hostname -m heapsize

ex) ./install.sh elasticsearch -s -m 4g

Install Kibana

Install Kibana.

./install.sh kibana -s hostname -e "elasticsearch_host"

ex) ./install.sh kibana -s -e "localhost"

Install Logstash

Install Logstash.

./install.sh logstash -c target_service -p protocol -s hostname -f config_file -g group_name [-k]

ex) ./install.sh logstash -c hadoop -p jmx -s -f ./hadoop.config -g development

Supported Components and Protocols

Below is the list of components that can be monitored with CloumonELK.

Component Protocol
Tajo ganglia
Hadoop jmx
Hadoop-poller jmx
Flume jmx
HBase jmx
ZooKeeper jmx
ElasticSearch http
System http

1) JMX protocol

In case you use JMX protocol to monitor certain component, you need to define the type of server in the configuration file. Below are the supported components and type of server:

For the services using JMX, you need to create the config file for each service. For example, hadoop.config would look like:

$ vi hadoop.config
namenode host001
namenode host002
journalnode host001
journalnode host002
journalnode host003
datanode host004
datanode host005
datanode host006

For other services, refer to the sample-installer.sh for more information.

2) hadoop-poller components

hadoop-poller is an additional compent for gathering full metrics of hadoop. hadoop-poller connects to web admin page of namenode, parses the metric information on the page and store the metrics to elasticsearch. To use hadoop-poller, create "namenodes.config" file and add host:port of hadoop namenode in it. For example, if you have a namenode and standby namenode for HA, namenodes.config is likely to be:

$ vi  namenodes.config

And then intall hadoop-poller components, like below:

./install logstash -s -c hadoop-poller -p ganglia -f  ./namenodes.config -k -g development

Setting up Kibana reports

When you've done with installing CloumonELK, setup the Kibana dashboards and chart templates. You can use built-in templates for the group "development" by the following command:

./kibana-data/restore.sh elasticsearch_host:port template_file group_name

ex) ./kibana-data/restore.sh hadoop-ganglia.json development

The following templates are included in CloumonELK:

Create your own installer from sample-installer.sh

sample-installer.sh has a bunch of install commands so that you can install multiple components at once. Build your own installer script based on sample-installer.sh and then install CloumonELK using your installer script.

Viewing Kibana Dashboards

OK, now you've done with CloumonELK setup. Visit http://kibana_hostname:5931 in your web browser and see the Kibana dashboard and charts to monitor your clusters.


Tajo Dashboard (Ganglia)

Tajo Dashboard (Ganglia)

Hadoop Dashboard (Ganglia)

Hadoop Dashboard (Ganglia)

HBase Dashboard (Ganglia)

HBase Dashboard (Ganglia)

ZooKeeper Dashboard (JMX)

ZooKeeper Dashboard (JMX)

Flume Dashboard (Ganglia)

Flume Dashboard (Ganglia)