performance-monitoring.txt

= Performance Monitoring
The performance monitoring toolchain consists of a number of components including

* A Collection Agent (collectd)
* A Collection Aggregator/Relay (graphite)
* A Data Store (whisperdb)
* An API/Presentation Layer (grafana)

These components and their interactions are laid out in the following diagram

image::performance_monitoring_single_node_collectd.png[]

== Installing the Performance Monitoring Suite
=== Installing the Collection Aggregator/Relay
Locate a bare metal system that meets the following minimum specifications

* 4GB of Memory
* Single Socket Xeon Class CPU
* 500GB of Disk Space

Do a fresh CentOS 7 basic install onto the machine. The required repositories
are enabled by

    yum install centos-release-opstools

There is also a repository for using grafana required. Details can be found at
https://grafana.com/grafana/download

First thing we will need to open the firewall on the machine to allow connections to graphite and grafana

 firewall-cmd --zone=public --add-port=2003/tcp --permanent
 firewall-cmd --zone=public --add-port=3000/tcp --permanent
 firewall-cmd --reload

Once that is done, install the graphite and grafana software by running the following command

 yum install -y python-carbon graphite-web grafana httpd

First we will configure the grafana web interface to allow access. Edit
or create the file /etc/httpd/conf.d/graphite-web.conf and modify the
following lines blow to look like the following (changing Require line)

    <Directory "/usr/share/graphite/">
        <IfModule mod_authz_core.c>
            # Apache 2.4
            Require all granted
        </IfModule>

Next we will synchronise the database behind graphite web. Run the following command (when prompted if you want to create a super user choose no)

 sudo -u apache /usr/bin/graphite-manage syncdb --noinput

Next we will start and enable all the graphite and grafana services

 systemctl start httpd
 systemctl enable httpd
 systemctl start carbon-cache
 systemctl enable carbon-cache
 systemctl start grafana-server
 systemctl enable grafana-server

Now we will configure grafana to talk to our graphite instance. Go to the
following address in your browser (where
$FQDN_OF_PERFORMANCE_MONITORING_HOST is the hostname or IP address of the
host we have configured to store the centralised monitoring data

 http://$FQDN_OF_PERFORMANCE_MONITORING_HOST:3000/

You should be presented with the grafana login page. Enter in the default
credentials of admin/admin to log into the system. Once you are logged in,
click on the grafana logo in the top left of the screen, then choose "Data
Sources". At the top of the page click "Add new", Enter in the following
details

 Name: graphite
 Default: yes (tick)
 Type: Graphite

 Url: http://localhost/
 Access: proxy
 Basic Auth: no (unticked)

Then click the "Add" button at the bottom.

== Installing the Performance Monitoring Collection Agent on all nodes
To monitor the performance of all machines in the OpenStack environment,
you will need to make sure the following channel is enabled in Subscription
Manager/Satellite on all machines

 rhel-7-server-openstack-7.0-optools-rpms

Once this is done, you will need to install the following packages on all
OpenStack machines

 yum install -y collectd

Next you will need to configure collectd on all machines to send their data
to the performance monitoring aggregator/relay. Edit the file
/etc/collectd.d/10-write_graphite.conf and place in it the following
contents (where $FQDN_OF_PERFORMANCE_MONITORING_HOST is the hostname or IP
address of the host we configured previously to be the performance
monitoring aggregator/relay)

 <LoadPlugin write_graphite>
   Globals false
 </LoadPlugin>

 <Plugin write_graphite>
   <Carbon>
     Host "$FQDN_OF_PERFORMANCE_MONITORING_HOST"
     Port "2003"
     Prefix "collectd."
     EscapeCharacter "_"
     StoreRates true
     LogSendErrors true
     Protocol "tcp"
   </Carbon>
 </Plugin>

When using SELinux, make sure to execute

 setsebool -P collectd_tcp_network_connect=1

== Configure OpenStack specific collectd plugins in collectd
Collectd in its standard distribution already provides a few plugins suitable
for monitoring OpenStack. However, some more advanced plugins can and should
be installed and enabled.

=== Configure apache related monitoring
To install the collectd plugin, run

 yum install collectd-apache

Create a file /etc/httpd/conf.d/16-server-status.conf:

 LoadModule status_module modules/mod_status.so
 <Location /server-status>
    SetHandler server-status
    Require host localhost
 </Location>
 ExtendedStatus On

Change the file /etc/collect.d/apache.conf to read

 LoadPlugin apache
 <Plugin apache>
   <Instance "localhost">
      URL "http://localhost/server-status?auto"
   </Instance>
 </Plugin>


=== Configure mysql monitoring

First, a collectd-plugin needs to be installed

 yum install collectd-mysql

Create a monitoring user for be able to connect to
mysql, and grant that user USAGE privileges:

  CREATE USER 'monitoring'@'localhost' IDENTIFIED BY 'monitorpassword';
   -- Give appropriate permissions
   -- ("GRANT USAGE" is synonymous to "no privileges")
  GRANT USAGE ON *.* TO 'monitoring'@'localhost';
  FLUSH PRIVILEGES;

Change the file /etc/collectd.d/mysql.conf:

 LoadPlugin mysql
 <Plugin mysql>
   <Database "nova">
     Host "localhost"
     User "monitoring"
     Password "monitorpassword"
     Socket "/var/lib/mysql/mysql.sock"
   </Database>
 </Plugin>

Now we need to start and enable collectd on all hosts

 systemctl start collectd
 systemctl enable collectd