Collect Logs in Cluster
Attention: We have migrated our documentation to our new platform, Ant Media Resources. Please follow this link for the latest and up-to-date documentation.
Graylog is an open source centeralized log collection and analysis software which uses elastic-search and MongoDB in its architecture. This guide will be about Graylog setup, configuration and how to send Ant Media Server logs to it.
If you are using the cluster structure and want to keep track of all logs from one place, this article is for you.
I will do this installation on Ubuntu 20.04, but it is similar in other Linux based operating systems.
System requirements:
- Ubuntu 18.04 or Ubuntu 20.04
- Minimum 4GB RAM
- Ant Media Servers
Test Environment:
Graylog Server: 192.168.1.250
Ant Media Server 1: 192.168.1.251
Ant Media Server 2: 192.168.1.252
In order to run Elasticsearch, you must install Java. Run the following commands to install.
sudo apt-get update
sudo apt-get install apt-transport-https openjdk-11-jre openjdk-11-jre-headless uuid-runtime pwgen
MongoDB stores the configurations and meta information.
Install MongoDB using the following commands.
sudo apt-get install gnupg
wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu `lsb_release -cs`/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt-get update & sudo apt-get install -y mongodb-org
Enable and restart MongoDB service by running the commands below.
sudo systemctl enable mongod.service & sudo systemctl restart mongod.service
Make sure the service is running:
sudo systemctl status mongod.service
Graylog can be used with Elasticsearch 6.x, please follow the below instructions to install the open-source version of Elasticsearch. Elasticsearch is software that acts as a search server, requiring Graylog to work.
Install Elasticsearch using the following commands.
wget -O - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add
echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install elasticsearch-oss
Once the installation of Elasticsearch 6.x is complete, set cluster name for Graylog.
Edit the following file:
vim /etc/elasticsearch/elasticsearch.yml
and then add the 2 lines below.
cluster.name: graylog
action.auto_create_index: false
Save the file and exit.
Enable and restart Elasticsearch service by running the commands below:
sudo systemctl enable elasticsearch.service
sudo systemctl restart elasticsearch.service
Make sure the service is running. To check the status of Elasticsearch, run the commands below:
sudo systemctl status elasticsearch.service
Make sure everything is correct by running the following commands:
curl -X GET http://localhost:9200
Output:
root@graylog:~# curl -X GET http://localhost:9200
{
"name" : "cdN0aJ1",
"cluster_name" : "graylog",
"cluster_uuid" : "hyWsngLVRqq_IWU1cr75AA",
"version" : {
"number" : "6.8.13",
"build_flavor" : "oss",
"build_type" : "deb",
"build_hash" : "be13c69",
"build_date" : "2020-10-16T09:09:46.555371Z",
"build_snapshot" : false,
"lucene_version" : "7.7.3",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Make sure the output status is green.
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 12,
"active_shards" : 12,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Graylog is a log parser. It collects the logs from various inputs. Now that we have installed MongoDB and Elasticsearch, it is time to install Graylog.
Install Graylog using the following commands:
wget https://packages.graylog2.org/repo/packages/graylog-3.3-repository_latest.deb
sudo dpkg -i graylog-3.3-repository_latest.deb
sudo apt-get update && sudo apt-get install graylog-server
To create your root_password_sha2 run the following command. You will need this password to login to the Graylog web interface.
echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
Output: 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
You will need to generate a secret to secure the user passwords. To generate the password_secret, you can use the pwgen tool to do.
pwgen -N 1 -s 96
Output: jyOQ188lAq1ssEMvCndsj2ImEOuWkC4v3aL4AQg9Dj4wvavkk3BAkSzMXFyH8aN8GiMoIJl2xmT4T5aGwS1r06Cz38SMsgDK
Edit the /etc/graylog/server/server.conf file then add root_password_sha2 and password_secret outputs.
password_secret = jyOQ188lAq1ssEMvCndsj2ImEOuWkC4v3aL4AQg9Dj4wvavkk3BAkSzMXFyH8aN8GiMoIJl2xmT4T5aGwS1r06Cz38SMsgDK
root_password_sha2 = 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
If you don't want to use reverse proxy with SSL termination, uncomment the following line then change according to your server ip address.
http_bind_address = 127.0.0.1:9000`
to
http_bind_address = your_server_public_ip:9000
If you want to use the reverse proxy with SSL termination, please go to this step.
save and exit the file.
Enable and restart Graylog Server service by running the commands below.
sudo systemctl enable graylog-server.service
sudo systemctl restart graylog-server.service
Make sure the service is running.
sudo systemctl status graylog-server.service
Run the following commands to install Nginx and certbot:
sudo apt install curl ca-certificates lsb-release -y
echo "deb http://nginx.org/packages/`lsb_release -d | awk '{print $2}' | tr '[:upper:]' '[:lower:]'` `lsb_release -cs` nginx" \
| sudo tee /etc/apt/sources.list.d/nginx.list
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install nginx certbot python-certbot-nginx -y
Run the following commands to create a certificate:
certbot --nginx -d yourdomain.com -d www.yourdomain.com
Edit crontab file crontab -e add below line to renew certificate each 80 days:
0 0 */80 * * root certbot -q renew --nginx
Backup default Nginx configuration.
mv /etc/nginx/conf.d/default.conf{,_bck}
Create a new file called graylog.conf and edit and save the following lines according to you.
vim /etc/nginx/conf.d/graylog.conf
server {
listen 443 ssl;
server_name yourdomain.com;
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1440m;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA HIGH !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
location / {
proxy_set_header HOST $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:9000;
}
}
Save and exit the file then restart nginx service as follows:
systemctl restart nginx
Now you can reach to Graylog server as follows.
https://yourdomain.com
Access Graylog web interface using its IP Address and port 9000
http://serverip_or_hostname:9000
or
https://yourdomain.com
Login to your servers where Ant Media is installed with ssh and create /etc/rsyslog.d/25-antmedia.conf file then add below lines:
$ModLoad imfile
$InputFileName /usr/local/antmedia/log/ant-media-server.log
$InputFileTag antmedia
$InputFileStateFile stat-antmedia
$InputRunFileMonitor
*.* @192.168.1.250:5144;RSYSLOG_SyslogProtocol23Format
Save and exit the file then restart rsyslog service.
sytemctl restart rsyslog
Open the dashboard and log in.
Click on Systems/Inputs and select Syslog UDP and click on Launch New Input.
Set the settings as in the screenshot and click Save.
Your input will appear as below.
If you have made the correct log settings on Ant Media servers, the logs as below will start to appear.
"stream1"
(stream1 OR stream2)
"stream1" AND NOT source:192.168.1.251
source:192.168.1.252
"stream*" NOT source:192.168.1.2