QuickStart

Data Import from Scala Applications

The 'fluent-logger-scala' library is used to post records from Scala applications to Fluentd.

This article explains how to use the fluent-logger-Scala library.

Prerequisites

• Basic knowledge of Scala and sbt
• Basic knowledge of Fluentd
• Scala 2.9.0/2.9.1/2.10.0
• sbt 0.12.0 or later

Installing Fluentd

Please refer to the following documents to install fluentd.

• Install Fluentd with rpm Package
• Install Fluentd with deb Package
• Install Fluentd with Ruby Gem
• Install Fluentd from source

Modifying the Config File

Next, please configure Fluentd to use the forward Input plugin as its data source.

:::text
<source>
  type forward
  port 24224
</source>
<match fluentd.test.**>
  type stdout
</match>

Please restart your agent once these lines are in place.

:::term
# for rpm/deb only
$ sudo /etc/init.d/td-agent restart

Using fluent-logger-scala

First, please add the following lines to build.sbt. The logger's revision information can be found in ChangeLog.

resolvers += "Apache Maven Central Repository" at "http://repo.maven.apache.org/maven2/"

libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.4.0"

or

resolvers += "Sonatype Repository" at "http://oss.sonatype.org/content/repositories/releases"

libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.4.0"

Next, please insert the following lines into your application. Further information regarding the API can be found here.

:::Scala
import org.fluentd.logger.scala.FluentLoggerFactory
import scala.collection.mutable.HashMap

object Sample {
  val LOG = FluentLoggerFactory.getLogger("fluentd.test")

  def main(args: Array[String]): Unit = {
    
    ...
    val data = new HashMap[String, String]();
    data.put("from", "userA");
    data.put("to", "userB");
    LOG.log("follow", data);
    ...
  }

}

Executing the script will send the logs to Fluentd.

:::term
$ sbt
> run

The logs should be output to /var/log/td-agent/td-agent.log or stdout of the Fluentd process via the stdout Output plugin.

Production Deployments

Output Plugins

Various output plugins are available for writing records to other destinations:

• Examples
  • Store Apache Logs into Amazon S3
  • Store Apache Logs into MongoDB
  • Data Collection into HDFS
• List of Plugin References
  • Output to Another Fluentd
  • Output to MongoDB or MongoDB ReplicaSet
  • Output to Hadoop
  • Output to File
  • etc...

High-Availablability Configurations of Fluentd

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

• High-Availability Configurations of Fluentd

Monitoring

Monitoring Fluentd itself is also important. The article below describes general monitoring methods for td-agent.

• Monitoring Fluentd