We would like to welcome you to the first in a series of monthly articles calledThe Monitoring Tip of the Month that focuses on the required changes that are necessary to install and configure tracing in an existing Akka project.
In this month’s edition, we examine the following topic: May’s Monitoring Tip of the Month: Telemetry OpenTracing. In future Monitoring Tips of the Month, we will dive deeper into how to use tracing to track activity in Akka systems.
In April 2017 Lightbend Telemetry version 2.4 was released. One of the most significant changes in this release is the addition of OpenTracing integration with support for Jaeger and Zipkin. OpenTracing is a “vendor-neutral open standard for distributed tracing.” Being that Akka is, by its nature is a, platform for building distributed systems tracing is a much needed and often requested feature that is a valuable tool for Akka developers.
This Monitoring Tip of the Month (MTM) focuses on the required changes that are necessary to install and configure tracing in an existing Akka project. In future MTM we will dive deeper into how to use tracing to track activity in Akka systems.
We are going to take an existing Akka project and make the necessary changes to enable Lightbend Telemetry with actor tracing.
We will be following the steps defined in the Lightbend Telemetry setup sbt documentation. The reason for this is that the sample project that we are using is built with SBT. Steps for setting up Maven and Gradle projects are also available.
The project used here can be found in the Lightbend Tech Hub / Get started with Lightbend technologies. The project is “Cluster Java.” Download this project if you wish to follow along. Another option is to take one of your sbt projects and follow along.
As instructed, edit the project/plugins.sbt file.
addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.8")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "1.1.1")
Add the lines as instructed in the documentation.
addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.0")
credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")
resolvers += Resolver.url("lightbend-commercial",
url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)
Here is the complete plugins.sbt file with the initial first 2 lines followed by the added lines.
addSbtPlugin("com.typesafe.sbt" % "sbt-multi-jvm" % "0.3.8")
addSbtPlugin("com.dwijnand" % "sbt-dynver" % "1.1.1")
addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.0")
credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")
resolvers += Resolver.url("lightbend-commercial",
url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)
Next make the necessary changes to the build.sbt file. The file initially looked like this.
import com.typesafe.sbt.SbtMultiJvm.multiJvmSettings
import com.typesafe.sbt.SbtMultiJvm.MultiJvmKeys.MultiJvm
val akkaVersion = "2.5.0"
val `akka-sample-cluster-java` = project
.in(file("."))
.settings(multiJvmSettings: _*)
.settings(
organization := "com.typesafe.akka.samples",
scalaVersion := "2.12.1",
scalacOptions in Compile ++= Seq("-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls", "-Xlint"),
javacOptions in Compile ++= Seq("-Xlint:unchecked", "-Xlint:deprecation"),
javacOptions in doc in Compile := Seq("-Xdoclint:none"),
javaOptions in run ++= Seq("-Xms128m", "-Xmx1024m", "-Djava.library.path=./target/native"),
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-remote" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster-metrics" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster-tools" % akkaVersion,
"com.typesafe.akka" %% "akka-multi-node-testkit" % akkaVersion,
"org.scalatest" %% "scalatest" % "3.0.1" % Test,
"io.kamon" % "sigar-loader" % "1.6.6-rev002"),
fork in run := true,
mainClass in (Compile, run) := Some("sample.cluster.simple.SimpleClusterApp"),
// disable parallel tests
parallelExecution in Test := false,
licenses := Seq(("CC0", url("http://creativecommons.org/publicdomain/zero/1.0")))
)
.configs (MultiJvm)
After adding the monitoring settings the file looks like this. Note that the new and modified lines all end with the “// Telemetry (Cinnamon)” comment.
import com.typesafe.sbt.SbtMultiJvm.multiJvmSettings
import com.typesafe.sbt.SbtMultiJvm.MultiJvmKeys.MultiJvm
val akkaVersion = "2.5.0"
val `akka-sample-cluster-java` = project
.in(file("."))
.enablePlugins(Cinnamon) // Telemetry (Cinnamon)
.settings(multiJvmSettings: _*)
.settings(
cinnamon in run := true, // Telemetry (Cinnamon)
cinnamon in test := true, // Telemetry (Cinnamon)
cinnamonLogLevel := "INFO", // Telemetry (Cinnamon)
organization := "com.typesafe.akka.samples",
scalaVersion := "2.12.1",
scalacOptions in Compile ++= Seq("-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls", "-Xlint"),
javacOptions in Compile ++= Seq("-Xlint:unchecked", "-Xlint:deprecation"),
javacOptions in doc in Compile := Seq("-Xdoclint:none"),
javaOptions in run ++= Seq("-Xms128m", "-Xmx1024m", "-Djava.library.path=./target/native"),
libraryDependencies += Cinnamon.library.cinnamonCHMetrics, // Telemetry (Cinnamon)
libraryDependencies += Cinnamon.library.cinnamonAkka, // Telemetry (Cinnamon)
libraryDependencies += Cinnamon.library.cinnamonOpenTracingJaeger, // Telemetry (Cinnamon)
libraryDependencies += Cinnamon.library.cinnamonOpenTracingZipkin, // Telemetry (Cinnamon)
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-remote" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster-metrics" % akkaVersion,
"com.typesafe.akka" %% "akka-cluster-tools" % akkaVersion,
"com.typesafe.akka" %% "akka-multi-node-testkit" % akkaVersion,
"org.scalatest" %% "scalatest" % "3.0.1" % Test,
// "io.kamon" % "sigar-loader" % "1.6.6-rev002"), // Telemetry (Cinnamon)
"org.slf4j" % "slf4j-api" % "1.7.25", // Telemetry (Cinnamon)
"org.slf4j" % "jcl-over-slf4j" % "1.7.25"), // Telemetry (Cinnamon)
fork in run := true,
mainClass in (Compile, run) := Some("sample.cluster.simple.SimpleClusterApp"),
// disable parallel tests
parallelExecution in Test := false,
licenses := Seq(("CC0", url("http://creativecommons.org/publicdomain/zero/1.0")))
)
.configs (MultiJvm)
The changes made to this build.sbt are somewhat different than what is shown in the documentation, but they are equivalent.
Now that the sbt changes have been completed the next thing that needs to be configured is the Akka configuration settings. The initial application.conf before making any monitoring changes is as follows.
akka {
actor {
provider = "cluster"
}
remote {
log-remote-lifecycle-events = off
netty.tcp {
hostname = "127.0.0.1"
port = 0
}
}
cluster {
seed-nodes = [
"akka.tcp://ClusterSystem@127.0.0.1:2551",
"akka.tcp://ClusterSystem@127.0.0.1:2552"]
# auto downing is NOT safe for production deployments.
# you may want to use it during development, read more about it in the docs.
auto-down-unreachable-after = 10s
}
}
# Disable legacy metrics in akka-cluster.
Akka.cluster.metrics.enabled = off
# Enable metrics extension in akka-cluster-metrics.
Akka.extensions = ["akka.cluster.metrics.ClusterMetricsExtension"]
# Sigar native library extract location during tests.
# Note: use per-jvm-instance folder when running multiple jvm on one host.
Akka.cluster.metrics.native-library-extract-folder = ${user.dir}/target/native<
Let’s make some minimal changes to get actor tracing working. First, there are some required setting for Actor configuration. The following is the complete application.conf file including the monitoring configuration settings added to the end of the file.
akka {
actor {
provider = "cluster"
}
remote {
log-remote-lifecycle-events = off
netty.tcp {
hostname = "127.0.0.1"
port = 0
}
}
cluster {
seed-nodes = [
"akka.tcp://ClusterSystem@127.0.0.1:2551",
"akka.tcp://ClusterSystem@127.0.0.1:2552"]
# auto downing is NOT safe for production deployments.
# you may want to use it during development, read more about it in the docs.
auto-down-unreachable-after = 10s
}
}
# Disable legacy metrics in akka-cluster.
akka.cluster.metrics.enabled=off
# Enable metrics extension in akka-cluster-metrics.
akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"]
# Sigar native library extract location during tests.
# Note: use per-jvm-instance folder when running multiple jvm on one host.
akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native
##### Monitoring Configuration Settings #####
# Actor configuration
cinnamon.akka {
actors {
"samle.cluster.*" {
report-by = class
traceable = on
}
"/user/*" {
report-by = class
traceable = on
}
}
}
Note that in cinnamon.akka.actors settings set traceable = on enables actor tracing. That’s it! Of course, there are other configuration settings available, but this is enough to trace actor activity. At this point we should be able to run the project and look at some tracing data.
Now that sbt and Akka are configured it is time to run some things to view some tracing data. If you are following along using the akka-samples-cluster-java project, please review the README file. This is an interesting project that showcases various Akka clustering features. For this initial tracing test, we will run the Router Example with Group of Routees processes.
Before we run anything, we are first going to start up Jaeger and Zipkin so that we can view the trace data in their respective UIs. Fortunately, both Jaeger and Zipkin provide Docker images that make it easy to use these components. The documentation provides all of the details. Here we cut to the chase and just run the two Docker instances.
docker run -d -p5775:5775/udp -p16686:16686 jaegertracing/all-in-one:latest
docker run -d -p 9411:9411 openzipkin/zipkin
The docker ps command shows the running instances and the tcp ports for their UIs.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
717712f95641 openzipkin/zipkin "/bin/sh -c 'test ..." 26 hours ago Up 4 seconds 9410/tcp, 0.0.0.0:9411->9411/tcp gallant_payne
a46c70f06192 jaegertracing/all-in-one:latest "/go/bin/standalon..." 28 hours ago Up 28 hours 0.0.0.0:5775->5775/udp, 5778/tcp, 6831-6832/udp, 0.0.0.0:16686->16686/tcp awesome_kirch
Note the Zipkin TCP port is 9411 and the Jaeger TCP port is 16686. Access the Jaeger UI at http://localhost:16686 and the Zipkin UI at http://localhost:9411. There will not be anything to see until we run something so let’s do that next.
The akka-samples-cluster-java project README file provides instructions for running various Akka clustering examples. We are going to run some tests with Router Example with Group of Routees. There are two approaches for running the cluster. You can either run one program that starts up the cluster or start each Akka cluster node separately. Since we are mainly interested in tracing Akka actors, it does not matter how you run the cluster. That said, it is more interesting to run each cluster node separately in different terminal windows.
To run everything with a single command cd into the project directory and enter the command:
sbt "runMain sample.cluster.stats.StatsSampleMain"
To run each cluster node separately open four terminal windows, in each terminal window cd into the project directory and enter the following commands, one in each terminal window:
sbt "runMain sample.cluster.stats.StatsSampleMain 2551"
sbt "runMain sample.cluster.stats.StatsSampleMain 2552"
sbt "runMain sample.cluster.stats.StatsSampleClientMain"
sbt "runMain sample.cluster.stats.StatsSampleMain 0"
Once the Akka cluster has been running for a few minutes go take a look at the Jaeger UI and Zipkin UI. Each UI is similar. First select the service, which in this case defaults to the main class name.
Shown above are screenshots of the Jaeger UI and the Zipkin UI.
So that is it! A few changes to the build configuration plus a few changes to the Akka configuration and Akka actor tracing is enabled. No code changes. No annotations. Just configuration changes.
At this point, you are ready to explore the tracing data via either or both UIs. There are also other configuration settings available that you can use to fine tune the monitoring settings. Give it a try and learn more about how you can use this very useful and new monitoring telemetry feature.
In future articles, we will explore tracing and telemetry in more detail. We will also explore monitoring with the exciting new advanced monitoring solution OpsClarity.
Your subscription to the Lightbend Reactive Platform includes enterprise capabilities that extend functionality beyond what’s available in the open source projects to deliver an enterprise-grade platform ready for your mission-critical systems. Learn more about how to use the Monitoring commercial features every month with our new Monitoring Tip Of The Month.
If you have general questions about how to utilize your subscription better, please contact support. If you have additional questions, or need assistance with Monitoring open a support ticket or email support@lightbend.com, and we will be happy to help.
