Monitoring guide

boms/bom/pom.xml

@@ -177,13 +177,13 @@
       <dependency>
         <groupId>com.github.dozermapper</groupId>
         <artifactId>dozer-core</artifactId>
-        <version>6.4.1</version>
+        <version>6.5.0</version>
       </dependency>
       <!-- Orika for bean-mapping/conversion (Alternative to Dozer) -->
       <dependency>
         <groupId>ma.glasnost.orika</groupId>
         <artifactId>orika-core</artifactId>
-        <version>1.5.2</version>
+        <version>1.5.4</version>
       </dependency>
       <!-- JSR303 API for annotation based validation -->
       <dependency>

boms/minimal/pom.xml

@@ -54,6 +54,16 @@
         <groupId>com.devonfw.java.modules</groupId>
         <artifactId>devon4j-beanmapping</artifactId>
         <version>${project.version}</version>
+      </dependency>
+       <dependency>
+        <groupId>com.devonfw.java.modules</groupId>
+        <artifactId>devon4j-beanmapping-dozer</artifactId>
+        <version>${project.version}</version>
+      </dependency>
+       <dependency>
+        <groupId>com.devonfw.java.modules</groupId>
+        <artifactId>devon4j-beanmapping-orika</artifactId>
+        <version>${project.version}</version>
       </dependency>
       <dependency>
         <groupId>com.devonfw.java.modules</groupId>

documentation/guide-beanmapping.asciidoc

@@ -21,16 +21,29 @@ public UserEto mapUser(UserEntity source) {
 Therefore we are using a `BeanMapper` for this purpose that makes our lives a lot easier.
 
 == Bean-Mapper Dependency
-To get access to the `BeanMapper` we use this dependency in our POM:
+To get access to the `BeanMapper` we have to use either of below dependency in our POM:
+
+We started with dozer.sourceforge.net/[dozer] in devon4j and still support it. However, we now recommend https://orika-mapper.github.io/orika-docs/[orika] (for new projects) as it is much faster (see https://www.baeldung.com/java-performance-mapping-frameworks#2-orika[Performance of Java Mapping Frameworks]).
+
+[source,xml]
+----
+    <dependency>
+      <groupId>com.devonfw.java</groupId>
+      <artifactId>devon4j-beanmapping-orika</artifactId>
+    </dependency>
+----
+
+or
 
 [source,xml]
 ----
     <dependency>
       <groupId>com.devonfw.java</groupId>
-      <artifactId>devon4j-beanmapping</artifactId>
+      <artifactId>devon4j-beanmapping-dozer</artifactId>
     </dependency>
 ----
-== Bean-Mapper Configuration
+== Bean-Mapper Configuration using Dozer
+
 The `BeanMapper` implementation is based on an existing open-source bean mapping framework. 
 In case of Dozer the mapping is configured `src/main/resources/config/app/common/dozer-mapping.xml`.
 

documentation/guide-configuration.asciidoc

@@ -57,26 +57,12 @@ Beans are configured via annotations in your java code (see link:guide-dependenc
 
 For technical configuration you will typically write additional spring config classes annotated with `@Configuration` that provide bean implementations via methods annotated with `@Bean`. See http://docs.spring.io/spring-javaconfig/docs/1.0.0.M4/reference/html/ch02s02.html[spring @Bean documentation] for further details. Like in XML you can also use `@Import` to make a `@Configuration` class include other configurations.
 
-=== XML-based beans configuration
-It is still possible and allowed to provide (bean-) configurations using XML, though not recommended. These configuration files are no more bundled via a main xml config file but loaded individually from their respective owners, e.g. for unit-tests:
-
-[source, java]
-----
-@SpringApplicationConfiguration(classes = { SpringBootApp.class }, locations = { "classpath:/config/app/batch/beans-productimport.xml" })
-public class ProductImportJobTest extends AbstractSpringBatchIntegrationTest {
-...
-----
-
-Configuration XML-files reside in an adequately named subfolder of:
+More specific configuration files (as required) reside in an adequately named subfolder of:
 
 `src/main/resources/app`
 
-=== Batch configuration
-In the directory `src/main/resources/config/app/batch` we place the configuration for the batch jobs. Each file within this directory represents one batch job. See link:guide-batch-layer.asciidoc[batch-layer] for further details.
-
 === BeanMapper Configuration
-In the directory `src/main/resources/config/app/common` we place the configuration for the bean-mapping.
-See link:guide-beanmapping#bean-mapper-configuration.asciidoc[bean-mapper configuration] for further details.
+In case you are still using dozer, you will find further details in link:guide-beanmapping.asciidoc#bean-mapper-configuration[bean-mapper configuration].
 
 === Security configuration
 The abstract base class `BaseWebSecurityConfig` should be extended to configure web application security thoroughly.
@@ -150,17 +136,17 @@ In this `application.properties` you only define the minimum properties that are
 Make sure your properties are thoroughly documented by providing a comment to each property. This inline documentation is most valuable for your operating department. 
 
 === Business Configuration
-The business configuration contains all business configuration values of the application, which can be edited by administrators through the GUI. The business configuration values are stored in the database in key/value pairs.
+Often applications do not need business configuration. In case they do it should typically be editable by administrators via the GUI. The business configuration values should therefore be stored in the database in key/value pairs.
 
-The database table `business_configuration` has the following columns:
+Therefore we suggest to create a dedicated table with (at least) the following columns:
 
 * ID
 * Property name
 * Property type (Boolean, Integer, String)
 * Property value
 * Description
 
-According to the entries in this table, the administrative GUI shows a generic form to change business configuration. The hierarchy of the properties determines the place in the GUI, so the GUI bundles properties from the same hierarchy level and name. Boolean values are shown as checkboxes, integer and string values as text fields. The properties are read and saved in a typed form, an error is raised if you try to save a string in an integer property for example.
+According to the entries in this table, an administrative GUI may show a generic form to modify business configuration. Boolean values should be shown as checkboxes, integer and string values as text fields. The values should be validated according to their type so an error is raised if you try to save a string in an integer property for example.
 
 We recommend the following base layout for the hierarchical business configuration:
 

documentation/guide-logging.asciidoc

@@ -111,4 +111,34 @@ In order to correlate separate HTTP requests to services belonging to the same u
 == Monitoring
 In highly distributed systems (from clustering up to microservices) it might get tedious to search for problems and details in log files. Therefore, we recommend to setup a central log and analysis server for your application landscape. Then you feed the logs from all your applications (using http://logstash.net/[logstash]) into that central server that adds them to a search index to allow fast searches (using http://www.elasticsearch.org/[elasticsearch]). This should be completed with a UI that allows dashboards and reports based on data aggregated from all the logs.
 This is addressed by https://www.elastic.co/webinars/introduction-elk-stack[ELK] or https://www.graylog.org/[Graylog].
-
+
+=== Adding custom values to JSON log
+
+When you use a external system for gathering distrubited logs, we strongly suggest that you use a JSON based log format (e.g. by using the provided `logback.xml`, see above).
+In that case it might be useful to log customs field to the produced JSON. This is fully supported by slf4j together with logstash. The _trick_ is to use the class `net.logstash.logback.argument.StructuredArguments` for adding the arguments to you log message, e.g.
+
+[source,java]
+----
+import static net.logstash.logback.argument.StructuredArguments.v;
+
+...
+
+  public void processMessage(MyMessage msg) {
+    LOG.info("Processing message with {}", v("msgId", msg.getId());
+  ...
+  }
+----
+
+This will produce something like:
+[source,javascript]
+----
+{ "timestamp":"2018-11-06T18:36:37.638+00:00",
+  "@version":"1",
+  "message":"Processing message with msgId=MSG-999-000",
+  "msgId":"MSG-999-000",
+  "logger_name":"com.myapplication...Processor",
+  "thread_name":"http-nio-8081-exec-6",
+  "level":"INFO",
+  "level_value":20000,
+  "appname":"basic",}
+----

documentation/guide-monitoring.asciidoc

@@ -3,25 +3,113 @@ toc::[]
 
 = Monitoring
 
-TODO
+Monitoring is a very comprehensive topic. For devon4j we will focus on selected core topics which are most important when developing production-ready applications.
+On a high level view we strongly suggest to separate the application to be monitoring from the "monitoring tool" itself.
+The monitoring system covers aspects like
 
+- Collect monitoring information
+- Aggregate, process and visualizate data, e.g. in dashboards
+- Provide alarms
+- ...
 
-== JMX
-We use JMX to provide monitoring information via MBeans.
+In distributed systems it is crucial that a monitoring systems provides a central overview over all applications in your application landscape. So it is not feasible to provide a monitoring system as part of your application. Your application is responsible for providing information to the monitoring system.
 
-=== MBeans
-[source,java]
-TODO MBean example
+== How to provide monitoring information
 
-=== Using JMX
-TODO: explain +jvisualvm+ and +VisualVM-MBeans+ Plugin
-give some hints and links on heap analysis and samplers
+=== Java Management Extensions (JMX)
 
-=== Integration with Nagios
+JMX is the official java monitoring solution. It is part of the JDK. Your application may provide monitoring information or receive monitoring related commands via MBeans. There is a huge amount of information about JMX available. A good starting point might be link:https://en.wikipedia.org/wiki/Java_Management_Extensions:[Wikipedia].
+Traditionally JMX uses RMI for communication. In many environments HTTP is preffered, so be careful on deciding if JMX is the right solution.
+Alternativly your application could provide a monitoring API via REST.
 
-== GC-Logging
-TODO: Explain JVM options, minimum overhead, always turn on, link to gcvisualizer
-Explain G1 collector that scales for large heaps
+=== REST APIs
 
-== Loadbalancing
-TODO explain status URL and integration with LB
+Since REST APIs are very popular it is quite natural to provide monitoring information via REST. If your application already uses REST and there is no JMX/RMI based monitoring solution in place, it is often a better approach to provide monitoring APIs via REST than introducing a new protocol with JMX.
+
+=== Logging
+
+Some aspects of monitoring are covered by "logging". A typical case might be the requirement to "monitor" the application your REST APIs. For that it is perfectly fine to log the perfomance of all (or some) request and ship this information to a logging system like link:www.graylog.org[Graylog]. So please carefully read the link:guide-logging.asciidoc[logging guide]. To allow efficient processing of those logs you should use JSON based logs.
+
+== Which monitoring informaton to provide?
+
+It is not possible to provide a final list of which monitoring information is exactly required for your application. But we give you a general advice what type of information your application should provide at a minimum.
+
+=== General information
+
+It is often useful if your application provides basic information about itself. This should cover
+
+* The name of the application
+* The version (or buildno., or commit-id, ...) of the application
+
+This is espicially useful in complex environments, e.g. to very that deployments went correctly.
+
+=== Metrics
+
+Metric means providing key figures about your applications like
+
+* performance key figures
+** request duration
+** queue lengths
+** duration of database queries
+** ..
+* business related information
+** number of successful / unsuccesful requests
+** size of result sets
+** worth of shopping baskets
+** ..
+* Technical key figures
+** JVM heap usage
+** cache sizes
+** (database) pool usage
+** ...
+* ...
+
+Remember that processing of this data should be done mainly in the monitoring system. You might have noticed that there are different types of metrics those that represent current values (like JVM heap usage, queue length, ...), others base on (timed) events like (duration of requests). Handling of different types of metrics might be different. 
+For handling events you may:
+
+* Write log statements for each (or a sample of) event. These logs must then be shipped to your monitoring systems.
+* Send data for the event via an API of your monitoring systems
+* Provide a REST API (or JMX MBeans) with pre-aggregated key figures, which is periodically polled by your monitoring system. This solution is a bit inferior since the aggregation is part of your application and might not fit to the desired visualization in your monitoring systems.
+
+For actual values you may:
+
+* Write them perodically to your log. These logs must then be shipped to your monitoring systems.
+* Send them peridocally via an API of your monitoring systems
+* Provide a REST API (or JMX MBeans) which is periodically polled by your monitoring system.
+
+[health]
+=== Health (Watchdog)
+
+For monitoring a complex application landscaper it is crucial to have a exact overview if all applications are up and running. So your application should offer an API for the monitoring systems which allows to easily check if the application is alive. Often this alive information is polled by the monitoring systems with a kind of watchdog.
+The health check should include checks if the application is working "correctly". For that we suggest to check if all required neighbour systems and infrastructure components are usable:
+
+* Check if your database can be queried (with a dummy query)
+* Check if you can reach your messaging system
+* Check if you can reach all your neighbour system, e.g. by querying their info-endpoint
+
+You should be very careful to not cascade those requests, e.g. your system should only test their direct neighbours. This test should not lead to additional tests in these systems.
+
+The healthcheck should a return a simple OK/NOK result for use in dashboards, and may addtionally include detail results for each check.
+
+
+== Implementation with Spring Boot Actuator
+
+To implement a monitoring API for your systems we suggest to use link:https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html[Spring Boot Actuator]. Actuator offers APIs which provide monitoring information including metrics via HTTP and JMX. It also contains a framework to implement xref:health[health checks].
+Please consult the original documentation for information about how to use it.
+
+=== Devonfw additions
+
+Devonfw includes the following additions for Spring boot actuator:
+
+* link:link:TODO[Kafka Health Check] in devon4j-kafka (WIP)
+
+== Integration of monitoring information
+
+=== Loadbalancers
+
+To loadbalance HTTP requests the loadbalancers needs to know which instances of the desired application are available and functioning. Often loadbalancers support reacting on the HTTP status code of an HTTP request to the service. The loadbalancer will periodically poll the service to find out if is available or not.
+To configure this you may use the healthcheck of the service to find out if the instance is functioning correctly or not.
+
+=== Docker
+
+Docker supports a link:https://docs.docker.com/engine/reference/builder/#healthcheck[healtcheck]. You may but a simple local curl to your application here to find out if the service is healthy or not. But be careful often unhealthy containers are automatically restarted. If you use the xref:health[health information] of your application this may lead to undesired effects. Since the health checks rellies on querying all neighbour systems and infrastucure components, applications often become unhealthy because of 3rd system has problems. Restarting the application itself will not heal the problem and be inexpedient. So generally it is better you query the info endpoint of your application to just check if the service itself is up and running.