Merge pull request #40 from f5devcentral/development

Release 0.7.0

README.md

@@ -7,8 +7,12 @@ update a legacy configuration, see [pages/config_migration.md](pages/config_migr
 >
 > Before you start, make sure to backup the /config/big-ips.json file!
 
+
 ## Overview
 
+> See the [AST Docsite](https://f5devcentral.github.io/application-study-tool/) for detailed
+configuration, troubleshooting info, etc.
+
 The Application Study Tool is intended to provide enhanced insights into (classic) BIG-IP products, leveraging best in class
 open source telemetry tools. The full installation includes:
 
@@ -27,7 +31,15 @@ the Openetlemetry Collector can be configured to send data to existing productio
 
 ### Prerequisites
 
-docker (or compatible) - [Installation Instructions](https://docs.docker.com/engine/install/)
+[Git Client](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+
+Docker (or compatible) container environment with compose.
+
+Installation Instructions:
+  * [General (docker engine)](https://docs.docker.com/engine/install/)
+  * [Ubuntu (docker engine)](https://docs.docker.com/engine/install/ubuntu/)
+  * [RHEL (docker engine)](https://docs.docker.com/engine/install/rhel/)
+  * [Podman](https://podman.io/docs/installation)
 
 ### Installation
 
@@ -206,9 +218,19 @@ as SENSOR_ID and SENSOR_SECRET_TOKEN (see [.env-example](./.env-example) for exa
 ## Run The Configuration Helper
 The config helper script can be run natively or via docker to merge the default and device
 level configs into the final OTEL Collector config from the project root directory as follows:
-```shell
+
+**Run With Docker**
+```bash
 # Run the configuration generator from the project root directory
-docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+# If `echo $PWD` doesn't give you the current directory on your system,
+# replace the '-v ${PWD}' section with '-v /path/to/your/directory'
+$ docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+```
+
+**Run With System Python**
+```bash
+$ pip install PyYAML==6.0.2
+$ python /app/src/config_helper.py --generate-config
 ```
 
 This will write 2 new files in the services/otel_collector directory:

VERSION

@@ -1 +1 @@
-v0.6.0
+v0.7.0

config/ast_defaults.yaml

@@ -8,6 +8,10 @@ bigip_receiver_defaults:
   # The password (not recommended) or a reference to an env variable (recommended, shown)
   # Below tells the collector to look for an environment variable named BIGIP_PASSWORD_1
   password: "${env:BIGIP_PASSWORD_1}"
+  # The timeout field can be used to adjust the amount of time the collector will wait for a response
+  # to BigIP iControl Rest requests. Larger boxes with more complex config may require setting this value
+  # higher. Set for individual devices in bigip_receivers.yaml
+  timeout: 10s
   # The data_types that should be enabled or disabled. DNS and GTM users can enable those modules
   # by setting the below to true. These will apply to all devices and may be better specified on the
   # per-reciever settings file below.
@@ -44,7 +48,7 @@ pipelines:
   # detail.
   metrics/local:
     #receivers list are generated via the config helper script
-    processors: [batch]
+    processors: [batch/local]
     exporters: [otlphttp/metrics-local, debug/bigip]
 
   # These pipeline configs are written to the OTEL config after having the configured receivers
@@ -53,5 +57,5 @@ pipelines:
   # detail.
   metrics/f5-datafabric:
     #receivers list are generated via the config helper script
-    processors: [batch, interval/f5-datafabric, attributes/f5-datafabric]
+    processors: [interval/f5-datafabric, attributes/f5-datafabric, batch/f5-datafabric]
     exporters: [otlp/f5-datafabric, debug/bigip]

config/bigip_receivers.yaml

@@ -13,6 +13,7 @@ bigip/1:
   # username: SOME_OVERRIDE_ACCOUNT_NAME
   # password: "${SOME_OTHER_ENV_VAR_WITH_ANOTHER_PASSWORD}"
   # collection_interval: 30s
+  # timeout: 20s
   # data_types:
   #   f5.dns:
   #     enabled: false

docker-compose.yaml

@@ -27,12 +27,11 @@ services:
       - 7lc_network
 
   otel-collector:
-    image: ghcr.io/f5devcentral/application-study-tool/otel_custom_collector:v0.6.0
+    image: ghcr.io/f5devcentral/application-study-tool/otel_custom_collector:v0.7.0
     restart: unless-stopped
     volumes:
       - ./services/otel_collector:/etc/otel-collector-config
-    command: 
-      - "--config=/etc/otel-collector-config/defaults/default-config.yaml"
+    command:
       - "--config=/etc/otel-collector-config/defaults/bigip-scraper-config.yaml"
     env_file:
       - ".env"

pages/.gitignore

@@ -1,6 +1,12 @@
-_site
-.sass-cache
-.jekyll-cache
+# These are directly copied from Jekyll's first-party docs on `.gitignore` files:
+# https://jekyllrb.com/tutorials/using-jekyll-with-bundler/#commit-to-source-control
+
+# Ignore the default location of the built site, and caches and metadata generated by Jekyll
+_site/
+.sass-cache/
+.jekyll-cache/
 .jekyll-metadata
-vendor
-Gemfile.lock
+
+# Ignore folders generated by Bundler
+.bundle/
+vendor/

pages/Gemfile

@@ -7,17 +7,16 @@ source "https://rubygems.org"
 #
 # This will help ensure the proper Jekyll version is running.
 # Happy Jekylling!
-# gem "jekyll", "~> 4.3.4"
-gem "github-pages", "~> 232", group: :jekyll_plugins
-# This is the default theme for new Jekyll sites. You may change this to anything you like.
-gem "minima", "~> 2.5"
+gem "jekyll", "~> 4.3.4"
+# gem "github-pages", "~> 232", group: :jekyll_plugins
+gem "just-the-docs", "0.10.0" # pinned to the current release
 # If you want to use GitHub Pages, remove the "gem "jekyll"" above and
 # uncomment the line below. To upgrade, run `bundle update github-pages`.
 # gem "github-pages", group: :jekyll_plugins
 # If you have any plugins, put them here!
-group :jekyll_plugins do
-  gem "jekyll-feed", "~> 0.12"
-end
+# group :jekyll_plugins do
+#   gem "jekyll-feed", "~> 0.12"
+# end
 
 # Windows and JRuby does not include zoneinfo files, so bundle the tzinfo-data gem
 # and associated library.

pages/Gemfile.lock

@@ -0,0 +1,118 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    addressable (2.8.7)
+      public_suffix (>= 2.0.2, < 7.0)
+    bigdecimal (3.1.8)
+    colorator (1.1.0)
+    concurrent-ruby (1.3.4)
+    diff-lcs (1.5.1)
+    em-websocket (0.5.3)
+      eventmachine (>= 0.12.9)
+      http_parser.rb (~> 0)
+    eventmachine (1.2.7)
+    ffi (1.17.0-arm64-darwin)
+    ffi (1.17.0-x86_64-linux-gnu)
+    forwardable-extended (2.6.0)
+    google-protobuf (4.28.1-arm64-darwin)
+      bigdecimal
+      rake (>= 13)
+    google-protobuf (4.28.1-x86_64-linux)
+      bigdecimal
+      rake (>= 13)
+    http_parser.rb (0.8.0)
+    i18n (1.14.6)
+      concurrent-ruby (~> 1.0)
+    jekyll (4.3.4)
+      addressable (~> 2.4)
+      colorator (~> 1.0)
+      em-websocket (~> 0.5)
+      i18n (~> 1.0)
+      jekyll-sass-converter (>= 2.0, < 4.0)
+      jekyll-watch (~> 2.0)
+      kramdown (~> 2.3, >= 2.3.1)
+      kramdown-parser-gfm (~> 1.0)
+      liquid (~> 4.0)
+      mercenary (>= 0.3.6, < 0.5)
+      pathutil (~> 0.9)
+      rouge (>= 3.0, < 5.0)
+      safe_yaml (~> 1.0)
+      terminal-table (>= 1.8, < 4.0)
+      webrick (~> 1.7)
+    jekyll-include-cache (0.2.1)
+      jekyll (>= 3.7, < 5.0)
+    jekyll-sass-converter (3.0.0)
+      sass-embedded (~> 1.54)
+    jekyll-seo-tag (2.8.0)
+      jekyll (>= 3.8, < 5.0)
+    jekyll-watch (2.2.1)
+      listen (~> 3.0)
+    just-the-docs (0.10.0)
+      jekyll (>= 3.8.5)
+      jekyll-include-cache
+      jekyll-seo-tag (>= 2.0)
+      rake (>= 12.3.1)
+    kramdown (2.4.0)
+      rexml
+    kramdown-parser-gfm (1.1.0)
+      kramdown (~> 2.0)
+    liquid (4.0.4)
+    listen (3.9.0)
+      rb-fsevent (~> 0.10, >= 0.10.3)
+      rb-inotify (~> 0.9, >= 0.9.10)
+    mercenary (0.4.0)
+    nokogiri (1.16.7-arm64-darwin)
+      racc (~> 1.4)
+    nokogiri (1.16.7-x86_64-linux)
+      racc (~> 1.4)
+    pathutil (0.16.2)
+      forwardable-extended (~> 2.6)
+    public_suffix (6.0.1)
+    racc (1.8.1)
+    rack (2.2.10)
+    rake (13.2.1)
+    rb-fsevent (0.11.2)
+    rb-inotify (0.11.1)
+      ffi (~> 1.0)
+    rexml (3.3.7)
+    rouge (4.3.0)
+    rspec (3.13.0)
+      rspec-core (~> 3.13.0)
+      rspec-expectations (~> 3.13.0)
+      rspec-mocks (~> 3.13.0)
+    rspec-core (3.13.1)
+      rspec-support (~> 3.13.0)
+    rspec-expectations (3.13.3)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-mocks (3.13.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-support (3.13.1)
+    safe_yaml (1.0.5)
+    sass-embedded (1.78.0-arm64-darwin)
+      google-protobuf (~> 4.27)
+    sass-embedded (1.78.0-x86_64-linux-gnu)
+      google-protobuf (~> 4.27)
+    terminal-table (3.0.2)
+      unicode-display_width (>= 1.1.1, < 3)
+    unicode-display_width (2.6.0)
+    webrick (1.8.1)
+
+PLATFORMS
+  arm64-darwin
+  x86_64-linux-gnu
+
+DEPENDENCIES
+  http_parser.rb (~> 0.6.0)
+  jekyll (~> 4.3.4)
+  just-the-docs (= 0.10.0)
+  nokogiri
+  rack (~> 2.2.4)
+  rspec
+  tzinfo (>= 1, < 3)
+  tzinfo-data
+  wdm (~> 0.1)
+
+BUNDLED WITH
+   2.5.9

pages/_config.yml

@@ -18,13 +18,50 @@
 # You can create any custom variable you would like, and they will be accessible
 # in the templates via {{ site.myvariable }}.
 
-title: Application Study Tool Docs
+title: Application Study Tool
+description: A tool for monitoring BigIP Devices with Grafana, Prometheus, and OpenTelemetry
 
 
 # Build settings
-theme: minima
-plugins:
-  - jekyll-feed
+theme: just-the-docs
+url: https://f5devcentral.github.io/application-study-tool/
+
+aux_links:
+  AST Repository: https://github.com/f5devcentral/application-study-tool/
+
+heading_anchors: true
+nav_enabled: true
+
+nav_sort: case_sensitive
+
+# External navigation links
+nav_external_links:
+  - title: Application Study Tool on GitHub
+    url: https://github.com/f5devcentral/application-study-tool/
+
+# Back to top link
+back_to_top: true
+back_to_top_text: "Back to top"
+
+# Footer last edited timestamp
+last_edit_timestamp: true # show or hide edit time - page must have `last_modified_date` defined in the frontmatter
+last_edit_time_format: "%b %e %Y at %I:%M %p" # uses ruby's time format: https://ruby-doc.org/stdlib-2.7.0/libdoc/time/rdoc/Time.html
+
+compress_html:
+  clippings: all
+  comments: all
+  endings: all
+  startings: []
+  blanklines: false
+  profile: false
+
+callouts:
+  warning:
+    title: Warning
+    color: red
+  important:
+    title: Important
+    color: blue
 
 # Exclude from processing.
 # The following items will not be processed, by default.

pages/components/components.md

@@ -0,0 +1,26 @@
+---
+layout: page
+title: Components
+nav_order: 4
+has_children: true
+permalink: /components
+---
+
+# Components
+
+The Application Study Tool is built using best-of-breed open source components, allowing for 
+flexibility, community support, and continuous improvement. By leveraging established technologies like 
+Grafana, Prometheus, and OpenTelemetry, the AST ensures a reliable and robust method for collecting and
+visualizing data from BigIP devices.
+
+## Grafana
+
+[**Grafana**](https://grafana.com/docs/grafana/latest/) serves as the visualization layer for the Application Study Tool, allowing users to create and manage dashboards that display metrics collected from F5 BigIP devices. With its powerful querying capabilities and customizable visualizations, Grafana enables users to gain insights into application performance and operational metrics effectively.
+
+## Prometheus
+
+[**Prometheus**](https://prometheus.io/docs/introduction/overview/) is the time series database utilized within the Application Study Tool to store and manage metric data. It collects data emitted by the OpenTelemetry Collector and provides powerful querying capabilities for analysis. Prometheus is designed for reliability and scalability, making it the backbone of metric storage for many monitoring applications.
+
+## OpenTelemetry Collector
+
+The [**OpenTelemetry Collector**](https://opentelemetry.io/docs/collector/) is a custom distribution integrated into the Application Study Tool that collects metrics from F5 BigIP devices. Featuring a **BigIP Receiver** component, it retrieves data from the BigIP iControl REST endpoint and transforms it into OpenTelemetry metrics. This data is then forwarded to Prometheus for storage and visualization, ensuring seamless integration across the monitoring stack.