Documentation

Monitoring with DataDog

This is a tutorial / how-to on setting up FeatureBase and Datadog to monitor FeatureBase and the system it’s running on. This will expose FeatureBase application metrics and basic system level metrics to Datadog. Go to FeatureBase Monitoring to see the application mertics that will be exposed. Go to Datadog System Checks for the list of system level metrics that will be exposed.

If you’re running on EC2 and would like CloudWatch metrics, follow the Getting Started with Datadog tutorial.

Configuration of FeatureBase

First, configure FeatureBase to write metrics that Datadog can scrape. This is done in your featurebase.conf file. featurebase.conf is likely located at /etc/featurebase.conf but you can search for it by running sudo find / -name featurebase.conf.

Include the following text in featurebase.conf:

[metric]
    service = "prometheus"
    poll-interval = "0m15s"

Note that FeatureBase metrics will be re-written every 15 seconds based on the configuration above. For more on configuring FeatureBase, visit Featurebase Configuration.

To verify FeatureBase is writing prometheus metrics, run curl localhost:10101/metrics on the FeatureBase host and confirm there are FeatureBase metrics being written. Again, for more on FeatureBase metrics, visit FeatureBase Monitoring.

Installation of Datadog Agent

The initial Datadog install can be done by running

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Make sure to add your Datadog API key to the command above.

Configuration of Datadog

There are two configuration files to update. The first is the Datadog configuration file. Save the original and then update datadog.yaml.

sudo cp /etc/datadog-agent/datadog.yaml /etc/datadog-agent/datadog_original.yaml
sudo truncate -s 0 /etc/datadog-agent/datadog.yaml
sudo nano /etc/datadog-agent/datadog.yaml

Copy, paste, save and exit:

api_key: <YOUR_API_KEY>

site: datadoghq.com

tags:
   - owner:<tag-value>
   - project:<tag-value>

logs_enabled: true

process_config:
  process_collection:
    enabled: true

network_config:
  enabled: true

The second file to update is the metrics configuration file. Again, save the original and update openmetrics.d/conf.yaml.

sudo cp /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf_original.yaml
sudo truncate -s 0 /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
sudo nano /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml

Copy, paste, save and exit:

init_config:

instances:

   # update the host and port as necessary
   - prometheus_url: http://localhost:10101/metrics
     namespace: "featurebase"
     metrics:
       - "*"

   # you can grab consumer metrics by adding an entry here
   # here is an example (given the default port for consumer metrics is 9093)
   # - prometheus_url: http://localhost:9093
   #   namespace: "consumer"
   #   metrics:
   #     - "*"

Restart Datadog Agent

sudo systemctl restart datadog-agent

Check that the service unit is running:

sudo systemctl status datadog-agent

Check that prometheus metrics are being pulled into datadog:

sudo -u dd-agent datadog-agent status

You should see something like the following in the returned message:

Running Checks
==============

...

openmetrics (2.0.0)
-------------------
    Instance ID: openmetrics:featurebase:2d670110d7af553d [OK]
    Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
    Total Runs: 1
    Metric Samples: Last Run: 1,302, Total: 1,302
    Events: Last Run: 0, Total: 0
    Service Checks: Last Run: 1, Total: 1
    Average Execution Time : 145ms
    Last Execution Date : 2022-04-27 00:14:25 UTC (1651018465000)
    Last Successful Execution Date : 2022-04-27 00:14:25 UTC (1651018465000)
Something missing or incorrect?
Help improve this article or join us on Discord!

© 2022 Molecula Corp. (DBA FeatureBase). All rights reserved.