Monitoring with DataDog
This is a tutorial / how-to on setting up FeatureBase and Datadog to monitor FeatureBase and the system it’s running on. This will expose FeatureBase application metrics and basic system level metrics to Datadog. Go to FeatureBase Monitoring to see the application mertics that will be exposed. Go to Datadog System Checks for the list of system level metrics that will be exposed.
If you’re running on EC2 and would like CloudWatch metrics, follow the Getting Started with Datadog tutorial.
Configuration of FeatureBase
First, configure FeatureBase to write metrics that Datadog can scrape. This is done in your featurebase.conf file. featurebase.conf is likely located at /etc/featurebase.conf but you can search for it by running sudo find / -name featurebase.conf.
Include the following text in featurebase.conf:
[metric]
service = "prometheus"
poll-interval = "0m15s"
Note that FeatureBase metrics will be re-written every 15 seconds based on the configuration above. For more on configuring FeatureBase, visit Featurebase Configuration.
To verify FeatureBase is writing prometheus metrics, run curl localhost:10101/metrics on the FeatureBase host and confirm there are FeatureBase metrics being written. Again, for more on FeatureBase metrics, visit FeatureBase Monitoring.
Installation of Datadog Agent
The initial Datadog install can be done by running
DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
Make sure to add your Datadog API key to the command above.
Configuration of Datadog
There are two configuration files to update. The first is the Datadog configuration file. Save the original and then update datadog.yaml.
sudo cp /etc/datadog-agent/datadog.yaml /etc/datadog-agent/datadog_original.yaml
sudo truncate -s 0 /etc/datadog-agent/datadog.yaml
sudo nano /etc/datadog-agent/datadog.yaml
Copy, paste, save and exit:
api_key: <YOUR_API_KEY>
site: datadoghq.com
tags:
- owner:<tag-value>
- project:<tag-value>
logs_enabled: true
process_config:
process_collection:
enabled: true
network_config:
enabled: true
The second file to update is the metrics configuration file. Again, save the original and update openmetrics.d/conf.yaml.
sudo cp /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf_original.yaml
sudo truncate -s 0 /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
sudo nano /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
Copy, paste, save and exit:
init_config:
instances:
# update the host and port as necessary
- prometheus_url: http://localhost:10101/metrics
namespace: "featurebase"
metrics:
- "*"
# you can grab consumer metrics by adding an entry here
# here is an example (given the default port for consumer metrics is 9093)
# - prometheus_url: http://localhost:9093
# namespace: "consumer"
# metrics:
# - "*"
Restart Datadog Agent
sudo systemctl restart datadog-agent
Check that the service unit is running:
sudo systemctl status datadog-agent
Check that prometheus metrics are being pulled into datadog:
sudo -u dd-agent datadog-agent status
You should see something like the following in the returned message:
Running Checks
==============
...
openmetrics (2.0.0)
-------------------
Instance ID: openmetrics:featurebase:2d670110d7af553d [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
Total Runs: 1
Metric Samples: Last Run: 1,302, Total: 1,302
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 1
Average Execution Time : 145ms
Last Execution Date : 2022-04-27 00:14:25 UTC (1651018465000)
Last Successful Execution Date : 2022-04-27 00:14:25 UTC (1651018465000)