Monitorig the process with Grafana

When deployed to the cloud service, handoff creates logging resources. The logs are easily parsed and visualized with the dashboarding tools like Grafana.

Here are some resources to get started.

Grafana for AWS CloudWatch logs

handoff’s default cloud provider is AWS. In this case, CloudWatch logs can be visualized with Grafana.

Tips:

  • When setting AWS CloudWatch data source on Grafana dashboard, make sure there is a .aws/credentials file accessible by the user running Grafana. When running on Ubuntu and authenticating AWS with a crendential file, you may need to keep a copy at /usr/share/grafana/.aws/credentials.
  • handoff creates a log group per task with the following naming convention: <resource-name>-<task-name>
  • When adding a query on Grafana Panel, set Query Mode to “CloudWatch Logs” and enter an Insigh query. For example, here is a query to extract Singer’s metrics:
fields @timestamp, @message
| filter @message like /METRIC/
| parse "* *: {\"type\": \"*\", \"metric\": \"*\", \"value\": *, *}" as log_level, log_type, singer_type, singer_metric, singer_value, rest
| filter singer_type = "counter"
| stats max(singer_value) as rows_loaded by bin(4h)
  • You can also count the errors and send an alert. Our suggestion for a beginner is to create a free PagerDuty account and create a new service from https://<your-domain>.pagerduty.com/service-directory. Select AWS CloudWatch as Integration Type and obtain the integration key to use it on Grafana alert setup.
  • Here is an example query for filtering errors from the logs and count:
fields @timestamp, @log, @message
| filter @message like /(CRITICAL|Error|error)/
| count() as errors by bin(1h)