io7m | single-page | multi-page | epub | Certusine User Manual
3. Configuration
Previous
Front Matter
Up
5. Certificate Outputs
Next

# 4. Monitoring

- 4.1. Overview
- 4.2. Metrics
  - 4.2.1. Names
  - 4.2.2. Alerting Rules
    - 4.2.2.1. PromQL
- 4.3. Logs
  - 4.3.1. Overview
  - 4.3.2. CSEventCertificateDNSChallengeFailed
  - 4.3.3. CSEventCertificateRenewalFailed
  - 4.3.4. CSEventCertificateRenewalSucceeded
  - 4.3.5. CSEventCertificateSigningFailed
  - 4.3.6. CSEventCertificateStored
  - 4.3.7. CSEventCertificateStoreFailed
  - 4.3.8. Alerting Rules
    - 4.3.8.1. Grafana Loki
- 4.4. Traces
- 4.5. Grafana Dashboard

## 4.1. Overview

1
The certusine package is extensively instrumented with OpenTelemetry in order to allow for the certificate renewal process to be continually monitored. The package publishes metrics, logs, and traces, all of which can be independently enabled or disabled. Most installations will only want to enable metrics or logs in production; traces are more useful when trying to diagnose performance problems, or for doing actual development on the certusine package.

## 4.2. Metrics

## 4.2.1. Names

1
The package publishes the following metrics that can be used for monitoring:

### 4.2.1.2. Metrics

Name | Description
--- | ---
certusine_up | A gauge that produces a constant value of 1 for as long as the certusine client is running.
certusine_dns_challenge_failures | A counter that is incremented every time a DNS challenge fails during certificate renewal.
certusine_signing_failures | A counter that is incremented every time certificate signing fails during certificate renewal.
certusine_renewal_failures | A counter that is incremented every time certificate renewal completely fails. This is an important metric for error monitoring; any non-zero value for this metric should be considered a problem.
certusine_renewal_successes | A counter that is incremented every time certificate renewal completely succeeds. This is an important metric for error monitoring; if this metric remains continually at zero, this should likely be considered a problem.
certusine_certificates_stored | A counter that is incremented every time certificates are stored. The package redundantly stores certificates many times, so this metric will be incremented frequently.
certusine_certificates_store_failures | A counter that is incremented every time certificates fail to be stored. If this metric is trending upwards, it may mean that certificates are not being distributed to systems that depend upon them.
certusine_certificate_time_remaining | A gauge that displays the age of all the certificates that are currently in the certificate store. Certificates can be distinguished by their domain and certificate name labels/attributes. This is an important metric for alerting.
certusine_certificate_expiration_threshold | A gauge that displays the configured expiration threshold in seconds. This is useful when combined with the certusine_certificate_time_remaining metric for alerting.

3
The package may produce other metrics, however these are undocumented and should not be relied upon.

## 4.2.2. Alerting Rules

## 4.2.2.1. PromQL

1
On a system that supports the querying of metrics using PromQL, alerting rules should be used to notify administrators that certificate renewals are failing.
2
Assuming that certificate renewal attempts are made at 30 minute intervals, the following rule can be used to capture failures:

### 4.2.2.1.3. Alert

    rate(certusine_renewal_failures_total[30m]) > bool 0

4
The rate of the certusine_renewal_failures_total metric should be zero over any given interval in a stable system.
5
Another useful metric to monitor for alerting is the certusine_certificate_time_remaining metric. ACME-issued certificates generally have relatively short expiration times. The de-facto standard Let's Encrypt service issues certificates with a 90-day expiration. If the certusine package is configured to renew any certificate with less than N seconds remaining before expiration (the conventional value being 72 hours, or 259200 seconds), and the expiration time of any certificate drops below, say, N / 2, then this clearly indicates that something is wrong and certificates are not being renewed as expected. An alerting rule for this scenario could be written such as:

### 4.2.2.1.6. Alert

    certusine_certificate_time_remaining{} < 252000

7
A value of 252000 (70 hours in seconds) is chosen arbitrarily. A working system should have renewed certificates minutes after the 72-hour renewal threshold. This rule could be improved by using the certusine_certificate_expiration_threshold metric:

### 4.2.2.1.8. Alert

    max(certusine_certificate_time_remaining{}) < bool max(certusine_certificate_expiration_threshold{})

9
The alerting rule would ideally be configured to fire if the above condition remains true for an hour or more. The rule is robust in the face of configuration changes; if the expiration threshold is changed, the value of the certusine_certificate_expiration_threshold metric will automatically update to reflect the change.

## 4.3. Logs

## 4.3.1. Overview

1
The certusine package is conservative in the amount of logging output it produces by default. The package is written to publish only a specific set of log messages to telemetry logs in order to increase the signal-to-noise ratio. The certusine package publishes strongly-typed events internally that are mapped directly to OpenTelemetry log messages, using attributes to hold the fields of the events, and using the certusine.type attribute to hold the name of the event type. The intention is to make it very easy to write alerting rules simply by counting occurrences of log messages that have a given type.

## 4.3.2. CSEventCertificateDNSChallengeFailed

1
The CSEventCertificateDNSChallengeFailed event type indicates that a DNS challenge failed during certificate renewal.

### 4.3.2.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateDNSChallengeFailed
certusine.domain | The domain containing the certificate that failed.
certusine.certificate | The name of the certificate that failed.

## 4.3.3. CSEventCertificateRenewalFailed

1
The CSEventCertificateRenewalFailed event type indicates that a certificate renewal completely failed. All occurrences of this log event should be considered a problem that needs to be fixed.

### 4.3.3.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateRenewalFailed
certusine.domain | The domain containing the certificate that failed.
certusine.certificate | The name of the certificate that failed.

## 4.3.4. CSEventCertificateRenewalSucceeded

1
The CSEventCertificateRenewalSucceeded event type indicates that a certificate renewal completely succeeded.

### 4.3.4.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateRenewalSucceeded
certusine.domain | The domain that succeeded.
certusine.certificate | The name of the certificate that was renewed.

## 4.3.5. CSEventCertificateSigningFailed

1
The CSEventCertificateSigningFailed event type indicates that a certificate could not be signed.

### 4.3.5.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateSigningFailed
certusine.domain | The domain containing the certificate that failed.
certusine.certificate | The name of the certificate that failed.

## 4.3.6. CSEventCertificateStored

1
The CSEventCertificateStored event type indicates that a certificate was stored.

### 4.3.6.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateStored
certusine.domain | The domain containing the certificate that was stored.
certusine.certificate | The name of the certificate that was stored.
certusine.target | The name of the output to which the certificate was stored.

## 4.3.7. CSEventCertificateStoreFailed

1
The CSEventCertificateStoreFailed event type indicates that a certificate could not be stored to an output. Occurrences of this event type may indicate that certificates are not reaching systems that depend on them (for example, if an application is repeatedly failing to publish certificates to directory, applications reading certificates from that directly may start serving expired certificates as newly renewed ones aren't getting through).

### 4.3.7.2. Attributes

Name | Description
--- | ---
certusine.type | CSEventCertificateStoreFailed
certusine.domain | The domain containing the certificate that could not be stored.
certusine.certificate | The name of the certificate that could not be stored.
certusine.target | The name of the output to which the certificate could not be stored.

## 4.3.8. Alerting Rules

## 4.3.8.1. Grafana Loki

1
Assuming that certificate renewal attempts are made at 30 minute intervals, the following rule should be used to capture log events of type CSEventCertificateRenewalFailed:

### 4.3.8.1.2. Alert

    count(rate({level="ERROR"} | json | attributes_certusine_type=`CSEventCertificateRenewalFailed`[30m])) > bool 0

3
The rule captures messages at ERROR severity, unpacks the included attributes using the json operator, and then filters messages based on the generated attributes_certusine_type label. Occurences of matching messages are counted over the previous 30-minute interval, and transformed to a "boolean" numeric value using the greater-than operator.

## 4.4. Traces

1
The certusine package publishes traces for all internal operations. No specific documentation is provided on the structures of the traces as they are effectively tied to the internal structure of the code and are subject to change.

### 4.4.2. Trace Display

## 4.5. Grafana Dashboard

1
The certusine package provides a basic example Grafana dashboard. The dashboard can be imported from https://raw.githubusercontent.com/io7m/certusine/develop/com.io7m.certusine.grafana/src/main/resources/com/io7m/certusine/grafana/dashboard.json.
3. Configuration
Previous
Front Matter
Up
5. Certificate Outputs
Next
io7m | single-page | multi-page | epub | Certusine User Manual