monitoring
This config controls the deployment of Pod/Service Monitors to connect Deep to Prometheus for metric gathering
Type: object
Path: monitoring
enabled
Is monitoring active, set to true to deploy the service monitor
Default: false
Type: boolean
Path: monitoring.enabled
serviceMonitor
ServiceMonitor configuration
Type: object
Path: monitoring.serviceMonitor
clusterLabel
the label to use to identify this cluster
Default: "cluster"
Type: string
Path: monitoring.serviceMonitor.clusterLabel
namespace
Alternative namespace for ServiceMonitor resources
Default: null
Type: string
Path: monitoring.serviceMonitor.namespace
namespaceSelector
Namespace selector for ServiceMonitor resources
Default: {}
Type: string
Path: monitoring.serviceMonitor.namespaceSelector
annotations
ServiceMonitor annotations
Default: {}
Type: string
Path: monitoring.serviceMonitor.annotations
labels
Additional ServiceMonitor labels
Default: {}
Type: string
Path: monitoring.serviceMonitor.labels
interval
ServiceMonitor scrape interval
Default: null
Type: string
Path: monitoring.serviceMonitor.interval
scrapeTimeout
ServiceMonitor scrape timeout in Go duration format (e.g. 15s)
Default: null
Type: string
Path: monitoring.serviceMonitor.scrapeTimeout
relabelings
ServiceMonitor relabel configs to apply to samples before scraping
https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
Default: []
Type: list
Path: monitoring.serviceMonitor.relabelings
metricRelabelings
ServiceMonitor metric relabel configs to apply to samples before ingestion
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#endpoint
Default: []
Type: list
Path: monitoring.serviceMonitor.metricRelabelings
scheme
ServiceMonitor will use http by default, but you can pick https as well
Default: http
Type: string
Path: monitoring.serviceMonitor.scheme
tlsConfig
ServiceMonitor will use these tlsConfig settings to make the health check requests
Default: null
Type: string
Path: monitoring.serviceMonitor.tlsConfig
dashboards
This controls the config for provided dashboards
Type: object
Path: monitoring.dashboards
enabled
if enabled the dashboards will be provided
Default: false
Type: boolean
Path: monitoring.dashboards.enabled
namespace
The namespace to deploy the dashboards in
Default: null
Type: string
Path: monitoring.dashboards.namespace
annotations
Additional annotations to add to the dashboards
Default: {}
Type: string
Path: monitoring.dashboards.annotations
labels
additional labels to add to the dashboards
Type: object
Path: monitoring.dashboards.labels
grafana_dashboard
Default: "1"
Type: string
Path: monitoring.dashboards.labels.grafana_dashboard
rules
This controls the prometheus rules
Type: object
Path: monitoring.rules
enabled
if enabled the prometheus rules will be deployed
Default: false
Type: boolean
Path: monitoring.rules.enabled
labels
additional labels to add to the PrometheusRule
Type: object
Path: monitoring.rules.labels
release
Default: prometheus
Type: string
Path: monitoring.rules.labels.release
annotations
Additional annotations to add to the PrometheusRule
Default: {}
Type: string
Path: monitoring.rules.annotations
namespace
The namespace to deploy the rules in
Default: null
Type: string
Path: monitoring.rules.namespace
alertLabels
Additional labels to add to the alerts
Type: object
Path: monitoring.rules.alertLabels
app_name
Default: '{{ include "deep.fullname" . }}'
Type: string
Path: monitoring.rules.alertLabels.app_name
alertAnnotations
Additional annotations to add to the alerts
Default: {}
Type: string
Path: monitoring.rules.alertAnnotations
additionalRuleExpr
Additional rules to add to all alert rules
Default: ""
Type: string
Path: monitoring.rules.additionalRuleExpr
rules
The rules to add
Type: list
Path: monitoring.rules.rules
0
Type: object
Path: monitoring.rules.rules.0
alert
Default: Unhealthy Ring Member
Type: string
Path: monitoring.rules.rules.0.alert
expr
Default: deep_ring_members{state="Unhealthy"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}} > 0
Type: string
Path: monitoring.rules.rules.0.expr
labels
Type: object
Path: monitoring.rules.rules.0.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.0.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.0.annotations
description
Default: "{{ $labels.service }} has {{ $value }} unhealthy nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.0.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.0.annotations.runbook_url
summary
Default: One or more ring members are unhealthy
Type: string
Path: monitoring.rules.rules.0.annotations.summary
1
Type: object
Path: monitoring.rules.rules.1
alert
Default: Required Ring Member Missing
Type: string
Path: monitoring.rules.rules.1.alert
expr
Default: deep_ring_members{state="ACTIVE", cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}, name=~"ingester|tracepoint"} == 0
Type: string
Path: monitoring.rules.rules.1.expr
labels
Type: object
Path: monitoring.rules.rules.1.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.1.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.1.annotations
description
Default: "{{ $labels.service }} has {{ $value }} active nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.1.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.1.annotations.runbook_url
summary
Default: One or more ring has missing required nodes
Type: string
Path: monitoring.rules.rules.1.annotations.summary
2
Type: object
Path: monitoring.rules.rules.2
alert
Default: Tracepoint ring missing member
Type: string
Path: monitoring.rules.rules.2.alert
expr
Default: deep_ring_members{state="ACTIVE", cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}, name="tracepoint"} != {{ .Values.tracepoint.replicas }}
Type: string
Path: monitoring.rules.rules.2.expr
labels
Type: object
Path: monitoring.rules.rules.2.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.2.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.2.annotations
description
Default: "{{ $labels.service }} has {{ $value }} active nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.2.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.2.annotations.runbook_url
summary
Default: One or more ring has missing required nodes
Type: string
Path: monitoring.rules.rules.2.annotations.summary
3
Type: object
Path: monitoring.rules.rules.3
alert
Default: Ingester ring missing member
Type: string
Path: monitoring.rules.rules.3.alert
expr
Default: deep_ring_members{state="ACTIVE", cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}, name="ingester"} != {{ .Values.ingester.replicas }}
Type: string
Path: monitoring.rules.rules.3.expr
labels
Type: object
Path: monitoring.rules.rules.3.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.3.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.3.annotations
description
Default: "{{ $labels.service }} has {{ $value }} active nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.3.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.3.annotations.runbook_url
summary
Default: One or more ring has missing required nodes
Type: string
Path: monitoring.rules.rules.3.annotations.summary
4
Type: object
Path: monitoring.rules.rules.4
alert
Default: Compactor ring missing member
Type: string
Path: monitoring.rules.rules.4.alert
expr
Default: deep_ring_members{state="ACTIVE", cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}, name="compactor"} != {{ .Values.compactor.replicas }}
Type: string
Path: monitoring.rules.rules.4.expr
labels
Type: object
Path: monitoring.rules.rules.4.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.4.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.4.annotations
description
Default: "{{ $labels.service }} has {{ $value }} active nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.4.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.4.annotations.runbook_url
summary
Default: One or more ring has missing required nodes
Type: string
Path: monitoring.rules.rules.4.annotations.summary
5
Type: object
Path: monitoring.rules.rules.5
alert
Default: Metrics Generator ring missing member
Type: string
Path: monitoring.rules.rules.5.alert
expr
Default: deep_ring_members{state="ACTIVE", cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}, name="metrics-generator"} != {{ if .Values.metricsGenerator.enabled }} {{.Values.distributor.replicas }} {{else}}0{{end}}
Type: string
Path: monitoring.rules.rules.5.expr
labels
Type: object
Path: monitoring.rules.rules.5.labels
severity
Default: critical
Type: string
Path: monitoring.rules.rules.5.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.5.annotations
description
Default: "{{ $labels.service }} has {{ $value }} active nodes in ring {{ $labels.name }}."
Type: string
Path: monitoring.rules.rules.5.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.5.annotations.runbook_url
summary
Default: One or more ring has missing required nodes
Type: string
Path: monitoring.rules.rules.5.annotations.summary
6
Type: object
Path: monitoring.rules.rules.6
alert
Default: Rapid increase in blocks
Type: string
Path: monitoring.rules.rules.6.alert
expr
Default: delta(deep_db_blocklist_length{cluster="{{ include "deep.clusterName" .}}"{{ tpl .Values.monitoring.rules.additionalRuleExpr $ }}}[24h]) > 30
Type: string
Path: monitoring.rules.rules.6.expr
for
Default: 48h
Type: string
Path: monitoring.rules.rules.6.for
labels
Type: object
Path: monitoring.rules.rules.6.labels
severity
Default: warning
Type: string
Path: monitoring.rules.rules.6.labels.severity
annotations
Type: object
Path: monitoring.rules.rules.6.annotations
description
Default: "Tenant
Type: string
Path: monitoring.rules.rules.6.annotations.description
runbook_url
Default: https
Type: string
Path: monitoring.rules.rules.6.annotations.runbook_url
summary
Default: Tenant has seen large increase in blocks.
Type: string
Path: monitoring.rules.rules.6.annotations.summary