> ## Documentation Index
> Fetch the complete documentation index at: https://docs.skyhook.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

> Prometheus, Mimir, and metrics collection for Kubernetes monitoring

## Overview

Metrics provide numerical measurements of your system's performance over time. The Observability Bundle includes four components that work together to collect, store, and visualize metrics from your Kubernetes clusters:

* **Prometheus**: Collects and stores short-term metrics
* **Grafana Mimir**: Long-term metrics storage and querying
* **Kube State Metrics**: Exposes Kubernetes object state as metrics
* **Prometheus Node Exporter**: Exposes node hardware and OS metrics

## Prometheus

Prometheus is the industry-standard monitoring system and time-series database for cloud-native environments. It automatically discovers services in Kubernetes and scrapes metrics from them.

### What Prometheus Monitors

**Application Metrics**:

* Request rates, latencies, error rates (RED metrics)
* Custom business metrics from your applications
* Metrics from `/metrics` endpoints

**Kubernetes Metrics**:

* Pod CPU and memory usage
* Container resource consumption
* Service endpoint availability

**Integration**:

* Scrapes metrics every 15-30 seconds (configurable)
* Stores data locally for 15-30 days (typical configuration)
* Remote writes to Mimir for long-term retention

### PromQL Basics

Prometheus uses PromQL (Prometheus Query Language) for querying metrics.

**Example queries**:

```promql theme={"system"}
# CPU usage by pod
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

# Memory usage by namespace
sum(container_memory_usage_bytes) by (namespace)

# Request rate for a service
rate(http_requests_total{job="my-service"}[5m])

# 95th percentile response time
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)
```

### Configuration

Prometheus is configured via the Observability Bundle's GitOps workflow. Key configuration options:

```yaml theme={"system"}
enabled: true
valuesObject:
  # Scrape interval
  scrapeInterval: 30s

  # Retention period
  retention: 15d

  # Remote write to Mimir
  remoteWrite:
    - url: http://mimir-gateway/api/v1/push

  # Resource limits
  resources:
    limits:
      memory: 4Gi
      cpu: 2
```

For detailed Helm chart values, see the [Prometheus chart documentation](https://artifacthub.io/packages/helm/prometheus-community/prometheus).

## Grafana Mimir

Mimir provides long-term, scalable storage for Prometheus metrics. It's designed to handle billions of active time series across multiple tenants.

### Why Mimir?

**Long-term Retention**:

* Stores metrics for months or years
* Prometheus typically keeps 15-30 days locally
* Historical analysis and capacity planning

**Horizontal Scalability**:

* Scales to handle millions of samples per second
* Distributed architecture handles large metric volumes
* No single point of failure

**Prometheus Compatible**:

* Receives data via Prometheus remote write
* Queries with PromQL
* Drop-in replacement for long-term storage

### How It Works

```
Prometheus ──[remote write]──> Mimir ──[PromQL queries]──> Grafana
    │                                                           │
    └─────────────[PromQL queries for recent data]─────────────┘
```

Grafana can query both Prometheus (recent data) and Mimir (historical data) simultaneously, providing seamless access to both real-time and long-term metrics.

### Configuration

```yaml theme={"system"}
enabled: true
valuesObject:
  # Storage backend (S3, GCS, Azure Blob)
  storage:
    backend: s3
    s3:
      endpoint: s3.amazonaws.com
      bucket: my-mimir-metrics

  # Retention period
  limits:
    compactor_blocks_retention_period: 1y

  # Ingestion limits
  ingester:
    ring:
      replication_factor: 3
```

For detailed configuration, see the [Mimir chart documentation](https://artifacthub.io/packages/helm/grafana/mimir-distributed).

## Kube State Metrics

Kube State Metrics generates metrics about the state of Kubernetes objects. Unlike metrics from the Kubernetes API server, kube-state-metrics focuses on the state of the objects (e.g., deployments, pods, nodes) rather than their resource consumption.

### What It Exposes

**Deployment Metrics**:

* `kube_deployment_status_replicas`: Number of desired replicas
* `kube_deployment_status_replicas_available`: Number of available replicas
* `kube_deployment_status_replicas_unavailable`: Number of unavailable replicas

**Pod Metrics**:

* `kube_pod_status_phase`: Pod phase (Pending, Running, Succeeded, Failed)
* `kube_pod_status_ready`: Whether pod is ready
* `kube_pod_container_status_restarts_total`: Container restart count

**Node Metrics**:

* `kube_node_status_condition`: Node conditions (Ready, MemoryPressure, DiskPressure)
* `kube_node_status_allocatable`: Allocatable resources per node
* `kube_node_status_capacity`: Total capacity per node

### Use Cases

**Cluster Health Monitoring**:

```promql theme={"system"}
# Pods not in Running state
count(kube_pod_status_phase{phase!="Running"})

# Nodes with conditions other than Ready
sum(kube_node_status_condition{condition!="Ready",status="true"})
```

**Capacity Planning**:

```promql theme={"system"}
# Available CPU capacity across cluster
sum(kube_node_status_allocatable{resource="cpu"})

# Pod resource requests vs node capacity
sum(kube_pod_container_resource_requests{resource="memory"}) /
sum(kube_node_status_capacity{resource="memory"})
```

**Deployment Monitoring**:

```promql theme={"system"}
# Deployments with unavailable replicas
kube_deployment_status_replicas_unavailable > 0

# Deployments not at desired replica count
kube_deployment_status_replicas != kube_deployment_spec_replicas
```

### Configuration

Kube State Metrics typically requires minimal configuration:

```yaml theme={"system"}
enabled: true
valuesObject:
  # Resource limits
  resources:
    limits:
      memory: 256Mi
      cpu: 100m

  # Which resources to monitor (default: all)
  collectors:
    - deployments
    - pods
    - nodes
    - services
    - configmaps
```

For more details, see the [Kube State Metrics chart](https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics).

## Prometheus Node Exporter

The Node Exporter exposes hardware and OS metrics from Kubernetes nodes. It runs as a DaemonSet (one pod per node) to collect node-level performance data.

### What It Exposes

**CPU Metrics**:

* `node_cpu_seconds_total`: CPU time spent in different modes (user, system, idle)
* Used to calculate CPU usage percentages

**Memory Metrics**:

* `node_memory_MemTotal_bytes`: Total memory
* `node_memory_MemAvailable_bytes`: Available memory
* `node_memory_Cached_bytes`: Cached memory

**Disk Metrics**:

* `node_filesystem_size_bytes`: Filesystem size
* `node_filesystem_avail_bytes`: Available space
* `node_disk_io_time_seconds_total`: Disk I/O time

**Network Metrics**:

* `node_network_receive_bytes_total`: Bytes received
* `node_network_transmit_bytes_total`: Bytes transmitted
* `node_network_receive_errors_total`: Receive errors

### Use Cases

**Node Resource Monitoring**:

```promql theme={"system"}
# CPU usage by node
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage by node
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Disk usage by mount point
100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)
```

**Network Monitoring**:

```promql theme={"system"}
# Network traffic by node
rate(node_network_receive_bytes_total[5m])

# Network errors
rate(node_network_receive_errors_total[5m]) > 0
```

**Disk I/O**:

```promql theme={"system"}
# Disk I/O operations
rate(node_disk_io_time_seconds_total[5m])
```

### Configuration

Node Exporter runs as a DaemonSet with host network and pid namespace access:

```yaml theme={"system"}
enabled: true
valuesObject:
  # Host network access for accurate metrics
  hostNetwork: true
  hostPID: true

  # Resource limits
  resources:
    limits:
      memory: 128Mi
      cpu: 100m
```

For more configuration options, see the [Node Exporter chart](https://artifacthub.io/packages/helm/prometheus-community/prometheus-node-exporter).

## Visualizing Metrics in Grafana

Once metrics are being collected, Grafana provides powerful visualization capabilities.

### Pre-built Dashboards

The Observability Bundle includes dashboards for common monitoring needs:

* **Kubernetes Cluster Monitoring**: Overview of cluster health and resource usage
* **Node Metrics**: Detailed node performance (from Node Exporter)
* **Pod Metrics**: CPU, memory, network by pod
* **Deployment Status**: Replica counts, pod states

### Creating Custom Dashboards

1. Navigate to Grafana (access via ingress URL configured during setup)
2. Click **Dashboards** → **New Dashboard**
3. Add a panel and select **Prometheus** or **Mimir** as data source
4. Write PromQL queries to visualize your metrics

**Example panel configuration**:

* **Panel Title**: "API Request Rate"
* **Data Source**: Prometheus
* **Query**: `rate(http_requests_total{job="api-service"}[5m])`
* **Visualization**: Time series graph

### Alerting

Grafana can alert based on metric thresholds:

1. Create an alert rule in a dashboard panel
2. Define alert conditions (e.g., CPU > 80% for 5 minutes)
3. Configure notification channels (email, Slack, PagerDuty)

## Accessing Grafana

To access Grafana and view your metrics:

1. Get the Grafana URL from your Observability Bundle configuration (configured during setup)
2. Log in with the credentials you configured:
   * **Username**: admin (default)
   * **Password**: Set during bundle configuration

If you need to retrieve the password:

```bash theme={"system"}
kubectl get secret grafana -o jsonpath="{.data.admin-password}" -n observability | base64 --decode
```

<Warning>
  For security, change the default admin password and create additional user accounts as needed through the Grafana UI.
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Prometheus not scraping metrics" icon="chart-bar">
    **Check Prometheus targets**:

    1. Access Prometheus UI (typically at `http://prometheus:9090`)
    2. Go to **Status** → **Targets**
    3. Look for targets in "down" state

    **Common issues**:

    * Network policies blocking Prometheus
    * Service selector not matching pods
    * Pods don't expose `/metrics` endpoint

    **Verify metrics endpoint**:

    ```bash theme={"system"}
    kubectl port-forward pod/my-app-pod 8080:8080
    curl http://localhost:8080/metrics
    ```
  </Accordion>

  <Accordion title="High memory usage in Prometheus" icon="memory">
    Prometheus memory usage scales with:

    * Number of time series (unique label combinations)
    * Scrape interval (more frequent = more data)
    * Retention period (longer = more data stored)

    **Solutions**:

    * Reduce retention period (keep 7-15 days, use Mimir for long-term)
    * Increase scrape interval (30s → 60s)
    * Drop unnecessary metrics using relabeling
    * Add more memory to Prometheus pods

    **Check cardinality**:

    ```bash theme={"system"}
    # Access Prometheus UI
    # Go to Status → TSDB Status
    # Look for series with high cardinality
    ```
  </Accordion>

  <Accordion title="Mimir remote write failing" icon="database">
    **Check Prometheus logs**:

    ```bash theme={"system"}
    kubectl logs -n observability prometheus-server-0 | grep "remote write"
    ```

    **Common issues**:

    * Mimir gateway not accessible (network/DNS issues)
    * Authentication failure (check credentials)
    * Rate limiting (ingestion rate too high)

    **Verify Mimir is running**:

    ```bash theme={"system"}
    kubectl get pods -n observability | grep mimir
    ```
  </Accordion>

  <Accordion title="Missing node or cluster metrics" icon="server">
    **Verify kube-state-metrics is running**:

    ```bash theme={"system"}
    kubectl get pods -n observability | grep kube-state-metrics
    ```

    **Verify node-exporter is running on all nodes**:

    ```bash theme={"system"}
    kubectl get pods -n observability -l app=prometheus-node-exporter -o wide
    # Should show one pod per node
    ```

    **Check Prometheus is scraping these exporters**:

    * Access Prometheus UI → Status → Targets
    * Look for `kube-state-metrics` and `node-exporter` targets
  </Accordion>
</AccordionGroup>

## Best Practices

<CardGroup cols={2}>
  <Card title="Metric Naming" icon="tag">
    Follow Prometheus naming conventions:

    * Use base unit (seconds, not milliseconds)
    * Append `_total` for counters
    * Append `_bucket` for histograms
    * Example: `http_request_duration_seconds`
  </Card>

  <Card title="Label Usage" icon="tags">
    Keep cardinality low:

    * Avoid high-cardinality labels (user IDs, timestamps)
    * Use service name, environment, region as labels
    * Don't create unique label values for every request
  </Card>

  <Card title="Retention Strategy" icon="clock">
    Balance cost and utility:

    * Prometheus: 7-15 days (recent, high-resolution)
    * Mimir: months to years (long-term, downsampled)
    * Adjust based on storage costs and query patterns
  </Card>

  <Card title="Resource Limits" icon="gauge">
    Right-size resource allocations:

    * Prometheus memory scales with active series
    * Monitor and adjust based on actual usage
    * Use horizontal scaling for very large deployments
  </Card>
</CardGroup>

## Next Steps

* [View Logs](/observability/logs) - Learn about log aggregation with Loki
* [Configure Tracing](/observability/tracing) - Set up distributed tracing
* [Observability Overview](/observability/overview) - Return to bundle overview