> ## Documentation Index
> Fetch the complete documentation index at: https://docs.skyhook.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Sealed Secrets Implementation

This page covers the technical implementation of Sealed Secrets in Skyhook for platform operators and infrastructure teams.

<Info>
  For developer-facing documentation on using environment variables and secrets, see [Environment Variables & Secrets](/application/services/envvars-secrets).
</Info>

## Overview

Skyhook uses [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) to encrypt sensitive configuration data before storing it in Git repositories. This ensures secrets remain secure even when version-controlled alongside application code.

## How Sealed Secrets Work

Sealed Secrets uses asymmetric cryptography to enable GitOps-compatible secret management:

### Architecture

```
Plain-text secret → Skyhook UI
                  ↓ (encrypt with public key)
    Encrypted SealedSecret → Git repository
                  ↓ (deploy via GitOps/ArgoCD)
        SealedSecret resource → Kubernetes cluster
                  ↓ (decrypt with private key)
        Sealed Secrets Controller → Standard Kubernetes Secret
                  ↓ (mount to pod)
            Application pods → Access secret values
```

### Key Components

**Public Key (Cluster-Specific)**:

* Each cluster's Sealed Secrets controller generates a unique public/private key pair
* Skyhook fetches the public key to encrypt secrets
* Public keys are safe to store or share

**Private Key (Cluster-Specific)**:

* Stored securely within the Sealed Secrets controller in the cluster
* Never leaves the cluster
* Used to decrypt SealedSecret resources into standard Kubernetes Secrets

**Sealed Secrets Controller**:

* Runs in the target Kubernetes cluster (typically in the `kube-system` namespace)
* Watches for SealedSecret custom resources
* Decrypts them into standard Secrets that pods can consume
* Manages key rotation and renewal

### Security Model

The security model ensures zero-knowledge architecture:

1. **Encryption happens client-side**: Skyhook encrypts secrets before they leave the UI
2. **Git stores encrypted data**: Only encrypted `SealedSecret` resources are committed to repositories
3. **Decryption happens in-cluster**: Only the target cluster's controller can decrypt secrets
4. **Skyhook has no access**: The platform cannot decrypt secrets; only your clusters can

**Benefits**:

* Secrets are safe to commit to version control
* Git provides audit trail (who changed what, when)
* No central secret store vulnerability
* Compatible with GitOps workflows
* Cluster isolation via cluster-specific keys

## Installing Sealed Secrets

The Sealed Secrets controller must be installed on all clusters where you want to use secrets.

### Installation via Skyhook GitOps

**Recommended**: Install via Skyhook's GitOps settings for automated management.

**Prerequisites**:

1. ArgoCD installed on your management cluster (see [GitOps Setup](/infrastructure/gitops/overview))
2. Clusters connected to Skyhook

**Installation Steps**:

1. Navigate to [GitOps Settings](https://app.skyhook.io/settings/gitops) in Skyhook
2. Under **Featured Addons**, locate **Sealed Secrets controller**
3. Select the clusters where you want secrets support (typically all clusters)
4. Review the default configuration or customize as needed
5. Generate and merge the pull request to deploy the controller

**Default Configuration**:

* Namespace: `kube-system`
* Controller image: `quay.io/bitnami/sealed-secrets-controller`
* Key renewal: Enabled (30-day rotation)
* Service type: `ClusterIP`

### Manual Installation

Alternatively, install Sealed Secrets manually using kubectl or Helm:

**Using kubectl**:

```bash theme={"system"}
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
```

**Using Helm**:

```bash theme={"system"}
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system
```

### Verifying Installation

Check that the controller is running:

```bash theme={"system"}
kubectl get pods -n kube-system -l name=sealed-secrets-controller
```

Expected output:

```
NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-xxxxxxxxxx-xxxxx   1/1     Running   0          5m
```

Fetch the public key (used by Skyhook for encryption):

```bash theme={"system"}
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d
```

## How Skyhook Integrates with Sealed Secrets

### Secret Encryption Flow

When a developer adds or updates a secret in Skyhook:

1. **Fetch Public Key**: Skyhook retrieves the public certificate from the target cluster's Sealed Secrets controller
2. **Encrypt Secret**: The secret value is encrypted client-side using the public key
3. **Create SealedSecret**: A `SealedSecret` Kubernetes custom resource is generated
4. **Store in Git**: The encrypted SealedSecret is committed to the deployment repository
5. **Deploy via GitOps**: ArgoCD or another GitOps tool syncs the SealedSecret to the cluster
6. **Controller Decrypts**: The Sealed Secrets controller decrypts the SealedSecret into a standard Kubernetes Secret
7. **Pods Consume**: Application pods mount the Secret as environment variables or files

### Secret Storage in Git

Sealed Secrets are stored in your deployment repository as Kubernetes custom resources:

```yaml theme={"system"}
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-service-secrets
  namespace: production
spec:
  encryptedData:
    DATABASE_URL: AgA...encrypted...
    STRIPE_API_KEY: AgB...encrypted...
```

The `encryptedData` fields contain encrypted values that are safe to commit to Git.

### Environment-Specific Secrets

Skyhook creates a SealedSecret resource for each service, named after the service itself (e.g., `my-service`) and deployed to the service's namespace. Secrets are environment-specific, with separate encrypted values for each environment (production, staging, development, etc.) stored within the same SealedSecret resource.

This ensures:

* Environment isolation through separate encrypted values
* Different credentials per environment
* Cluster-specific encryption (production cluster can't decrypt staging secrets if using different controllers)

## Reloader Integration

### Overview

Kubernetes does not automatically restart pods when ConfigMaps or Secrets change. [Reloader](https://github.com/stakater/Reloader) solves this by monitoring configuration resources and triggering rolling restarts when changes are detected.

### How Reloader Works

1. **Watch for Changes**: Reloader watches ConfigMaps and Secrets referenced by Deployments, StatefulSets, and DaemonSets
2. **Detect Updates**: When a ConfigMap or Secret is updated, Reloader detects the change
3. **Trigger Restart**: Reloader updates an annotation on the Deployment, triggering a rolling restart
4. **Zero Downtime**: Pods restart gracefully with the new configuration

### Installing Reloader

**Via Skyhook GitOps**:

1. Go to [GitOps Settings](https://app.skyhook.io/settings/gitops)
2. Locate **Reloader** under Featured Addons
3. Select target clusters
4. Generate and merge the pull request

**Manual Installation**:

```bash theme={"system"}
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
```

### Enabling Per-Service

Developers can enable auto-reload for individual services in the **Env Vars & Secrets** tab by checking **Reload on Environment Variables or Secrets Update**.

This adds the necessary annotations to the service's Deployment:

```yaml theme={"system"}
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  # ...
```

## Troubleshooting

### Secrets Not Decrypting

**Symptoms**: SealedSecret exists but no corresponding Secret is created

**Causes**:

* Sealed Secrets controller not installed or not running
* Wrong cluster public key used for encryption
* Controller doesn't have RBAC permissions

**Resolution**:

```bash theme={"system"}
# Check controller status
kubectl get pods -n kube-system -l name=sealed-secrets-controller

# Check controller logs
kubectl logs -n kube-system -l name=sealed-secrets-controller

# Verify SealedSecret resource
kubectl get sealedsecret -n <namespace>
kubectl describe sealedsecret <name> -n <namespace>
```

### Key Rotation Issues

**Symptoms**: Old secrets work but newly created secrets fail to decrypt

**Causes**:

* Cluster certificate was rotated but Skyhook is using old public key
* Manual certificate replacement without proper key backup

**Resolution**:

* Ensure Skyhook refetches the cluster's current public key
* Verify the controller's certificate:
  ```bash theme={"system"}
  kubectl get secret -n kube-system sealed-secrets-key -o yaml
  ```

### Pods Not Reloading

**Symptoms**: Secrets updated but pods still use old values

**Causes**:

* Reloader not installed
* Reloader annotations missing or incorrect
* Reloader not watching the correct namespace

**Resolution**:

```bash theme={"system"}
# Check if Reloader is running
kubectl get pods -n reloader

# Verify Deployment annotations
kubectl get deployment <deployment-name> -n <namespace> -o yaml | grep reloader

# Check Reloader logs
kubectl logs -n reloader -l app=reloader
```

## kubectl Integration

### Sealing Secrets Manually

The `kubeseal` CLI can manually encrypt secrets:

**Install kubeseal**:

```bash theme={"system"}
# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-linux-amd64
chmod +x kubeseal-linux-amd64
sudo mv kubeseal-linux-amd64 /usr/local/bin/kubeseal
```

**Seal a secret**:

```bash theme={"system"}
# Create a Secret (don't apply it)
kubectl create secret generic my-secret \
  --from-literal=DB_PASSWORD=supersecret \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Apply the SealedSecret
kubectl apply -f sealed-secret.yaml
```

### Backing Up Encryption Keys

**Important**: Back up the Sealed Secrets controller's private key to prevent data loss during cluster migrations.

**Export private key**:

```bash theme={"system"}
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml
```

**Restore private key** (on new cluster):

```bash theme={"system"}
kubectl apply -f sealed-secrets-key-backup.yaml
kubectl delete pod -n kube-system -l name=sealed-secrets-controller  # Restart controller
```

## Security Best Practices

### Key Management

* **Backup private keys**: Store sealed-secrets controller keys securely for disaster recovery
* **Rotate keys regularly**: Enable automatic key rotation (default: 30 days)
* **Separate keys per cluster**: Never share private keys between clusters

### Access Control

* **Limit RBAC**: Restrict who can view/edit Secrets and SealedSecrets
* **Audit changes**: Review Git history for unauthorized secret modifications
* **Encrypt Git repos**: Use private repositories with strict access controls

### Secret Hygiene

* **Rotate secrets**: Update API keys, passwords, and tokens periodically
* **Delete unused secrets**: Remove deprecated secrets to reduce attack surface
* **Monitor usage**: Track which secrets are actually being used

## Additional Resources

* [Sealed Secrets GitHub Repository](https://github.com/bitnami-labs/sealed-secrets)
* [Sealed Secrets Documentation](https://sealed-secrets.netlify.app/)
* [Reloader GitHub](https://github.com/stakater/Reloader)
* [Skyhook GitOps Setup](/infrastructure/gitops/overview)
* [How to Manage Secrets in Kubernetes: 5 Approaches Compared](https://www.skyhook.io/blog/how-to-manage-secrets-in-kubernetes) - comparison of Sealed Secrets, SOPS, vault SDKs, External Secrets Operator, and CSI Driver