Sealed Secrets Implementation

​

Overview

​

How Sealed Secrets Work

​

Architecture

Plain-text secret → Skyhook UI
                  ↓ (encrypt with public key)
    Encrypted SealedSecret → Git repository
                  ↓ (deploy via GitOps/ArgoCD)
        SealedSecret resource → Kubernetes cluster
                  ↓ (decrypt with private key)
        Sealed Secrets Controller → Standard Kubernetes Secret
                  ↓ (mount to pod)
            Application pods → Access secret values

​

Key Components

• Each cluster’s Sealed Secrets controller generates a unique public/private key pair
• Skyhook fetches the public key to encrypt secrets
• Public keys are safe to store or share

• Stored securely within the Sealed Secrets controller in the cluster
• Never leaves the cluster
• Used to decrypt SealedSecret resources into standard Kubernetes Secrets

• Runs in the target Kubernetes cluster (typically in the kube-system namespace)
• Watches for SealedSecret custom resources
• Decrypts them into standard Secrets that pods can consume
• Manages key rotation and renewal

​

Security Model

1. Encryption happens client-side: Skyhook encrypts secrets before they leave the UI
2. Git stores encrypted data: Only encrypted SealedSecret resources are committed to repositories
3. Decryption happens in-cluster: Only the target cluster’s controller can decrypt secrets
4. Skyhook has no access: The platform cannot decrypt secrets; only your clusters can

• Secrets are safe to commit to version control
• Git provides audit trail (who changed what, when)
• No central secret store vulnerability
• Compatible with GitOps workflows
• Cluster isolation via cluster-specific keys

​

Installing Sealed Secrets

​

Installation via Skyhook GitOps

1. ArgoCD installed on your management cluster (see GitOps Setup)
2. Clusters connected to Skyhook

1. Navigate to GitOps Settings in Skyhook
2. Under Featured Addons, locate Sealed Secrets controller
3. Select the clusters where you want secrets support (typically all clusters)
4. Review the default configuration or customize as needed
5. Generate and merge the pull request to deploy the controller

• Namespace: kube-system
• Controller image: quay.io/bitnami/sealed-secrets-controller
• Key renewal: Enabled (30-day rotation)
• Service type: ClusterIP

​

Manual Installation

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system

​

Verifying Installation

kubectl get pods -n kube-system -l name=sealed-secrets-controller

NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-xxxxxxxxxx-xxxxx   1/1     Running   0          5m

kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 -d

​

How Skyhook Integrates with Sealed Secrets

​

Secret Encryption Flow

1. Fetch Public Key: Skyhook retrieves the public certificate from the target cluster’s Sealed Secrets controller
2. Encrypt Secret: The secret value is encrypted client-side using the public key
3. Create SealedSecret: A SealedSecret Kubernetes custom resource is generated
4. Store in Git: The encrypted SealedSecret is committed to the deployment repository
5. Deploy via GitOps: ArgoCD or another GitOps tool syncs the SealedSecret to the cluster
6. Controller Decrypts: The Sealed Secrets controller decrypts the SealedSecret into a standard Kubernetes Secret
7. Pods Consume: Application pods mount the Secret as environment variables or files

​

Secret Storage in Git

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-service-secrets
  namespace: production
spec:
  encryptedData:
    DATABASE_URL: AgA...encrypted...
    STRIPE_API_KEY: AgB...encrypted...

​

Environment-Specific Secrets

• Environment isolation through separate encrypted values
• Different credentials per environment
• Cluster-specific encryption (production cluster can’t decrypt staging secrets if using different controllers)

​

Reloader Integration

​

Overview

​

How Reloader Works

1. Watch for Changes: Reloader watches ConfigMaps and Secrets referenced by Deployments, StatefulSets, and DaemonSets
2. Detect Updates: When a ConfigMap or Secret is updated, Reloader detects the change
3. Trigger Restart: Reloader updates an annotation on the Deployment, triggering a rolling restart
4. Zero Downtime: Pods restart gracefully with the new configuration

​

Installing Reloader

1. Go to GitOps Settings
2. Locate Reloader under Featured Addons
3. Select target clusters
4. Generate and merge the pull request

kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

​

Enabling Per-Service

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  # ...

​

Troubleshooting

​

Secrets Not Decrypting

• Sealed Secrets controller not installed or not running
• Wrong cluster public key used for encryption
• Controller doesn’t have RBAC permissions

# Check controller status
kubectl get pods -n kube-system -l name=sealed-secrets-controller

# Check controller logs
kubectl logs -n kube-system -l name=sealed-secrets-controller

# Verify SealedSecret resource
kubectl get sealedsecret -n <namespace>
kubectl describe sealedsecret <name> -n <namespace>

​

Key Rotation Issues

• Cluster certificate was rotated but Skyhook is using old public key
• Manual certificate replacement without proper key backup

• Ensure Skyhook refetches the cluster’s current public key
• Verify the controller’s certificate:
  Copy

  kubectl get secret -n kube-system sealed-secrets-key -o yaml
  

​

Pods Not Reloading

• Reloader not installed
• Reloader annotations missing or incorrect
• Reloader not watching the correct namespace

# Check if Reloader is running
kubectl get pods -n reloader

# Verify Deployment annotations
kubectl get deployment <deployment-name> -n <namespace> -o yaml | grep reloader

# Check Reloader logs
kubectl logs -n reloader -l app=reloader

​

kubectl Integration

​

Sealing Secrets Manually

# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-linux-amd64
chmod +x kubeseal-linux-amd64
sudo mv kubeseal-linux-amd64 /usr/local/bin/kubeseal

# Create a Secret (don't apply it)
kubectl create secret generic my-secret \
  --from-literal=DB_PASSWORD=supersecret \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Apply the SealedSecret
kubectl apply -f sealed-secret.yaml

​

Backing Up Encryption Keys

kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml

kubectl apply -f sealed-secrets-key-backup.yaml
kubectl delete pod -n kube-system -l name=sealed-secrets-controller  # Restart controller

​

Security Best Practices

​

Key Management

• Backup private keys: Store sealed-secrets controller keys securely for disaster recovery
• Rotate keys regularly: Enable automatic key rotation (default: 30 days)
• Separate keys per cluster: Never share private keys between clusters

​

Access Control

• Limit RBAC: Restrict who can view/edit Secrets and SealedSecrets
• Audit changes: Review Git history for unauthorized secret modifications
• Encrypt Git repos: Use private repositories with strict access controls

​

Secret Hygiene

• Rotate secrets: Update API keys, passwords, and tokens periodically
• Delete unused secrets: Remove deprecated secrets to reduce attack surface
• Monitor usage: Track which secrets are actually being used

​

Additional Resources

• Sealed Secrets GitHub Repository
• Sealed Secrets Documentation
• Reloader GitHub
• Skyhook GitOps Setup