Secret Management

This directory contains documentation for Blueberry IDP's secret management workflows and security patterns.

Overview

Blueberry IDP implements a comprehensive secret management system that integrates Google Secret Manager, Kubernetes secrets, and application-level security patterns to provide secure, scalable secret handling.

Architecture

Secret Storage Layers

Google Secret Manager (Primary)
Central secrets repository
Automatic encryption at rest and in transit
Fine-grained IAM access controls
Version history and rotation support
Kubernetes Secrets (Runtime)
Secrets mounted directly into pods
Synchronized from Google Secret Manager
Namespace-isolated for security
Automatic updates via External Secrets Operator
Application Configuration (Runtime)
Environment variables for simple configuration
Lazy loading from Secret Manager API
Caching for performance optimization
Fallback to environment variables

Secret Types

Authentication Secrets

GitLab Private Token: Repository access and API calls
Firebase API Key: Authentication service integration
OAuth Client Secrets: Third-party authentication

Integration Secrets

GitLab Webhook Token: Webhook verification
Slack Webhook URL: Notification delivery
Redis Password: Cache authentication

Infrastructure Secrets

GCP Service Account Keys: Cloud resource access
Container Registry Credentials: Image pull access
Database Passwords: Data store authentication

Lazy Loading Pattern

The system implements lazy loading for optimal performance:

def get_gitlab_token(self) -> str | None:
    """Get GitLab token, loading from Secret Manager if needed."""
    if not self.gitlab_private_token and not self._secrets_loaded:
        self._load_secrets_from_secret_manager()
    return self.gitlab_private_token

Benefits

Faster application startup
Reduced API calls to Secret Manager
Graceful degradation when secrets unavailable
Improved development experience

Security Features

Workload Identity

No stored service account keys
Automatic credential rotation
Kubernetes service account binding
Fine-grained IAM permissions

Access Control

Least privilege access model
Namespace-based isolation
Role-based access control (RBAC)
Audit logging for all access

Lifecycle Management

Automatic secret rotation
Version history tracking
Graceful secret updates
Terraform-managed provisioning

Configuration

Secret Manager IDs

Secrets are referenced by ID in configuration:

# Secret Manager configuration
gitlab_token_secret_id: gitlab-token
redis_password_secret_id: redis-password
firebase_api_key_secret_id: firebase-api-key
gitlab_webhook_token_secret_id: gitlab-webhook-token
slack_webhook_url_secret_id: slack-webhook-url

Environment Variable Fallback

Direct environment variables provide fallback:

export GITLAB_PRIVATE_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx
export REDIS_PASSWORD=super-secret-password
export FIREBASE_API_KEY=AIzaSyxxxxxxxxxxxxxxxxxxxxxxxxx

External Secrets Operator

The platform uses External Secrets Operator for secret synchronization:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secret-store
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: gitlab-token
    remoteRef:
      key: gitlab-token

Best Practices

Secret Creation

Use Google Secret Manager for all sensitive data
Implement proper secret rotation policies
Use descriptive secret names with consistent prefixes
Document secret purpose and usage

Access Patterns

Use lazy loading for performance
Implement proper error handling
Cache secrets appropriately
Log access patterns for monitoring

Security

Never log secret values
Use Workload Identity instead of service account keys
Implement proper secret rotation
Monitor secret access patterns

Implementation Examples

Secret Manager Access

from blueberry.infrastructure.secret_manager import get_secret_manager_client

def _access_secret(secret_id: str) -> str | None:
    """Helper to fetch secret from Secret Manager."""
    try:
        client = get_secret_manager_client()
        name = f"projects/{self.project_id}/secrets/{secret_id}/versions/latest"
        response = client.access_secret_version(request={"name": name})
        return response.payload.data.decode("utf-8").strip()
    except Exception as exc:
        logging.debug("Failed to fetch secret '%s': %s", secret_id, exc)
        return None

Kubernetes Secret Mount

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    env:
    - name: GITLAB_PRIVATE_TOKEN
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: gitlab-token
    volumeMounts:
    - name: secrets-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets-volume
    secret:
      secretName: app-secrets

Troubleshooting

Secret Access Errors

Check Workload Identity binding between K8s SA and GCP SA
Verify IAM permissions for secret access
Ensure External Secrets Operator is running
Check secret names and versions

Configuration Loading Issues

Check pod logs for secret mount errors
Verify ConfigMap and Secret resources exist
Test Secret Manager API access manually
Review application configuration parsing

Performance Issues

Monitor secret loading times
Check for unnecessary secret reloads
Verify caching is working correctly
Review error handling patterns

blueberry/config.py - Secret loading implementation
blueberry/infrastructure/secret_manager.py - Secret Manager client
charts/blueberry/templates/secrets.yml - Kubernetes secret templates
argocd-apps/base/external-secrets/ - External Secrets configuration

Document ID: reference/configuration/secrets/README

Table of Contents