Overview

Blueberry IDP automates the entire lifecycle of ephemeral environments using GitOps principles. This guide covers all core workflows from initial setup to advanced troubleshooting scenarios.

1. Environment Creation Workflow

Complete Creation Process

Step 1: Trigger & Authentication

Environment creation can be triggered through multiple entry points:

• Web UI: POST /api/environments/ (HTML form)
• API: POST /api/v2/environments (JSON)
• GitLab CI: Pipeline scripts using API tokens

Authentication: Firebase ID tokens or API tokens with environments:create scope

Step 2: Validation & Naming

Name Generation:

• Manual environments: {name}-{repo}-{commit}
• PR environments: pr-{number}
• Duplicate checking in Firestore

Configuration Validation:

• Parse and validate JSON config overrides
• Check repository existence and access
• Validate branch and commit references

Step 3: Environment Record Creation

Creates metadata record in Firestore:

{
  "id": "env-abc123",
  "name": "pr-456",
  "status": "PENDING",
  "created_by": "user@example.com",
  "kubernetes": {
    "namespace": "pr-456",
    "argocd_app": "env-pr-456"
  },
  "git": {
    "repo_name": "sample-be-1",
    "branch": "feature/new-api",
    "commit_sha": "abc12345"
  },
  "ttl_hours": 72
}

Step 4: ArgoCD Application Creation

Creates Kubernetes Custom Resource for GitOps deployment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: env-pr-456
  namespace: argocd
  labels:
    blueberry.io/environment: pr-456
spec:
  project: blueberry
  source:
    repoURL: https://gitlab.com/obtain-blew/blueberry
    path: charts/environment
    helm:
      values: |
        environment:
          name: pr-456
          namespace: pr-456
        backend:
          image:
            tag: abc12345
  destination:
    server: https://kubernetes.default.svc
    namespace: pr-456
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Step 5: GitOps Sync & Deployment

ArgoCD Process:

• Detects new Application resource
• Clones Git repository
• Renders Helm templates
• Applies resources to target namespace

Deployment Order (Sync Waves):

• Wave 0: Namespace + Redis
• Wave 1: Backend services
• Wave 2: Frontend services
• Wave 3: Ingress resources

Expected Timeline

< 2s

API Response

30-60s

ArgoCD Sync

1-3m

Pod Startup

1-2m

DNS Ready

2. Environment Deletion Workflow

Complete Deletion Process

1

Delete Request
DELETE /api/environments/{id}

→

2

Remove ArgoCD
Application

→

3

Cascading Deletion
of K8s Resources

→

4

Archive Metadata
& Clear Cache

Cleanup Details

• Kubernetes Resources: All pods, services, ingresses, and configmaps removed
• Namespace: Entire namespace deleted (if no other resources)
• DNS: Ingress rules removed, DNS records cleaned up
• Firestore: Environment record marked as TERMINATED
• Cache: Redis cache entries evicted

3. CI/CD Integration Workflows

GitLab CI/CD Pipeline

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy-ephemeral
  - cleanup

variables:
  BLUEBERRY_API_URL: "https://blueberry.florenciacomuzzi.com"

deploy_ephemeral:
  stage: deploy-ephemeral
  image: curlimages/curl:latest
  script:
    - |
      RESPONSE=$(curl -s -X POST "$BLUEBERRY_API_URL/api/v2/environments" \
        -H "Authorization: Bearer $BLUEBERRY_API_TOKEN" \
        -H "Content-Type: application/json" \
        -d '{
          "name": "mr-'$CI_MERGE_REQUEST_IID'",
          "repository_id": "'$REPOSITORY_ID'",
          "branch": "'$CI_COMMIT_REF_NAME'",
          "ttl_hours": 72
        }')
      echo "$RESPONSE" | jq '.urls[]'
  only:
    - merge_requests

cleanup_ephemeral:
  stage: cleanup
  image: curlimages/curl:latest
  script:
    - |
      curl -X DELETE "$BLUEBERRY_API_URL/api/v2/environments/mr-$CI_MERGE_REQUEST_IID" \
        -H "Authorization: Bearer $BLUEBERRY_API_TOKEN"
  when: manual
  only:
    - merge_requests

GitHub Actions Workflow

# .github/workflows/ephemeral-env.yml
name: Deploy Ephemeral Environment

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy Environment
        run: |
          RESPONSE=$(curl -s -X POST "${{ secrets.BLUEBERRY_API_URL }}/api/v2/environments" \
            -H "Authorization: Bearer ${{ secrets.BLUEBERRY_API_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d '{
              "name": "pr-${{ github.event.number }}",
              "repository_id": "${{ secrets.REPOSITORY_ID }}",
              "branch": "${{ github.head_ref }}",
              "ttl_hours": 72
            }')
          echo "Environment URLs:"
          echo "$RESPONSE" | jq -r '.urls[]'

      - name: Comment PR
        uses: actions/github-script@v6
        with:
          script: |
            const urls = JSON.parse(process.env.RESPONSE).urls;
            const comment = `🚀 **Ephemeral Environment Deployed**\n\n${urls.map(url => `- [${url}](${url})`).join('\n')}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });

4. Troubleshooting Workflows

Environment Stuck in PROVISIONING

Diagnosis Steps:

1. Check ArgoCD Application sync status in UI
2. Verify container images exist in registry
3. Check Kubernetes events for pod scheduling issues
4. Review environment logs for specific error messages

Common Fixes:

# Check ArgoCD app status
kubectl get application env-pr-456 -n argocd -o yaml

# Check pod events
kubectl describe pods -n pr-456

# Check image availability
docker pull us-docker.pkg.dev/project/blueberry/app:commit-sha

Images Not Found

Verification Checklist:

• Verify GitLab CI pipeline completed successfully
• Check Artifact Registry for pushed images
• Ensure commit SHA matches the built image tag
• Check registry access permissions

Manual Build Check:

# Check if image exists
gcloud artifacts docker images list \
  --repository=blueberry \
  --location=us \
  --filter="package=app AND version=commit-sha"

ArgoCD Sync Failures

Investigation Steps:

• Check ArgoCD UI for specific error messages
• Verify Git repository access and credentials
• Review Helm template syntax and values
• Check RBAC permissions for target namespace

Manual Sync:

# Force manual sync
kubectl patch application env-pr-456 -n argocd \
  --type merge -p '{"operation":{"sync":{"force":true}}}'

5. Best Practices

✅ Recommended Practices

• Use descriptive environment names that include PR/feature context
• Set appropriate TTL values (24-72 hours for most use cases)
• Clean up environments when no longer needed
• Use API tokens with minimal required scopes
• Monitor resource usage and costs
• Test environment creation in staging before production

❌ Common Pitfalls

• Creating environments with very long TTL values
• Using production secrets in ephemeral environments
• Not cleaning up failed environments
• Hardcoding environment names in CI/CD pipelines
• Ignoring resource limit warnings
• Creating environments without proper authentication

Workflows & Processes