docs: [#28] document configuration architecture and override system

josecelano · josecelano · commit cc0c4d6c0a08 · 2025-08-04T12:01:00.000+01:00
- Add comprehensive configuration-architecture.md documentation
- Explain two-layer hierarchy: environment configs override provider defaults
- Document loading order: environment first, then provider
- Clarify why variables appear in both environment and provider configs
- Add practical examples of override scenarios
- Update provider README.md with hierarchy explanation
- Add inline comments to hetzner.env explaining loading order
- Resolves confusion about apparent variable duplication
diff --git a/infrastructure/config/providers/README.md b/infrastructure/config/providers/README.md
@@ -33,22 +33,60 @@ infrastructure/config/templates/providers/
 
 ## Usage Instructions
 
+### Configuration Hierarchy and Override System
+
+The project uses a **hierarchical configuration system** where environment configurations
+can override provider defaults:
+
+#### Loading Order
+
+1. **Environment config** loaded first: `infrastructure/config/environments/{environment}-{provider}.env`
+2. **Provider config** loaded second: `infrastructure/config/providers/{provider}.env`
+3. **Result**: Provider config can override environment values, but environments can override
+   provider defaults
+
+#### Override Strategy
+
+- **Provider configs**: Set sensible defaults that work for most environments
+- **Environment configs**: Override only when necessary (performance, geography, cost)
+- **Commented variables**: Environment files contain commented provider variables for easy overriding
+
+#### Example Scenario
+
+```bash
+# Provider config (hetzner.env) - defaults for ALL environments
+HETZNER_SERVER_TYPE=cpx31      # Default: 4 vCPU, 8GB RAM
+HETZNER_LOCATION=fsn1          # Default: Falkenstein datacenter
+
+# Environment config (production-hetzner.env) - environment-specific overrides
+# HETZNER_SERVER_TYPE=cx41     # Uncomment: Use higher performance for production
+# HETZNER_LOCATION=nbg1        # Uncomment: Use Nuremberg for production
+```
+
+### Setup Instructions
+
 1. **Copy the appropriate template:**
 
    ```bash
    # For local testing
    cp infrastructure/config/templates/providers/libvirt.env.tpl infrastructure/config/providers/libvirt.env
-   
+
    # For Hetzner Cloud
    cp infrastructure/config/templates/providers/hetzner.env.tpl infrastructure/config/providers/hetzner.env
    ```
 
 2. **Edit the copied file** with your actual values:
+
    - Replace placeholder tokens with real API keys
    - Configure VM sizes and locations
    - Set appropriate defaults for your use case
 
-3. **Never commit these files** - they contain secrets and are automatically git-ignored
+3. **Override in environment configs when needed**:
+
+   - Uncomment variables in environment files only when you need environment-specific settings
+   - Document why you're overriding the provider default
+
+4. **Never commit these files** - they contain secrets and are automatically git-ignored
 
 ## Security Notes
 
diff --git a/infrastructure/config/providers/hetzner.env b/infrastructure/config/providers/hetzner.env
@@ -1,5 +1,19 @@
-# Hetzner Cloud Provider Configuration Template
-# Copy this file to hetzner.env and replace placeholder values
+# Hetzner Cloud Provider Configuration
+# 
+# CONFIGURATION HIERARCHY: This file provides provider-wide defaults that apply 
+# to all environments using the Hetzner provider. These values can be overridden 
+# by environment-specific configurations when needed.
+#
+# Loading order during deployment:
+# 1. Environment config is loaded first (e.g., production-hetzner.env)
+# 2. Provider config is loaded second (this file) 
+# 3. Provider values can override environment values, but environments can override provider defaults
+#
+# BEST PRACTICES:
+# - Set sensible defaults here that work for most environments
+# - Use environment configs to override only when necessary (performance, geography, etc.)
+# - Keep authentication tokens and API keys only in this file
+#
 # Location: infrastructure/config/providers/hetzner.env
 
 # === HETZNER CLOUD AUTHENTICATION ===
diff --git a/infrastructure/docs/configuration-architecture.md b/infrastructure/docs/configuration-architecture.md
@@ -0,0 +1,308 @@
+# Configuration Architecture Guide
+
+This guide explains the hierarchical configuration system used by the Torrust Tracker Demo
+for managing environment and provider configurations.
+
+## Overview
+
+The project uses a **two-level configuration hierarchy** that allows flexible management
+of deployment settings across different environments (development, staging, production)
+and providers (libvirt, Hetzner Cloud).
+
+## Configuration Layers
+
+### Layer 1: Environment Configuration
+
+**Location**: `infrastructure/config/environments/`  
+**Purpose**: Environment-specific settings (VM specs, secrets, SSL, backups)  
+**Examples**: `development-libvirt.env`, `production-hetzner.env`
+
+### Layer 2: Provider Configuration
+
+**Location**: `infrastructure/config/providers/`  
+**Purpose**: Provider-wide defaults and authentication  
+**Examples**: `libvirt.env`, `hetzner.env`
+
+## Loading Order and Override System
+
+During deployment, configurations are loaded in this specific order:
+
+```bash
+# 1. Environment configuration loaded first
+source "infrastructure/config/environments/production-hetzner.env"
+
+# 2. Provider configuration loaded second (can override environment values)
+source "infrastructure/config/providers/hetzner.env"
+```
+
+### Why This Order?
+
+This loading order provides **environment-specific override capability**:
+
+- **Provider configs** set sensible defaults for all environments using that provider
+- **Environment configs** can override provider defaults when specific needs arise
+- **Later loaded values win**, so provider configs have final say on their core settings
+
+## Practical Configuration Examples
+
+### Example 1: Default Behavior (No Overrides)
+
+**Provider config** (`hetzner.env`):
+
+```bash
+HETZNER_SERVER_TYPE=cpx31      # 4 vCPU, 8GB RAM - good default
+HETZNER_LOCATION=fsn1          # Falkenstein datacenter
+HETZNER_TOKEN=actual_api_token # Authentication
+```
+
+**Environment config** (`production-hetzner.env`):
+
+```bash
+VM_MEMORY=8192                 # Application-specific setting
+DOMAIN_NAME=tracker.example.com # Environment-specific domain
+# HETZNER_SERVER_TYPE not set   # Uses provider default (cpx31)
+```
+
+**Result**: Production uses `cpx31` server in `fsn1` location.
+
+### Example 2: Environment-Specific Override
+
+**Provider config** (`hetzner.env`):
+
+```bash
+HETZNER_SERVER_TYPE=cpx31      # Default for most environments
+HETZNER_LOCATION=fsn1          # Default datacenter
+```
+
+**Environment config** (`production-hetzner.env`):
+
+```bash
+VM_MEMORY=16384                # Higher memory requirement
+HETZNER_SERVER_TYPE=cx41       # Override: Higher performance for production
+# HETZNER_LOCATION not set     # Uses provider default (fsn1)
+```
+
+**Result**: Production uses `cx41` server (higher performance) in `fsn1` location.
+
+### Example 3: Geographic Distribution
+
+**Provider config** (`hetzner.env`):
+
+```bash
+HETZNER_SERVER_TYPE=cpx31      # Standard performance
+HETZNER_LOCATION=fsn1          # EU default
+```
+
+**Environment configs**:
+
+```bash
+# staging-hetzner.env (EU staging)
+# HETZNER_LOCATION not set     # Uses provider default (fsn1)
+
+# production-us-hetzner.env (US production)
+HETZNER_LOCATION=ash           # Override: US East Coast
+```
+
+**Result**: Staging in EU, production in US, both using same server type.
+
+## Configuration Variable Categories
+
+### Environment-Only Variables
+
+These appear only in environment configurations:
+
+- `VM_MEMORY`, `VM_VCPUS`, `VM_DISK_SIZE` - VM specifications
+- `MYSQL_ROOT_PASSWORD`, `MYSQL_PASSWORD` - Application secrets
+- `DOMAIN_NAME`, `CERTBOT_EMAIL` - SSL configuration
+- `ENABLE_SSL`, `ENABLE_DB_BACKUPS` - Feature flags
+
+### Provider-Only Variables
+
+These appear only in provider configurations:
+
+- `HETZNER_TOKEN`, `PROVIDER_LIBVIRT_URI` - Authentication
+- Server type and location defaults
+- Provider-specific settings and references
+
+### Shared Variables (Override Candidates)
+
+These can appear in both layers, with environment overriding provider:
+
+- `HETZNER_SERVER_TYPE` - Server performance level
+- `HETZNER_LOCATION` - Datacenter location
+- `HETZNER_IMAGE` - Operating system image
+- Provider-specific performance or regional settings
+
+## Best Practices
+
+### For Provider Configurations
+
+1. **Set sensible defaults** that work for most environments
+2. **Include comprehensive documentation** about available options
+3. **Keep authentication tokens secure** and never commit to git
+4. **Test defaults** across different environment types
+
+### For Environment Configurations
+
+1. **Override sparingly** - only when truly needed for that environment
+2. **Document the reason** for any provider overrides
+3. **Use comments** to show available override options
+4. **Keep secrets separate** from non-sensitive configuration
+
+### When to Override
+
+Override provider defaults in environment configs when:
+
+- **Performance requirements differ** (production needs more power)
+- **Geographic requirements** (regulatory or latency concerns)
+- **Cost optimization** (development can use smaller instances)
+- **Testing specific features** (particular server types or locations)
+
+### When NOT to Override
+
+Don't override unless necessary:
+
+- **API tokens and authentication** should stay in provider configs
+- **Standard configurations** work fine with provider defaults
+- **Adds unnecessary complexity** without clear benefit
+
+## Configuration Templates
+
+The project includes templates for both layers:
+
+### Template Structure
+
+```text
+infrastructure/config/
+├── templates/                 # Version-controlled templates
+│   ├── environments/         # Environment templates
+│   │   ├── base.env.tpl
+│   │   ├── production.env.tpl
+│   │   └── staging.env.tpl
+│   └── providers/            # Provider templates
+│       ├── hetzner.env.tpl
+│       └── libvirt.env.tpl
+├── environments/             # User-generated (git-ignored)
+│   ├── development-libvirt.env
+│   └── production-hetzner.env
+└── providers/                # User-generated (git-ignored)
+    ├── hetzner.env
+    └── libvirt.env
+```
+
+### Template Usage
+
+1. **Copy templates** to create user configurations
+2. **Customize values** for your specific deployment
+3. **Never modify templates directly** - they're shared across all users
+4. **User configs are git-ignored** to protect secrets
+
+## Troubleshooting Configuration Issues
+
+### Common Problems
+
+1. **Variable not taking effect**
+
+   - Check loading order: provider config loads after environment
+   - Verify variable name spelling and format
+   - Check if variable is commented out
+
+2. **Override not working**
+
+   - Ensure variable is uncommented in environment config
+   - Check for typos in variable names
+   - Verify the variable is loaded by the provisioning script
+
+3. **Missing configuration**
+
+   - Check that both environment and provider configs exist
+   - Verify file naming convention: `{environment}-{provider}.env`
+   - Ensure files are in correct directories
+
+### Debugging Configuration Loading
+
+Enable debug logging to see configuration loading:
+
+```bash
+# Add to provisioning script for debugging
+set -x  # Enable debug output
+echo "Loading environment: ${env_file}"
+echo "Loading provider: ${provider_config}"
+set +x  # Disable debug output
+```
+
+## Security Considerations
+
+### Sensitive Data Handling
+
+- **Provider configs**: Contain API tokens and credentials
+- **Environment configs**: Contain application secrets and passwords
+- **Both layers**: Are git-ignored to prevent accidental commits
+- **Templates**: Contain no sensitive data, safe to version control
+
+### Access Control
+
+- Limit access to configuration directories
+- Use proper file permissions (600) for config files
+- Rotate API tokens and passwords regularly
+- Use encrypted storage for backup configurations
+
+This architecture provides flexibility while maintaining security and simplicity for
+managing complex multi-environment, multi-provider deployments.### Template Usage
+
+1. **Copy templates** to create user configurations
+2. **Customize values** for your specific deployment
+3. **Never modify templates directly** - they're shared across all users
+4. **User configs are git-ignored** to protect secrets
+
+## Troubleshooting Configuration Issues
+
+### Common Problems
+
+1. **Variable not taking effect**
+
+   - Check loading order: provider config loads after environment
+   - Verify variable name spelling and format
+   - Check if variable is commented out
+
+2. **Override not working**
+
+   - Ensure variable is uncommented in environment config
+   - Check for typos in variable names
+   - Verify the variable is loaded by the provisioning script
+
+3. **Missing configuration**
+   - Check that both environment and provider configs exist
+   - Verify file naming convention: `{environment}-{provider}.env`
+   - Ensure files are in correct directories
+
+### Debugging Configuration Loading
+
+Enable debug logging to see configuration loading:
+
+```bash
+# Add to provisioning script for debugging
+set -x  # Enable debug output
+echo "Loading environment: ${env_file}"
+echo "Loading provider: ${provider_config}"
+set +x  # Disable debug output
+```
+
+## Security Considerations
+
+### Sensitive Data Handling
+
+- **Provider configs**: Contain API tokens and credentials
+- **Environment configs**: Contain application secrets and passwords
+- **Both layers**: Are git-ignored to prevent accidental commits
+- **Templates**: Contain no sensitive data, safe to version control
+
+### Access Control
+
+- Limit access to configuration directories
+- Use proper file permissions (600) for config files
+- Rotate API tokens and passwords regularly
+- Use encrypted storage for backup configurations
+
+This architecture provides flexibility while maintaining security and simplicity for
+managing complex multi-environment, multi-provider deployments.
