refactor: simplify configuration management system

- Remove Proposal 2 (simplified configuration approach)
- Remove TypeScript implementation assumptions
- Convert to language-agnostic design documentation
- Remove generic-to-concrete provider value mappings
- Use direct concrete values in provider contexts
- Update provider context structure to match file organization
- Focus on single advanced YAML-based configuration approach

The configuration system now uses concrete provider-specific values
directly instead of generic mappings, making it simpler and more
maintainable.

Author: josecelano

docs/redesign/phase3-design/configuration-management-implementation.md

@@ -0,0 +1,293 @@
+# Configuration Management System Implementation
+
+## Overview
+
+This document describes an advanced configuration management system implementation
+that can handle multi-environment, multi-provider_context deployments with proper defaults, validation,
+and secret management.
+
+## Advanced Configuration Management System
+
+### Design Concept
+
+A sophisticated configuration management system using YAML files with nested structures,
+JSON Schema validation, file inheritance, and template processing.
+
+### Architecture
+
+#### Configuration File Structure
+
+```text
+config/
+├── schemas/                # JSON Schema definitions
+│   ├── environment.schema.json
+│   ├── provider.schema.json
+│   └── composite.schema.json
+├── defaults/               # Base configuration templates
+│   ├── common.yaml         # Universal defaults
+│   ├── development.yaml    # Development-specific defaults
+│   ├── staging.yaml        # Staging-specific defaults
+│   └── production.yaml     # Production-specific defaults
+├── provider_contexts/      # Provider context definitions
+│   ├── libvirt.yaml        # Local development provider context
+│   ├── hetzner-staging.yaml # Hetzner Cloud staging provider context
+│   ├── hetzner-production.yaml # Hetzner Cloud production provider context
+│   └── aws.yaml            # AWS provider context
+└── environments/           # User environment configurations
+    ├── dev-alice.yaml      # Alice's personal dev environment
+    ├── staging-main.yaml   # Main staging environment
+    └── prod-primary.yaml   # Primary production environment
+```
+
+#### Example Configuration Format
+
+**Environment Configuration** (`environments/staging-main.yaml`):
+
+```yaml
+# Environment identification
+environment_type: staging
+provider_context: hetzner
+
+# General configuration
+general:
+  domains:
+    tracker: tracker.staging-torrust-demo.com
+    grafana: grafana.staging-torrust-demo.com
+  certbot_email: [email protected]
+
+# Application configuration
+application:
+  tracking:
+    enable_stats: true
+    log_level: info
+  database:
+    enable_backups: true
+    retention_days: 7
+
+# Secret references (resolved from environment variables)
+secrets:
+  mysql_root_password: ${MYSQL_ROOT_PASSWORD}
+  tracker_admin_token: ${TRACKER_ADMIN_TOKEN}
+  grafana_admin_password: ${GF_SECURITY_ADMIN_PASSWORD}
+```
+
+**Provider Context** (`provider_contexts/hetzner-staging.yaml`):
+
+```yaml
+# Provider identification
+provider_name: hetzner
+provider_type: cloud
+
+# Concrete provisioning values for this provider context
+provisioning:
+  server_type: cx31 # Hetzner-specific server type
+  location: fsn1 # Hetzner datacenter location
+  image: ubuntu-24.04 # Hetzner image name
+  networking:
+    floating_ip: true
+    ipv6: true
+    private_network: false
+
+# Provider-specific configuration
+ssl:
+  method: letsencrypt
+  email: "{{ general.certbot_email }}"
+
+# Hetzner API configuration
+api:
+  token_env_var: HCLOUD_TOKEN
+  dns_token_env_var: HDNS_TOKEN
+```
+
+**Provider Context** (`provider_contexts/libvirt.yaml`):
+
+```yaml
+# Provider identification
+provider_name: libvirt
+provider_type: local
+
+# Concrete provisioning values for this provider context
+provisioning:
+  memory: 2048 # Memory in MB
+  vcpus: 2 # Number of virtual CPUs
+  disk_size: 20 # Disk size in GB
+  base_image_url: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
+  networking:
+    network: default # libvirt network name
+    nat: true
+
+# Provider-specific configuration
+ssl:
+  method: self_signed # Use self-signed certificates for local testing
+
+# LibVirt configuration
+libvirt:
+  uri: "qemu:///system"
+  pool: "user-default"
+```
+
+### Implementation Components
+
+#### 1. Configuration Parser
+
+A component that loads and parses YAML configuration files, handling the nested structure
+and converting them into internal configuration objects.
+
+#### 2. Schema Validation System
+
+JSON Schema-based validation system that ensures all configuration files conform to
+expected structure and data types.
+
+#### 3. Template Resolution Engine
+
+A template processing system that resolves references between configurations and
+applies variable substitution with conditional logic.
+
+#### 4. File Merging with Priority System
+
+A configuration merging system that combines multiple configuration layers (base,
+defaults, environment, provider_context) according to priority rules and inheritance
+patterns.
+
+### Pros and Cons Analysis
+
+#### Advantages ✅
+
+1. **Powerful and Flexible**
+
+   - Supports complex nested configurations
+   - Rich template system with conditional logic
+   - Proper inheritance and composition patterns
+   - Provider context abstraction enables multi-cloud
+
+2. **Robust Validation**
+
+   - JSON Schema provides comprehensive validation
+   - Type safety and format validation
+   - Custom validation rules for business logic
+   - Clear error messages with schema violations
+
+3. **Professional Configuration Management**
+
+   - Follows enterprise configuration management patterns
+   - Separates concerns clearly (environment vs provider_context vs defaults)
+   - Enables configuration reuse and DRY principles
+   - Supports complex deployment scenarios
+
+4. **Extensible Architecture**
+
+   - Easy to add new provider contexts
+   - Template system supports custom logic
+   - Schema-driven validation allows evolution
+   - Plugin architecture for custom processors
+
+5. **Developer Experience**
+   - Rich IDE support with JSON Schema integration
+   - Auto-completion and validation in editors
+   - Clear separation of user vs system configuration
+   - Comprehensive error reporting
+
+#### Disadvantages ❌
+
+1. **Implementation Complexity**
+
+   - **Custom configuration system required** - No existing library handles this complexity
+   - **Multi-layer validation nightmare** - Common + conditional parts make validation extremely complex
+   - **Complex file merging** - Priority-based merging with inheritance requires custom implementation
+   - **Template engine development** - Need to build conditional template processing from scratch
+
+2. **Maintenance Burden**
+
+   - **High learning curve** - New contributors need to understand complex configuration system
+   - **Debugging complexity** - Multi-layer inheritance makes troubleshooting difficult
+   - **Schema evolution** - Changes require careful coordination across all layers
+   - **Custom tooling required** - Need to build validation, debugging, and migration tools
+
+3. **Development Time**
+
+   - **Months of custom development** - Building robust configuration management takes significant time
+   - **Testing complexity** - Need extensive test coverage for all configuration combinations
+   - **Documentation overhead** - Complex system requires comprehensive documentation
+   - **Tool ecosystem** - Need to build CLI tools, validators, and documentation generators
+
+4. **Technical Risks**
+
+   - **No existing libraries** - Building from scratch introduces bugs and edge cases
+   - **Secret injection complexity** - Secure credential handling in templates is non-trivial
+   - **Performance concerns** - Complex processing can be slow for large configurations
+   - **Vendor lock-in** - Custom system creates dependency on proprietary configuration format
+
+5. **Operational Complexity**
+   - **Hard to debug** - Nested YAML structures with inheritance are difficult to trace
+   - **Complex validation errors** - Multi-layer schemas produce confusing error messages
+   - **Tool dependency** - Requires custom tools for configuration management
+   - **Migration complexity** - Changes to configuration format require migration tools
+
+### Critical Implementation Challenges
+
+#### 1. File Merging with Priorities
+
+**Challenge**: Need to merge multiple YAML files with complex inheritance rules.
+**Reality**: No standard library exists that can handle conditional merging with provider context resolution.
+
+#### 2. Secret Injection from Environment Variables
+
+**Challenge**: Inject classical environment variables into nested YAML while keeping secrets out of files.
+**Reality**: Building secure template processing that handles secrets properly is extremely complex.
+
+#### 3. Multi-layer Validation
+
+**Challenge**: Validate configurations across multiple file layers with
+provider_context-specific rules.
+
+#### 3. Multi-layer Validation
+
+**Challenge**: Validate configurations that have common parts and conditional/extensible parts.
+**Reality**: JSON Schema with conditional validation becomes unwieldy and hard to maintain.
+
+#### 4. Provider Context Resolution
+
+**Challenge**: Map abstract configuration to provider_context-specific implementations.
+**Reality**: Building abstraction layers that work across different cloud provider contexts
+is a massive undertaking.
+
+### Conclusion
+
+While this approach offers powerful capabilities and follows enterprise patterns, the
+implementation complexity is prohibitive for a project of this scope. The lack of existing
+libraries to handle the specific combination of requirements (nested YAML merging,
+conditional validation, secret injection, provider abstraction) means building a custom
+configuration management system from scratch.
+
+**Recommendation**: This approach is too complex for the current project needs and would
+require significant development resources to implement properly.
+
+## Implementation Roadmap
+
+### Phase 1: Core Configuration System
+
+1. **Schema Definition**: Create base environment and provider context schemas
+2. **Configuration Parser**: Implement YAML loading and validation
+3. **Provider Context Resolution**: Build reference resolution system
+4. **Basic Templates**: Implement simple template resolution
+
+### Phase 2: Provider_context Integration
+
+1. **Hetzner Provider Context**: Implement Hetzner-specific mappings and capabilities
+2. **Libvirt Provider Context**: Implement local development provider context
+3. **Validation Integration**: Add composite schema validation
+4. **Error Handling**: Comprehensive error reporting and validation messages
+
+### Phase 3: Advanced Features
+
+1. **Template Engine**: Full template resolution with conditional logic
+2. **Multiple Provider Contexts**: Support for multiple accounts per provider
+3. **Configuration Inheritance**: Implement goal-based defaults and inheritance
+4. **CLI Integration**: Command-line tools for configuration management
+
+### Phase 4: Production Features
+
+1. **Credential Management**: Secure handling of provider context authentication
+2. **Configuration Validation**: Pre-deployment validation and dry-run capabilities
+3. **Migration Tools**: Tools for migrating between provider contexts
+4. **Documentation**: Complete configuration reference and examples