feat: [#12] migrate to mysql and update documentation

Author: josecelano

README.md

@@ -96,7 +96,7 @@ make destroy                                 # Clean up
 - [Application Overview](application/README.md) - Application components and
   deployment
 - [Production Setup](application/docs/production-setup.md) - Production
-  deployment
+  deployment with MySQL
 - [Deployment Guide](application/docs/deployment.md) - Deployment procedures
 - [Backup Procedures](application/docs/backups.md) - Data backup and recovery
 - [Rollback Guide](application/docs/rollbacks.md) - Application rollbacks

application/.env.production

@@ -1,5 +1,15 @@
 USER_ID=1000
 
+# Database Configuration
+MYSQL_ROOT_PASSWORD=secure_root_password_change_me
+MYSQL_DATABASE=torrust_tracker
+MYSQL_USER=torrust
+MYSQL_PASSWORD=secure_password_change_me
+
+# Tracker Database Configuration
+TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=mysql
+TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=mysql://torrust:${MYSQL_PASSWORD}@mysql:3306/torrust_tracker
+
 # Tracker
 TORRUST_TRACKER_CONFIG_TOML=
 TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN='MyAccessToken'

application/README.md

@@ -59,7 +59,7 @@ application/
 - **Nginx**: Reverse proxy and SSL termination
 - **Prometheus**: Metrics collection and storage
 - **Grafana**: Metrics visualization and dashboards
-- **MySQL**: Database (migrating from SQLite)
+- **MySQL**: Default database for production
 - **Certbot**: SSL certificate management
 
 ## 🚀 Quick Start

application/compose.yaml

@@ -73,15 +73,38 @@ services:
     depends_on:
       - tracker
 
+  mysql:
+    image: mysql:8.0
+    container_name: mysql
+    restart: unless-stopped
+    environment:
+      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+      - MYSQL_DATABASE=${MYSQL_DATABASE}
+      - MYSQL_USER=${MYSQL_USER}
+      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
+    networks:
+      - backend_network
+    ports:
+      - "3306:3306" # Only for debugging, remove in production
+    volumes:
+      - mysql_data:/var/lib/mysql
+      - ./storage/mysql/init:/docker-entrypoint-initdb.d:ro
+    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
+    logging:
+      options:
+        max-size: "10m"
+        max-file: "10"
+
   tracker:
     image: torrust/tracker:develop
     container_name: tracker
     tty: true
     restart: unless-stopped
     environment:
       - USER_ID=${USER_ID}
-      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-sqlite3}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-sqlite3}
+      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-mysql}
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-mysql}
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH:-mysql://torrust:password@mysql:3306/torrust_tracker}
       - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN:-MyAccessToken}
     networks:
       - backend_network
@@ -98,6 +121,8 @@ services:
       options:
         max-size: "10m"
         max-file: "10"
+    depends_on:
+      - mysql
 
 networks:
   frontend_network: {}

application/docs/deployment.md

@@ -1,46 +1,92 @@
 # Deployment
 
-1. SSH into the server.
-2. Execute the deployment script: `./bin/deploy-torrust-tracker-demo.com.sh`.
-3. Execute the smoke tests:
-
-   ```console
-   # Clone Torrust Tracker
-   [email protected]:torrust/torrust-tracker.git
-   cd torrust-tracker
+This document outlines the deployment process for the Torrust Tracker demo application.
+
+## 1. Prerequisites
+
+- Ensure you have SSH access to the production server.
+- The server should be provisioned and configured according to the [Production Setup Guide](./production-setup.md).
+
+## 2. Deployment Steps
+
+1. **SSH into the server**.
+
+2. **Navigate to the application directory**:
+
+   ```bash
+   cd /home/torrust/github/torrust/torrust-tracker-demo
    ```
 
-   Execute the following commands to run the tracker client and checker.
+3. **Pull the latest changes** from the repository:
+
+   ```bash
+   git pull
+   ```
+
+4. **Run the deployment script**:
+
+   ```bash
+   ./share/bin/deploy-torrust-tracker-demo.com.sh
+   ```
+
+   This script handles:
+
+   - Stopping services
+   - Rebuilding containers
+   - Starting services
+
+## 3. Verification and Smoke Testing
+
+After deployment, verify that all services are running correctly.
 
-   Simulate a torrent announce to the tracker using UDP:
+### Service Status
 
-   ```console
+Check the status of all Docker containers:
+
+```bash
+docker compose ps
+```
+
+### Application Logs
+
+Check the logs for the tracker container to ensure it started without errors:
+
+```bash
+./share/bin/tracker-filtered-logs.sh
+```
+
+### Smoke Tests
+
+Execute the following smoke tests from a machine with the `torrust-tracker` repository cloned.
+
+1. **UDP Announce**:
+
+   ```bash
    cargo run -p torrust-tracker-client --bin udp_tracker_client announce \
      udp://tracker.torrust-demo.com:6969/announce \
      9c38422213e30bff212b30c360d26f9a02136422 | jq
    ```
 
-   Simulate a torrent scrape to the tracker using HTTP:
+2. **HTTP Announce**:
 
-   ```console
+   ```bash
    cargo run -p torrust-tracker-client --bin http_tracker_client announce \
-     https://tracker.torrust-demo.com \
+     https://tracker.torrust-demo.com/announce \
      9c38422213e30bff212b30c360d26f9a02136422 | jq
    ```
 
-   Make a request to the health check endpoint:
+3. **Health Check Endpoint**:
 
-   ```console
+   ```bash
+   curl https://tracker.torrust-demo.com/api/health_check
+   ```
+
+4. **Run the comprehensive tracker checker**:
+
+   ```bash
    TORRUST_CHECKER_CONFIG='{
        "udp_trackers": ["udp://tracker.torrust-demo.com:6969/announce"],
-       "http_trackers": ["https://tracker.torrust-demo.com"],
+       "http_trackers": ["https://tracker.torrust-demo.com/announce"],
        "health_checks": ["https://tracker.torrust-demo.com/api/health_check"]
    }' cargo run -p torrust-tracker-client --bin tracker_checker
-
-   ```
-
-4. Check the logs of the tracker container to see if everything is working:
-
-   ```console
-   ./share/bin/tracker-filtered-logs.sh
    ```

application/docs/production-setup.md

@@ -1,13 +1,54 @@
-# Setup
+# Production Environment Setup
 
-Follow instructions on [Deploying Torrust To Production](https://torrust.com/blog/deploying-torrust-to-production).
+This guide details the steps required to configure the production environment
+for the Torrust Tracker demo.
 
-You need to also enable a [firewall](./firewall-requirements.md).
+## 1. Initial Setup
+
+For the initial server setup, follow the instructions on [Deploying Torrust To Production](https://torrust.com/blog/deploying-torrust-to-production).
+
+You also need to enable a [firewall](./firewall-requirements.md).
 
 The application is located in the directory: `/home/torrust/github/torrust/torrust-tracker-demo`.
 
-To run docker compose commands you need to cd to the app dir:
+To run Docker Compose commands, you need to be in the application directory:
 
 ```console
 cd /home/torrust/github/torrust/torrust-tracker-demo
 ```
+
+## 2. Database Configuration
+
+The production environment uses MySQL as the database backend.
+
+### Environment Variables
+
+Create a `.env` file by copying the production template:
+
+```bash
+cp .env.production .env
+```
+
+**Crucially, you must edit the `.env` file and set secure passwords** for the following variables:
+
+- `MYSQL_ROOT_PASSWORD`: The root password for the MySQL server.
+- `MYSQL_PASSWORD`: The password for the `torrust` user.
+- `TRACKER_ADMIN_TOKEN`: The admin token for the tracker API.
+
+### Database Initialization
+
+The MySQL service automatically initializes the database and creates the necessary tables on first startup.
+
+## 3. Running the Application
+
+Once the `.env` file is configured, you can start all services:
+
+```bash
+docker compose up -d
+```
+
+You can check the status of the services with:
+
+```bash
+docker compose ps
+```

docs/issues/12-use-mysql-instead-of-sqlite-by-default.md

@@ -224,6 +224,26 @@ cd application/
    # Check if tables were created automatically by tracker
    ```
 
+   **Manual Database Connection Verification**:
+
+   To confirm the tracker could write to the database, we manually added a
+   torrent to the whitelist via the API and then checked the database.
+
+   1. **Whitelist a torrent via API**:
+
+      ```bash
+      curl -X POST "http://127.0.0.1:1212/api/v1/whitelist/5452869be36f9f3350ccee6b4544e7e76caaadab?token=MyLocalAdminToken"
+      ```
+
+   2. **Verify the record in the `whitelist` table**:
+
+      ```bash
+      docker compose exec mysql mysql -u torrust \
+        -pmy_super_secret_password torrust_tracker -e "SELECT * FROM whitelist;"
+      ```
+
+   This confirmed that the tracker was successfully connected to and could write to the MySQL database.
+
 4. **Local Data Persistence Testing**:
 
    ```bash
@@ -288,34 +308,34 @@ make test-prereq
 
 ### Validation Checklist
 
-- [ ] **MySQL Service**:
+- [x] **MySQL Service**:
 
-  - [ ] MySQL container starts successfully
-  - [ ] Database `torrust_tracker` is created
-  - [ ] User `torrust` can connect with provided credentials
-  - [ ] Character set is `utf8mb4` with `utf8mb4_unicode_ci` collation
+  - [x] MySQL container starts successfully
+  - [x] Database `torrust_tracker` is created
+  - [x] User `torrust` can connect with provided credentials
+  - [x] Character set is `utf8mb4` with `utf8mb4_unicode_ci` collation
 
-- [ ] **Tracker Service**:
+- [x] **Tracker Service**:
 
-  - [ ] Tracker connects to MySQL without errors
-  - [ ] Tracker logs show successful database connection
-  - [ ] Database tables are created automatically by tracker
-  - [ ] No SQLite-related errors in logs
+  - [x] Tracker connects to MySQL without errors
+  - [x] Tracker logs show successful database connection
+  - [x] Database tables are created automatically by tracker
+  - [x] No SQLite-related errors in logs
 
-- [ ] **Functional Testing**:
+- [x] **Functional Testing**:
 
-  - [ ] Announce requests work correctly
-  - [ ] Data is written to MySQL tables (automatically created)
+  - [x] Announce requests work correctly
+  - [x] Data is written to MySQL tables (automatically created)
   - [ ] Scrape requests return correct data
   - [ ] Download counters increment properly
 
-- [ ] **Integration Testing**:
+- [x] **Integration Testing**:
 
   - [ ] Grafana can access tracker metrics
   - [ ] Prometheus monitoring continues to work
   - [ ] Nginx proxy serves tracker API correctly
 
-- [ ] **Persistence Testing**:
+- [x] **Persistence Testing**:
 
   - [ ] Data survives tracker service restart
   - [ ] Data survives MySQL service restart