Skip to content

Development

Jump to bottom
sysid edited this page Oct 12, 2025 · 2 revisions

Development

This guide covers development setup, testing, contributing, and the technical architecture of bkmr for developers who want to contribute or understand the codebase.

Overview

bkmr is built with:

  • Language: Rust 2021 edition
  • Database: SQLite with Diesel ORM
  • Search: FTS5 full-text search
  • CLI Framework: Clap v4
  • LSP: tower-lsp with async tokio runtime
  • Testing: Cargo test with shared database strategy

Getting Started

Prerequisites

Required:

  • Rust 1.70+ (latest stable recommended)
  • SQLite 3.35+
  • Git

Optional (for full development):

  • fzf (for fuzzy finder testing)
  • Python 3.8+ (for LSP test scripts)
  • jq (for JSON processing in examples)

Clone and Build

# Clone repository
git clone https://github.com/sysid/bkmr.git
cd bkmr

# Build debug version (includes all functionality)
cargo build

# Build release version (optimized)
cargo build --release

# Binary location
ls -la target/debug/bkmr        # Debug
ls -la target/release/bkmr      # Release

Development Setup

1. Create test database:

# Create at standard test location
./target/debug/bkmr create-db ../db/bkmr.db

# Set environment variable
export BKMR_DB_URL=../db/bkmr.db

2. Verify build:

# Check version
./target/debug/bkmr --version

# Test basic functionality
./target/debug/bkmr add "https://example.com" test
./target/debug/bkmr search "test"

3. Configure editor:

# For code navigation and development
# Rust Analyzer recommended for VS Code/Neovim

Project Structure

bkmr/
├── bkmr/                    # Main Rust project
│   ├── src/
│   │   ├── main.rs          # Entry point, CLI routing
│   │   ├── cli/             # CLI commands and handlers
│   │   ├── application/     # Use cases and business logic
│   │   ├── domain/          # Core entities and traits
│   │   ├── infrastructure/  # External systems (DB, HTTP, embeddings)
│   │   └── lsp/             # LSP server implementation
│   ├── migrations/          # Diesel database migrations
│   ├── tests/               # Integration tests
│   └── Cargo.toml           # Dependencies and configuration
├── docs/                    # Documentation (to be deprecated)
├── bkmr.wiki/               # GitHub wiki (main documentation)
├── scripts/                 # Utility scripts
│   └── lsp/                 # LSP testing scripts
├── Makefile                 # Development commands
└── README.md                # Project overview

Architecture Overview

bkmr follows Clean Architecture principles:

Layer Structure:

  1. Domain (src/domain/) - Core business entities and traits

    • No external dependencies
    • Pure business logic
    • Entity definitions (Bookmark, Tag, etc.)

  2. Application (src/application/) - Use cases and orchestration

    • Service traits and implementations
    • Depends only on Domain interfaces
    • Business workflow coordination

  3. Infrastructure (src/infrastructure/) - External systems

    • SQLite repository implementations
    • OpenAI embeddings integration
    • HTTP client for metadata fetching
    • Clipboard operations

  4. CLI (src/cli/) - User interface

    • Command parsing (Clap)
    • Command handlers
    • Output formatting

  5. LSP (src/lsp/) - Editor integration

    • tower-lsp protocol implementation
    • Async/sync bridging
    • Language-aware filtering

Dependency Injection:

  • ServiceContainer pattern for dependency wiring
  • All services accept dependencies via constructors
  • No global state (eliminated in v4.31+)
  • Arc for shared service ownership

Running Tests

Critical Testing Requirements

⚠️ MANDATORY: All tests MUST run single-threaded

# Primary test command (recommended)
make test

# Manual execution
cargo test -- --test-threads=1

# With output visible
cargo test -- --test-threads=1 --nocapture

# Specific test
cargo test --lib test_name -- --test-threads=1 --nocapture

Why Single-Threaded?

Critical reasons:

  1. Database contention - Tests share SQLite database file (../db/bkmr.db)
  2. Lock prevention - Eliminates SQLite lock conflicts during parallel access
  3. Environment isolation - Prevents race conditions in environment variable manipulation
  4. Reliable execution - Ensures consistent, deterministic test results

Without --test-threads=1:

  • ❌ Random SQLite lock errors
  • ❌ Test flakiness
  • ❌ Intermittent CI failures
  • ❌ Race conditions in environment setup

Test Categories

Unit Tests (cargo test --lib):

# Run only library tests
env RUST_LOG=error BKMR_DB_URL=../db/bkmr.db cargo test --lib --manifest-path bkmr/Cargo.toml -- --test-threads=1

Integration Tests (tests/ directory):

# Run integration tests
cargo test --test '*' -- --test-threads=1

Documentation Tests (embedded in code):

# Included in cargo test
# Tests example code in doc comments

Test Infrastructure

TestServiceContainer (src/util/test_service_container.rs):

// Modern dependency injection for tests
let test_container = TestServiceContainer::new();
let service = test_container.bookmark_service.clone();

// Use service with explicit dependencies
let bookmarks = service.get_all_bookmarks(None, None)?;

Test Conventions:

  • Naming: given_X_when_Y_then_Z() pattern
  • Structure: Arrange/Act/Assert
  • Environment: Use EnvGuard for variable isolation
  • Services: Obtain via TestServiceContainer

Running Test Suites

Quick tests (library only):

make test-lib
# Or
env RUST_LOG=error BKMR_DB_URL=../db/bkmr.db cargo test --lib --manifest-path bkmr/Cargo.toml -- --test-threads=1 --quiet

Full test suite:

make test

With debug output:

RUST_LOG=debug cargo test -- --test-threads=1 --nocapture 2>&1 | tee test-output.log

Development Commands

Makefile Targets

# Build
make build              # Build release version
make                    # Same as 'make build'

# Testing
make test               # Run all tests (single-threaded)
make test-lib           # Run only library tests

# Code Quality
make format             # Format code with cargo fmt
make lint               # Run clippy linter with auto-fix

# Maintenance
make clean              # Clean build artifacts
make install            # Install to ~/bin/ with version suffix

# Database
make create-test-db     # Create test database

Manual Commands

Format code:

cargo fmt
cargo fmt --check  # Check without modifying

Linting:

cargo clippy --all-targets --all-features
cargo clippy --fix  # Auto-fix warnings

Build variations:

# Debug build (fast compile, unoptimized)
cargo build

# Release build (optimized)
cargo build --release

# With specific features (no longer needed, all features always enabled)
cargo build --release

Run from source:

# Debug binary
./target/debug/bkmr search "test"

# Release binary
./target/release/bkmr search "test"

# Or use cargo run
cargo run -- search "test"
cargo run --release -- search "test"

Database Management

Schema Changes

Create new migration:

# Install diesel CLI
cargo install diesel_cli --no-default-features --features sqlite

# Create migration
diesel migration generate add_new_column

# Edit generated files in migrations/
# - up.sql: Schema changes
# - down.sql: Rollback changes

# Apply migration
diesel migration run

# Rollback if needed
diesel migration revert

Migration Best Practices:

  1. Always create backup before schema changes
  2. Test migrations on copy of production database
  3. Write both up.sql and down.sql
  4. Document complex migrations in comments

Test Database

Reset test database:

# Remove existing
rm -f ../db/bkmr.db

# Create fresh
./target/debug/bkmr create-db ../db/bkmr.db

# Run migrations (automatic on first access)
./target/debug/bkmr search ""

Debugging

Debug Logging

Enable debug output:

# Application debug logs
RUST_LOG=debug cargo run -- search "test"

# Trace level (very verbose)
RUST_LOG=trace cargo run -- search "test"

# Specific modules
RUST_LOG=bkmr::application=debug cargo run -- search "test"
RUST_LOG=bkmr::lsp=debug cargo run -- lsp

# Multiple modules
RUST_LOG=bkmr::application=debug,bkmr::infrastructure=debug cargo run -- search "test"

Debug flags:

# Single debug flag
cargo run -- -d search "test"

# Double debug flag (more verbose)
cargo run -- -d -d search "test"

# Save logs to file
RUST_LOG=debug cargo run -- search "test" 2>/tmp/bkmr-debug.log

Testing LSP

Test LSP server:

# Start LSP in debug mode
RUST_LOG=debug ./target/debug/bkmr lsp 2>/tmp/bkmr-lsp.log

# In another terminal, watch logs
tail -f /tmp/bkmr-lsp.log

# Use Python test scripts
python3 scripts/lsp/list_snippets.py --debug
python3 scripts/lsp/get_snippet.py 123
python3 scripts/lsp/test_lsp_client.py

Test basic LSP connectivity:

# Echo initialize request
echo '{"jsonrpc":"2.0","method":"initialize","id":1,"params":{}}' | ./target/debug/bkmr lsp

# Should return JSON with capabilities

Common Debug Scenarios

Search not working:

# Debug search execution
RUST_LOG=bkmr::application::services::bookmark_service=debug cargo run -- search "test" 2>&1 | grep -i search

Template not interpolating:

# Debug template rendering
RUST_LOG=bkmr::application::services::template_service=debug cargo run -- open <id> 2>&1 | grep -i template

Database issues:

# Debug database operations
RUST_LOG=bkmr::infrastructure::repository=debug cargo run -- search "test" 2>&1 | grep -i database

Contributing

Before Starting

  1. Check existing issues: Look for similar feature requests or bug reports
  2. Open discussion: For significant changes, open an issue first
  3. Read architecture: Understand Clean Architecture principles
  4. Run tests: Ensure existing tests pass

Development Workflow

1. Create feature branch:

git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix

2. Make changes:

  • Write code following Rust conventions
  • Add tests for new functionality
  • Update documentation
  • Run tests frequently

3. Test thoroughly:

# Format code
make format

# Run linter
make lint

# Run all tests (CRITICAL)
make test

# All tests must pass before committing

4. Commit changes:

# Write clear commit messages
git commit -m "Add: New feature description"
git commit -m "Fix: Bug description"
git commit -m "Docs: Documentation update"

5. Push and create PR:

git push origin feature/your-feature-name

# Create pull request on GitHub
# Fill in PR template with:
# - Description of changes
# - Related issues
# - Testing performed

Code Style Guidelines

Rust Conventions:

  • Follow rustfmt formatting (run make format)
  • Use snake_case for functions and variables
  • Use PascalCase for types and traits
  • Add documentation comments for public APIs
  • Use thiserror for error types
  • Prefer Result<T> over panics

Error Handling:

// Good: Propagate errors with context
let bookmark = repository.get_bookmark(id)
    .context("Failed to retrieve bookmark")?;

// Good: Descriptive error messages
return Err(DomainError::NotFound(format!("Bookmark {} not found", id)));

// Avoid: Generic errors
return Err(DomainError::Unknown);

Dependency Injection:

// Good: Explicit dependencies via constructor
pub struct BookmarkServiceImpl {
    repository: Arc<dyn BookmarkRepository>,
    embedder: Arc<dyn Embedder>,
}

impl BookmarkServiceImpl {
    pub fn new(
        repository: Arc<dyn BookmarkRepository>,
        embedder: Arc<dyn Embedder>,
    ) -> Arc<dyn BookmarkService> {
        Arc::new(Self { repository, embedder })
    }
}

// Avoid: Global state or factory methods
// These patterns have been completely eliminated

Testing:

// Good: Clear test names and structure
#[test]
fn given_valid_url_when_adding_bookmark_then_succeeds() {
    // Arrange
    let test_container = TestServiceContainer::new();
    let service = test_container.bookmark_service.clone();

    // Act
    let result = service.add_bookmark(
        "https://example.com",
        &["test"],
        Some("Test Bookmark"),
        None,
        false,
    );

    // Assert
    assert!(result.is_ok());
}

Documentation

Code Documentation:

/// Retrieves a bookmark by its ID.
///
/// # Arguments
/// * `id` - The unique identifier of the bookmark
///
/// # Returns
/// Returns the bookmark if found, or an error if not found or inaccessible.
///
/// # Errors
/// * `DomainError::NotFound` - Bookmark with given ID doesn't exist
/// * `DomainError::DatabaseError` - Database access failed
pub fn get_bookmark(&self, id: i32) -> DomainResult<Bookmark> {
    // Implementation
}

Wiki Documentation:

  • Update wiki pages for user-facing changes
  • Add examples for new features
  • Update troubleshooting for new issues
  • Keep documentation in sync with code

Pull Request Guidelines

PR Description Should Include:

  • Clear description of changes
  • Motivation and context
  • Related issues (closes #123)
  • Breaking changes (if any)
  • Testing performed

Before Submitting:

  • ✅ All tests pass (make test)
  • ✅ Code formatted (make format)
  • ✅ No clippy warnings (make lint)
  • ✅ Documentation updated
  • ✅ Commit messages are clear

Review Process:

  1. Automated CI checks must pass
  2. Code review by maintainers
  3. Address feedback
  4. Final approval and merge

Release Process

Version Numbers

Follow Semantic Versioning (SemVer):

  • Major: Breaking changes (e.g., 4.0.0 → 5.0.0)
  • Minor: New features, backward compatible (e.g., 4.31.0 → 4.32.0)
  • Patch: Bug fixes (e.g., 4.31.0 → 4.31.1)

Release Checklist

1. Prepare release:

  • Update CHANGELOG.md
  • Update version in Cargo.toml
  • Run full test suite
  • Test on multiple platforms

2. Create release:

# Tag version
git tag -a v4.32.0 -m "Release 4.32.0"
git push origin v4.32.0

# GitHub Actions handles:
# - Building binaries
# - Publishing to crates.io
# - Creating GitHub release
# - Building Python wheels

3. Verify release:

  • Check crates.io publication
  • Verify GitHub release artifacts
  • Test installation from crates.io

Architecture Details

Clean Architecture Layers

Domain Layer (innermost):

// Pure business logic, no external dependencies
pub trait BookmarkRepository: Send + Sync {
    fn get_bookmark(&self, id: i32) -> DomainResult<Bookmark>;
    fn add_bookmark(&self, bookmark: &Bookmark) -> DomainResult<i32>;
}

Application Layer:

// Use cases and orchestration
pub trait BookmarkService: Send + Sync {
    fn get_all_bookmarks(&self, limit: Option<i32>, offset: Option<i32>)
        -> ApplicationResult<Vec<Bookmark>>;
}

Infrastructure Layer:

// External system implementations
pub struct SqliteBookmarkRepository {
    pool: Pool<ConnectionManager<SqliteConnection>>,
}

impl BookmarkRepository for SqliteBookmarkRepository {
    fn get_bookmark(&self, id: i32) -> DomainResult<Bookmark> {
        // SQLite-specific implementation
    }
}

Dependency Flow

main.rs
  ↓
ServiceContainer (composition root)
  ↓ Creates all services with dependencies
CLI Commands
  ↓ Receives ServiceContainer + Settings
Application Services
  ↓ Business logic orchestration
Domain Entities
  ↓ Pure business logic
Infrastructure
  ↓ External systems

Error Handling Strategy

Layer-Specific Errors:

  • DomainError - Business rule violations
  • ApplicationError - Use case failures
  • InfrastructureError - External system errors
  • CliError - User interface errors

Error Conversion:

// Infrastructure → Domain
impl From<diesel::result::Error> for DomainError {
    fn from(err: diesel::result::Error) -> Self {
        DomainError::DatabaseError(err.to_string())
    }
}

// Domain → Application
impl From<DomainError> for ApplicationError {
    fn from(err: DomainError) -> Self {
        ApplicationError::DomainError(err)
    }
}

Advanced Topics

LSP Implementation

Architecture:

  • BkmrLspBackend - Main LSP server
  • tower-lsp for protocol handling
  • tokio runtime for async operations
  • Sync bkmr services wrapped in spawn_blocking

Key Files:

  • src/lsp/server.rs - LSP backend implementation
  • src/lsp/services/ - LSP-specific services
  • src/lsp/domain/ - Language registry and mappings

Testing Philosophy

Principles:

  • Test behavior, not implementation
  • Use dependency injection for mockability
  • Single-threaded execution for reliability
  • Arrange/Act/Assert structure

Test Categories:

  • Unit tests for business logic
  • Integration tests for CLI commands
  • LSP tests with Python scripts
  • Documentation tests for examples

Performance Considerations

Database:

  • Connection pooling (r2d2, max_size=15)
  • FTS5 for efficient search
  • Indexed columns for common queries

Memory:

  • Arc for shared ownership
  • Lazy loading for large content
  • Efficient string handling

Concurrency:

  • Single-threaded database access (SQLite limitation)
  • Arc for thread-safe sharing
  • No global mutable state

Community

Resources:

Getting Help:

  • Check wiki for documentation
  • Search existing issues
  • Open new issue with details
  • Join discussions

Related Pages

Clone this wiki locally