|
| 1 | +# Developer Guide |
| 2 | + |
| 3 | +Welcome to the python-common-expression-language development guide! This document is for contributors who want to understand the codebase architecture, development workflow, and how we maintain compatibility with the upstream CEL specification. |
| 4 | + |
| 5 | +## Project Architecture |
| 6 | + |
| 7 | +### Core Components |
| 8 | + |
| 9 | +This Python package provides bindings for Google's Common Expression Language (CEL) using a Rust backend: |
| 10 | + |
| 11 | +```mermaid |
| 12 | +%%{init: {"flowchart": {"padding": 20}}}%% |
| 13 | +flowchart LR |
| 14 | + subgraph Python[" 🐍 Python Layer "] |
| 15 | + API[" cel.evaluate()<br/> Context class<br/> CLI tool "] |
| 16 | + end |
| 17 | + |
| 18 | + subgraph Rust[" 🦀 Rust Wrapper (PyO3) "] |
| 19 | + Wrapper[" Type conversion<br/> Error handling<br/> Function calls "] |
| 20 | + end |
| 21 | + |
| 22 | + subgraph CEL[" ⚡ CEL Engine (upstream) "] |
| 23 | + Engine[" CEL parser<br/> Expression evaluation<br/> Built-in functions "] |
| 24 | + end |
| 25 | + |
| 26 | + Python --> Rust |
| 27 | + Rust --> CEL |
| 28 | + |
| 29 | + style Python fill:#e8f4f8,color:#2c3e50 |
| 30 | + style Rust fill:#fdf2e9,color:#2c3e50 |
| 31 | + style CEL fill:#f0f9ff,color:#2c3e50 |
| 32 | +``` |
| 33 | + |
| 34 | +**Key Files:** |
| 35 | + |
| 36 | +- `src/lib.rs` - Main evaluation engine and type conversions |
| 37 | +- `src/context.rs` - Context management and Python function integration |
| 38 | +- `python/cel/` - Python module structure and CLI |
| 39 | +- `tests/` - Comprehensive test suite with 300+ tests |
| 40 | + |
| 41 | +### Dependencies |
| 42 | + |
| 43 | +- **[cel crate](https://crates.io/crates/cel)** v0.11.0 - The Rust CEL implementation we wrap |
| 44 | +- **[PyO3](https://pyo3.rs/)** - Python-Rust bindings framework |
| 45 | +- **[maturin](https://www.maturin.rs/)** - Build system for Python extensions |
| 46 | + |
| 47 | +## Development Workflow |
| 48 | + |
| 49 | +### Setup |
| 50 | + |
| 51 | +```bash |
| 52 | +# Clone and setup development environment |
| 53 | +git clone https://github.com/hardbyte/python-common-expression-language.git |
| 54 | +cd python-common-expression-language |
| 55 | + |
| 56 | +# Install development dependencies |
| 57 | +uv sync --dev |
| 58 | + |
| 59 | +# Build the Rust extension |
| 60 | +uv run maturin develop |
| 61 | + |
| 62 | +# Run tests to verify setup |
| 63 | +uv run pytest |
| 64 | +``` |
| 65 | + |
| 66 | +### Code Organization |
| 67 | + |
| 68 | +``` |
| 69 | +python-common-expression-language/ |
| 70 | +├── src/ # Rust source code |
| 71 | +│ ├── lib.rs # Main module & evaluation engine |
| 72 | +│ └── context.rs # Context management |
| 73 | +├── python/ # Python module |
| 74 | +│ └── cel/ # Python package |
| 75 | +├── tests/ # Test suite (300+ tests) |
| 76 | +│ ├── test_basics.py # Core functionality |
| 77 | +│ ├── test_arithmetic.py # Arithmetic operations |
| 78 | +│ └── test_upstream_improvements.py # Future compatibility |
| 79 | +├── docs/ # Documentation |
| 80 | +└── pyproject.toml # Python package configuration |
| 81 | +``` |
| 82 | + |
| 83 | +### Testing Strategy |
| 84 | + |
| 85 | +We maintain comprehensive test coverage across multiple categories: |
| 86 | + |
| 87 | +```bash |
| 88 | +# Run all tests |
| 89 | +uv run pytest |
| 90 | + |
| 91 | +# Run specific test categories |
| 92 | +uv run pytest tests/test_basics.py # Core functionality |
| 93 | +uv run pytest tests/test_arithmetic.py # Math operations |
| 94 | +uv run pytest tests/test_context.py # Variable handling |
| 95 | +uv run pytest tests/test_upstream_improvements.py # Future compatibility |
| 96 | + |
| 97 | +# Run with coverage |
| 98 | +uv run pytest --cov=cel |
| 99 | +``` |
| 100 | + |
| 101 | + |
| 102 | +## Upstream Compatibility Strategy |
| 103 | + |
| 104 | +One of our key challenges is staying compatible with the evolving upstream `cel` crate while providing a stable Python API. |
| 105 | + |
| 106 | +### Monitoring Upstream Changes |
| 107 | + |
| 108 | +We use a proactive detection system to monitor for upstream improvements: |
| 109 | + |
| 110 | +**Location**: `tests/test_upstream_improvements.py` |
| 111 | + |
| 112 | +#### Detection Methodology |
| 113 | + |
| 114 | +1. **Negative Detection**: Tests that verify current limitations still exist |
| 115 | +2. **Positive Detection**: Expected failures (`@pytest.mark.xfail`) ready to pass when features arrive |
| 116 | + |
| 117 | +```python |
| 118 | +import pytest |
| 119 | +import cel |
| 120 | + |
| 121 | +# Example: Detecting when string functions become available |
| 122 | +def test_lower_ascii_not_implemented(self): |
| 123 | + """When this test starts failing, lowerAscii() has been implemented.""" |
| 124 | + with pytest.raises(RuntimeError, match="Undefined variable or function.*lowerAscii"): |
| 125 | + cel.evaluate('"HELLO".lowerAscii()') |
| 126 | + |
| 127 | +@pytest.mark.xfail(reason="String utilities not implemented in cel v0.11.0", strict=False) |
| 128 | +def test_lower_ascii_expected_behavior(self): |
| 129 | + """This test will pass when upstream implements lowerAscii().""" |
| 130 | + assert cel.evaluate('"HELLO".lowerAscii()') == "hello" |
| 131 | +``` |
| 132 | + |
| 133 | +#### Monitored Categories |
| 134 | + |
| 135 | +| Category | Status | Impact | |
| 136 | +|----------|--------|---------| |
| 137 | +| **String Functions** (`lowerAscii`, `upperAscii`, `indexOf`, etc.) | 8 functions monitored | Medium - String processing | |
| 138 | +| **Type Introspection** (`type()` function) | Ready to detect | Medium - Dynamic typing | |
| 139 | +| **Mixed Arithmetic** (`int + uint` operations) | Comprehensive detection | Medium - Type safety | |
| 140 | +| **Optional Values** (`optional.of()`, `?.` chaining) | Future feature detection | Low - Advanced use cases | |
| 141 | +| **🚨 OR Operator** (CEL spec compliance) | **Critical behavioral difference** | **High - Logic errors** | |
| 142 | +| **Math Functions** (`ceil`, `floor`, `round`) | Standard library functions | Low - Mathematical operations | |
| 143 | + |
| 144 | +#### Running Detection Tests |
| 145 | + |
| 146 | +```bash |
| 147 | +# Check current upstream compatibility status |
| 148 | +uv run pytest tests/test_upstream_improvements.py -v |
| 149 | + |
| 150 | +# Look for XPASS results indicating new capabilities |
| 151 | +uv run pytest tests/test_upstream_improvements.py -v --tb=no | grep -E "(XPASS|FAILED)" |
| 152 | +``` |
| 153 | + |
| 154 | +**Interpreting Results:** |
| 155 | +- **PASSED** = Limitation still exists (expected) |
| 156 | +- **XFAIL** = Expected failure (ready for when feature arrives) |
| 157 | +- **XPASS** = 🎉 Feature now available! (remove xfail marker) |
| 158 | + |
| 159 | +### Dependency Update Process |
| 160 | + |
| 161 | +When updating the `cel` crate dependency: |
| 162 | + |
| 163 | +1. **Run detection tests first** to identify new capabilities |
| 164 | +2. **Update Cargo.toml** with new version |
| 165 | +3. **Fix compilation issues** (API changes) |
| 166 | +4. **Remove xfail markers** for now-passing tests |
| 167 | +5. **Update documentation** to reflect new features |
| 168 | +6. **Test thoroughly** to ensure no regressions |
| 169 | + |
| 170 | +## Code Style & Conventions |
| 171 | + |
| 172 | +### Rust Code |
| 173 | + |
| 174 | +```rust |
| 175 | +// Follow standard Rust conventions |
| 176 | +use ::cel::objects::TryIntoValue; |
| 177 | +use ::cel::Value; |
| 178 | + |
| 179 | +// Document complex functions |
| 180 | +/// Converts a Python object to a CEL Value with proper error handling |
| 181 | +pub fn python_to_cel_value(obj: &PyAny) -> PyResult<Value> { |
| 182 | + // Implementation... |
| 183 | +} |
| 184 | +``` |
| 185 | + |
| 186 | +### Python Code |
| 187 | + |
| 188 | +```python |
| 189 | +from typing import Optional, Union, Dict, Any, Callable |
| 190 | +import cel |
| 191 | + |
| 192 | +# Type hints for public APIs |
| 193 | +def evaluate(expression: str, context: Optional[Union[Dict[str, Any], 'Context']] = None) -> Any: |
| 194 | + """Evaluate a CEL expression with optional context.""" |
| 195 | + pass |
| 196 | + |
| 197 | +# Comprehensive docstrings |
| 198 | +def add_function(self, name: str, func: Callable) -> None: |
| 199 | + """Add a Python function to the CEL evaluation context. |
| 200 | + |
| 201 | + Args: |
| 202 | + name: Function name to use in CEL expressions |
| 203 | + func: Python callable to invoke |
| 204 | + |
| 205 | + Example: |
| 206 | + >>> context = cel.Context() |
| 207 | + >>> context.add_function("double", lambda x: x * 2) |
| 208 | + >>> cel.evaluate("double(21)", context) |
| 209 | + 42 |
| 210 | + """ |
| 211 | +``` |
| 212 | + |
| 213 | +## Debugging & Troubleshooting |
| 214 | + |
| 215 | +### Common Issues |
| 216 | + |
| 217 | +**Build Failures:** |
| 218 | +```bash |
| 219 | +# Clean rebuild |
| 220 | +uv run maturin develop --release |
| 221 | + |
| 222 | +# Check Rust toolchain |
| 223 | +rustc --version |
| 224 | +cargo --version |
| 225 | +``` |
| 226 | + |
| 227 | +**Test Failures:** |
| 228 | +```bash |
| 229 | +# Run with verbose output |
| 230 | +uv run pytest tests/test_failing.py -v -s |
| 231 | + |
| 232 | +# Debug specific test |
| 233 | +uv run pytest tests/test_file.py::test_name --pdb |
| 234 | +``` |
| 235 | + |
| 236 | +**Type Conversion Issues:** |
| 237 | +```bash |
| 238 | +# Check Python-Rust boundary |
| 239 | +uv run pytest tests/test_types.py -v --tb=long |
| 240 | +``` |
| 241 | + |
| 242 | +### Performance Profiling |
| 243 | + |
| 244 | +```bash |
| 245 | +# Basic performance verification |
| 246 | +uv run pytest tests/test_performance_verification.py |
| 247 | + |
| 248 | +# Memory profiling (if needed) |
| 249 | +uv run pytest --profile tests/test_performance.py |
| 250 | +``` |
| 251 | + |
| 252 | +## Release Process |
| 253 | + |
| 254 | +1. **Version Bump** - Update version in `pyproject.toml` |
| 255 | +2. **Changelog** - Document changes in `CHANGELOG.md` |
| 256 | +3. **Release** - Create a release in GitHub to trigger publishing to PyPI |
| 257 | + |
0 commit comments