|
| 1 | +# Developer Guide |
| 2 | + |
| 3 | +Welcome to the python-common-expression-language development guide! This document is for contributors who want to understand the codebase architecture, development workflow, and how we maintain compatibility with the upstream CEL specification. |
| 4 | + |
| 5 | +## Project Architecture |
| 6 | + |
| 7 | +### Core Components |
| 8 | + |
| 9 | +This Python package provides bindings for Google's Common Expression Language (CEL) using a Rust backend: |
| 10 | + |
| 11 | +``` |
| 12 | +┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ |
| 13 | +│ Python API │───▶│ Rust Wrapper │───▶│ cel crate │ |
| 14 | +│ │ │ (PyO3) │ │ (upstream) │ |
| 15 | +├─────────────────┤ ├──────────────────┤ ├─────────────────┤ |
| 16 | +│ • cel.evaluate │ │ • Type conversion│ │ • CEL parser │ |
| 17 | +│ • Context class │ │ • Error handling │ │ • Expression │ |
| 18 | +│ • CLI tool │ │ • Function calls │ │ evaluation │ |
| 19 | +└─────────────────┘ └──────────────────┘ └─────────────────┘ |
| 20 | +``` |
| 21 | + |
| 22 | +**Key Files:** |
| 23 | +- `src/lib.rs` - Main evaluation engine and type conversions |
| 24 | +- `src/context.rs` - Context management and Python function integration |
| 25 | +- `python/cel/` - Python module structure and CLI |
| 26 | +- `tests/` - Comprehensive test suite with 300+ tests |
| 27 | + |
| 28 | +### Dependencies |
| 29 | + |
| 30 | +- **[cel crate](https://crates.io/crates/cel)** v0.11.0 - The Rust CEL implementation we wrap |
| 31 | +- **[PyO3](https://pyo3.rs/)** - Python-Rust bindings framework |
| 32 | +- **[maturin](https://www.maturin.rs/)** - Build system for Python extensions |
| 33 | + |
| 34 | +## Development Workflow |
| 35 | + |
| 36 | +### Setup |
| 37 | + |
| 38 | +```bash |
| 39 | +# Clone and setup development environment |
| 40 | +git clone https://github.com/hardbyte/python-common-expression-language.git |
| 41 | +cd python-common-expression-language |
| 42 | + |
| 43 | +# Install development dependencies |
| 44 | +uv sync --dev |
| 45 | + |
| 46 | +# Build the Rust extension |
| 47 | +uv run maturin develop |
| 48 | + |
| 49 | +# Run tests to verify setup |
| 50 | +uv run pytest |
| 51 | +``` |
| 52 | + |
| 53 | +### Code Organization |
| 54 | + |
| 55 | +``` |
| 56 | +python-common-expression-language/ |
| 57 | +├── src/ # Rust source code |
| 58 | +│ ├── lib.rs # Main module & evaluation engine |
| 59 | +│ └── context.rs # Context management |
| 60 | +├── python/ # Python module |
| 61 | +│ └── cel/ # Python package |
| 62 | +├── tests/ # Test suite (300+ tests) |
| 63 | +│ ├── test_basics.py # Core functionality |
| 64 | +│ ├── test_arithmetic.py # Arithmetic operations |
| 65 | +│ └── test_upstream_improvements.py # Future compatibility |
| 66 | +├── docs/ # Documentation |
| 67 | +└── pyproject.toml # Python package configuration |
| 68 | +``` |
| 69 | + |
| 70 | +### Testing Strategy |
| 71 | + |
| 72 | +We maintain comprehensive test coverage across multiple categories: |
| 73 | + |
| 74 | +```bash |
| 75 | +# Run all tests |
| 76 | +uv run pytest |
| 77 | + |
| 78 | +# Run specific test categories |
| 79 | +uv run pytest tests/test_basics.py # Core functionality |
| 80 | +uv run pytest tests/test_arithmetic.py # Math operations |
| 81 | +uv run pytest tests/test_context.py # Variable handling |
| 82 | +uv run pytest tests/test_upstream_improvements.py # Future compatibility |
| 83 | + |
| 84 | +# Run with coverage |
| 85 | +uv run pytest --cov=cel |
| 86 | +``` |
| 87 | + |
| 88 | +**Test Categories:** |
| 89 | +- **Basic Operations** (42 tests) - Core CEL evaluation |
| 90 | +- **Arithmetic** (31 tests) - Math operations and mixed types |
| 91 | +- **Type Conversion** (23 tests) - Python ↔ CEL type mapping |
| 92 | +- **Context Management** (11 tests) - Variables and functions |
| 93 | +- **Upstream Detection** (26 tests) - Future compatibility monitoring |
| 94 | + |
| 95 | +## Upstream Compatibility Strategy |
| 96 | + |
| 97 | +One of our key challenges is staying compatible with the evolving upstream `cel` crate while providing a stable Python API. |
| 98 | + |
| 99 | +### Monitoring Upstream Changes |
| 100 | + |
| 101 | +We use a proactive detection system to monitor for upstream improvements: |
| 102 | + |
| 103 | +**Location**: `tests/test_upstream_improvements.py` |
| 104 | + |
| 105 | +#### Detection Methodology |
| 106 | + |
| 107 | +1. **Negative Detection**: Tests that verify current limitations still exist |
| 108 | +2. **Positive Detection**: Expected failures (`@pytest.mark.xfail`) ready to pass when features arrive |
| 109 | + |
| 110 | +```python |
| 111 | +import pytest |
| 112 | +import cel |
| 113 | + |
| 114 | +# Example: Detecting when string functions become available |
| 115 | +def test_lower_ascii_not_implemented(self): |
| 116 | + """When this test starts failing, lowerAscii() has been implemented.""" |
| 117 | + with pytest.raises(RuntimeError, match="Undefined variable or function.*lowerAscii"): |
| 118 | + cel.evaluate('"HELLO".lowerAscii()') |
| 119 | + |
| 120 | +@pytest.mark.xfail(reason="String utilities not implemented in cel v0.11.0", strict=False) |
| 121 | +def test_lower_ascii_expected_behavior(self): |
| 122 | + """This test will pass when upstream implements lowerAscii().""" |
| 123 | + assert cel.evaluate('"HELLO".lowerAscii()') == "hello" |
| 124 | +``` |
| 125 | + |
| 126 | +#### Monitored Categories |
| 127 | + |
| 128 | +| Category | Status | Impact | |
| 129 | +|----------|--------|---------| |
| 130 | +| **String Functions** (`lowerAscii`, `upperAscii`, `indexOf`, etc.) | 8 functions monitored | Medium - String processing | |
| 131 | +| **Type Introspection** (`type()` function) | Ready to detect | Medium - Dynamic typing | |
| 132 | +| **Mixed Arithmetic** (`int + uint` operations) | Comprehensive detection | Medium - Type safety | |
| 133 | +| **Optional Values** (`optional.of()`, `?.` chaining) | Future feature detection | Low - Advanced use cases | |
| 134 | +| **🚨 OR Operator** (CEL spec compliance) | **Critical behavioral difference** | **High - Logic errors** | |
| 135 | +| **Math Functions** (`ceil`, `floor`, `round`) | Standard library functions | Low - Mathematical operations | |
| 136 | + |
| 137 | +#### Running Detection Tests |
| 138 | + |
| 139 | +```bash |
| 140 | +# Check current upstream compatibility status |
| 141 | +uv run pytest tests/test_upstream_improvements.py -v |
| 142 | + |
| 143 | +# Look for XPASS results indicating new capabilities |
| 144 | +uv run pytest tests/test_upstream_improvements.py -v --tb=no | grep -E "(XPASS|FAILED)" |
| 145 | +``` |
| 146 | + |
| 147 | +**Interpreting Results:** |
| 148 | +- **PASSED** = Limitation still exists (expected) |
| 149 | +- **XFAIL** = Expected failure (ready for when feature arrives) |
| 150 | +- **XPASS** = 🎉 Feature now available! (remove xfail marker) |
| 151 | + |
| 152 | +### Dependency Update Process |
| 153 | + |
| 154 | +When updating the `cel` crate dependency: |
| 155 | + |
| 156 | +1. **Run detection tests first** to identify new capabilities |
| 157 | +2. **Update Cargo.toml** with new version |
| 158 | +3. **Fix compilation issues** (API changes) |
| 159 | +4. **Remove xfail markers** for now-passing tests |
| 160 | +5. **Update documentation** to reflect new features |
| 161 | +6. **Test thoroughly** to ensure no regressions |
| 162 | + |
| 163 | +Example recent upgrade (cel-interpreter 0.10.0 → cel 0.11.0): |
| 164 | +- Crate was renamed from `cel-interpreter` to `cel` |
| 165 | +- Function registration API completely changed (new `IntoFunction` trait) |
| 166 | +- All Python API remained backward compatible |
| 167 | +- 287 tests continued passing after migration |
| 168 | + |
| 169 | +## Code Style & Conventions |
| 170 | + |
| 171 | +### Rust Code |
| 172 | + |
| 173 | +```rust |
| 174 | +// Follow standard Rust conventions |
| 175 | +use ::cel::objects::TryIntoValue; |
| 176 | +use ::cel::Value; |
| 177 | + |
| 178 | +// Document complex functions |
| 179 | +/// Converts a Python object to a CEL Value with proper error handling |
| 180 | +pub fn python_to_cel_value(obj: &PyAny) -> PyResult<Value> { |
| 181 | + // Implementation... |
| 182 | +} |
| 183 | +``` |
| 184 | + |
| 185 | +### Python Code |
| 186 | + |
| 187 | +```python |
| 188 | +from typing import Optional, Union, Dict, Any, Callable |
| 189 | +import cel |
| 190 | + |
| 191 | +# Type hints for public APIs |
| 192 | +def evaluate(expression: str, context: Optional[Union[Dict[str, Any], 'Context']] = None) -> Any: |
| 193 | + """Evaluate a CEL expression with optional context.""" |
| 194 | + pass |
| 195 | + |
| 196 | +# Comprehensive docstrings |
| 197 | +def add_function(self, name: str, func: Callable) -> None: |
| 198 | + """Add a Python function to the CEL evaluation context. |
| 199 | + |
| 200 | + Args: |
| 201 | + name: Function name to use in CEL expressions |
| 202 | + func: Python callable to invoke |
| 203 | + |
| 204 | + Example: |
| 205 | + >>> context = cel.Context() |
| 206 | + >>> context.add_function("double", lambda x: x * 2) |
| 207 | + >>> cel.evaluate("double(21)", context) |
| 208 | + 42 |
| 209 | + """ |
| 210 | +``` |
| 211 | + |
| 212 | +### Testing Conventions |
| 213 | + |
| 214 | +```python |
| 215 | +import pytest |
| 216 | +import cel |
| 217 | + |
| 218 | +class TestFeatureCategory: |
| 219 | + """Test [specific feature] with [scope] coverage.""" |
| 220 | + |
| 221 | + def test_specific_behavior(self): |
| 222 | + """Test [what] [under what conditions].""" |
| 223 | + # Arrange |
| 224 | + context = {"key": "value"} |
| 225 | + |
| 226 | + # Act |
| 227 | + result = cel.evaluate("key", context) |
| 228 | + |
| 229 | + # Assert |
| 230 | + assert result == "value" |
| 231 | + |
| 232 | + def test_error_condition(self): |
| 233 | + """Test that [condition] raises [exception type].""" |
| 234 | + with pytest.raises(RuntimeError, match="Undefined variable"): |
| 235 | + cel.evaluate("undefined_variable") |
| 236 | +``` |
| 237 | + |
| 238 | +## Contributing Guidelines |
| 239 | + |
| 240 | +### Development Process |
| 241 | + |
| 242 | +1. **Issue Discussion** - Open an issue to discuss significant changes |
| 243 | +2. **Branch Creation** - Create feature branch from main |
| 244 | +3. **Implementation** - Follow code style and add tests |
| 245 | +4. **Testing** - Ensure all tests pass (`uv run pytest`) |
| 246 | +5. **Documentation** - Update docs for user-facing changes |
| 247 | +6. **Pull Request** - Submit with clear description and examples |
| 248 | + |
| 249 | +### What We're Looking For |
| 250 | + |
| 251 | +**High Priority Contributions:** |
| 252 | +- **Enhanced error handling** - Better Python exception mapping |
| 253 | +- **Performance improvements** - Optimization of type conversions |
| 254 | +- **Local utility functions** - Python implementations of missing CEL functions |
| 255 | +- **Documentation improvements** - Examples, guides, edge cases |
| 256 | + |
| 257 | +**Upstream Contributions (cel crate):** |
| 258 | +- **String utilities** - `lowerAscii`, `upperAscii`, `indexOf`, etc. |
| 259 | +- **Type introspection** - `type()` function implementation |
| 260 | +- **Mixed arithmetic** - Better signed/unsigned integer support |
| 261 | +- **CEL spec compliance** - OR operator boolean return values |
| 262 | + |
| 263 | +### Testing Requirements |
| 264 | + |
| 265 | +All contributions must include: |
| 266 | +- **Unit tests** for new functionality |
| 267 | +- **Integration tests** for user-facing features |
| 268 | +- **Error condition tests** for edge cases |
| 269 | +- **Documentation tests** for examples in docs |
| 270 | + |
| 271 | +```bash |
| 272 | +# Full test suite (required before PR) |
| 273 | +uv run pytest |
| 274 | + |
| 275 | +# Documentation examples (must pass) |
| 276 | +uv run --group docs pytest tests/test_docs.py |
| 277 | + |
| 278 | +# Upstream compatibility (monitoring) |
| 279 | +uv run pytest tests/test_upstream_improvements.py |
| 280 | +``` |
| 281 | + |
| 282 | +## Debugging & Troubleshooting |
| 283 | + |
| 284 | +### Common Issues |
| 285 | + |
| 286 | +**Build Failures:** |
| 287 | +```bash |
| 288 | +# Clean rebuild |
| 289 | +uv run maturin develop --release |
| 290 | + |
| 291 | +# Check Rust toolchain |
| 292 | +rustc --version |
| 293 | +cargo --version |
| 294 | +``` |
| 295 | + |
| 296 | +**Test Failures:** |
| 297 | +```bash |
| 298 | +# Run with verbose output |
| 299 | +uv run pytest tests/test_failing.py -v -s |
| 300 | + |
| 301 | +# Debug specific test |
| 302 | +uv run pytest tests/test_file.py::test_name --pdb |
| 303 | +``` |
| 304 | + |
| 305 | +**Type Conversion Issues:** |
| 306 | +```bash |
| 307 | +# Check Python-Rust boundary |
| 308 | +uv run pytest tests/test_types.py -v --tb=long |
| 309 | +``` |
| 310 | + |
| 311 | +### Performance Profiling |
| 312 | + |
| 313 | +```bash |
| 314 | +# Basic performance verification |
| 315 | +uv run pytest tests/test_performance_verification.py |
| 316 | + |
| 317 | +# Memory profiling (if needed) |
| 318 | +uv run pytest --profile tests/test_performance.py |
| 319 | +``` |
| 320 | + |
| 321 | +## Release Process |
| 322 | + |
| 323 | +1. **Version Bump** - Update version in `pyproject.toml` |
| 324 | +2. **Changelog** - Document changes in `CHANGELOG.md` |
| 325 | +3. **Testing** - Full test suite across Python versions |
| 326 | +4. **Documentation** - Update any version-specific docs |
| 327 | +5. **Release** - Tag and publish to PyPI via CI |
| 328 | + |
| 329 | +## Resources |
| 330 | + |
| 331 | +### Documentation |
| 332 | +- **User Docs**: https://python-common-expression-language.readthedocs.io/ |
| 333 | +- **CEL Specification**: https://github.com/google/cel-spec |
| 334 | +- **cel crate**: https://docs.rs/cel/latest/cel/ |
| 335 | + |
| 336 | +### Development Tools |
| 337 | +- **PyO3 Guide**: https://pyo3.rs/ |
| 338 | +- **maturin**: https://www.maturin.rs/ |
| 339 | +- **Rust Book**: https://doc.rust-lang.org/book/ |
| 340 | + |
| 341 | +### Community |
| 342 | +- **Issues**: https://github.com/hardbyte/python-common-expression-language/issues |
| 343 | +- **Discussions**: Use GitHub Discussions for questions and ideas |
| 344 | +- **CEL Community**: https://github.com/google/cel-spec/discussions |
| 345 | + |
| 346 | +--- |
| 347 | + |
| 348 | +Thank you for contributing to python-common-expression-language! Your efforts help provide a robust, performant CEL implementation for the Python ecosystem. |
0 commit comments