Skip to content

Commit 0f57936

Browse files
author
Daniele Briggi
committed
feat(python): workflow to build python package
1 parent 92890f0 commit 0f57936

File tree

9 files changed

+334
-0
lines changed

9 files changed

+334
-0
lines changed

‎.github/workflows/python-package.yml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: Build and Publish Python Package
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
version:
7+
description: "Version to use for the Python package (e.g. 0.9.9)"
8+
required: true
9+
type: string
10+
release:
11+
types: [published]
12+
13+
jobs:
14+
build-and-publish:
15+
runs-on: ${{ matrix.os }}
16+
permissions:
17+
id-token: write # mandatory for Pypi trusted publishing
18+
strategy:
19+
matrix:
20+
include:
21+
- os: ubuntu-latest
22+
platform: linux
23+
python-version: "3.10"
24+
arch: x86_64
25+
plat_name: manylinux2014_x86_64
26+
- os: ubuntu-latest
27+
platform: linux
28+
python-version: "3.10"
29+
arch: arm64
30+
plat_name: manylinux2014_aarch64
31+
- os: ubuntu-latest
32+
platform: windows
33+
python-version: "3.10"
34+
arch: x86_64
35+
plat_name: win_amd64
36+
- os: ubuntu-latest
37+
platform: macos
38+
python-version: "3.10"
39+
arch: x86_64
40+
plat_name: macosx_10_9_x86_64
41+
- os: ubuntu-latest
42+
platform: macos
43+
python-version: "3.10"
44+
arch: arm64
45+
plat_name: macosx_11_0_arm64
46+
defaults:
47+
run:
48+
shell: bash
49+
steps:
50+
- uses: actions/checkout@v4
51+
with:
52+
submodules: false
53+
54+
- name: Set up Python
55+
uses: actions/setup-python@v5
56+
with:
57+
python-version: ${{ matrix.python-version }}
58+
59+
- name: Install build dependencies
60+
run: |
61+
cd packages/python
62+
python3 -m pip install --upgrade pip
63+
python3 -m pip install -r requirements-dev.txt
64+
65+
- name: Get version
66+
id: get_version
67+
run: |
68+
if [[ "${{ github.event_name }}" == "release" ]]; then
69+
VERSION="${{ github.event.release.tag_name }}"
70+
else
71+
VERSION="${{ github.event.inputs.version }}"
72+
fi
73+
VERSION=${VERSION#v}
74+
echo "version=$VERSION" >> $GITHUB_OUTPUT
75+
76+
- name: Download artifacts for current platform
77+
run: |
78+
cd packages/python
79+
python3 download_artifacts.py "${{ matrix.plat_name }}" "${{ steps.get_version.outputs.version }}"
80+
81+
- name: Build wheel
82+
env:
83+
PACKAGE_VERSION: ${{ steps.get_version.outputs.version }}
84+
run: |
85+
cd packages/python
86+
python setup.py bdist_wheel --plat-name "${{ matrix.plat_name }}"
87+
88+
- name: Publish to PyPI
89+
uses: pypa/gh-action-pypi-publish@release/v1
90+
with:
91+
packages-dir: packages/python/dist
92+
verbose: true
93+
# Avoid workflow to fail if the version has already been published
94+
skip-existing: true
95+
# Upload to Test Pypi for testing
96+
repository-url: https://test.pypi.org/legacy/

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,16 @@ SELECT load_extension('./vector');
5252

5353
Or embed it directly into your application.
5454

55+
### Python Package
56+
57+
Python developers can quickly get started using the ready-to-use `sqlite-vector` package available on PyPI:
58+
59+
```bash
60+
pip install sqlite-vector
61+
```
62+
63+
For usage details and examples, see the [Python package documentation](./packages/python/README.md).
64+
5565
## Example Usage
5666

5767
```sql

‎packages/python/MANIFEST.in‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
include README.md
2+
include LICENSE
3+
recursive-include src/sqlite-vector/binaries *

‎packages/python/README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
## SQLite Vector Python package
2+
3+
This package provides the sqlite-vector extension prebuilt binaries for multiple platforms and architectures.
4+
5+
### SQLite Vector
6+
7+
SQLite Vector is a cross-platform, ultra-efficient SQLite extension that brings vector search capabilities to your embedded database. It works seamlessly on iOS, Android, Windows, Linux, and macOS, using just 30MB of memory by default. With support for Float32, Float16, BFloat16, Int8, and UInt8, and highly optimized distance functions, it's the ideal solution for Edge AI applications.
8+
9+
More details on the official repository [sqliteai/sqlite-vector](https://github.com/sqliteai/sqlite-vector).
10+
11+
### Documentation
12+
13+
For detailed information on all available functions, their parameters, and examples, refer to the [comprehensive API Reference](https://github.com/sqliteai/sqlite-vector/blob/main/API.md).
14+
15+
### Supported Platforms and Architectures
16+
17+
| Platform | Arch | Subpackage name | Binary name |
18+
| ------------- | ------------ | ------------------- | ------------ |
19+
| Linux (CPU) | x86_64/arm64 | sqlite-vector.binaries | vector.so |
20+
| Windows (CPU) | x86_64 | sqlite-vector.binaries | vector.dll |
21+
| macOS (CPU) | x86_64/arm64 | sqlite-vector.binaries | vector.dylib |
22+
23+
## Usage
24+
25+
> **Note:** Some SQLite installations on certain operating systems may have extension loading disabled by default.
26+
If you encounter issues loading the extension, refer to the [sqlite-extensions-guide](https://github.com/sqliteai/sqlite-extensions-guide/) for platform-specific instructions on enabling and using SQLite extensions.
27+
28+
```python
29+
import importlib.resources
30+
import sqlite3
31+
32+
# Connect to your SQLite database
33+
conn = sqlite3.connect("example.db")
34+
35+
# Load the sqlite-vector extension
36+
# pip will install the correct binary package for your platform and architecture
37+
ext_path = importlib.resources.files("sqlite-vector.binaries") / "vector"
38+
39+
conn.enable_load_extension(True)
40+
conn.load_extension(str(ext_path))
41+
conn.enable_load_extension(False)
42+
43+
44+
# Now you can use sqlite-vector features in your SQL queries
45+
print(conn.execute("SELECT vector_version();").fetchone())
46+
```

‎packages/python/download_artifacts.py‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
import sys
2+
import zipfile
3+
import requests
4+
from pathlib import Path
5+
import shutil
6+
7+
8+
# == USAGE ==
9+
# python3 download_artifacts.py PLATFORM VERSION
10+
# eg: python3 download_artifacts.py linux_x86_64 "0.5.9"
11+
12+
REPO = "sqliteai/sqlite-vector"
13+
RELEASE_URL = f"https://github.com/{REPO}/releases/download"
14+
15+
# Map Python plat_name to artifact names
16+
ARTIFACTS = {
17+
"manylinux2014_x86_64": ["vector-linux-x86_64"],
18+
"manylinux2014_aarch64": [
19+
"vector-linux-arm64",
20+
],
21+
"win_amd64": ["vector-windows-x86_64"],
22+
"macosx_10_9_x86_64": ["vector-macos"],
23+
"macosx_11_0_arm64": ["vector-macos"],
24+
}
25+
26+
BINARY_NAME = {
27+
"manylinux2014_x86_64": "vector.so",
28+
"manylinux2014_aarch64": "vector.so",
29+
"win_amd64": "vector.dll",
30+
"macosx_10_9_x86_64": "vector.dylib",
31+
"macosx_11_0_arm64": "vector.dylib",
32+
}
33+
34+
BINARIES_DIR = Path(__file__).parent / "src/sqlite-vector/binaries"
35+
36+
37+
def download_and_extract(artifact_name, bin_name, version):
38+
artifact = f"{artifact_name}-{version}.zip"
39+
url = f"{RELEASE_URL}/{version}/{artifact}"
40+
print(f"Downloading {url}")
41+
42+
r = requests.get(url)
43+
if r.status_code != 200:
44+
print(f"Failed to download {artifact}: {r.status_code}")
45+
sys.exit(1)
46+
47+
zip_path = BINARIES_DIR / artifact
48+
with open(zip_path, "wb") as f:
49+
f.write(r.content)
50+
51+
out_dir = BINARIES_DIR
52+
out_dir.mkdir(parents=True, exist_ok=True)
53+
54+
with zipfile.ZipFile(zip_path, "r") as zip_ref:
55+
for member in zip_ref.namelist():
56+
if member.endswith(bin_name):
57+
zip_ref.extract(member, out_dir)
58+
59+
# Move to expected name/location
60+
src = out_dir / member
61+
dst = out_dir / bin_name
62+
src.rename(dst)
63+
64+
print(f"Extracted {dst}")
65+
66+
zip_path.unlink()
67+
68+
69+
def main():
70+
version = None
71+
platform = None
72+
if len(sys.argv) == 3:
73+
platform = sys.argv[1].lower()
74+
version = sys.argv[2]
75+
76+
if not version or not platform:
77+
print(
78+
'Error: Version is not specified.\nUsage: \n python3 download_artifacts.py linux_x86_64 "0.5.9"'
79+
)
80+
sys.exit(1)
81+
82+
print(BINARIES_DIR)
83+
if BINARIES_DIR.exists():
84+
shutil.rmtree(BINARIES_DIR)
85+
BINARIES_DIR.mkdir(parents=True, exist_ok=True)
86+
87+
platform_artifacts = ARTIFACTS.get(platform, [])
88+
if not platform_artifacts:
89+
print(f"Error: Unknown platform '{platform}'")
90+
sys.exit(1)
91+
92+
for artifact_name in platform_artifacts:
93+
download_and_extract(artifact_name, BINARY_NAME[platform], version)
94+
95+
96+
if __name__ == "__main__":
97+
main()

‎packages/python/pyproject.toml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
[build-system]
2+
requires = ["setuptools>=61.0", "wheel", "toml"]
3+
build-backend = "setuptools.build_meta"
4+
5+
[project]
6+
name = "sqlite-vector"
7+
dynamic = ["version"]
8+
description = "Python prebuilt binaries for SQLite Vector extension for all supported platforms and architectures."
9+
authors = [
10+
{ name = "SQLite AI Team" }
11+
]
12+
readme = "README.md"
13+
requires-python = ">=3"
14+
classifiers = [
15+
"Programming Language :: Python :: 3",
16+
"Operating System :: POSIX :: Linux",
17+
"Operating System :: Microsoft :: Windows",
18+
"Operating System :: MacOS :: MacOS X"
19+
]
20+
21+
[project.urls]
22+
Homepage = "https://sqlite.ai"
23+
Documentation = "https://github.com/sqliteai/sqlite-vector/blob/main/API.md"
24+
Repository = "https://github.com/sqliteai/sqlite-vector"
25+
Issues = "https://github.com/sqliteai/sqlite-vector/issues"

‎packages/python/requirements-dev.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
requests
2+
toml
3+
wheel

‎packages/python/setup.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import setuptools
2+
import toml
3+
import os
4+
import sys
5+
6+
usage = """
7+
Usage: python setup.py bdist_wheel --plat-name <platform>
8+
The PACKAGE_VERSION environment variable must be set to the desired version.
9+
10+
Example:
11+
PACKAGE_VERSION=0.5.9 python setup.py bdist_wheel --plat-name linux_x86_64
12+
"""
13+
14+
with open("pyproject.toml", "r") as f:
15+
pyproject = toml.load(f)
16+
17+
project = pyproject["project"]
18+
19+
# Get version from environment or default
20+
version = os.environ.get("PACKAGE_VERSION", "")
21+
if not version:
22+
print("PACKAGE_VERSION environment variable is not set.")
23+
print(usage)
24+
sys.exit(1)
25+
26+
# Get Python platform name from --plat-name argument
27+
plat_name = None
28+
for i, arg in enumerate(sys.argv):
29+
if arg == "--plat-name" and i + 1 < len(sys.argv):
30+
plat_name = sys.argv[i + 1]
31+
break
32+
33+
if not plat_name:
34+
print("Error: --plat-name argument is required")
35+
print(usage)
36+
sys.exit(1)
37+
38+
with open("README.md", "r", encoding="utf-8") as f:
39+
long_description = f.read()
40+
41+
setuptools.setup(
42+
name=project["name"],
43+
version=version,
44+
description=project.get("description", ""),
45+
author=project["authors"][0]["name"],
46+
long_description=long_description,
47+
long_description_content_type="text/markdown",
48+
url=project["urls"]["Homepage"],
49+
packages=setuptools.find_packages(where="src"),
50+
package_dir={"": "src"},
51+
include_package_data=True,
52+
python_requires=project.get("requires-python", ">=3"),
53+
classifiers=project.get("classifiers", []),
54+
)

‎packages/python/src/sqlite-vector/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)