Package polaris client as python package #2049

MonkeyCanCode · 2025-07-14T23:04:01Z

Currently we are using <git_root_dir>/polaris as the entrypoint to interact with polaris CLI. This is a okay for local usage but it is not ideal for others who doesn't want to downlaod the entire code base to just use the CLI as well as making polaris client distributable as part of release. To address this issue, we will need to public polaris as an CLI without depends on the bash script.

This PR addressed this problem with following changes:

create main in polaris_cli.py as that will be the entrypoint for us to use
set scripts directly (which is point to item 1) and include both polaris and cli directories as part of packaging. Due to client/python/polais is auto generated from OpenAPI and poetry.core.masonry.api doe build backend (with poetry.core.masonry.api as build backend, it only includes files that are tracked by git. in this case, then auto generated files won't be track without including adding an include clause)
Update the invoke command when calling python polaris within bash script polaris

Here is a quick demo on the packaging:

setup poetry then perform a poetry build (which we can use later to distribute the client code as well as push to Python Package Index):

➜  polaris git:(polaris_cli_package) python3 -m venv venv
➜  polaris git:(polaris_cli_package) source venv/bin/activate
(venv) ➜  polaris git:(polaris_cli_package) pip install -r regtests/requirements.txt
...
[notice] A new release of pip is available: 24.2 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
(venv) ➜  python git:(polaris_cli_package) poetry build
Building polaris (1.0.0)
Building sdist
  - Building sdist
  - Built polaris-1.0.0.tar.gz
Building wheel
  - Building wheel
  - Built polaris-1.0.0-py3-none-any.whl

now assuming I am an user who doesn't want to download the code base but want to use the cli:

➜  Desktop python3 -m venv venv_dist
➜  Desktop source venv_dist/bin/activate
(venv_dist) ➜  Desktop pip freeze
(venv_dist) ➜  Desktop
(venv_dist) ➜  Desktop pip install GitHome/polaris/client/python/dist/polaris-1.0.0.tar.gz
...
Building wheels for collected packages: polaris
  Building wheel for polaris (pyproject.toml) ... done
  Created wheel for polaris: filename=polaris-1.0.0-py3-none-any.whl size=520413 sha256=9c9a25c6edb2a0b642666ab07abf5bc52f0a51f939095b18f12404bb118ec2e5
  Stored in directory: /Users/yong/Library/Caches/pip/wheels/b8/f1/20/1dd4b05f93820742954b32d0d44f211007e0d06e5742927628
Successfully built polaris
Installing collected packages: urllib3, typing-extensions, six, jmespath, annotated-types, typing-inspection, python-dateutil, pydantic-core, pydantic, botocore, s3transfer, boto3, polaris
Successfully installed annotated-types-0.7.0 boto3-1.38.36 botocore-1.38.46 jmespath-1.0.1 polaris-1.0.0 pydantic-2.11.7 pydantic-core-2.33.2 python-dateutil-2.9.0.post0 s3transfer-0.13.0 six-1.17.0 typing-extensions-4.14.1 typing-inspection-0.4.1 urllib3-2.5.0

[notice] A new release of pip is available: 24.2 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
(venv_dist) ➜  Desktop which polaris
/Users/yong/Desktop/venv_dist/bin/polaris
(venv_dist) ➜  Desktop cat /Users/yong/Desktop/venv_dist/bin/polaris
#!/Users/yong/Desktop/venv_dist/bin/python3.13
# -*- coding: utf-8 -*-
import re
import sys
from cli.polaris_cli import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
(venv_dist) ➜  Desktop SCRIPT_DIR="dummy" polaris --help

input: polaris --help
options:
  polaris
    --host  hostname
    --port  port
    --base-url  complete base URL instead of hostname:port
    --client-id  client ID for token-based authentication
    --client-secret  client secret for token-based authentication
    --access-token  access token for token-based authentication
    --profile  profile for token-based authentication
    --proxy  proxy URL
...

Here is a quick demo on the updated bash script:

➜  polaris git:(polaris_cli_package) ./polaris
--> Performing first-time setup for the Python client...
--> Upgrading pip in virtual environment...
Requirement already satisfied: pip in ./polaris-venv/lib/python3.13/site-packages (24.2)
Collecting pip
  Using cached pip-25.1.1-py3-none-any.whl.metadata (3.6 kB)
Using cached pip-25.1.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.2
    Uninstalling pip-24.2:
      Successfully uninstalled pip-24.2
Successfully installed pip-25.1.1
--> Installing test requirements (including Poetry) into virtual environment...
Collecting poetry==2.1.3 (from -r ./regtests/requirements.txt (line 20))
  Using cached poetry-2.1.3-py3-none-any.whl.metadata (7.1 kB)
Collecting build<2.0.0,>=1.2.1 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached build-1.2.2.post1-py3-none-any.whl.metadata (6.5 kB)
Collecting cachecontrol<0.15.0,>=0.14.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached cachecontrol-0.14.3-py3-none-any.whl.metadata (3.1 kB)
Collecting cleo<3.0.0,>=2.1.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached cleo-2.1.0-py3-none-any.whl.metadata (12 kB)
Collecting dulwich<0.23.0,>=0.22.6 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached dulwich-0.22.8-cp313-cp313-macosx_11_0_arm64.whl.metadata (4.9 kB)
Collecting fastjsonschema<3.0.0,>=2.18.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached fastjsonschema-2.21.1-py3-none-any.whl.metadata (2.2 kB)
Collecting findpython<0.7.0,>=0.6.2 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached findpython-0.6.3-py3-none-any.whl.metadata (5.3 kB)
Collecting installer<0.8.0,>=0.7.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached installer-0.7.0-py3-none-any.whl.metadata (936 bytes)
Collecting keyring<26.0.0,>=25.1.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached keyring-25.6.0-py3-none-any.whl.metadata (20 kB)
Collecting packaging>=24.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached packaging-25.0-py3-none-any.whl.metadata (3.3 kB)
Collecting pbs-installer<2026.0.0,>=2025.1.6 (from pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached pbs_installer-2025.7.12-py3-none-any.whl.metadata (991 bytes)
Collecting pkginfo<2.0,>=1.12 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached pkginfo-1.12.1.2-py3-none-any.whl.metadata (13 kB)
Collecting platformdirs<5,>=3.0.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached platformdirs-4.3.8-py3-none-any.whl.metadata (12 kB)
Collecting poetry-core==2.1.3 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached poetry_core-2.1.3-py3-none-any.whl.metadata (3.5 kB)
Collecting pyproject-hooks<2.0.0,>=1.0.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached pyproject_hooks-1.2.0-py3-none-any.whl.metadata (1.3 kB)
Collecting requests<3.0,>=2.26 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached requests-2.32.4-py3-none-any.whl.metadata (4.9 kB)
Collecting requests-toolbelt<2.0.0,>=1.0.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)
Collecting shellingham<2.0,>=1.5 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting tomlkit<1.0.0,>=0.11.4 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached tomlkit-0.13.3-py3-none-any.whl.metadata (2.8 kB)
Collecting trove-classifiers>=2022.5.19 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached trove_classifiers-2025.5.9.12-py3-none-any.whl.metadata (2.3 kB)
Collecting virtualenv<21.0.0,>=20.26.6 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached virtualenv-20.31.2-py3-none-any.whl.metadata (4.5 kB)
Collecting xattr<2.0.0,>=1.0.0 (from poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached xattr-1.2.0-cp313-cp313-macosx_11_0_arm64.whl.metadata (3.8 kB)
Collecting msgpack<2.0.0,>=0.5.2 (from cachecontrol<0.15.0,>=0.14.0->cachecontrol[filecache]<0.15.0,>=0.14.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached msgpack-1.1.1-cp313-cp313-macosx_11_0_arm64.whl.metadata (8.4 kB)
Collecting filelock>=3.8.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)
Collecting crashtest<0.5.0,>=0.4.1 (from cleo<3.0.0,>=2.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached crashtest-0.4.1-py3-none-any.whl.metadata (1.1 kB)
Collecting rapidfuzz<4.0.0,>=3.0.0 (from cleo<3.0.0,>=2.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached rapidfuzz-3.13.0-cp313-cp313-macosx_11_0_arm64.whl.metadata (12 kB)
Collecting urllib3>=1.25 (from dulwich<0.23.0,>=0.22.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached urllib3-2.5.0-py3-none-any.whl.metadata (6.5 kB)
Collecting jaraco.classes (from keyring<26.0.0,>=25.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring<26.0.0,>=25.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached jaraco_functools-4.2.1-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring<26.0.0,>=25.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Collecting httpx<1,>=0.27.0 (from pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)
Collecting zstandard>=0.21.0 (from pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached zstandard-0.23.0-cp313-cp313-macosx_11_0_arm64.whl.metadata (3.0 kB)
Collecting anyio (from httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached anyio-4.9.0-py3-none-any.whl.metadata (4.7 kB)
Collecting certifi (from httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached certifi-2025.7.14-py3-none-any.whl.metadata (2.4 kB)
Collecting httpcore==1.* (from httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)
Collecting idna (from httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting h11>=0.16 (from httpcore==1.*->httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Collecting charset_normalizer<4,>=2 (from requests<3.0,>=2.26->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached charset_normalizer-3.4.2-cp313-cp313-macosx_10_13_universal2.whl.metadata (35 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv<21.0.0,>=20.26.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached distlib-0.3.9-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting cffi>=1.16.0 (from xattr<2.0.0,>=1.0.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached cffi-1.17.1-cp313-cp313-macosx_11_0_arm64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.16.0->xattr<2.0.0,>=1.0.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Collecting sniffio>=1.1 (from anyio->httpx<1,>=0.27.0->pbs-installer[download,install]<2026.0.0,>=2025.1.6->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting more-itertools (from jaraco.classes->keyring<26.0.0,>=25.1.0->poetry==2.1.3->-r ./regtests/requirements.txt (line 20))
  Using cached more_itertools-10.7.0-py3-none-any.whl.metadata (37 kB)
Using cached poetry-2.1.3-py3-none-any.whl (278 kB)
Using cached poetry_core-2.1.3-py3-none-any.whl (332 kB)
Using cached build-1.2.2.post1-py3-none-any.whl (22 kB)
Using cached cachecontrol-0.14.3-py3-none-any.whl (21 kB)
Using cached cleo-2.1.0-py3-none-any.whl (78 kB)
Using cached crashtest-0.4.1-py3-none-any.whl (7.6 kB)
Using cached dulwich-0.22.8-cp313-cp313-macosx_11_0_arm64.whl (915 kB)
Using cached fastjsonschema-2.21.1-py3-none-any.whl (23 kB)
Using cached findpython-0.6.3-py3-none-any.whl (20 kB)
Using cached installer-0.7.0-py3-none-any.whl (453 kB)
Using cached keyring-25.6.0-py3-none-any.whl (39 kB)
Using cached msgpack-1.1.1-cp313-cp313-macosx_11_0_arm64.whl (78 kB)
Using cached pbs_installer-2025.7.12-py3-none-any.whl (59 kB)
Using cached httpx-0.28.1-py3-none-any.whl (73 kB)
Using cached httpcore-1.0.9-py3-none-any.whl (78 kB)
Using cached pkginfo-1.12.1.2-py3-none-any.whl (32 kB)
Using cached platformdirs-4.3.8-py3-none-any.whl (18 kB)
Using cached pyproject_hooks-1.2.0-py3-none-any.whl (10 kB)
Using cached rapidfuzz-3.13.0-cp313-cp313-macosx_11_0_arm64.whl (1.4 MB)
Using cached requests-2.32.4-py3-none-any.whl (64 kB)
Using cached charset_normalizer-3.4.2-cp313-cp313-macosx_10_13_universal2.whl (199 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB)
Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Using cached tomlkit-0.13.3-py3-none-any.whl (38 kB)
Using cached urllib3-2.5.0-py3-none-any.whl (129 kB)
Using cached virtualenv-20.31.2-py3-none-any.whl (6.1 MB)
Using cached distlib-0.3.9-py2.py3-none-any.whl (468 kB)
Using cached filelock-3.18.0-py3-none-any.whl (16 kB)
Using cached xattr-1.2.0-cp313-cp313-macosx_11_0_arm64.whl (19 kB)
Using cached certifi-2025.7.14-py3-none-any.whl (162 kB)
Using cached cffi-1.17.1-cp313-cp313-macosx_11_0_arm64.whl (178 kB)
Using cached h11-0.16.0-py3-none-any.whl (37 kB)
Using cached packaging-25.0-py3-none-any.whl (66 kB)
Using cached trove_classifiers-2025.5.9.12-py3-none-any.whl (14 kB)
Using cached zstandard-0.23.0-cp313-cp313-macosx_11_0_arm64.whl (633 kB)
Using cached anyio-4.9.0-py3-none-any.whl (100 kB)
Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Using cached jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Using cached jaraco_functools-4.2.1-py3-none-any.whl (10 kB)
Using cached more_itertools-10.7.0-py3-none-any.whl (65 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Installing collected packages: trove-classifiers, fastjsonschema, distlib, zstandard, urllib3, tomlkit, sniffio, shellingham, rapidfuzz, pyproject-hooks, pycparser, poetry-core, platformdirs, pkginfo, pbs-installer, packaging, msgpack, more-itertools, jaraco.context, installer, idna, h11, filelock, crashtest, charset_normalizer, certifi, virtualenv, requests, jaraco.functools, jaraco.classes, httpcore, findpython, dulwich, cleo, cffi, build, anyio, xattr, requests-toolbelt, keyring, httpx, cachecontrol, poetry
Successfully installed anyio-4.9.0 build-1.2.2.post1 cachecontrol-0.14.3 certifi-2025.7.14 cffi-1.17.1 charset_normalizer-3.4.2 cleo-2.1.0 crashtest-0.4.1 distlib-0.3.9 dulwich-0.22.8 fastjsonschema-2.21.1 filelock-3.18.0 findpython-0.6.3 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 idna-3.10 installer-0.7.0 jaraco.classes-3.4.0 jaraco.context-6.0.1 jaraco.functools-4.2.1 keyring-25.6.0 more-itertools-10.7.0 msgpack-1.1.1 packaging-25.0 pbs-installer-2025.7.12 pkginfo-1.12.1.2 platformdirs-4.3.8 poetry-2.1.3 poetry-core-2.1.3 pycparser-2.22 pyproject-hooks-1.2.0 rapidfuzz-3.13.0 requests-2.32.4 requests-toolbelt-1.0.0 shellingham-1.5.4 sniffio-1.3.1 tomlkit-0.13.3 trove-classifiers-2025.5.9.12 urllib3-2.5.0 virtualenv-20.31.2 xattr-1.2.0 zstandard-0.23.0
--> Installing 'polaris' package using Poetry...
Updating dependencies
Resolving dependencies... (2.5s)

Package operations: 46 installs, 0 updates, 0 removals

  - Installing six (1.17.0)
  - Installing mdurl (0.1.2)
  - Installing jmespath (1.0.1)
  - Installing python-dateutil (2.9.0.post0)
  - Installing typing-extensions (4.14.1)
  - Installing pluggy (1.6.0)
  - Installing iniconfig (2.1.0)
  - Installing botocore (1.38.46)
  - Installing markdown-it-py (3.0.0)
  - Installing annotated-types (0.7.0)
  - Installing pydantic-core (2.33.2)
  - Installing pygments (2.19.2)
  - Installing typing-inspection (0.4.1)
  - Installing cachetools (5.5.2)
  - Installing cfgv (3.4.0)
  - Installing chardet (5.2.0)
  - Installing click (8.2.1)
  - Installing colorama (0.4.6)
  - Installing coverage (7.9.2)
  - Installing fsspec (2025.5.1)
  - Installing identify (2.6.12)
  - Installing mccabe (0.7.0)
  - Installing mmh3 (5.1.0)
  - Installing mypy-extensions (1.1.0)
  - Installing nodeenv (1.9.1)
  - Installing pathspec (0.12.1)
  - Installing pycodestyle (2.14.0)
  - Installing pydantic (2.11.7)
  - Installing pyflakes (3.4.0)
  - Installing pyparsing (3.2.3)
  - Installing pytest (8.4.1)
  - Installing pyproject-api (1.9.1)
  - Installing rich (13.9.4)
  - Installing s3transfer (0.13.0)
  - Installing sortedcontainers (2.4.0)
  - Installing pyyaml (6.0.2)
  - Installing strictyaml (1.7.3)
  - Installing tenacity (9.1.2)
  - Installing boto3 (1.38.36)
  - Installing pytest-cov (6.2.1)
  - Installing tox (4.27.0)
  - Installing types-python-dateutil (2.9.0.20250708)
  - Installing flake8 (7.3.0)
  - Installing mypy (1.16.1)
  - Installing pre-commit (4.2.0)
  - Installing pyiceberg (0.9.1)

Writing lock file

Installing the current project: polaris (1.0.0)
✅ Dependencies repaired.
--> Activating virtual environment for CLI execution: source ./polaris-venv/bin/activate
--> SCRIPT_DIR is set to: .
--> Running Polaris CLI...
Traceback (most recent call last):
  File "/Users/yong/Desktop/GitHome/polaris/polaris-venv/bin/polaris", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/yong/Desktop/GitHome/polaris/client/python/cli/polaris_cli.py", line 195, in main
    PolarisCli.execute()
    ~~~~~~~~~~~~~~~~~~^^
  File "/Users/yong/Desktop/GitHome/polaris/client/python/cli/polaris_cli.py", line 68, in execute
    client_builder = PolarisCli._get_client_builder(options)
  File "/Users/yong/Desktop/GitHome/polaris/client/python/cli/polaris_cli.py", line 151, in _get_client_builder
    raise Exception(
    ...<5 lines>...
    )
Exception: Please provide credentials via either --client-id & --client-secret or --access-token. Alternatively, you may set the environment variables CLIENT_ID & CLIENT_SECRET.

By doing so, we really don't need bash script polaris (at least the name is confusing with this change...instead, it should be client_setup.sh or something similar) and end-users can do regular pip install to setup Polaris CLI. To avoid too much change, I will keep the bash script polaris for now (this script is still useful for initial setup as well as refresh dependencies etc. but with proper Python Package Index, we should just do pip upgrade instead).

MonkeyCanCode · 2025-07-17T17:06:23Z

@eric-maynard / @HonahX may I have a review here? This will get us a proper packaging for polaris client as well as to help us to remove dependency on the bash script (and use it only for repair or initial setup)

HonahX · 2025-07-21T01:49:07Z

Thanks @MonkeyCanCode for working on this! This is a great step towards polaris client packaging and publication. One small question to confirm, at this stage users will still need to run ./gradlew regeneratePythonClient before pip install?

I once had an experimental PR to use poetry's custom build file to auto-generate the open api client code when building the package. But I did not have time to fix all the CI for that PR. Do you think that could be the next step of the python packaging?

MonkeyCanCode · 2025-07-21T01:53:39Z

Thanks @MonkeyCanCode for working on this! It's great to make python client a package. One small question to confirm, at this stage users will still need to run ./gradlew regeneratePythonClient before pip install?

I once had an experimental PR to use poetry's custom build file to auto-generate the open api client code when building the package. But I did not have time to fix all the CI for that PR. Do you think that could be the next step of the python packaging?

So the bash script polaris already does it. Also, I added those to the new Makefile PR for adding python client related operations in. But yes, if that is more preferred, I can update poetry to do so as opposed to using an external bash script for those.

But ideally, people should be using pip install Polaris from a python repo instead. From the changes introduced in this PR, those generated files will be included.

Let me know which route is more preferred. @HonahX

HonahX · 2025-07-21T02:10:42Z

But ideally, people should be using pip install Polaris from a python repo instead.
+1, this should be the right direction moving on.

I also chatted with Eric offline, and we both think it would be ideal to auto-generate those files as part of the packaging or installation, rather than relying on a separate script. Like you mentioned, the bash script isn’t very helpful for users who “just want to use the CLI” without having to download the entire codebase. Ideally, users should be able to pip install polaris with a plain clone of the repo, without running any one-time setup script first.

I think we can merge this first as it already made a huge step to set CLI as the entrypoint

* chore(deps): update dependency mypy to >=1.17, <=1.17.0 (apache#2114) * Spark 3.5.6 and Iceberg 1.9.1 (apache#1960) * Spark 3.5.6 and Iceberg 1.9.1 * Cleanup * Add `pathStyleAccess` to AwsStorageConfigInfo (apache#2012) * Add `pathStyleAccess` to AwsStorageConfigInfo This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). * Add TestFileIOFactory helper (apache#2105) * Add FileIOFactory.wrapExisting helper * fix(deps): update dependency gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext to v1.2 (apache#2125) * fix(deps): update dependency boto3 to v1.39.7 (apache#2124) * Abstract polaris-runtime-service tests for all persistence implementations (apache#2106) The NoSQL persistence implementation has to run the Iceberg table & view catalog plus the Polaris specific tests as well. Reusing existing tests is beneficial to avoid a lot of code duplcation. This change moves the actual tests to `Abstract*` classes and refactors the existing tests to extend those. The NoSQL persistence work extends the same `Abstract*` classes but runs with different Quarkus test profiles. * Add IMPLICIT authentication support to the CLI (apache#2121) PRs apache#1925 and apache#1912 were merged around the same time. This PR connects the two changes and enables the CLI to accept IMPLICIT authentication type. Since Hadoop federated catalogs rely purely on IMPLICIT authentication, the CLI parsing test has been updated to reflect the same. * feat(helm): Add support for external authentication (apache#2104) * fix(deps): update dependency org.apache.iceberg:iceberg-bom to v1.9.2 (apache#2126) * fix(deps): update quarkus platform and group to v3.24.4 (apache#2128) * fix(deps): update dependency boto3 to v1.39.8 (apache#2129) * fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.13.3 (apache#2130) * Add newIcebergCatalog helper (apache#2134) creation of `IcebergCatalog` instances was quite redundant as tests mostly use the same parameters most of the time. also remove an unused field in 2 other tests. * Add server and client support for the new generic table `baseLocation` field (apache#2122) * Use Makefile to simplify setup and commands (apache#2027) * Use Makefile to simplify setup and commands * Add targets for minikube state management * Add podman support and spark plugin build * Add version target * Update README.md for Makefile usage and relation to the project * Fix nit * Package polaris client as python package (apache#2049) * Package polaris client as python package * Package polaris client as python package * Change owner to spark when copying files from local into Dockerfile * CI: Address failure from accessing GH API (apache#2132) CI sometimes fails with this failure: ``` * What went wrong: Execution failed for task ':generatePomFileForMavenPublication'. > Unable to process url: https://api.github.com/repos/apache/polaris/contributors?per_page=1000 ``` The sometimes failing request fetches the list of contributors to be published in the "root" POM. Unauthorized GH API requests have an hourly(?) limit of 60 requests per source IP. Authorized requests have a much higher rate limit. We do have a GitHub token available in every CI run, which can be used in GH API requests. This change adds the `Authorization` header for the failing GH API request to leverage the higher rate limit and let CI not fail (that often). * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4 (apache#2139) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.0 (apache#2142) * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.4 (apache#2146) * fix(deps): update dependency org.xerial.snappy:snappy-java to v1.1.10.8 (apache#2138) * fix(deps): update dependency org.junit:junit-bom to v5.13.4 (apache#2147) * fix(deps): update dependency boto3 to v1.39.9 (apache#2137) * fix(deps): update dependency com.fasterxml.jackson:jackson-bom to v2.19.2 (apache#2136) * Python client: add support for endpoint, sts-endpoint, path-style-access (apache#2127) This change adds support for endpoint, sts-endpoint, path-style-access to the Polaris Python client. Amends apache#1913 and apache#2012 * Remove PolarisEntityManager.getCredentialCache (apache#2133) `PolarisEntityManager` itself is not using the `StorageCredentialCache` but just hands it out via `getCredentialCache`. the only caller of `getCredentialCache` is `FileIOUtil.refreshAccessConfig`, which in in turn is only called by `DefaultFileIOFactory` and `IcebergCatalog`. note that in a follow-up we will likely be able to remove `PolarisEntityManager` usage completely from `IcebergCatalog`. additional cleanups: - use `StorageCredentialCache` injection in tests (but we need to invalidate all entries on test start) - remove unused `UserSecretsManagerFactory` from `PolarisCallContextCatalogFactory` * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.22-1.1752676419 (apache#2150) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.1 (apache#2152) * fix(deps): update dependency boto3 to v1.39.10 (apache#2151) * chore: fix class reference in the javadoc of TableLikeEntity (apache#2157) * fix(deps): update dependency commons-codec:commons-codec to v1.19.0 (apache#2160) * fix(deps): update dependency boto3 to v1.39.11 (apache#2159) * Last merged commit 395459f --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Yong Zheng <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Pooja Nilangekar <[email protected]> Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: Yun Zou <[email protected]>

Package polaris client as python package

ff70ff5

github-project-automation bot added this to Basic Kanban Board Jul 14, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jul 14, 2025

MonkeyCanCode mentioned this pull request Jul 14, 2025

No client in binary distribution #2042

Open

MonkeyCanCode marked this pull request as draft July 14, 2025 23:16

Package polaris client as python package

c4c40ff

MonkeyCanCode marked this pull request as ready for review July 14, 2025 23:43

MonkeyCanCode requested review from HonahX and eric-maynard July 14, 2025 23:44

Change owner to spark when copying files from local into Dockerfile

0fb5ec1

HonahX approved these changes Jul 21, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 21, 2025

MonkeyCanCode merged commit 3650c68 into apache:main Jul 21, 2025
12 of 13 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jul 21, 2025

MonkeyCanCode mentioned this pull request Jul 28, 2025

Python client auto generate #2192

Merged

eric-maynard mentioned this pull request Aug 1, 2025

Fix CLI error polaris: not found #2228

Closed

MonkeyCanCode mentioned this pull request Aug 10, 2025

Perform force repair when polaris not found #2313

Merged

MonkeyCanCode mentioned this pull request Aug 22, 2025

Add support for poetry build with wheel #2425

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Package polaris client as python package #2049

Package polaris client as python package #2049

Uh oh!

MonkeyCanCode commented Jul 14, 2025 •

edited

Loading

Uh oh!

MonkeyCanCode commented Jul 17, 2025

Uh oh!

HonahX commented Jul 21, 2025 •

edited

Loading

Uh oh!

MonkeyCanCode commented Jul 21, 2025

Uh oh!

HonahX commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Package polaris client as python package #2049

Package polaris client as python package #2049

Uh oh!

Conversation

MonkeyCanCode commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MonkeyCanCode commented Jul 17, 2025

Uh oh!

HonahX commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MonkeyCanCode commented Jul 21, 2025

Uh oh!

HonahX commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MonkeyCanCode commented Jul 14, 2025 •

edited

Loading

HonahX commented Jul 21, 2025 •

edited

Loading