You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Problem
While researching how to integrate some rust extensions into our python,
I heard about this alternative tool for code formatting that also does
linting. This tool is much faster than black and can also do codestyle
linting to enforce some best practices for python.
Ruff is implemented by [Astral](https://docs.astral.sh/), which is a
company aiming to create a "cargo for python" type experience with
python tools that are implemented in rust making them both fast and
reliable. I'm interested in some other stuff they are doing (e.g. uv for
dependency management), but this seemed like a lower stakes way to test
the waters with their stuff.
## Solution
Main changes are in:
- Dev dependency changes: add ruff, remove black
- Remove black configs in pyproject.toml.
- Add ruff configs to pyproject.toml. Mostly stuck with defaults,
although I disabled a few of the more annoying lint rules for now on a
per-file basis.
- Adjust CI to run ruff checks instead of black.
- Update CONTRIBUTING
The rest of this large diff is due to formatting and lint fixes for
various code style things.
These checks can be run manually with `poetry run ruff check --fix` and
`poetry run ruff format`, but otherwise they should automatically be
triggered by pre-commit hooks. I documented this in CONTRIBUTING.
## Type of Change
- [x] Infrastructure change (CI configs, etc)
## Test Plan
Tests should still be green. No functional impact expected.
[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client
23
-
as a library.
23
+
as a library.
24
24
25
-
A common need when making changes to the Pinecone client is to test your changes against existing Python code or Jupyter Notebooks that `pip install` the Pinecone Python client as a library.
25
+
A common need when making changes to the Pinecone client is to test your changes against existing Python code or Jupyter Notebooks that `pip install` the Pinecone Python client as a library.
26
26
27
-
Developers want to be able to see their changes to the library immediately reflected in their main application code, as well as to track all changes they make in git, so that they can be contributed back in the form of a pull request.
27
+
Developers want to be able to see their changes to the library immediately reflected in their main application code, as well as to track all changes they make in git, so that they can be contributed back in the form of a pull request.
28
28
29
-
The Pinecone Python client therefore supports Poetry as its primary means of enabling a consistent local development experience. This guide will walk you through the setup process so that you can:
29
+
The Pinecone Python client therefore supports Poetry as its primary means of enabling a consistent local development experience. This guide will walk you through the setup process so that you can:
30
30
1. Make local changes to the Pinecone Python client that are separated from your system's Python installation
31
31
2. Make local changes to the Pinecone Python client that are immediately reflected in other local code that imports the pinecone client
32
32
3. Track all your local changes to the Pinecone Python client so that you can contribute your fixes and feature additions back via GitHub pull requests
33
33
34
34
### Step 1. Fork the Pinecone python client repository
35
35
36
-
On the [GitHub repository page](https://github.com/pinecone-io/pinecone-python-client) page, click the fork button at the top of the screen and create a personal fork of the repository:
36
+
On the [GitHub repository page](https://github.com/pinecone-io/pinecone-python-client) page, click the fork button at the top of the screen and create a personal fork of the repository:
37
37
38
38
39
39
40
-
It will take a few seconds for your fork to be ready. When it's ready, **clone your fork** of the Pinecone python client repository to your machine.
40
+
It will take a few seconds for your fork to be ready. When it's ready, **clone your fork** of the Pinecone python client repository to your machine.
41
41
42
-
Change directory into the repository, as we'll be setting up a virtualenv from within the root of the repository.
42
+
Change directory into the repository, as we'll be setting up a virtualenv from within the root of the repository.
43
43
44
-
### Step 1. Install Poetry
44
+
### Step 1. Install Poetry
45
45
46
-
Visit [the Poetry site](https://python-poetry.org/) for installation instructions.
46
+
Visit [the Poetry site](https://python-poetry.org/) for installation instructions.
47
47
48
-
### Step 2. Install dependencies
48
+
### Step 2. Install dependencies
49
49
50
-
Run `poetry install` from the root of the project.
50
+
Run `poetry install` from the root of the project.
51
51
52
52
### Step 3. Activate the Poetry virtual environment and verify success
53
53
54
-
Run `poetry shell` from the root of the project. At this point, you now have a virtualenv set up in this directory, which you can verify by running:
54
+
Run `poetry shell` from the root of the project. At this point, you now have a virtualenv set up in this directory, which you can verify by running:
55
55
56
56
`poetry env info`
57
57
58
-
You should see something similar to the following output:
58
+
You should see something similar to the following output:
If you want to extract only the path to your new virtualenv, you can run `poetry env info --path`
75
75
76
-
##Loading your virtualenv in another shell
76
+
### Step 4. Enable pre-commit hooks.
77
77
78
-
It's a common need when developing against this client to load it as part of some other application or Jupyter Notebook code, modify
79
-
it directly, see your changes reflected immediately and also have your changes tracked in git so you can contribute them back.
78
+
Run `poetry run pre-commit install` to enable checks to run when you commit so you don't have to find out during your CI run that minor lint issues need to be addressed.
80
79
81
-
It's important to understand that, by default, if you open a new shell or terminal window, or, for example, a new pane in a tmux session,
82
-
your new shell will not yet reference the new virtualenv you created in the previous step.
80
+
## Common tasks
81
+
82
+
### Running tests
83
+
84
+
- Unit tests: `make test-unit`
85
+
- Integration tests: `PINECONE_API_KEY="YOUR API KEY" make test-integration`
86
+
- Run the tests in a single file: `poetry run pytest tests/unit/data/test_bulk_import.py -s -vv`
87
+
88
+
### Running the ruff linter / formatter
89
+
90
+
These should automatically trigger if you have enabled pre-commit hooks with `poetry run pre-commit install`. But in case you want to trigger these yourself, you can run them like this:
91
+
92
+
```
93
+
poetry run ruff check --fix # lint rules
94
+
poetry run ruff format # formatting
95
+
```
96
+
97
+
If you want to adjust the behavior of ruff, configurations are in `pyproject.toml`.
98
+
99
+
100
+
### Consuming API version upgrades
101
+
102
+
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
103
+
104
+
Prerequisites:
105
+
- You must be an employee with access to private Pinecone repositories
106
+
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
107
+
- You must have initialized the git submodules under codegen
108
+
109
+
```sh
110
+
git submodule
111
+
```
112
+
113
+
To regenerate the generated portions of the client with the latest version of the API specifications, you need to have Docker Desktop running on your local machine.
114
+
115
+
```sh
116
+
./codegen/
117
+
```
118
+
119
+
120
+
## Loading your virtualenv in another shell
121
+
122
+
It's a common need when developing against this client to load it as part of some other application or Jupyter Notebook code, modify
123
+
it directly, see your changes reflected immediately and also have your changes tracked in git so you can contribute them back.
124
+
125
+
It's important to understand that, by default, if you open a new shell or terminal window, or, for example, a new pane in a tmux session,
126
+
your new shell will not yet reference the new virtualenv you created in the previous step.
83
127
84
128
### Step 1. Get the path to your virtualenv
85
129
86
-
We're going to first get the path to the virtualenv we just created, by running:
130
+
We're going to first get the path to the virtualenv we just created, by running:
87
131
88
132
```bash
89
133
poetry env info --path
@@ -93,75 +137,52 @@ You'll get a path similar to this one: `/home/youruser/.cache/pypoetry/virtuale
93
137
94
138
### Step 2. Load your existing virtualenv in your new shell
95
139
96
-
Within this path is a shell script that lives at `<your-virtualenv-path>/bin/activate`. Importantly, you cannot simply run this script, but you
97
-
must instead source it like so:
140
+
Within this path is a shell script that lives at `<your-virtualenv-path>/bin/activate`. Importantly, you cannot simply run this script, but you
In the above example, ensure you're using your own virtualenv path as returned by `poetry env info --path`.
103
147
104
-
### Step 3. Test out your virtualenv
148
+
### Step 3. Test out your virtualenv
105
149
106
-
Now, we can test that our virtualenv is working properly by adding a new test module and function to the `pinecone` client within our virtualenv
107
-
and running it from the second shell.
150
+
Now, we can test that our virtualenv is working properly by adding a new test module and function to the `pinecone` client within our virtualenv
151
+
and running it from the second shell.
108
152
109
153
#### Create a new test file in pinecone-python-client
110
-
In the root of your working directory of the `pinecone-python-client` where you first ran `poetry shell`, add a new file named `hello_virtualenv.py` under the `pinecone` folder.
154
+
In the root of your working directory of the `pinecone-python-client` where you first ran `poetry shell`, add a new file named `hello_virtualenv.py` under the `pinecone` folder.
111
155
112
-
In that file write the following:
156
+
In that file write the following:
113
157
114
158
```python
115
159
defhello():
116
160
print("Hello, from your virtualenv!")
117
161
```
118
-
Save the file.
162
+
Save the file.
119
163
120
-
#### Create a new test file in your second shell
121
-
This step demonstrates how you can immediately test your latest Pinecone client code from any local Python application or Jupyter Notebook:
164
+
#### Create a new test file in your second shell
165
+
This step demonstrates how you can immediately test your latest Pinecone client code from any local Python application or Jupyter Notebook:
122
166
123
-
In your second shell, where you ran `source` to load your virtualenv, create a python file named `test.py` and write the following:
167
+
In your second shell, where you ran `source` to load your virtualenv, create a python file named `test.py` and write the following:
124
168
125
169
```python
126
170
from pinecone import hello_virtualenv
127
171
128
172
hello_virtualenv.hello()
129
173
```
130
174
131
-
Save the file. Run it with your Python binary. Depending on your system, this may either be `python` or `python3`:
175
+
Save the file. Run it with your Python binary. Depending on your system, this may either be `python` or `python3`:
132
176
133
177
```bash
134
178
python3 test.py
135
179
```
136
180
137
-
You should see the following output:
181
+
You should see the following output:
138
182
139
183
```bash
140
184
❯ python3 test.py
141
185
Hello, from your virtualenv!
142
186
```
143
187
144
188
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
145
-
146
-
147
-
## Consuming API version upgrades
148
-
149
-
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
150
-
151
-
Prerequisites:
152
-
- You must be an employee with access to private Pinecone repositories
153
-
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
154
-
- You must have initialized the git submodules under codegen
155
-
156
-
```sh
157
-
git submodule
158
-
```
159
-
160
-
161
-
To regenerate the generated portions of the client with the latest version of the API specifications, you need to have Docker Desktop running on your local machine.
0 commit comments