Skip to content

Commit e60590d

Browse files
vincentsaragovincentsarago
committed
Merge branch 'main' of https://github.com/developmentseed/eoapi-devseed into feature/add-stac-HTML-output
2 parents a9287e7 + 767c4d1 commit e60590d

File tree

24 files changed

+1346
-207
lines changed

24 files changed

+1346
-207
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 58 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ on:
1313
- '.github/workflows/ci.yml'
1414
- '.pre-commit-config.yaml'
1515
pull_request:
16+
workflow_dispatch:
17+
release:
18+
types: [published]
1619

1720
jobs:
1821
tests:
@@ -21,32 +24,23 @@ jobs:
2124
fail-fast: false
2225
steps:
2326
- uses: actions/checkout@v4
24-
- name: Set up Python
25-
uses: actions/setup-python@v5
26-
with:
27-
python-version: '3.12'
28-
27+
- uses: astral-sh/setup-uv@v5
28+
- name: Install
29+
run: uv sync
2930
- name: Run pre-commit
30-
run: |
31-
python -m pip install --upgrade pip
32-
python -m pip install pre-commit
33-
pre-commit run --all-files
31+
run: uv run pre-commit run --all-files
3432

3533
- name: Launch services
3634
run: docker compose up -d
3735

3836
- name: install lib postgres
3937
uses: nyurik/action-setup-postgis@v2
4038

41-
- name: Install python dependencies
42-
run: |
43-
python -m pip install pytest httpx pypgstac==0.9.2 psycopg[pool]
44-
4539
- name: Ingest Stac Items/Collection
4640
run: |
47-
pypgstac pgready --dsn postgresql://username:[email protected]:5439/postgis
48-
pypgstac load collections .github/workflows/data/noaa-emergency-response.json --dsn postgresql://username:[email protected]:5439/postgis --method insert_ignore
49-
pypgstac load items .github/workflows/data/noaa-eri-nashville2020.json --dsn postgresql://username:[email protected]:5439/postgis --method insert_ignore
41+
uv run pypgstac pgready --dsn postgresql://username:[email protected]:5439/postgis
42+
uv run pypgstac load collections .github/workflows/data/noaa-emergency-response.json --dsn postgresql://username:[email protected]:5439/postgis --method insert_ignore
43+
uv run pypgstac load items .github/workflows/data/noaa-eri-nashville2020.json --dsn postgresql://username:[email protected]:5439/postgis --method insert_ignore
5044
psql postgresql://username:[email protected]:5439/postgis -f .github/workflows/data/my_data.sql
5145
5246
# see https://github.com/developmentseed/tipg/issues/37
@@ -59,7 +53,54 @@ jobs:
5953
shell: bash
6054

6155
- name: Integrations tests
62-
run: python -m pytest .github/workflows/tests/
56+
run: uv run pytest .github/workflows/tests/
6357

6458
- name: Stop services
6559
run: docker compose stop
60+
61+
deploy:
62+
name: Deploy
63+
environment: dev
64+
needs: [tests]
65+
permissions:
66+
id-token: write
67+
contents: read
68+
runs-on: ubuntu-latest
69+
if: github.event_name == 'release' || github.event_name == 'workflow_dispatch'
70+
env:
71+
STACK_NAME: eoapi-dev
72+
73+
steps:
74+
75+
- uses: actions/checkout@v4
76+
77+
- name: Create config file
78+
run: |
79+
echo "${{ vars.CONFIG_YAML }}" > config.yaml
80+
81+
- name: Configure AWS credentials
82+
uses: aws-actions/configure-aws-credentials@v4
83+
with:
84+
role-to-assume: arn:aws:iam::390960605471:role/eoapi-devseed
85+
role-session-name: eoapi-devseed
86+
aws-region: us-west-2
87+
88+
- name: Set up node
89+
uses: actions/setup-node@v4
90+
with:
91+
node-version: 20
92+
93+
- name: Install uv
94+
uses: astral-sh/setup-uv@v5
95+
96+
- name: Install dependencies
97+
run: |
98+
uv sync --only-group deploy
99+
uv run --only-group deploy npm install
100+
101+
- name: CDK Synth
102+
run: uv run --only-group deploy npx cdk synth --all
103+
104+
- name: CDK Deploy
105+
run: |
106+
uv run --only-group deploy npx cdk deploy --all --require-approval never

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ repos:
77
args: ["-m", "3","--trailing-comma", "-l", "88"]
88

99
- repo: https://github.com/astral-sh/ruff-pre-commit
10-
rev: v0.4.4
10+
rev: v0.9.4
1111
hooks:
1212
- id: ruff
1313
args: ["--fix"]

‎.python-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.12

‎README.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,9 @@ Once the applications are *up*, you'll need to add STAC **Collections** and **It
6464

6565
Then you can start exploring your dataset with:
6666

67-
- the STAC Metadata service [http://localhost:8081](http://localhost:8081)
68-
- the Raster service [http://localhost:8082](http://localhost:8082)
69-
- the browser UI [http://localhost:8085](http://localhost:8085)
67+
- the STAC Metadata service [http://localhost:8081](http://localhost:8081)
68+
- the Raster service [http://localhost:8082](http://localhost:8082)
69+
- the browser UI [http://localhost:8085](http://localhost:8085)
7070

7171
If you've added a vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083).
7272

@@ -85,31 +85,32 @@ If you've added a vector dataset to the `public` schema in the Postgres database
8585
Install python dependencies with
8686

8787
```
88-
python -m venv .venv
89-
source .venv/bin/activate
90-
python -m pip install -r requirements.txt
88+
uv sync --group deploy
9189
```
9290

91+
> [!NOTE]
92+
> [install `uv`](https://docs.astral.sh/uv/getting-started/installation/#installing-uv)
93+
9394
And node dependencies with
9495

9596
```
96-
npm install
97+
uv run npm install
9798
```
9899

99100
Verify that the `cdk` CLI is available. Since `aws-cdk` is installed as a local dependency, you can use the `npx` node package runner tool, that comes with `npm`.
100101

101102
```
102-
npx cdk --version
103+
uv run npx cdk --version
103104
```
104105

105106
First, synthesize the app
106107

107108
```
108-
npx cdk synth --all
109+
uv run npx cdk synth --all
109110
```
110111

111112
Then, deploy
112113

113114
```
114-
npx cdk deploy --all --require-approval never
115+
uv run npx cdk deploy --all --require-approval never
115116
```

‎docker-compose.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,11 @@ services:
3535
- POSTGRES_PORT=5432
3636
- DB_MIN_CONN_SIZE=1
3737
- DB_MAX_CONN_SIZE=1
38-
# - EOAPI_STAC_TITILER_ENDPOINT=raster
39-
- EOAPI_STAC_TITILER_ENDPOINT=http://127.0.0.1:8082
38+
# - TITILER_ENDPOINT=raster
39+
- TITILER_ENDPOINT=http://127.0.0.1:8082
4040
# PgSTAC extensions
41-
# - EOAPI_STAC_EXTENSIONS=["filter", "query", "sort", "fields", "pagination", "titiler", "transaction"] # defaults
42-
# - EOAPI_STAC_CORS_METHODS='GET,POST,PUT,OPTIONS'
41+
# - EXTENSIONS=["filter", "query", "sort", "fields", "pagination", "titiler", "transaction"] # defaults
42+
# - CORS_METHODS='GET,POST,PUT,OPTIONS'
4343
env_file:
4444
- path: .env
4545
required: false
@@ -123,7 +123,7 @@ services:
123123
- POSTGRES_PORT=5432
124124
- DB_MIN_CONN_SIZE=1
125125
- DB_MAX_CONN_SIZE=10
126-
- EOAPI_VECTOR_DEBUG=TRUE
126+
- DEBUG=TRUE
127127
env_file:
128128
- path: .env
129129
required: false

‎infrastructure/app.py‎

Lines changed: 17 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ def __init__(
104104
self,
105105
"pgstac-db",
106106
vpc=vpc,
107+
add_pgbouncer=True,
107108
engine=aws_rds.DatabaseInstanceEngine.postgres(
108109
version=aws_rds.PostgresEngineVersion.VER_14
109110
),
@@ -121,34 +122,25 @@ def __init__(
121122
"context": True,
122123
"mosaic_index": True,
123124
},
125+
pgstac_version="0.9.2",
126+
)
127+
128+
# allow connections from any ipv4 to pgbouncer instance security group
129+
assert pgstac_db.security_group
130+
pgstac_db.security_group.add_ingress_rule(
131+
aws_ec2.Peer.any_ipv4(), aws_ec2.Port.tcp(5432)
124132
)
125-
pgstac_db.db.connections.allow_default_port_from_any_ipv4()
126133

127134
#######################################################################
128135
# Raster service
129136
raster = TitilerPgstacApiLambda(
130137
self,
131138
"raster-api",
132139
api_env={
133-
"EOAPI_RASTER_NAME": app_config.build_service_name("raster"),
140+
"NAME": app_config.build_service_name("raster"),
134141
"description": f"{app_config.stage} Raster API",
135-
"POSTGRES_HOST": pgstac_db.pgstac_secret.secret_value_from_json(
136-
"host"
137-
).to_string(),
138-
"POSTGRES_DBNAME": pgstac_db.pgstac_secret.secret_value_from_json(
139-
"dbname"
140-
).to_string(),
141-
"POSTGRES_USER": pgstac_db.pgstac_secret.secret_value_from_json(
142-
"username"
143-
).to_string(),
144-
"POSTGRES_PASS": pgstac_db.pgstac_secret.secret_value_from_json(
145-
"password"
146-
).to_string(),
147-
"POSTGRES_PORT": pgstac_db.pgstac_secret.secret_value_from_json(
148-
"port"
149-
).to_string(),
150142
},
151-
db=pgstac_db.db,
143+
db=pgstac_db.connection_target,
152144
db_secret=pgstac_db.pgstac_secret,
153145
# If the db is not in the public subnet then we need to put
154146
# the lambda within the VPC
@@ -193,30 +185,12 @@ def __init__(
193185
self,
194186
"stac-api",
195187
api_env={
196-
"EOAPI_STAC_NAME": app_config.build_service_name("stac"),
188+
"NAME": app_config.build_service_name("stac"),
197189
"description": f"{app_config.stage} STAC API",
198-
"POSTGRES_HOST_READER": pgstac_db.pgstac_secret.secret_value_from_json(
199-
"host"
200-
).to_string(),
201-
"POSTGRES_HOST_WRITER": pgstac_db.pgstac_secret.secret_value_from_json(
202-
"host"
203-
).to_string(),
204-
"POSTGRES_DBNAME": pgstac_db.pgstac_secret.secret_value_from_json(
205-
"dbname"
206-
).to_string(),
207-
"POSTGRES_USER": pgstac_db.pgstac_secret.secret_value_from_json(
208-
"username"
209-
).to_string(),
210-
"POSTGRES_PASS": pgstac_db.pgstac_secret.secret_value_from_json(
211-
"password"
212-
).to_string(),
213-
"POSTGRES_PORT": pgstac_db.pgstac_secret.secret_value_from_json(
214-
"port"
215-
).to_string(),
216-
"EOAPI_STAC_TITILER_ENDPOINT": raster.url.strip("/"),
217-
"EOAPI_STAC_EXTENSIONS": '["filter", "query", "sort", "fields", "pagination", "titiler"]',
190+
"TITILER_ENDPOINT": raster.url.strip("/"),
191+
"EXTENSIONS": '["filter", "query", "sort", "fields", "pagination", "titiler", "collection_search", "free_text"]',
218192
},
219-
db=pgstac_db.db,
193+
db=pgstac_db.connection_target,
220194
db_secret=pgstac_db.pgstac_secret,
221195
# If the db is not in the public subnet then we need to put
222196
# the lambda within the VPC
@@ -259,26 +233,11 @@ def __init__(
259233
TiPgApiLambda(
260234
self,
261235
"vector-api",
262-
db=pgstac_db.db,
236+
db=pgstac_db.connection_target,
263237
db_secret=pgstac_db.pgstac_secret,
264238
api_env={
265-
"EOAPI_VECTOR_NAME": app_config.build_service_name("vector"),
239+
"NAME": app_config.build_service_name("vector"),
266240
"description": f"{app_config.stage} tipg API",
267-
"POSTGRES_HOST": pgstac_db.pgstac_secret.secret_value_from_json(
268-
"host"
269-
).to_string(),
270-
"POSTGRES_DBNAME": pgstac_db.pgstac_secret.secret_value_from_json(
271-
"dbname"
272-
).to_string(),
273-
"POSTGRES_USER": pgstac_db.pgstac_secret.secret_value_from_json(
274-
"username"
275-
).to_string(),
276-
"POSTGRES_PASS": pgstac_db.pgstac_secret.secret_value_from_json(
277-
"password"
278-
).to_string(),
279-
"POSTGRES_PORT": pgstac_db.pgstac_secret.secret_value_from_json(
280-
"port"
281-
).to_string(),
282241
},
283242
# If the db is not in the public subnet then we need to put
284243
# the lambda within the VPC
@@ -342,7 +301,7 @@ def __init__(
342301
stage=app_config.stage,
343302
data_access_role=data_access_role,
344303
stac_db_secret=pgstac_db.pgstac_secret,
345-
stac_db_security_group=pgstac_db.db.connections.security_groups[0],
304+
stac_db_security_group=pgstac_db.security_group,
346305
# If the db is not in the public subnet then we need to put
347306
# the lambda within the VPC
348307
vpc=vpc if not app_config.public_db_subnet else None,

‎infrastructure/handlers/raster_handler.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,23 @@
55
import os
66

77
from eoapi.raster.app import app
8+
from eoapi.raster.config import ApiSettings
89
from mangum import Mangum
910
from titiler.pgstac.db import connect_to_db
1011

1112
logging.getLogger("mangum.lifespan").setLevel(logging.ERROR)
1213
logging.getLogger("mangum.http").setLevel(logging.ERROR)
1314

15+
settings = ApiSettings()
16+
1417

1518
@app.on_event("startup")
1619
async def startup_event() -> None:
1720
"""Connect to database on startup."""
18-
await connect_to_db(app)
21+
await connect_to_db(
22+
app,
23+
settings=settings.load_postgres_settings(),
24+
)
1925

2026

2127
handler = Mangum(app, lifespan="off")

‎infrastructure/handlers/vector_handler.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,15 @@
55
import os
66

77
from eoapi.vector.app import app
8+
from eoapi.vector.config import ApiSettings
89
from mangum import Mangum
910
from tipg.collections import register_collection_catalog
1011
from tipg.database import connect_to_db
11-
from tipg.settings import PostgresSettings
1212

1313
logging.getLogger("mangum.lifespan").setLevel(logging.ERROR)
1414
logging.getLogger("mangum.http").setLevel(logging.ERROR)
1515

16-
postgres_settings = PostgresSettings()
16+
settings = ApiSettings()
1717

1818
try:
1919
from importlib.resources import files as resources_files # type: ignore
@@ -31,7 +31,7 @@ async def startup_event() -> None:
3131
"""Connect to database on startup."""
3232
await connect_to_db(
3333
app,
34-
settings=postgres_settings,
34+
settings=settings.load_postgres_settings(),
3535
# We enable both pgstac and public schemas (pgstac will be used by custom functions)
3636
schemas=["pgstac", "public"],
3737
user_sql_files=sql_files,

‎package-lock.json‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)