Metastore on sqlite #1877

YaroslavLitvinov · 2025-10-23T22:30:15Z

Partial implementation of Metastore on sqlite

Volumes, databases, schemas are migrated on SQLITE (no vfs used)
We use diesel orm to map mentioned ^^ entities to sql.
Tables remained on slatedb, at least temporary.
ExecutionService::abort_query turned into async function as now it waits for abort confirmation

Known issues

UI should support some minor API changes in responses. Also didn't fix openapi spec, yet.
Slatedb sql view we use in execution engine renamed to sqlite view, and maybe need some fixes.
List of failing tests, probably due to some minor bugs in new Metastore implementation. Fixing them is a matter of time.
tests::test_schemas
tests::service::test_max_concurrency_level
tests::service::test_query_recording
tests::sql::commands::show::query_show_databases
tests::sql::ddl::database::query_drop_database
tests::sql::ddl::database::query_drop_database_error_in_use
tests::test_generic_sqls::snowflake_generic::submit_ok_query_with_concurrent_limit
tests::test_rest_quick_sqls::snowflake_compatibility::abort_ok_query
tests::test_rest_quick_sqls::snowflake_compatibility::cancel_ok_query
tests::test_rest_quick_sqls::snowflake_compatibility::cancel_ok_sleeping_query

github-actions · 2025-10-23T22:30:57Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-10-23T23:08:23Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

… from metastor; make abort_query async; fix flake tests

github-actions · 2025-10-25T16:33:56Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-10-26T12:16:25Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-10-28T00:41:46Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

crates/core-metastore/src/sqlite/diesel_gen.rs

github-actions · 2025-10-29T13:55:32Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-10-30T00:21:32Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-01T02:50:29Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-02T02:39:07Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-03T01:42:27Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-04T04:16:35Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-05T03:26:08Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-05T20:05:07Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-06T17:22:20Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-06T17:23:44Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

rampage644 · 2025-11-06T17:35:22Z

crates/core-metastore/src/sqlite_metastore.rs

+///
+const KEY_TABLE: &str = "tbl";
+
+pub struct SlateDBMetastore {


Sqlite or SlateDB metastore?

Slatedb still used for tables

github-actions · 2025-11-06T17:43:34Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

crates/api-ui/src/volumes/models.rs

github-actions · 2025-11-07T12:18:29Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

github-actions · 2025-11-07T12:19:55Z

✅ SLT Targeted Testing: No relevant SLT tests found for the changes in this PR. No testing required.

YaroslavLitvinov added 3 commits October 23, 2025 23:14

added sqlite conn to metastore, metastore new changed from sync to async

94ebd97

Merge remote-tracking branch 'origin/main' into yaro/sqlite-metastore

ebc781e

add patchdog workflow

f551732

add workflow dispatch

0f10c4f

YaroslavLitvinov marked this pull request as draft October 24, 2025 13:50

YaroslavLitvinov added 3 commits October 24, 2025 20:37

stage before merging main

5914c47

Merge remote-tracking branch 'origin/main' into yaro/sqlite-metastore

0851f8b

metastore tests separated from implemetation; remove utoipa::ToSchema…

9aa26a3

… from metastor; make abort_query async; fix flake tests

YaroslavLitvinov added 2 commits October 25, 2025 22:29

metastore api simplify

ed61103

added ID to RwObject

76c45c8

YaroslavLitvinov added 2 commits October 27, 2025 18:15

broken, staged

dd7aed6

staged wip

5d3f7f4

merged

5305557

rampage644 reviewed Oct 29, 2025

View reviewed changes

crates/core-metastore/src/sqlite/diesel_gen.rs Outdated Show resolved Hide resolved

stage volumes basic ops ok

b588851

YaroslavLitvinov added 2 commits October 29, 2025 18:46

move from iter_volumes to get_volumes

51e493c

volume tests fixes

82f8e9a

wip

47b0224

make it compilable after databases support

145ea36

run test server in separate thread

5f2f634

YaroslavLitvinov added 3 commits November 3, 2025 13:27

staged

c793523

Merge remote-tracking branch 'origin/main' into yaro/sqlite-metastore

7776c57

wip

82fe602

YaroslavLitvinov added 2 commits November 5, 2025 00:00

before changes

c1433fa

wip

5877ce7

draft

797b957

YaroslavLitvinov added 5 commits November 6, 2025 03:52

some fixes, tests failing

2e52499

fix known metastore bugs

c8dce28

lint

f839468

fmt

e337143

merged with main

dbd70da

Merge remote-tracking branch 'origin' into yaro/sqlite-metastore

1bd415d

YaroslavLitvinov changed the title ~~WIP: DO NOT MERGE sqlite metastore~~ Metastore on sqlite Nov 6, 2025

YaroslavLitvinov marked this pull request as ready for review November 6, 2025 17:31

rampage644 reviewed Nov 6, 2025

View reviewed changes

couple of snapshots

ea97b3e

YaroslavLitvinov commented Nov 6, 2025

View reviewed changes

crates/api-ui/src/volumes/models.rs Show resolved Hide resolved

staged work, use ids for compile time checks

27774f7

YaroslavLitvinov marked this pull request as draft November 7, 2025 12:18

fmt

e328080

Metastore on sqlite #1877

Are you sure you want to change the base?

Metastore on sqlite #1877

Uh oh!

Conversation

YaroslavLitvinov commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Partial implementation of Metastore on sqlite

Known issues

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 25, 2025

Uh oh!

github-actions bot commented Oct 26, 2025

Uh oh!

github-actions bot commented Oct 28, 2025

Uh oh!

Uh oh!

github-actions bot commented Oct 29, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Nov 1, 2025

Uh oh!

github-actions bot commented Nov 2, 2025

Uh oh!

github-actions bot commented Nov 3, 2025

Uh oh!

github-actions bot commented Nov 4, 2025

Uh oh!

github-actions bot commented Nov 5, 2025

Uh oh!

github-actions bot commented Nov 5, 2025

Uh oh!

github-actions bot commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025

Uh oh!

rampage644 Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

YaroslavLitvinov Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 6, 2025

Uh oh!

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YaroslavLitvinov commented Oct 23, 2025 •

edited

Loading