[Discussion] Crates separation proposal

[rampage644]

I propose we split the crates with an idea of how it was previously before we merged most of them into single runtime.

Problems I would like to address primarily are compilation and thus developer velocity: currently it takes ~30s (at least) on my machine (M1 Max) to iterate on any change within runtime crate. Project codebase grows pretty quickly and compilation times gets worse very quickly.

Another problem to address is clearer dependency split. Example: nothing should depend on UI module and currently (due to Rust allowing cycle dependencies in a crate) there are suboptimal dependencies already. I would like to avoid that and keep dependency graph simple.

With that being said, here is the proposal (pseudo naming is used, feel free to suggest better naming):

• Have single "fat" main where all dependencies are constructed, set up and wired up together. This is the only crate/module that will have all other dependencies (with possible conditional compilation flags, i.e. "compile with ui disabled", etc)
• embucket-executor provides ExecutionService trait and CoreExecutor implementation, depends on Metastore
• embucket-metastore provides Metastore trait and `SlatedbMetastore implementation, leaf dependency (doesn't depend on other domain services and entities)
• embucket-http-ui provides axum Router, depends on possibly all/many services, serves UI
• embucket-http-v1 provides axum Router, depends on possibly all/many services, serves snowflake v1 REST API
• embucket-http-iceberg provides axum Router, depends on possibly all/many services, serves Iceberg REST API
• embucket-history, provides WorksheetsStore trait and SlateDBWorksheetsStore + RecordingExecutionService implementations, depends on dyn ExecutionService

There are other crates I'm less confident about (mostly cross application dependencies like iceberg spec, utils and such)

• embucket-datafusion-functions: does it make sense to also separate from original crate?
• embucket-datafusion-catalog: same, not sure
• embucket-utils: historically it provided slatedb related utility functions that were reused across entities storing data in slatedb
• anything else I forgot

[rampage644]

@Vedin @ravlio @osipovartem @DanCodedThis @YaroslavLitvinov

[Vedin]

I'd say it absolutely make sense to separate functions and catalog. A lot of work happens there so it makes no sense to rebuild all runtime every time.

[DanCodedThis]

RecordingExecutionService had a circular dependency with runtime when I tried implementing it in history. If we decouple the crates, maybe it will change.

@YaroslavLitvinov Proposed to make a trait for all our stores (history, auth, meta). I could see the utils crate becoming that store crate (which would make a lot of sense).

The other thing we should definitely look into is how to implement errors when we decouple the crates such that it wouldn't result in us converting them all the time (many soure: source: source: etc. or at least make it easy to convert)

Nitpick: if we rename the utils to store I would avoid using the word "store" in other crate names embucket-metastore -> embucket-meta (not the structs, just crate names) so to avoid confusion. Or go from the other side and add "store" to every crate (or even inside crate directory, ex. ui's auth-store)

Question: embucket-datafusion-catalog is it our default catalog (internal)?

embucket-http-v1 -> embucket-data-forge? (Or similar embucket-data-pipe, embucket-data-catalyst or playing into it embucket-sleet)

[osipovartem]

WDYT about removing crates dir
and make

• metastore crate
• execution dir
  • catalog crate (catalogs lists, catalogs, schema providers, etc.)
  • functions crate (all udf, udaf, udwf, etc.)
  • executor crate (service, query, session, visitors, etc.)
• http
  • UI crate
  • iceberg crate
  • v1 crate
• history crate
• utils crate

[rampage644]

WDYT about removing crates dir

Some directory is still needed to group all rust related code (since currently embucket is a monorepo), it can be renamed to rust/. And still crate(s) is quite a common pattern I see in other repos, though not all of them (datafusion repo follows what you suggested).

[rampage644]

Question: embucket-datafusion-catalog is it our default catalog (internal)?

embucket-http-v1 -> embucket-data-forge? (Or similar embucket-data-pipe, embucket-data-catalyst or playing into it embucket-sleet)

Yes, catalog is our internal catalog for query processing.
Naming looks interesting, though I would prefer them to be more descriptive and explicit.

[DanCodedThis]

what about (no crates folder):

• bucketd (main)
• runtime dir
  • execution crate
  • functions crate
  • catalog crate
• rest (http, perhaps services?) dir
  • ui crate
    • js (we can put ui js ts static inside the crate)
    • rust (include both the router and static file builder server)
  • iceberg crate
  • data-forge (v1) crate
• storage dir
  • meta crate
  • history crate
  • auth crate (can also be put inside the /rest/ui/rust/..., but it breaks the pattern I described a little)
  • store (utils) crate

[YaroslavLitvinov]

@YaroslavLitvinov Proposed to make a trait for all our stores (history, auth, meta). I could see the utils crate becoming that store crate (which would make a lot of sense).

We could combine all the stores implementing traits like: metastore, history, ... any other we add later.
into stores crate

And i would rename utils crate into db or embucket-db

[ravlio]

move functions to separate crate — sounds reasonable

[rampage644]

We could combine all the stores implementing traits like: metastore, history, ... any other we add later. into stores crate

I could see those traits combined in a single place with an idea that "everything that stores data in slatedb resides in a single crate", however I view this differently: everything that is required to provide some functionality stays together in a single crate.

For example, history crate will have dependency on store crate and thus it automatically depends on other entities it never interacts with (literally, all other services/traits/objects that aren't WorksheetStore).

I view crate separation based on domain boundaries: parts that evolve together or rely on each other conceptually (and semantically) should stay together.