A key-value store API

[SuperSonicHub1]

I'm still working on my Mastodon client. In order to fully fulfill the Riffle philosophy, we must put as much application state in the database as humanly possible. I think the best way to go about this is to have a bunch of key-value stores that store the state of all my components. Having a KV store will also come in handy for an eventual settings menu. Because TinyBase is so well-designed, I was able to implement a reactive key-store on top of the library in a svelte 79 lines of code, including a unidirectional wrapper for Svelte.

Usage

export const store = createStore()
const kv = new TinyKV(store, 'settings')
kv.setValue('number', 5)
console.log(kv.getValue('number')) // 5

In Svelte (I plan to talk about Svelte interop more in my eventual post-mortem):

<script>
	export const store = createStore()
	const kv = new TinyKV(store, 'settings')
	kv.setValue('number', Math.random())
	const number = entry(kv, 'number')

	onMount(() => {
		setInterval(() => {
			kv.setValue('number', Math.random())
		}, 5000)
	})
</script>

<p>Reactive entry: {$number}</p>

Why?

My current implementation isn't super fleshed out, but you get a surprising amount for building on top of TinyBase:

• reactivity
• transactions and checkpoints
• persistence across multiple sources (great for settings menus on cross-platform apps)

Future Improvements

• properly implement transactions
• make my Svelte wrapper bidirectional
• mirror more methods of the Store interface

Implementation

License: https://unlicense.org/

/** A key-value store implemented in TinyBase. */

import { readable } from "svelte/store"
import type { Id } from "tinybase/common"
import type { Cell, GetCellChange, Store } from "tinybase/store"

const CELL_NAME: Id = 'value'


type EntryListener = (kv: TinyKV, key: Id, newValue: Cell, oldValue: Cell, getCellChange: GetCellChange | undefined) => void
type KeysListener = (kv: TinyKV) => void

export default class TinyKV {
	private store: Store
	private tableId: Id
 
	constructor(store: Store, tableId: Id) {
		this.store = store
		this.tableId = tableId
	}

	getStore() {
		return this.store
	}

	getTableId() {
		return this.tableId
	}

	getBackingTable() {
		return this.store.getTable(this.tableId)
	}

	getValue(key: Id) {
		return this.store.getCell(this.tableId, key, CELL_NAME)
	}

	setValue(key: Id, value: Cell) {
		return this.store.setCell(this.tableId, key, CELL_NAME, value)
	}

	getKeys() {
		return this.store.getRowIds(this.tableId)
	}

	getValues() {
		return Object.values(this.getBackingTable()).map(row => row.value)
	}

	getEntries(): [key: Id, value: Cell][] {
		return Object.entries(this.getBackingTable()).map(([key, row]) => [key, row.value])
	}

	addEntryListener(key: Id, listener: EntryListener, mutator?: boolean) {
		const cellListener = (store, tableId: Id, rowId: Id, cellId: Id, newCell: Cell, oldCell: Cell, getCellChange: GetCellChange | undefined,) => {
			queueMicrotask(() => listener(this, rowId, newCell, oldCell, getCellChange))
		}
		return this.store.addCellListener(this.tableId, key, CELL_NAME, cellListener, mutator)
	}

	addKeysListener(listener: KeysListener, mutator?: boolean) {
		const rowIdsListener = () => {
			queueMicrotask(() => listener(this))
		}
		return this.store.addRowIdsListener(this.tableId, rowIdsListener, mutator)
	}

	delListener(listenerId: string) {
		this.store.delListener(listenerId)
		return this
	}
}

export function entry (kv: TinyKV, key: Id) {
	return readable(kv.getValue(key), set => {
		const listenerId = kv.addEntryListener(key, (_kv, _key, newValue) => set(newValue))
		return () => kv.delListener(listenerId)
	})
}

[jamesgpearce]

Wow that's great! Would you propose something like this as a first class feature too?

[SuperSonicHub1]

Definitely! Having a KV store directly in TinyBase would make the implementation a lot simpler and make TinyBase's advanced features a lot more approachable to users who may feel that they don't need a relational model but still want easy reactivity. Being part of TinyBase would also open up new features that cannot be done with TinkKV's current architecture, like having a schema. Would you be interested in me sending a pull request?

[jamesgpearce]

The above is enough to get me thinking!

I am tempted to say we put it into the core Store interface, with basically a "special" table and row to hide it in that won't collide with user tables. The advantage of that is that all the persistence, schema, listening, crdt infra will work for free. (Versus having a special KV module that then needs all its own new stuff).

A very rare ascii control character or perhaps a symbol might do the trick for the table and row Ids. The api methods are then, as you show above, basically super thin aliases.

[jamesgpearce]

Design notes

Thought about this some more. Basically we want to be able to provide a key/value API that works a little bit like operating on a single row of a single table. Methods like setValue, getValue, addValueListener etc. This is useful for people who want a simple flat store, and who otherwise have to have empty table and row names to store what is essentially just a reactive map.

But there are three (at least) ways to do this, all with pros and cons. Things to consider:

• minimize breaking changes; eg a persisted v2 store should be able to start up in v3
• try and minimize massive renames or codemods
• persisters listen to cell changes
• persisters make assumptions about ‘initial’ content being tabular
• checkpoints listen to cell changes
• forthcoming synchronization (crdts) listen to cell changes
• store schemas are tabular
• references to each type of things in ui-react’s context
• reduce ‘magic’ like secret table names

Option A: Create new type of first-class store alongside the existing one

We create ‘KeyStore’ which has a more limited API than today’s ‘Store’. It just stores and listens to Id/Value pairs.

Pros:

• Clean & clear API
• Separate module, only install if needed

Cons:

• Would really need to rename today’s ‘Store’ to ‘Tabular’ store for symmetry; big codemod.
• Persisters, checkpoints, synchronizations, react-ui would ALL need to know about this new type of store. Elegant with interfaces like ‘Persistable’, ‘Checkpointable’ etc, but lots of breaking changes

Option B: Create new type of wrapper around a store

We create a ‘KeyStore’ wrapper (with a similar pattern to Metrics, Indexes etc) which takes a fully fledged Store as a creation argument: eg createKeyStore(myStore). A single table row in the tabular store is used as a reactive store for the key/value pairs.

Pros:

• Clean & clear API
• No need to rename existing tabular Store interface
• No need to change checkpoints, persistence, schema behavior, etc since that’s all handled by the underlying store.
• Simple to implement

Cons:

• Reveals that the underlying implementation is a tabular store! User has to remember to persist that directly. This feels yucky.
• Danger of ‘magic’ table and row Ids colliding with real ones in the store.
• The other ‘wrapper’ interfaces like Metrics, Queries, etc are generally read-based. This is a bit inconsistent since it’s read/write.

Option C: Existing Store knows how to handle key/value pairs, and just gains additional methods like setValue, getValue.

Pros:

• No need to rename existing Store interface
• “The one store that does everything!”

Cons:

• Less clean API since it’s woven in amongst the tabular API.
• Breaking changes: the store’s getJson and schema etc would need to expose the key/values concept alongside the tabular concept.
• changes required to persisters (visible to user) and checkpoints/sync (not visible to user)

--

B seems the least work but I worry about the leaky abstraction of having to explicitly assign a tabular store to instantiate it. My personal preference is option C because the changes and codemods are more surgical than A and the resulting API doesn’t leak. The only breaking change for users (I think) is that schemas will need to get another part to them (to schematize key values) and persisters will need to be able to take additional ‘initial’ content when loading.

[jamesgpearce]

In which case, the task list looks like this. Follow the keyvalues branch for progress.

• Enhance Store schema into two parts so that key-values can be schematized. Make backwards compatible where possible
• Test cases for the above
• Enhance JSON for store into two parts so that key-values can be serialized. Make backwards compatible where possible.
• Test cases for the above
• Create internal structure for key-values in the Store (rather than reusing tables)
• Create API for key-values in Store
• Test cases for the above
• Use addValueListener in checkpoints
• Test cases for the above
• Use addValueListener in persisters and sync
• Test cases for the above
• Consider reconciling getTablesSchema and getValuesSchema into a new getSchema
• Consider reconciling getTablesJson and getValuesJson into a new getJson
• Consider reconciling setTablesSchema and setValuesSchema into a new (backward compat) setSchema
• Consider reconciling setTablesJson and setValuesJson into a new (backward compat) setJson
• Document the new Store API
• React hooks
• React components
• Guides for key-value usage
• Update demos that have key-value-ish usage (like those that store view/route state)
• Look at where it says 'Cell value' in the docs to see if it needs rephrasing

[SuperSonicHub1]

Excellent! I would agree that C makes the least compromises. I'll see where I can pitch in.

[SuperSonicHub1]

It might also be a good idea to move all of this to an issue so that progress on the key store is more visible.

[jamesgpearce]

Please give [email protected] a quick try! https://github.com/tinyplex/tinybase/releases/tag/v3.0.0-beta.1

[SuperSonicHub1]

Just saw that you dropped v3.0! I sadly didn't get to test the beta due to being backlogged last weekend, but certainly will give it a shot today! Thanks for taking my idea to heart and making it a part of TinyBase. :)

[jamesgpearce]

Yep, just went live on https://tinybase.org/guides/releases/#v3-0 ... it was quite a journey to weave it in everywhere (persisters, checkpoints, react, tools etc), but I think it came out quite well. Let me know!