Defining a JSON-based incremental upgrade path for schemas and associated instance docs

[brettz9]

Another topic related to IndexedDB (as in issue #17) and perhaps leveraging the same syntax proposed in #15 ...

Although versioning of schemas may not be of large consequence whenever server-side databases are in use--since an upgrade can in many cases be forced at once on all new visitors, with client-side IndexedDB in particular (though also for server-side databases being interacted with by Ajax and maintaining separate stores for a given user) users may need to be allowed to continue operating with an old version of the database, but when the version change can occur, there needs to be a safe migration path (even potentially needing to go through multiple schema upgrades if the user is making changes to the database offline long before visiting the site online again).

IndexedDB has an upgradeneeded event which can be leveraged for such migrations (and service workers could be used to grab the latest upgrades without the user needing to load a new page or refresh the old one), but it would be handy for the IndexedDB-friendly JSON Schema (proposed in issue #17 ) to also have a formal JSON definition for expressing diffs between schemas (even if it would not be able to have the robustness of all potential programmatic changes such as changing individual records between versions) and in a way which would also cause changes in the instance documents.

For example, one might wish to indicate that for version 2 of a schema, such-and-such a store should be added and an object modified, while for version 3, one store should be deleted, one schema object should be renamed, and one object should be moved elsewhere within the schema (and data also migrated--at least when "move" and "copy" operations are used on the schema diffs).

[awwright]

This has been discussed before, I've even written up a draft and a vocabulary for how you could implement such a thing (JSON Transform, I believe I called it).

But the problem is most upgrade operations aren't simple things like renaming, but more frequently things like pre-computing indexes or splitting apart a string. I've sort of resigned to using a full scripting language for doing this sort of stuff, otherwise we're just re-inventing XSLT.

I'm not opposed to this as such, but this is outside the scope of JSON Schema validation, which by definition is strictly structural, meaning a validation doesn't depend on the state of any other documents -- you need only the schema and the instance to determine valid or invalid.

There's been proposals for keywords that would depend on external values, like "ensure this integer refers to a valid user id", but that would be better suited for a separate meta-schema (like how we have JSON Hyper-Schema, in addition to JSON Schema validation). I don't know what we would call it... verification, referential consistency, ...?

[brettz9]

As far as the JSON syntax, it's pretty funny--I actually have been working on an "XSLT for JSON" (very alpha and--at least for now--with JavaScript-based templates rather than JSON ones). I even thought about retrieving IndexedDB databases and stores (like XSLT's document()) but was tunnel-visioned to only thinking of template usage, and it didn't occur to me to use for versioning, so thank you for the aha moment. I see how it would be necessary to overcome limitations otherwise requiring a programmatic approach (including other nuanced behaviors like copying from within one store into another store).

As far as proposals for keywords depending on external values, a separate meta-schema would be fine with me. "Referential consistency" sounds like a very good name choice to me.

(Incidentally, the readOnly instance property from Hyper Schema (which oddly shows in the changelog as having been removed despite having its own section in the "latest" version) seemed to me like one that might be better placed in a more general meta-schema. Although IndexedDB only has readonly transactions, and not readonly stores, it does seem to fit along with the types of suggestions raised in issue #17 , at least in the sense of its exact manner of enforcement being out of scope of the spec.)

[handrews]

@brettz9 it sounds like this is a good idea that is, as noted, a bit outside the scope of JSON Schema proper.

Is there any action we need to take here within the project? If not could you please close this?

[brettz9]

Well, is there any place for tracking suggestions to be added to different meta-schemas?

[handrews]

The validation and hyper-schema meta-schemas (schema.json and hyper-schema.json in this repo) would be tracked here. Any other meta-schemas would be tracked wherever those meta-schemas live.

[awwright]

I think this is outside the scope of JSON Schema right now, so I'll close this out. However, it's certainly something of interest.

If you have a particular solution for doing this that uses JSON Schema, I'd love to see it.

[naught101]

I understand why this is out of scope, but I wonder if it might be something worth addressing in the docs. For example, a "recommended versioning/upgrade pathway methodology" or similar.

I'm trying to implement something like this in a private project, and thinking of using something along the lines of:

• A single repository of schemas and upgrade code that can be used stand-alone or as a library in multiple languages
• Schema versions for each object type in subdirectories named after the version.
• A current_version file that keeps track of the current stable version of the schema.
• A previousVersion tag in each version of the schema (e.g. v1.1.0 might have previousVersion: 1.0.7)
• A requirement in each schema that all objects have a schemaVersion property.
• Upgrade function for each version change that can be imported into my apps, with a wrapper script for stand alone operation, in each language I care about (e.g. upgrade_v1.1.0_from_v.1.0.7.py, upgrade_v1.1.0_from_v.1.0.7.js), which handle the upgrade, and which can be chained (with pre-and post-validation).
• Some master scripts which detect the current stable schema version, object schema version, and get and run the upgrade function chain.

This feels like a sensible idea, because it keeps all of the validation and upgrade logic in the same place, and allows each part to be used as-needed in each app (in particular we have a large object store that will have objects with various different versions that need upgrading as they are accessed and used by other apps in the system).

I thought I'd chuck this up here because it might be useful as a basis for a more sensible methodology recommendation, and for the possibility that I might get some feedback on it :)

[handrews]

There's a difference between a specification and "docs." The spec should really be pretty minimalist. This sort of idea should go on the web site or maybe the "understanding JSON Schema" book (although it's not entirely clear what we're doing with the book right now). Both of them have their own repos.

If you just want to discuss things, the slack is a better option (click "Join our Slack" in the upper right on json-schema.org).

[naught101]

Yeah, OK. I thought the "understanding JSON Schema" book was part of this project, I'll see if there's a relevant issue there.

[handrews]

@naught101 it's part of this project, just not part of this repo :-)