Draft 2019+ tests incorrectly depend on implementations supporting `$schema`-less schemas but they are not required to process them

[gregsdennis]

For example, the first test in the const file is

{
    "description": "const validation",
    "schema": {"const": 2},
    "tests": [
        {
            "description": "same value is valid",
            "data": 2,
            "valid": true
        },
        {
            "description": "another value is invalid",
            "data": 5,
            "valid": false
        },
        {
            "description": "another type is invalid",
            "data": "a",
            "valid": false
        }
    ]
}

There is no $schema declaration. That means that this is a valid draft-6, -7, and -2019-09 schema.

My implementation assumes latest, unless it can be determined that another one should be used (via keywords used or $schema declaration), but that may not be the case for others.

This means that while we intend for this to be evaluated as draft 2019-09, it may not be, depending on how the implementation reads non-specific schemas.

[Julian]

There's no real reason other than that it'd be noise (a property needed in all schemas) -- it's also valid to leave it out even though it's highly recommended, so even if we did add it everywhere you'd still have a few (one at least) that tests the behavior without it.

Your implementation doesn't allow explicitly choosing which version to apply?

E.g. in mine, when I run the suite, for each folder, I explicitly use the appropriate draft, even though mine works as you say defaulting to the latest if you don't specify.

[handrews]

@Julian given that we're strongly encouraging folks to start using $schema and the $vocabulary keyword in the referenced meta-schema to control how to process schemas, we should really include this.

The idea that $schema is a needless formality has significantly degraded the usefulness of the keyword to the point that many implementations outright ignore it and just default to doing whatever they feel like regardless of what the schema declares. That's a problem.

[gregsdennis]

Your implementation doesn't allow explicitly choosing which version to apply?

Quite the opposite. If you specify with $schema, it applies at that draft. But if you don't, it makes a best guess given the keywords that are used.

[gregsdennis]

The idea that $schema is a needless formality has significantly degraded the usefulness of the keyword to the point that many implementations outright ignore it and just default to doing whatever they feel like regardless of what the schema declares. That's a problem.

Exactly. Moreover, we should be the example of how schemas should be authored.

[Julian]

Quite the opposite. If you specify with $schema, it applies at that draft. But if you don't, it makes a best guess given the keywords that are used.

Right that's what I mean -- in mine, there are 3 choices -- if you specify with $schema you get that one. If you specify nothing you get latest. But if you want to, you can explicitly choose which draft to use. Seems pretty important to me to support the latter if you're going to support multiple drafts, and that was definitely how the suite was written (that you explicitly apply the validator for each corresponding draft).

That being said though I'm definitely not against adding $schema to all the schemas for what @handrews you said -- I do think we should have at least 1 test for each draft that doesn't use it and asserts you still get the right draft-specific behavior so long as the drafts don't say it is literally required, otherwise there's a valid schema we won't cover.

(My usual neuroses about not using large sweeping automated formatting changes there though still may mean we have to do this slowly enough to review the patches individually)

Exactly. Moreover, we should be the example of how schemas should be authored.

I don't think the test suite should be a guide there -- it contains in many cases intentionally pathological examples, but yeah fair enough I guess I shouldn't push back if I'm not disagreeing :D

[handrews]

But if you want to, you can explicitly choose which draft to use. Seems pretty important to me to support the latter if you're going to support multiple drafts

I'd say it's far more important to support reading and respecting $schema and $vocabulary. Sure, its fine to put in an option to explicitly set it when $schema is absent, but I would find the ability to outright contradict it to be of dubious value.

Exactly. Moreover, we should be the example of how schemas should be authored.

Yup. This is particularly critical with $vocabulary now. In the past, there wasn't really all that much you could do with $schema. For most implementations, it understood only the http://json-schema.org/{version}/schema URI for each version anyway. Custom meta-schema URIs were essentially useless. With $vocabulary that is no longer the case.

I do think we should have at least 1 test for each draft that doesn't use it and asserts you still get the right draft-specific behavior so long as the drafts don't say it is literally required, otherwise there's a valid schema we won't cover.

Sure, we can have a test for this, but we only need one, and it should only test behavior specified by the specification. So I'm not sure what it should tests. The most recent specification does state a default meta-schema, but I believe it is the first to explicitly do so.

My usual neuroses about not using large sweeping automated formatting changes there though still may mean we have to do this slowly enough to review the patches individually

@Julian are you really saying that you're not OK with mechanically inserting the same $schema everywhere in each version tree? It's so easy to just do that and review them. If there's a problem with file formatting not being stable enough, then we should absolutely fix that first.

[Julian]

but I would find the ability to outright contradict it to be of dubious value.

I didn't mention contradicting -- though yes, mine supports that too. Just specifying it external to a schema. Can't tell if you two disagree, but yeah that's been pretty fundamental to me, for literally this reason, that from what I can tell, very few people use $schema, but still write their schema to a specific draft.

Fixing that human problem is one thing. Not allowing actually getting correct behavior though seems like another, hence me saying I think it's quite important for an implementation to support that -- but whatever, if we do disagree probably on this ticket isn't the greatest place to discuss that (and I'm not sure what ramifications that disagreement has, if your library's users aren't asking you for it who cares what I think :).

@Julian are you really saying that you're not OK with mechanically inserting the same $schema everywhere in each version tree? It's so easy to just do that and review them. If there's a problem with file formatting not being stable enough, then we should absolutely fix that first.

It's not a file formatting issue -- it's an automated change issue -- in my world, there are only 2 kinds of changes:

• changes so transparently correct that they do not need code review and could never be wrong -- these changes don't get code review
• changes that are not transparently correct and do need code review -- these cannot be large, because it is humanly impossible to read large patchsets (like, in a way literally supported by at least decent research -- one that says specifically to keep changes around 200 lines essentially)

A large automated change adding $schema to all schemas is essentially impossible to code review (for me or any human being), and also is likely to introduce unintended bugs due to the specific JSON parser's peculiarities -- so so far yeah I have pushed back on any large automated change.

[handrews]

A large automated change adding $schema to all schemas is essentially impossible to code review

Well I strongly disagree with that but this is your package so I won't push. I'm not submitting manual PRs to individually change it though, and I find the amount that that's going to slow things down frustrating.

[Julian]

Well -- how about this -- if you and @gregsdennis feel confident one can submit it and the other review it that I don't object to -- I do think it is impossible even for you :), and like I said I think there's some research, logic (that your eyes glaze over from reading long mechanical changes and you literally do not notice things) and anecdotal evidence (I have literally never seen such a change not introduce a bug, including the last time in this repo, when I happened to trust my gut and force myself to try to review the change and then sure enough find a bug)

I do understand the sentiment, this isn't the first time I've had this discussion (though I did not invent the above :) -- I got it from other projects that value high quality software and programmers smarter than me, and so far it's served me well as a programmer) -- but yeah if the two of you are confident in the change, it's not like I've got an iron fist here, this repo is useful to all hopefully, and you both are pretty reasonable :) (though yeah I don't want to see this kind of change on any at all frequent basis)

[handrews]

@Julian I apologize for pushing on the bulk change topic. Please feel free to edit out any comments, in whole or in part, that you feel detract from the point of this issue. 🙏

[karenetheridge]

There should at least be some tests in each version's suite that contain a $schema value for a different draft, and some assertions that would work only when the version is properly detected.

For example, one could copy a test from draft4 which uses a keyword whose semantics later changed (say exclusiveMinimum, that used to specify a boolean value but is now numeric), adding a "$schema": "http://json-schema.org/draft-04/schema#" to it -- indicating that the parser should recognize that this is not a draft7 schema and should not be evaluated as such (it could either error out, or downgrade to draft4 semantics).

Another tricky test would be a schema with both a $id and id keywords, but with different values -- the test being that the proper one is used and the other is ignored, according to the value of $schema.

[Julian]

That part is/was json-schema-org/json-schema-spec#210 but I don't remember what happened there.

But agreed though on we should have them (though I think it'd be nice if we actually did it in a draft-agnostic way -- i.e. ideally we shouldn't rely on downstream validators supporting multiple drafts -- we should instead do this by e.g. fixing a single draft, then redefining a new metaschema, then using the new metaschema in a new schema or something, and then having the test verify that it's validated using the new metaschema. This way someone can have them pass without implementing multiple drafts).

[Julian]

Can we separate out that one from this ticket which I'm inclined to even close since we got all wrapped up in unrelated things?

This one was originally about test case style, but I'm inclined to be bold given all the back and forth we got tangled in and say we should just come back to that later, let's get 2019-09 tests all done first...