-
Notifications
You must be signed in to change notification settings - Fork 210
RFD to introduce an RFD process #405
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFD to introduce an RFD process #405
Conversation
8f91091 to
e3176dc
Compare
|
First comment to track any open points of disagreement as they arise... (Checked off items have been resolved via updates to the PR materials)
|
|
I love this. Thank you for putting the effort in! For the benefit of of anyone else trying to read the full document here's a link to see it rendered. Some nits:
For the sake of procedure; does this mean that the CVE board should record their approval via PRs? eg. One/many board member(s) approve/reject a PR? A (minor) disagreement: example: github/advisory-database#5234 |
|
Thanks! I'll update the reference link for "Seeing Like a State." It's really only included for the purpose of explaining the term "legible," which is covered in the Wikipedia page for the book, so I'll link to that. I can also amend the first post of the PR to include rendered links to the RFD and the template. For final approval, I'm inclined to make it the job of a QWG Co-Chair to update the PR with the result of a CVE Board vote, since this RFD process is something the QWG is adopting for itself, and thus ought not to bind / demand new process from anyone outside of the QWG. I'm also happy to amend with an activity requirement to stave off automated closure, assuming someone in the QWG is willing to set up that automation as you describe. |
|
Ah, actually @darakian I think the current language matches what you want pretty well. There's no set requirement for the QWG to resolve an RFD with a final determination on a set timetable, but there is a requirement that RFD pull requests remain active, with inactive PRs being closed and the RFD then requiring re-submittal. I'll change the closure language from "may be closed" to "shall be closed" to make it non-discretionary on the part of the QWG Co-Chairs. |
e3176dc to
51fd2dc
Compare
Wikipedia is also good and my nit there is of course minor. I just want to be mindful of open access principles as much as possible.
That seems reasonable to me as well, but I do think the board should make it clear that they accept that delegation of responsibility (maybe via RFD even 🤔).
That's fair too. The automation aspect is mostly just a forcing function to keep things moving forward. |
axeberg
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small typo in list of items.
51fd2dc to
a53be9e
Compare
|
is the anchor text in the Value for Field or is it supposed to be a unique identifier? I'm guessing (but I'm not sure) that and are supposed to have different anchor text such as versus and maybe the .md filenames are not both supposed to start with |
|
Per the RFD procedures specified in the RFD document, they only get assigned a number if/when they are approved. Until that point they use 0000 as a placeholder. On approval then yes the document file name and contents will be updated to reflect the number assigned to the RFD. |
|
I have now opened a poll on whether to adopt this RFD: #412 |
Great work. Here are my suggested edits for the template:
|
This introduces a new Request for Discussion (RFD) process for considering changes to the CVE Record Format. The details of the proposal are in the commit. This commit includes both the first RFD, proposing the introduction an RFD process and modeling what the process looks like, as well as a template for future RFDs to reuse. Signed-off-by: Andrew Lilley Brinker <[email protected]>
a53be9e to
7de8fbb
Compare
|
Thanks @boblord! I've amended the template with your suggestions. Regarding alternative options, I opted not to modify the "Proposed Solution" section, as alternative options are already given space to be listed in the "Related Issues or Proposals" section of the template. |
|
@alilleybrinker one meta comment I have is that you should avoid all the squashing and force pushing of commits. It has the effect of deleting the history of how the final form of the document comes together. Some commits are worth squashing together and this is an art form more than a science, but I would encourage you to err more on the side of leaving more breadcrumbs rather than fewer 👍 |
|
@darakian yeah this is a frustration I have with how GitHub pull requests handle code review. In other systems, comments against prior versions of the patchset included in a change are much clearer to navigate and review. In GitHub, the history isn't lost (if you scroll up, the force-pushed-over commits are still accessible and show prior revisions of the documents), but they are inconvenient to access. In general, I prefer to keep the commits in a PR in a merge-able state, rather than either a squash-at-the-end approach immediately prior to merge or merging all intermediate commits. That said, if the group's preference is to merge intermediate commits, or to squash at the end, I'm happy to adjust. This may be a good thing to document in a EDIT: I'll do additional commits going forward, and have now added one to address the feedback from @andrewpollock. |
andrewpollock
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a fantastic and revolutionary leap forward in structural discipline, thank you!
This clarifies the procedures around updating the RFD number in an RFD pull request after approval of an RFD by the CVE Board. Signed-off-by: Andrew Lilley Brinker <[email protected]>
|
I feel that SchemaVer has only a small value in characterizing the change proposed in an RFD, and that additional characterization of the change is needed within an RFD. In other words, I don't agree that "describe any migration paths" is an adequate alternative. SchemaVer is only about interaction between a new schema and historical data. Sometimes, it only represents the amount of work that the CVE Program administration would need to perform in order to convert historical data to the new format. Often, it does not capture the amount of work that downstream consumers would need to perform for continuity of operations. I don't know of previous research into measuring downstream impacts of JSON schema changes that alter the contract between producers and consumers. This type of research may exist. I believe that relevant yes/no-style characterization includes:
Depending on how the schema change is characterized, it may be easier to make decisions on:
There once was work on the SOAP specification that I think is tangentially related: https://w3.org/TR/soap12-part1 sections 2.4 and 5.2.3. Essentially, SOAP has the concept of Today, One might imagine similar use of By contrast, suppose that CVSS 3.1 were the only choice in Again, |
|
@ElectricNroff, the template purposefully leaves the level of required detail on a migration path open to the discretion of the QWG. Some proposals are substantial and will, as you describe, entail a more substantial migration requiring detailed planning and public communication. Other changes may be minor and require little work. I don't want to be overly prescriptive about the required process here, because I think the odds of getting a one-size-fits-all process right up-front in a document like this is low. We require outlining the level of SchemaVer impact because SchemaVer is what the project uses for versioning, and the migration path is intended to cover the remaining details (to the level necessary) to support an ecosystem migration on both the production and consumption sides to support a change. |
At Matt Powers' recommendation, this amends the "Compatibility and Migration" section of the RFD template to describe expectations for analyzing and planning migrations more clearly, including specific questions which ought to be answered, and clarifying the limits of SchemaVer in expressing the adoption burden of new features. Signed-off-by: Andrew Lilley Brinker <[email protected]>
|
@ElectricNroff, I've amended the RFD's "Compatibility and Migration" section to hopefully address your concerns. Take a look and let me know if you think it still needs revision! |
Based on conversations with Matt Power, I've amended the text of the "Compatibility and Migration" section of the RFD template in a couple of ways: 1. Removed explicit reference to SchemaVer as the versioning scheme of choice for the CVE Record Format. 2. Introduced a set of questions which must specifically be answered in any submitted RFD. The key goal here is to balance the need for a high level of rigor around compatibility and migration in CVE because it is a systemically-important and large multi-stakeholder system with the need to not be overly prescriptive in the RFD template in ways which may in fact reduce rigor by over-specializing RFD requirements on today's considerations for what the important questions are. The questions included are specific, and intended to capture key concerns about forward compatibility and the impact of changes on CVE consumers particularly. Separately, references to SchemaVer were removed because, while it is my understanding that SchemaVer is the version scheme of choice for the CVE Record Format, that is not currently codified and itself likely ought to go through an RFD process to firmly resolve. In fact, Matt Power has raised concerns that SchemaVer may be insufficiently expressive for the versioning constraints under which CVE operates, and I'd rather not attempt to resolve that question in the context of this process-focused RFD. Signed-off-by: Andrew Lilley Brinker <[email protected]>
|
@ElectricNroff I've updated the Compatibility and Migration section based on our conversation yesterday. Please take a look and let me know if this satisfies what you're looking for. (CC @ccoffin) |
|
I feel that the term "forward compatibility" is needlessly confusing because there are different vantage points with different compatibility concerns. For example, suppose that the CVE Record Format stated that vulnerabilities would be described in English in either an HTML document or a PDF document. Then someone proposed that a Word document would also be OK. That would be an "ADDITION" type of change, but would not be backwards compatible with the processes of consumers to read the documents. |
I believe this is an incorrect use of the term "backward compatibility." Backward compatibility is when a new version of something is still compatible with prior instances of that thing. So adding a field (an ADDITION-level change in SchemaVer) is inherently backwards compatible, and thus the proposed change in your hypothetical is backwards compatible. To be more succinct:
The one wrinkle in forward compatibility is that, if it's not already documented, we'd need to document what additions we expect CVE consumers to be resilient to, at minimum: adding new optional fields to an object. |
|
In the hypothetical, a client could sent a GET request today to the CVE Services server for /api/cve/CVE-2026-0001 with the header "Accept: application/pdf, text/html" and they would get the CVE-2026-0001 document in either PDF or HTML (whichever the server chose to send). However, the CNA can choose to update the record to Word tomorrow if the proposal were adopted. Then, if I send the same GET request for /api/cve/CVE-2026-0001 with the "Accept: application/pdf, text/html" header, I receive an error message to the effect that the server does not have CVE-2026-0001 in either PDF or HTML. In this situation, it is completely valid to say that the API service is not backward compatible. (It could be made backward compatible if on-the-fly data conversion were implemented.) That is what I meant by "different vantage points." |
|
@ElectricNroff I don't follow with the pdf/word/etc... stuff, but the term forward compatibility is not some new invention. Wikipedia has an article with edits going back to at least 2003 A question for you. Say we have a schema of the form A single named field that holds a string. |
In the way you've written it, no, because a producer might have already used field_b in an incompatible way. When the schema was: a producer may have already composed a historical document: and this won't validate against the new schema. This is analogous to "we decide to add a new cost property" in SchemaVer. Possibly you meant that In that situation, the most common JSON schema social contract is that ordinary consumers aren't allowed to raise compatibility objections for a schema change that only introduces extra data elements. (For example, the extra data has no effect on what data is required, the valid ways of expressing required data, what other data may be present, what values are allowed in other data, or the steps for extracting information from any data.) If they aren't interested in field_b, they are expected to ignore it. There can be domain-specific differences in the social contract, e.g., if everyone using this schema produces multi-TB strings and understands that two of them aren't going to fit in memory on the users' machines, but I think these are uncommon. |
Say "additionalProperties": false is not present, but we can be 100% sure that no records have ever been published using |
|
I don't think we need to resolve the versioning question here. For the purposes of the RFD template, the goal is to include enough detail on what should go into an RFD so that the QWG can be well-informed when making decisions. This template could also be updated in the future after an RFD on how to version the CVE Record Format if there's a need to reflect new consensus on the right questions. I think the current text is probably sufficient at asking the right questions and probing for the right detail. I also think backwards compatible and forward compatible are sufficiently well-attested terms with common and well-understood meanings, and I am comfortable leaving them in the template. |
|
I've opened an issue on the need to develop a CVE Record Format versioning RFD, which links to this and two other open discussions that touch on the versioning question. As I said, I think this RFD can and should move forward without waiting to resolve the versioning questions raised in the linked issue. (#418) |
|
@ElectricNroff and @darakian, how do you both feel about the text of the RFD as it stands, with the understanding that we won't try to resolve the versioning commitment question here, and with the clarification that "forward compatibility" is a standard term with a well-understood definition? I think, given those two notes, that the open issues are addressed, but I want to confirm. |
No, for APIs that depend on JSON schemas, the reality is that there are two different meanings of (for example) backward compatible that are both in common use, neither one is the result of any mistakes or misunderstandings, and they represent two different perspectives. For example: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html https://docs.specmatic.io/documentation/backward_compatibility_rules.html Suppose the old schema is: and the new schema is: There is v1.0 of an API service that responds with documents that conform to the old schema. Then, the API service is changed to v1.1 and responds with documents that conform to the new schema. In the confluent.io perspective, this is a backward compatible change because a client who is using the new schema can certainly read and understand data produced with the old schema (i.e., all documents are guaranteed to have both x and y). In the specmatic.io perspective, this is not a backward compatible change, because a mandatory key y has now become optional. The server is now free to send documents that do not have y. Client code that previously was accessing the y key may now fail with an error message indicating that the key was not found, or the client may not be able to compute a y value that is needed for an important business purpose. The confluent.io perspective is occasionally called schema-backward-compatible, whereas the specmatic.io perspective is occasionally called consumer-backward-compatible. (Normally, they are both just called backward compatible, perpetuating the confusion.) They are both of interest, but they affect the CVE Program in different ways. We are concerned about the consumer experience. We don't have evidence of any significant consumer population that retrieves JSON documents (either from our API service, or from a known filename on GitHub) and then executes JSON schema validation. We do know that consumers access JSON keys that they expect are present, and use the values in important decision processes of varying complexity. So, I feel that it's best to offer explanations that apply both to the consumer audience and the RFD author audience. Regardless of what terminology we choose, we have to explain what we mean: we cannot rely on an assumption that something might be "well-attested" one way or the other. For example, some relatively simple initial statements could be:
|
|
@ElectricNroff I appreciate you digging into different examples of how others use those terms. That said, the RFD provides definitions for both "backward compatibility" and "forward compatibility." |
I believe that there would be substantial challenges in using its definitions: Separately, you wrote: (shown here just as background information - it's not part of the RFD document) To restate what I've said in a different way: "impact users of the schema's ability to parse existing CVE records produced under prior versions of the CVE Record Format" does not mention that, within CVE Program operations, these users are not expected to try to parse "records produced under prior versions of the CVE Record Format." When the schema changes, the official copies of all CVE Records are changed (e.g., administratively) to comply with the current version of the CVE Record Format. "explain if current users of the schema would be able to accept" does not indicate what "accept" means. Typically, it is easy for users to download the latest copy of the schema, but not necessarily easy for them to change how they extract and use data within the now validated documents, and not necessarily easy for them to change a UI so that (for example) users can enter any now valid data but are dynamically guided away from entry of invalid data. For "When a new version of the schema accepts records compatible with the past version of the schema": a previous update to the CVE Record Format (5.0.0 to 5.1.0) was widely favored, and had zero known user complaints afterward, even though this property was not present. For "When a new version of the schema does not permit records which would not be permitted by the past version of the schema": a previous update to the CVE Record Format (5.1.0 to 5.1.1) was mostly favored, and had zero known user complaints afterward, even though this property was not present. By contrast, the material at https://docs.specmatic.io/documentation/backward_compatibility_rules.html offers objective criteria about what disrupts a consumer's compatibility experience in realistic scenarios. (It also offers automation around this.) It may not all be accurate, and it may not be complete, and it may not be adapted to our exact use case. But, my guess is that it would allow evaluation of schema-change ideas with weeks or months less elapsed time. |
|
@ElectricNroff I've updated my tracking of open issues to reflect some of the recent things we've covered and where I believe the conversation stands: #405 (comment) The current question is whether the RFD's definitions of "backward compatibility" and "forward compatibility" are sufficiently clear. Based on your latest comment, I will amend the language slightly, but I will not produce a detailed breakdown of the precise versioning guarantees like is found in the linked Specmatic documentation, because I do not believe I can. The QWG has not resolved the rules for CVE Record Format versioning to that level of detail. In fact, we only just finished a discussion earlier in this thread objecting to SchemaVer, which was previously the best candidate (and indeed, which I had understood to be the group's consensus) model for versioning the Record Format. I understand your desire for more rigor in this section, and I agree with the goal and want to see the QWG get there. I think for now the best we can do is to have general requirements for describing compatibility in the RFD template, with an understanding that the QWG will need to closely scrutinize this section and ask pertinent questions during review. At the same time, we can pursue an RFD to codify precise versioning guarantees (see: #418), and when that's resolved we can amend the RFD template to be more detailed. |
Amend the definitions of "backward compatibility" and "forward compatibility" in the "Compatibility and Migration" section of the RFD template to more clearly explain requirements for RFD writers. In the future we'd like this section to be more rigorous, and ideally to provide a detailed breakdown of what kinds of changes in the schema are considered "breaking," but the versioning rules for the CVE Record Format aren't sufficiently defined yet. I recommend developing an RFD to solidify the versioning rules, and then amending the template to require greater rigor around compatibility in this section. Signed-off-by: Andrew Lilley Brinker <[email protected]>
|
Which downstream consumers of CVE data have we polled on this topic? Should
we bring more into the conversation?
Thanks,
-Bob
…On Wed, Jun 11, 2025 at 10:58 AM Andrew Lilley Brinker < ***@***.***> wrote:
*alilleybrinker* left a comment (CVEProject/cve-schema#405)
<#405 (comment)>
@ElectricNroff <https://github.com/ElectricNroff> I've updated my
tracking of open issues to reflect some of the recent things we've covered
and where I believe the conversation stands: #405 (comment)
<#405 (comment)>
The current question is whether the RFD's definitions of "backward
compatibility" and "forward compatibility" are sufficiently clear.
Based on your latest comment, I will amend the language slightly, but I
will not produce a detailed breakdown of the precise versioning guarantees
like is found in the linked Specmatic documentation, because I do not
believe I can.
The QWG has not resolved the rules for CVE Record Format versioning to
that level of detail. In fact, we only just finished a discussion earlier
in this thread objecting to SchemaVer, which was previously the best
candidate (and indeed, which I had understood to be the group's consensus)
model for versioning the Record Format.
I understand your desire for more rigor in this section, and I agree with
the goal and want to see the QWG get there. I think for now the best we can
do is to have general requirements for describing compatibility in the RFD
template, with an understanding that the QWG will need to closely
scrutinize this section and ask pertinent questions during review.
At the same time, we can pursue an RFD to codify precise versioning
guarantees (see: #418
<#418>), and when that's
resolved we can amend the RFD template to be more detailed.
—
Reply to this email directly, view it on GitHub
<#405 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACBH7WYN4XATPQOHPVRAPX33DBU5BAVCNFSM6AAAAAB4PRVUZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRTG4YDKNZWGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I think consumer engagement would make sense for the versioning conversation, but for this RFD which is solely focused on the process the QWG follows for change management I don't think it makes sense. |
|
Note Final Comment Period called!This proposal is entering a Final Comment Period (FCP). There is one week remaining to give comments on this proposal, closing at 1pm PDT / 4pm EDT, Thursday, June 19th. |
Co-authored-by: Jon <[email protected]>
|
Note Final Comment Period ClosedThe Final Comment Period (FCP) for this proposal has closed, with no new issues raised. This proposal is now closed for further issues, and goes to the QWG Co-Chairs to determine whether to adopt or reject the proposal. As this is a QWG-specific procedural proposal, the QWG Co-Chairs will be the final deciders, and the issue will not advance to the CVE Board. |
|
Update from the last QWG meeting, held yesterday. With the FCP closed, @ccoffin has brought the RFD to the attention of the CVE Board for any comments. As the RFD only proposes procedural changes for the QWG itself, it is not necessary for the Board of directors to vote to approve it, as they would for changes to the CVE Record Format. However, it is being offered to the Board for comment, as it is a substantive change that impacts the nature and format of proposals which will in the future be presented to the Board after QWG approval. Barring the raising of additional concerns by members of the CVE Board, this RFD (and thereby the RFD process itself) will be adopted by the QWG. |
|
Update from the 2025/07/03 QWG meeting: the RFD will be adopted, as we have not received any contrary feedback from the CVE Board. Next steps will be:
@ccoffin, I'll update the PR with the RFD number and ping you when it's ready to merge. |
Per the RFD rules, assign ID CVEProject#1 to the first RFD. Signed-off-by: Andrew Lilley Brinker <[email protected]>
|
@ccoffin, the RFD ID number has been updated. This is ready for merge! |
This introduces a new Request for Discussion (RFD) process for considering changes to the CVE Record Format. The details of the proposal are in the commit.
This commit includes both the first RFD, proposing the introduction an RFD process and modeling what the process looks like, as well as a template for future RFDs to reuse.
EDIT: