Split protobuf package into libraries #1026

osa1 · 2025-07-18T09:51:55Z

This is the first part of the PRs that sync internal JSON decoding chages.

To be able to conditionally import different JSON decoders, this splits the
monolithic protobuf package into libraries.

These parts are now internal libraries:

consts.dart
json_parsing_context.dart
permissive_compare.dart
type_registry.dart
utils.dart

These changes are not useful on their own, they're to prepare the library for
the rest of the PRs and to keep the PRs small and easier to review.

In preparation for adding conditional imports to be able to use different encoding/decoding implementation based on the platform, this PR creates an "internal" library and exports some of the important types for encoding/decoding like `FieldInfo`, `PbFieldType` (renamed as `PbFieldTypeInternal`). This syncs some of cl/613649886.

This reverts commit 32911d8.

devoncarew

This change looks good from the POV of a general refactoring. My main question is what the actual impact on the public API surface area is. If you know it off the top of your head, and do not have concerns, then sgtm. Otherwise, you might look into:

that API diff tool from BMW
running dart doc before and after the change, and manually inspecting the output
try a small, api => normalized markdown tool I wrote a while ago, and see how that works; the results are intended to be something you'd diff. https://github.com/devoncarew/app_dirs/blob/main/tool/api.dart

protobuf/lib/protobuf.dart

protoc_plugin/lib/src/gen/google/api/client.pb.dart

protobuf/lib/src/protobuf/field_type.dart

osa1 · 2025-07-22T10:17:54Z

@devoncarew I compared the API before and after these set of PRs using apitool. Somehow it seems to generate lots of false positives. I've pasted the full diff it generates below. You'll see a lot of lines like:

│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)

and things like:

├── Function "parseLongInt" removed (CE10)
...
├── Function "parseLongInt" added (CE11) (minor)

I don't know why it generates these changes..

When I manually get rid of the false positives the remaining changes are:

BREAKING CHANGES
├── Interface "PbFieldType" removed (CI01)
├── Interface "_FieldSet" removed (CI01)
Non-Breaking changes
├── Interface "PbFieldTypeInternal" added (CI02) (minor)
├── Interface "FieldSet" added (CI02) (minor)

I also don't understand what it means by "Interface FieldSet added". When I document the library I can't find a public type named FieldSet, because we hide it in the public library. No idea why apitool is listing it as an addition.

Both the library and plugin needs a major version bump with these changes.

@devoncarew do we want to merge these all together, or as separate PRs?

full apitool output

BREAKING CHANGES
├── Interface "PbFieldType" removed (CI01)
├── Interface "_FieldSet" removed (CI01)
├── Function "parseLongInt" removed (CE10)
├── Function "getCheckFunction" removed (CE10)
├── Function "unpackIntoHelper" removed (CE10)
├── Function "canUnpackIntoHelper" removed (CE10)
├── Function "getTagFieldNumber" removed (CE10)
├── Function "getTagWireType" removed (CE10)
├── Function "makeTag" removed (CE10)
├── Field "WIRETYPE_VARINT" removed (CF01)
├── Field "WIRETYPE_FIXED64" removed (CF01)
├── Field "WIRETYPE_LENGTH_DELIMITED" removed (CF01)
├── Field "WIRETYPE_START_GROUP" removed (CF01)
├── Field "WIRETYPE_END_GROUP" removed (CF01)
├── Field "WIRETYPE_FIXED32" removed (CF01)
├─┬ Class BuilderInfo
│ ├─┬ Method addMapField
│ │ ├── Type of parameter "mapEntryBuilderInfo" changed. BuilderInfo -> BuilderInfo (CE08)
│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
│ ├─┬ Method addRepeated
│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
│ ├─┬ Method pc
│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
│ ├─┬ Method m
│ │ ├── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
│ │ └── Type of parameter "packageName" changed. PackageName -> PackageName (CE08)
│ └─┬ Constructor new
│   └── Type of parameter "package" changed. PackageName -> PackageName (CE08)
├─┬ Class FieldInfo
│ ├── Field "enumValues" removed (CF01)
│ ├── Field "defaultEnumValue" removed (CF01)
│ ├── Field "valueOf" removed (CF01)
│ ├─┬ Constructor new
│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
│ └─┬ Constructor repeated
│   └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
├─┬ Class GeneratedMessage
│ ├─┬ Method writeToCodedBufferWriter
│ │ └── Type of parameter "output" changed. CodedBufferWriter -> CodedBufferWriter (CE08)
│ ├─┬ Method mergeFromCodedBufferReader
│ │ ├── Type of parameter "input" changed. CodedBufferReader -> CodedBufferReader (CE08)
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromBuffer
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromJson
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromJsonMap
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method addExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method clearExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method getExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method hasExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method mergeFromMessage
│ │ └── Type of parameter "other" changed. GeneratedMessage -> GeneratedMessage (CE08)
│ ├─┬ Method mergeUnknownFields
│ │ └── Type of parameter "unknownFieldSet" changed. UnknownFieldSet -> UnknownFieldSet (CE08)
│ ├─┬ Method setExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ └─┬ Method $_get
│   └── Type of parameter "defaultValue" changed. T -> T (CE08)
├─┬ Class UnknownFieldSet
│ ├─┬ Method addField
│ │ └── Type of parameter "field" changed. UnknownFieldSetField -> UnknownFieldSetField (CE08)
│ ├─┬ Method mergeField
│ │ └── Type of parameter "field" changed. UnknownFieldSetField -> UnknownFieldSetField (CE08)
│ ├─┬ Method mergeFieldFromBuffer
│ │ └── Type of parameter "input" changed. CodedBufferReader -> CodedBufferReader (CE08)
│ ├─┬ Method mergeFromCodedBufferReader
│ │ └── Type of parameter "input" changed. CodedBufferReader -> CodedBufferReader (CE08)
│ ├─┬ Method mergeFromUnknownFieldSet
│ │ └── Type of parameter "other" changed. UnknownFieldSet -> UnknownFieldSet (CE08)
│ ├─┬ Method mergeGroupField
│ │ └── Type of parameter "value" changed. UnknownFieldSet -> UnknownFieldSet (CE08)
│ └─┬ Method writeToCodedBufferWriter
│   └── Type of parameter "output" changed. CodedBufferWriter -> CodedBufferWriter (CE08)
├─┬ Class UnknownFieldSetField
│ ├─┬ Method writeTo
│ │ └── Type of parameter "output" changed. CodedBufferWriter -> CodedBufferWriter (CE08)
│ └─┬ Method addGroup
│   └── Type of parameter "value" changed. UnknownFieldSet -> UnknownFieldSet (CE08)
├─┬ Class CodedBufferReader
│ ├─┬ Method readGroup
│ │ ├── Type of parameter "message" changed. GeneratedMessage -> GeneratedMessage (CE08)
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ └─┬ Method readMessage
│   ├── Type of parameter "message" changed. GeneratedMessage -> GeneratedMessage (CE08)
│   └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
├─┬ Class ExtensionRegistry
│ ├─┬ Method add
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ └─┬ Method reparseMessage
│   └── Type of parameter "message" changed. T -> T (CE08)
├─┬ Class Extension
│ ├── Field "enumValues" removed (CF01)
│ ├── Field "defaultEnumValue" removed (CF01)
│ └── Field "valueOf" removed (CF01)
├─┬ Class MapFieldInfo
│ ├── Field "enumValues" removed (CF01)
│ ├── Field "defaultEnumValue" removed (CF01)
│ ├── Field "valueOf" removed (CF01)
│ └─┬ Constructor new
│   ├── Type of parameter "mapEntryBuilderInfo" changed. BuilderInfo -> BuilderInfo (CE08)
│   └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
├─┬ Class GeneratedService
│ └─┬ Method handleCall
│   ├── Type of parameter "ctx" changed. ServerContext -> ServerContext (CE08)
│   └── Type of parameter "request" changed. GeneratedMessage -> GeneratedMessage (CE08)
├─┬ Class $_MessageSet
│ ├─┬ Method writeToCodedBufferWriter
│ │ └── Type of parameter "output" changed. CodedBufferWriter -> CodedBufferWriter (CE08)
│ ├─┬ Method mergeFromCodedBufferReader
│ │ ├── Type of parameter "input" changed. CodedBufferReader -> CodedBufferReader (CE08)
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromBuffer
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromJson
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method mergeFromJsonMap
│ │ └── Type of parameter "extensionRegistry" changed. ExtensionRegistry -> ExtensionRegistry (CE08)
│ ├─┬ Method addExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method clearExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method getExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method hasExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ ├─┬ Method mergeFromMessage
│ │ └── Type of parameter "other" changed. GeneratedMessage -> GeneratedMessage (CE08)
│ ├─┬ Method mergeUnknownFields
│ │ └── Type of parameter "unknownFieldSet" changed. UnknownFieldSet -> UnknownFieldSet (CE08)
│ ├─┬ Method setExtension
│ │ └── Type of parameter "extension" changed. Extension<dynamic> -> Extension<dynamic> (CE08)
│ └─┬ Method $_get
│   └── Type of parameter "defaultValue" changed. T -> T (CE08)
└─┬ Class RpcClient
  └─┬ Method invoke
    ├── Type of parameter "ctx" changed. ClientContext? -> ClientContext? (CE08)
    ├── Type of parameter "request" changed. GeneratedMessage -> GeneratedMessage (CE08)
    └── Type of parameter "emptyResponse" changed. T -> T (CE08)
Non-Breaking changes
├── Interface "PbFieldTypeInternal" added (CI02) (minor)
├── Interface "FieldSet" added (CI02) (minor)
├── Function "parseLongInt" added (CE11) (minor)
├── Function "getCheckFunction" added (CE11) (minor)
├── Function "unpackIntoHelper" added (CE11) (minor)
├── Function "canUnpackIntoHelper" added (CE11) (minor)
├── Function "getTagFieldNumber" added (CE11) (minor)
├── Function "getTagWireType" added (CE11) (minor)
├── Function "makeTag" added (CE11) (minor)
├── Field "mapKeyFieldNumber" added (CF02) (minor)
├── Field "mapValueFieldNumber" added (CF02) (minor)
├── Field "WIRETYPE_VARINT" added (CF02) (minor)
├── Field "WIRETYPE_FIXED64" added (CF02) (minor)
├── Field "WIRETYPE_LENGTH_DELIMITED" added (CF02) (minor)
├── Field "WIRETYPE_START_GROUP" added (CF02) (minor)
├── Field "WIRETYPE_END_GROUP" added (CF02) (minor)
└── Field "WIRETYPE_FIXED32" added (CF02) (minor)

osa1 · 2025-07-22T10:47:51Z

I just reverted the renaming because I think it's not necessary (see comments above). This should be backwards compatible now.

(apitool still reports lots of false positives...)

devoncarew · 2025-07-22T14:50:06Z

I just reverted the renaming because I think it's not necessary (see comments above). This should be backwards compatible now.

(apitool still reports lots of false positives...)

I have not used apitool myself; I think what it does is impressive but am not surprised that it gets tripped up on some things.

Re: the renaming and "the type was already public, but it's a mysterious undocumented type that (I hope) no one's using". I suspect that is wasn't used by users but was used by the generated code. I believe you're correct above that to ship this rename we'd need a major version bump for both packages. This is something we can do if we want to keep in sync w/ internal; I think a revert here (which you've done) and look into reverting the renaming internally is also something we can do and probably less friction.

do we want to merge these all together, or as separate PRs?

Separate PRs seems fine to me.

devoncarew

lgtm!

osa1 · 2025-07-23T09:59:46Z

Since protoc_plugin did not change at all (other than a test file where we have file paths for dart:mirrors), and the user API should also not change (unless I did something wrong), I'm bumping minor version. Files generated with the current protoc_plugin will be compatible with these changes and the user code should also not break.

mosuem · 2025-07-23T13:26:37Z

cc @devmil

@devoncarew I compared the API before and after these set of PRs using apitool. Somehow it seems to generate lots of false positives. I've pasted the full diff it generates below. You'll see a lot of lines like:
│ │ └── Type of parameter "defaultEnumValue" changed. ProtobufEnum? -> ProtobufEnum? (CE08)
and things like:
├── Function "parseLongInt" removed (CE10)
...
├── Function "parseLongInt" added (CE11) (minor)
I don't know why it generates these changes..

When I manually get rid of the false positives the remaining changes are:
BREAKING CHANGES
├── Interface "PbFieldType" removed (CI01)
├── Interface "_FieldSet" removed (CI01)
Non-Breaking changes
├── Interface "PbFieldTypeInternal" added (CI02) (minor)
├── Interface "FieldSet" added (CI02) (minor)
I also don't understand what it means by "Interface FieldSet added". When I document the library I can't find a public type named FieldSet, because we hide it in the public library. No idea why apitool is listing it as an addition.

Both the library and plugin needs a major version bump with these changes.

@devoncarew do we want to merge these all together, or as separate PRs?

full apitool output

devmil · 2025-07-23T17:28:54Z

thanks for the hint @mosuem
Those two commits uncovered some quite big issues in dart_apitool so they made it into the ever growing list of dart_apitool integration tests

mosuem · 2025-07-24T06:38:40Z

thanks for the hint @mosuem Those two commits uncovered some quite big issues in dart_apitool so they made it into the ever growing list of dart_apitool integration tests

Thanks to @osa1 for the digging!

osa1 added 3 commits July 18, 2025 09:54

Also make consts a lib

35bb15a

Make utils a lib

9a2207f

osa1 marked this pull request as ready for review July 18, 2025 09:55

osa1 requested a review from devoncarew July 18, 2025 09:55

osa1 mentioned this pull request Jul 18, 2025

Make PbFieldType members public #1027

Merged

osa1 added 2 commits July 18, 2025 13:05

Export ExtensionRegistry

32911d8

Revert "Export ExtensionRegistry"

e821977

This reverts commit 32911d8.

devoncarew reviewed Jul 18, 2025

View reviewed changes

protobuf/lib/protobuf.dart Show resolved Hide resolved

protoc_plugin/lib/src/gen/google/api/client.pb.dart Outdated Show resolved Hide resolved

protobuf/lib/src/protobuf/field_type.dart Outdated Show resolved Hide resolved

osa1 added 2 commits July 22, 2025 09:40

Merge remote-tracking branch 'origin/master' into split_into_libs

9cb7ca2

Revert some of the changes

be75e06

osa1 added 2 commits July 22, 2025 11:36

Revert renaming

34afcf8

Formatting

38e9653

devoncarew approved these changes Jul 22, 2025

View reviewed changes

Bump version, update CHANGELOG

231934e

osa1 merged commit 7f7d776 into master Jul 23, 2025
16 checks passed

osa1 deleted the split_into_libs branch July 23, 2025 10:05

devmil mentioned this pull request Jul 23, 2025

Fix weird protobuf changes bmw-tech/dart_apitool#232

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Split protobuf package into libraries #1026

Split protobuf package into libraries #1026

Uh oh!

osa1 commented Jul 18, 2025 •

edited

Loading

Uh oh!

devoncarew left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

osa1 commented Jul 22, 2025

Uh oh!

osa1 commented Jul 22, 2025

Uh oh!

devoncarew commented Jul 22, 2025

Uh oh!

devoncarew left a comment

Uh oh!

osa1 commented Jul 23, 2025

Uh oh!

Uh oh!

mosuem commented Jul 23, 2025

Uh oh!

devmil commented Jul 23, 2025

Uh oh!

mosuem commented Jul 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Split protobuf package into libraries #1026

Split protobuf package into libraries #1026

Uh oh!

Conversation

osa1 commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devoncarew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

osa1 commented Jul 22, 2025

Uh oh!

osa1 commented Jul 22, 2025

Uh oh!

devoncarew commented Jul 22, 2025

Uh oh!

devoncarew left a comment

Choose a reason for hiding this comment

Uh oh!

osa1 commented Jul 23, 2025

Uh oh!

Uh oh!

mosuem commented Jul 23, 2025

Uh oh!

devmil commented Jul 23, 2025

Uh oh!

mosuem commented Jul 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

osa1 commented Jul 18, 2025 •

edited

Loading