From 4303ca50ad0a5c6934bfb640c8142298efc38c0f Mon Sep 17 00:00:00 2001 From: Stephane Bersier Date: Fri, 19 Jan 2024 06:25:37 -0500 Subject: [PATCH 01/27] Update 42.type.md Tried to improve superficial grammar. --- content/42.type.md | 90 +++++++++++++++++++++++----------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/content/42.type.md b/content/42.type.md index f143c9a8..c35552cf 100644 --- a/content/42.type.md +++ b/content/42.type.md @@ -33,17 +33,17 @@ their role is to give the meaning of paths selecting types and terms from nested paths have an intuitive meaning to programmers from a wide range of backgrounds which belies their underpinning by a somewhat "advanced" concept in type theory. -Nevertheless, by pairing a type with it's unique inhabitant, singleton types bridge the gap between -types and values, and their presence in Scala has over the years allowed Scala programmers to explore -techniques which would typically only be available in languages, such as Agda or Idris, with support +Nevertheless, by pairing a type with its unique inhabitant, singleton types bridge the gap between +types and values, and their presence in Scala has, over the years, allowed Scala programmers to explore +techniques which would typically only be available in languages such as Agda or Idris, with support for full-spectrum dependent types. Scala's semantics have up until now been richer than its syntax. The only singleton types which are currently _directly_ expressible are those of the form `p.type` where `p` is a path pointing to a value of some subtype of `AnyRef`. Internally the Scala compiler also represents singleton types for -individual values of subtypes of `AnyVal`, such as `Int` or values of type `String` which don't +individual values of subtypes of `AnyVal`, such as `Int` or values of type `String`, which don't correspond to paths. These types are inferred in some circumstances, notably as the types of `final` -vals. Their primary purpose has been to represent compile time constants (see [6.24 Constant +vals. Their primary purpose has been to represent compile-time constants (see [6.24 Constant Expressions](https://scala-lang.org/files/archive/spec/2.12/06-expressions.html#constant-expressions) and the discussion of "constant value definitions" in [4.1 Value Declarations and Definitions](https://scala-lang.org/files/archive/spec/2.12/04-basic-declarations-and-definitions.html#value-declarations-and-definitions)). @@ -89,15 +89,15 @@ Lightbend Scala compiler. foo(1: 1) // type ascription ``` -+ The `.type` singleton type forming operator can be applied to values of all subtypes of `Any`. - To prevent the compiler from widening our return type we assign to a final val. ++ The `.type` singleton-type-forming operator can be applied to values of all subtypes of `Any`. + To prevent the compiler from widening our return type, we assign to a final val. ``` def foo[T](t: T): t.type = t final val bar = foo(23) // result is bar: 23 ``` + The presence of an upper bound of `Singleton` on a formal type parameter indicates that - singleton types should be inferred for type parameters at call sites. To help see this + singleton types should be inferred for type parameters at call sites. To help see this, we introduce type constructor `Id` to prevent the compiler from widening our return type. ``` type Id[A] = A @@ -118,7 +118,7 @@ Lightbend Scala compiler. ``` + A `scala.ValueOf[T]` type class and corresponding `scala.Predef.valueOf[T]` operator has been - added yielding the unique value of types with a single inhabitant. + added, yielding the unique value of types with a single inhabitant. ``` def foo[T](implicit v: ValueOf[T]): T = v.value foo[13] // result is 13: Int @@ -129,13 +129,13 @@ Lightbend Scala compiler. Many of the examples below use primitives provided by the Scala generic programming library [shapeless](https://github.com/milessabin/shapeless/). It provides a `Witness` type class and a -family of Scala macro based methods and conversions for working with singleton types and shifting +family of Scala-macro-based methods and conversions for working with singleton types and shifting from the value to the type level and vice versa. One of the goals of this SIP is to enable Scala programmers to achieve similar results without having to rely on a third party library or fragile and non-portable macros. The relevant parts of shapeless are excerpted in [Appendix 1](#appendix-1--shapeless-excerpts). -Given the definitions there, some of forms summarized above can be expressed in current Scala, +Given the definitions there, some of the forms summarized above can be expressed in current Scala, ``` val wOne = Witness(1) val one: wOne.T = wOne.value // wOne.T is the type 1 @@ -147,13 +147,13 @@ foo[wOne.T] // result is 1: 1 "foo" ->> 23 // shapeless record field constructor // result type is FieldType["foo", Int] ``` -The syntax is awkward and hiding it from library users is challenging. Nevertheless they enable many +The syntax is awkward, and hiding it from library users is challenging. Nevertheless they enable many constructs which have proven valuable in practice. #### shapeless records shapeless models records as HLists (essentially nested pairs) of record values with their types -tagged with the singleton types of their keys. The library provides user friendly mechanisms for +tagged with the singleton types of their keys. The library provides user-friendly mechanisms for constructing record _values_, however it is extremely laborious to express the corresponding _types_. Consider the following record value, ``` @@ -165,7 +165,7 @@ val book = HNil ``` -Using shapeless and current Scala the following would be required to give `book` an explicit type +Using shapeless and current Scala, the following would be required to give `book` an explicit type annotation, ``` val wAuthor = Witness("author") @@ -241,20 +241,20 @@ val c: Int Refined Greater[w6.T] = a ^ ``` -Under this proposal we can express these refinements much more succinctly, +Under this proposal, we can express these refinements much more succinctly, ``` val a: Int Refined Greater[5] = 10 val b: Int Refined Greater[4] = a ``` -Type level predicates of this kind have proved to be useful in practice and are supported by modules +Type-level predicates of this kind have proved to be useful in practice and are supported by modules of a [number of important libraries](https://github.com/fthomas/refined#external-modules). Experience with those libraries has led to a desire to compute directly over singleton types, in -effect to lift whole term-level expressions to the type-level which has resulted in the development +effect to lift whole term-level expressions to the type level, which has resulted in the development of the [singleton-ops](https://github.com/fthomas/singleton-ops) library. singleton-ops is built -with Typelevel Scala which allows it to use literal types as discussed in this SIP. +with Typelevel Scala, which allows it to use literal types, as discussed in this SIP. ``` import singleton.ops._ @@ -279,7 +279,7 @@ singleton-ops is used by a number of libraries, most notably our next motivating [Libra](https://github.com/to-ithaca/libra) is a a dimensional analysis library based on shapeless, spire and singleton-ops. It support SI units at the type level for all numeric types. Like -singleton-ops Libra is built using Typelevel Scala and so is able to use literal types as discussed +singleton-ops, Libra is built using Typelevel Scala and so is able to use literal types, as discussed in this SIP. Libra allows numeric computations to be checked for dimensional correctness as follows, @@ -324,7 +324,7 @@ case class Residue[M <: Int](n: Int) extends AnyVal { } ``` -Given this definition we can work with modular numbers without any danger of mixing numbers with +Given this definition, we can work with modular numbers without any danger of mixing numbers with different moduli, ``` @@ -342,7 +342,7 @@ fiveModTen + fourModEleven ``` Also note that the use of `ValueOf` as an implicit argument of `+` means that the modulus does not -need to be stored along with the `Int` in the `Residue` value which could be beneficial in +need to be stored along with the `Int` in the `Residue` value, which could be beneficial in applications which work with large datasets. ### Proposal details @@ -360,7 +360,7 @@ applications which work with large datasets. | ‘(’ Types ‘)’ ``` - Examples, + Examples: ``` val one: 1 = 1 // val declaration def foo(x: 1): Option[1] = Some(x) // param type, type arg @@ -368,7 +368,7 @@ applications which work with large datasets. foo(1: 1) // type ascription ``` -+ The restriction that the singleton type forming operator `.type` can only be appended to ++ The restriction that the singleton-type-forming operator `.type` can only be appended to stable paths designating a value which conforms to `AnyRef` is dropped -- the path may now conform to `Any`. Section [3.2.1](https://scala-lang.org/files/archive/spec/2.12/03-types.html#singleton-types) of the SLS is @@ -385,7 +385,7 @@ applications which work with large datasets. > denoted by `p` (i.e., the value `v` for which `v eq p`). Where the path does not conform to > `scala.AnyRef` the type denotes the set consisting of only the value denoted by `p`. - Example, + Example: ``` def foo[T](t: T): t.type = t final val bar = foo(23) // result is bar: 23 @@ -471,7 +471,7 @@ applications which work with large datasets. > corresponding to a singleton-apt definition, or (2) The upper bound Ui of Ti conforms to > `Singleton`. - Example, + Example: ``` type Id[A] = A def wide[T](t: T): Id[T] = t @@ -483,17 +483,17 @@ applications which work with large datasets. Note that we introduce the type constructor `Id` simply to avoid widening of the return type. + A `scala.ValueOf[T]` type class and corresponding `scala.Predef.valueOf[T]` operator has been - added yielding the unique value of types with a single inhabitant. + added, yielding the unique value of types with a single inhabitant. Type inference allows us to infer a singleton type from a literal value. It is natural to want to be able to go in the other direction and infer a value from a singleton type. This latter capability was exploited in the motivating `Residue` example given earlier, and is widely relied - on in current Scala in uses of shapeless's records, and `LabelledGeneric` based type class + on in current Scala in uses of shapeless's records, and `LabelledGeneric`-based type class derivation. - Implicit resolution is Scala's mechanism for inferring values from types and in current Scala + Implicit resolution is Scala's mechanism for inferring values from types, and in current Scala, shapeless provides a macro-based materializer for instances of its `Witness` type class. This SIP - adds a directly compiler supported type class as a replacement, + adds a directly compiler-supported type class as a replacement: ``` final class ValueOf[T](val value: T) extends AnyVal @@ -502,20 +502,20 @@ applications which work with large datasets. Instances are automatically provided for all types with a single inhabitant, which includes literal and non-literal singleton types and `Unit`. - Example, + Example: ``` def foo[T](implicit v: ValueOf[T]): T = v.value foo[13] // result is 13: Int ``` - A method `valueOf` is also added to `scala.Predef` analogously to existing operators such as + A method `valueOf` is also added to `scala.Predef`, analogously to existing operators such as `classOf`, `typeOf` etc. ``` def valueOf[T](implicit vt: ValueOf[T]): T = vt.value ``` - Example, + Example: ``` object Foo valueOf[Foo.type] // result is Foo: Foo.type @@ -531,11 +531,11 @@ applications which work with large datasets. where the `TypePat` is a literal type is translated as a match against the subsuming non-singleton type followed by an equality test with the value corresponding to the literal type. - Where applied to literal types `isInstanceOf` is translated to a test against + Where applied to literal types, `isInstanceOf` is translated to a test against the subsuming non-singleton type and an equality test with the value corresponding to the literal type. - Examples, + Examples: ``` (1: Any) match { case one: 1 => true @@ -544,28 +544,28 @@ applications which work with large datasets. (1: Any).isInstanceOf[1] // result is true: Boolean ``` - Importantly, that doesn't include `asInstanceOf` as that is a user assertion to the compiler, with + Importantly, that doesn't include `asInstanceOf`, as that is a user assertion to the compiler, with the compiler inserting in the generated code just enough code for the underlying runtime to not give a `ValidationError`. The compiler should not, for instance, generate code such that an expression like `(1: Any).asInstanceOf[2]` would throw a `ClassCastException`. + Default initialization for vars with literal types is forbidden. - The default initializer for a var is already mandated to be it's natural zero element (`0`, - `false`, `null` etc.). This is inconsistent with the var being given a non-zero literal type, + The default initializer for a var is already mandated to be its natural zero element (`0`, + `false`, `null` etc.). This is inconsistent with the var being given a non-zero literal type: ``` var bad: 1 = _ ``` - Whilst we could, in principle, provide an implicit non-default initializer for cases such as these + Whilst we could, in principle, provide an implicit non-default initializer for cases such as these, it is the view of the authors of this SIP that there is nothing to be gained from enabling this - construction and that default initializer should be forbidden. + construction, and that default initializer should be forbidden. -## Follow on work from this SIP +## Follow-on work from this SIP Whilst the authors of this SIP believe that it stands on its own merits, we think that there are two -areas where follow on work is desirable, and one area where another SIP might improve the implementation of SIP-23. +areas where follow-on work is desirable, and one area where another SIP might improve the implementation of SIP-23. ### Infix and prefix types @@ -573,7 +573,7 @@ areas where follow on work is desirable, and one area where another SIP might im has emerged from the work on refined types and computation over singleton types mentioned in the motivation section above. -Once literal types are available it is natural to want to lift entire expressions to the type level +Once literal types are available, it is natural to want to lift entire expressions to the type level as is done already in libraries such as [singleton-ops](https://github.com/fthomas/singleton-ops). However, the precedence and associativity of symbolic infix _type constructors_ don't match the precedence and associativity of symbolic infix _value operators_, and prefix type constructors don't @@ -583,12 +583,12 @@ terms. ### Byte and short literals `Byte` and `Short` have singleton types, but lack any corresponding syntax either at the type or at the term level. -These types are important in libraries which deal with low level numerics and protocol implementation +These types are important in libraries which deal with low-level numerics and protocol implementation (see eg. [Spire](https://github.com/non/spire) and [Scodec](https://github.com/scodec/scodec)) and elsewhere, and the ability to, for instance, index a type class by a byte or short literal would be valuable. -A prototype of this syntax extension existed at an early stage in the development of Typelevel Scala +A prototype of this syntax extension existed at an early stage in the development of Typelevel Scala, but never matured. The possibility of useful literal types adds impetus. ### Opaque types @@ -610,7 +610,7 @@ would be elided, and the `valueOf[A]` method would be compiled to an identity fu ## Appendix 1 -- shapeless excerpts -Extracts from shapeless relevant to the motivating examples for this SIP, +Extracts from shapeless relevant to the motivating examples for this SIP: ``` trait Witness { From 41de7e77f978066c57abb5597427bba05b9d4ca5 Mon Sep 17 00:00:00 2001 From: Oron Port Date: Fri, 19 Jan 2024 20:42:49 +0200 Subject: [PATCH 02/27] update SIP-56 status to accepted --- content/match-types-spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/match-types-spec.md b/content/match-types-spec.md index ca20238e..6609d931 100644 --- a/content/match-types-spec.md +++ b/content/match-types-spec.md @@ -1,8 +1,8 @@ --- layout: sip permalink: /sips/:title.html -stage: implementation -status: waiting-for-implementation +stage: completed +status: accepted title: SIP-56 - Proper Specification for Match Types --- From cf1fb75c7df49ad783cadbe88cc94a0cc54a33f6 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Wed, 14 Feb 2024 19:27:40 +0800 Subject: [PATCH 03/27] wip . . . --- content/unroll-default-arguments.md | 755 ++++++++++++++++++++++++++++ 1 file changed, 755 insertions(+) create mode 100644 content/unroll-default-arguments.md diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md new file mode 100644 index 00000000..7552a940 --- /dev/null +++ b/content/unroll-default-arguments.md @@ -0,0 +1,755 @@ +--- +layout: sip +permalink: /sips/:title.html +stage: completed +status: under-review +title: SIP-61 - Unroll Default Arguments for Binary Compatibility +--- + +**By: Li Haoyi** + +## History + +| Date | Version | +|---------------|--------------------| +| Feb 14th 2024 | Initial Draft | + +## Summary + +This SIP proposes an `@unroll` annotation lets you add additional parameters +to method `def`s,`class` construtors, or `case class`es, without breaking binary +compatibility. `@unroll` works by generating "unrolled" or "telescoping" forwarders: + +```scala +// Original +def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + +// Generated +def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) +def foo(s: String, n: Int) = foo(s, n, true, 0) +``` + +In contrast to most existing or proposed alternatives that require you to contort your +code to become binary compatible (see [Major Alternatives](#major-alternatives)), +`@unroll` allows you to write Scala in the most straightforward way. You add +a single annotation, and your `def`/`class`/`case class` will maintain binary +compatibility as new default parameters and fields are added over time. + +`@unroll`'s only constraints are that: + +1. New parameters need to have a default value +2. New parameters can only be added on the right + +These are both existing industry-wide standard when dealing with data and schema evolution +(e.g. [Schema evolution in Avro, Protocol Buffers and Thrift — Martin Kleppmann’s blog](https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html)), +and are also the way the new parameters interact with _source compatibility_ in +the Scala language. Thus these constraints should be immediately familiar to any +experienced programmers, and would be easy to follow without confusion. + +Prior Discussion can be found [here](https://contributors.scala-lang.org/t/can-we-make-adding-a-parameter-with-a-default-value-binary-compatible/6132) + +## Motivation + +Maintaining binary compatibility of Scala libraries as they evolve over time is +difficult. Although tools like https://github.com/lightbend/mima help _surface_ +issues, actually _resolving_ those issues is a different challenge. + +Some kinds of library changes are fundamentally impossible to make compatible, +e.g. removing methods or classes. But there is one big class of binary compatibility +issues that are "spurious": adding default parameters to methods, `class` constructors, +or `case class`es. + +Adding a default parameter is source-compatible, but not binary compatible: a user +downstream of a library that adds a default parameter does not need to make any +changes to their code, but _does_ need to re-compile it. This is "spurious" because +there is no _fundamental_ incompatibility here: semantically, a new default parameter +is meant to be optional! Old code invoking that method without a new default parameter +is exactly the user intent, and works just fine if the downstream code is re-compiled. + +Other languages, such as Python, have the same default parameter language feature but face +no such compatibility issues with their use. Even Scala codebases compiled from source +(e.g. in a mono-repo setup) do not suffer these restrictions: adding a default parameter +to the right side of a parameter list is for all intents and purposes backwards compatible. +The fact that such addition is binary incompatible is purely an implementation restriction +of Scala's binary artifact format and distribution strategy. + +**Binary compatibility is generally more important than Source compatibility**. When +you hit a source compatibility issue, you can always change the source code you are +compiling, whether manually or via your build tool. In contrast, when you hit binary +compatibility issues, it can come in the form of diamond dependencies that would require +_re-compiling all of your transitive dependencies_, a task that is far more difficult and +often impractical. + +There are many approaches to resolving these "spurious" binary compatibility issues, +but most of them involve either tremendous amounts of boilerplate writing +binary-compatibility forwarders, giving up on core language features like Case Classes +or Default Parameters, or both. Consider the following code snippet +([link](https://github.com/com-lihaoyi/mainargs/blob/1d04a6bd19aaca401d11fe26da31615a8bc9213c/mainargs/src/Parser.scala)) +from the [com-lihaoyi/mainargs](https://github.com/com-lihaoyi/mainargs) library, which +duplicates the parameters of `def constructEither` no less than five times in +order to maintain binary compatibility as the library evolves and more default +parameters are added to `def constructEither`: + +```scala + def constructEither( + args: Seq[String], + allowPositional: Boolean, + allowRepeats: Boolean, + totalWidth: Int, + printHelpOnExit: Boolean, + docsOnNewLine: Boolean, + autoPrintHelpAndExit: Option[(Int, PrintStream)], + customName: String, + customDoc: String, + sorted: Boolean, + ): Either[String, T] = constructEither( + args, + allowPositional, + allowRepeats, + totalWidth, + printHelpOnExit, + docsOnNewLine, + autoPrintHelpAndExit, + customName, + customDoc, + sorted, + ) + + def constructEither( + args: Seq[String], + allowPositional: Boolean = false, + allowRepeats: Boolean = false, + totalWidth: Int = 100, + printHelpOnExit: Boolean = true, + docsOnNewLine: Boolean = false, + autoPrintHelpAndExit: Option[(Int, PrintStream)] = Some((0, System.out)), + customName: String = null, + customDoc: String = null, + sorted: Boolean = true, + nameMapper: String => Option[String] = Util.kebabCaseNameMapper + ): Either[String, T] = ??? + + /** binary compatibility shim. */ + private[mainargs] def constructEither( + args: Seq[String], + allowPositional: Boolean, + allowRepeats: Boolean, + totalWidth: Int, + printHelpOnExit: Boolean, + docsOnNewLine: Boolean, + autoPrintHelpAndExit: Option[(Int, PrintStream)], + customName: String, + customDoc: String, + nameMapper: String => Option[String] + ): Either[String, T] = constructEither( + args, + allowPositional, + allowRepeats, + totalWidth, + printHelpOnExit, + docsOnNewLine, + autoPrintHelpAndExit, + customName, + customDoc, + sorted = true, + nameMapper = nameMapper + ) +``` + +Apart from being extremely verbose and full of boilerplate, like any boilerplate this is +also extremely error-prone. Bugs like [com-lihaoyi/mainargs#106](https://github.com/com-lihaoyi/mainargs/issues/106) +slip through when a mistake is made in that boilerplate. These bugs are impossible to catch +using a normal test suite, as they only appear in the presence of version skew. The above code +snippet actually _does_ have such a bug, that the test suite _did not_ catch. See if you can +spot it! + +Sebastien Doraene's talk [Designing Libraries for Source and Binary Compatibility](https://www.youtube.com/watch?v=2wkEX6MCxJs) +explores some of the challenges, and discusses the workarounds. + +## Proposed solution + + +The proposed solution is to provide a `scala.annotation.unroll` annotation, that +can be applied to methods `def`s, `class` constructors, or `case class`es to generate +"unrolled" or "telescoping" versions of a method that forward to the primary implementation: + +```scala + def constructEither( + args: Seq[String], + allowPositional: Boolean = false, + allowRepeats: Boolean = false, + totalWidth: Int = 100, + printHelpOnExit: Boolean = true, + docsOnNewLine: Boolean = false, + autoPrintHelpAndExit: Option[(Int, PrintStream)] = Some((0, System.out)), + customName: String = null, + customDoc: String = null, + @unroll sorted: Boolean = true, + nameMapper: String => Option[String] = Util.kebabCaseNameMapper + ): Either[String, T] = ??? +``` + +This allows the developer to write the minimal amount of code they _want_ to write, +and add a single annotation to allow binary compatibility to old versions. In this +case, we annotated `sorted` with `@unroll`, which generates forwarders that make +`def constructEither` binary compatible with older versions that have fewer parameters, +up to a version before `sorted` was added. Any existing method `def`, `class`, or +`case class` can be evolved in this way, by addition of `@unroll` the first time +a default argument is added to their signature after its initial definition. + +### Unrolling `def`s + +Consider a library that is written as follows: + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1) = s + n + b + l +} +``` + +If over time a new default parameter is added: + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b + l +} +``` + +And another + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, b: Boolean = true, l: Long = 0) = s + n + b + l +} +``` + +This is a source-compatible change, but not binary-compatible: JVM bytecode compiled against an +earlier version of the library would be expecting to call `def foo(String, Int)`, but will fail +because the signature is now `def foo(String, Int, Boolean)` or `def foo(String, Int, Boolean, Long)`. +On the JVM this will result in a `MethodNotFoundError` at runtime, a common experience for anyone +who upgrading the versions of their dependencies. Similar concerns are present with Scala.js and +Scala-Native, albeit the failure happens at link-time rather than run-time + +`@unroll` is an annotation that can be applied as follows, to the first "additional" default +parameter that was added (in this case, `b: Boolean = true`) + + +```scala +import scala.annotation.unroll + +object Unrolled{ + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l +} +``` + +The `@unroll` annotation takes `def foo` and generates synthetic forwarders for the purpose +of maintaining binary compatibility for old callers who may be expecting the previous signature. +These forwarders do nothing but forward the call to the current implementation, using the +given default parameter values: + +```scala +import scala.annotation.unroll + +object Unrolled{ + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + + def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) + def foo(s: String, n: Int) = foo(s, n, true, 0) +} +``` + +As a result, old callers who expect `def foo(String, Int, Boolean)` or `def foo(String, Int, Boolean, Long)` +can continue to work, even as new parameters are added to `def foo`. The only restriction is that +new parameters can only be added on the right, and they must be provided with a default value. + +If there are multiple parameter lists (e.g. for curried methods or methods taking implicits) only one +parameter list can be unrolled (though it does not need to be the first one). e.g. this works: + +```scala +object Unrolled{ + def foo(s: String, + n: Int = 1, + @unroll b: Boolean = true, + l: Long = 0) + (implicit blah: Blah) = s + n + b + l +} +``` + +As does this + +```scala +object Unrolled{ + def foo(blah: Blah) + (s: String, + n: Int = 1, + @unroll b: Boolean = true, + l: Long = 0) = s + n + b + l +} +``` + +`@unroll`ed methods can be defined in `object`s, `class`es, or `trait`s. Other cases are shown below. + +### Unrolling `class`es + +Class constructors and secondary constructors are treated by `@unroll` just like any +other method: + +```scala +import scala.annotation.unroll + +class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ + def foo = s + n + b + l +} +``` + +Unrolls to: + +```scala +import scala.annotation.unroll + +class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ + def foo = s + n + b + l + + def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0) + def this(s: String, n: Int) = this(s, n, true, 0) +} +``` + +### Unrolling `class` secondary constructors + +```scala +import scala.annotation.unroll + +class Unrolled() { + var foo = "" + + def this(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = { + this() + foo = s + n + b + l + } +} +``` + +Unrolls to: + +```scala +import scala.annotation.unroll + +class Unrolled() { + var foo = "" + + def this(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = { + this() + foo = s + n + b + l + } + + def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0) + def this(s: String, n: Int) = this(s, n, true, 0) +} +``` + +### Case Classes + +`case class`es can also be `@unroll`ed. Unlike normal `class` constructors +and method `def`s, `case class`es have several generated methods (`apply`, `copy`) +that need to be kept in sync with their primary constructor. `@unroll` thus +generates forwarders for those methods as well, based on the presence of the +`@unroll` annotation in the primary constructor: + +```scala +import scala.annotation.unroll + +case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true){ + def foo = s + n + b +} +``` + +Unrolls to: + +```scala +import scala.annotation.unroll + +case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0L){ + def this(s: String, n: Int) = this(s, n, true, 0L) + def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0L) + + def copy(s: String, n: Int) = copy(s, n, true, 0L) + def copy(s: String, n: Int, b: Boolean) = copy(s, n, b, 0L) + + def foo = s + n + b +} +object Unrolled{ + def apply(s: String, n: Int) = apply(s, n, true, 0L) + def apply(s: String, n: Int, b: Boolean) = apply(s, n, b, , 0L) +} +``` + +Notes: + +1. `.unapply` does not need to be duplicated in Scala 3.x, as its signature + `def unapply(x: Unrolled): Unrolled` does not change when new `case class` fields are + added. + +2. Even in Scala 2.x, where `def unapply(x: Unrolled): Option[TupleN]` is not + binary compatible, pattern matching on `case class`es is already binary compatible + to addition of new fields due to + [Option-less Pattern Matching](https://docs.scala-lang.org/scala3/reference/changed-features/pattern-matching.html). + Thus, only direct calls to `.unapply` on an unrolled `case class` in Scala 2.x (shown below) + will cause a crash if additional fields were added. + +```scala +def foo(t: (String, Int)) = println(t) +Unrolled.unapply(unrolled).map(foo) +``` + +In Scala 3, `@unroll`ing a `case class` also needs to generate a `fromProduct` +implementation in the companion object, as shown below: + +```scala +def fromProduct(p: Product): CaseClass = p.productArity match + case 2 => + CaseClass( + p.productElement(0).asInstanceOf[...], + p.productElement(1).asInstanceOf[...], + ) + case 3 => + CaseClass( + p.productElement(0).asInstanceOf[...], + p.productElement(1).asInstanceOf[...], + p.productElement(2).asInstanceOf[...], + ) + ... +``` + +This is not necessary for preserving binary compatibility - the method signature of +`def fromProduct` does not change depending on the number of fields - but it is +necessary to preserve semantic compatibility. `fromProduct` by default does not +take into account field default values, and this change is necessary to make it +use them when the given `p: Product` has a smaller `productArity` than the current +`CaseClass` implementation + + +## Limitations + +1. Only the one parameter list of multi-parameter list methods (i.e. curried or taking + implicits) can be `@unroll`ed. Unrolling multiple parameter lists would generate a number + of forwarder methods exponential with regard to the number of parameter lists unrolled, + and the generated forwarders may begin to conflict with each other. We can choose to spec + this out and implement it later if necessary, but for 99% of use cases `@unroll`ing one + parameter list should be enough. Typically, only one parameter list in a method has default + arguments, with other parameter lists being `implicit`s or a single callback/blocks, neither + of which usually has default values. + +2. As unrolling generates synthetic forwarder methods for binary compatibility, it is + possible for them to collide if your unrolled method has manually-defined overloads + +3. As mentioned earlier, `@unroll`ed case classes are only fully binary compatible in Scala 3, + though they are _almost_ binary compatible in Scala 2. Direct calls to `unapply` are binary + incompatible, but most common pattern matching of `case class`es goes through a different + code path that _is_ binary compatible. In practice this should be sufficient for 99% of use + cases, but it does mean that it is possible for code written as below to fail in Scala 2 + if a new unrolled parameter is added to the case class and `.unapply` is called directly. + +4. While `@unroll`ed `case class`es are fully binary compatible, they are *not* fully + _source_ compatible, due to the fact that pattern matching requires all arguments to + be specified. This proposal does not change that. Future improvements related to + [Pattern Matching on Named Fields](https://github.com/scala/improvement-proposals/pull/44) + may bring improvements here. But as we discussed earlier, binary compatibility is generally + more important than source compatibility, and so we do not need to wait for any source + compatibility improvements to land before proceeding with these binary compatibility + improvements. + +5. This proposal does not address how macros will derive typeclasses for `case class`es, and + whether or not those will be binary/source/semantically compatible. That is up to the + individual macro implementations to decide. e.g., [uPickle](https://github.com/com-lihaoyi/upickle) + has a very similar rule about adding `case class` fields, except that field ordering + does not matter. Trying to standardize this across all possible macros and all possible + typeclasses is out of scope + +6. `@unroll` only supports `final` methods. `object` methods and constructors are naturally + final, but `class` or `trait` methods that are `@unroll`ed need to be explicitly marked `final`. + It has proved difficult to implement the semantics of `@unroll` in the presence of downstream + overrides, `super`, etc. where the downstream overrides can be compiled against by different + versions of the upstream code. If we can come up with some implementation that works, we can + lift this restriction later, but for now I have not managed to do so and so this restriction + stays. + +7. `@unroll` generates a quadratic amount of generated bytecode as more default parameters + are added: each forwarder has `O(num-params)` size, and there are `O(num-default-params)` + forwarders. We do not expect this to be a problem in practice, as the small size of the + generated forwarder methods means the constant factor is small, but one could imagine + the `O(n^2)` asymptotic complexity becoming a problem if a method accumulates hundreds of + default parameters over time. In such extreme scenarios, some kind of builder pattern + (such as those listed in [Major Alternatives](#major-alternatives)) may be preferable. + + +## Major Alternatives + +The major alternatives to `@unroll` are listed below: + +1. [data-class](https://index.scala-lang.org/alexarchambault/data-class) +2. [SBT Datatype](https://www.scala-sbt.org/1.x/docs/Datatype.html) +3. [Structural Data Structures](https://contributors.scala-lang.org/t/pre-sip-structural-data-structures-that-can-evolve-in-a-binary-compatible-way/5684) +4. Avoiding language features like `case class`es or default parameters, as suggested by the + [Binary Compatibility for Library Authors](https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html) documentation. + +While those alternate approaches _do work_ - `data-class` and `SBT Datatype` are used heavily +in various open-source projects - I believe they are inferior to the approach that `@unroll` +takes: + +### Case Class v.s. not-a-Case-Class + +The first major difference between `@unroll` and the above alternatives is that these alternatives +all introduce something new: some kind of _not-a-case-class_ `class` that is to be used +when binary compatibility is desired. This _not-a-case-class_ has different syntax from +`case class`es, different semantics, different methods, and so on. + +In contrast, `@unroll` does not introduce any new language-level or library-level constructs. +The `@unroll` annotation is purely a compiler-backend concern for maintaining binary +compatibility. At a language level, `@unroll` allows you to keep using normal method `def`s, +`class`es and `case class`es with exactly the same syntax and semantics you have been using +all along. + +Having people be constantly choosing between _case-class_ and _not-a-case-class_ when +designing their data types, is inferior to simply using `case class`es all the time + + +### Scala Syntax v.s. Java-esque Syntax + + +The alternatives linked above all build a +Java-esque "[inner platform](https://en.wikipedia.org/wiki/Inner-platform_effect)" +on top of the Scala language, with its own conventions like `.withFoo` methods. + +In contrast, `@unroll` makes use of the existing Scala language's default parameters +to achieve the same effect. + +If we think Scala is nicer to write then Java due to its language +features, then `@unroll`'s approach of leveraging those language features is nicer +to use than the alternative's Java-esque syntax. + +Having implementation-level problems - which is what binary compatibility across version +skew is - bleed into the syntax and semantics of the language is also inferior to having it +be controlled by an annotation. Martin Odersky has said that annotations are intended for +things that do not affect typechecking, and `@unroll` fits the bill perfectly. + + +### Evolving Any Class v.s. Evolving Pre-determined Classes + +The alternatives given require that the developer has to decide _up front_ whether their +data type needs to be evolved while maintaining binary compatibility. + +In contrast, `@unroll` allows you to evolve any existing `class` or `case class`. + +In general, trying to decide which classes _will need to evolve later on_ is a difficult +task that is easy to get wrong. `@unroll` totally removes that requirement, allowing +you to take _any_ `class` or `case class` and evolve it later in a binary compatible way. + + +### Binary Compatibility for Methods and Classes + +Lastly, the above alternatives only solve _half_ the problem: how to evolve `case class`es. + +In contrast, `@unroll` allows the evolution of `def`s and normal `class`es, in addition +to `case class`es, all using the same approach. + +Binary compatility is not just a problem for `case class`es adding new fields: normal +`class` constructors, instance method `def`s, static method `def`s, etc. have default +parameters added all the time as well. `@unroll` solves all these problems at once, +using the same implementation and same user-facing semantics. + + +## Minor Alternatives: + + +### `@unrollOnly` + +Currently, `@unroll` generates forwarders for every default parameter to the right of the one +annotated. This is not always necessary, e.g. if multiple parameters are added at once, we +should only need a single forwarder for the entire set. This can be supported by requiring +supporting an `@unrollOnly` is provided for every default parameter that needs a forwarder generated. That +would mean generating two forwarders would look like this: + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, @unrollOnly l: Long = 0) = s + n + b + l +} +``` +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, @unrollOnly l: Long = 0) = s + n + b + l + + def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) + def foo(s: String, n: Int) = foo(s, n, true, 0) +} +``` + +And generating one forwarder would look like this: + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, l: Long = 0) = s + n + b + l +} +``` +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, l: Long = 0) = s + n + b + l + + def foo(s: String, n: Int) = foo(s, n, true, 0) +} +``` + +`@unrollOnly` would provide a more granular replacement for `@unroll`. This probably does not +make a huge difference in most cases, but it could be useful in scenarios where there +are a large number of default parameters being added every version, as it would minimize the +amount of generated code. + + +### Abstract and Virtual Methods + +In [Limitations](#limitations), I mentioned that `@unroll` only supports `final` methods. +It is likely possible for abstract methods which are `@unrolled` to have concrete forwarder +methods generated on their behalf. + +```scala +import scala.annotation.unroll + +trait Unrolled{ + def foo(s: String, n: Int = 1, @unroll b: Boolean = true): String +} + +object Unrolled extends Unrolled{ + def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b +} +``` + +Unrolls to: + +```scala +trait Unrolled{ + def foo(s: String, n: Int = 1, @unroll b: Boolean = true): String = foo(s, n) + def foo(s: String, n: Int = 1): String = foo(s, n, true) +} + +object Unrolled extends Unrolled{ + def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b +} +``` + +As the forwarders are concrete, the implementor of the abstract method does +not need to `@unroll` the implementation: they only need to provide the implementation +for the primary method `def` and not the forwarders. + +One thing to note is that the `@unroll`ed abstract method needs to _itself_ become a +forwarder method, despite originally being abstract! That is because downstream code +compiled against an old version may define classes which `extends Unrolled` and +define a concrete `def foo(s: String, n: Int = 1): String`, while other downstream code compiled +against a newer version may define a concrete +`def foo(s: String, n: Int = 1, b: Boolean = true): String`. Thus, we need both overloads +of `foo` to be forwarders, so that downstream code can override either version and still work. + +This handling for abstract methods is not fully fleshed out or implemented, so I'm +not sure if it can truly be made to work. + +### Should the Generated Methods be Deprecated or Invisible? + +It is not clear to me if we should discourage usage of the generated forwarders +methods, or hide them: + +1. On one hand, downstream code should not be compiling against these methods: they are + purely for binary compatibility + +2. On the other hand, downstream code does not control which overload of the method is + selected: that is up to the Scala compiler and how overloading interacts with default + parameter values. I have encountered scenarios with manually-written forwarders where + selected over the "primary" implementation + +3. The forwarders are meant to have the exact same semantics as the "primary" implementation. + Thus it _does not matter_ whether you call the primary and pass it a default value, + or you call a forwarder which then calls the primary and passes it the same default value. + +For now, I have left the generated methods "as is", though choosing to hide them or deprecate +them or something else is definitely an option + +### Generating Forwarders For Parameter Type Widening or Result Type Narrowing + +While this proposal focuses on generating forwarders for addition of default parameters, +you can also imagine similar forwarders being generated if method parameter types +are widened or if result types are narrowed: + +```scala +// Before +def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b + l + +// After +def foo(@unrollType[String] s: Object, n: Int = 1, b: Boolean = true) = s.toString + n + b + l + +// Generated +def foo(s: Object, n: Int = 1, b: Boolean = true) = s.toString + n + b + l +def foo(s: String, n: Int = 1, b: Boolean = true) = foo(s, n, b) +``` + +This would follow the precedence of how Java's and Scala's covariant method return +type overrides are implemented: when a class overrides a method with a new +implementation with a narrower return type, a forwarder method is generated to +allow anyone calling the original signature \to be forwarded to the narrower signature. + +This is not currently implemented in `@unroll`, but would be a straightforward addition. + +### Incremental Forwarders or Direct Forwarders + +Given this: + +```scala +def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l +``` + +There are two ways to do the forwarders. First option, which I used in above, is +to have each forwarder directly call the primary method: + +```scala +def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) +def foo(s: String, n: Int) = foo(s, n, true, 0) +``` + +Second option is to have each forwarder incrementally call the next forwarder, which +will eventually end up calling the primary method: + +```scala +def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) +def foo(s: String, n: Int) = foo(s, n, true) +``` + +The first option results in shorter stack traces, while the second option results in +roughly half as much generated bytecode in the method bodies (though it's still `O(n^2)`). + +For now I chose the first option. + + +## Implementation & Testing + +This SIP has a full implementation for Scala {2.12, 2.13, 3} X {JVM, JS, Native} +in the following repository, as a compiler plugin: + +- https://github.com/com-lihaoyi/unroll + +As the `@unroll` annotation is purely a compile-time construct and does not need to exist +at runtime, `@unroll` can be added to Scala 2.13.x without breaking forwards compatibility. + +The linked repo also contains an extensive test suite that uses both MIMA as well +as classpath-mangling to validate that it provides both the binary and semantic +compatibility benefits claimed in this document. In fact, it has even discovered +bugs in the upstream Scala implementation related to binary compatibility, e.g. +[scala-native/scala-native#3747](https://github.com/scala-native/scala-native/issues/3747) + +I have also opened pull requests to a number of popular OSS Scala libraries, +using `@unroll` as a replacement for manually writing binary compatibility stubs, +and the 100s of lines of boilerplate reduction can be seen in the links below: + +- https://github.com/com-lihaoyi/mainargs/pull/113/files +- https://github.com/com-lihaoyi/mill/pull/3008/files +- https://github.com/com-lihaoyi/upickle/pull/555/files + +These pull requests all pass both the test suite as well as the MIMA +`check-binary-compatibility` job, demonstrating that this approach does work +in real-world codebases. From 870aab6c181184bae789abc604771c96c3e068d4 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 15 Feb 2024 21:53:44 +0800 Subject: [PATCH 04/27] fix --- content/unroll-default-arguments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index a319a6d5..2ebed547 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -380,7 +380,7 @@ case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = } object Unrolled{ def apply(s: String, n: Int) = apply(s, n, true, 0L) - def apply(s: String, n: Int, b: Boolean) = apply(s, n, b, , 0L) + def apply(s: String, n: Int, b: Boolean) = apply(s, n, b, 0L) } ``` From c3ae803684d3584bb0781aa6525c34aaaf3b6761 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 15 Feb 2024 22:50:14 +0800 Subject: [PATCH 05/27] fix --- content/unroll-default-arguments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 2ebed547..8cd00f02 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -373,8 +373,8 @@ case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = def this(s: String, n: Int) = this(s, n, true, 0L) def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0L) - def copy(s: String, n: Int) = copy(s, n, true, 0L) - def copy(s: String, n: Int, b: Boolean) = copy(s, n, b, 0L) + def copy(s: String, n: Int) = copy(s, n, this.b, this.l) + def copy(s: String, n: Int, b: Boolean) = copy(s, n, b, this.l) def foo = s + n + b } From 47da9eaf9a4034c31f80120b6aa73f0fdd1177b0 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Fri, 16 Feb 2024 20:48:51 +0800 Subject: [PATCH 06/27] @unroll/@unrollAll --- content/unroll-default-arguments.md | 89 +++++++++++------------------ 1 file changed, 34 insertions(+), 55 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 8cd00f02..0db33bae 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -185,7 +185,7 @@ can be applied to methods `def`s, `class` constructors, or `case class`es to gen customName: String = null, customDoc: String = null, @unroll sorted: Boolean = true, - nameMapper: String => Option[String] = Util.kebabCaseNameMapper + @unroll nameMapper: String => Option[String] = Util.kebabCaseNameMapper ): Either[String, T] = ??? ``` @@ -238,7 +238,7 @@ parameter that was added (in this case, `b: Boolean = true`) import scala.annotation.unroll object Unrolled{ - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l } ``` @@ -248,10 +248,8 @@ These forwarders do nothing but forward the call to the current implementation, given default parameter values: ```scala -import scala.annotation.unroll - object Unrolled{ - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) def foo(s: String, n: Int) = foo(s, n, true, 0) @@ -262,6 +260,18 @@ As a result, old callers who expect `def foo(String, Int, Boolean)` or `def foo( can continue to work, even as new parameters are added to `def foo`. The only restriction is that new parameters can only be added on the right, and they must be provided with a default value. +If multiple default parameters are added at once (e.g. `b` and `l` below) you can also +choose to only `@unroll` the first default parameter of each batch, to avoid generating +unnecessary forwarders: + +```scala +object Unrolled{ + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + + def foo(s: String, n: Int) = foo(s, n, true, 0) +} +``` + If there are multiple parameter lists (e.g. for curried methods or methods taking implicits) only one parameter list can be unrolled (though it does not need to be the first one). e.g. this works: @@ -269,8 +279,8 @@ parameter list can be unrolled (though it does not need to be the first one). e. object Unrolled{ def foo(s: String, n: Int = 1, - @unroll b: Boolean = true, - l: Long = 0) + @unroll b: Boolean = true, + @unroll l: Long = 0) (implicit blah: Blah) = s + n + b + l } ``` @@ -282,8 +292,8 @@ object Unrolled{ def foo(blah: Blah) (s: String, n: Int = 1, - @unroll b: Boolean = true, - l: Long = 0) = s + n + b + l + @unroll b: Boolean = true, + @unroll l: Long = 0) = s + n + b + l } ``` @@ -295,9 +305,7 @@ Class constructors and secondary constructors are treated by `@unroll` just like other method: ```scala -import scala.annotation.unroll - -class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ +class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0){ def foo = s + n + b + l } ``` @@ -305,9 +313,7 @@ class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ Unrolls to: ```scala -import scala.annotation.unroll - -class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ +class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0){ def foo = s + n + b + l def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0) @@ -318,12 +324,10 @@ class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0){ ### Unrolling `class` secondary constructors ```scala -import scala.annotation.unroll - class Unrolled() { var foo = "" - def this(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = { + def this(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = { this() foo = s + n + b + l } @@ -333,12 +337,10 @@ class Unrolled() { Unrolls to: ```scala -import scala.annotation.unroll - class Unrolled() { var foo = "" - def this(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = { + def this(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = { this() foo = s + n + b + l } @@ -357,8 +359,6 @@ generates forwarders for those methods as well, based on the presence of the `@unroll` annotation in the primary constructor: ```scala -import scala.annotation.unroll - case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true){ def foo = s + n + b } @@ -367,9 +367,7 @@ case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true){ Unrolls to: ```scala -import scala.annotation.unroll - -case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0L){ +case class Unrolled(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0L){ def this(s: String, n: Int) = this(s, n, true, 0L) def this(s: String, n: Int, b: Boolean) = this(s, n, b, 0L) @@ -490,7 +488,7 @@ against different versions of each other (hence the varying number of parameters ```scala class Upstream{ // V2 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l } ``` @@ -624,48 +622,29 @@ using the same implementation and same user-facing semantics. ## Minor Alternatives: -### `@unrollOnly` +### `@unrollAll` -Currently, `@unroll` generates forwarders for every default parameter to the right of the one -annotated. This is not always necessary, e.g. if multiple parameters are added at once, we -should only need a single forwarder for the entire set. This can be supported by requiring -supporting an `@unrollOnly` is provided for every default parameter that needs a forwarder generated. That -would mean generating two forwarders would look like this: +Currently, `@unroll` generates a forwarder only for the annotated default parameter; +if you want to generate multiple forwarders, you need to `@unroll` each one. In the +vast majority of scenarios, we want to unroll every default parameters we add, and in +many cases default parameters are added one at a time. In this case, an `@unrollAll` +annotation may be useful, a shorthand for applying `@unroll` to the annotated default +parameter and every parameter to the right of it: ```scala object Unrolled{ - def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, @unrollOnly l: Long = 0) = s + n + b + l + def foo(s: String, n: Int = 1, @unrollAll b: Boolean = true, l: Long = 0) = s + n + b + l } ``` ```scala object Unrolled{ - def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, @unrollOnly l: Long = 0) = s + n + b + l + def foo(s: String, n: Int = 1, b: Boolean = true, l: Long = 0) = s + n + b + l def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) def foo(s: String, n: Int) = foo(s, n, true, 0) } ``` -And generating one forwarder would look like this: - -```scala -object Unrolled{ - def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, l: Long = 0) = s + n + b + l -} -``` -```scala -object Unrolled{ - def foo(s: String, n: Int = 1, @unrollOnly b: Boolean = true, l: Long = 0) = s + n + b + l - - def foo(s: String, n: Int) = foo(s, n, true, 0) -} -``` - -`@unrollOnly` would provide a more granular replacement for `@unroll`. This probably does not -make a huge difference in most cases, but it could be useful in scenarios where there -are a large number of default parameters being added every version, as it would minimize the -amount of generated code. - ### Abstract Methods From 2dfee7fbd0cbff06a265678274b0eb2a918028fd Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Fri, 16 Feb 2024 21:01:50 +0800 Subject: [PATCH 07/27] . --- content/unroll-default-arguments.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 0db33bae..851c437c 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -231,7 +231,8 @@ who upgrading the versions of their dependencies. Similar concerns are present w Scala-Native, albeit the failure happens at link-time rather than run-time `@unroll` is an annotation that can be applied as follows, to the first "additional" default -parameter that was added (in this case, `b: Boolean = true`) +parameter that was added in each published version of the library (in this case, +`b: Boolean = true` and `l: Long = 0`) ```scala From 1f2cedc8270365ea749ee9f709afa1501029c2a9 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 17 Feb 2024 09:37:57 +0800 Subject: [PATCH 08/27] contraband --- content/unroll-default-arguments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 851c437c..2d57fdae 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -549,7 +549,7 @@ are implemented only once by a final method. See the section about The major alternatives to `@unroll` are listed below: 1. [data-class](https://index.scala-lang.org/alexarchambault/data-class) -2. [SBT Datatype](https://www.scala-sbt.org/1.x/docs/Datatype.html) +2. [SBT Contrabad](https://www.scala-sbt.org/contraband/) 3. [Structural Data Structures](https://contributors.scala-lang.org/t/pre-sip-structural-data-structures-that-can-evolve-in-a-binary-compatible-way/5684) 4. Avoiding language features like `case class`es or default parameters, as suggested by the [Binary Compatibility for Library Authors](https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html) documentation. From 05b0bdc6b581d5fbdc09943973f9b6577eabe2d9 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:43:51 +0800 Subject: [PATCH 09/27] add spec for abstract methods --- content/unroll-default-arguments.md | 313 +++++++++++++++++++--------- 1 file changed, 215 insertions(+), 98 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 2d57fdae..5a860019 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -166,6 +166,36 @@ spot it! Sebastien Doraene's talk [Designing Libraries for Source and Binary Compatibility](https://www.youtube.com/watch?v=2wkEX6MCxJs) explores some of the challenges, and discusses the workarounds. + +## Requirements + +### Backwards Compatibility + +Given: + +* Two libraries, **Upstream** and **Downstream**, where **Downstream** depends on **Upstream** + +* If we use a _newer_ version of **Upstream** which contains an added + default parameter together with an _older_ version of **Downstream** compiled + against an _older_ version of **Upstream** before that default parameter was added + +* The behavior should be binary compatible and semantically indistinguishable from using + a verion of **Downstream** compiled against the _newer_ version of **Upstream** + +**Note:** we do not aim for _Forwards_ compatibility. Using an _older_ +version of **Upstream** with a _newer_ version of **Downstream** compiled against a +_newer_ version of **Upstream** is not a use case we want to support. The vast majority +of OSS software does not promise forwards compatibility, including software such as +the JVM, so we should just follow suite + +### All Overrides Are Equivalent + +All versions of an `@unroll`ed method `def foo` should have the same semantics when called +with the same parameters. + +**Note:** this includes forwarder methods that may never get called in the scenarios +required by our [Backwards Compatibility](#backwards-compatibility) requirement. + ## Proposed solution @@ -427,59 +457,187 @@ take into account field default values, and this change is necessary to make it use them when the given `p: Product` has a smaller `productArity` than the current `CaseClass` implementation +### Abstract Methods + +Apart from `final` methods, `@unroll` also supports purely abstract methods. Consider +the following example with a trait `Unrolled` and an implementation `UnrolledObj`: + +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b +} +``` + +This unrolls to: +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) + def foo(s: String, n: Int, b: Boolean): String = foo(s, n) + def foo(s: String, n: Int): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l + def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) + def foo(s: String, n: Int) = foo(s, n, true) +} +``` + +Note that both the abstract methods from `trait Unrolled` and the concrete methods +from `object UnrolledObj` generate forwarders when `@unroll`ed, but the forwarders +are generated _in opposite directions_! Unrolled concrete methods forward from longer +parameter lists to shorter parameter lists, while unrolled abstract methods forward +from shorter parameter lists to longer parameter lists. For example, we may have a +version of `object UnrolledObj` that was compiled against an earlier version of `trait Unrolled`: + + +```scala +object UnrolledObj extends Unrolled{ // version 2 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b + def foo(s: String, n: Int) = foo(s, n, true) +} +``` + +But further downstream code calling `.foo` on `UnrolledObj` may expect any of the following signatures, +depending on what version of `Unrolled` and `UnrolledObj` it was compiled against: + +```scala +UnrolledObj.foo(String, Int) +UnrolledObj.foo(String, Int, Boolean) +UnrolledObj.foo(String, Int, Boolean, Long) +``` + +Because such downstream code cannot know which version of `Unrolled` that `UnrolledObj` +was compiled against, we need to ensure all such calls find their way to the correct +implementation of `def foo`, which may be at any of the above signatures. This "double +forwarding" strategy ensures that regardless of _which_ version of `.foo` gets called, +it ends up eventually forwarding to the actual implementation of `foo`, with +the correct combination of passed arguments and default arguments + +```scala +UnrolledObj.foo(String, Int) // forwards to UnrolledObj.foo(String, Int, Boolean) +UnrolledObj.foo(String, Int, Boolean) // actual implementation +UnrolledObj.foo(String, Int, Boolean, Long) // forwards to UnrolledObj.foo(String, Int, Boolean) +``` + +As is the case for `@unroll`ed methods on `trait`s and `class`es, `@unroll`ed +implementations of an abtract method must be final. + +#### Are Reverse Forwarders Really Necessary? + +This "double forwarding" strategy is not strictly necessary to support +[Backwards Compatibility](#backwards-compatibility): the "reverse" forwarders +generated for abstract methods are only necessary when a downstream callsite +of `UnrolledObj.foo` is compiled against a newer version of the original +`trait Unrolled` than the `object UnrolledObj` was, as shown below: + +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) + // generated + def foo(s: String, n: Int, b: Boolean): String = foo(s, n) + def foo(s: String, n: Int): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 2 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b + // generated + def foo(s: String, n: Int) = foo(s, n, true) +} +``` +```scala +// version 3 +UnrolledObj.foo("hello", 123, true, 456L) +``` + +If we did not have the reverse forwarder from `foo(String, Int, Boolean, Long)` to +`foo(String, Int, Boolean)`, this call would fail at runtime with an `AbstractMethodError`. +It also will get caught by MiMa as a `ReversedMissingMethodProblem`. + +This configuration of version is not allowed given our definition of backwards compatibility: +that definition assumes that `Unrolled` must be of a greater version than `UnrolledObj`, +which itself must be of a greater version than the final call to `UnrolledObj.foo`. However, +the reverse forwarders are to fulfill our requirement +[All Overrides Are Equivalent](#all-overrides-are-equivalent): +looking at `trait Unrolled // version 3` and `object UnrolledObj // version 2` in isolation, +we find that without the reverse forwarders the signature `foo(String, Int, Boolean, Long)` +is defined but not implemented. Such an un-implemented abstract method is something +we want to avoid, even if our artifact version constraints mean it should technically +never get called. ## Limitations -1. Only the one parameter list of multi-parameter list methods (i.e. curried or taking - implicits) can be `@unroll`ed. Unrolling multiple parameter lists would generate a number - of forwarder methods exponential with regard to the number of parameter lists unrolled, - and the generated forwarders may begin to conflict with each other. We can choose to spec - this out and implement it later if necessary, but for 99% of use cases `@unroll`ing one - parameter list should be enough. Typically, only one parameter list in a method has default - arguments, with other parameter lists being `implicit`s or a single callback/blocks, neither - of which usually has default values. - -2. As unrolling generates synthetic forwarder methods for binary compatibility, it is - possible for them to collide if your unrolled method has manually-defined overloads - -3. As mentioned earlier, `@unroll`ed case classes are only fully binary compatible in Scala 3, - though they are _almost_ binary compatible in Scala 2. Direct calls to `unapply` are binary - incompatible, but most common pattern matching of `case class`es goes through a different - code path that _is_ binary compatible. In practice this should be sufficient for 99% of use - cases, but it does mean that it is possible for code written as below to fail in Scala 2 - if a new unrolled parameter is added to the case class and `.unapply` is called directly. - -4. While `@unroll`ed `case class`es are fully binary compatible, they are *not* fully - _source_ compatible, due to the fact that pattern matching requires all arguments to - be specified. This proposal does not change that. Future improvements related to - [Pattern Matching on Named Fields](https://github.com/scala/improvement-proposals/pull/44) - may bring improvements here. But as we discussed earlier, binary compatibility is generally - more important than source compatibility, and so we do not need to wait for any source - compatibility improvements to land before proceeding with these binary compatibility - improvements. - -5. This proposal does not address how macros will derive typeclasses for `case class`es, and - whether or not those will be binary/source/semantically compatible. That is up to the - individual macro implementations to decide. e.g., [uPickle](https://github.com/com-lihaoyi/upickle) - has a very similar rule about adding `case class` fields, except that field ordering - does not matter. Trying to standardize this across all possible macros and all possible - typeclasses is out of scope - -6. `@unroll` generates a quadratic amount of generated bytecode as more default parameters - are added: each forwarder has `O(num-params)` size, and there are `O(num-default-params)` - forwarders. We do not expect this to be a problem in practice, as the small size of the - generated forwarder methods means the constant factor is small, but one could imagine - the `O(n^2)` asymptotic complexity becoming a problem if a method accumulates hundreds of - default parameters over time. In such extreme scenarios, some kind of builder pattern - (such as those listed in [Major Alternatives](#major-alternatives)) may be preferable. - -7. `@unroll` only supports `final` methods. `object` methods and constructors are naturally - final, but `class` or `trait` methods that are `@unroll`ed need to be explicitly marked `final`. - It has proved difficult to implement the semantics of `@unroll` in the presence of downstream - overrides, `super`, etc. where the downstream overrides can be compiled against by different - versions of the upstream code. If we can come up with some implementation that works, we can - lift this restriction later, but for now I have not managed to do so and so this restriction - stays. +### Only the one parameter list of multi-parameter list methods can be `@unroll`ed. + +Unrolling multiple parameter lists would generate a number +of forwarder methods exponential with regard to the number of parameter lists unrolled, +and the generated forwarders may begin to conflict with each other. We can choose to spec +this out and implement it later if necessary, but for 99% of use cases `@unroll`ing one +parameter list should be enough. Typically, only one parameter list in a method has default +arguments, with other parameter lists being `implicit`s or a single callback/blocks, neither +of which usually has default values. + +### Unrolled forwarder methods can collide with manually-defined overrides + +This is similar to any other generated methods. We can raise an error to help users +debug such scenarios, but such name collisions are inevitably possible given how binary +compatibility on the JVM works. + +### `@unroll`ed case classes are only fully binary compatible in Scala 3 + + +They are _almost_ binary compatible in Scala 2. Direct calls to `unapply` are binary +incompatible, but most common pattern matching of `case class`es goes through a different +code path that _is_ binary compatible. There are also the `AbstractFunctionN` traits, from +which the companion object inherits `.curried` and `.tupled` members. Luckily, `unapply` +was made binary compatible in Scala 3, and `AbstractFunctionN`, `.curried`, and `.tupled` +were removed + +### While `@unroll`ed `case class`es are *not* fully _source_ compatible + +This is due to the fact that pattern matching requires all arguments to +be specified. This proposal does not change that. Future improvements related to +[Pattern Matching on Named Fields](https://github.com/scala/improvement-proposals/pull/44) +may bring improvements here. But as we discussed earlier, binary compatibility is generally +more important than source compatibility, and so we do not need to wait for any source +compatibility improvements to land before proceeding with these binary compatibility +improvements. + +### Binary and semantic compatibility for macro-derived derive typeclasses is out of scope + + +This propsosal does not have any opinion on whether or not macro-derivation is be binary/source/semantically +compatible. That is up to the +individual macro implementations to decide. e.g., [uPickle](https://github.com/com-lihaoyi/upickle) +has a very similar rule about adding `case class` fields, except that field ordering +does not matter. Trying to standardize this across all possible macros and all possible +typeclasses is out of scope + +### `@unroll` generates a quadratic amount of generated bytecode as more default parameters are added + +Each forwarder has `O(num-params)` size, and there are `O(num-default-params)` +forwarders. We do not expect this to be a problem in practice, as the small size of the +generated forwarder methods means the constant factor is small, but one could imagine +the `O(n^2)` asymptotic complexity becoming a problem if a method accumulates hundreds of +default parameters over time. In such extreme scenarios, some kind of builder pattern +(such as those listed in [Major Alternatives](#major-alternatives)) may be preferable. + +###`@unroll` only supports `final` methods. + +`object` methods and constructors are naturally +final, but `class` or `trait` methods that are `@unroll`ed need to be explicitly marked `final`. +It has proved difficult to implement the semantics of `@unroll` in the presence of downstream +overrides, `super`, etc. where the downstream overrides can be compiled against by different +versions of the upstream code. If we can come up with some implementation that works, we can +lift this restriction later, but for now I have not managed to do so and so this restriction +stays. ### Challenges of Non-Final Methods and Overriding @@ -647,52 +805,6 @@ object Unrolled{ ``` -### Abstract Methods - -In [Limitations](#limitations), I mentioned that `@unroll` only supports `final` methods. -It is likely possible for abstract methods which are `@unrolled` to have concrete forwarder -methods generated on their behalf. - -```scala -import scala.annotation.unroll - -trait Unrolled{ - def foo(s: String, n: Int = 1, @unroll b: Boolean = true): String -} - -object Unrolled extends Unrolled{ - def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b -} -``` - -Unrolls to: - -```scala -trait Unrolled{ - def foo(s: String, n: Int = 1, @unroll b: Boolean = true): String = foo(s, n) - def foo(s: String, n: Int = 1): String = foo(s, n, true) -} - -object Unrolled extends Unrolled{ - def foo(s: String, n: Int = 1, b: Boolean = true) = s + n + b -} -``` - -As the forwarders are concrete, the implementor of the abstract method does -not need to `@unroll` the implementation: they only need to provide the implementation -for the primary method `def` and not the forwarders. - -One thing to note is that the `@unroll`ed abstract method needs to _itself_ become a -forwarder method, despite originally being abstract! That is because downstream code -compiled against an old version may define classes which `extends Unrolled` and -define a concrete `def foo(s: String, n: Int = 1): String`, while other downstream code compiled -against a newer version may define a concrete -`def foo(s: String, n: Int = 1, b: Boolean = true): String`. Thus, we need both overloads -of `foo` to be forwarders, so that downstream code can override either version and still work. - -This handling for abstract methods is not fully fleshed out or implemented, so I'm -not sure if it can truly be made to work. - ### Should the Generated Methods be Deprecated or Invisible? It is not clear to me if we should discourage usage of the generated forwarders @@ -743,7 +855,7 @@ This is not currently implemented in `@unroll`, but would be a straightforward a Given this: ```scala -def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l +def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l ``` There are two ways to do the forwarders. First option, which I used in above, is @@ -765,7 +877,12 @@ def foo(s: String, n: Int) = foo(s, n, true) The first option results in shorter stack traces, while the second option results in roughly half as much generated bytecode in the method bodies (though it's still `O(n^2)`). -For now I chose the first option. +In order to allow `@unroll`ing of [Abstract Methods](#abstract-methods), we had to go with +the second option. This is because when an abstract method is overriden, it is not necessarily +true that the longest override that contains the implementation. Thus we need to forward +between the different `def foo` overrides one at a time until the override containing the +implementation is found. + ## Implementation & Testing From 3ece9cbf8afa51d61c46b988256b3968f1cd4d6f Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:46:07 +0800 Subject: [PATCH 10/27] . --- content/unroll-default-arguments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 5a860019..a568fab4 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -39,6 +39,7 @@ compatibility as new default parameters and fields are added over time. 1. New parameters need to have a default value 2. New parameters can only be added on the right +3. The `@unroll`ed methods must be abstract or final These are both existing industry-wide standard when dealing with data and schema evolution (e.g. [Schema evolution in Avro, Protocol Buffers and Thrift — Martin Kleppmann’s blog](https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html)), From e6d816aa9aaa0896d03fd32689320c5b9ee3a621 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:47:10 +0800 Subject: [PATCH 11/27] . --- content/unroll-default-arguments.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index a568fab4..03b34785 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -31,9 +31,9 @@ def foo(s: String, n: Int) = foo(s, n, true, 0) In contrast to most existing or proposed alternatives that require you to contort your code to become binary compatible (see [Major Alternatives](#major-alternatives)), -`@unroll` allows you to write Scala in the most straightforward way. You add -a single annotation, and your `def`/`class`/`case class` will maintain binary -compatibility as new default parameters and fields are added over time. +`@unroll` allows you to write Scala with vanilla `def`s/`class`es/`case class`es, add +a single annotation, and your code will maintain binary compatibility as new default +parameters and fields are added over time. `@unroll`'s only constraints are that: From b9b9ec859dec2876816024e9a247932df0f8a59e Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:48:28 +0800 Subject: [PATCH 12/27] . --- content/unroll-default-arguments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 03b34785..9c3dd1dc 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -69,8 +69,8 @@ is exactly the user intent, and works just fine if the downstream code is re-com Other languages, such as Python, have the same default parameter language feature but face no such compatibility issues with their use. Even Scala codebases compiled from source -(e.g. in a mono-repo setup) do not suffer these restrictions: adding a default parameter -to the right side of a parameter list is for all intents and purposes backwards compatible. +do not suffer these restrictions: adding a default parameter to the right side of a parameter +list is for all intents and purposes backwards compatible in a mono-repo setup. The fact that such addition is binary incompatible is purely an implementation restriction of Scala's binary artifact format and distribution strategy. From f31a3761977a33e274e83d3ec94b308b2733b8ab Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:51:22 +0800 Subject: [PATCH 13/27] . --- content/unroll-default-arguments.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 9c3dd1dc..c01e358b 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -192,10 +192,13 @@ the JVM, so we should just follow suite ### All Overrides Are Equivalent All versions of an `@unroll`ed method `def foo` should have the same semantics when called -with the same parameters. +with the same parameters. We must be careful to ensure: -**Note:** this includes forwarder methods that may never get called in the scenarios -required by our [Backwards Compatibility](#backwards-compatibility) requirement. +1. All our different method overrides point at the same underlying implementation +2. Abstract methods are properly implemented, and no method would fail with an + `AbstractMethodError` when called +3. We properly forward the necessary argument and default parameter values when + calling the respective implementation. ## Proposed solution From ec9890e166ebaef92ad02c6dc67eb305a11360dd Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:52:21 +0800 Subject: [PATCH 14/27] . --- content/unroll-default-arguments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index c01e358b..f0796a58 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -225,9 +225,9 @@ can be applied to methods `def`s, `class` constructors, or `case class`es to gen This allows the developer to write the minimal amount of code they _want_ to write, and add a single annotation to allow binary compatibility to old versions. In this -case, we annotated `sorted` with `@unroll`, which generates forwarders that make +case, we annotated `sorted` and `nameMapper` with `@unroll`, which generates forwarders that make `def constructEither` binary compatible with older versions that have fewer parameters, -up to a version before `sorted` was added. Any existing method `def`, `class`, or +up to a version before `sorted` or `nameMapper` was added. Any existing method `def`, `class`, or `case class` can be evolved in this way, by addition of `@unroll` the first time a default argument is added to their signature after its initial definition. From dc0417a93ff1ad475cf2308c85ee172e6002e658 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:54:02 +0800 Subject: [PATCH 15/27] . --- content/unroll-default-arguments.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index f0796a58..3915cc4a 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -428,7 +428,8 @@ Notes: to addition of new fields due to [Option-less Pattern Matching](https://docs.scala-lang.org/scala3/reference/changed-features/pattern-matching.html). Thus, only direct calls to `.unapply` on an unrolled `case class` in Scala 2.x (shown below) - will cause a crash if additional fields were added. + will cause a crash if additional fields were added, or calls to `.tupled` or `.curried` on the + `case class` companion `object. ```scala def foo(t: (String, Int)) = println(t) From fb06141dfcb4fb66c9a83e605a3c3c579563fe17 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:54:26 +0800 Subject: [PATCH 16/27] . --- content/unroll-default-arguments.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 3915cc4a..3b0d52c6 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -427,9 +427,9 @@ Notes: binary compatible, pattern matching on `case class`es is already binary compatible to addition of new fields due to [Option-less Pattern Matching](https://docs.scala-lang.org/scala3/reference/changed-features/pattern-matching.html). - Thus, only direct calls to `.unapply` on an unrolled `case class` in Scala 2.x (shown below) - will cause a crash if additional fields were added, or calls to `.tupled` or `.curried` on the - `case class` companion `object. + Thus, only calls to `.tupled` or `.curried` on the `case class` companion `object`, or direct calls + to `.unapply` on an unrolled `case class` in Scala 2.x (shown below) + will cause a crash if additional fields were added: ```scala def foo(t: (String, Int)) = println(t) From c2bb54f4907f770fb8775bd9b2ef508828ec2e65 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:55:20 +0800 Subject: [PATCH 17/27] . --- content/unroll-default-arguments.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 3b0d52c6..9cb54ec1 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -419,11 +419,13 @@ object Unrolled{ Notes: -1. `.unapply` does not need to be duplicated in Scala 3.x, as its signature +1. `@unroll`ed `case class`es are fully binary and backwards compatible in Scala 3, but not in Scala 2 + +2. `.unapply` does not need to be duplicated in Scala 3.x, as its signature `def unapply(x: Unrolled): Unrolled` does not change when new `case class` fields are added. -2. Even in Scala 2.x, where `def unapply(x: Unrolled): Option[TupleN]` is not +3. Even in Scala 2.x, where `def unapply(x: Unrolled): Option[TupleN]` is not binary compatible, pattern matching on `case class`es is already binary compatible to addition of new fields due to [Option-less Pattern Matching](https://docs.scala-lang.org/scala3/reference/changed-features/pattern-matching.html). From 57e6b6103439630ed0365b2d71804218c49f34a3 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:57:06 +0800 Subject: [PATCH 18/27] . --- content/unroll-default-arguments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 9cb54ec1..904ffe56 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -569,8 +569,8 @@ If we did not have the reverse forwarder from `foo(String, Int, Boolean, Long)` It also will get caught by MiMa as a `ReversedMissingMethodProblem`. This configuration of version is not allowed given our definition of backwards compatibility: -that definition assumes that `Unrolled` must be of a greater version than `UnrolledObj`, -which itself must be of a greater version than the final call to `UnrolledObj.foo`. However, +that definition assumes that `Unrolled` must be of a greater or equal version than `UnrolledObj`, +which itself must be of a greater or equal version than the final call to `UnrolledObj.foo`. However, the reverse forwarders are to fulfill our requirement [All Overrides Are Equivalent](#all-overrides-are-equivalent): looking at `trait Unrolled // version 3` and `object UnrolledObj // version 2` in isolation, From 2f3f4498a0f7b8b0106cd7335b332d110b7c0c70 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:57:20 +0800 Subject: [PATCH 19/27] . --- content/unroll-default-arguments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 904ffe56..9c8da754 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -571,7 +571,7 @@ It also will get caught by MiMa as a `ReversedMissingMethodProblem`. This configuration of version is not allowed given our definition of backwards compatibility: that definition assumes that `Unrolled` must be of a greater or equal version than `UnrolledObj`, which itself must be of a greater or equal version than the final call to `UnrolledObj.foo`. However, -the reverse forwarders are to fulfill our requirement +the reverse forwarders are needed to fulfill our requirement [All Overrides Are Equivalent](#all-overrides-are-equivalent): looking at `trait Unrolled // version 3` and `object UnrolledObj // version 2` in isolation, we find that without the reverse forwarders the signature `foo(String, Int, Boolean, Long)` From b244838545d0e1d5f4b06881164ea8987879524a Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 19:58:39 +0800 Subject: [PATCH 20/27] . --- content/unroll-default-arguments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 9c8da754..08a4b92a 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -636,7 +636,7 @@ the `O(n^2)` asymptotic complexity becoming a problem if a method accumulates hu default parameters over time. In such extreme scenarios, some kind of builder pattern (such as those listed in [Major Alternatives](#major-alternatives)) may be preferable. -###`@unroll` only supports `final` methods. +### `@unroll` only supports `final` methods. `object` methods and constructors are naturally final, but `class` or `trait` methods that are `@unroll`ed need to be explicitly marked `final`. From 7f87fc29790ca940590625b5351da6cc79ac4acc Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Mon, 19 Feb 2024 20:01:48 +0800 Subject: [PATCH 21/27] . --- content/unroll-default-arguments.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 08a4b92a..c6943088 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -700,10 +700,10 @@ from `Downstream#foo`'s `super.foo` is meant to resolve to `Upstream#foo`. But a method implementation cannot know how it was called, and thus it is impossible for `def foo` to forward the call to the right place. -This issue only arises in the presence of version skew between `Upstream` and -`Downstream` code, but that scenario is precisely where `@unroll` is meant to -provide benefits. Thus, unless we can find some solution, we cannot properly support -virtual methods and overrides in `@unroll`. +Like our treatment of [Abstract Methods](#abstract-methods), this scenario can never +happen according to what version combinations are supported by our definition of +[Backwards Compatibility](#backwards-compatibility), but nevertheless is a real +concern due to the requirement that [All Overrides Are Equivalent](#all-overrides-are-equivalent). It may be possible to loosen this restriction to also allow abstract methods that are implemented only once by a final method. See the section about From f1fdf2a97a45c09c4907969161f58c31681f06f7 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Wed, 21 Feb 2024 15:44:36 +0800 Subject: [PATCH 22/27] . --- content/unroll-default-arguments.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index c6943088..e2428ceb 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -775,15 +775,28 @@ you to take _any_ `class` or `case class` and evolve it later in a binary compat ### Binary Compatibility for Methods and Classes Lastly, the above alternatives only solve _half_ the problem: how to evolve `case class`es. +This is _schema evolution_. + +Binary compatility is not just a problem for `case class`es adding new fields: normal +`class` constructors, instance method `def`s, static method `def`s, etc. have default +parameters added all the time as well. In contrast, `@unroll` allows the evolution of `def`s and normal `class`es, in addition -to `case class`es, all using the same approach. +to `case class`es, all using the same approach: -Binary compatility is not just a problem for `case class`es adding new fields: normal -`class` constructors, instance method `def`s, static method `def`s, etc. have default -parameters added all the time as well. `@unroll` solves all these problems at once, -using the same implementation and same user-facing semantics. +1. `@unroll`ing `case class`es is about _schema evolution_ +2. `@unroll`ing concrete method `def`s is about _API evolution_ +3. `@unroll`ing abstract method `def`s is about _protocol evolution_ + +All three cases above have analogous best practices in the broader software engineering +world: whether you are adding an optional column to a database table, adding an +optional flag to a command-line tool, are extending an existing protocol with optional +fields that may need handling by both clients and servers implementing that protocol. +`@unroll` solves all three problems at once - schema evolution, API evolution, and protocol +evolution. It does so with the same Scala-level syntax and semantics, with the same requirements +and limitations that common schema/API/protocol-evolution best-practices have in the broader +software engineering community. ## Minor Alternatives: From 3a369644fbe0ce8e272112076b3e70f93b437bb2 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 22 Feb 2024 12:55:07 +0800 Subject: [PATCH 23/27] add section on hiding generated methods --- content/unroll-default-arguments.md | 32 ++++++++++++----------------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index e2428ceb..bb13488d 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -579,6 +579,19 @@ is defined but not implemented. Such an un-implemented abstract method is someth we want to avoid, even if our artifact version constraints mean it should technically never get called. +### Hiding Generated Forwarder Methods + +As the generated forwarder methods are intended only for binary compatibility purposes, +we should generally hide them: IDEs, downstream compilers, ScalaDoc, etc. should behave as +if the generated methods do not exist. + +This is done in two different ways: + +1. In Scala 2, we generate the methods in a post-`pickler` phase. This ensures they do + not appear in the scala signature, and thus are not exposed to downstream tooling + +2. In Scala 3, the generated methods are flagged as `Invisible` + ## Limitations ### Only the one parameter list of multi-parameter list methods can be `@unroll`ed. @@ -825,25 +838,6 @@ object Unrolled{ ``` -### Should the Generated Methods be Deprecated or Invisible? - -It is not clear to me if we should discourage usage of the generated forwarders -methods, or hide them: - -1. On one hand, downstream code should not be compiling against these methods: they are - purely for binary compatibility - -2. On the other hand, downstream code does not control which overload of the method is - selected: that is up to the Scala compiler and how overloading interacts with default - parameter values. I have encountered scenarios with manually-written forwarders where - selected over the "primary" implementation - -3. The forwarders are meant to have the exact same semantics as the "primary" implementation. - Thus it _does not matter_ whether you call the primary and pass it a default value, - or you call a forwarder which then calls the primary and passes it the same default value. - -For now, I have left the generated methods "as is", though choosing to hide them or deprecate -them or something else is definitely an option ### Generating Forwarders For Parameter Type Widening or Result Type Narrowing From 977c13f60dc593467f551cf829a90d59b526073a Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 21 Mar 2024 13:36:47 +0800 Subject: [PATCH 24/27] . --- content/unroll-default-arguments.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index bb13488d..ba5c9a72 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -925,4 +925,8 @@ and the 100s of lines of boilerplate reduction can be seen in the links below: These pull requests all pass both the test suite as well as the MIMA `check-binary-compatibility` job, demonstrating that this approach does work -in real-world codebases. +in real-world codebases. At time of writing, these are published under the following +artifacts and can be used in your own projects already: + +- Compiler Plugin: `ivy"com.lihaoyi::unroll-plugin:0.1.12"` +- Annotation: `ivy"com.lihaoyi::unroll-annotation:0.1.12"` From b6ceef53cf627547379c00c8fde843c47d474a21 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Thu, 21 Mar 2024 14:08:53 +0800 Subject: [PATCH 25/27] . --- content/unroll-default-arguments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index ba5c9a72..1947fca9 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -922,6 +922,7 @@ and the 100s of lines of boilerplate reduction can be seen in the links below: - https://github.com/com-lihaoyi/mainargs/pull/113/files - https://github.com/com-lihaoyi/mill/pull/3008/files - https://github.com/com-lihaoyi/upickle/pull/555/files +- https://github.com/com-lihaoyi/os-lib/pull/254 These pull requests all pass both the test suite as well as the MIMA `check-binary-compatibility` job, demonstrating that this approach does work From f2b5b354019edcff684f287b794c7bea4617059e Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Fri, 24 May 2024 19:08:46 +0800 Subject: [PATCH 26/27] Update unroll-default-arguments.md --- content/unroll-default-arguments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 1947fca9..15a79413 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -22,7 +22,7 @@ compatibility. `@unroll` works by generating "unrolled" or "telescoping" forward ```scala // Original -def foo(s: String, n: Int = 1, @unroll b: Boolean = true, l: Long = 0) = s + n + b + l +def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l // Generated def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) From f916b31f10c3c20b973d2a9de48209f194aa3af1 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Fri, 24 May 2024 22:07:46 +0800 Subject: [PATCH 27/27] Move `Abstract Methods` section out of main proposal into alternatives --- content/unroll-default-arguments.md | 229 ++++++++++++++-------------- 1 file changed, 115 insertions(+), 114 deletions(-) diff --git a/content/unroll-default-arguments.md b/content/unroll-default-arguments.md index 15a79413..42ea8ac3 100644 --- a/content/unroll-default-arguments.md +++ b/content/unroll-default-arguments.md @@ -464,120 +464,6 @@ take into account field default values, and this change is necessary to make it use them when the given `p: Product` has a smaller `productArity` than the current `CaseClass` implementation -### Abstract Methods - -Apart from `final` methods, `@unroll` also supports purely abstract methods. Consider -the following example with a trait `Unrolled` and an implementation `UnrolledObj`: - -```scala -trait Unrolled{ // version 3 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String -} -``` -```scala -object UnrolledObj extends Unrolled{ // version 3 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b -} -``` - -This unrolls to: -```scala -trait Unrolled{ // version 3 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) - def foo(s: String, n: Int, b: Boolean): String = foo(s, n) - def foo(s: String, n: Int): String -} -``` -```scala -object UnrolledObj extends Unrolled{ // version 3 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l - def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) - def foo(s: String, n: Int) = foo(s, n, true) -} -``` - -Note that both the abstract methods from `trait Unrolled` and the concrete methods -from `object UnrolledObj` generate forwarders when `@unroll`ed, but the forwarders -are generated _in opposite directions_! Unrolled concrete methods forward from longer -parameter lists to shorter parameter lists, while unrolled abstract methods forward -from shorter parameter lists to longer parameter lists. For example, we may have a -version of `object UnrolledObj` that was compiled against an earlier version of `trait Unrolled`: - - -```scala -object UnrolledObj extends Unrolled{ // version 2 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b - def foo(s: String, n: Int) = foo(s, n, true) -} -``` - -But further downstream code calling `.foo` on `UnrolledObj` may expect any of the following signatures, -depending on what version of `Unrolled` and `UnrolledObj` it was compiled against: - -```scala -UnrolledObj.foo(String, Int) -UnrolledObj.foo(String, Int, Boolean) -UnrolledObj.foo(String, Int, Boolean, Long) -``` - -Because such downstream code cannot know which version of `Unrolled` that `UnrolledObj` -was compiled against, we need to ensure all such calls find their way to the correct -implementation of `def foo`, which may be at any of the above signatures. This "double -forwarding" strategy ensures that regardless of _which_ version of `.foo` gets called, -it ends up eventually forwarding to the actual implementation of `foo`, with -the correct combination of passed arguments and default arguments - -```scala -UnrolledObj.foo(String, Int) // forwards to UnrolledObj.foo(String, Int, Boolean) -UnrolledObj.foo(String, Int, Boolean) // actual implementation -UnrolledObj.foo(String, Int, Boolean, Long) // forwards to UnrolledObj.foo(String, Int, Boolean) -``` - -As is the case for `@unroll`ed methods on `trait`s and `class`es, `@unroll`ed -implementations of an abtract method must be final. - -#### Are Reverse Forwarders Really Necessary? - -This "double forwarding" strategy is not strictly necessary to support -[Backwards Compatibility](#backwards-compatibility): the "reverse" forwarders -generated for abstract methods are only necessary when a downstream callsite -of `UnrolledObj.foo` is compiled against a newer version of the original -`trait Unrolled` than the `object UnrolledObj` was, as shown below: - -```scala -trait Unrolled{ // version 3 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) - // generated - def foo(s: String, n: Int, b: Boolean): String = foo(s, n) - def foo(s: String, n: Int): String -} -``` -```scala -object UnrolledObj extends Unrolled{ // version 2 - def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b - // generated - def foo(s: String, n: Int) = foo(s, n, true) -} -``` -```scala -// version 3 -UnrolledObj.foo("hello", 123, true, 456L) -``` - -If we did not have the reverse forwarder from `foo(String, Int, Boolean, Long)` to -`foo(String, Int, Boolean)`, this call would fail at runtime with an `AbstractMethodError`. -It also will get caught by MiMa as a `ReversedMissingMethodProblem`. - -This configuration of version is not allowed given our definition of backwards compatibility: -that definition assumes that `Unrolled` must be of a greater or equal version than `UnrolledObj`, -which itself must be of a greater or equal version than the final call to `UnrolledObj.foo`. However, -the reverse forwarders are needed to fulfill our requirement -[All Overrides Are Equivalent](#all-overrides-are-equivalent): -looking at `trait Unrolled // version 3` and `object UnrolledObj // version 2` in isolation, -we find that without the reverse forwarders the signature `foo(String, Int, Boolean, Long)` -is defined but not implemented. Such an un-implemented abstract method is something -we want to avoid, even if our artifact version constraints mean it should technically -never get called. ### Hiding Generated Forwarder Methods @@ -811,6 +697,121 @@ evolution. It does so with the same Scala-level syntax and semantics, with the s and limitations that common schema/API/protocol-evolution best-practices have in the broader software engineering community. +### Abstract Methods + +Apart from `final` methods, `@unroll` also supports purely abstract methods. Consider +the following example with a trait `Unrolled` and an implementation `UnrolledObj`: + +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b +} +``` + +This unrolls to: +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) + def foo(s: String, n: Int, b: Boolean): String = foo(s, n) + def foo(s: String, n: Int): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0) = s + n + b + l + def foo(s: String, n: Int, b: Boolean) = foo(s, n, b, 0) + def foo(s: String, n: Int) = foo(s, n, true) +} +``` + +Note that both the abstract methods from `trait Unrolled` and the concrete methods +from `object UnrolledObj` generate forwarders when `@unroll`ed, but the forwarders +are generated _in opposite directions_! Unrolled concrete methods forward from longer +parameter lists to shorter parameter lists, while unrolled abstract methods forward +from shorter parameter lists to longer parameter lists. For example, we may have a +version of `object UnrolledObj` that was compiled against an earlier version of `trait Unrolled`: + + +```scala +object UnrolledObj extends Unrolled{ // version 2 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b + def foo(s: String, n: Int) = foo(s, n, true) +} +``` + +But further downstream code calling `.foo` on `UnrolledObj` may expect any of the following signatures, +depending on what version of `Unrolled` and `UnrolledObj` it was compiled against: + +```scala +UnrolledObj.foo(String, Int) +UnrolledObj.foo(String, Int, Boolean) +UnrolledObj.foo(String, Int, Boolean, Long) +``` + +Because such downstream code cannot know which version of `Unrolled` that `UnrolledObj` +was compiled against, we need to ensure all such calls find their way to the correct +implementation of `def foo`, which may be at any of the above signatures. This "double +forwarding" strategy ensures that regardless of _which_ version of `.foo` gets called, +it ends up eventually forwarding to the actual implementation of `foo`, with +the correct combination of passed arguments and default arguments + +```scala +UnrolledObj.foo(String, Int) // forwards to UnrolledObj.foo(String, Int, Boolean) +UnrolledObj.foo(String, Int, Boolean) // actual implementation +UnrolledObj.foo(String, Int, Boolean, Long) // forwards to UnrolledObj.foo(String, Int, Boolean) +``` + +As is the case for `@unroll`ed methods on `trait`s and `class`es, `@unroll`ed +implementations of an abtract method must be final. + +#### Are Reverse Forwarders Really Necessary? + +This "double forwarding" strategy is not strictly necessary to support +[Backwards Compatibility](#backwards-compatibility): the "reverse" forwarders +generated for abstract methods are only necessary when a downstream callsite +of `UnrolledObj.foo` is compiled against a newer version of the original +`trait Unrolled` than the `object UnrolledObj` was, as shown below: + +```scala +trait Unrolled{ // version 3 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true, @unroll l: Long = 0): String = foo(s, n, b) + // generated + def foo(s: String, n: Int, b: Boolean): String = foo(s, n) + def foo(s: String, n: Int): String +} +``` +```scala +object UnrolledObj extends Unrolled{ // version 2 + def foo(s: String, n: Int = 1, @unroll b: Boolean = true) = s + n + b + // generated + def foo(s: String, n: Int) = foo(s, n, true) +} +``` +```scala +// version 3 +UnrolledObj.foo("hello", 123, true, 456L) +``` + +If we did not have the reverse forwarder from `foo(String, Int, Boolean, Long)` to +`foo(String, Int, Boolean)`, this call would fail at runtime with an `AbstractMethodError`. +It also will get caught by MiMa as a `ReversedMissingMethodProblem`. + +This configuration of version is not allowed given our definition of backwards compatibility: +that definition assumes that `Unrolled` must be of a greater or equal version than `UnrolledObj`, +which itself must be of a greater or equal version than the final call to `UnrolledObj.foo`. However, +the reverse forwarders are needed to fulfill our requirement +[All Overrides Are Equivalent](#all-overrides-are-equivalent): +looking at `trait Unrolled // version 3` and `object UnrolledObj // version 2` in isolation, +we find that without the reverse forwarders the signature `foo(String, Int, Boolean, Long)` +is defined but not implemented. Such an un-implemented abstract method is something +we want to avoid, even if our artifact version constraints mean it should technically +never get called. + ## Minor Alternatives: