sbt 1.0 roadmap and beta-1

By Eugene Yokota (@eed3si9n) April 18, 2017

sbt 1.0 has been in the works for some years now, long enough that there may be skepticism in the community that it’s going to happen at all. We hope the following points will help dispel that skepticism:

  • Features originally planned for 1.0 (such as AutoPlugin and Dotty support) have already shipped in the 0.13.x series as Technology Previews.
  • Some more ambitious aspects of plans for 1.0 have been postponed or scaled back to make 1.0 achievable soon. (For example, cached compilation and replacing Ivy)
  • sbt 1.0 isn’t vaporware; there are milestones, most recently 1.0.0-M5, that you can try today.

As a result, we believe the current sbt 1.0 plan is achievable by summer of 2017.

Highlights

sbt 1.0 is aimed to be a new stable series for the next few years with smooth migration from sbt 0.13 series. sbt 0.12 style operators and Build trait are removed. Many originally planned 1.0 improvements already shipped in the 0.13 series as Tech Preview.

The major benefits of sbt 1.0 are

  • Scala 2.12 for build definitions and plugins
  • a new incremental compiler (called Zinc 1) with speedups especially noticeable in large codebase.
  • sbt server, which lays the groundwork for better IDE integration

Timing

sbt 1.0.0-M5 is our first beta release. The most significant remaining challenges are:

  • Finalize module APIs for Zinc 1, librarymanagement API, etc.
  • Implement guards in build.sbt DSL where we plan to change the semantics.
  • Implement tooling to assist automatic migration of older builds.
  • Validate migration paths for commonly-used sbt plugins.
  • Validate that the major community projects can migrate to sbt 1.0.

None of these challenges are open-ended and all of them are achievable in the coming months. We plan to release beta-2 in a month, and a release candidate in June.

  • 2017-04-18: sbt 1.0.0-M5 “beta-1”
  • 2017-05-18: sbt 1.0.0-M6 “beta-2”
  • June, 2017 July 10, 2017: sbt 1.0.0-RC1
  • July, 2017 August, 2017: sbt 1.0.0

For anyone interested in helping with the release of sbt 1.0, here are the areas the sbt team is most in need of community participation and assistance:

  • Testing/porting sbt plugins against the sbt 1.0 milestones.
  • Testing migration tool and sbt 1.0 milestones against community projects.

sbt 0.13.5+ as Technology Preview series

In March 2014, the sbt team labelled sbt 0.13.5 as “Technology Preview for sbt 1”, and started adding new features while keeping the backward compatibility with sbt 0.13.0. The purpose of the Technology Preview was to experiment with new ideas during sbt 0.13.x and reduce the jump when we go to sbt 1.0. Here are some of the features that were added in the last three years:

  • AutoPlugin
  • Name hashing incremental compiler
  • Natural white space handling
  • Cached dependency resolution
  • Configurable compiler bridge and Dotty support
  • Deprecation of 0.12.x style DSL
  • sbt new command
  • Preliminary JDK 9 support

There are also some features we are introducing newly in sbt 1.0 that required breaking binary compatibility.

Scala 2.12

In sbt 1.0, the build definitions and the plugin ecosystem will be built with Scala 2.12. Since we can break binary compatibility, we are adopting the latest stable Scala release. This will have positive impact to the plugin ecosystem that can now leverage recent libraries and tooling.

Modularization and Zinc incremental compiler

sbt 1.0 splits the code base into multiple modules with separate repositories such as sbt/sbt, sbt/zinc, sbt/librarymanagement, and sbt/io. The goal for the modularization is to clarify what is API and what is implementation details, and also to reduce the size of the code base.

In the sbt 0.13 code base roughly half of the code is around incremental compiler, and its usage has impact on Scala tooling in general, not just sbt. As Lightbend’s Tooling Team, we would like to split the incremental compiler apart so other build tools authors can participate in the development and benefit from it. The incremental compiler is called Zinc 1. In the sbt/zinc-contrib Gitter room we have engineers from Lightbend, Scala Center, VirtusLab, and Twitter regularly exchanging ideas and code.

Library management is another module that is another module that’s split out from sbt/sbt. The purpose is to avoid exposing the Ivy implementation directly to sbt plugins, so we can improve or swap out to different implementations in the future.

Zinc 1: Class-based name hashing

A major improvement brought into Zinc 1.0 by Grzegorz Kossakowski (commissioned by Lightbend) is class-based name hashing, which will speed up the incremental compilation of Scala in large projects.

Zinc 1.0’s name hashing tracks your code dependendencies at the class level, instead of at the source file level. The GitHub issue sbt/sbt#1104 lists some comparisons of adding a method to an existing class in some projects:

ScalaTest   AndHaveWord class:          Before 49s, After 4s (12x)
Specs2      OptionResultMatcher class:  Before 48s, After 1s (48x)
scala/scala Platform class:             Before 59s, After 15s (3.9x)
scala/scala MatchCodeGen class:         Before 48s, After 17s (2.8x)

This depends on some factors such as how your classes are organized, but you can see 3x ~ 40x improvements. The reason for the speedup is because it compiles fewer source files than before by untangling the classes from source files. In the example adding a method to scala/scala’s Platform class, sbt 0.13’s name hashing used to compile 72 sources, but the new Zinc compiles 6 sources.

sbt server: JSON API for tooling integration

sbt 1.0 includes a new server command, which allows IDEs and other tools to query the build for settings, and invoke commands via a JSON API. Similar to the way that the interactive shell in sbt 0.13 is implemented with a shell command, server is just a command that listens to both human input and network input. As a user, there should be minimal impact because of the server.

In March 2016, we rebooted the “server” feature to make it as small as possible. We worked in collaboration with JetBrains’ @jastice who works on IntelliJ’s sbt interface to come up with the feature. sbt 1.0 will not have all the things we originally wanted, but in the long term, we hope to see better integration between IDE and sbt ecosystem using this system. For example, IDEs will be able to issue the compile task and retrieve compiler warning as JSON events:

{"type":"xsbti.Problem","message":{"category":"","severity":"Warn","message":"a pure expression does nothing in statement position; you may be omitting necessary parentheses","position":{"line":2,"lineContent":"  1","offset":29,"pointer":2,"pointerSpace":"  ","sourcePath":"/tmp/hello/Hello.scala","sourceFile":"file:/tmp/hello/Hello.scala"}},"level":"warn"}

Another related feature that was added is the bgRun task which, for example, enables a server process to be run in the background while you run tests against it.

Cross building sbt 1.0 plugin from sbt 0.13

Like we are able to cross build against multiple Scala versions, we can cross build sbt 1.0 plugins while staying on sbt 0.13. This is useful because we can port one plugin at a time.

  1. Clone a plugin in a separate working directory.
  2. If the plugin depends on libraries, make sure there are Scala 2.12 artifacts for them.
  3. Use the sbt 0.13.16-M1.
$ sbt
> ++2.12.2
> ^^1.0.0-M5
> compile

If you run into problems upgrading a plugin, please report to GitHub issue.

Removal of features

sbt 1.0 will remove the following support:

  • Non-auto sbt.Plugin trait. Please migrate to AutoPlugin. Auto plugins are easier to configure, and work better with each other.
  • sbt 0.12 style Build trait that was deprecated in sbt 0.13.12, is removed. Please migrate to build.sbt. Auto plugins and Build trait do not work well together, and its feature is now largely subsumed by multi-project build.sbt.
  • sbt 0.12 style key dependency operators <<=, <+=, <++= are removed. Please migrate to :=, +=, and ++=. These operators have been sources of confusion for many users, and have long been removed from 0.13 docs, and have been formally deprecated since sbt 0.13.13.
  • Support for Scala 2.9 and earlier is dropped. Scala 2.10 must use 2.10.2 and above. Scala 2.11 must use 2.11.2 and above. (latest patch releases are recommended)

We are working with Scala Center to provide an automatic migration tool for these changes.

Feedback welcome

If you have questions or feedback for this roadmap, please comment on Asking for your feedback on sbt + Scala Center announcement in the next two weeks.