Each pull request on Gearpump will be checked for code coverage thanks to sbt-scoverage plugin. sbt coverage test had been added to .travis.yml to trigger the check.

The coverage suddenly dropped to 0 when I upgraded Gearpump's Scala version to 2.12. The suspicious change was upgrading sbt-scoverage from 1.2.0 to 1.5.1. I thought the problem was related to the sbt version 0.13.16 and asked about it on sbt-scoverage's issues. Meanwhile, I went on to try my luck with sbt 1.2.7 which turned a minor issue into significant work because all SBT plugins had to be upgraded to versions that work with SBT 1.x

One roadblock was we'd maintained forked versions of sbt-assembly and sbt-pack from 0.13.x and it looked very hard to port those changes.

Upgrading sbt-assembly

Gearpump uses sbt-assembly to create fat jars with shaded dependencies for sub-modules. Tinkering with sbt-assembly has contributed to my blog series Shade with SBT I, II and III. There was one more issue. Running gearpump examples in Intellij or SBT would fail with ClassNotFoundException since they had provided dependency on gearpump-core and gearpump-streaming. My solution then was changing the scope to compile and manually filtering out everything other than examples' own source codes with the built-in assemblyExcludedJars option. There was one limitation with the option that it only excluded jars. Hence, I went on to make a tweak extending the exclusion to all files.

Apparently, it's not worth maintaining a tweak and porting it from one forked version to another. Therefore, I switched to the official sbt-assembly 0.14.9 and used the assemblyMergeStrategy option this time.

assemblyMergeStrategy in assembly := {
x =>
  // core and streaming dependencies are not marked as provided
  // such that the examples can be run with sbt or Intellij
  // so they have to be excluded manually here
  if (x.contains("examples")) {
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
  } else {
    MergeStrategy.discard
  }
}

It's working! At least 50%.

This brought in another issue that the assembled jar were not published as before with the following.

addArtifact(Artifact("gearpump-core"), sbtassembly.AssemblyKeys.assembly)

After taking a closer look, I found the previous way was fragile where the unassembled and assembled jars were created with the same name in different directories and the latter happened to override the former when publishing. The fix was to append an assembly to the assembled jar and alter the artifact setting as documented.

artifact in (Compile, assembly) := {
  val art = (artifact in (Compile, assembly)).value
  art.withClassifier(Some("assembly"))
}

By the way, it's not recommended to publish fat jars so I think this merits revisiting later.

Upgrading sbt-pack

sbt-pack has served to create distributable Gearpump packages with launch scripts under bin and system jars (output from sbt-assembly) under lib. The lib directory is by default put on all classpaths in the launch scripts. Not to conflict with users' dependencies, Gearpump tried to put system dependencies under subdirectories of lib. My colleague Vincent hacked into the packLibDir option and made the following possible.

packLibDir := Map(	      
  "lib/hadoop" -> new ProjectsToPack(gearpumpHadoop.id).
      exclude(services.id, core.id),
  "lib/services" -> new ProjectsToPack(services.id).exclude(core.id)
)

Like sbt-assembly, I decided not to maintain the forked sbt-pack given the cost of porting. The solution was to pack system dependencies into other directories on the same level as lib and I was able to upgrade sbt-pack to 0.11.

Other changes

• sbt-unidoc has been upgraded to 0.4.2 with unresolved Javadoc errors.

• License headers have been updated removing ASF declarations with the help of Intellij's updating copyright text button.

• Build scripts have been moved from classes extending sbt.Build in various .scala files to build.sbt.

Is the coverage issue solved ?

No! Unfortunately the coverage issue was not solved even when SBT had been upgraded 1.2.7. It's turned out that SBT version is not the scapegoat but that I have not read sbt-scoverage's README carefully enough. sbt coverage only outputs the coverage data and sbt coverageReport is needed to generate the report.

sbt coverage test
sbt coverageReport

Note that coverageReport can't be put in the same command as coverage.

It has left me thinking whether my fight with SBT is worthwhile.