Integration testing the Typelevel toolkit

The Typelevel toolkit is a metalibrary including some great libraries by Typelevel, that was created to speed up the development of cross-platform applications in Scala and that I happily maintain since its creation. It's the Typelevel's flavour of the official Scala Toolkit, a set of libraries to perform common programming tasks, that has its own section, full of examples, in the official Scala documentation.

One of the vaunts of the Typelevel's stack is the fact that (almost) every library is published for the all the three officially supported Scala platforms: JVM, JS and Native, and for this reason every library is heavily tested against every supported platform and Scala version, to ensure a near perfect cross-compatibility.

Since its creation the Typelevel toolkit was lacking any sort of testing, mainly due to the fact that it is a mere collection of already battle tested libraries, so why bothering writing tests for it? As this bug promptly reminded us, the main goal of the toolkit is to provide the most seamless experience while using scala-cli.

Ideally you should be able to write:

helloWorld.scala
//> using toolkit typelevel:latest

import cats.effect.*

object Hello extends IOApp.Simple:
  def run = IO.println("Hello World!")

and calling scala-cli run {,--js,--native} helloWorld.scala should Just Work™ printing "Hello World!" to the console.

To be 100% sure we needed CI tests indeed.

Planning the tests

What had to be tested though? All the included libraries are already tested, some of them are built using other included libraries, so some sort of cross testing was already done. What we were really interested in was always being sure that scala-cli is always able to compile scripts written using the toolkit. And what's the best way to ensure that scala-cli can compile a script written with the toolkit if not using scala-cli itself?

Pause for dramatic effect

The coarse idea that Arman and I had in mind was to have a CI doing the following:

The third step in particular could have been implemented in a couple of ways:

  1. Installing scala-cli in the CI image via GitHub Actions, call it from the tests code, and gather the results
  2. Since scala-cli is a native executable generated by GraalVM Native Image and the corresponding jvm artifact is distributed, using it as a dependency and calling its main method in the tests.

We decided to follow the latter, as we didn't want to mangle the GitHub Actions CI file or relying on the timely publication of the updated scala-cli GitHub Action: whenever any continuous integration setting is changed, every developer should apply the same or an equivalent change to its local environment to reflect the testing/building remote environment change. This also means more testing/contributing documentation that needs to be constantly updated (and that risks becoming outdated at every CI setting changed) and that the contributing/developing curve becomes steeper for newcomers (it's easier to ask a Scala developer to have just one build tool installed locally, right?).

Also, sbt is a superb tool for implementing this kind of tests: since it downloads automatically the specified scala-cli artifact we didn't need to have scala-cli installed locally, the version we are testing in particular. The build would be more self-contained, the scala-cli artifact version will be managed as every other dependency by scala-steward and developers and contributors could test locally the repository with ease with a simple sbt test.

BONUS EXAMPLE: Using scala-cli in scala-cli to run a scala-cli script that runs itself

recursiveScalaCli.scala
//> using dep org.virtuslab.scala-cli::cli::1.0.4

import scala.cli.ScalaCli

object ScalaCliApp extends App:
    ScalaCli.main(Array("run", "recursiveScalaCli.scala"))

First tentative: using the dependency in tests

In order to publish the artifacts locally before testing we needed a new tests project and to establish this relationship:

build.sbt
//...
lazy val root = tlCrossRootProject.aggregate(
  toolkit, 
  toolkitTest,
  tests
)
//...
lazy val tests = project
  .in(file("tests"))
  .settings(
    name := "tests",
    Test / test := (Test / test).dependsOn(toolkit.jvm / publishLocal).value
  )
//...

In this way the test sbt command will always run a publishLocal of the jvm flavor of the toolkit artifact. The project then needed to be set to not publish its artifact and to have some dependencies added to actually write the tests. The scala-cli dependency needed some trickery (.cross(CrossVersion.for2_13Use3)) to use the Scala 3 artifact, the only one published, in Scala 2.13 as well.

build.sbt
//...
lazy val tests = project
  .in(file("tests"))
  .settings(
    name := "tests",
    Test / test := (Test / test).dependsOn(toolkit.jvm / publishLocal).value,
    // Required to use the scala 3 artifact with scala 2.13
    scalacOptions ++= {
      if (scalaBinaryVersion.value == "2.13") Seq("-Ytasty-reader") else Nil
    },
    libraryDependencies ++= Seq(
      "org.typelevel" %% "munit-cats-effect" % "2.0.0-M3" % Test,
      // This is needed to write scripts' body into files
      "co.fs2" %% "fs2-io" % "3.9.2" % Test,
      "org.virtuslab.scala-cli" %% "cli" % "1.0.4" % Test cross (CrossVersion.for2_13Use3)
    )
  )
  .enablePlugins(NoPublishPlugin)
//...

The last bit needed was a way to add to the scripts' body which version of the artifact we were publishing right before the testing step and which Scala version we were running on, in order to test it properly. The only place were this (non-static) information was present was the build itself, but we needed to have them as an information in the source code. We definitively needed some sbt trickery to make it happen.

There is an unspoken rule about the Scala community (or in the sbt users community to be precise) that you may already know about:

If you need some kind of sbt trickery, eed3si9n probably wrote a sbt plugin for that.

This was our case with sbt-buildinfo, a sbt plugin whose punchline is "I know this because build.sbt knows this". As you'll discover later, sbt-buildinfo has been the corner stone of our second and more exhausting approach, but what briefly does is generating Scala source from your build definitions, and thus makes build information available in the source code too.

As scalaVersion and version are two information that are injected by default, we just needed to add the plugin into project/plugins.sbt and enabling it on tests in the build:

projects/plugins.sbt
//...
addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.11.0")
build.sbt
//...
lazy val tests = project
  .in(file("tests"))
  .settings(
    name := "tests",
    Test / test := (Test / test).dependsOn(toolkit.jvm / publishLocal).value,
    // Required to use the scala 3 artifact with scala 2.13
    scalacOptions ++= {
      if (scalaBinaryVersion.value == "2.13") Seq("-Ytasty-reader") else Nil
    },
    libraryDependencies ++= Seq(
      "org.typelevel" %% "munit-cats-effect" % "2.0.0-M3" % Test,
      // This is needed to write scripts' body into files
      "co.fs2" %% "fs2-io" % "3.9.2" % Test,
      "org.virtuslab.scala-cli" %% "cli" % "1.0.4" % Test cross (CrossVersion.for2_13Use3)
    )
  )
  .enablePlugins(NoPublishPlugin, BuildInfoPlugin)
//...

Time to write the tests! The first thing that was needed was a way to write on a temporary file the body of the script, including the artifact and Scala version, and then submit the file to scala-cli main method:

ToolkitTests.scala
package org.typelevel.toolkit

import munit.CatsEffectSuite
import cats.effect.IO
import fs2.Stream
import fs2.io.file.Files
import scala.cli.ScalaCli
import buildinfo.BuildInfo.{version, scalaVersion}

class ToolkitCompilationTest extends CatsEffectSuite {

  testRun("Toolkit should compile a simple Hello Cats Effect") {
    s"""|import cats.effect._
        |
        |object Hello extends IOApp.Simple {
        |  def run = IO.println("Hello toolkit!")
        |}"""
  }

  // We'll describe this method in a later section of the post
  def testRun(testName: String)(scriptBody: String): Unit = test(testName)(
    Files[IO].tempFile(None, "", "-toolkit.scala", None)
      .use { path =>
          val header = List(
            s"//> using scala ${BuildInfo.scalaVersion}",
            s"//> using toolkit typelevel:${BuildInfo.version}",
          ).mkString("", "\n", "\n")
        Stream(header, scriptBody.stripMargin)
          .through(Files[IO].writeUtf8(path))
          .compile
          .drain >> IO.delay(
          ScalaCli.main(Array("run", path.toString))
        )
      }
  )
}

And with this easy and lean approach we were finally able to test the toolkit! 🎉🎉🎉

Another pause for dramatic effect

Except we weren't really testing everything: the js and native artifact weren't tested by this approach, as the tests project is a jvm only project depending on toolkit.jvm. Also, the toolkit-test artifact wasn't even taken in consideration. We needed a more general/agnostic solution.

Second approach: Invoking Java as an external process

The first tentative was good but not satisfying at all: we had to find a way to test the js and native artifacts too, but how? The scala-cli artifact is JVM Scala 3 only, and there's no way to use it as a dependency on other platforms. The only way to use it is just through the jvm, and that's precisely what we decided to do.

Given that:

we knew that was possible, there was just some sbt-fu needed.

The thing we needed to intelligently invoke was a mere java -cp <scala-cli + transitive deps classpath> scala.cli.ScalaCli, pass to it run <scriptFilename>.scala and wait for the exit code, for each (scalaVersion,platform) combination.

BuildInfo magic

To begin we had to transform the tests project in to a cross project (using sbt-crossproject, that is embedded in sbt-typelevel) and make every subproject test command depend on the publication of the respective artifacts:

build.sbt
//...
lazy val tests = crossProject(JVMPlatform, JSPlatform, NativePlatform)
  .in(file("tests"))
  .settings(
    name := "tests",
    scalacOptions ++= {
      if (scalaBinaryVersion.value == "2.13") Seq("-Ytasty-reader") else Nil
    },
    libraryDependencies ++= Seq(
      "org.typelevel" %%% "munit-cats-effect" % "2.0.0-M3" % Test,
      "co.fs2" %%% "fs2-io" % "3.9.2" % Test,
      "org.virtuslab.scala-cli" %% "cli" % "1.0.4" cross (CrossVersion.for2_13Use3)
    )
  )
  .jvmSettings(
    Test / test := (Test / test).dependsOn(toolkit.jvm / publishLocal, toolkitTest.jvm / publishLocal).value
  )
  .jsSettings(
    Test / test := (Test / test).dependsOn(toolkit.js / publishLocal, toolkitTest.js / publishLocal).value
    scalaJSLinkerConfig ~= { _.withModuleKind(ModuleKind.CommonJSModule) }
  )
  .nativeSettings(
    Test / test := (Test / test).dependsOn(toolkit.native / publishLocal, toolkitTest.native / publishLocal).value
  )
  .enablePlugins(BuildInfoPlugin, NoPublishPlugin)
//...

One thing to note is that we deliberately made a "mistake". The munit-cats-effect and fs2-io dependencies are declared using %%% the operator that not only appends _${scalaBinaryVersion} to the end of the artifact name but also the platform name (appending i.e. for a Scala 3 native dependency _native0.4_3), but the scala-cli one was declared using just %% and the % Test modifier was removed. In this way we were sure that, for every platform, the Compile / dependencyClasspath would have included just the jvm version of scala-cli.

To inject the classpath into the source code we leveraged our beloved friend sbt-buildinfo, that it's not limited to inject just SettingKey[T]s and/or static information (computed at project load time), but using its own syntax can inject TaskKey[T]s after they've been evaluated (and re-evaluated each time at compile). So in the common .settings we added:

build.sbt
///...
  buildInfoKeys += scalaBinaryVersion,
  buildInfoKeys += BuildInfoKey.map(Compile / dependencyClasspath) {
      case (_, v) =>
        "classPath" -> v.seq
          .map(_.data.getAbsolutePath)
          .mkString(File.pathSeparator) // That's the way java -cp accepts classpath info
    },
    buildInfoKeys += BuildInfoKey.action("javaHome") {
      val path = sys.env.get("JAVA_HOME").orElse(sys.props.get("java.home")).get
      if (path.endsWith("/jre")) {
        // handle JDK 8 installations
        path.replace("/jre", "")
      } else path
    },
    buildInfoKeys += "scala3" -> (scalaVersion.value.head == '3')
///...

and in each platform specific section we added to buildInfo the platform's name:

build.sbt
//...
  .jvmSettings(
    //...
    buildInfoKeys += "platform" -> "jvm"
  )
  .jsSettings(
    //...
    buildInfoKeys += "platform" -> "js",
  )
  .nativeSettings(
    //...
    buildInfoKeys += "platform" -> "native"
  )
//...

in this way we could leverage in our source code all the information required to run scala-cli and test our snippets:

private val classPath: String          = BuildInfo.classPath
private val javaHome: String           = BuildInfo.javaHome
private val platform: String           = BuildInfo.platform
private val scalaBinaryVersion: String = BuildInfo.scalaBinaryVersion
private val scala3: Boolean            = BuildInfo.scala3

Invoking Java via fs2 Process

Once we had all the required components, invoking java was easy, we just created and spawned a Process from the package fs2.io.process, that is implemented for every platform under the very same API:

ScalaCliTest.scala
import buildinfo.BuildInfo
import cats.effect.kernel.Resource
import cats.effect.std.Console
import cats.effect.IO
import cats.syntax.parallel.*
import fs2.Stream
import fs2.io.file.Files
import fs2.io.process.ProcessBuilder
import munit.Assertions.fail

object ScalaCliProcess {

  private def scalaCli(args: List[String]): IO[Unit] = ProcessBuilder(
    s"${BuildInfo.javaHome}/bin/java",
    args.prependedAll(List("-cp", BuildInfo.classPath, "scala.cli.ScalaCli"))
  ).spawn[IO]
    .use(process =>
      (
        process.exitValue,
        process.stdout.through(fs2.text.utf8.decode).compile.string,
        process.stderr.through(fs2.text.utf8.decode).compile.string
      ).parFlatMapN {
        case (0, _, _) => IO.unit
        case (exitCode, stdout, stdErr) =>
          IO.println(stdout) >> Console[IO].errorln(stdErr) >> IO.delay(
            fail(s"Non zero exit code ($exitCode) for ${args.mkString(" ")}")
          )
      }
    )

  //..

}

Let's dissect this function:

Now we needed a method to write in a temporary file the source of each scala-cli script with all the information needed to correctly test the toolkit. Luckily for us fs2 makes it easy:

ScalaCliTest.scala
//...
  private def writeToFile(scriptBody: String)(isTest: Boolean): Resource[IO, String] =
    Files[IO].tempFile(None,"",if (isTest) "-toolkit.test.scala" else "-toolkit.scala", None)
      .evalTap { path =>
        val header = List(
          s"//> using scala ${BuildInfo.scalaVersion}",
          s"//> using toolkit typelevel:${BuildInfo.version}",
          s"//> using platform ${BuildInfo.platform}"
        ).mkString("", "\n", "\n")
        Stream(header, scriptBody.stripMargin)
          .through(Files[IO].writeUtf8(path))
          .compile
          .drain
      }
      .map(_.toString)
//...

Dissecting this function too we'll see that:

The only thing we needed to do was to combine the two methods into a testing method:

ScalaCliTest.scala
//...
  def testRun(testName:String)(body: String): IO[Unit] = 
   test(testName)(writeToFile(body)(false).use(f => scalaCli("run" :: f :: Nil)))

  def testTest(testName:String)(body: String): IO[Unit] = 
    test(testName)(writeToFile(body)(true).use(f => scalaCli("test" :: f :: Nil)))
//...

To recap, each of the two methods will run a munit test that:

The produced files will look, for example, like this:

//> using scala 3
//> using toolkit typelevel:typelevel:0.1.14-29-d717826-20231004T153011Z-SNAPSHOT
//> using platform jvm

import cats.effect.*

object Hello extends IOApp.Simple:
  def run = IO.println("Hello toolkit!")

where 0.1.14-29-d717826-20231004T153011Z-SNAPSHOT is the version of the toolkit that was just published locally by sbt.

Test writing

It was then Time to write and run the actual tests!

ToolkitTests.scala
import munit.CatsEffectSuite
import buildinfo.BuildInfo.scala3
import ScalaCliTest.{testRun, testTest}

class ToolkitTests extends CatsEffectSuite {

  testRun("Toolkit should run a simple Hello Cats Effect") {
    if (scala3)
      """|import cats.effect.*
         |
         |object Hello extends IOApp.Simple:
         |  def run = IO.println("Hello toolkit!")"""
    else
      """|import cats.effect._
         |
         |object Hello extends IOApp.Simple {
         |  def run = IO.println("Hello toolkit!")
         |}"""
  }

  testTest("Toolkit should execute a simple munit suite") {
    if (scala3)
      """|import cats.effect.*
         |import munit.*
         |
         |class Test extends CatsEffectSuite:
         |  test("test")(IO.unit)"""
    else
      """|import cats.effect._
         |import munit._
         |
         |class Test extends CatsEffectSuite {
         |  test("test")(IO.unit)
         |}"""
  }
  //...
}

The little testing framework we wrote is now capable of both running and testing scala-cli scripts that use the typelevel toolkit, and it will test it in every platform and scala version. sbt test will now publish both the toolkit and the test toolkit, for every platform, right before running the unit tests, achieving in this way a complete coverage and adding reliability to our releases! 🎉

And all of this was done without even touching our GitHub Actions, just with some sbt-fu, and just using the libraries that are included in the toolkit itself 😎