assertTrue("Test DSLs" == "Legacy")

Over decades in programming I’ve had a few moments where I make big paradigm shifts, finding it very hard to go back. One of these moments was with Functional Programming and my latest is the move from Test Assertion DSLs to just using assertTrue with boolean-only tests.

Let’s look at some examples of Test Assertion DSLs. Suppose we have a string and we want to assert it starts with a letter and ends with a letter.

With ScalaTest:

"Bob" should (startWith ("B") and endWith ("c"))

With AssertJ:

assertThat("Bob")
    .startsWith("B")
    .endsWith("c")

These assertion DSLs provide excellent error messages when the assertion fails:

With ScalaTest:

"Bob" started with substring "B", but "Bob" did not end with substring "c"

With AssertJ:

Expecting actual:
    "Bob"
to end with:
    "c"

All assertions boil down to a boolean (either the test succeeded or failed) so why don’t we just use regular code and boolean assertions, like:

("Bob".startsWith("B") && "Bob".endsWith("c")) should be (true)

Because the test failure messages are not helpful:

false was not true

With JUnit:

assertTrue("Bob".startsWith("B") && "Bob".endsWith("c"));

The error is:

Test failed: java.lang.AssertionError: null

Test Assertion DSLs add another set of things to learn and the code isn’t “normal” code, i.e. not the way you would test something outside of a test and not necessarily reusable in the same way as normal code. Ideally we have our cake and eat it too - write normal boolean-based test assertions AND get good error messages. Luckily some test frameworks have support for this. For example, with ScalaTest a failing test defined as:

assert("Bob".startsWith("B") && "Bob".endsWith("c"))

Results in the error message:

"Bob" started with "B", but "Bob" did not end with "c"

No DSL needed! Which for me has significantly reduced my reluctance to test. I spend most of my day writing code that is not in the test DSLs so there is often some additional lag when I have to lookup how to do something in a test DSL.

Yet there is much more that test DSLs do for you so let’s dive a bit deeper, switching gears to ZIO Test which also has support for the “good error messages without the DSL” approach. Let’s say we have a basic data structure and test data:

case class Role(title: String)
case class Person(name: String, role: Role)
val people = Seq(
  Person("Joe", Role("Engineer")),
  Person("Sally", Role("Manager")),
  Person("Bob", Role("QA")),
)

We may want to find a person and assert on their title:

assert(people.find(_.name == "Ralph"))(
  isSome(hasField("title", _.role.title, equalsIgnoreCase("qa")))
)

Since the person wasn’t found, our error message is quite helpful:

Thanks to some great work by the ZIO Test folks, you can also get the same helpful error message but without the DSL, using assertTrue on a boolean:

assertTrue(people.find(_.name == "Ralph").get.role.title.equalsIgnoreCase("qa"))

In all of my new projects where these “smart” boolean assertions are supported (currently Scala ZIO projects), I exclusively use them and have not felt any need to go back to the legacy assertion DSLs. I hope more test frameworks will add support for this style so I can forever move away from Test Assertion DSLs. If you know of test frameworks beyond ScalaTest and ZIO Test that support the “smart” boolean assertion style, let me know!

Update Sept 27, 2023

Kotlin also has a similar library for great error messages without the DSL called kotlin-power-assert and the maintainers helped get it working with my examples. Given the test:

assert("Bob".startsWith("B") && "Bob".endsWith("c"))

The nice error is:

MyTest > one FAILED
    java.lang.AssertionError: Assertion failed
    assert("Bob".startsWith("B") && "Bob".endsWith("c"))
                 |                        |
                 |                        false
                 true
        at MyTest.one(MyTest.kt:16)

Going back to the more nested data structure example, here is how it looks:

data class Role(val title: String)
data class Person(val name: String, val role: Role)
val people = listOf(
    Person("Joe", Role("Engineer")),
    Person("Sally", Role("Manager")),
    Person("Bob", Role("QA")),
)

assert(people.find { it.name == "Ralph" }?.role?.title.equals("qa", ignoreCase = true))

Which results in:

MyTest > two FAILED
    java.lang.AssertionError: Assertion failed
    assert(people.find { it.name == "Ralph" }?.role?.title.equals("qa", ignoreCase = true))
           |      |                            |     |     |
           |      |                            |     |     false
           |      |                            |     null
           |      |                            null
           |      null
           [Person(name=Joe, role=Role(title=Engineer)), Person(name=Sally, role=Role(title=Manager)), Person(name=Bob, role=Role(title=QA))]
        at MyTest.two(MyTest.kt:20)

Amazing! Huge thank you to the kotlin-power-assert maintainers for helping figure that out.