Evolving test setup and test data management

Evolving test setup and test data management

One of the things I do constantly learn anew is ways to create test data and manage it. Also, setting up tests evolves every time I start afresh.

Starting a new project at work set evolution in motion again. This time we also slightly changed the team which helped to trigger it. Marc added experience with test data creation to the project. More of this later. Building on that, a new pattern began to emerge for me. Again, nothing fancy. But something I want to write down to share with as much as to clarify it for me.

The context

I do like to work with little stories in my head to set a context in which the example lives. This time I picked one from my time when I was still working as a biochemist in a wet lab. There we had the issue of organizing all the samples we put into different freezers in different locations and boxes. It was a constant battle with entropy to keep an overview. We did not have a LIMS (Laboratory Information and Management System). I got into touch which flavors of this software later at a different employer where I started switching my careers. So this context will have something to do with a LIMS.

Imagine you are part of a team responsible for the StorageCabinet module of such a LIMS. The company just got started with this, and you start working iteratively on it. We will do some iterations in this example, and every time we do a new one we will refactor our tests. The team writes code with Kotlin and tries to build a rich domain model. They also decide to go with in-memory repository for now, planning on changing this later as it is still unclear which type of database will be used or even if there will be multiple options.

Here are the iterations we will do (you can find the code here and links to a corresponding branch after each point)

  1. Allow Commission and Decommission as well as name changes of StorageCabinets (Branch)
  2. Add a room to the StorageCabinet to describe where it is located (Branch)
  3. Adding storage boxes to the cabinet (Branch)
  4. Refactor the tests to make them easier to write (this is a purely technical task) (Branch)

Managing Storage Cabinets

Your first task is to create a possibility to commission and decommission storage cabinets. Each cabinet shall have a user defined name. The name can be changed and must be unique over all cabinets. The team creates an application service for each use case and of course the StorageCabinet class which will reside in the domain package and will implement the domain logic. The domain itself also contains a sealed class of DomainErrors, e.g. NotFound and AlreadyExists and a StorageValidator which contains the validation logic maybe needed in multiple places - for now this checks if the name of the cabinet is already taken.

We will not look at the implementation, if you like you can have a look here. But we will look at some test cases.

The team realizes using an in-memory repository has a big advantage when writing tests. No need to fire up a database or mock the repository. Each test case can just create its own in-memory repository. They also decide to use KoTest with the DescribeSpec to write tests.

//...
describe("Commission a new StorageCabinet") {
    it("adds new cabinet to repository and returns it") {
        val repository = StorageInMemoryPersistence()
        val validator = StorageValidator(repository)
        val newName = StorageCabinetName("Cabinet 000")
        val expectedCabinet = StorageCabinet(StorageCabinetId.new(), newName)
  
        val newCabinet = StorageCabinet.commission(newName, repository, validator)

        newCabinet shouldBeSuccess {
            it.shouldBeEqualToIgnoringFields(expectedCabinet, StorageCabinet::id)
        }
    }

    it("returns already existing when cabinet with same name is already present") {
        val repository = StorageInMemoryPersistence()
        val validator = StorageValidator(repository)
        val duplicatedName = StorageCabinetName("Cabinet 000")
        StorageCabinet.commission(duplicatedName, repository, validator)

        val newCabinet = StorageCabinet.commission(duplicatedName, repository, validator)

        newCabinet shouldBeFailure {
            it shouldBeEqualUsingFields ConstraintViolation("Name already used")
        }
    }
}
//...

The team just finished the implementation and the tests. They are quite happy with how quickly this went. And the tests do not look too bad, right? A bit of setup to create the repository, the validator and storage cabinets for the test. But this is easily copied.

Link to the code

Adding rooms

The next task is to add rooms to where the cabinets are located. The team discusses if there should be a room containing storage cabinets, or if the cabinet holds the room where it is located? They decided to go with the latter model and the StorageCabinet just will hold a RoomName for now.

Adding a room was easily done. Add it to the constructor of the StorageCabinet, add it to the StorageCabinet.commission factory method, create an updateRoom method in StorageCabinet and a ChangeRoomOfStorageCabinet UseCase. Next up are the tests. All of them fail. They even stopped compiling. What happened? Well, adding a new constructor parameter and a new parameter to the factory method broke them.

A quick fix is to add the parameter where needed.

//...
describe("Commission a new StorageCabinet") {
    it("adds new cabinet to repository and returns it") {
        val repository = StorageInMemoryPersistence()
        val validator = StorageValidator(repository)
        val newName = StorageCabinetName("Cabinet 000")
        val expectedCabinet = StorageCabinet(StorageCabinetId.new(), newName, Room(RoomName("Room")))

        val newCabinet = StorageCabinet.commission(expectedCabinet.name, expectedCabinet.room, repository, validator)

        newCabinet shouldBeSuccess {
            it.shouldBeEqualToIgnoringFields(expectedCabinet, StorageCabinet::id)
        }
    }

    //...
}
//...

This was done quickly for the handful of tests currently existing. And also for only one parameter. You realize this may be an issue later when the code base has grown. You remember a talk with a colleague, let’s call im Marc (yes, the one from above), talking about good experience with a central test data factory allowing you to create a default test object and build domain objects easily. The team is willing, the schedule not pressing, and you find the idea of a test data factory quite appealing anyway. Time to finally do this from the start on not always plan to create one later. Also, the team can rely on Marc’s experience and may be able to not learn the pitfalls of this strategy by falling into them.

Link to the code

The TestData Factory

There are multiple goals and things needed to get this working. First let us look at the goals.

  1. We want to be able to create domain objects with custom values
  2. We want a single stop where we can branch out from for creating test data objects
  3. We want a single stop to get a default domain object. It may contain random values e.g., has a room set

The first goal is easily achieved using a builder pattern:

class StorageCabinetBuilder {

    private var name = StorageCabinetName("Storage-${generateRandomString(10)}")
    private var room = RoomBuilder().build()
    private var id = StorageCabinetId.new()

    fun withId(id: StorageCabinetId): StorageCabinetBuilder {
        this.id = id
        return this
    }
    //...
     fun build(): StorageCabinet {
        return StorageCabinet(id, name, room)
    }
}

Fulfilling the second and the third goal is also easily done by using a singleton TestDataFactory:

object TestDataFactory {

    fun storageCabinetBuilder() = StorageCabinetBuilder()
    fun createRandomDefaultStorageCabinet() = StorageCabinetBuilder().build()

    fun roomBuilder() = RoomBuilder()
    fun createRandomDefaultRoom() = RoomBuilder().build()
}

Last but not least, the team refactors the tests:

//...
describe("Commission a new StorageCabinet") {
    it("adds new cabinet to repository and returns it") {
        val repository = StorageInMemoryPersistence()
        val validator = StorageValidator(repository)
        val expectedCabinet = TestDataFactory.createRandomDefaultStorageCabinet()

        val newCabinet = StorageCabinet.commission(expectedCabinet.name, expectedCabinet.room, repository, validator)

        newCabinet shouldBeSuccess {
            it.shouldBeEqualToIgnoringFields(expectedCabinet, StorageCabinet::id)
        }
    }

    //...
}
//...

Not much changed but, in the first test example, we do not create all the cabinets values line by line but take it from the random default cabinet. At least the team now feels a bit more confident in keeping the test data generation manageable.

Putting Boxes into the Cabinet

Next up is adding StorageBoxes into the cabinet. Each box needs a name and a description. Each box has a specific content and only exists once. Therefore, the Box also needs an ID. Easily done. The team models this box as a domain entity. Clarifying requirements lead to two more discoveries. When a StorageCabinet is commissioned, it must be empty, it must also be empty when it is decommissioned.

The team takes this step by step. First, they add the StorageBox domain object together with a StorageBoxBuilder which is registered in the TestDataFactory. They also add the logic to add and remove a StorageBox from the StorageCabinet and make sure a StorageCabinet is commissioned as an empty one. This also includes the tests.

Then, they add a storage box into the default StorageCabinet created via the TestDataFactory. This leads to a failure of the test commissioning a StorageCabinet and adding a method to create an empty StorageCabinet via TestDataFactory and clarifying the test by using the empty cabinet in the code and renaming it to adds new empty cabinet to repository and returns it".

Next up is making sure to not allow decommissioning a storage cabinet while there are still boxes in it. Running the test, the team sees the test for decommission a cabinet failing. They are thrilled to see this. This is the first time to see the creation of a default test storage cabinet in action. The default cabinet has a box in it and should not be decommissioned. If this had been a regression it would have been caught! As this is desired behavior, the team now adds a new test case testing if decommission works with an empty StorageCabinet and changed the test name for the case when the StorageCabinet is not empty.

While this looks promising and the team is quite happy time allows you to revisit the tests again.

Link to the code

Using a Fixture

Looking at the tests created so far, you see a lot of repeating code. Every time you start a test case which needs a StorageCabinet already created and in the repository you see something like this.

val repository = StorageInMemoryPersistence()
val existingCabinet = TestDataFactory.createRandomDefaultStorageCabinet()
repository.add(existingCabinet)

Yes, this is not much. And it is easily copied. But somehow you wonder if this could be done more cleanly. For the lack of a better name you decide to create a StorageCabinetTestFixture together with a TestFixtureFactory. While the factory makes sure to create always the same fixture with the same StorageCabinetName and so on, the fixture should allow for easy access to the default cabinet with its box. The fixture also should allow easily adding more cabinets and offering general helper methods when writing tests.

class StorageCabinetTestFixture(val cabinet: StorageCabinet, val repository: StorageRepository) {
    val boxInCabinet = cabinet.storageBoxes.first()
    val room = cabinet.room
    val validator = StorageValidator(repository)

    fun createRandomNewCabinet() = TestDataFactory.createEmptyDefaultStorageCabinet()
    fun createRandomCabinet() = TestDataFactory.createRandomDefaultStorageCabinet()

    fun addNewCabinet() = repository.add(TestDataFactory.createEmptyDefaultStorageCabinet()).getOrThrow()
    fun addRandomCabinet() = repository.add(TestDataFactory.createRandomDefaultStorageCabinet()).getOrThrow()
}

The tests become even clearer

it("updates cabinet name if the name is not yet existing in the repository") {
    val repository = StorageInMemoryPersistence()
    val validator = StorageValidator(repository)
    repository.add(TestDataFactory.createRandomDefaultStorageCabinet())
    val existingCabinet = TestDataFactory.createRandomDefaultStorageCabinet()
    repository.add(existingCabinet)

    val expectedCabinet = existingCabinet.copy(name = StorageCabinetName("newName"))
    require(repository.getAll().size == 2)

    val result = existingCabinet.updateName(expectedCabinet.name, validator)

    // returns the updated cabinet
    result shouldBeSuccess {
        it shouldBeEqualUsingFields expectedCabinet
    }

    // makes sure the instance is updated
    existingCabinet shouldBeEqualUsingFields expectedCabinet
}

becomes

it("updates cabinet name if the name is not yet existing in the repository") {
    val fixture = TestFixtureFactory.createStorageCabinetFixture()
    fixture.addRandomCabinet()
    val existingCabinet = fixture.cabinet

    val expectedCabinet = existingCabinet.copy(name = StorageCabinetName("newName"))
    require(fixture.repository.getAll().size == 2)

    val result = existingCabinet.updateName(expectedCabinet.name, fixture.validator)

    // returns the updated cabinet
    result shouldBeSuccess {
        it shouldBeEqualUsingFields expectedCabinet
    }

    // makes sure the instance is updated
    existingCabinet shouldBeEqualUsingFields expectedCabinet
}

No need to manually create the repository, validator, adding a cabinet. You even can write val cabinet = fixuture.addEmptyCabinet() to add a new empty cabinet to the test case (and into the repository for this test). Further, you realize you only have one place where you can change the creation of e.g., the repository used in tests. Here an in-memory repository is used. But what if you want to change it with a database backed? Or mock it altogether? This is the one place where you can address this. The test case itself should not bother. At least this is what you are hoping for. You can’t wait to discuss this with the team and see to what this evolves in the future.

Link to the code

Conclusion

I do hope this little story helps you when thinking about test data management and test design. It surely helped me bring structure into my thoughts. If you have any comments about this story or concept, feel free to contact me on Linkedin - or if you read this on Medium write a comment.