Rolling Your Own Monad To Deal With Nested Monads In Scala

In Thoughts On Working With Nested Monad Within The Future Monad In Scala, I wrote about how nested contexts (or technically accurate: nested monads) usually end up creeping into any Scala codebase. And how these nested contexts end up leading to unwieldy code base, that is hard to read and hard to work with.

In that same post, I then mentioned two techniques that can be used to regain readability when dealing with such nested monads. Since the post used the Future Monad as its focus, the first suggested technique was to encode the failure in the Future. The second technique was to use monad transformers.

The blogpost inspired a talk that I gave at the Amsterdam Scala Meet-up group. (You can find the slides from that talk here by the way). In the discussions that ensued after the talk, Devlaam suggested an alternative technique for dealing with the situation: which is to basically create a monadic structure that wraps around the value in the nested context.  Doing this would allow us to regain readability by being able to use for comprehensions with the created monadic structure.

In this post I explore how the idea of rolling your own monad can be applied as a third approach to dealing with the clunkiness of working with nested monad in Scala. Just as with the previous post, Future[Either[E, A] would be the nested monads that would be used for illustrations.


As a refresher, here is a quick recap of the updated code snippet used in the illustration.

Objective

1. Look up a user Id by email
2. Convert id to author id
3. Use the author id to lookup associated posts

Provided Functions
1. Look up a user Id by email

def getIdByEmail(email:String): Future[Either[String, Int]] = Future {
 if (email.endsWith("@gmail.com")) {
   Right(10)
 } else if (email.endsWith("@yahoo.com")) {
   Right(20)
 } else {
   Left("No User with given email")
 }
}

2. Convert id to author id

def convertToAuthorId(id:Int): Future[Either[String, Int]] = Future {
 if (id == 10 || id == 20) {
   Right(id / 10)
 } else {
   Left("No Author ID")
 }
}

3. Use the author id to lookup associated posts

def getPostsByAuthorId(id:Int): Future[Either[String, List[String]]] 
  = Future {
 if (id == 1) {
   Right(List("Post title 1", "Post title 2"))
 } else {
   Left("No posts found")
 }
}

Monads, Sequencing Computations and The Problem of Nested Monads.

For the purpose of this post, we will safely approximate the definition of a monad-like structure in Scala to be any type with the flatMap and apply method.

I qualified this an an approximate definition because in reality Monads are more than a method with a certain type signature. There are also laws that needs to satisfied before something can be regarded as a Monad. But since I do not intend to turn this post into yet another Monad tutorial, we will stick with this approximate definition.

The other thing that I will quickly point out, also without going into details is the motivation for having Monads. Monads provide us with a mechanism for chaining function applications, with the caveats that the intermediate values generated as output from the functions may or may not be in a form that allows it to be easily fed as input into the next function in the computation chain.

Which is basically the situation we have with these three functions above.

We want to take the value generated by the getIdByEmail function and pass it as input to the second function which is convertToAuthorId. The second function then takes this value and use it to generate another value that would be fed into the 3rd function, getPostsByAuthorId which then computes the final value.

But the intermediate values generated by these functions are not guaranteed to be there right away, neither are they guaranteed to be valid. They are not guaranteed to be there right away because the functions are processed in an asynchronous fashion and their values only materialised after some time. They are also not guaranteed to be valid values that can be used by the next function. For instance, it is possible that no user id exist for a given email and hence the value returned from calling getIdByEmail can never be possibly one that can be fed to the convertToAuthorId function.

But because Future and Either are more or less Monads in Scala (they have flatMaps) we can still sequence these three computations together.

This we do with the following code:

val result:Future[Either[String, List[String]]] 
   = getIdByEmail("me@gmail.com").flatMap({
 case Right(id) => convertToAuthorId(id).flatMap({
   case Right(authorId) => getPostsByAuthorId(authorId)
   case Left(err) => Future {Left(err)}
 })
 case Left(err) => Future{Left(err)}
})

println(Await.result(result, 3.seconds))

But because we are dealing with nested monads, this sequencing, this taking of output and feeding it as input into the next function ends up being clunky. And we can’t totally do away with some manually checking of the validity of the intermediate values even when we use flatMaps (which is supposed to allow us sequence these kind of computation without having to directly worry about checking if the intermediate values are valid and usable as inputs to the next functions)

Monads and For Comprehension in Scala

In Scala, a for comprehension without the yield portion signifies iteration. For example:

for (x <- List(1,2,3,4,5,6,7,8,9,10)) {
 if (x % 2 == 0) println(x)
}

Loops through a list containing numbers from 1 to 10 and if a number is even, it is printed.

This is the basic idea of iteration that you would expect from a syntax that uses the "for" keyword, especially if you have experience with Java, C#, JavaScript etc where similar construct denotes iteration.

On the other hand, a for comprehension with a yield portion, stops being about iteration. Such a construct in Scala, is then about sequencing computations over Functors (things with a map method) and Monads (things with a flatMap method). For example this:

for {
 x <- Option(1)
 y <- Option(2)
} yield x + y

Is about taking the value from the first Option monad, take the value from the second Option monad, add the values together and return it as a new value in an Option monad. Nothing to do with iteration.

The above operation could have been written without using for comprehension as:

Option(1).flatMap(x => Option(2).map(y => x + y))

But as can be seen, directly using flatMaps and Maps to sequence computations do not usually lead to code that is easy to read.

It so happens that Scala provides the for comprehension as a more pleasant syntax for writing computations that would normally be written by chaining flatMaps and maps. In essence, the for comprehension provides a more pleasant syntactic sugar to flatMaps and maps.

The nice thing about the for comprehension is that its usage is not limited to Option, or Future, or Try or any other type in the standard library that has Monadic structure. They can also be used with any custom type, as long as that custom type defines a set or subset of certain methods.

These methods include map, flatMap, withFilter, and filter. For more on the exact mechanism of how this work, you can consult How does Yield work

So in essence, at minimal, any type with map and flatMap defined on them, can easily be used in a for comprehension to sequence monadic computations.

So how does this insight help us with the problem we started of with? Which is dealing with nested monads in Scala? How does it lead to a third alternative, different from the two specified in Thoughts On Working With Nested Monad Within The Future Monad In Scala?

The idea is this, we create a custom type that wraps over the nested context we want to deal with. We make sure to implement the map and flatMap method for this custom type, making it monadic, And because we defined these methods, we can then use our custom type easily within the for comprehension.

Let's see how this look in practice.

Rolling Your Own Monad

Going back to our original set of functions:

1. getIdByEmail: String => Future[Either[String, Int]]
2. convertToAuthorId: Int => Future[Either[String, Int]]
3. getPostsByAuthorId Int => Future[Either[String, List[String]]]

..and the computation which we want to perform:

1. Look up a user Id by email
2. Convert id to author id
3. Use the author id to lookup associated posts

We can create a custom type that wraps over the nested Future[Either[E, A]] that we are dealing with.

Such custom type would look like this

case class FutureOfEither[E,A](value: Future[Either[E, A]]) {
 def map[B](f: A => B): Future[Either[E, B]] = {
   value.map {
     case Right(thing) => Right(f(thing))
     case Left(err) => Left(err)
   }
 }

 def flatMap[B](f: A => Future[Either[E, B]]): Future[Either[E, B]] = {
   value.flatMap {
     case Right(thing) => f(thing)
     case Left(err) => Future.successful(Left(err))
   }
 }
}

Because we defined map and flatMap on FutureOfEither class, We can then use it in a for comprehension, together with the set of functions that returns Future[Either[E, A].

This will look like this:

val result1 = for {
 id <- FutureOfEither(getIdByEmail("me@gmail.com"))
 authorId <- FutureOfEither(convertToAuthorId(id))
 posts <- FutureOfEither(getPostsByAuthorId(authorId))
} yield posts

This way, we regain the ability to simply chain our computation using the for comprehensions. This improves readability.

In case we think having to write FutureOfEither in front of the methods could end up being tiresome, we can use extension methods, via Scala’s implicit class machinery to reduce some of this verbosity. This we do by adding a method to Future[Either[E, A]] type that automatically converts it to FutureOfEither.

Such an implicit class would look like this:

implicit class PointSyntax[E, A](value: Future[Either[E, A]]) {
 def point:FutureOfEither[E,A] = FutureOfEither(value)
}

Having this implicit class in the implicit scope would enable the ability to write the code as follows:

val result2  = for {
 id <- getIdByEmail("me@gmail.com").point
 authorId <- convertToAuthorId(id).point
 posts <- getPostsByAuthorId(authorId).point
} yield posts 

Which can be said to be less verbose and removes some of the obfuscation with having FutureOfEither in front of the actual functions involved in the computation.

The full code snippet that makes use of this approach is presented below:

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}
import scala.language.implicitConversions

case class FutureOfEither[E,A](value: Future[Either[E, A]]) {
  def map[B](f: A => B): Future[Either[E, B]] = {
    value.map {
      case Right(thing) => Right(f(thing))
      case Left(err) => Left(err)
    }
  }

  def flatMap[B](f: A => Future[Either[E, B]]): Future[Either[E, B]] = {
    value.flatMap {
      case Right(thing) => f(thing)
      case Left(err) => Future.successful(Left(err))
    }
  }
}

// lookup a user Id by email
def getIdByEmail(email:String): Future[Either[String, Int]] = Future {
  if (email.endsWith("@gmail.com")) {
    Right(10)
  } else if (email.endsWith("@yahoo.com")) {
    Right(20)
  } else {
    Left("No User with given email")
  }
}

def convertToAuthorId(id:Int): Future[Either[String, Int]] = Future {
  if (id == 10 || id == 20) {
    Right(id / 10)
  } else {
    Left("No Author ID")
  }
}

def getPostsByAuthorId(id:Int): Future[Either[String, List[String]]] = Future {
  if (id == 1) {
    Right(List("Post title 1", "Post title 2"))
  } else {
    Left("No posts found")
  }
}

implicit class PointSyntax[E, A](value: Future[Either[E, A]]) {
  def point:FutureOfEither[E,A] = FutureOfEither(value)
}

val result1 = for {
  id <- FutureOfEither(getIdByEmail("me@gmail.com"))
  authorId <- FutureOfEither(convertToAuthorId(id))
  posts <- FutureOfEither(getPostsByAuthorId(authorId))
} yield posts

println(Await.result(result1, 5.seconds))

// using the extension method.
val result2  = for {
  id <- getIdByEmail("me@gmail.com").point
  authorId <- convertToAuthorId(id).point
  posts <- getPostsByAuthorId(authorId).point
} yield posts

println(Await.result(result2, 5.seconds))

Conclusion

The approach of "rolling your own monad" would involve having to write separate monadic wrapper for each unique monadic stack that appears in your codebase. In the examples in this post, the stack was Future[Either[E, A]]. It could also very well be a Future[Option[A]] stack or List[Option[A]] stack or any other combination of Monads. It does comes with the cost of having to manually set up some machinery code for each unique case of nested monads.

That been said, it could be considered as a homegrown alternative to having to use Monad Transformers, which would involve having to include a library like Cats in a project.

Looking at the 3 approaches I have been able to identify for dealing with stacks of monads, I would still recommend introducing Cats and going with monad transformers. This is because Cats is more than just Monad transformers. It is basically a toolkit of useful abstractions that makes functional programming in Scala more principled, less tedious and if I may say, more fun.

So you should use Cats, because you will find other areas to derive value from it, apart from helping to deal with stack of Monads.

But in a case you can’t use Cats (for whatever reasons that may be), then I would rather go with rolling your own monad: yes it requires having to write some boilerplate, infrastructure-like code, it still beats having to deal with nested flatMaps and maps any day.

As I mentioned in the comment section in the previous post.

"I am continually learning and on the lookout for best practices on how to deal with scenarios like this, so If I find a better approach: which might be based on machinery provided in up-coming Scala 3, or in another abstraction/pattern, I would definitely update this post."

So do expect another post once I learn a new approach for dealing with nested Monads situation in Scala or better still how to avoid having Nested monads in the first place :)

If you’ve got some insights around this topic; perhaps another technique, please do not hesitate to share in the comment section 👇 It will be appreciated!