Thoughts On Working With Nested Monad Within The Future Monad In Scala

This post is about nested contexts, specifically with Future, or more accurately, nested monad within the Future monad. eg: Future[Either[E, A]], Future[Option[A]] etc. And how such nested context could easily lead to hard to read and hence hard to maintain codebase.

These sort of context nesting and its impact is not limited to Future, but I would be using Future as the reference in this post, strictly because they seem to be the form that appears the most in a particular Scala codebase I work on at my day job.

Also in this post, I would use Monad and Context interchangeable to mean the same thing. I would also mention terms like Monads, Monad Stack, Monad Transformers, or For Comprehension without providing any explanation of these terms. This is because the thrust of this post is not to expound on these concepts but to share some recent thoughts I have been having regarding nested context within Future.

A reader not familiar with these concepts should still be able to follow along though.

Why We End Up With Nested Context Within Future.

Why do we end up with Future of an Either or Future of an Option or, in a play application Future of a JsResult?

If you are developing in Scala and you are only using the standard library (i.e no Cats or Scalaz), then sooner or later, you would end up having to use/deal with Future. 

This is almost inevitable as you would have to perform asynchronous operations like making a remote HTTP call or executing a database query. The library or framework you use for such operations would most likely return the results of these operations in a Future.

And since you also want to make use of the Scala type system to model your program properly, you end up with API’s that encode the result of their asynchronous actions not just as a Future of a plain value, but as a Future of values of type Either, Option etc.

For example, an HTTP request to get a value of type A, can successfully return with that value or fail with an exception E. Using Scala's type system to model our application to be as descriptive as possible, we make use of the Either type to capture this possibility of an error. So instead of having an API that returns a Future[A], we end up with a return type of Future[Either[E, A]]

And there goes our first embedded context within a Future.

My submission in this post is that, often than not, such nested contexts in Future is a code smell and should be avoided at all cost since it leads to a hard to read code. This conclusion is borne out of my experience with a Scala code base at work which is riddled with such nested Future. 

The rest of this post makes this case.

Why nesting Contexts within Future is bad

To make this case, let us imagine we have two functions. A getIdByEmail function that retrieves a user ID given an email, and a getPostsById function that retrieves a list of posts given a user ID. These two functions return a value of Future of an Either.

Let us imagine the implementations of these two functions is as follows:

// getIdByEmail

import scala.concurrent.{Await, Future}
import scala.language.implicitConversions
import scala.concurrent.ExecutionContext.Implicits.global

def getIdByEmail(email:String): Future[Either[String, Int]] = Future {
 if (email.endsWith("@gmail.com")) {
   Right(1)
 } else if (email.endsWith("@yahoo.com")) {
   Right(2)
 } else {
   Left("No User with given email")
 }
}

and

// getPostsById

import scala.concurrent.{Await, Future}
import scala.language.implicitConversions
import scala.concurrent.ExecutionContext.Implicits.global


def getPostsById(id:Int): Future[Either[String, List[String]]] = Future {
 if (id == 1) {
   Right(List("Post title 1", "Post title 2"))
 } else {
   Left("No posts found")
 }
}

We can then make use of these two functions to retrieve a list of post given an email. Such an implementation that makes use of these functions might look like this:

val result = getIdByEmail("mee@gmail.com").flatMap({
 case Right(id) =>
   getPostsById(id).map {
     case Right(posts) => posts
     case Left(error) => Future {Left(error)}
   }
 case Left(error) => Future {Left(error)}
})

println(Await.result(result, 3.seconds))

//updated: initially was 'case Right(books) => books'

I do not know about you, but code that looks like the above always leaves a bad taste in my mouth.

The combination of nested map/flatMap coupled together with pattern matching does not make for code that is easy to read.

The snippet above is using only just two functions that return nested context in a Future, and yet we are already having code that does not read easily at first glance. What happens if we throw a third or fourth function into the mix? It gets unwieldy really quickly.

So is there anything we can do to improve the readability? What if we try using for comprehension? Since we know it provides syntax sugar for maps and flatMaps, maybe that can help in improving the readability? Let us see.

Our first attempt at using for comprehension does not work out well as intended. The following code:

val result = for {
  id <- getIdByEmail("mee@gmail.com")
  posts <- getPostsById(id)
} yield posts

println(Await.result(result, 3.seconds))

Leads to the following compilation error:

Error:(44, 30) type mismatch;
 found   : Either[String,Int]
 required: Int
      posts <- getPostsById(id)

The reason why we get this error is due to the fact that Monads do not compose

Is there a possibility to wrangle the code, still using for comprehension and make it compile?. It turns out we can pull that off.

Such an incarnation would look like this:

val result = for {
    eitherAnIdValue <- getIdByEmail("dadepo@outlook.com")
} yield for {
    id <- eitherAnIdValue
} yield for {
   posts <- getPostsById(id)
} yield posts

println(Await.result(result, 3.seconds))
//updated: initially was 'books <- getPostsById(id)'

Again, I do not know about you, but I do not want to write, neither do I want to read code like this just to perform an operation that technically involves two steps: look up an Id by email, use the found Id to lookup associated posts.

If this is the kind code that Functional programming in Scala leads to, then I do not want to have anything to do with it!

The two code snippet above is why I have come to the conclusion that a function with the return type of Future[Either[E, A]], or Future[Option[A]] or something similar is usually a code smell. This is because, as seen above, they lead to unnecessary convoluted and unwieldy code.

How to deal with this? Is there a better way?

I currently can make two propositions for preventing or avoiding this unwieldy code that nested context leads to: Encode the failure in Future or Use Monad Transformers.

Let’s explore these two propositions.

Encode the Failure in Future

The Future type in Scala comes with a mechanism for encoding error within it. Although it is might not be the most robust way, it is definitely an option.

This can be done by calling the failed method with an instance of an exception representing the error. For example:

Future.failed(new UnsupportedOperationException("not implemented"))

This mechanism can then be used to encode exceptional scenario instead of using Either (or Option, or JsResult etc). Leading to a simpler return type of the form Future[A] where A represents the type of the returned value of interests.

This then mean that the functions can be used easily in a for comprehension. The only caveat being that a recover block needs to be introduced to take care of the situation where an exception is encountered.

Applying this approach would see the getIdByEmail and getPostsById method take the following form:

def getIdByEmail(email:String): Future[Int] = {
 if (email.endsWith("@gmail.com")) {
   Future.successful(1)
 } else if (email.endsWith("@yahoo.com")) {
   Future.successful(2)
 } else {
   Future.failed(new Exception("No User with given email"))
 }
}

and

def getPostsById(id:Int): Future[List[String]] = {
 if (id == 1) {
   Future.successful(List("Post title 1", "Post title 2"))
 } else {
   Future.failed(new Exception("No posts found"))
 }
}

And the code that makes use of these two methods would now look like this:

val result = {
  for {
    id <- getIdByEmail("dadepo@outlook.com")
    posts <- getPostsById(id)
  } yield posts
} recover {
    case e:Exception => Future.successful(e.getMessage)
}

println(Await.result(result, 3.seconds))
// updated: initially was 'books <- getPostsById(id)'

Or better still, to make it more readable, break into two steps like this:

val futureResult = for {
   id <- getIdByEmail("dadepo@outlook.com")
   posts <- getPostsById(id)
} yield posts

val result = futureResult recover {
  case e:Exception => Future.successful(e.getMessage)
}

println(Await.result(result, 3.seconds))
// updated: initially was 'books <- getPostsById(id)'

I personally think this approach of encoding the error in the Future results in more readable code.

The only drawback though, is that this approach undermines the type safety since it effectively removes the exception from the type system. It then makes it easier for a user to forget about handling the possible exceptions that can emanate from the Future in a "recover" step, hence leading to things blowing up at runtime.

I am still inclined to adopt this approach than having to deal the cumbersomeness of nested context within Future.

The second approach, which uses Monad transformers is explained next.

Use Monad Transformers

The other approach that helps in dealing with a nested context within Future is the use of monad Transformers.

Types like Future[Either[A, B]] can be more accurately described to be a monad inside another monad or stack of monads. And due to the nature of monads, having a stack of them this way does not lend to convenient usage within a for comprehension.

The way to make them less cumbersome is to make use of Monad Transformers which effectively creates a monad from the combination of two or more monads. The resulting monad can then, for all intent and purpose be treated as a single standalone monad entity.

The Scala standard library gives us monads (types like Future, List, Option, Try etc are all monadic if not monads in the true, law-abiding sense of what a monad is) but does not give us the tools to effectively deal with a stack of monads.

To make use of Monad transformers, we would need to bring in libraries like ScalaZ or Cats.

In this post, I will make use of Cats. The code snippet in the rest of the post below shows how Monad Transformers, has provided by Cats, makes programming with types like Future[Either[E, A]] less cumbersome.

Adding Cats to your project, if you use sbt would look like this:

libraryDependencies +=
"org.typelevel" %% "cats-core" % "current.version.number"
scalacOptions ++= Seq(
"-Xfatal-warnings",
"-Ypartial-unification"
)

After Cats is added, we can then make use of the EitherT monad transformer.

Let us have the getIdByEmail and getPostsById function in their initial form where they return an Either inside a Future. That is:

// getIdByEmail

import scala.concurrent.{Await, Future}
import scala.language.implicitConversions
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import cats.data.EitherT
import cats.instances.future._

def getIdByEmail(email:String): Future[Either[String, Int]] = Future {
 if (email.endsWith("@gmail.com")) {
   Right(1)
 } else if (email.endsWith("@yahoo.com")) {
     Right(2)
 } else {
   Left("No User with given email")
 }
}

and

// getPostsById

import scala.concurrent.{Await, Future}
import scala.language.implicitConversions
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import cats.data.EitherT
import cats.instances.future._


def getPostsById(id:Int): Future[Either[String, List[String]]] = Future {
 if (id == 1) {
   Right(List("Post title 1", "Post title 2"))
 } else {
   Left("No posts found")
 }
}

With the EitherT monad transformer, we can then make use of these two methods in a for comprehension in the following manner:

val result = for {
    id <- EitherT(getIdByEmail("mee@gmail.com"))
    posts <- EitherT(getPostsById(id))
} yield posts

println(Await.result(result.value, 3.seconds))

As you can see, this removes all the boilerplate and clunkiness of working with nested Either in a Future. Problem solved!

Conclusion

The Monad Transformer approach is definitely the more robust way to deal with a stack of monads. Which is what a type of Future[Either[E, A]] is. It makes working with them easier without sacrificing type safety.

I definitely recommend using Monad transformers over the other approach of encoding the failure scenarios in the Future itself.

In the case where you cannot use libraries like Scalaz or Cats then I think encoding the exceptional scenarios in the Future would be the way to go, and it would lead to a more readable code when compared to all the nested map/flatMap and pattern matching that would be required otherwise.

So far, these are the two approaches I can think of when it comes to dealing with having another monad nested in the Future monad in Scala. If by chance, you know of any other best practice regarding dealing with a stack of monads in Scala? then please do share in the comment section.👇

Updates
Published Rolling Your Own Monad To Deal With Nested Monads In Scala which looks at the approach of creating a wrapper Monad as a means for making working with nested monads easier.