Understanding Implicits in Scala

I knew about the maxim: "Implicits are evil" way before I started learning and using Scala. So by the time I started running into them in codebases, I was already biased. It was thus easier to raise my hands in despair at the "magic" that is implicit and just condemn their existence.

Obviously such a view is not healthy. So I decided to do what any sane human would do: If I am going to continually bitch about something, I should at least know some one or two things about what I am bitching about.

This post captures my understanding of how implicits work in Scala thus far. Not in the sense of how they work at the compiler or implementation level, but how they work in the sense of what they do and how to reason about them when encountered in a codebase.

It is not about when to make the decision to use or not use implicits. Or how to use them "tastefully". My current level of mastery of the Scala language does not permit me to make such pontification. Hopefully in the nearest feature, a post like this would follow.

Should in case you were like me a couple of months back, and just want a clue about what the hell is the magic going on with implicit, then you will find this post useful.

So let us get right to it.

Grokking Implicits

I have seen many explanation of implicits in scala, and most of these explanations approach the subject from the point of view of the features provided by implicits.

I, on the other hand, came to grokk implicits by viewing them from the perspective of the mechanics of what seems to go on when the use of an implicits is triggered in Scala.

Which is, the last attempt by the Scalar compiler to prevent a compilation error.

Compilation Errors and implicits

Compilation errors occur when the compiler fails to compile a source code due to faults in the way the program has been constructed. There are various types of faults that can lead to a compilation error: From erroneous syntax, to improper use of the type system etc.

The implicits mechanism in Scala kicks in when the Scalar compiler encounters a certain set of compilation errors. Before giving up, it then tries to use the facilities provided by implicits to resolve the problem, prevent the compilation error and continue with a successful compilation.

The preceding paragraphs mentions that the implicit mechanism kicks in, only for a certain set of compilation errors. It is important to stress this because the Scalar compiler does not attempt to use implicits to resolve all kinds of compilation errors. For example if you have a compilation error due to wrong syntax, the compilation would fail without the compiler trying to resolve it.

The set of compilation errors which kicks in the usage of implicits include:

Compilation Errors Due To Type Mismatch

1. Variable Type Mismatch: A variable expects a value of a certain type. Another kind was given.
2. Parameter Type Mismatch: A function or method which requires to be applied to values of a certain type, but is applied to values of a different type.

Compilation Errors Due to Missing Function/Method Parameters
A method/function defined to accept a parameter is called without the parameter (or complete parameter list)

Member Mismatch
A method is called on an object, but the object does not have the method defined.

Now let us explore, a little deeper, these cases of compilation errors and how implicits come to play.

Compilation Errors Due To Type Mismatch

As stated above, there are two kinds of compilation errors that fall under this category. We look at each.

Variable Type Mismatch
A variable expects a value of a certain type. Another kind was given. For example:

val answerToLifeUniverseAndEverything:Int = "42"

The variable above is defined as an Int, but a String of value 42 was given. This would lead to a compilation error:

Error:(14, 51) type mismatch;
 found   : String("42")
 required: Int
lazy val answerToLifeUniverseAndEverything:Int = "42"

How does the compiler use implicits to attempt to prevent this kind of compilation errors? By looking for a function that can be used to convert the given type to the required type. i.e. a function that converts from a String to an Int.

For example, the code snippet below would resolve the error and the program would successfully compile.

implicit def toInt(given:String): Int = {
  given.toInt
}

val answerToLifeUniverseAndEverything:Int = "42"

This is because the function toInt has been added. The function is defined to take in a String and returns an Int. Note the implicit keyword before the definition. This keyword is what marks the function as one that can be used by the scalar compiler to resolve compilation error.

So when the compiler encounters an attempt to assign a String into a variable defined to take an Int, before resulting into a compilation error, it first looks to see if there is any function marked with implicit keyword that can be used to convert the given String value into an Int value. It finds the toInt method, which is used to implicitly convert the String to Int, and thus the compilation error is avoided.

To be more explicit, what the compiler does is in effect this:

val answerToLifeUniverseAndEverything:Int = toInt("42")

Parameter Type Mismatch

Having a method/function called with arguments of types that differ to the ones it is defined to accept would lead to a compilation error:

def sayAnswer(ans:Int) = {
  println(s"The answer to Life, Universe and Everything is $ans")
}

sayAnswer("42")

sayAnswer is defined to take an argument of type Int, the function was called with a String, this would lead to the compilation error:

Error:(22, 12) type mismatch;
found : String("42")
required: Int
sayAnswer("42")
^

This can also be resolved by having a method marked as implicit that can convert the wrong type into the expected type. We use the same toInt method we used before.

implicit def toInt(given:String): Int = {
  given.toInt
}

def sayAnswer(ans:Int) = {
  println(s"The answer to Life, Universe and Everything is $ans")
}

sayAnswer("42")

This would compile because of the presence of the implicit toInt method. In effect, what the compiler does in other to prevent this compilation error is this:

sayAnswer(toInt("42"))

Seeing it explicitly, the magic of what implicits do, kind of disappears.

Compilation Errors Due to Missing Function/Method Parameters

A method/function defined to accept a parameter is called without the parameter. For example:

def whatIsTheAnswer(ans:Int): Unit = {
  println(
  if (ans == 42) 
    s"Yes. 42 is the answer to Life, Universe and Everything"
  else
    s"Nope. $ans is not the answer"
  )
}

// calling the method
whatIsTheAnswer

This would lead to an error:

Error:(33, 2) missing arguments for method whatIsTheAnswer in class A$A122;
follow this method with `_' if you want to treat it as a partially applied function
whatIsTheAnswer
^

Because whatIsTheAnswer is defined to be called with an argument but was called without one.

Two steps are needed to make the compiler be able to implicitly recover from this compilation error and still compile successfully.

Basically if we want to be able to call whatIsTheAnswer without supplying an argument (even though it is defined to take the argument), then what we want to do is have the compiler implicitly supply this missing argument for us.

In other to achieve that, we must mark the definition of the parameter in the function as implicit. And then provide a value which the compiler should use as a parameter to the method call if one is not given i.e

implicit val answer = 42

def whatIsTheAnswer(implicit ans:Int): Unit = {
  println(
  if (ans == 42)
    s"Yes. 42 is the answer to Life, Universe and Everything"
  else
    s"Nope. $ans is not the answer"
  )
}

The implicit keyword in the method definition tells the compiler that if an argument is not given, then try to supply one implicitly. The val definition marked as implicit implicit val answer = 42 tells the compiler which value to use in such a situation.

With the above, calling whatIsTheAnswer would work without any compilation error.

You usually see this pattern used with curried function, a facility that allows the calling of a function by supplying only some of the required parameters and have the remaining supplied later one.

For example:

scala> def multiply(x: Int)(y: Int): Int = x * y
multiply: (x: Int)(y: Int)Int

Then calling it with only one of the required two parameters

scala> val timesThree = multiply(3) _
timesThree: (Int) => Int = <function1>

Then later supplying the remaining required parameter.

scala> timesThree(4)
res1: Int = 12

When paired with implicits, you have a pattern where the library supplies the implicit value while the user of the method only supply the absolutely required parameters.

This pattern is used a great deal, within Scala and 3rd party libraries out there. As an exercise for the reader, explore the "!" method in Akka and the apply method of Future object in Scala. You would quickly spot the use of implicits in this fashion.

Member Mismatch

When a method is called on an object, but the object does not have the method defined. This leads to a compilation error. For example:

class AnswerToAll {
  def answer = {
    "The answer to Life, Universe and Everything is 42"
  }
}

then:

println((new AnswerToAll).answer)

Prints the answer because it is defined in the class AnswerToAll, while this:

print((new AnswerToAll).getAnswer)

causes a compilation error:

Error:(33, 26) value getAnswer is not a member of A$A85.this.AnswerToAll
print((new AnswerToAll).getAnswer)
^

Implicits can be used to resolve this issue. This is done by "enriching" the target object, (in this case, an instance of AnswerToAll) with the missing method. Or seen another way, by converting the object into a type which has the missing method.

To do this, we define a class, mark it as implicit and define the primary constructor to take the type to convert from as its only single parameter.

implicit class WithGetAnswer(convertFrom: AnswerToAll) {
  def getAnswer = convertFrom.answer
}

class AnswerToAll {
  def answer = {
    "The answer to Life, Universe and Everything is 42"
  }
}

print((new AnswerToAll).getAnswer)

The above compiles successfully. This is because when the compiler attempts to call getAnswer on an instance of AnswerToAll, it sees the requested method is not defined, but before letting this lead to a compilation error, the compiler sees if there is a class defined with the keyword implicit that takes AnswerToAll as its constructor parameter and defines the getAnswer method. It finds WithGetAnswer method and thus the compilation error is averted.

To see things in more explicit fashion, this is what the compiler in essence is doing:

print(new WithGetAnswer((new AnswerToAll)).getAnswer)

Another method is to define the enriched class and use a method marked as implicit to handle the conversion. For example:

implicit def toGetAnswer(convertFrom: AnswerToAll) = 
                                              new WithGetAnswer(convertFrom)

class WithGetAnswer(convertFrom: AnswerToAll) {
  def getAnswer = convertFrom.answer
}

class AnswerToAll {
  def answer = {
    "The answer to Life, Universe and Everything is 42"
  }
}

print((new AnswerToAll).getAnswer)

This approach uses an older syntax. It is definitely more verbose. The newer syntax of using the implicit with the enriched class is generally advised.

Using implicit in this fashion leads to what is sometimes referred to as "Pimp my Library" pattern.

The Scala standard library also makes extensive use of this pattern. For example, in Scala, the String class is implemented using the String class from Java, but Scala's String class exposes a lot of methods you won’t find in Java’s String class. For example the reverse method.

An exercise for the reader. Find how Scala accomplishes its enrichment of the String with the use of implicits. Tip: look at the StringOps class, and the augmentString method in the Predef object.

This mechanism of attempting to resolve a set of compilation errors by the compiler provides a powerful extension mechanism that can be used to implement interesting features and patterns.

This is why implicits are such a powerful feature in Scala.

Finding Implicits

In all the examples in this post, the implicits are defined right in the same class where the implicit is needed. This is not always the case. Implicits could be defined in other accepted places and the compiler can search for them in those locations.

The other places where the compiler looks for an implicits includes:

1. Right in the same file
2. Implicits brought into scope by an explicit Import statement
3. Implicits defined in the companion object (of both the class being converted to and from)

When the compiler encounters compilation error that it can resolve by finding implicits definition, the compiler would look at these places first. Only when it does not find an implicit that can be used to resolve the error does the compilation fail.

They are other search paths, for more on this topic, consult the article Where does Scala finds Implicits from.

Thinking Implicits

So far so good we have seen the mechanics of implicits. We now know how Implicit works, but what is the best way to think about them?

Here are some guidelines that helps personally in approaching that question.

• If you see a variable marked as implicits in a library or framework, know it is a valid candidate to be inserted into a function call or method call somewhere in your code or the library's.
• If in your code base you see a variable marked as implicits, know it might be a requirement for a library or framework which requires such an implicit to be defined. It could also mean another part of your codebase requires this variable, implicitly.
• If you see a function marked as implicits, know it is used to convert from one type into another. Something that happens when the compiler needs to do some type conversion.
• if you see a class marked with implicit, know it is used to implement the "pimp my library" pattern.
• When you see a method call on an object but that method is not defined in the class, do not pull out your hair at the magic going on, just know there is an implicit conversion going on somewhere.

So when to use implicits? That advice I can't readily give now. But a scenario of "when not to use implicits" that kind of seem obvious because of its apparent "evil" is using implicits to convert from one basic type e.g String into another basic type e.g. Int. Doing such feels like an unnecessary endeavour to cause confusing in a codebase without any apparent advantage.

The short section on Implicit in Effective Scala, should help in providing some guidance on using implicit tastefully.