Four years of Scala

Last December, I finished my work on ProQuest Flow. It was a great experience, with occasionally some periods of sheer agony, because of some technological choices we made. This is the first post in (what might become) a series to see what learned from it. This episode: Scala.

Now, let me first clarify that I joined the team since they had already chosen Scala. Java – from my point of view – is well beyond its expiry date, and it was about time we started working on something new.

I remember visiting JavaOne a decade ago, and being in a room where Joshua Bloch, Neal Gafter and probably some other people were discussing one of the many closure proposals. There were doing a show of hands, to see how many people considered closures to be a valuable addition to the language. Everyone raised his hand, apart from me. Then they were asking who did not want closures in Java. I raised my hand.

Don't get me wrong, I think support for closures is extremely relevant, for any language, but Java – being built with a particular idea in mind – would have to be turned upside down to make it into something that made sense. Either you'd have something that is fragmented and not all that powerful, or you'd have something that would be like a total new language. Apparently the decision has been made to with the first option.

Fine by me. But I'd rather have a language with some conceptual integrity. And Scala, being a statically typed language without the penalty of being statically typed, seemed to be a better approach. And I still think it is.

So I don't regret the decision to go with Scala. However, we did have to learn a few things while we were at it.

Cake

The Cake pattern is often advertised as the way to do dependency injection in Scala. We happily took the advice, until we found ourself in a corner where compilation would take ages and we frequently ran into NullPointerExceptions, because of lazy vals not being initialized yet.

At some point, the pain just became unbearable, and we ripped it out. After that, we just did the wiring ourselves. I guess Cake pattern based dependency injection could be fine for a component kit, but not for doing the plumbing of an entire application.

In general, I think it would be best if I remove the cake pattern from my toolbox, for the future.

Specs2, ScalaTest, …

One of the benefits of Scala is that creating a domain specific language is easy. As it happens, that's also one of the disadvantages. During the lifetime of our project, we went through different incarnations of Specs and Specs2 and more than once felt like running away in despair

We did however find that the immutable specs were impossible to maintain. From my point of view, this is an area where the ambition to be truly functional causes the code to be no longer readable. We didn't feel like sacrificing readability to the Gods of functional programming, and I'm glad we eventually got rid of all (most?) of them.

So the question is: leave it in, or drop it from my toolbox? I think I'm going to keep it in. Despite the fact that it's sometimes annoying, still hard to read and there are some naming conflicts that can drive you mad.

Unfiltered

I like Unfiltered. Unfiltered is like a 2CV. You can look under the hood, and understand what's going on. And handling HTTP requests really is like pattern matching, so it seems like a good fit.

We did make a couple of mistakes though. First of all, Unfiltered itself changed its opinion on how to deal with query parameters. We didn't like what they had first, so we went with our own solution. Our own solution wasn't necessarily great, but it was painless enough to ignore the changes that were made to the framework to make handling query parameters a lot better.

Also, in hindsight, it would have been better to use Unfiltered's primitives to build some higher level pattern matching constructs, instead of piecing together the different matchers for every type of request we had to respond to. Some policies, like naming policies and policies for authorization and authentication, would have probably been quite easily captured in higher level abstractions, like the kits that the Unfiltered documentation started mentioning at some point.

Anyway, we didn't do it, and therefore had to make changes in quite a few places whenever we had to change our conventions. Which is annoying.

The last thing I think we probably shouldn't have done is to have our own Plan abstraction, with some additional imports. That's just gut feeling though.

I'm going to leave Unfiltered in. I still think it's easier to grasp than Spray. Spray might be a Ferrari, but I can't fix Ferraris and I might be able to fix 2CVs. So I'm going to keep Unfiltered in my toolbox.

SBT

Whatever people are saying: it's not simple. It just isn't. Things I don't like about it:

  • <understatement>Not exactly an out of the box experience</understatement>
  • Simple things should be obvious, but unless you really get your head around the underlying model, nothing is obvious at all, and some things are plain impossible
  • You could argue that build tools should be scripting language based, and not require recompilation
  • Keeping up with the changes in SBT is a pain

Things I do like about it:

  • Once you start digging around in the model, it is actually making sense
  • As a result of its structure, it definitely can do some things way faster than normal build tools
  • I think it still is the fastest way of getting your Scala code compiled

So, regardless of what I said before, SBT is a keeper. However, you do need to understand what you're doing. (This book came too late, but it helps a lot!) If you don't know what you're doing, it's easy to turn it into a mess that will be impossible to refactor into something useful.

JSON4S

I've said quite a few things about JSON4S before. It migth be fast. It might be clever. But it's not very user friendly. And having multiple ways to get something done doesn't help.

If there ever is a reasonable alternative for JSON4S, I'd go for it. Until then, we're stuck with it. And I bet it's better than any Java alternative.

Scalate

We were using Scalate for rendering our server pages. Now, I hated SSP, with a passion. SSP seemed like PHP but then worse, and the output looked awful. However, Scalate also has a HAML implementation, called SCAML, and that definitely made it a better experience.

The one thing I'm not to sure about is the compilation to Scala code, in the way it's done. Compiling SCAML to Scala code might generate code that is really fast at runtime, but time it takes to compile SCAML to Scala is making the build taking even longer than it already is. Somehow it's not painful enough to have somebody really look into it.

Also, under the hood Scalate is pretty yucky. I tried to get some i18n support in the code generation, and couldn't believe what I had to do to make it work.

Having said that, I don't see a valid alternative. I hate the arrogance of the Play framework, forcing another template language upon you, and I don't like it. Dry, indent-based languages are the future for template language. So. Scalate. I'm going to keep it in my toolbox.

Configgy

I know! It has been dead for three years, but we never took the time to take it out. And I hated myself for it for three years in a row. Don't want to talk about it anymore.

Compilation time

I already referred to it a couple of times: Scala compilation is slow. I remember that at some point it just became unbearable, and I took some time to see how we could reduce our compilation times. One of the solutions turned out to be moving from traits to abstract base classes. That prevented a lot of unvoluntary and unecessary compilation.

The lesson learned there is that you need to monitor your compilation time continously and actively work on making sure it stays as fast as it possibly can. It's way easier to find the code causing compilation to take longer when you've just comitted it. Finding it after a few months is just too darn painful.

Akka

We used it in two places. And it never felt right. Admittedly, it had to do with ① our limited understanding of Akka at the time we started using it, ② the arkane version of Akka we were using originally, ③ the transition we had to make to roll forward. I guess that's the downside of being an early adopter.

One of the things I was really missing is persistent actors. That has been solved by now. And then we wanted a version of Akka that supported Camel. Unfortunately, it wasn't working with our version of Scala. So I backported Akka to our version. Perhaps not the best idea ever, but – you know – you just do whatever it takes to get it to work.

Does that mean I would not use Akka ever again? I wouldn't go that far. I think there's definitely a good use for Akka. However it might be on the peak of inflated expectations right now (or heading there), and I would definitely consider some alternatives instead of jumping on Akka since the rest of the Scala community is doing it.

Mongo drivers

Our projects originally started using the Java driver for Mongo; Casbah wasn't around yet, or perhaps still in its infancy. At some point, that just didn't feel write. The amount of code we had to write to map our objects into BSON data structures was just ridiculous. So we created our own wrapper around BasicDBObject, taking care of some of the mapping issues.

It must have been around that time that the Casbah driver became a viable alternative to the Java driver. So I included that in our project as well. Cashbah has a way to transparently map from and to the Java classes, so that seemed like a good approach.

But was it? Admittedly, Cashbah makes a whole bunch of things a lot easier to do. But knowing what to import to get the magic going continued to be painful. And the implicit conversions from and to the Java object model were sometimes just really confusing. Also, our own non-strictly layered abstractions imposed on top of the Java classes, didn't help to make sure Casbah knew what to operate on.

All in all, I'm not a huge fan of the Casbah driver. That could be caused by the fact that we didn't go Casbah all the way. It might. Would I use it again? Perhaps. In any case, I think I will make sure that in the future we stick to one abstraction instead of trying to juggle with three different abstractions. And perhaps that should be Reactive Mongo instead.

Scala version updates

Continued to be painful.

Scala maturity

So, what about Scala's maturity? I guess there are at least two answers to this question. If the question is strictly about the language, then I'd say yes, definitely mature. If the question is about the platform, then I think we're heading in the right direction.

When we started, there was no official standard for writing Scala code yet. There were a couple of schools of thought though. And some schools of thought promoted leaving the Java coding conventions. That has been biting us. Meanwhile, the dust settled down, and I think most of us are now agreeing on the suggestions here.

However, I haven't seen too much standardization on APIs just yet. And because of the way Scala works, the APIs do have an impact on what the language feels like. That means it sometimes seems as if there is no single Scala yet, but that there are many Scalas, with codebases that really don't share a lot of coding patterns in common. That's a problem. Hopefully there will be some convergence in the future.