The Rise of the Smiths

Beating metalsmith into shape

I have a thing for static site generators. I think Middleman was the first I ever used. It introduced me to HAML based templating, and a bunch of other things. Unfortunately, my Ruby skills are – uhm – underdeveloped, so I could never figure out how to configure it correctly.

Still, I liked the idea. So I created my own, called Monkeyman, done in Scala. Now, even though I like Scala, and it still serves this blog well, the truth is that JavaScript is the language of the Web, and all the tools you will ever need have been written in JavaScript.

So even though Monkeyman serves its purpose, extending it is actually quite hard. That's partly to blame to its architecture, but also to blame to the eco system. JavaScripts web eco system has just evolved so much more that it's impossible to imagine a world where you simply ignore all that goodness.

Originally, I also used Monkeyman whenever I wanted to create a single page application. Monkeyman knows about all the files it's managing, which makes creating a manifest a breeze, for instance. However, I still had to add support for it myself, and hack it directly into Monkeyman.

After a while, I started looking for an alternative, and found Brunch. I like Brunch. A lot. Brunch shares some similarity with Middleman in that it has support for all the minification, concatenation and a whole bunch of other tools readily available. Adding support for a certain transformation normally means just npm installing a package.

Now, for creating a single page application, that's great. But there's also something missing. Creating an entire web site with is actually a little harder. It doesn't seem to be the tool that lends itself perfectly for that task. I'm sure you can get it done, but IMHO, its's mainly tries to take care of some tedious tasks that you'd normally have to grunt script together yourself.

So when I considered updating my Middleman based web site with something else, I started looking for something that was just geared a little bit more towards generating a web site, applying Jade layouts to Markdown files and tying everything together with YAML frontmatter based metadata.

My first stop was Wintersmith. I already found Wintersmith a couple of months ago, and back then it seemed like the best option. However getting it to do the right thing required more effort than I was willing to put in. It's hard to tell now, but looking back, I think it was the lack over an overarching mental picture on how this system was supposed to work, and readable documentation on the plugin architecture, and it didn't seem to be maintained any longer.

So I continued searching the web for something better, and through StaticGen found Metalsmith, another descendant of Blacksmith. Lesson learned: whenever you're creating a static web site generator in 2015, find a name that has the word 'smith' in it.

Metalsmith

Metalsmith advertises itself as system in which everything is a plugin. That itself is not necessarily a good thing. On my Mac, all of the applications are basically plugins of the Operating System. However that doesn't mean they all work together in a nice way.

But Metalsmith makes sure all these plugins are built around a very simple and small core. As with Brunch, all plugins are npm modules. These plugins are then registered with Metalsmith in a particular order. Each plugin gets a chance to generate new content or modify existing content base on what was there before.

There are two ways to configure Metalsmith. One way is to create a metalsmith.json configuration file. That configuration file specifies in which order the npm-installed plugins should run, and passes configuration data for those plugins. However, you can also do exactly the same thing using code only, which – from my point of view - really isn't all that different, apart from the added flexibility. (More on that later.)

At it's core, a plugin is a function that takes some configuration parameters, and then produces another function. The signature of that function is like this:

function(files, metalsmith, done)

The files object is an object in which every key represents a path in a virtual file system, and its value an object that contains both metadata and the actual contents, as a Buffer, stored under the contents key.

So a simple example adding a robots.txt file would look like this:

function robots(opts) {
  return function(files, metalsmith, done) {
    fs.readFile(opts.source, function(err, data) {
      files['robots.txt'] = {
        contents: data
      };
      done();
    });
  };
}

Here's how you add that plugin:

metalsmith(__dirname)
  .use(markdown())
  .use(robots({
    source: 'sample-robots.txt'
  })
  .destination('./build');

In this particular example, I've added the markdown plugin (one that is installed with npm install -S metalsmith-markdown), and also added the example robots.txt plugin I created a minute ago. The destination(…) call makes sure all plugins run in order, and the result is written to de destination folder. Alternatively, you could also use the metalsmith-serve plugin to serve the web site directly.

Using it for real

Using Metalsmith for real made me realize a couple of things:

The configuration

When you starting using Metalsmith, it probably won't take long before you realize you really need to configure it programmatically. Here are some of the reasons why:

  1. I think it won't be uncommon that you want to run Metalsmith either as a generator, or serve content directly while you're creating your web pages. There are some plugins you do want to have executed while you're running it as a server (like the live reload plugin) and other plugins if you're generating a final version. Configuring Metalsmith programmatically allows you to build a common configuration and extend that common configuration in different profiles, dependening on your needs.
  2. The collection of plugins is impressive, but the structure of Metalsmith doesn't enforce the plugins to play together in a nice way. You often need code to bridge the gap between two plugins. The good news is: you can do that, by creating your own plugins. But then creating npms out of those is a lot of overhead. Still, if you're using the scripted configuration, you can just define new plugins on the spot, without having to go through the npm packaging process. (And you can make them a little bit more specific to your context.)

Existing plugins

They don't all work together all that well. Let's take the metalsmith-concat plugin as an example. This one will concatenate all your CSS and turn it into a single CSS file. It works great. Unless you use it in combination with the metalsmith-watch plugin. If you use it in combination with the metalsmith-watch plugin, the concat plugin will – after every change – start generating a blank file.

In this case, the underlying issue is that metalsmiths model is just a little too simple: the concat plugin takes the existing CSS files, merges them, and stores the results in a new file. But when the watch plugin fires, it will only pass the changed files to the chain of plugins. As a result, the concat plugin no longer gets to see the original files, and therefore constructs an empty file.

This actually took me quite a while to figure out. However, there's some good news too. Since Metalsmith's model and plugin structure are so incredibly simple, it's actually quite ease to examine the plugins and understand what they're doing. In most cases, it's nothing but a few lines around an existing library.

And that's really the good news. Nothing is perfect, and Metalsmith is not an exception. But since the programming model is so simple, it's easy to 'beat it into shape'.

The plugins I kept using are these:

  • metalsmith-markdown
  • metalsmith-templates
  • metalsmith-serve
  • metalsmith-watch
  • metalsmith-collections
  • metalsmith-filepath
  • metalsmith-less
  • metalsmith-uglify
  • metalsmith-clean-css
  • metalsmith-coffee

Existing vs homegrown plugins

I created a bunch of plugins myself, just for sake of the things I needed. And since these plugins all live in my own project, I can cheat. There are a few less hoops you need to jump through as a result.

And even if you would create a fully fledged plugin from your own work, it really is not much work at all.

I had to add my own plugins for:

  • Extracting metadata from this blog site: getting the titles, background images and dates from existing blog posts, and add that as resources to the virtual file system.
  • Having bower support. I'll write a little more on that in another post.
  • Having a working concat implementation.
  • Adding global metadata on the CSS and JavaScript files in my project, and expose that to the templates plugin, allowing the Jade files to access it.

Conclusion

All in all, using Metalsmith was a much better experience than I imagined. The most important benefit of Metalsmith is its simplicity. If you know a little bit about node, then Metalsmith is really simple to understand. And with the myriad of libraries you have at your disposal, extending Metalsmith to be exactly the tool you need is really simple.

I do think that Metalsmith's model is a little flawed. No longer having access to all of the files during a second pass of your plugin is confusing, and it's hard to compensate for it. And perhaps rather than storing the contents of a file as a Buffer, it would have been better to store it as a function, to prevent polluting the heap too much, and have the ability to keep existing files on the file system. At the other hand, that would make most plugins more complicated, and some plugins really need to look at the data directly.

In that sense, balancing all of these little trade-offs reminded me of Monkeyman. In case of Monkeyman, there are many things I would do differently now. Metalsmith is taking it to some extent into the direction I would have taken it myself as well (having plugins being just simple functions).

Interestingly, Metalsmith doesn't implement a whole bunch of things I considered to be required for a next version of Monkeyman. One of the things I considered to be important is to have proper dependency tracking: make sure you remember how a file came into existince, and on which previous versions it was based. That way, if one file changed, you would immediately know which files had to be updated.

Metalsmith doesn't have that, and it doesn't suffer too much from the consequences. Whatever doesn't work because of the underlying model, you can fix yourself. And I like it for it.