Here at Tumblr, we use a JS bundler to compile our client-side code. As time went by, we started to feel the very real effects of bit rot in the form increasingly slow build times and the realization that were were 9 major versions behind on Browserify with no straightforward way to upgrade.

We decided to take a fresh look at our process and give Webpack a try. We laughed, we cried, we saved a bunch of time on our builds.

---

About two years ago, Tumblr embarked on a journey to create and apply cohesive clientside architecture to the website. Our Product Engineers had lived without things like JS bundling, npm libraries, and CSS preprocessors, and a lingua franca for application-level functionalities to share between Product teams was a considerable step forward for our codebase.

One of the things that came out of this initiative was our use of Browserify to create JavaScript bundles that used npm modules as well as our own libraries. Suddenly, we had a straightforward way to share classes and utilities without polluting the global namespace! We decided on Browserify for building and Gulp as our taskrunner.

We decided early on that we wouldn’t just start from scratch and rewrite the entire site, but rather we would pull things over piecemeal from the old way to the new way. We needed a way of splitting code between various pages on the site during this transition. Thus, “contexts” were born.

A context bundle is essentially a mini Single Page App. For example, the dashboard is a context. The search page is a different context. The help docs are a different context. Each distinct context meant a different set of JS and CSS build artifacts, which meant a Browserify build for each.

These contexts were still sharing plenty of code between them, particularly vendor libraries, which necessitated another bundle to avoid code duplication (and downloading identical code). We used a vendor bundle to address this. Another Browserify build! We manually maintained a list of modules that would be kept in the vendor bundle so the context bundle builds would know not to include them.

The Browserify build process

Fast forward a year or so and we had added the header bundle, which loaded above the fold, and standalone bundles, which are entirely self-contained and don’t rely on the full vendor bundle. Our builds had turned into something like this (and this isn’t even including CSS):

Browserify
 ├─ Header *
 ├─ Vendor *
 ├─ Context
 │   ├─ Default *
 │   ├─ Dashboard *
 │   ├─ Search *
 │   ├─ Explore *
 │   └─ ...
 └─ Standalone
     ├─ Blog Network *
     ├─ Mobile Web *
     ├─ Share Button *
     ├─ Embed-A-Post *
     └─ ...

Each starred thing up there was a separate Browserify build, and it got slower every time there was a new context or standalone bundle. Furthermore, the version of Browserify we were using was falling further out of date, because newer versions were even slower in our case. One engineer created a system to parallelize gulp tasks using the cluster, which sped things up and had the added benefit of turning our boxes into loud, fan-spinning space heaters.

Rethinking the build

Luckily, we did have an idea why things were so slow. Many modules were shared across contexts, but not in the vendor bundle, and our builds parsed them repeatedly. Browserify couldn’t share cached information about these modules across build processes. We could have fixed that by passing multiple entry points into Browserify, but that required rewriting our JS build scripts entirely, which was way too scary.

In the meantime while we were furrowing our brows at the situation we were in, Webpack was emerging as a popular new solution to bundling. It had some neat features that weren’t in Browserify, or not as easy to configure. We were particularly interested in automatic bundle splitting, async loading, and hot reloading. One engineer had looked into it early on, but some of the magic in our Browserify configuration didn’t translate over easily. We shelved it.

From Browserify to Webpack

At this point, our backs were against the wall. Our build process was so fucked up that we really had nothing to lose by trying something completely different except time.

“Okay, fine. Where do I sign up?”

Baby steps

The first step was trying to get something building in Webpack. Since we were still committed to using Gulp, we opted for webpack-stream. I tossed together a basic Webpack configuration and tried it.

We encountered problems immediately. Each of our context bundles used a “bootloader” to kick off the bundled JS with some bootstrapped data generated by the server. Being able to do require('context') and pass in the bootstrap data in an inline <script> tag seemed like a convenient way to share code, but we ended up with a circular require and this little slice of evil in our Browserify configuration:

{
  expose: 'context',
  // ...
  requireOpts: {expose: 'context'},
}

It seemed like a good idea at first, but it had to go, so it went. We refactored our contexts so that the entry points ran the bootloader immediately rather than exposing it as a static method on the class exported by the context.

Loaders

The next obstacle was that we overloaded require to include non-JS files and had been relying on Browserify transforms for templates and styles. We needed to handle these using Loaders, the Webpack equivalent to transforms. Fortunately, the Webpack community had already created loaders to handle these cases.

For styles, we went from using sassr to postcss and autoprefixer to style-loader!css-loader!postcss-loader!sass-loader.

For HTML templates, we went from using jstify to underscore-template-loader!html-minifier-loader.

Whereas our Browserify transforms did several things at once (loading the file, postprocessing, converting to a JS module), Webpack Loaders tend to be chainable single-purpose steps that allow the same end result.

Bundling and splitting

We used bundle splitting in Browserify, but it was a manual process that required separate per-bundle build scripts and a list of modules.

This isn’t the exact code we used, but in was essentially something like this:

Header bundle

var browserifyInstance = browserify();
var requires = ['some-config'];
var externals = [];
var bundleOptions = {};
requires.forEach(function(requireModule) {
  browserifyInstance.require(requireModule);
});
externals.forEach(function(externalModule) {
  browserifyInstance.require(externalModule);
});
browserifyInstance.require('header/index.js');
browserifyInstance.bundle(bundleOptions);

Vendor bundle

var browserifyInstance = browserify();
var externals = ['lodash', 'backbone', 'jquery', '...']; // list of vendor dependencies
var externals = ['some-config']; // Grab this from the header bundle
var bundleOptions = {exposeAll: true};
requires.forEach(function(requireModule) {
  browserifyInstance.require(requireModule);
});
externals.forEach(function(externalModule) {
  browserifyInstance.require(externalModule);
});
browserifyInstance.require('vendor/index.js');
browserifyInstance.bundle(bundleOptions);

The exposeAll option isn’t even documented. I couldn’t tell you how we discovered it.

Context bundle

var browserifyInstance = browserify();
var requires = [];
var externals = ['lodash', 'backbone', 'jquery', '...']; // same list of vendor dependencies
var bundleOptions = {};
requires.forEach(function(requireModule) {
  browserifyInstance.require(requireModule);
});
externals.forEach(function(externalModule) {
  browserifyInstance.require(externalModule);
});
browserifyInstance.add('context/*/index.js'); // add instead of require since it's the entry point
browserifyInstance.bundle(bundleOptions);

Our Browserify build was abstracted into a common class and each of the bundles extended it, so for each bundle, we were effectively listing internal things and external things, tweaking some config options, and then giving the whole thing to gulp to deal with.

Webpack bundle splitting

Automatic bundle splitting was one of the Webpack features we were most excited about. Webpack has a reputation for requiring a lot of configuration, but using the CommonsChunkPlugin, we were able to remove a lot of the manual configuration we had been using to maintain the header/vendor/context separation in Browserify.

Our context bundles were smaller as the automatic code splitting pulled many shared modules into the new global bundle. This would never have been feasible in our Browserify build process. Here’s what our configuration started to look like:

{
  // ...
  plugins: [
    new webpack.optimize.CommonsChunkPlugin({
      names: [
        'global',
        'vendor',
        'header',
      ],
    }),
  ],
  entry: _.extend({
    'global': [],
    'vendor': [
        'vendor', // entry point
        'lodash',
        'backbone',
        'jquery',
        // etc...
    ],
    'app/header': [
        'header', // entry point
        'some-config',
    ],
  }, createEntryPoints('context/*/index.js')),
  // ...
}

That createEntryPoints function expanded the glob and created a full mapping of of our context entry points. CommonsChunkPlugin decided if a module was “global enough” and pulled it into the global bundle. In early tests without the global bundle, we found that a lot of those modules were automatically dropped into the vendor bundle.

The other interesting thing we found was that the order of modules in the CommonsChunkPlugin options mattered. The last entry was assumed to be the first script loaded on the page. This is important it defines webpackJsonp, which subsequent bundles rely on to communicate with each other.

Putting it all together

With all of these changes implemented, our builds looked like this:

Webpack
 ├─ Context *
 │   ├─ Header
 │   ├─ Vendor
 │   ├─ Global
 │   ├─ Default
 │   ├─ Dashboard
 │   ├─ Search
 │   ├─ Explore
 │   └─ ...
 └─ Standalone *
     ├─ Blog Network
     ├─ Mobile Web
     ├─ Share Button
     ├─ Embed-A-Post
     └─ ...

We were down to two build processes, each using multiple entry points, so each process benefitted from sharing parsed modules between those entry points.

How much faster did it get?

A lot faster. Dev builds took less than a third of the time to run. Production build times were cut in half. Incremental builds using the Webpack watcher were almost instant.

These build times are a bottleneck in the development process, and by saving hundreds (estimated) of developer-hours, we’re freeing up time to work on more features, ship faster, and spend more time with our families.

So did we need to ditch Browserify to speed things up?

Maybe, maybe not. The process of migrating our Browserify configuration to Webpack exposed several foolish things we were doing that we could have fixed without switching bundlers. On the other hand, our build scripts are easier to read now because we’re using core Webpack features that we accomplished with clever (in the bad way) tricks in Browserify.

Either way, you can pry chainable loaders out of my cold, dead hands.

Overall, it’s always a good exercise to make sure you still understand the code you’re responsible for maintaining. Learning a new tool is fun, but when it improves your development flow as much as it did for us, the proof is, as they say, in the pudding.

— @keithmcknight