Static Generators, Powerful Pipelines and Wyam Websites

04/13/2017

Static Generators, Powerful Pipelines and Wyam Websites

We rebuilt our website using a static site generator and, in the process, learned to appreciate the power of pipelines.

If you're reading this, then we have managed to successfully deploy our newly revamped company website. After trying many different CMS platforms on for size (the last being Orchard CMS which experienced a horrible migration from our hosting provider to Azure), a couple of conclusions were made:

  • CMS platforms manage content, but trying to customize them for optimizing SEO gets very cumbersome
  • We want to keep our Javascript usage/exposure to a minimum
  • We can find a better way to deploy what should be a pretty simple website

After plenty of exploration (and a blog post from Scott Hanselman), we went with a static site generator instead - notably, Wyam. Wyam is built on top of .NET (and moving to dotnetcore as we speak) and lets you build blogs, documentation, and websites (such as this one) by chaining together modules inside of one or more pipelines that reside in a configuration file.

Content and Metadata

All blog entries are written using Markdown files, with metadata written in the top, like this:

Title: Notes about the new site
Published: 1/7/2017
Description: We rebuilt our website using a static site generator and lived to tell the tale.
---
(blog post Markdown goes here)

When running the Wyam build, this Markdown file will be picked up with the other Markdown and/or .cshtml files, run through the Razor view engine, and the rendered HTML will be written into the output folder. Our website keeps a blog, a news feed, and the pages for services in separate folders. Each folder, including the top-level, gets a Pipeline in order to process the content files.

Pipelines

The more powerful features of Wyam is its use of a pipeline for generating the site. A pipeline is a series of modules executed in a sequence that result in output documents. If you're familiar with the Cake build automation platform you'll be right at home here; if you're not, you'll be ready for Cake when you're done, as both use a very similar approach.

Here's an example of the pipeline used to generate the blog as part of this website:

Pipelines.Add(
    "Blog",
    ReadFiles("blog/**.md"),
    FrontMatter(Yaml()),
    If(
        @doc.FilePath(Keys.RelativeFilePath).Extension == ".md",
        Markdown()
    ),
    Razor(),
    WriteFiles(".html"),
    Meta("BlogFile", string.Format(@"blog/{0}", @doc.String("DestinationFileName"))),
    Meta("SitemapItem", (string)@doc["BlogFile"]), 
    Branch(
        GenerateFeeds().WithRssPath(ctx=> "blog/feed.rss")
        .WithRdfPath(ctx=> null).WithAtomPath(ctx => null)
        .WithFeedCopyright(ctx=> "2007-2017 Endpoint Systems. All rights reserved.")
        .WithFeedTitle(ctx=> "Endpoint Systems Blog")
        .WithFeedDescription(ctx=> "APIs, XML, BizTalk and cloud development")
        .WithFeedLink(ctx=> "https://endpointsystems.com/blog/index.html")
        .WithItemPublished((doc, ctx) => doc["Published"])
        WriteFiles()               
        )    
);

Taking it from the top, this pipeline does the following:

  • Creates a pipeline named "Blog"
  • Picks up all Markdown files in the directory
  • Checks for any YML files containing additional metadata
  • Runs an If module to check for a condition to see if I managed to pick up any Markdown files. If I have, they are run through the Markdown module. For all intents and purposes this is redundant considering all blog posts will be written into Markdown files, but it's a great demonstration of the versatility of the pipeline model being used.
  • Runs the Markdown files (now HTML) and any .cshtml files through the Razor templating engine
  • Writes the files to an .html extension
  • Pushes the HTML file names into the metadata using the Meta. Note we are adding to the path here instead of just using the DestinationFileName key
  • Push a SitemapItem into the file's metadata using our previously-created BlogFile value. This will get picked up in a Sitemap module we call in our parent diretory, which as the name implies, generates a site map for the entire site
  • Use a Branch module to iterate through the files and generate the RSS feed for the blog using the RssFeed module. Note that we also set several different feed properties, including making sure we use the Published metadata value for the

Metadata

As you may have surmised by now, there is a ton of metadata getting generated and passed around. If you're new to Wyam it can be difficult to comprehend what's already available vs. what you're looking for or may want to create on your own. The best place to start when trying to understand the metadata system is the Keys class, which you can use to get or set core module properties with. There's also FeedKeys for the GenerateFeeds module, BlogKeys for the blog recipe (recipes are great if you're looking for something preconfigured for a special purpose, like a blog), DocsKeys for the docs recipe, HtmlKeys for the various HTML processing modules, GitKeys if you want to use the GitCommits and/or GitContributors modules within the pipeline to do something with your SCM, SearchIndexKeys for building a SearchIndex for your site, and CodeAnalysisKeys for the code analysis modules which help generate documentation pages.

To set your own metadata values, go to your config.wyam configuration file and, up before the pipeline code, you can add things like:

Settings[Keys.Host] = "endpointsystems.com";
Settings[Keys.LinkHideExtensions] = false;
Settings[Keys.LinkHideIndexPages] = false;
Settings[FeedKeys.Copyright] = "2007-2017 Endpoint Systems";

Or you can set your own, like this:

Settings["figaroSlogan"] = "Figaro: Real Document Storage for .NET Developers";

Settings["summary"] = "Endpoint Systems is a software and services firm helping customers with APIs, microservices, integration and cloud migration.";

When I want to access these in my Razor templates, I find these properties within the @Model:

<p><a href="https://bdbxml.net" target="_blank">@Html.Raw(@Model["figaroSlogan"])</a></p>

Deploying

The best thing about deploying a static website is how cheap your options can go. While there are many free services available, we opted for Google Cloud Storage buckets to host our static sites. Not only is it easy to set up and dirt cheap for what you get, you can also take advantage of Multi-Regional Storage and have instant geographic redundancy across the entire United States where we deployed our storage.

Since our deployment needs are pretty simple at the moment, we use a .bat file containing our gcloud commands for uploading the files and setting the permissions for everyting deployed. As we finish revamping the Figaro and Figaro documentation websites, however, a formalized build process for Wyam static site CI/CD process will be built, and given the recent changes in VSTS (which we use to build all of the diferent Figaro packages we have to generate), we're going to be looking at using Cake going forward for all of our project needs. No doubt there will be some insights to share there as well.

Enjoy!