I am an engineer that loves docs. Well, OK, I don’t love all docs but I believe docs are a crucial, yet often neglected element to a great developer experience. I work on the developer experience team for Cloudflare Workers focusing on several components of Workers, particularly on the docs that we recently migrated to Gatsby.

? We’ve moved the Cloudflare Workers docs to @gatsbyjs

The new documentation is...

?‍♀️ faster⭐️ more accessible? a perfect foundation for the redesign later this year?️ open-source

shout out to @exvuma for this incredible work ??https://t.co/k3huvCvash pic.twitter.com/MBWxVtlrin

— Cloudflare Developers (@CloudflareDev) March 4, 2020

Through porting our documentation site to Gatsby I learned a lot. In this post, I share some of the learnings that could’ve saved my former self from several headaches. This will hopefully help others considering a move to Gatsby or another static site generator.

Prior to our migration to Gatsby, we used Hugo for our developer documentation. There are a lot of positives about working with Hugo - fast build times, fast load times - that made building a simple static site a great use case for Hugo. Things started to turn sour when we started making our docs more interactive and expanding the content being generated.

Going from writing JSX with TypeScript back to string-based templating languages is difficult. Trying to perform complicated tasks, like generating a sidebar, cost me - a developer who knows nothing about liquid code or Go templating (though with Golang experience) - several tears not even to implement but to just understand what was happening.

Here is the code to template an item in the sidebar in Hugo:

<!-- templates -->
{{ define "section-tree-nav" }}
{{ $currentNode := .currentnode }}
{{ with .sect }}
 {{ if not .Params.Hidden }}
  {{ if .IsSection }}
    {{safeHTML .Params.head}}
    <li data-nav-id="{{.URL}}" class="dd-item
        {{ if .IsAncestor $currentNode }}parent{{ end }}
        {{ if eq .UniqueID $currentNode.UniqueID}}active{{ end }}
        {{ if .Params.alwaysopen}}parent{{ end }}
        {{ if .Params.alwaysopen}}always-open{{ end }}
        ">
      <a href="{{ .RelPermalink}}">
        <span>{{safeHTML .Params.Pre}}{{.Title}}{{safeHTML .Params.Post}}</span>
 
        {{ if .Params.new }}
          <span class="new-badge">NEW</span>
        {{ end }}
 
        {{ $numberOfPages := (add (len .Pages) (len .Sections)) }}
        {{ if ne $numberOfPages 0 }}
 
          {{ if or (.IsAncestor $currentNode) (.Params.alwaysopen)  }}
            <i class="triangle-up"></i>
          {{ else }}
            <i class="triangle-down"></i>
          {{ end }}
 
        {{ end }}
      </a>
      {{ if ne $numberOfPages 0 }}
        <ul>
          {{ .Scratch.Set "pages" .Pages }}
          {{ if .Sections}}
          {{ .Scratch.Set "pages" (.Pages | union .Sections) }}
          {{ end }}
          {{ $pages := (.Scratch.Get "pages") }}
 
        {{ if eq .Site.Params.ordersectionsby "title" }}
          {{ range $pages.ByTitle }}
            {{ if and .Params.hidden (not $.showhidden) }}
            {{ else }}
            {{ template "section-tree-nav" dict "sect" . "currentnode" $currentNode }}
            {{ end }}
          {{ end }}
        {{ else }}
          {{ range $pages.ByWeight }}
            {{ if and .Params.hidden (not $.showhidden) }}
            {{ else }}
            {{ template "section-tree-nav" dict "sect" . "currentnode" $currentNode }}
            {{ end }}
          {{ end }}
        {{ end }}
        </ul>
      {{ end }}
    </li>
  {{ else }}
    {{ if not .Params.Hidden }}
      <li data-nav-id="{{.URL}}" class="dd-item
     {{ if eq .UniqueID $currentNode.UniqueID}}active{{ end }}
      ">
        <a href="{{.RelPermalink}}">
        <span>{{safeHTML .Params.Pre}}{{.Title}}{{safeHTML .Params.Post}}</span>
        {{ if .Params.new }}
          <span class="new-badge">NEW</span>
        {{ end }}
 
        </a></li>
     {{ end }}
  {{ end }}
 {{ end }}
{{ end }}
{{ end }}

Whoa. I may be exceptionally oblivious, but I had to squint at the snippet above for an hour before I realized this was the code for a sidebar item (the li element was the eventual giveaway, but took some parsing to discover where the logic actually started).

(Disclaimer: I am in no way a pro at Hugo and in any situation there are always several ways to code a solution; thus I am in no way claiming this was the only way to write the template nor am I chastising the author of the code. I am just displaying the differences in pieces of code I came across)

Now, here is what the TSX (I will get into the JS later in the article) for the Gatsby project using the exact same styling would look like:

 <li data-nav-id={pathToServe} className={'dd-item ' + ddClass}>
   <Link className="" to={pathToServe} title="Docs Home" activeClassName="active">
     {title || 'No title'}
     {numberOfPages ? <Triangle isAncestor={isAncestor} alwaysopen={showChildren} /> : ''}
     {showNew ? <span className="new-badge">NEW</span> : ''}
   </Link>
   {showChildren ? (
     <ul>
       {' '}
       {myChildren.map((child: mdx) => {
         return (
           <SidebarLi
             frontmatter={child.frontmatter}
             fields={child.fields}
             depth={++depth}
             key={child.frontmatter.title}
           />
         )
       })}
     </ul>
   ) : (
     ''
   )}
 </li>

This code is clean and compact because Gatsby is a static content generation tool based on React. It’s loved for a myriad of reasons, but my honest main reason to migrate to it was to make the Hugo code above much less ugly.

For our purposes, less ugly was important because we had dreams of redesigning our docs to be interactive with support for multiple coding languages and other features.

For example, the template gallery would be a place to go to for how-to recipes and examples. The templates themselves would live in a template registry service and turn into static pages via an API.

We wanted the docs to not be constrained by Go templating. The Hugo docs admit their templates aren’t the best for complicated logic:

Go Templates provide an extremely simple template language that adheres to the belief that only the most basic of logic belongs in the template or view layer.

Gatsby and React enable the more complex logic we were looking for. After our team built workers.cloudflare.com and Built with Workers on Gatsby, I figured this was my shot to really give Gatsby a try on our Workers developer docs.

I’m normally not a fan of fixing things that aren’t broken. Though I didn’t like working with Hugo, did love working in React, and had all the reasons to. I was timid about being the one in charge of switching from Hugo. I was scared. I hated looking at the liquid code of Go templates. I didn’t want to have to port all the existing templates to React without truly understanding what I might be missing.

There comes a point with tech debt though where you have to tackle the tech debt you are most scared of.

The easiest solution would be of course to throw the Hugo code away. Start from scratch. A clean slate. But this means taking something that was not broken and breaking it. The styling, SEO, tagging, and analytics of the site took small iterations over the course of a few years to get right and I didn’t want to be the one to break them. Instead of throwing all the styling and logic tied in for search, SEO, etc..., our plan was to maintain as much of the current design and logic as possible while converting it to React piece-by-piece, component-by-component.

Also there were existing developer docs still using Hugo on Cloudflare by other teams (e.g. Access, Argo Tunnel, etc...). I wanted a team at Cloudflare to be able to import their existing markdown files with frontmatter into the Gatsby repo and preserve the existing design.

I wanted to migrate instead of teleport to Gatsby.

In this blog post, I go through some but not all of the steps of how I ported to Gatsby from Hugo for our complex doc site. The few examples here help to convey the issues that caused the most pain.

Let’s start with getting the markdown files to turn into HTML pages.

One goal was to keep all the existing markdown and frontmatter we had set up in Hugo as similar as possible. The reasoning for this was to not break existing content and also maintain the version history of each doc.

Gatsby is built on top of GraphQL. All the data and most all content for Gatsby is put into GraphQL during startup likely via a plugin, then Gatsby will query for this data upon actual page creation. This is quite different from Hugo’s much more abstract model of putting all your content in a folder named content and then Hugo figures out which template to apply based on the logic in the template.

MDX is a sophisticated tool that parses markdown into Gatsby so it can later be represented as HTML (it actually can do much more than that but, I won’t get into it here). I started with Gatsby’s MDX plugin to create nodes from my markdown files. Here is the code to set up the plugin to get all the markdown files (files ending in .md and .mdx) I had in the src/content folder into GraphQL:

gatsby-config.js

const path = require('path')
 
module.exports = {
 plugins: [
   {
     resolve: `gatsby-source-filesystem`,
     options: {
       name: `mdx-pages`,
       path: `${__dirname}/src/content`,
       ignore: [`**/CONTRIBUTING*`, '/styles/**'],
     },
   },
   {
     resolve: `gatsby-plugin-mdx`,
     options: {
       extensions: [`.mdx`, `.md`],
     },
   }, 
]}

Now that Gatsby knows about these files as nodes, we can create pages for them. In gatsby-node.js, I tell Gatsby to grab these MDX pages and use a template markdownTemplate.tsx to create pages for them:

const path = require(`path`)
const { createFilePath } = require(`gatsby-source-filesystem`)
exports.createPages = async ({ actions, GraphQL, reporter }) => {
 const { createPage } = actions
 
 const markdownTemplate = path.resolve(`src/templates/markdownTemplate.tsx`)
 
 result = await GraphQL(`
   {
     allMdx(limit: 1000) {
       edges {
         node {
           fields {
             pathToServe
           }
           frontmatter {
             alwaysopen
             weight
           }
           fileAbsolutePath
         }
       }
     }
   }
 `)
 // Handle errors
 if (result.errors) {
   reporter.panicOnBuild(`Error while running GraphQL query.`)
   return
 }
 result.data.allMdx.edges.forEach(({ node }) => {
   return createPage({
     path: node.fields.pathToServe,
     component: markdownTemplate,
     context: {
       parent: node.fields.parent,
       weight: node.frontmatter.weight,
     }, // additional data can be passed via context, can use as variable on query
   })
 })
}
exports.onCreateNode = ({ node, getNode, actions }) => {
 const { createNodeField } = actions
 // Ensures we are processing only markdown files
 if (node.internal.type === 'Mdx') {
   // Use `createFilePath` to turn markdown files in our `content` directory into `/workers/`pathToServe
   const originalPath = node.fileAbsolutePath.replace(
     node.fileAbsolutePath.match(/.*content/)[0],
     ''
   )
   let pathToServe = createFilePath({
     node,
     getNode,
     basePath: 'content/',
   })
   let parentDir = path.dirname(pathToServe)
   if (pathToServe.includes('index')) {
     pathToServe = parentDir
     parentDir = path.dirname(parentDir) // "/" dirname will = "/"
   }
   pathToServe = pathToServe.replace(/\/+$/, '/') // always end the path with a slash
   // Creates new query'able field with name of 'pathToServe', 'parent'..
   // for allMdx edge nodes
   createNodeField({
     node,
     name: 'pathToServe',
     value: `/workers${pathToServe}`,
   })
   createNodeField({
     node,
     name: 'parent',
     value: parentDir,
   })
   createNodeField({
     node,
     name: 'filePath',
     value: originalPath,
   })
 }
}

Now every time Gatsby runs, it starts running through each node on onCreateNode. If the node is MDX, it passes the node’s content (the markdown, fileAbsolutePath, etc.) and all the node fields (filePath, parent and pathToServe) to the markdownTemplate.tsx component so that the component can render the appropriate information for that markdown file.

The barebone component for a page that renders a React component from the MDX node looks like this:

markdownTemplate.tsx

import React from "react"
import { graphql } from "gatsby"
import { MDXRenderer } from "gatsby-plugin-mdx"
 
export default function PageTemplate({ data: { mdx } }) {
 return (
   <div>
     <h1>{mdx.frontmatter.title}</h1>
     <MDXRenderer>{mdx.body}</MDXRenderer>
   </div>
 )
}
 
export const pageQuery = graphql`
 query BlogPostQuery($id: String) {
   mdx(id: { eq: $id }) {
     id
     body
     frontmatter {
       title
     }
   }
 }
`

Now let’s get into where I wasted the most time, but learned hard lessons upfront: turning the Hugo template into a React component. At the beginning of this article, I showed that scary sidebar.

To set up the li element we had the Hugo logic looks like:

{{ define "section-tree-nav" }}
{{ $currentNode := .currentnode }}
{{ with .sect }}
 {{ if not .Params.Hidden }}
  {{ if .IsSection }}
    {{safeHTML .Params.head}}
    <li data-nav-id="{{.URL}}" class="dd-item
        {{ if .IsAncestor $currentNode }}parent{{ end }}
        {{ if eq .UniqueID $currentNode.UniqueID}}active{{ end }}
        {{ if .Params.alwaysopen}}parent{{ end }}
        {{ if .Params.alwaysopen}}always-open{{ end }}
        ">

I see that the code is defining some section-tree-nav component-like thing and taking in some currentNode. To be honest, I still don’t know exactly what the variables .sect, IsSection, Params.head, Params.Hidden mean. Although I can take a wild guess, they're not that important for understanding what the logic is doing. The logic is setting the classes on the li element which is all I really care about: parent, always-open and active.

When focusing on those three classes, we can port them to React in a much more readable way by defining a variable string ddClass:

 let ddClass = ''
 let isAncestor = numberOfPages > 0
 if (isAncestor) {
   ddClass += ' parent'
 }
 if (frontmatter.alwaysopen) {
   ddClass += ' parent alwaysOpen'
 }
 return (
   <Location>
     {({ location }) => {
       const currentPathActive = location.pathname === pathToServe
       if (currentPathActive) {
         ddClass += ' active'
       }
       return (
         <li data-nav-id={pathToServe} className={'dd-item ' + ddClass}>

There are actually a few nice things about the Hugo code, I admit. Using the Location component in React was probably less intuitive than Hugo’s ability to access currentNode to get the active page. Also isAncestor is predefined in Hugo as Whether the current page is an ancestor of the given page. For me though, having to track down the definitions of the predefined variables was frustrating and I appreciate the local explicitness of the definition, but I admit I’m a bit jaded.

The most complex part of the sidebar is getting the children. Now this is a story that really gets me starting to appreciate GraphQL.

Here’s getting the children for the sidebar in Hugo:

    {{ $numberOfPages := (add (len .Pages) (len .Sections)) }}
        {{ if ne $numberOfPages 0 }}
 
          {{ if or (.IsAncestor $currentNode) (.Params.alwaysopen)  }}
            <i class="triangle-up"></i>
          {{ else }}
            <i class="triangle-down"></i>
          {{ end }}
 
        {{ end }}
      </a>
      {{ if ne $numberOfPages 0 }}
        <ul>
          {{ .Scratch.Set "pages" .Pages }}
          {{ if .Sections}}
          {{ .Scratch.Set "pages" (.Pages | union .Sections) }}
          {{ end }}
          {{ $pages := (.Scratch.Get "pages") }}
 
        {{ if eq .Site.Params.ordersectionsby "title" }}
          {{ range $pages.ByTitle }}
            {{ if and .Params.hidden (not $.showhidden) }}
            {{ else }}
            {{ template "section-tree-nav" dict "sect" . "currentnode" $currentNode }}
            {{ end }}
          {{ end }}
        {{ else }}
          {{ range $pages.ByWeight }}
            {{ if and .Params.hidden (not $.showhidden) }}
            {{ else }}
            {{ template "section-tree-nav" dict "sect" . "currentnode" $currentNode }}
            {{ end }}
          {{ end }}
        {{ end }}
        </ul>
      {{ end }}
    </li>
  {{ else }}
    {{ if not .Params.Hidden }}
      <li data-nav-id="{{.URL}}" class="dd-item
     {{ if eq .UniqueID $currentNode.UniqueID}}active{{ end }}
      ">
        <a href="{{.RelPermalink}}">
        <span>{{safeHTML .Params.Pre}}{{.Title}}{{safeHTML .Params.Post}}</span>
        {{ if .Params.new }}
          <span class="new-badge">NEW</span>
        {{ end }}
 
        </a></li>
     {{ end }}
  {{ end }}
 {{ end }}
{{ end }}
{{ end }}

This is just the first layer of children. No grandbabies, sorry. And I won’t even get into all that is going on there exactly. When I started porting this over, I realized a lot of that logic was not even being used.

In React, we grab all the markdown pages and see which have parents that match the current page:

  const topLevelMarkdown: markdownRemarkEdge[] = useStaticQuery(
   GraphQL`
     {
       allMdx(limit: 1000) {
         edges {
           node {
             frontmatter {
               title
               alwaysopen
               hidden
               showNew
               weight
             }
             fileAbsolutePath
             fields {
               pathToServe
               parent
               filePath
             }
           }
         }
       }
     }
   `
 ).allMdx.edges
 const myChildren: mdx[] = topLevelMarkdown
   .filter(
     edge =>
       fields.pathToServe === '/workers' + edge.node.fields.parent &&
       fields.pathToServe !== edge.node.fields.pathToServe
   )
   .map(child => child.node)
   .filter(child => !child.frontmatter.hidden)
   .sort(sortByWeight)
 const numberOfPages = myChildren.length

And then we render the children, so the full JSX becomes:

<li data-nav-id={pathToServe} className={'dd-item ' + ddClass}>
   <Link
     to={pathToServe}
     title="Docs Home"
     activeClassName="active"
   >
     {title || 'No title'}
     {numberOfPages ? (
       <Triangle isAncestor={isAncestor} alwaysopen={showChildren} />
     ) : (
       ''
     )}
     {showNew ? <span className="new-badge">NEW</span> : ''}
   </Link>
   {showChildren ? (
     <ul>
       {' '}
       {myChildren.map((child: mdx) => {
         return (
           <SidebarLi
             frontmatter={child.frontmatter}
             fields={child.fields}
             depth={++depth}
             key={child.frontmatter.title}
           />
         )
       })}
     </ul>
   ) : (
     ''
   )}
 </li>

Ok now that we have a component, and we have Gatsby creating the pages off the markdown, I can go back to my PageTemplate component and render the sidebar:

import Sidebar from './Sidebar'
export default function PageTemplate({ data: { mdx } }) {
 return (
   <div>
     <Sidebar />
     <h1>{mdx.frontmatter.title}</h1>
     <MDXRenderer>{mdx.body}</MDXRenderer>
   </div>
 )
}

I don’t have to pass any props to Sidebar because the GraphQL static query in Sidebar.tsx gets all the data about all the pages that I need. I don’t even maintain state because Location is used to determine which path is active. Gatsby generates pages using the above component for each page that’s a markdown MDX node.

This was just the beginning of the full migration to Gatsby. I repeated the process above for turning templates, partials, and other HTML component-like parts in Hugo into React, which was actually pretty fun, though turning vanilla JS that once manipulated the DOM into React would probably be a nightmare if I wasn’t somewhat comfortable working in React.

Main lessons learned:

• Being careful about breaking things and being scared to break things are two very different things. Being careful is good; being scared is bad. If I were to complete this migration again, I would’ve used the Hugo templates as a reference but not as a source of truth. Staging environments are what testing is for. Don’t sacrifice writing things the right way to comply with the old way.

• When doing a migration like this on a static site, get just a few pages working before moving the content over to avoid intermediate PRs from breaking. It seems obvious but, with the large amounts of content we had, a lot of things broke when porting over content. Get everything polished with each type of page before moving all your content over.

• When doing a migration like this, it’s OK to compromise some features of the old design until you determine whether to add them back in, just make sure to test this with real users first. For example, I made the mistake of assuming others wouldn’t mind being without anchor tags. (Note Hugo templates create anchor tags for headers automatically as in Gatsby you have to use MDX to customize markdown components). Test this on a single, popular page with real users first to see if it matters before giving it up.

• Even for those with React background, the ramp up with GraphQL and setting up Gatsby isn’t as simple as it seems at first. But once you’re set up it’s pretty dang nice.

Overall the process of moving to Gatsby was well worth the effort. As we implement a redesign in React it’s much easier to apply the designs in this cleaner code base. Also though Hugo was already very performant with a nice SEO score, in Gatsby we are able to increase the performance and SEO thanks to the framework’s flexibility.

Lastly, working with the Gatsby team was awesome and they even give free T-shirts for your first PR!

There's no reason to manage all these layers on separate platforms. For a site on Cloudflare, these layers can be on a single platform. Serverless technology simplifies developers' lives and reframes our current definition of backend.

In this article I will breeze through a simple example of how converting a former Node server into a Worker untangled a part of my teams’ code base. The conversion to Workers for this example can be found at this PR on Github.

Cloudflare Marketplace hosts a variety of apps, most of which are produced by third party developers, but some are produced by Cloudflare employees.

The Spotify app is one of those apps that was written by the Cloudflare apps team. This app requires an OAuth flow with Spotify to retrieve the user’s token and gather the playlist, artists, other Spotify profile specific information. While Cloudflare manages the OAuth authentication portion, the app owner - in this case Cloudflare Apps - manages the small integration service that uses the token to call Spotify and formats an appropriate response. Mysteriously, this Spotify OAuth integration broke.

Teams at Cloudflare are keen to remain  agile, adaptive, and constantly learning. The current Cloudflare Apps team no longer comprises the original team that developed the Spotify OAuth integration. As such, this current team had no idea why the app broke. Although we had various alerting and logging systems, the Spotify OAuth server was lost in the cloud.

Our first step to tackling the issue was tracking down, where exactly did the OAuth flow live. After shuffling through several of the potential platforms - GCloud, AWS, Digital Ocean.. - we discovered the service was on Heroku.  The more platforms introduced, the more complexity in deploys and access management.

I decided to reduce the number of layers in our service by simply creating a serverless Cloudflare Worker with no maintenance, no new logins, and no unique backend configuration.

Here’s how I did it.

The old service used the Node.js and Express.

app.post('/blah', function(request, response) {

This states that for every POST to an endpoint /blah, execute the callback function with a request and response object as arguments.

Cloudflare Workers are built on top of the Service Workers spec. Instead of mutating the response and calling methods on the response object like in Express, we need to respond to ‘fetch’ events. The code below adds an event listener for fetch events (incoming requests to the worker), receiving a FetchEvent as the first parameter. The FetchEvent has a special method called respondWith that accepts an instance of Response or a Promise which resolves to a Response.

addEventListener("fetch", event => {
         event.respondWith(new Response(‘Hello world!’));
});

To avoid reimplementation of the routing logic in my worker, I made my own app .

const app = {
   get: (endpoint, fn) => {
     url = new URL(request.url);
     if (url.pathname === endpoint && request.method === "GET")
       return fn(request);
     return null;
   },
   post: (endpoint, fn) => {
     url = new URL(request.url);
     if (url.pathname === endpoint && request.method === "POST")
       return fn(request);
     return null;
   }
 };

Now with app set, I call app.get(..) similar to how I did in Node in my handler. I just need to make sure the handler returns at the correct app.

async function handleRequest(request) {
  lastResponse = app.post("/", async function (request) {..}
  if (lastResponse) {
      return lastResponse;
    }
 lastResponse =  app.get("/", async function (request) {
 if (lastResponse) {
      return lastResponse;
    }

lastResponse ensures that we keep listening for all the endpoint methods.

The other thing that needs to change is the return of the response. Before that return used response.json(), so the final response would be of JSON type.

response.json({
         proceed: false,
         errors: [{type: '400', message: error.toString()}]

In workers, I need to return a type Response to the respondWith function. I replaced every instance of response.json or response.sendStatus with a new Response object.

return new Response(
         JSON.stringify({
         proceed: false,
         errors: [{ type: "400", message: res.error }]
         }, { headers: { ‘Content-Type’: ‘application/json’ } })

Now for the most beautiful part of the transition: delete useless config.

Our Express server was set to export app as a module insert credentials so that Heroku or whatever non-serverless server could pick up, run, and build.

Though I can import libraries for workers via webpack, for this application, it’s unreasoned. Also, I have access to fetch and other native service worker functions.

const {getJson} = require('simple-fetch')
module.exports = function setRoutes (app) {

Getting rid of modules and deployment config, I removed the files: Procfile, credentials.json, package.json, development.js, heroku.js, and create-app.js.Routes.js simply becomes worker.js.

This was a demo of how workers made my life as a programmer easier. Future developers working with my code can read this code without ever looking at any configuration. Even a purely vanilla bean Javascript developer can come in since there is no managing builds and pulling hair out.

With serverless I can now spend time on doing what I love - development.

Implementing an OAuth Authentication server is conceptually simple but a pain in practice. We can leverage the power of Cloudflare Workers to simplify the implementation, reduce latency, and segregate our service logic from the authentication layer.

For those unfamiliar with OAuth, I highly recommend reading a more in depth article.

The steps of the OAuth 2.0 workflow are as follows:

1. The consumer service redirects the user to a callback URL that was setup by the auth server. At this callback URL, the auth server asks the user to sign in and accept the consumer permissions requests.

2. The auth server redirects the user to the consumer service with a code.

3. The consumer service asks to exchange this code for an access token. The consumer service validates their identity by including their client secret in the callback URL.

4. The auth server gives the consumer the access token.

5. The consumer service can now use the access token to get resources on behalf of the user.

In the rest of this post, I will be walking through my implementation of an OAuth Authentication server using a Worker. For simplicity, I will make the assumption the user has already logged in and obtained a session token in the form of a JWT that I will refer to as “token” herein. My full implementation has a more thorough flow that includes initial user login and registration.

We must be able to reference valid user sessions, codes and login information. Because Workers do not maintain state between executions, we will store this information using Cloudflare Storage. We setup up three namespaces called: USERS, CODES, and TOKENS .

On your OAuth server domain, create two empty worker scripts called auth and token. Bind the three namespaces to the two workers scripts. Then configure the namespaces to the scripts so that your resources end up looking like:

To put and get items from storage using KV Storage syntax:

// @ts-ignore 
CODES.get(“user@ex.com”) 

We include // @ts-ignore preceding all KV storage commands. We do not have type definitions for these variables locally, so Typescript would throw an error at compile time otherwise.

To set up a project using Typescript and the Cloudflare Previewer, follow this blog post. Webpack will allow us to import which we will need to use the JWT library jsonwebtoken.

import * as jwt from "jsonwebtoken";

Remember to run:

npm install jsonwebtoken && npm install @types/jsonwebtoken

Optionally, we can set up a file to specify endpoints and credentials.

import { hosts } from "./private";

export const credentials = {/* for demo purposes, ideally use KV to store secrets */
  clients: [{
    id: "victoriasclient",
    secret: "victoriassecret"
  }],
  storage: {
    secret: "somesecrettodecryptfromtheKV"
  }

};

export const paths = {
  auth: {
    authorize: hosts.auth + "/authorize",
    login: hosts.auth + "/login",
    code: hosts.auth + "/code",
  },
  token: {
    resource: hosts.token + "/resource",
    token: hosts.token + "/authorize",
  }
}

The consumer service generates some callback URL that redirects the user to our authentication server. The authentication server then presents the user with a login or accept page to generate a code. The authentication server thus must listen on the authorize url endpoint and return giveLoginPageResponse.

addEventListener("fetch", (event: FetchEvent) => {
 const url = new URL(event.request.url);
 if (url.pathname.includes("/authorize"))
  return event.respondWith(giveLoginPageResponse(event.request));
}

export async function giveLoginPageResponse(request: Request) {
 ...checks for cases where I am not necessarily logged in... 
 let token = getTokenFromRequest(request)
 if (token) { //user already signed in
  return new Response(giveAcceptPage(request)
 }

Since the user already has a stored session, we can use a method giveAcceptPage. To display the accept page and return a redirect to generate the code.

export function giveAcceptPage(request: Request) {
  let req_url = new URL(request.url);
  let params = req_url.search
  let fetchCodeURL = paths.auth.code + params
  return `<!DOCTYPE html>
  <html>
    <body>
      <a href="${fetchCodeURL}"> Accept</button>
    </body>
  </html>
    `;
}

At the endpoint for fetchCodeURL, the authentication server will redirect the user’s browser to the consumer’s page as specified by redirect_uri in the original URL params of the callback with the code.

addEventListener("fetch", (event: FetchEvent) => {
 ...
 if (url.pathname.includes("/code"))
  return event.respondWith(redirectCodeToConsumer(event.request));
}
export async function redirectCodeToConsumer(request: Request) {
 let session = await verifyUser(request)

 if (session.msg == "403") return new Response(give403Page(), { status: 403 })
 if (session.msg == "dne") return registerNewUser(session.email, session.pwd)
 let code = Math.random().toString(36).substring(2, 12)
 try {
  let req_url = new URL(request.url)
  let redirect_uri = new URL(encodeURI(req_url.searchParams.get("redirect_uri")))
  let client_id = new URL(encodeURI(req_url.searchParams.get("client_id")))
  // @ts-ignore
  await CODES.put(client_id + email, code)
  redirect_uri.searchParams.set("code", code);
  redirect_uri.searchParams.set("response_type", "code");
  return Response.redirect(redirect_uri.href, 302);
 } catch (e) {
  // @ts-ignore
  await CODES.delete(email, code)
  return new Response(
   JSON.stringify(factoryIError({ message: "error with the URL passed in" + e})),
  { status: 500 });
 }
}

Now the consumer has the code. They can use this code to request a token. On our token worker, configure the endpoint to exchange the code for the consumer service to grant a token.

addEventListener("fetch", (event: FetchEvent) => {
...
 if (url.pathname.includes("/token"))
  return event.respondWith(giveToken(event.request));

Grab the code from the request and validate this code matches the code that is stored for this client. Once the code is verified, deliver the token by grabbing the existing token from the KV storage or by signing the user information to generate a new token.

export async function giveToken(request: Request) {
 let req_url = new URL(request.url);
 let code = req_url.searchParams.get("code");
 let email = req_url.searchParams.get("email");
 if (code){
   if(!validClientSecret(request) return errorResponse()
  // @ts-ignore
  let storedCode = await CODES.get(email)
  if(code != storedCode) return new Response(give403Page(), { status:403})

  let tokenJWT = jwt.sign(email, credentials.client.secret);
  ... return token

Continuing where step 3 left off from the giveToken method, respond to the consumer with this valid token.

  ...
  headers.append("set-cookie", "token=Bearer " + tokenJWT);
  // @ts-ignore
  await TOKENS.put(email, tokenJWT)
  var respBody = factoryTokenResponse({
   "access_token": tokenJWT,
   "token_type": "bearer",
   "expires_in": 2592000,
   "refresh_token": token,
   "token": token
  })
 } else {
  respBody.errors.push(factoryIError({ message: "there was no code sent to the authorize token url" }))
 }
 return new Response(JSON.stringify(respBody), { headers });
}

At this point voila, your duty as an OAuth 2.0 Authentication server is complete! The consumer service that wishes to use your service has the token that you have not so magically generated.

The consumer server would send a request including the token:

GET /resource/some-goods
Authorization: Bearer eyJhbGci..bGAqA

Authentication server would validate the token and give the goods:

export async function giveResource(request: Request) {
 var respBody: HookResponse = factoryHookResponse({})
 let token = ""
 let decodedJWT = factoryJWTPayload()
 try { //validate request is who they claim
  token = getCookie(request.headers.get("cookie"), "token")
  if (!token) token = request.headers.get("Authorization").substring(7)
  decodedJWT = jwt.verify(token, credentials.storage.secret)
  // @ts-ignore
  let storedToken = await TOKENS.get(decodedJWT.sub)
  if (isExpired(storedToken)) throw new Error("token is expired") /* TODO instead of throwing error send to refresh */
  if (storedToken != token) throw new Error("token does not match what is stored")
 }
 catch (e) {
  respBody.errors.push(factoryIError({ message: e.message, type: "oauth" }))
  return new Response(JSON.stringify(respBody), init)
 }
 respBody.body = getUsersPersonalBody(decodedJWT.sub)
 return new Response(JSON.stringify(respBody), init)
}

The boundaries of serverless are pushed everyday, though if your app just needs to authorize users, you may be better off using Cloudflare Access. We've demonstrated that a full blown OAuth 2.0 authentication server implementation can be achieved with Cloudflare Workers and Storage.

Stay tuned for a follow-up blog post on an OAuth consumer implementation.

When installing an app, a site owner must select where - what URLs on their site - they want what apps installed. Our original plan for selecting the URLs an app would be installed on took a few twists and turns. Our end decision was to utilize our Always Online crawler to pre-populate a tree of the user’s site. Always Online is a feature that crawls Cloudflare sites and serves pages from our cache if the site goes down.

The benefits to this original setup are:1. Only valid pages appearAn app only allows installations on html pages. For example, since injecting Javascript into a JPEG image isn’t possible, we would prevent the installer from trying it by not showing that path. Preventing the user from that type of phony installation prevents the user from being confused later when it doesn’t work.2. The user was not required to know any URL of their siteThe URLs are available right there in the UI. With the click of a check mark, the user would not have to type a thing.

The disadvantage of this setup is the dependency of the Always Online crawler.

First off, some users do not wish to have Always Online turned on. Without the consent of the site owner to crawl the site via Always Online, the page loader tree will not load and the user had no options of pages to install an app on.

When a user does have Always Online enabled properly, the crawler might not crawl every page the site owner wishes to install an app on.

The duty of Always Online is to make sure in the most catastrophic event for a site owner - their site being down - users can still see a version of the site via cached static HTML. Once upon a time before Always Online v2, we actually used the Google bot and other search engine crawlers’ activity to decide what to cache for the Always Online feature. We found that implementing our own crawler made more sense. Our goal is to make sure the most vital pages of a site are crawled and stored on our cache, contrasting with search engine crawler’s priority of get the most information possible from the site, thus going “deep” into the depths of a site map.

The duty of an app install on Cloudflare’s Apps platform is to seamlessly enable users to select pages in which to inject Javascript, HTML, CSS, and in the near future, Cloudflare Service Workers into. Since the objectives of the Always Online crawler differ from that of the Cloudflare Apps platform, there were inevitable consequences. Here are some examples where a page would not be crawled:

• The page’s subdomain was not "orange-clouded".

• The page was not be accessible from the site's homepage via links.

• The site’s homepage had too many links for us to follow.

• The page was password-protected, preventing us from accessing it and adding it to your site map.

• The page was added before we had a chance to crawl the site.

Although our custom crawler works well for the Always Online feature, this limited control for our customers who are installing apps. We decided to do something about it. Combining the advantages of the crawler data we already had implemented with the ability to enter any URL in an install, we created the best of both worlds.

Now, site owners can type in whatever URL they wish to install an app. There is also an option for selecting an entire directory or strictly that page. For simplicity, no regex patterns are supported.

As the apps on the Cloudflare Apps platform advance, it is vital that the platform itself advance. In the near future, the App’s platform will have the power of Cloudflare Workers, local testing, and much more to come.