Cloudforce One is publishing the results of our investigation and real-time effort to detect, deny, degrade, disrupt, and delay threat activity by the Russia-aligned threat actor FlyingYeti during their latest phishing campaign targeting Ukraine. At the onset of Russia’s invasion of Ukraine on February 24, 2022, Ukraine introduced a moratorium on evictions and termination of utility services for unpaid debt. The moratorium ended in January 2024, resulting in significant debt liability and increased financial stress for Ukrainian citizens. The FlyingYeti campaign capitalized on anxiety over the potential loss of access to housing and utilities by enticing targets to open malicious files via debt-themed lures. If opened, the files would result in infection with the PowerShell malware known as COOKBOX, allowing FlyingYeti to support follow-on objectives, such as installation of additional payloads and control over the victim’s system.

Since April 26, 2024, Cloudforce One has taken measures to prevent FlyingYeti from launching their phishing campaign – a campaign involving the use of Cloudflare Workers and GitHub, as well as exploitation of the WinRAR vulnerability CVE-2023-38831. Our countermeasures included internal actions, such as detections and code takedowns, as well as external collaboration with third parties to remove the actor’s cloud-hosted malware. Our effectiveness against this actor prolonged their operational timeline from days to weeks. For example, in a single instance, FlyingYeti spent almost eight hours debugging their code as a result of our mitigations. By employing proactive defense measures, we successfully stopped this determined threat actor from achieving their objectives.

• On April 18, 2024, Cloudforce One detected the Russia-aligned threat actor FlyingYeti preparing to launch a phishing espionage campaign targeting individuals in Ukraine.

• We discovered the actor used similar tactics, techniques, and procedures (TTPs) as those detailed in Ukranian CERT's article on UAC-0149, a threat group that has primarily targeted Ukrainian defense entities with COOKBOX malware since at least the fall of 2023.

• From mid-April to mid-May, we observed FlyingYeti conduct reconnaissance activity, create lure content for use in their phishing campaign, and develop various iterations of their malware. We assessed that the threat actor intended to launch their campaign in early May, likely following Orthodox Easter.

• After several weeks of monitoring actor reconnaissance and weaponization activity (Cyber Kill Chain Stages 1 and 2), we successfully disrupted FlyingYeti’s operation moments after the final COOKBOX payload was built.

• The payload included an exploit for the WinRAR vulnerability CVE-2023-38831, which FlyingYeti will likely continue to use in their phishing campaigns to infect targets with malware.

• We offer steps users can take to defend themselves against FlyingYeti phishing operations, and also provide recommendations, detections, and indicators of compromise.

FlyingYeti is the cryptonym given by Cloudforce One to the threat group behind this phishing campaign, which overlaps with UAC-0149 activity tracked by CERT-UA in February and April 2024. The threat actor uses dynamic DNS (DDNS) for their infrastructure and leverages cloud-based platforms for hosting malicious content and for malware command and control (C2). Our investigation of FlyingYeti TTPs suggests this is likely a Russia-aligned threat group. The actor appears to primarily focus on targeting Ukrainian military entities. Additionally, we observed Russian-language comments in FlyingYeti’s code, and the actor’s operational hours falling within the UTC+3 time zone.

In the days leading up to the start of the campaign, Cloudforce One observed FlyingYeti conducting reconnaissance on payment processes for Ukrainian communal housing and utility services:

• April 22, 2024 – research into changes made in 2016 that introduced the use of QR codes in payment notices

• April 22, 2024 – research on current developments concerning housing and utility debt in Ukraine

• April 25, 2024 – research on the legal basis for restructuring housing debt in Ukraine as well as debt involving utilities, such as gas and electricity

Cloudforce One judges that the observed reconnaissance is likely due to the Ukrainian government’s payment moratorium introduced at the start of the full-fledged invasion in February 2022. Under this moratorium, outstanding debt would not lead to evictions or termination of provision of utility services. However, on January 9, 2024, the government lifted this ban, resulting in increased pressure on Ukrainian citizens with outstanding debt. FlyingYeti sought to capitalize on that pressure, leveraging debt restructuring and payment-related lures in an attempt to increase their chances of successfully targeting Ukrainian individuals.

The disrupted phishing campaign would have directed FlyingYeti targets to an actor-controlled GitHub page at hxxps[:]//komunalka[.]github[.]io, which is a spoofed version of the Kyiv Komunalka communal housing site https://www.komunalka.ua. Komunalka functions as a payment processor for residents in the Kyiv region and allows for payment of utilities, such as gas, electricity, telephone, and Internet. Additionally, users can pay other fees and fines, and even donate to Ukraine’s defense forces.

Based on past FlyingYeti operations, targets may be directed to the actor’s Github page via a link in a phishing email or an encrypted Signal message. If a target accesses the spoofed Komunalka platform at hxxps[:]//komunalka[.]github[.]io, the page displays a large green button with a prompt to download the document “Рахунок.docx” (“Invoice.docx”), as shown in Figure 1. This button masquerades as a link to an overdue payment invoice but actually results in the download of the malicious archive “Заборгованість по ЖКП.rar” (“Debt for housing and utility services.rar”).

Figure 1: Prompt to download malicious archive “Заборгованість по ЖКП.rar”

A series of steps must take place for the download to successfully occur:

• The target clicks the green button on the actor’s GitHub page hxxps[:]//komunalka.github[.]io

• The target’s device sends an HTTP POST request to the Cloudflare Worker worker-polished-union-f396[.]vqu89698[.]workers[.]dev with the HTTP request body set to “user=Iahhdr”

• The Cloudflare Worker processes the request and evaluates the HTTP request body

• If the request conditions are met, the Worker fetches the RAR file from hxxps[:]//raw[.]githubusercontent[.]com/kudoc8989/project/main/Заборгованість по ЖКП.rar, which is then downloaded on the target’s device

Cloudforce One identified the infrastructure responsible for facilitating the download of the malicious RAR file and remediated the actor-associated Worker, preventing FlyingYeti from delivering its malicious tooling. In an effort to circumvent Cloudforce One's mitigation measures, FlyingYeti later changed their malware delivery method. Instead of the Workers domain fetching the malicious RAR file, it was loaded directly from GitHub.

During remediation, Cloudforce One recovered the RAR file “Заборгованість по ЖКП.rar” and performed analysis of the malicious payload. The downloaded RAR archive contains multiple files, including a file with a name that contains the unicode character “U+201F”. This character appears as whitespace on Windows devices and can be used to “hide” file extensions by adding excessive whitespace between the filename and the file extension. As highlighted in blue in Figure 2, this cleverly named file within the RAR archive appears to be a PDF document but is actually a malicious CMD file (“Рахунок на оплату.pdf[unicode character U+201F].cmd”).

Figure 2: Files contained in the malicious RAR archive “Заборгованість по ЖКП.rar” (“Housing Debt.rar”)

FlyingYeti included a benign PDF in the archive with the same name as the CMD file but without the unicode character, “Рахунок на оплату.pdf” (“Invoice for payment.pdf”). Additionally, the directory name for the archive once decompressed also contained the name “Рахунок на оплату.pdf”. This overlap in names of the benign PDF and the directory allows the actor to exploit the WinRAR vulnerability CVE-2023-38831. More specifically, when an archive includes a benign file with the same name as the directory, the entire contents of the directory are opened by the WinRAR application, resulting in the execution of the malicious CMD. In other words, when the target believes they are opening the benign PDF “Рахунок на оплату.pdf”, the malicious CMD file is executed.

The CMD file contains the FlyingYeti PowerShell malware known as COOKBOX. The malware is designed to persist on a host, serving as a foothold in the infected device. Once installed, this variant of COOKBOX will make requests to the DDNS domain postdock[.]serveftp[.]com for C2, awaiting PowerShell cmdlets that the malware will subsequently run.

Alongside COOKBOX, several decoy documents are opened, which contain hidden tracking links using the Canary Tokens service. The first document, shown in Figure 3 below, poses as an agreement under which debt for housing and utility services will be restructured.

Figure 3: Decoy document Реструктуризація боргу за житлово комунальні послуги.docx

The second document (Figure 4) is a user agreement outlining the terms and conditions for the usage of the payment platform komunalka[.]ua.

Figure 4: Decoy document Угода користувача.docx (User Agreement.docx)

The use of relevant decoy documents as part of the phishing and delivery activity are likely an effort by FlyingYeti operators to increase the appearance of legitimacy of their activities.

The phishing theme we identified in this campaign is likely one of many themes leveraged by this actor in a larger operation to target Ukrainian entities, in particular their defense forces. In fact, the threat activity we detailed in this blog uses many of the same techniques outlined in a recent FlyingYeti campaign disclosed by CERT-UA in mid-April 2024, where the actor leveraged United Nations-themed lures involving Peace Support Operations to target Ukraine’s military. Due to Cloudforce One’s defensive actions covered in the next section, this latest FlyingYeti campaign was prevented as of the time of publication.

Cloudforce One mitigated FlyingYeti’s campaign through a series of actions. Each action was taken to increase the actor’s cost of continuing their operations. When assessing which action to take and why, we carefully weighed the pros and cons in order to provide an effective active defense strategy against this actor. Our general goal was to increase the amount of time the threat actor spent trying to develop and weaponize their campaign.

We were able to successfully extend the timeline of the threat actor’s operations from hours to weeks. At each interdiction point, we assessed the impact of our mitigation to ensure the actor would spend more time attempting to launch their campaign. Our mitigation measures disrupted the actor’s activity, in one instance resulting in eight additional hours spent on debugging code.

Due to our proactive defense efforts, FlyingYeti operators adapted their tactics multiple times in their attempts to launch the campaign. The actor originally intended to have the Cloudflare Worker fetch the malicious RAR file from GitHub. After Cloudforce One interdiction of the Worker, the actor attempted to create additional Workers via a new account. In response, we disabled all Workers, leading the actor to load the RAR file directly from GitHub. Cloudforce One notified GitHub, resulting in the takedown of the RAR file, the GitHub project, and suspension of the account used to host the RAR file. In return, FlyingYeti began testing the option to host the RAR file on the file sharing sites pixeldrain and Filemail, where we observed the actor alternating the link on the Komunalka phishing site between the following:

• hxxps://pixeldrain[.]com/api/file/ZAJxwFFX?download=one

• hxxps://1014.filemail[.]com/api/file/get?filekey=e_8S1HEnM5Rzhy_jpN6nL-GF4UAP533VrXzgXjxH1GzbVQZvmpFzrFA&pk_vid=a3d82455433c8ad11715865826cf18f6

We notified GitHub of the actor’s evolving tactics, and in response GitHub removed the Komunalka phishing site. After analyzing the files hosted on pixeldrain and Filemail, we determined the actor uploaded dummy payloads, likely to monitor access to their phishing infrastructure (FileMail logs IP addresses, and both file hosting sites provide view and download counts). At the time of publication, we did not observe FlyingYeti upload the malicious RAR file to either file hosting site, nor did we identify the use of alternative phishing or malware delivery methods.

A timeline of FlyingYeti’s activity and our corresponding mitigations can be found below.

Cloudforce One leveraged industry relationships to provide advanced warning and to mitigate the actor’s activity. To further protect the intended targets from this phishing threat, Cloudforce One notified and collaborated closely with GitHub’s Threat Intelligence and Trust and Safety Teams. We also notified CERT-UA and Cloudflare industry partners such as CrowdStrike, Mandiant/Google Threat Intelligence, and Microsoft Threat Intelligence.

There are several ways to hunt FlyingYeti in your environment. These include using PowerShell to hunt for WinRAR files, deploying Microsoft Sentinel analytics rules, and running Splunk scripts as detailed below. Note that these detections may identify activity related to this threat, but may also trigger unrelated threat activity.

Consider running a PowerShell script such as this one in your environment to identify exploitation of CVE-2023-38831. This script will interrogate WinRAR files for evidence of the exploit.

CVE-2023-38831
Description:winrar exploit detection 
open suspios (.tar / .zip / .rar) and run this script to check it 

function winrar-exploit-detect(){
$targetExtensions = @(".cmd" , ".ps1" , ".bat")
$tempDir = [System.Environment]::GetEnvironmentVariable("TEMP")
$dirsToCheck = Get-ChildItem -Path $tempDir -Directory -Filter "Rar*"
foreach ($dir in $dirsToCheck) {
    $files = Get-ChildItem -Path $dir.FullName -File
    foreach ($file in $files) {
        $fileName = $file.Name
        $fileExtension = [System.IO.Path]::GetExtension($fileName)
        if ($targetExtensions -contains $fileExtension) {
            $fileWithoutExtension = [System.IO.Path]::GetFileNameWithoutExtension($fileName); $filename.TrimEnd() -replace '\.$'
            $cmdFileName = "$fileWithoutExtension"
            $secondFile = Join-Path -Path $dir.FullName -ChildPath $cmdFileName
            
            if (Test-Path $secondFile -PathType Leaf) {
                Write-Host "[!] Suspicious pair detected "
                Write-Host "[*]  Original File:$($secondFile)" -ForegroundColor Green 
                Write-Host "[*] Suspicious File:$($file.FullName)" -ForegroundColor Red

                # Read and display the content of the command file
                $cmdFileContent = Get-Content -Path $($file.FullName)
                Write-Host "[+] Command File Content:$cmdFileContent"
            }
        }
    }
}
}
winrar-exploit-detect

Microsoft Sentinel

In Microsoft Sentinel, consider deploying the rule provided below, which identifies WinRAR execution via cmd.exe. Results generated by this rule may be indicative of attack activity on the endpoint and should be analyzed.

DeviceProcessEvents
| where InitiatingProcessParentFileName has @"winrar.exe"
| where InitiatingProcessFileName has @"cmd.exe"
| project Timestamp, DeviceName, FileName, FolderPath, ProcessCommandLine, AccountName
| sort by Timestamp desc

Splunk

Consider using this script in your Splunk environment to look for WinRAR CVE-2023-38831 execution on your Microsoft endpoints. Results generated by this script may be indicative of attack activity on the endpoint and should be analyzed.

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.parent_process_name=winrar.exe `windows_shells` OR Processes.process_name IN ("certutil.exe","mshta.exe","bitsadmin.exe") by Processes.dest Processes.user Processes.parent_process_name Processes.parent_process Processes.process_name Processes.process Processes.process_id Processes.parent_process_id 
| `drop_dm_object_name(Processes)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)` 
| `winrar_spawning_shell_application_filter`

Cloudflare Email Security (CES) customers can identify FlyingYeti threat activity with the following detections.

• CVE-2023-38831

• FLYINGYETI.COOKBOX

• FLYINGYETI.COOKBOX.Launcher

• FLYINGYETI.Rar

Cloudflare recommends taking the following steps to mitigate this type of activity:

• Implement Zero Trust architecture foundations:    

• Deploy Cloud Email Security to ensure that email services are protected against phishing, BEC and other threats

• Leverage browser isolation to separate messaging applications like LinkedIn, email, and Signal from your main network

• Scan, monitor and/or enforce controls on specific or sensitive data moving through your network environment with data loss prevention policies

• Ensure your systems have the latest WinRAR and Microsoft security updates installed

• Consider preventing WinRAR files from entering your environment, both at your Cloud Email Security solution and your Internet Traffic Gateway

• Run an Endpoint Detection and Response (EDR) tool such as CrowdStrike or Microsoft Defender for Endpoint to get visibility into binary execution on hosts

• Search your environment for the FlyingYeti indicators of compromise (IOCs) shown below to identify potential actor activity within your network.

If you’re looking to uncover additional Threat Intelligence insights for your organization or need bespoke Threat Intelligence information for an incident, consider engaging with Cloudforce One by contacting your Customer Success manager or filling out this form.

You’ve had a chance to build a Cloudflare Worker. You’ve tried KV Storage and have a great use case for your Worker. You’ve even demonstrated the usefulness to your product or organization. Now you need to go from writing a single file in the Cloudflare Dashboard UI Editor to source controlled code with multiple environments deployed using your favorite CI tool.

Fortunately, we have a powerful and flexible API for managing your workers. You can customize your deployment to your heart’s content. Our blog has already featured many things made possible by that API:

• The Wrangler CLI

• CI/CD Pipeline

• Github Actions

• Worker bootstrap template

These tools make deployments easier to configure, but it still takes time to manage. The Serverless Framework Cloudflare Workers plugin removes that deployment overhead so you can spend more time working on your application and less on your deployment.

Here at Cloudflare, we’ve been working to rebuild our Access product to run entirely on Workers. The move will allow Access to take advantage of the resiliency, performance, and flexibility of Workers. We’ll publish a more detailed post about that migration once complete, but the experience required that we retool some of our process to match or existing development experience as much as possible.

To us this meant:

• Git

• Easily deploy

• Different environments

• Unit Testing

• CI Integration

• Typescript/Multiple Files

• Everything Must Be Automated

The Cloudflare Access team looked at three options for automating all of these tools in our pipeline. All of the options will work and could be right for you, but custom scripting can be a chore to maintain and Terraform lacked some extensibility.

1. Custom Scripting

2. Terraform

3. Serverless Framework

We decided on the Serverless Framework. Serverless Framework provided a tool to mirror our existing process as closely as possible without too much DevOps overhead. Serverless is extremely simple and doesn’t interfere with the application code. You can get a project set up and deployed in seconds. It’s obviously less work than writing your own custom management scripts. But it also requires less boiler plate than Terraform because the Serverless Framework is designed for the “serverless” niche. However, if you are already using Terraform to manage other Cloudflare products, Terraform might be the best fit.

Everything for the project happens in a YAML file called serverless.yml. Let’s go through the features of the configuration file.

To get started, we need to install serverless from npm and generate a new project.

npm install serverless -g
serverless create --template cloudflare-workers --path myproject
cd myproject
npm install

If you are an enterprise client, you want to use the cloudflare-workers-enterprise template as it will set up more than one worker (but don’t worry, you can add more to any template). Also, I’ll touch on this later, but if you want to write your workers in Rust, use the cloudflare-workers-rust template.

You should now have a project that feels familiar, ready to be added to your favorite source control. In the project should be a serverless.yml file like the following.

service:
    name: hello-world

provider:
  name: cloudflare
  config:
    accountId: CLOUDFLARE_ACCOUNT_ID
    zoneId: CLOUDFLARE_ZONE_ID

plugins:
  - serverless-cloudflare-workers

functions:
  hello:
    name: hello
    script: helloWorld  # there must be a file called helloWorld.js
    events:
      - http:
          url: example.com/hello/*
          method: GET
          headers:
            foo: bar
            x-client-data: value

The service block simply contains the name of your service. This will be used in your Worker script names if you do not overwrite them.

Under provider, name must be ‘cloudflare’  and you need to add your account and zone IDs. You can find them in the Cloudflare Dashboard.

The plugins section adds the Cloudflare specific code.

Now for the good part: functions. Each block under functions is a Worker script.

name: (optional) If left blank it will be STAGE-service.name-script.identifier. If I removed name from this file and deployed in production stage, the script would be named production-hello-world-hello.

script: the relative path to the javascript file with the worker script. I like to organize mine in a folder called handlers.

events: Currently Workers only support http events. We call these routes. The example provided says that GET https://example.com/hello/ will  cause this worker to execute. The headers block is for testing invocations.

At this point you can deploy your worker!

CLOUDFLARE_AUTH_EMAIL=you@yourdomain.com CLOUDFLARE_AUTH_KEY=XXXXXXXX serverless deploy

This is very easy to deploy, but it doesn’t address our requirements. Luckily, there’s just a few simple modifications to make.

Here’s a more complex YAML file.

service:
  name: hello-world

package:
  exclude:
    - node_modules/**
  excludeDevDependencies: false

custom:
  defaultStage: development
  deployVars: ${file(./config/deploy.${self:provider.stage}.yml)}

kv: &kv
  - variable: MYUSERS
    namespace: users

provider:
  name: cloudflare
  stage: ${opt:stage, self:custom.defaultStage}
  config:
    accountId: ${env:CLOUDFLARE_ACCOUNT_ID}
    zoneId: ${env:CLOUDFLARE_ZONE_ID}

plugins:
  - serverless-cloudflare-workers

functions:
  hello:
    name: ${self:provider.stage}-hello
    script: handlers/hello
    webpack: true
    environment:
      MY_ENV_VAR: ${self:custom.deployVars.env_var_value}
      SENTRY_KEY: ${self:custom.deployVars.sentry_key}
    resources: 
      kv: *kv
    events:
      - http:
          url: "${self:custom.deployVars.SUBDOMAIN}.mydomain.com/hello"
          method: GET
      - http:
          url: "${self:custom.deployVars.SUBDOMAIN}.mydomain.com/alsohello*"
          method: GET

We can add a custom section where we can put custom variables to use later in the file.

defaultStage: We set this to development so that forgetting to pass a stage doesn’t trigger a production deploy. Combined with the stage option under provider we can set the stage for deployment.

deployVars: We use this custom variable to load another YAML file dependent on the stage. This lets us have different values for different stages. In development, this line loads the file ./config/deploy.development.yml. Here’s an example file:

env_var_value: true
sentry_key: XXXXX
SUBDOMAIN: dev

kv: Here we are showing off a feature of YAML. If you assign a name to a block using the ‘&’, you can use it later as a YAML variable. This is very handy in a multi script account. We could have named this variable anything, but we are naming it kv since it holds our Workers Key Value storage settings to be used in our function below.

Inside of the kv block we're creating a namespace and binding it to a variable available in your Worker. It will ensure that the namespace “users” exists and is bound to MYUSERS.

kv: &kv
  - variable: MYUSERS
    namespace: users

provider: The only new part of the provider block is stage.

stage: ${opt:stage, self:custom.defaultStage}

This line sets stage to either the command line option or custom.defaultStage if opt:stage is blank. When we deploy, we pass —stage=production to serverless deploy.

Now under our function we have added webpack, resources, and environment.

webpack: If set to true, will simply bundle each handler into a single file for deployment. It will also take a string representing a path to a webpack config file so you can customize it. This is how we add Typescript support to our projects.

resources: This block is used to automate resource creation. In resources we're linking back to the kv block we created earlier.

Side note: If you would like to include WASM bindings in your project, it can be done in a very similar way to how we included Workers KV. For more information on WASM, see the documentation.

environment: This is the butter for the bread that is managing configuration for different stages. Here we can specify values to bind to variables to use in worker scripts. Combined with YAML magic, we can store our values in the aforementioned config files so that we deploy different values in different stages. With environments, we can easily tie into our CI tool. The CI tool has our deploy.production.yml. We simply run the following command from within our CI.

sls deploy --stage=production

Finally, I added a route to demonstrate that a script can be executed on multiple routes.

At this point I’ve covered (or hinted) at everything on our original list except Unit Testing. There are a few ways to do this.

We have a previous blog post about Unit Testing that covers using cloud worker, a great tool built by Dollar Shave Club.

My team opted to use the classic node frameworks mocha and sinon. Because we are using Typescript, we can build for node or build for v8. You can also make mocha work for non-typescript projects if you use an experimental feature that adds import/export support to node.

--experimental-modules

We’re excited about moving more and more of our services to Cloudflare Workers, and the Serverless Framework makes that easier to do. If you’d like to learn even more or get involved with the project, see us on github.com. For additional information on using Serverless Framework with Cloudflare Workers, check out our documentation on the Serverless Framework.

Want to automatically deploy Workers directly from a GitHub repository? Now you can with our official GitHub Action. This Action is an extension of our existing integration with the Serverless Framework. It runs in a containerized GitHub environment and automatically deploys your Worker to Cloudflare. We chose to utilize the Serverless Framework within our GitHub Action to raise awareness of their awesome work and to enable even more serverless applications to be built with Cloudflare Workers. This Action can be used to deploy individual Worker scripts as well; the Serverless Framework is being used in the background as the deployment mechanism.

Before going into the details, we’ll quickly go over what GitHub Actions are.

GitHub Actions allow you to trigger commands in reaction to GitHub events. These commands run in containers and can receive environment variables. Actions could trigger build, test, or deployment commands across a variety of providers. They can also be linked and run sequentially (i.e. ‘if the build passes, deploy the app’). Similar to many CI/CD tools, these commands run in an isolated container and receive environment variables. You can pass any command to the container that enables your development workflow.

Actions are a powerful way to your workflow on GitHub, including automating parts of your deployment pipeline directly from where your codebase lives. To that end, we’ve built an Action to deploy a Worker to your Cloudflare zone via our existing Serverless Framework integration for Cloudflare Workers. To visualize the entire flow see below:

To see some of the other actions out there today, please see here.

Serverless applications are deployed without developers needing to worry about provisioning hardware, capacity planning, scaling or paying for equipment when your application isn't running. Unlike most providers who ask you to choose a region for your serverless app to run in, all Cloudflare Workers deploy into our entire global network. Learn more about the benefits of serverless.

The Serverless Framework is a popular toolkit for deploying applications that are serverless. The advantage of the Serverless Framework is that it offers a common CLI to use across multiple providers which support serverless applications. In late 2018, Cloudflare integrated Workers deployment into the Serverless CLI. Please check out our docs here to get started.

If you run an entire application in a Worker, there is no cost to a business when the application is idle. If the application runs on our network (Cloudflare has 165 PoPs as of writing this), the app can be incredibly close to the end user, reducing latency by proximity. Additionally, Workers can be a powerful way to augment what you've already built in an existing technology, moving just the authentication or performance-sensitive components into Workers.

Configuration of the Action is straightforward, with the side benefit of giving you just a ‘little bit’™ of exposure to the Serverless Framework if desired. A repo using this Action can just contain the Worker script to be deployed. If you feed the Action the right ENV variables, we’ll take care of the rest.

Alternatively you can also provide a serverless.yml in the root of your repo with your worker if you want to override the defaults. Get started learning about our integration with Serverless here.

Your Worker script, and optional serverless.yml are passed into the container which runs the Action for deployment. The Serverless Framework picks up these files and deploys the Worker for you.

All the relevant variables must be passed to the Action as well, which include various account identifiers as well as your API key. You can check out this tutorial from GitHub on how to pass environmental variables to an Action (hint: use the secret variable type for your API key).

Support

The repository is publicly available here which goes over the configuration in more technical detail. Any question/suggestions feel free to let us know!

PAL is available on Github: github.com/cloudflare/pal

Although PAL is not currently under active development, we have found it a useful tool and we think the community will benefit from its source being available. We believe that it's better to open source this tool and allow others to use the code than leave it hidden from view and unmaintained.

CC BY 2.0 image by Personal Creations

How do you get these secrets to your services? If you’re the only developer, or one of a few on a project, you might put the secrets with your source code in your version control system. But if you just store the secrets in plain text with your code, everyone with access to your source repository can read them and use them for nefarious purposes (for example, stealing an API token and pretending to be an authorized client). Furthermore, distributed version control systems like Git will download a copy of the secret everywhere a repository is cloned, regardless of whether it’s needed there, and will keep that copy in the commit history forever. For a company where many people (including people who work on unrelated systems) have access to source control this just isn’t an option.

Another idea is to keep your secrets in a secure place and then embed them into your application artifacts (binaries, containers, packages, etc.) at build time. This can be awkward for modern CI/CD workflows because it results in multiple parallel sets of artifacts for different environments (e.g. production, staging, development). Once you have artifacts with secrets, they become secret themselves, and you will have to restrict access to the “armed” packages that contain secrets after they’re built. Consider the discovery last year that the source code of Twitter’s Vine service was available in public Docker repositories. Not only was the source code for the service leaked, but the API keys that allow Vine to interact with other services were also available. Vine paid over $10,000 when they were notified about this.

A more advanced technique to manage and deploy secrets is to use a secret management service. Secret management services can be used to create, store and rotate secrets as well as distribute them to applications. The secret management service acts as a gatekeeper, allowing access to some secrets for some applications as prescribed by an access control policy. An application that wants to gain access to a secret authenticates itself to the secret manager, the secret manager checks permissions to see if the application is authorized, and if authorized, sends the secret. There are many options to choose from, including Vault, Keywhiz, Knox, Secretary, dssss and even Docker’s own secret managment service.

Secret managers are a good solution as long as an identity/authorization system is already in place. However, since most authentication systems involve the client already being in possession of a secret, this presents a chicken and egg problem.

Once we have verified a service’s identity, we can make access control decisions about what that service can access. Therefore, the real problem we must solve is the problem of bootstrapping service identity.

This problem has many solutions when services are tightly bound to individual machines (for example, we can simply install host-level credentials on each machine or even use a machine’s hardware to identify it. Virtual machine platforms like Amazon AWS have machine-based APIs for host-level identity, like IAM and KMS). Containerized services have a much more fluid lifecycle - instances may appear on many machines and may come and go over time. Furthermore, any number of trusted and untrusted containers might be running on the same host at the same time. So what we need instead is an identity that belongs to a service, not to a machine.

Every Application needs an ID to prove to the bouncer that they’re on the guest list for Club Secret.

Bootstrapping the identity of a service that lives in a container is not a solved problem, and most of the existing are deeply integrated into the container orchestration (Kubernetes, Docker Swarm, Mesos, etc.). We ran into the problem of container identity bootstrapping, and wanted something that worked with or current application deployment stack (Docker/Mesos/Marathon) but wasn’t locked down to a given orchestration platform.

We use Docker containers to deploy many services across a shared, general-purpose Mesos cluster. To solve the service identity bootstrapping problem in our Docker environment, we developed PAL, which stands for Permissive Action Link, a security device for nuclear weapons. PAL makes sure secrets are only available in production, and only when jobs are authorized to run.

PAL makes it possible to keep only encrypted secrets in the configuration for a service while ensuring that those secrets can only be decrypted by authorized service instances in an approved environment (say, a production or staging environment). If those credentials serve to identify the service requesting access to secrets, PAL becomes a container identity bootstrapping solution that you can easily deploy a secret manager on top of.

The model for PAL is that the secrets areprovided in an encrypted form and either embedded in containers, or provided as runtime configuration for jobs running in an orchestration framework such as Apache Mesos.

PAL allows secrets to be decrypted at runtime after the service’s identity has been established. These credentials could allow authenticated inter-service communication, which would allow you to keep service secrets in a central repository such as Hashicorp’s Vault, KeyWhiz, or others. The credentials could also be used to issue service-level credentials (such as certificates for an internal PKI). Without PAL, you must distribute the identity credentials, required by tools like these themselves, inside your infrastructure.

PAL consists of two components: a small in-container initialization tool, pal, that requests secrets decryption and installs decrypted secret, and a daemon called pald that runs on every node in the cluster. pal and pald communicate with each other via a Unix socket. The pal tool is set as each job’s entrypoint, and it sends pald the encrypted secrets. pald then identifies the process making the job and determines whether it is allowed to access the requested secret. If so, it decrypts the secret on behalf of the job and returns the plaintext to pal, which installs the plaintext within the calling job’s container as either an environment variable or a file.

PAL currently supports two methods of encryption—PGP and Red October—but it can be extended to support more.

PGP is a popular form of encryption that has been around since the early 90s. PAL allows you to use secrets that are encrypted with PGP keys that are installed on the host. The current version of PAL does not apply policies at a per-key level (e.g. only containers with Docker Label A can use key 1), but it could easily be extended to do so.

The Red October mode is used for secrets that are very high value and need to be managed manually or with multi-person control. We open sourced Red October back in 2013. It has the nice feature of being able to encrypt a secret in such a way that multiple people are required to authorize the decryption.

In the PAL-RO typical setup, each machine in your cluster will be provisioned a Red October account. Before a container is scheduled to run, the secret owners delegate the ability to decrypt the secret to the host on which the container is going to run. When the container starts, pal calls pald which uses the machine’s Red October credentials to decrypt the secret via a call to the Red October server. Delegations can be of a limited time or for a number of decryptions. Once the delegations are used up, Red October has no way to decrypt the secret.

These two modes have been invaluable for protecting high-value secrets, where Red October provides additional oversight and control. For lower-value secrets, PGP provides a non-interactive experience that works well with ephemeral containers.

An important part of a secret management tool is ensuring that only authorized entities can decrypt a secret. PAL enables you to control which containers can decrypt a secret by leveraging existing code signing infrastructure. Both secrets and containers can be given optional labels that PAL will respect. Labels define which containers can access which secrets—a container must have the label of any secret it accesses. Labels are named references to security policies. An example label could be “production-team-secret” which denotes that a secret should conform to the production team’s secret policy. Labels bind cipher texts to an authorization to decrypt. These authorizations allow you to use PAL to control when and in what environment secrets can be decrypted.

By opening the Unix socket with the option SO_PASSCRED, we enable pald to obtain the process-level credentials (uid, pid, and gid) of the caller for each request. These credentials can then be used to identify containers and assign them a label. Labels allow PAL to consult a predefined policy and authorize containers to receive secrets. To get the list of labels on a container, pald uses the process id (pid) of the calling process to get its cgroups from Linux (by reading and parsing /proc/<pid>/cgroups). The names of the cgroups contain the Docker container id, which we can use to get container metadata via Docker’s inspect call. This container metadata carries a list of labels assigned by the Docker LABEL directive at build time.

Containers and their labels must be bound together using code integrity tools. PAL supports using Docker’s Notary, which confirms that a specific container hash maps to specific metadata like a container’s name and label.

PAL represents one solution for identity bootstrapping for our environment. Other service identity bootstrapping tools bootstrap at the host-level or are highly environmentally coupled. AWS IAM, for example, only works at the level of virtual machines running on AWS; Kubernetes secrets and Docker secrets management only work in Kubernetes and Docker Swarm, respectively. While we’ve developed PAL to work alongside Mesos, we designed it to used as a service identity mechanism for many other environments simply by plugging in new ways for PAL to read identities from service artifacts (containers, packages, or binaries).

Recall the issue where Vine disclosed their source code in their Docker containers on a public repository, Docker Hub. With PAL, Vine could have kept their API keys (or even the entire codebase) encrypted in the container, published that safe version of the container to Docker Hub, and decrypted the code at container startup in their particular production environment.

Using PAL, you can give your trusted containers an identity that allows them to safely receive secrets only in production, without the risks associated with other secret distribution methods. This identity can be a secret like a cryptographic key, allowing your service to decrypt its sensitive configuration, or it could be a credential that allows it to access sensitive services such as secret managers or CAs. PAL solves a key bootstrapping problem for service identity, making it simple to run trusted and untrusted containers side-by-side while ensuring that your secrets are safe.

PAL was created by Joshua Kroll, Daniel Dao, and Ben Burkert with design prototyping by Nick Sullivan. This post was adapted from an internal blog by Joshua Kroll and a presentation I made at O’Reilly Security Amsterdam in 2016 , and BSides Las Vegas.

CC-BY 2.0 image by Kevin Dooley

We got to thinking. Are DNS records not just a piece of the configuration? We concluded that they are and decided to manage our own records from Salt too.

We are strong believers in eating our own dog food, so we make our employees use the next version of our service before rolling it to everyone else. That way if there's a problem visiting one of the 5 million websites that use Cloudflare it'll get spotted quickly internally. This is also why we keep our own DNS records on Cloudflare itself.

Cloudflare has an API that allows you to manage your zones programmatically without ever logging into the dashboard. Until recently, we were using handcrafted scripts to manage our own DNS records via our API. These scripts were in exotic languages like PHP for historical reasons and had interesting behavior that not everybody enjoyed. While we were dogfooding our own APIs, these scripts were pushing the source of truth for DNS out of Salt.

When we decided to move some zones to Salt, we had a few key motivations:

1. Single source of truth

2. Peer-reviewed, audited and versioned changes

3. Making things that our customers would want to use

Points 1 and 2 were achieved by having DNS records in a Salt repo. The Salt configuration is itself in git, so we get peer reviews and audit trail for free. We think that we made progress on point 3 also.

After extensive internal testing and finding a few bugs in our API (that's what we wanted!), we are happy to announce the public availability of Cloudflare Salt module.

If you are familiar with Salt, it should be easy to see how to configure your records via Salt. All you need is the following:

Create the state cloudflare to deploy your zones:

example.com:
  cloudflare.manage_zone_records:
    - zone: {{ pillar["cloudflare_zones"]["example.com"]|yaml }}

Add a pillar to configure your zone:

cloudflare_zones:
  example.com:
    auth_email: ivan@example.com
    auth_key: auth key goes here
    zone_id: 0101deadbeefdeadbeefdeadbeefdead
    records:
      - name: blog.example.com
        content: 93.184.216.34
        proxied: true

Here we configure zone example.com to only have one record blog.example.com pointing to 93.184.216.34 behind Cloudflare.

You can test your changes before you deploy:

salt-call state.apply cloudflare test=true

And then deploy if you are happy with the dry run:

salt-call state.apply cloudflare

After the initial setup, all you need is to change the records array in pillar and re-deploy the state. See the README for more details.

DNS records are only one part of configuration you may want to change for your Cloudflare domain. We have plans to "saltify" other settings like WAF, caching, page rules and others too.

Come work with us if you want to help!

In this tutorial, I will show you how to use CloudFlare and GitHub together. By taking advantage of CloudFlare’s global network, you can utilise our CDN service to improve your site's performance and security.

Whilst GitHub Pages doesn't ordinarily support SSL on custom domains, CloudFlare's Universal SSL allows your users to access your site over SSL, thus opening up the performance advantages of HTTP/2.

GitHub Pages is designed to host sites that only serve static HTML. The ability to only host static content isn’t as big of a restriction as you might think.

Static site generators avoid repetitive update tasks of updating “latest posts” feeds, pagination or sitemaps; whilst generating static HTML that can be uploaded to any web hosting service without a scripting engine. Unlike ancient desktop tools like FrontPage and Dreamweaver which lacked a content model, modern static site generators have the design decisively separate from content.

Typically, CMS-based sites must query a database for content, then render the HTML to be served to the end user; all this to serve the same content for request after request. Even with caching this combination is hardly elegant for sites where only the administrator changes the content.

With static sites the web server merely needs to serve static HTML to an end user. This has profound performance benefits; static sites walk whilst dynamic sites crawl. Above this, the ability to track all site changes in a Git repository adds better control when it comes to collaborative editing.

With static sites there is no CMS, no database; just HTML. No need to worry about patching PHP or plugins with insecure database queries.

Clearly static sites can’t do everything, namely anything that’s dynamic; though you can utilise JavaScript APIs to add some dynamic functionality if that's a route you want to pursue.

Chicago: National Prtg. & Engr. Co.

For this tutorial, I will be using Jekyll as a static static site generator. Jekyll works by taking a bunch of markdown files and outputting the HTML necessary for a static blog.

There is a great list of generators on a site called StaticGen, including a static site generator written in Go called Hugo. I’m using Jekyll here due to the integration with GitHub Pages.

If you want to host a JavaScript app or a simple static site, just skip this step.

Assuming you have a Ruby version greater than 2.0.0, then you can just install Jekyll by running:

gem install jekyll

For this example I’m going to be creating a blog about about plants using Jekyll. To create the blog simply run:

jekyll new plants

which will output something like:

New jekyll site installed in /Users/junade/plants.

From here I can cd into the plants directory and serve the blog on my local computer as follows:

cd plants
jekyll serve

A web server will spin up and you’ll find some useful information in your terminal prompt. You should be able to access the site from localhost at port 4000 in your browser: http://127.0.0.1:4000

Whilst this server is running you can update your site; you will find some useful variables to edit in _config.yml. To add a new post or edit the existing demo, simply add a new markdown file in the _posts directory.

Whilst ordinarily you would need to generate your site’s HTML using jekyll build then upload it to the web server of your choice, GitHub allows for raw Jekyll projects to be uploaded to its service; it will handle the building and serving of the HTML itself from the Jekyll project.

There is a Ruby Gem for Jekyll sites that ensures they are rendered the same way locally as they do when they are hosted on GitHub Pages if you’re interested.

Create a GitHub repository which contains the files of the site we want to serve (such as our Jekyll source or our HTML). As my GitHub username is IcyApril, I can create a repository called icyapril.github.io. Be sure that the repository name is all lowercase.

In this repository, whatever is in the main branch ends up published on icyapril.github.io.

If you haven’t already, let’s initiate a repository where our site files are:

git init
git add -A
git commit -m “Initial commit.”

We can now push files to our host by adding the origin as GitHub; make sure the URL of the origin is customised to be your own repository:

git remote add origin git@github.com:IcyApril/icyapril.github.io.git
git push -u origin master

You should now see your site when you visit [username].github.io.

I’ll assume you have registered a domain and added it to your CloudFlare account.In order to for GitHub to accept traffic from this domain, we need to create a CNAME file in our repository which contains the hostname to accept traffic for.

The following rules apply:

• If the CNAME file contains example.com, then www.example.com will redirect to example.com.

• If the CNAME file contains www.example.com, then example.com will redirect to www.example.com.

In the Git repository we created in the previous section, let’s add a CNAME file to that repository and commit our changes:

echo "www.ju.je" > CNAME
git add -A
git commit -m “Added CNAME file.”
git push origin master

We can add the records to point our DNS records to our GitHub Pages account (we can use a CNAME at the root thanks to CNAME Flattening:

You can find the most up-to-date IP Addresses from the GitHub Pages documentation.

Unfortunately GitHub Pages doesn’t yet support SSL on GitHub Pages for custom domains which would ordinarily rule out using HTTP/2. Whilst the HTTP/2 specification (RFC 7540) allows for HTTP/2 over plain-text HTTP/2, all popular browsers require HTTP/2 to run on top of Transport Layer Security; meaning HTTP/2 only being able to run over HTTPS is the de-facto standard.

Fortunately, CloudFlare’s Universal SSL option allows us provide a signed SSL certificate to site visitors. This allows us to gain the performance benefits of HTTP/2 and potentially improve search engine rankings.

In the Crypto tab of your CloudFlare site you should ensure your SSL mode is set to Full but not Full (Strict):

We can now add a Page Rule to enforce HTTPS, as you add other Page Rules make sure this is the primary Page Rule:

We can also create a Page Rule to ensure that non-www is redirected to www securely when using HTTPS:

Enabling HTTP Strict Transport Security (HSTS) will help ensure that your visitors have to communicate to your site over HTTPS, by telling browsers that they should always communicate over encrypted HTTPS. Be careful if you choose to set this though, it may render your site inaccessible if you wish to choose to ever turn HTTPS off.

CloudFlare has a “Cache Everything” option in Page Rules. For static sites, it allows your HTML to be cached and served directly from CloudFlare's CDN.

When deploying your site you can use the Purge Cache option in the Cache tab on CloudFlare to remove the cached version of the static pages. If you’re using a Continuous Integration system to deploy your site, you can use our API to clear the cache programmatically.

WillMcC on WikiMedia

Firstly a word on security. If you are deploying a JavaScript app which communicates with remote APIs, be sure not to use this for sensitive data submissions. As GitHub themselves put it: “GitHub Pages sites shouldn't be used for sensitive transactions like sending passwords or credit card numbers.” Also bear in mind your website source files are publicly accessible in a Git repository, so be extra careful about what you put there.

There are some things we can’t do; GitHub Pages doesn’t let us set custom headers, which unfortunately means we can’t do HTTP/2 Server Push right now.

GitHub Pages, CloudFlare and a static site generator combine to create fast, secure, free hosting for static sites.

At CloudFlare, we are always looking for better ways to secure the data we’re entrusted with. This means hardening our system against outside threats such as hackers, but it also means protecting against insider threats. According to a recent Verizon report, insider threats account for around 14% of data breaches in 2013. While we perform background checks and carefully screen team members, we also implement technical barriers to protect the data with which we are entrusted.

One good information security practice is known as the “two-man rule.” It comes from military history, where a nuclear missile couldn’t be launched unless two people agreed and turned their launch keys simultaneously. This requirement was introduced in order to prevent one individual from accidentally (or intentionally) starting World War III.

To prevent the risk of rogue employees misusing sensitive data we built a service in Go to enforce the two-person rule. We call the service Red October after the famous scene from “The Hunt for Red October.” In line with our philosophy on security software, we are open sourcing the technology so you can use it in your own organization (here’s a link to the public Github repo). If you are interested in the nitty-gritty details, read on.

Red October is a cryptographically-secure implementation of the two-person rule to protect sensitive data. From a technical perspective, Red October is a software-based encryption and decryption server. The server can be used to encrypt a payload in such a way that no one individual can decrypt it. The encryption of the payload is cryptographically tied to the credentials of the authorized users.

Authorized persons can delegate their credentials to the server for a period of time. The server can decrypt any previously-encrypted payloads as long as the appropriate number of people have delegated their credentials to the server.

This architecture allows Red October to act as a convenient decryption service. Other systems, including CloudFlare’s build system, can use it for decryption and users can delegate their credentials to the server via a simple web interface. All communication with Red October is encrypted with TLS, ensuring that passwords are not sent in the clear.

Setting up a Red October server is simple; all it requires is a locally-readable path and an SSL key pair. After that, all control is handled remotely through a set of JSON-based APIs.

Red October is backed by a database of accounts stored on disk in a portable password vault. The server never stores the account password there, only a salted hash of the password for each account. For each user, the server creates an RSA key pair and encrypts the private key with a key derived from the password and a randomly generated salt using a secure derivation function.

Any administrator can encrypt any piece of data with the encrypt API. This request takes a list of users and the minimum number of users needed to decrypt it. The server returns a somewhat larger piece of data that contains an encrypted version of this data. The encrypted data can then be stored elsewhere.

This data can later be decrypted with the decrypt API, but only if enough people have delegated their credentials to the server. The delegation API lets a user grant permission to a server to use their credentials for a limited amount of time and a limited number of uses.

Red October was designed from cryptographic first principles, combining trusted and understood algorithms in known ways. CloudFlare is also opening the source of the server to allow others to analyze its design.

Red October is based on combinatorial techniques and trusted cryptographic primitives. We investigated using complicated secret primitives like Shamir's sharing scheme, but we found that a simpler combinatorial approach based on primitives from Go's standard library was preferable to implementing a mathematical algorithm from scratch. Red October uses 128-bit AES, 2048-bit RSA and scrypt as its cryptographic primitives.

Each user is assigned a unique, randomly-generated RSA key pair when creating an account on a Red October server. The private key is encrypted with a password key derived from the user’s password and salt using scrypt. The public key is stored unencrypted in the vault with the encrypted private key.

When asked to encrypt a piece of data, the server generates a random 128-bit AES key. This key is used to encrypt the data. For each user that is allowed to decrypt the data, a user-specific key encryption key is chosen. For each unique pair of users, the data key is doubly encrypted, once with the key encryption key of each user. The key encryption keys are then encrypted with the public RSA key associated with their account. The encrypted data, the set of doubly-encrypted data keys, and the RSA-encrypted key encryption keys are all bundled together and returned. The encrypted data is never stored on the server.

Delegating credentials to the server

When a user delegates their key to the server, they submit their username and password over TLS using the delegate JSON API. For each account, the password is verified against the salted hash. If the password is correct, a password key is derived from the password and used to decrypt the user’s RSA private key. This key is now “Live” for the length of time and number of decryptions chosen by the user.

Decrypting data

To decrypt a file, the server validates that the requesting user is an administrator and has the correct password. If two users of the list of valid users have delegated their keys, then decryption can occur. First the RSA private key is used to decrypt the key encryption key for these two users, then the key encryption keys are used to decrypt the doubly encrypted data key, which is then used to decrypt the data.

Some other key points:

1. Cryptographic security. The Red October server does not have the ability to decrypt user keys without their password. This prevents someone with access to the vault from decrypting data.

2. Password flexibility. Passwords can be changed without changing the encryption of a given file. Key encryption keys ensure that password changes are decoupled from data encryption keys.

The version of Red October we are releasing to GitHub is in beta. It is licensed under the 3-clause BSD license. We plan to continue to release our improvements to the open source community. Here is the project on GitHub: Red October.

Writing the server in Go allowed us to design the different components of this server in a modular way. Our hope is this modularity will make it easy for anyone to build in support for different authentication methods that are not based on passwords (for example, TLS client certificates, time-based one-time-passwords) and new core cryptographic primitives (for example, elliptic curve cryptography).

CloudFlare is always looking to improve the state of security on the Internet. It is important to us to share our advances with the world and contribute back to the community. See the CloudFlare GitHub page for the list of our open source projects and initiatives.

And, of course, we make use of open source tools such as gcc, make, the Go programming language, Lua, python, Perl, and PHP, and projects like Sentry, Kibana, and nagios. And, naturally, we use Linux.

It would take a while to write down all the software that we use to build CloudFlare, but all that software has one thing in common: it's open source or free software. Our stack consists of either software we've built ourselves or an open source project (which we've sometimes forked).

It's probably obvious to most readers why we use open source software: it's reliable, it's easy to modify and it's easy to maintain. But there's another benefit that should not be overlooked: using and working on open source software brings a great deal of job satisfaction for programmers and it helps us hire the best.

We encourage our programmers to release changes they've made to open source software and to release projects through the CloudFlare GitHub page.

At GitHub you'll find projects such as golog (a high-performance Go logger), lua-cmsgpack (an implementation of MessagePack for Lua), a Python-based CNAME flattener, and a macro language for systemtap, amongst others.

You'll also find the ngx_lua module which embeds Lua in nginx. That's not something CloudFlare initially wrote, but we make such extensive use of it that we hired Yichun Zhang. He continues to work full-time on it while at CloudFlare.

And, if you've ever delved into the internals of nginx, you'll know another CloudFlare employee, Piotr Sikora, who recently added the ability to set keys for TLS Session Tickets to nginx.

So, at CloudFlare, open source can get you a job, be your job or, at least, be a significant part of your job.

Where appropriate (i.e. where we think we make the biggest impact and get something we need) we've sponsored external open source projects and paid for improvements that all can use.

We make wide use of the excellent LuaJIT project and after much profiling by engineers we discovered areas where more JITing would improve our performance. Rather than do the work ourselves we sponsored the LuaJIT project. These speedups will be appearing in LuaJIT 2.1 when it is released.

Of course, it would be easy for us to use open source software, make modifications and not release them. None of the licenses for the software we use force us to release our modifications. But we prefer to give back and not just because of karma.

There are two big advantages to releasing modifications we've made to existing projects: the many eyeballs effect and reducing fork cost.

The first, many eyeballs, is common to any open source project: the more people look at code, the better it gets. And that applies equally to code written by a core team of developers and code written by outsiders. When we contribute changes we've made, others look at the changes and improve them.

For example, back in 2012 we contributed an improvement to Go's log/syslog module. This year that work was improved.

And the cost of maintaining a fork provides useful economic pressure making releasing our modifications make sense. It's cheaper for us to release than to maintain a fork and merge as the core of a project changes.

But, what about CloudFlare's secret sauce?

Our strong bias is to open source everything we've built. When we don't, it's usually because it's highly specific to us and/or because the support cost is high. Ultimately, we'd like to open source all our major components so that they can be used to build a faster, safer, smarter web.

Many of our smaller components are really glue code that don't make any sense to open source as they are so specific to our implementation of the overall system.

We don't believe that there is any chunk of code so clever that it gives us a long term competitive advantage. Instead, our advantage comes from the network we've built, the data we collect on making the web faster and safer, and, most importantly, the people we're able to attract.

A commitment to open source builds trust in the community which helps us continue to build our service and attract the best people.

In fact, the best way to get hired at CloudFlare is to make good contributions to open source projects we find interesting and useful. Contributions like that often speak more loudly than a resume.