This blog post is part of Crypto Week 2019.

Trust on the Internet is underpinned by the Public Key Infrastructure (PKI). PKI grants servers the ability to securely serve websites by issuing digital certificates, providing the foundation for encrypted and authentic communication.

Certificates make HTTPS encryption possible by using the public key in the certificate to verify server identity. HTTPS is especially important for websites that transmit sensitive data, such as banking credentials or private messages. Thankfully, modern browsers, such as Google Chrome, flag websites not secured using HTTPS by marking them “Not secure,” allowing users to be more security conscious of the websites they visit.

This blog post introduces a new, free tool Cloudflare offers to CAs so they can further secure certificate issuance. But before we dive in too deep, let’s talk about where certificates come from.

Certificate Authorities (CAs) are the institutions responsible for issuing certificates.

When issuing a certificate for any given domain, they use Domain Control Validation (DCV) to verify that the entity requesting a certificate for the domain is the legitimate owner of the domain. With DCV the domain owner:

1. creates a DNS resource record for a domain;

2. uploads a document to the web server located at that domain; OR

3. proves ownership of the domain’s administrative email account.

The DCV process prevents adversaries from obtaining private-key and certificate pairs for domains not owned by the requestor.  

Preventing adversaries from acquiring this pair is critical: if an incorrectly issued certificate and private-key pair wind up in an adversary’s hands, they could pose as the victim’s domain and serve sensitive HTTPS traffic. This violates our existing trust of the Internet, and compromises private data on a potentially massive scale.

For example, an adversary that tricks a CA into mis-issuing a certificate for gmail.com could then perform TLS handshakes while pretending to be Google, and exfiltrate cookies and login information to gain access to the victim’s Gmail account. The risks of certificate mis-issuance are clearly severe.

To prevent attacks like this, CAs only issue a certificate after performing DCV. One way of validating domain ownership is through HTTP validation, done by uploading a text file to a specific HTTP endpoint on the webserver they want to secure.  Another DCV method is done using email verification, where an email with a validation code link is sent to the administrative contact for the domain.

Suppose Alice buys the domain name aliceswonderland.com and wants to get a dedicated certificate for this domain. Alice chooses to use Let’s Encrypt as their certificate authority. First, Alice must generate their own private key and create a certificate signing request (CSR). She sends the CSR to Let’s Encrypt, but the CA won’t issue a certificate for that CSR and private key until they know Alice owns aliceswonderland.com. Alice can then choose to prove that she owns this domain through HTTP validation.

When Let’s Encrypt performs DCV over HTTP, they require Alice to place a randomly named file in the /.well-known/acme-challenge path for her website. The CA must retrieve the text file by sending an HTTP GET request to http://aliceswonderland.com/.well-known/acme-challenge/<random_filename>. An expected value must be present on this endpoint for DCV to succeed.

For HTTP validation, Alice would upload a file to http://aliceswonderland.com/.well-known/acme-challenge/YnV0dHNz

where the body contains:

curl http://aliceswonderland.com/.well-known/acme-challenge/YnV0dHNz

GET /.well-known/acme-challenge/YnV0dHNz
Host: aliceswonderland.com

HTTP/1.1 200 OK
Content-Type: application/octet-stream

YnV0dHNz.TEST_CLIENT_KEY

The CA instructs them to use the Base64 token YnV0dHNz. TEST_CLIENT_KEY in an account-linked key that only the certificate requestor and the CA know. The CA uses this field combination to verify that the certificate requestor actually owns the domain. Afterwards, Alice can get her certificate for her website!

Another way users can validate domain ownership is to add a DNS TXT record containing a verification string or token from the CA to their domain’s resource records. For example, here’s a domain for an enterprise validating itself towards Google:

$ dig TXT aliceswonderland.com
aliceswonderland.com.	 28 IN TXT "google-site-verification=COanvvo4CIfihirYW6C0jGMUt2zogbE_lC6YBsfvV-U"

Here, Alice chooses to create a TXT DNS resource record with a specific token value. A Google CA can verify the presence of this token to validate that Alice actually owns her website.

Certificate issuance is required for servers to securely communicate with clients. This is why it’s so important that the process responsible for issuing certificates is also secure. Unfortunately, this is not always the case.

Researchers at Princeton University recently discovered that common DCV methods are vulnerable to attacks executed by network-level adversaries. If Border Gateway Protocol (BGP) is the “postal service” of the Internet responsible for delivering data through the most efficient routes, then Autonomous Systems (AS) are individual post office branches that represent an Internet network run by a single organization. Sometimes network-level adversaries advertise false routes over BGP to steal traffic, especially if that traffic contains something important, like a domain’s certificate.

Bamboozling Certificate Authorities with BGP highlights five types of attacks that can be orchestrated during the DCV process to obtain a certificate for a domain the adversary does not own. After implementing these attacks, the authors were able to (ethically) obtain certificates for domains they did not own from the top five CAs: Let’s Encrypt, GoDaddy, Comodo, Symantec, and GlobalSign. But how did they do it?

There are two main approaches to attacking the DCV process with BGP hijacking:

1. Sub-Prefix Attack

2. Equally-Specific-Prefix Attack

These attacks create a vulnerability when an adversary sends a certificate signing request for a victim’s domain to a CA. When the CA verifies the network resources using an HTTP GET  request (as discussed earlier), the adversary then uses BGP attacks to hijack traffic to the victim’s domain in a way that the CA’s request is rerouted to the adversary and not the domain owner. To understand how these attacks are conducted, we first need to do a little bit of math.

Every device on the Internet uses an IP (Internet Protocol) address as a numerical identifier. IPv6 addresses contain 128 bits and follow a slash notation to indicate the size of the prefix. So, in the network address 2001:DB8:1000::/48, “/48” refers to how many bits the network contains. This means that there are 80 bits left that contain the host addresses, for a total of 10,240 host addresses. The smaller the prefix number, the more host addresses remain in the network. With this knowledge, let’s jump into the attacks!

When BGP announces a route, the router always prefers to follow the more specific route. So if 2001:DB8::/32 and 2001:DB8:1000::/48 are advertised, the router will use the latter as it is the more specific prefix. This becomes a problem when an adversary makes a BGP announcement to a specific IP address while using the victim’s domain IP address. Let’s say the IP address for our victim, leagueofentropy.com, is 2001:DB8:1000::1 and announced as 2001:DB8::/32. If an adversary announces the prefix 2001:DB8:1000::/48, then they will capture the victim’s traffic, launching a sub-prefix hijack attack.

In an IPv4 attack, such as the attack during April 2018, this was /24 and /23 announcements, with the more specific /24 being announced by the nefarious entity. In IPv6, this could be a /48 and /47 announcement. In both scenarios, /24's and /48's are the smallest blocks allowed to be routed globally. In the diagram below, /47 is Texas and /48 is the more specific Austin, Texas. The new (but nefarious) routes overrode the existing routes for portions of the Internet. The attacker then ran a nefarious DNS server on the normal IP addresses with DNS records pointing at some new nefarious web server instead of the existing server. This attracted the traffic destined for the victim’s domain within the area the nefarious routes were being propagated. The reason this attack was successful was because a more specific prefix is always preferred by the receiving routers.

In the last attack, the adversary was able to hijack traffic by offering a more specific announcement, but what if the victim’s prefix is /48 and a sub-prefix attack is not viable? In this case, an attacker would launch an equally-specific-prefix hijack, where the attacker announces the same prefix as the victim. This means that the AS chooses the preferred route between the victim and the adversary’s announcements based on properties like path length. This attack only ever intercepts a portion of the traffic.

There are more advanced attacks that are covered in more depth in the paper. They are fundamentally similar attacks but are more stealthy.

Once an attacker has successfully obtained a bogus certificate for a domain that they do not own, they can perform a convincing attack where they pose as the victim’s domain and are able to decrypt and intercept the victim’s TLS traffic. The ability to decrypt the TLS traffic allows the adversary to completely Monster-in-the-Middle (MITM) encrypted TLS traffic and reroute Internet traffic destined for the victim’s domain to the adversary. To increase the stealthiness of the attack, the adversary will continue to forward traffic through the victim’s domain to perform the attack in an undetected manner.

Another way an adversary can gain control of a domain is by spoofing DNS traffic by using a source IP address that belongs to a DNS nameserver. Because anyone can modify their packets’ outbound IP addresses, an adversary can fake the IP address of any DNS nameserver involved in resolving the victim’s domain, and impersonate a nameserver when responding to a CA.

This attack is more sophisticated than simply spamming a CA with falsified DNS responses. Because each DNS query has its own randomized query identifiers and source port, a fake DNS response must match the DNS query’s identifiers to be convincing. Because these query identifiers are random, making a spoofed response with the correct identifiers is extremely difficult.

Adversaries can fragment User Datagram Protocol (UDP) DNS packets so that identifying DNS response information (like the random DNS query identifier) is delivered in one packet, while the actual answer section follows in another packet. This way, the adversary spoofs the DNS response to a legitimate DNS query.

Say an adversary wants to get a mis-issued certificate for victim.com by forcing packet fragmentation and spoofing DNS validation. The adversary sends a DNS nameserver for victim.com a ICMP "fragmentation needed" packet with a small Maximum Transmission Unit, or maximum byte size. This gets the nameserver to start fragmenting DNS responses. When the CA sends a DNS query to a nameserver for victim.com asking for victim.com’s TXT records, the nameserver will fragment the response into the two packets described above: the first contains the query ID and source port, which the adversary cannot spoof, and the second one contains the answer section, which the adversary can spoof. The adversary can continually send a spoofed answer to the CA throughout the DNS validation process, in the hopes of sliding their spoofed answer in before the CA receives the real answer from the nameserver.

In doing so, the answer section of a DNS response (the important part!) can be falsified, and an adversary can trick a CA into mis-issuing a certificate.

At first glance, one could think a Certificate Transparency log could expose a mis-issued certificate and allow a CA to quickly revoke it. CT logs, however, can take up to 24 hours to include newly issued certificates, and certificate revocation can be inconsistently followed among different browsers. We need a solution that allows CAs to proactively prevent this attacks, not retroactively address them.

We’re excited to announce that Cloudflare provides CAs a free API to leverage our global network to perform DCV from multiple vantage points around the world. This API bolsters the DCV process against BGP hijacking and off-path DNS attacks.

Given that Cloudflare runs 175+ datacenters around the world, we are in a unique position to perform DCV from multiple vantage points. Each datacenter has a unique path to DNS nameservers or HTTP endpoints, which means that successful hijacking of a BGP route can only affect a subset of DCV requests, further hampering BGP hijacks. And since we use RPKI, we actually sign and verify BGP routes.

This DCV checker additionally protects CAs against off-path, DNS spoofing attacks. An additional feature that we built into the service that helps protect against off-path attackers is DNS query source IP randomization. By making the source IP unpredictable to the attacker, it becomes more challenging to spoof the second fragment of the forged DNS response to the DCV validation agent.

By comparing multiple DCV results collected over multiple paths, our DCV API makes it virtually impossible for an adversary to mislead a CA into thinking they own a domain when they actually don’t. CAs can use our tool to ensure that they only issue certificates to rightful domain owners.

Our multipath DCV checker consists of two services:

1. DCV agents responsible for performing DCV out of a specific datacenter, and

2. a DCV orchestrator that handles multipath DCV requests from CAs and dispatches them to a subset of DCV agents.

When a CA wants to ensure that DCV occurred without being intercepted, it can send a request to our API specifying the type of DCV to perform and its parameters.

The DCV orchestrator then forwards each request to a random subset of over 20 DCV agents in different datacenters. Each DCV agent performs the DCV request and forwards the result to the DCV orchestrator, which aggregates what each agent observed and returns it to the CA.

This approach can also be generalized to performing multipath queries over DNS records, like Certificate Authority Authorization (CAA) records. CAA records authorize CAs to issue certificates for a domain, so spoofing them to trick unauthorized CAs into issuing certificates is another attack vector that multipath observation prevents.

As we were developing our multipath checker, we were in contact with the Princeton research group that introduced the proof-of-concept (PoC) of certificate mis-issuance through BGP hijacking attacks. Prateek Mittal, coauthor of the Bamboozling Certificate Authorities with BGP paper, wrote:

“Our analysis shows that domain validation from multiple vantage points significantly mitigates the impact of localized BGP attacks. We recommend that all certificate authorities adopt this approach to enhance web security. A particularly attractive feature of Cloudflare’s implementation of this defense is that Cloudflare has access to a vast number of vantage points on the Internet, which significantly enhances the robustness of domain control validation.”

Our DCV checker follows our belief that trust on the Internet must be distributed, and vetted through third-party analysis (like that provided by Cloudflare) to ensure consistency and security. This tool joins our pre-existing Certificate Transparency monitor as a set of services CAs are welcome to use in improving the accountability of certificate issuance.

Building our multipath DCV checker also allowed us to dogfood multiple Cloudflare products.

The DCV orchestrator as a simple fetcher and aggregator was a fantastic candidate for Cloudflare Workers. We implemented the orchestrator in TypeScript using this post as a guide, and created a typed, reliable orchestrator service that was easy to deploy and iterate on. Hooray that we don’t have to maintain our own dcv-orchestrator  server!

We use Argo Tunnel to allow Cloudflare Workers to contact DCV agents. Argo Tunnel allows us to easily and securely expose our DCV agents to the Workers environment. Since Cloudflare has approximately 175 datacenters running DCV agents, we expose many services through Argo Tunnel, and have had the opportunity to load test Argo Tunnel as a power user with a wide variety of origins. Argo Tunnel readily handled this influx of new origins!

If you and/or your organization are interested in trying our DCV checker, email dcv@cloudflare.com and let us know! We’d love to hear more about how multipath querying and validation bolsters the security of your certificate issuance.

As a new class of BGP and IP spoofing attacks threaten to undermine PKI fundamentals, it’s important that website owners advocate for multipath validation when they are issued certificates. We encourage all CAs to use multipath validation, whether it is Cloudflare’s or their own. Jacob Hoffman-Andrews, Tech Lead, Let’s Encrypt, wrote:

“BGP hijacking is one of the big challenges the web PKI still needs to solve, and we think multipath validation can be part of the solution. We’re testing out our own implementation and we encourage other CAs to pursue multipath as well”

Hopefully in the future, website owners will look at multipath validation support when selecting a CA.

We’re also happy to announce CFSSL’s new home: cfssl.org. From there you can try out CFSSL’s user interface, download binaries, and test some of its features.

Licensing: Public Domain

This 2013 National Security Agency (NSA) slide describing how data from Google’s internal network was collected by intelligence agencies was eye-opening—and shocking—to many technology companies. The idea that an attacker could read messages passed between services wasn’t technically groundbreaking, but it did reveal a security flaw in the way many distributed systems were designed. Many companies only encrypted the data to the border of their datacenter, not inside. The slide showed that private physical networks are being subverted to extract data passing through them. And just because a network has a security perimeter, it doesn’t mean that data can be safely sent between applications unencrypted inside that perimeter. In short: treat your own network as hostile.

This mentality helped shape CloudFlare’s philosophy for securing internal services and resulted in a simple rule:

Services should only communicate with each other using encrypted and mutually authenticated protocols.

With this in mind we started tackling the harder problem of how to manage the encryption keys for these services. To tackle the issue of service-to-service encryption, we built our own public key infrastructure using CFSSL. Much of the new features we’re introducing in this post came about from our effort to make this system robust.

We have also made an effort to use standards-compliant and interoperable technology. By incorporating support for certificate transparency, OSCP, and CRL, the standards used by the public Internet can now be used in your private infrastructure. Now, on to the new features.

CFSSL now has a full-featured TLS endpoint scanner.

Just because a server uses encryption, it doesn’t mean that it is secure. There have been a series of vulnerabilities in TLS that only affect some configurations. To keep your server and its visitors protected against the nearly monthly new attacks you need to pick the right configuration. Staying secure requires testing your configuration against the latest vulnerabilities, and keeping your configuration updated against new threats.

History of vulnerabilities in SSL/TLS

The gold standard for testing a website’s TLS configuration is Ivan Ristić’s SSL Labs. It provides a simple letter grade for your site’s configuration (sites using CloudFlare get an A, by the way, and A+ if you enable HSTS). The drawback of SSL Labs is that it only works on public websites: you can’t use it for internal services. At CloudFlare, we needed an easy way to check the configuration of our services as well as our customers’ origins (which are typically not publicly accessible).

To solve this, CloudFlare added functionality to CFSSL to scan a TLS endpoint to evaluate how securely it’s configured. With it, we are able to check the configuration of internal services and protected customer origins for the following configuration issues:

• IPv4/IPv6 connectivity

• Certificate validity (expiration, trust chain, hostnames, etc.)

• Supported cipher suites and algorithms

• Session resumption

• Revoked certificates

Each scan provides a grade of "Good" or "Bad". CFSSL Scan can also be used to scan entire IP ranges or lists of hosts. It can be used either as a CLI or as API-driven server.

Using the CLI is a simple command:

$ cfssl scan cloudflare.com
{
  "Connectivity": {
	"DNSLookup": {
	  "grade": "Good",
	  "output": [
		"198.41.215.162",
		"198.41.214.162",
		"2400:cb00:2048:1::c629:d6a2",
		"2400:cb00:2048:1::c629:d7a2"
	  ]
	},
	"TCPDial": {
	  "grade": "Good"
	},
	"TLSDial": {
	  "grade": "Good"
	}
  },
  "PKI": {
	"ChainExpiration": {
	  "grade": "Good",
	  "output": "2016-11-30T23:59:59Z"
	},
	"ChainValidation": {
	  "grade": "Warning",
	  "output": [
		"Certificate for COMODO ECC Extended Validation Secure Server CA is valid for too long"
	  ]
	},
	"MultipleCerts": {
	  "grade": "Good"
	}
  },
  "TLSHandshake": {
	"CertsByCiphers": {
	  "grade": "Good",
	  "output": {
		"TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384": "SHA256WithRSA",
		"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_3DES_EDE_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_CBC_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_128_GCM_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_CBC_SHA": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_CBC_SHA256": "SHA256WithRSA",
		"TLS_RSA_WITH_AES_256_GCM_SHA384": "SHA256WithRSA"
	  }
	}
  }
}

CFSSL Scan also accessible as part of the new CFSSL UI.

An important design pattern in security engineering is secure defaults. Developers want to write secure software and aren’t always security experts, let alone crypto gurus. The two trickiest parts of deploying an application that speaks TLS are:

1. Configuration

2. Key management

Simplicity is the key to empowering developers to use encryption in their services. We created the CFSSL Transport package to make these two tasks easy for our Go developers.

Transport is a Go library that takes regular HTTP or TCP connections, and transparently turns them into encrypted connections. Transport handles all the sticky points so that the developer doesn’t have to. This includes creating a private key, getting a certificate for it using a CFSSL CA, renewing certificates before they expire, and choosing the correct cryptographic parameters. If you’re writing a service in Go, you no longer need to know how PKI works.

Certificate Issuance with CFSSL CA

Not only does the Transport handle setting up and rotating certificates, it automatically checks to make sure the services your service are connecting to are using a valid certificate, including checking for revocation (more on that later).

OCSP Check with CFSSL CA

Internal CAs can be used to set up coarse-grained authorization between services. For example, if you have both an API server and a database, you can set up a dedicated CA for each of them. In the example below, the API server CA is in orange and the DB CA is in red. You can then configure the DB to only trust connections from the API server and vice versa. This type of setup can provide a baseline level of authorization enforcement for your applications. The transport package lets you automate the setup of these mutually-authenticated connections. This type of setup is covered in a previous blog post.

Once you have a CFSSL CA (or a multi-root CA) up and running, it just takes a few lines of code to start using TLS in your Go application. Just swap your standard net.Dial or net.Listen/Accept with transport.Dial and transport.Listen/Accept.

Before:

conn, err := net.Dial("tcp", addr)
if err != nil {
	// handle error
}

After (configuration file location stored in the conf variable):

var id = new(core.Identity)
data, err := ioutil.ReadFile(conf)
if err != nil {
	// handle error
}
err = json.Unmarshal(data, id)
if err != nil {
	// handle error
}

    // Renew 5 minutes before expiry
tr, err := transport.New(5 * time.Minute, id)
if err != nil {
	// handle error
}
conn, err := transport.Dial(addr, tr)
if err != nil {
	// handle error
}

You can start playing around with the transport package with some examples from Github:

https://github.com/cloudflare/cfssl/tree/master/transport/example

CC Creative Commons Attribution-Share Alike 3.0 Unported

One of the nice things about CFSSL is that you can easily spin it up inside your infrastructure and have a certificate authority. One of the risks of running a PKI is infrastructure compromise. If the private key material for a certificate falls into the wrong hands, there need to be mechanisms so that the rest of the system knows to no longer trust that certificate.

The first step in knowing which certificates are trusted is knowing which certificates have been issued. To solve this, we added the ability to keep track, in a persistent database, of which certificates have been issued and the subset of those that have been revoked. We are big fans of PostgreSQL, so we built a database backend for CFSSL in PostgreSQL, but other backends like MySQL are in development. You can now set up CFSSL to use a certificate database with very little work, and we leveraged that integration to create an automated revocation system.

The two standard mechanisms for signaling that a certificate is no longer trusted are certificate revocation lists (CRLs) and the online certificate status protocol (OCSP). CFSSL now fully supports both of these mechanisms.

A CRL is simply a list of revoked certificate serial numbers. It covers all certificates issued by a CA that have not expired, and is digitally signed by the CA’s private key. When a client obtains a certificate, it can simply look at this list to check to see if the certificate has been revoked. CRL files can grow quite a bit if a lot of certificates are revoked, and can therefore cause some scalability issues. We saw this after Heartbleed, when we revoked a large number of customer certificates at once.

Partly to combat these scalability issues, OCSP was introduced. OCSP provides on-demand answers about the revocation status of a given certificate. An OCSP responder is a service that returns signed answers to the question "is this certificate revoked?". The response is either "Yes" or "No". Each response is signed by the CA and has a validity period so the client knows how long to cache the response.

CFSSL now has an OCSP responder service that can be configured to run in a distributed way, without access to the CA. There are also OCSP management tools in CFSSL to automatically populate the data for the OCSP responder and keep it fresh using the certificate database.

In CFSSL, you can now programmatically create CRLs and OCSP responses for certificates issued by your CA. Using standards-compatible revocation mechanisms allows these certificates to be shared outside of our infrastructure and to work with most software that implements TLS.

Another exciting new PKI standard is Certificate Transparency (CT). It helps provide (as the name implies) transparency into the workings of a certificate authority by providing an append-only log of issued certificates.

You can think of CT as a public ledger of all certificates issued. Any certificate on the list (even if issued for use on a private network) is made public and can be checked to see if it was issued according to the rules of the CA/Browser forum. If you encounter a certificate that is not on the ledger, then it may have been created fraudulently. Google Chrome currently requires all Extended Validation certificates used by websites to be in the CT log.

CFSSL now allows you to submit certificates to a CT log at issuance time and automatically embed the proof that it has been logged into the certificate. Running a CT log inside your internal infrastructure is a nice way to audit your CA and catch mis-issuances.

CFSSL is great for software deployments, as you can spin it up anywhere and run it on any platform that Go supports. You can even use our convenient Dockerfiles to deploy it in a containerized environment. However, in some situations (like running a publicly-trusted CA), keeping a private key in software is not secure enough. For these situations, hardware-based protection is needed.

The industry standard protocol for working with cryptographic hardware is called PKCS#11. With help from Richard Barnes of Mozilla and others we were able to add support for PKCS#11 into CFSSL. This feature is can be enabled in programs that use the signer/local package and the pkcs11key package. We also have plans to add command line support using the PKCS#11 URI specification. If you have a PKCS#11 interface to your HSM, certificate creation using that key is fully supported by the cfssl/signer package. Power users including Let’s Encrypt use CFSSL to run their publicly trusted CA while keeping the private key in a FIPS 140-2 certified HSM.

Open source is hard. Different people have different needs, and as a project maintainer you have to be respectful of these needs while honoring the spirit of the project. CloudFlare’s needs for CFSSL are not identical to the needs of its other users. We have attempted to strike a balance between building a tool for our own specific use cases and building a great general-purpose toolkit for PKI/TLS. We are grateful to the open source community for their valuable contributions to this project and are proud to be part of the tradition of free and open source software.

I’d like to thank one of the largest contributors to CFSSL over the last year: the Let’s Encrypt project. They have contributed code reviews and useful features while integrating CFSSL into Boulder, the software that manages their certificate authority. I’d also like to thank the Open Academy participants from Cornell and UCSD who worked on the project for a semester, and everyone else who helped contribute to this release. The core CFSSL team is Kyle Isom, Zi Lin, Jacob Haven, and Nick Sullivan.

CC BY-SA 2.0 image by kris krüg

Great security architecture requires a defense system with multiple layers of protection. As CloudFlare’s services have grown, the need to secure application-to-application communication has grown with it. As a result, we needed a simple and maintainable way to ensure that all communication between CloudFlare’s internal services stay protected, so we built one based on known and reliable protocols.

Our system of trust is based on a Public Key Infrastructure (PKI) using internally-hosted Certificate Authorities (CAs). In this post we will describe how we built our PKI, how we use it internally, and how to run your own with our open source software. This is a long post with lots of information, grab a coffee!

Most reasonably complex modern web services are not made up of one monolithic application. In order to handle complex operations, an application is often split up into multiple “services” that handle different portions of the business logic or data storage. Each of these services may live on different machines or even in different datacenters.

The software stacks of large service providers are made up of many components. For CloudFlare, this includes a web application to handle user actions, a main database to maintain DNS records and user rules, a data pipeline to distribute these rules to the edge network, services for caching, a log transport pipeline, data analysis services and much, much more.

Some service-to-service communication can happen within a machine, some within a datacenter and some across a broader network like the Internet. Managing which communications should use which type of network in our evolving services is not a simple task. A single accidental configuration change could result in messages that are supposed to never leave a machine going through an untrusted connection on the Internet. The system should be designed so that these messages are secure even if they go over the wrong network.

One approach to mitigate the risks posed by attackers is to encrypt and authenticate data in transit. Our approach is to require that all new services use an encrypted protocol, Transport Layer Security (TLS), to keep inter-service communication protected. It was a natural choice: TLS is the “S” in HTTPS and is the foundation of the encrypted web. Furthermore, modern web services and APIs have embraced TLS as the de-facto standard for application layer encryption. It works seamlessly with RESTful services, is supported in Kyoto Tycoon, PostgreSQL, and the Go standard library.

As we have described in previous blog posts, unauthenticated encryption can be foiled by on-path attackers. Encryption without authentication does not protect data in transit. For connections to be safe, each party needs to prove their identity to the other. Public key cryptography provides many mechanisms for trust, including PGP’s “web of trust” and HTTPS’s public key infrastructure (PKI) model. We chose the PKI model because of ease of use and deployment. TLS with PKI provides trusted communication.

Trust is the bedrock of secure communication. For two parties to securely exchange information, they need to know that the other party is legitimate. PKI provides just that: a mechanism for trusting identities online.

The tools that enable this are digital certificates and public key cryptography. A certificate lets a website or service prove its identity. Practically speaking, a certificate is a file with some identity information about the owner, a public key, and a signature from a certificate authority (CA). Each certificate also contains a public key. Each public key has an associated private key, which is kept securely under the certificate owner’s control. The private key can be used to create digital signatures that can be verified by the associated public key.

A certificate typically contains

• Information about the organization that the certificate is issued to

• A public key

• Information about the organization that issued the certificate

• The rights granted by the issuer

• The validity period for the certificate

• Which hostnames the certificate is valid for

• The allowed uses (client authentication, server authentication)

• A digital signature by the issuer certificate’s private key

A certificate is a powerful tool for proving your identity online. The owner of a certificate can digitally sign data, and a verifier can use the public key from the certificate to verify it. The fact that the certificate is itself digitally signed by a third party CA means that if the verifier trusts the third party, they have assurances that the certificate is legitimate. The CA can give a certificate certain rights, such as a period of time in which the identity of the certificate should be trusted.

Sometimes certificates are signed by what’s called an intermediate CA, which is itself signed by a different CA. In this case, a certificate verifier can follow the chain until they find a certificate that they trust — the root.

This chain of trust model can be very useful for the CA. It allows the root certificate’s private key to be kept offline and only used for signing intermediate certificates. Intermediate CA certificates can be shorter lived and be used to sign endpoint certificates on demand. Shorter-lived online intermediates are easier to manage and revoke if compromised.

This is the same system used for HTTPS on the web. For example, cloudflare.com has a certificate signed by Comodo’s Intermediate certificate, which is in turn signed by the Comodo offline root. Browsers trust the Comodo root, and therefore also trust the intermediate and web site certificate.

This model works for the web because browsers only trust a small set of certificate authorities, who each have stringent requirements to only issue certificates after validating the ownership of a website. For internal services that are not accessed via browsers, there is *no need * to go through a third party certificate authority. Trusted certificates do not have to be from Globalsign, Comodo, Verisign or any other CA — they can be from a CA you operate yourself.

The most painful part of getting a certificate for a website is going through the process of obtaining it. For websites, we eliminated this pain by launching Universal SSL. The most painful part of running a CA is the administration. When we decided to build our internal CA, we sought to make both obtaining certificates and operating the CA painless and even fun.

The software we are using is CFSSL, CloudFlare’s open source PKI toolkit. This tool was open sourced last year and has all the capabilities needed to run a certificate authority. Although CFSSL was built for an internal CA, it’s robust enough to be use a publicly trusted CA; in fact, the Let’s Encrypt project is using CFSSL as a core part of their CA infrastructure.

To run a CA, you need the CA certificate and corresponding private key. This private key is extremely sensitive. Any person who knows the value of the key can act as the CA and issue certificates. Browser-trusted certificate authorities are required to keep their private keys inside of specialized hardware known as Hardware Security Modules (HSMs). The requirements for protecting private keys for corporate infrastructures are not necessarily as stringent, so we provided several mechanisms to protect keys.

CFSSL supports three different modes of protection for private keys:

1. Hardware Security Module (HSM)CFSSL allows the CA server to use an HSM to compute digital signatures. Most HSMs use an interface called PKCS#11 to interact with them, and CFSSL natively supports this interface. Using an HSM ensures that private keys do not live in memory and it provides tamper protection against physical adversaries.

2. Red OctoberPrivate Keys can be encrypted using Red October (another open source CloudFlare project). A key protected with Red October can only be decrypted with the permission of multiple key owners. In order to use CFSSL with a Red October key, the key owners need to authorize the use of the private key. This ensures that the CA key is never unencrypted on disk, in source control, or in configuration management. Note: Red October support in CFSSL is experimental and subject to change.

3. PlaintextCFSSL accepts plain unencrypted private keys. This works well when the private key is generated on the machine running CFSSL or by another program. If the machine that is running the CA is highly secure, this mode is a compromise between security, cost, and usability. This is also useful in development mode, allowing users to test changes to their infrastructure designs.

Next I’ll show you how to quickly configure an internal CA using plaintext private keys. Note: The following expects CFSSL to be installed.

To start, you need some information about what metadata you want to include in your certificate. Start by creating a file called csr_ca.json containing this basic information (feel free to fill in your own organization's details):

{
  "CN": "My Awesome CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
    "names": [
       {
         "C": "US",
         "L": "San Francisco",
         "O": "My Awesome Company",
         "OU": "CA Services",
         "ST": "California"
       }
    ]
}

With this you can create your CA with a single call"$ cfssl gencert -initca csr_ca.json | cfssljson -bare ca

This generates the two files you need to start your CA:

• ca-key.pem: your private key

• ca.pem: your certificate

• ca.csr: your certificate signing request (needed to get your CA cross-signed by another CA)

The key and certificate are the bare minimum you need to start running a CA.

Once the CA certificate and key are created, the CA software needs to know what kind of certificates it will issue. This is determined in the CFSSL configuration file’s signing policy section.

Here’s an example of a simple policy for a CA that can issue certificates that are valid for a year and can be used for server authentication.

config_ca.json
{
  "signing": {
    "default": {
      "auth_key": "key1",
      "expiry": "8760h",
      "usages": [
         "signing",
         "key encipherment",
         "server auth"
       ]
     }
  },
  "auth_keys": {
    "key1": {
      "key": <16 byte hex API key here>,
      "type": "standard"
    }
  }
}

We also added an authentication key to this signing policy. This authentication key should be randomly generated and kept private. The API key is a basic authentication mechanism that prevents unauthorized parties from requesting certificates. There are several other features you can use for the CA (subject name allowlisting, etc.), CFSSL documentation for more information.

To run the service, call$ cfssl serve -ca-key ca-key.pem -ca ca.pem -config config_ca.json

This opens up a CA service listening on port 8888.

Certificate authorities do not just create certificates out of a private key and thin air, they need a public key and metadata to populate the certificate’s data fields. This information is typically communicated to a CA via a certificate signing request (CSR).

A CSR is very similar in structure to a certificate. The CSR contains:

• Information about the organization that is requesting the certificate

• A public key

• A digital signature by the requestor’s private key

Given a CSR, a certificate authority can create a certificate. First, it verifies that the requestor has control over the associated private key. It does this by checking the CSR’s signature. Then the CA will check to see if the requesting party should be given a certificate and which domains/IPs it should be valid for. This can be done with a database lookup or through a registration authority. If everything checks out, the CA uses its private key to create and sign the certificate to send back to the requestor.

Let’s say you have CFSSL set up as CA as described above and it’s running on a server called “ca1.mysite.com” with an authentication API key. How do you get this CA to issue a certificate? CFSSL provides two commands to help with that: gencert and sign. They are available as JSON API endpoints or command line options.

The gencert command will automatically handle the whole certificate generation process. It will create your private key, generate a CSR, send the CSR to the CA to be signed and return your signed certificate.

There are two configuration files needed for this. One to tell the local CFSSL client where the CA is and how to authenticate the request, and a CSR configuration used to populate the CSR.

Here’s an example for creating a certificate for a generic database server, db1.mysite.com.

csr_client.json

{
  "hosts": [
    	"db1.mysite.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "San Francisco",
      "O": “My Awesome Company",
      "OU": “Data Services",
      "ST": "California"
    }
  ]
}

config_client.json

{
  "signing": {
    "default": {
      "auth_key": "key1",
      "remote": "caserver"
    }
  },
  "auth_keys": {
    "key1": {
    "key": <16 byte hex API key here>,
    "type": "standard"
    }
  },
  "remotes": {
    "caserver": “ca1.mysite.com:8888"
  }
}

With these two configuration files set, you can create your certificate.$ cfssl gencert -config config_client.json csr_client.json | cfssljson -bare db

This results in three files:

• db-key.pem: your private key

• db.pem: your certificate

• db.csr: your CSR

The CSR can be resubmitted to the CA to be signed again at any point with the sign command

$ cfssl sign -config config_client.json db.csr | cfssljson -bare db-new

resulting in:

• db-new.pem: your re-signed certificate

These two commands let you easily and conveniently set up a private PKI. As a startup or a growing business moving to a service-oriented or even a microservice architecture, having a PKI can be very convenient. Next we’ll describe how CloudFlare set up its own internal PKI to help make its services encrypted by default.

So now you have a complex set of services that can all speak TLS, and a central certificate authority server that can issue certificates. What’s next? Getting certificates and keys for the applications. There are several ways to do this including a centrally managed way and a distributed way.

One way to create certificates and keys for your applications is to create them all on a central provisioning server and then send them out to each of the servers. In this model, a central server creates certificates and keys and sends them over a secure tunnel to the application servers.

This model creates a sort of chicken and egg problem. How do you transport private keys if you need those private keys to encrypt your transport?

A distributed key management model fits better with the way modern services are typically deployed and run. The trick is creating the private keys directly in the application server. At install or run time, a service creates its own private key and sends a request to a certificate authority to issue a certificate. This can be repeated on demand if the current certificate is close to expiring.

For example, many companies are starting to use Docker, or other lightweight container technologies, to encapsulate and run individual services. Under load, services can be scaled up by automatically running new containers. In a centralized distribution model, certificates and keys for each container need to be created before the containers are deployed.

In the centralized distribution model, the provisioning service needs to create and manage a key and certificate for each service. Keeping this sort of central database in a complex and dynamic topology seems like the wrong approach. It would be preferable if the CA itself was stateless and generated a set of logs.

The idea that keys need to be transported to their destination instead of generated locally is also troubling. Transporting private keys introduces an unnecessary risk to the system. When a new service comes into being, it should generate its key locally and request a certificate for use.

Internal services need to trust each other. Browsers validate website certificates by checking the signature on the certificate and checking the hostname against the list of Subject Alternative Names (SANs) in the certificate. This type of explicit check is useful, but it has a record of not working as expected. Another way for services to trust each other is an implicit check based on per-service CAs.

The idea is simple: use a different CA for each set of services. Issue all database certificates from a database CA and all API servers from an API server CA.

When setting these services up to talk to each other with mutual TLS authentication configure the trust stores as follows:

• API server only trusts DB CA

• DB only trusts API CA

This approach is less fine-grained than an explicit check against a SAN, but it is more robust and easier to manage on the CA policy side. With an implicit trust system in place, you can guarantee that services of type A can only communicate with services of type B.

The following diagram describes how two applications can trust each other with mutually authenticated TLS.

In this diagram, the API server trusts the DB CA (in red). It will therefore only accept certificates that are signed by the DB CA (i.e. with a red ribbon). Conversely, the database server only accepts certificates with a signature from the API CA (orange ribbon). To establish a trusted connection, each party sends a key share to the other, signed with their certificate’s private key. The key shares are combined to create a session key, with which both parties can encrypt their data. The chain of verification from key share to certificate to CA assure that the other party is authentic.

Establishing a mutually authenticated trusted tunnel between services prevents attackers from accessing or modifying data in transit and causing havoc on your services. With a strong PKI in place, every service can communicate securely and confidentially over any network, even the Internet.

Internal PKIs are very flexible and can be used to issue certificates to third parties who are integrating with your network. For example, CloudFlare has a service called Railgun that can be used to optimize connections between CloudFlare and an origin server. Communication between Railgun and CloudFlare is done over an encrypted and authenticated channel using certificates from a CloudFlare certificate authority.

This ensures that data is secure in transit. When a new Railgun instance is set up on the origin server, it creates a private key and sends a CSR to CloudFlare, which then issues the appropriate certificate. The Railgun server keeps the private key in memory and deletes it when it shuts down, preventing other services from getting access to it.

This model works great for not only Railgun, but several other initiatives at CloudFlare such as the Origin CA and Keyless SSL.

Securing data at the application level is an important step for securing a distributed systems architecture, but is only truly effective with a strong PKI in place.

Portal uses open-source software and services to take inexpensive, commodity travel routers and turn them into powerful security devices. Since this is pretty far from CloudFlare's core business, it warrants a brief digression into why we support projects like this.

Computer security was for a very long time only of interest to hobbyists, academics, and obscure government agencies. Cryptography was an interesting offshoot of number theory, a foundational but very abstract part of mathematics, and many of the early infrastructure components of the Internet didn't include security at all -- there was an assumption that anyone who could gain access would be responsible and well-intentioned, a consequence of the academic origins; after all, why would they want to break or steal things which were freely available.

Before the "cambrian explosion" of commercial computer security, there was still a lot of great security research -- it was just done by academics and by individuals in the "security community", who were motivated by a desire to understand how things worked, and to make tools because they loved the technology and wanted to solve their own problems. Some of the most interesting and powerful security tools available today trace their origins to rather humble open-source, hobbyist, or academic beginnings -- PGP, Tor, OTR, various forms of electronic cash, and many others. Many of today's most respected people in computer security entered the field during this period, out of personal curiosity or academic interest.

While CloudFlare is an eager participant in the commercial security world (we're the easiest and fastest way to set up TLS for any website, and we provide edge security and performance to millions of sites, including some of the largest sites on the Internet -- both with free service and paid services in various tiers), we are also very aware of the broad and deep foundation of security tools and research on which we're built.

CloudFlare makes extensive use of open source software, such as the Nginx web server, community collections of Web Application Firewall (WAF) rules originally generated by OWASP, and powerful cryptographic algorithms developed in academia and implemented by open source efforts such as the OpenSSL Project.

Where possible, CloudFlare also contributes back to the community in those areas. We contribute bugfixes and new functionality back to open source packages, and we employ developers who in their spare time make additional contributions to open source software. CloudFlare's GitHub Open Source page is a great collection of many of our contributions to open source.

One of our biggest contributions to date has been CFSSL, CloudFlare's PKI toolkit. We're constantly hearing from various projects and companies how CFSSL has been helpful to them -- one of the most exciting being the Let's Encrypt community Certificate Authority project. Nick Sullivan has written in the CloudFlare blog announcing CFSSL, and exciting things are continuing to happen with that software.

CloudFlare, like many other companies in computer security, makes other contributions to the security community. One of the most interesting is that we, like some other companies, values having employees participate in the security community in a variety of ways. Encouraging side projects independent of work -- research, finding new vulnerabilities and responsibly disclosing them, creating new tools, participating in conferences or working groups, running tutorials, and being active in standards bodies -- sometimes doesn't have a direct connection to the company's products, but contributes to a vibrant security ecosystem. There are often unforeseen benefits of these collaborations -- learning about new tools, finding great engineers -- we're actively hiring for a variety of roles -- and many others.

Marc and I are grateful to CloudFlare for the time to work on this open source tool and to present it to the world, and we're looking forward to presenting at RSA.

Last September, CloudFlare unveiled Universal SSL, enabling HTTPS support for all sites by default. All sites using CloudFlare now support strong cryptography from the browser to CloudFlare’s servers. One of the most popular requests for Universal SSL was to make it easier to encrypt the other half of the connection: from CloudFlare to the origin server.

Until today, encryption from CloudFlare to the origin required the purchase of a trusted certificate from a third party. The certificate purchasing process can be tedious and sometimes costly. To remedy this, CloudFlare has created a new Origin CA service in which we provide free limited-function certificates to customer origin servers.

Today we are excited to announce the public beta of this service, providing full encryption of all data from the browser to the origin, for free.

CloudFlare offers three modes for HTTPS: Flexible, Full and Strict. In Flexible mode, traffic from browsers to CloudFlare is encrypted, but traffic from CloudFlare to a site's origin server is not. In Full and Strict modes, traffic between CloudFlare and the origin server is encrypted. Strict mode adds validation of the origin server’s certificate. We strongly encourage customers to select Strict mode for their websites to ensure their visitors get the strongest data security possible.

As we previously discussed, sites on CloudFlare’s Free plan default to Flexible SSL mode. To take advantage of our Strict SSL mode it’s necessary to install a certificate on the origin server, which until now required them to buy one from a third party. Now customers can get that certificate directly from CloudFlare, for free.

This certificate is only used to protect the traffic between the origin server and CloudFlare; it is never presented to browsers. For now you should only use it behind orange-clouded sites on CloudFlare.

If you are a CloudFlare customer and want to sign up for the beta, just send an email to origin-ca-beta@cloudflare.com with the following:

• A certificate signing request (CSR)

• The domain name of the orange-clouded zone you want to install the certificate on

The first ten brave beta customers will get a shiny new certificate to install on their web server. Note: do not send your private key to CloudFlare, only the CSR is needed.

Update: The beta is full! Thanks to those who are participating.

In order to grant certificates to customer origins, CloudFlare had to create its own Certificate Authority. This consists of a set of processes and systems to validate certificate requests and create new certificates. For the Origin CA, CloudFlare created a private key and certificate for the specific purpose of signing certificates for origin servers.

The certificate authority software we use is CFSSL, our open source PKI toolkit written in Go. It allows us to validate CSRs and use them to create new certificates for sites. These certificates are signed with our certificate authority private key, and validated when CloudFlare connects to the origin in Strict SSL mode.

In collaboration with other members of the industry (such as Richard Barnes from the Let's Encrypt project), we have updated CFSSL with several new features that help make it a viable certificate authority tool. These include PKCS#11 support, which makes it possible for CFSSL to use a Hardware Security Module (HSM) to store private keys and OCSP support, which lets CFSSL answer questions about the revocation status of a certificate.

CAs are supposed to only give certificates to sites that own the domain(s) listed in the certificate. Domain validation is usually done in one of three ways:

• Putting a challenge in the DNS zone

• Putting a challenge into a meta-tag of an HTML page hosted on the domain

• Sending an email challenge to the domain registrant from the WhoIs DB

Since CloudFlare is both a content delivery network and a DNS provider, both DNS and HTML validation can be done by CloudFlare on behalf of the site. If your site is on CloudFlare and orange-clouded, we will give you a certificate for your site.

The CloudFlare Origin CA is currently not trusted by browsers, so these certificates should not be used on sites that are not behind CloudFlare. To issue certificates that are trusted by browsers, we would have to convince a publicly trusted certificate authority to cross-sign our CA certificate. This is not necessary in this case since it is CloudFlare that determines which certificates we trusted and the Origin CA is on our list.

The certificate signing request (CSR) is the standard mechanism for obtaining a certificate from a certificate authority. It contains a public key, some metadata such as which domain it is for and is digitally signed by a private key. It lets CloudFlare know that you own the private key.

CFSSL is not only a tool that can be used for running a CA, but it can be used to create CSRs too. Following these instructions will get you a private key and a CSR to submit to a certificate authority.

1) Install Go:

https://golang.org/doc/install

2) Install CFSSL

$ go get github.com/cloudflare/cfssl/cmd/...

3) Create a CSR template

Use the following template for csr.json and replace “mysite.com” with your site’s domain name and names with your company's information.

csr.json:

{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts": [
    “mysite.com"
  ],
 "CN": "mysite.com",
  "names": [
    {
      "C": "US",
      "L": "San Francisco",
      "ST": "California",
      "O": "My Company, Inc.",
      "OU": "My Company’s IT Department"
    }
  ]
}

4) Create the certificate

$ cfssl genkey csr.json | cfssljson -bare site

This creates two files:

• site.csr: your CSR

• site-key.pem: your private key

If CFSSL is not working for you, here are some more resources for creating CSRs:

• Comodo

• GlobalSign

• Digicert

In the future we plan on releasing tools to make certificate generation even easier and more automatic.

CloudFlare’s Flexible SSL mode is the default for CloudFlare sites on the Free plan. Flexible SSL mode means that traffic from browsers to CloudFlare will be encrypted, but traffic from CloudFlare to a site's origin server will not be. To take advantage of our Full and Strict SSL mode—which encrypts the connection between CloudFlare and the origin server—it’s necessary to install a certificate on the origin server.

We made Universal SSL free so that everyone can use modern, strong encryption tools to protect their web traffic. More encrypted traffic helps build a safer, better Internet. In keeping with CloudFlare’s goal to help build a better Internet, we have some tips on how to upgrade your site from Flexible SSL to Full or Strict SSL.

Dealing with Certificate Authorities (CAs) can be frustrating, and the process of obtaining a certificate can be time consuming. In the meantime, you can get started by installing a self-signed certificate on your origin server. This allows CloudFlare to encrypt the communication with the origin, protecting the communication against passive surveillance, but not against active attackers.

Our handy CSR guide for CFSSL describes how to generate a self-signed certificate. Using OpenSSL to create it is another option.

Once you have created a self-signed certificate and private key, you can install them on your origin server. Digicert has a guide for installing a certificate that covers the most popular server software.

Keep in mind that a self-signed certificate is not signed by a trusted CA. This means that you can change your SSL setting from Flexible SSL to Full, but not Full (strict). Full SSL won’t be able to provide authentication, but it will make sure the connection to the origin is encrypted and protected from passive snoopers.

Most CAs offer low-cost or even free certificates. A popular CA that offers free SSL certificates is StartSSL. Buying and installing a trusted certificate on your origin server is currently the simplest way to enable Strict SSL on your site.

To enable TLS on your server, you need both a certificate and a corresponding private key. The first step in obtaining a certificate from a CA is creating a Certificate Signing Request (CSR). A CSR contains your public key and a proof that you have the associated private key. The CA will verify it and give you back a certificate that you install on your web server. We put together a guide to creating a private key and CSR with CloudFlare’s CFSSL tool that you can use, or alternatively, there’s always OpenSSL.

Once you have a certificate installed on your origin server, you can change your SSL setting from Flexible to Full (strict) and have the added benefit of an authenticated and encrypted connection to your origin server.

Soon you will be able to send your CSR to CloudFlare to get a certificate instantaneously, speeding up the certificate acquisition process. This process will be like that of a regular CA, but much faster. These certificates aren't yet trusted by browsers, but will be trusted by CloudFlare, allowing the back end connection to be both encrypted and authenticated. This also protects your site if one of the publicly trusted certificate authorities is compromised by attackers and used to issue illegitimate certificates.

We’re also investigating the possibility of adding a feature called Certificate Pinning. Certificate Pinning would allow you to tell CloudFlare exactly which certificate to trust for your origin. This would allow customers to use hosting services that don’t allow custom certificates to have the benefit of a fully encrypted tunnel, or to simply use a self-signed certificate and get the benefit of both authentication and encryption.

Creating a certificate bundle is a common pain point for website operators, and doing it right is important for website security AND speed (CloudFlare does both). Getting the correct bundle together is a black art, and can quickly become a debugging nightmare if it's not done correctly. We wrote CFSSL to make bundling easier. By picking the right chain of certificates, CFSSL solves the balancing act between performance, security, and compatibility.

Staying true to our promise to make the web fast and safe, we’re open sourcing CFSSL. We believe CFSSL will be useful for anyone building a site using HTTPS—from website owners to large software-as-a-service companies. CFSSL is written in Go and available on the CloudFlare Github account. It can be used as a web service with a JSON API, and as a handy command line tool.

CFSSL is the result of real-world expertise about how the TLS ecosystem on the Web works that you can gain by working at CloudFlare’s scale. These are hard-won lessons, applied in code. The rest of this blog post serves as an under-the-hood look at how and why CFSSL works, and how you can use it as a certificate bundler or as a lightweight CA.

As you can see in the above image, the key is owned by CloudFlare, Inc., and the issuer is GlobalSign (a well-known CA). GlobalSign has issued a certificate named “GlobalSign Extended Validation CA - SHA256 - G2”; this G2 certificate is signed by another certificate called “GlobalSign Root CA - R2”. Note that the root certificate does not have an issuer—it is signed by its own private key. In other words, the root vouches for itself.

The reason your browser trusts a root certificate is because browsers have a list of root certificates that they implicitly trust, and when a site is trusted it will show the lock icon to the left of the web address (see image below). Certain certificates are on the list typically because they have been around for some time, and they belong to certificate authorities that have gone through a rigorous security audit. GlobalSign is one of these trusted authorities; therefore, its root certificate is in the list of trusted root certificates for nearly every browser.

SSL certificates form the core of trust on the web by assuring the identity of websites. This trust is built by digitally binding a cryptographic key to an organization’s identity. An SSL certificate will bind domain names to server names, and company names to locations. This ensures that if you go to your bank’s web site, for example, you know for sure it is your bank, and you are not giving out your information to a phishing scam.

A certificate is a file that contains a public key which is bound to a record of its owner. The mechanism that makes this binding trustworthy is called the Public Key Infrastructure and public-key cryptography is the glue that makes this possible.

A certificate is associated with a private key that corresponds to the certificate's public key, which is stored separately. The certificate comes with a digital signature from a trusted third-party called a certificate authority or CA. Let's examine the cloudflare.com certificate.

The length of intermediate certificates in a chain can vary, but chains will always have one leaf and one root.

Unfortunately, there is a catch for the owner of the leaf certificate: presenting only the leaf certificate to the browser is usually not enough. The intermediate certificates are not always known to the browser, requiring the website to include them with the leaf certificate. The list of certificates that the browser needs to validate a certificate is called a certificate bundle. It should contain all the certificates in the chain up to the first certificate known to the browser. In the case of the CloudFlare website, this bundle contains both the cloudflare.com certificate and the GlobalSign G2 intermediate.

CFSSL was written to make certificate bundling easier.

If you are running a website (or perhaps some other TLS-based service) and need to install a certificate, CFSSL can create the certificate bundle for you. Start with the following command:

$ cfssl bundle -cert mycert.crt

This will output a JSON blob containing the chain of certificates along with relevant information extracted from that chain. Alternatively, you can run the CFSSL service that responds to requests with a JSON API:

$ cfssl serve

This command opens up an HTTP service on localhost that accepts requests. To bundle using this API, send a POST request to this service, [http://localhost:8888/api/v1/cfssl/bundle](http://localhost:8888/api/v1/cfssl/bundle), using a JSON request such as:

{ "certificate": }

CloudFlare’s SSL service will return a JSON response of the form:

{ "result": {}, "success": true, "errors": [], "messages": [], }

(Developers take note: this response format is a preview of our upcoming CloudFlare API rewrite; with this API, we can use CFSSL as a service for certificate bundling and more—stay tuned.)

If you upload your certificate to CloudFlare, this is what is used to create a certificate bundle for your website.

To create a certificate bundle with CFSSL, you need to know which certificates are trusted by the browsers you hope to display content to. In a controlled corporate environment, this is usually easy since every browser is set up with the same configuration; however, it becomes more difficult when creating a bundle for the web.

Each browser has unique capabilities and configurations, and a certificate bundle that’s trusted in one browser might not be trusted in another; this can happen for several reasons:

1. Different browsers trust different root certificates.Some browsers trust more root certificates than others. For example, the NSS root store used in Firefox trusts 143 certificates; however, Windows 8.1 trusts 391 certificates. So a bundle with a chain that expects the browser to trust a certificate exclusive to the Windows root store will appear as untrusted in Firefox.

2. Older systems might have old root stores.Some browsers run on older systems that have not been updated to trust recently-created certificate authorities. For example, Android 2.2 and earlier don't trust the “GoDaddy Root Certificate Authority G2” because that root certificate was created after they were released.

3. Older systems don't support modern cryptography.Users on Windows XP SP2 and earlier can't validate certificates signed by certain intermediates. This includes the “DigiCert SHA2 High Assurance Server CA” because the hash function SHA-2 used in this certificate was standardised after Windows XP was released.

In order to provide maximum compatibility between SSL chains and browsers, you often have to pick a different certificate chain than the one originally provided to you by your CA. This alternate chain might contain a different set of intermediates that are signed by a different root certificate. Alternate chains can be troublesome. They tend to contain a longer list of certificates than the default chain from the CA, and longer chains cause site connections to be slower. This lag is because the web server needs to send more certificates (i.e. more data) to the browser, and the browser has to spend time verifying more certificates on its end. Picking the right chain can be tricky.

CFSSL helps pick the right certificate chain, selecting for performance, security, and compatibility.

The chain of trust works by following the keys used to sign certificates, and there can be multiple chains of trust for the same keys.

In this diagram, all of the intermediates have the same public key and represent the same identity (GlobalSign's intermediate signing certificate), and they are signed by the GlobalSign root key; however, the issuance date for each chain and signature type are different.

For some outdated browsers, the current GlobalSign root certificate is not trusted, so GlobalSign used an older CA (GlobalSign nv-sa) to sign their root certificate. This allows older browsers to trust certificates issued by GlobalSign.

Each of these are valid chains for some browsers, but each has drawbacks:

• CloudFlare leaf → GlobalSign SHA2 Intermediate.This chain is trusted by any browser that trusts the GS root G2 and supports SHA2 (i.e., this chain would not be trusted by a browser on Windows XP SP2).

• CloudFlare leaf → 2012 GlobalSign Intermediate → GS Root G2.This chain is trusted by any browser that trusts GS Root G2, but does not benefit from the stronger cryptographic properties of the SHA-2 hashing algorithm.

• CloudFlare leaf → 2012 GlobalSign Intermediate → GS cross-signed root.This chain is trusted by any browser that trusts the GlobalSign nv-sa root, but uses the older (and weaker) GlobalSign nv-sa root certificate.

This last chain is the most common because it’s trusted by more browsers; however, it’s also the longest, and has weaker cryptography.

CFSSL can create either the most common or the optimal bundle, and if you need help deciding, the documentation that ships with CFSSL has tips on choosing.

If you decide to create the optimal bundle, there’s a chance it might not work in some browsers; however, CFSSL is configured to let you know specifically which browsers it will not work with. For example, it will warn the user about bundles that contains certificates signed with advanced hash functions such as SHA2; they can be problematic for certain operating systems like Windows XP SP2.

All paid CloudFlare accounts receive HTTPS service automatically; to make this happen, our engineers do a lot of work behind scenes.

CloudFlare must obtain a certificate for each site on the service, and we want these certificates to be valid on as many browsers as possible; getting a certificate that works in multiple browsers is a challenge, but CFSSL makes things easier.

To start, a key-pair is created for the customer through a call to CFSSL's genkey API with the required certificate information:

{ "CN": "customer.com", "hosts": [ "customer.com", "www.customer.com" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "US", "L": "San Francisco", "O": "Customer", "OU": "Website", "ST": "California" } ] }

Next, the CFSSL service responds with a key and a Certificate Signature Request (CSR). The CSR is sent to the CA to verify a site’s identity. Once the CA has validated the CSR, and the identity of the site, they then return a certificate signed by one of their issuing intermediates.

Once we have a certificate for a site (whether created by our CA partner or uploaded by the customer), we send the certificate to CFSSL's bundler to create a certificate bundle for the customer.

By default, we pick the most common chain. For customers who prefer performance to compatibility, we’ll soon introduce an option to rebundle certificates for optimum chains.

CFSSL is not only a tool for bundling a certificate, it can also be used as a CA. This is possible because it covers the basic features of certificate creation including creating a private key, building a certificate signature request, and signing certificates.

You can create a brand new CA with CFSSL using the -initca option. As we saw above, this takes a JSON blob with the certificate request, and creates a new CA key and certificate:

$ cfssl gencert -initca ca_csr.json

This will return a CA certificate and CA key that is valid for signing. The CA is meant to function as an internal tool for creating certificates. CFSSL should be used behind a middle layer that processes incoming requests, and ensures they conform to policy; we use it behind the CloudFlare site as an internal service.

Here’s an example of signing a certificate with CFSSL on the command line:

$ cfssl sign -ca-key=test-key.pem -ca=test-cert.pem www.example.com example.csr

Alternatively, a POST request containing the CSR and hostname can be sent to the CFSSL service via the /api/v1/cfssl/sign endpoint. To run the service call the serve command as follows:

$ cfssl serve -address=localhost -port=8888 -ca-key=test-key.pem -ca=test-cert.pem

If you already have a CFSSL instance running (in this case on localhost, but it can be anywhere), you can automate certificate creation with the gencert command’s -remote option. For example, if CFSSL is running on localhost, running the following gives you a private key and a certificate signed by the CA:

$ cfssl gencert -remote="localhost:8888" www.example.com csr.json

At CloudFlare we use CFSSL heavily for Railgun and other internal services. Special thanks go to Kyle Isom, Zi Lin, Sebastien Pahl, and Dane Knecht, and the rest of the CloudFlare team for making this release possible.

CloudFlare’s mission is to help build a better web, and we believe that improved certificate bundling and certificate authority tools are a step in the right direction. Encrypted sites create a safer, more private internet for everyone; by open-sourcing CFSSL, we’re making this process easier.

We have big plans for the CA part of this tool. Currently, we run CFSSL on secure locked-down machines, but plan to add stronger hardware security. Adding stronger hardware security involves integrating CFSSL with low-cost Trusted Platform Modules (TPMs). This ensures that private keys stay private, even in the event of a breach.

For additional information on CFSSL and our other open source projects, check out our Github page. We encourage users to file issues on Github as they find bugs.