In December 2022, Cloudflare joined the AS112 Project, which helps the Internet deal with misdirected DNS queries. In March 2023, we launched an AS112 statistics page on Radar, providing insight into traffic trends and query types for this misdirected traffic. Extending the basic analysis presented on that page, and building on the analysis of resolver data used for the Domains page, today we are excited to launch a dedicated DNS page on Cloudflare Radar to provide increased visibility into aggregate traffic and usage trends seen across 1.1.1.1 resolver traffic. In addition to looking at global, location, and autonomous system (ASN) traffic trends, we are also providing perspectives on protocol usage, query and response characteristics, and DNSSEC usage.

The traffic analyzed for this new page may come from users that have manually configured their devices or local routers to use 1.1.1.1 as a resolver, ISPs that set 1.1.1.1 as the default resolver for their subscribers, ISPs that use 1.1.1.1 as a resolver upstream from their own, or users that have installed Cloudflare’s 1.1.1.1/WARP app on their device. The traffic analysis is based on anonymised DNS query logs, in accordance with Cloudflare’s Privacy Policy, as well as our 1.1.1.1 Public DNS Resolver privacy commitments.

Below, we walk through the sections of Radar’s new DNS page, reviewing the included graphs and the importance of the metrics they present. The data and trends shown within these graphs will vary based on the location or network that the aggregated queries originate from, as well as on the selected time frame.

As with many Radar metrics, the DNS page leads with traffic trends, showing normalized query volume at a worldwide level (default), or from the selected location or autonomous system (ASN). Similar to other Radar traffic-based graphs, the time period shown can be adjusted using the date picker, and for the default selections (last 24 hours, last 7 days, etc.), a comparison with traffic seen over the previous period is also plotted.

For location-level views (such as Latvia, in the example below), a table showing the top five ASNs by query volume is displayed alongside the graph. Showing the network’s share of queries from the selected location, the table provides insights into the providers whose users are generating the most traffic to 1.1.1.1.

When a country/region is selected, in addition to showing an aggregate traffic graph for that location, we also show query volumes for the country code top level domain (ccTLD) associated with that country. The graph includes a line showing worldwide query volume for that ccTLD, as well as a line showing the query volume based on queries from the associated location. Anguilla’s ccTLD is .ai, and is a popular choice among the growing universe of AI-focused companies. While most locations see a gap between the worldwide and “local” query volume for their ccTLD, Anguilla’s is rather significant — as the graph below illustrates, this size of the gap is driven by both the popularity of the ccTLD and Anguilla’s comparatively small user base. (Traffic for .ai domains from Anguilla is shown by the dark blue line at the bottom of the graph.) Similarly, sizable gaps are seen with other “popular” ccTLDs as well, such as .io (British Indian Ocean Territory), .fm (Federated States of Micronesia), and .co (Colombia). A higher “local” ccTLD query volume in other locations results in smaller gaps when compared to the worldwide query volume.

Depending on the strength of the signal (that is, the volume of traffic) from a given location or ASN, this data can also be used to corroborate reported Internet outages or shutdowns, or reported blocking of 1.1.1.1. For example, the graph below illustrates the result of Venezuelan provider CANTV reportedly blocking access to 1.1.1.1 for its subscribers. A comparable drop is visible for Supercable, another Venezuelan provider that also reportedly blocked access to Cloudflare’s resolver around the same time.

Individual domain pages (like the one for cloudflare.com, for example) have long had a choropleth map and accompanying table showing the popularity of the domain by location, based on the share of DNS queries for that domain from each location. A similar view is included at the bottom of the worldwide overview page, based on the share of total global queries to 1.1.1.1 from each location.

While traffic trends are always interesting and important to track, analysis of the characteristics of queries to 1.1.1.1 and the associated responses can provide insights into the adoption of underlying transport protocols, record type popularity, cacheability, and security.

Published in November 1987, RFC 1035 notes that “The Internet supports name server access using TCP [RFC-793] on server port 53 (decimal) as well as datagram access using UDP [RFC-768] on UDP port 53 (decimal).” Over the subsequent three-plus decades, UDP has been the primary transport protocol for DNS queries, falling back to TCP for a limited number of use cases, such as when the response is too big to fit in a single UDP packet. However, as privacy has become a significantly greater concern, encrypted queries have been made possible through the specification of DNS over TLS (DoT) in 2016 and DNS over HTTPS (DoH) in 2018. Cloudflare’s 1.1.1.1 resolver has supported both of these privacy-preserving protocols since launch. The DNS transport protocol graph shows the distribution of queries to 1.1.1.1 over these four protocols. (Setting up 1.1.1.1 on your device or router uses DNS over UDP by default, although recent versions of Android support DoT and DoH. The 1.1.1.1 app uses DNS over HTTPS by default, and users can also configure their browsers to use DNS over HTTPS.)

Note that Cloudflare's resolver also services queries over DoH and Oblivious DoH (ODoH) for Mozilla and other large platforms, but this traffic is not currently included in our analysis. As such, DoH adoption is under-represented in this graph.

Aggregated worldwide between February 19 - February 26, distribution of transport protocols was 86.6% for UDP, 9.6% for DoT, 2.0% for TCP, and 1.7% for DoH. However, in some locations, these ratios may shift if users are more privacy conscious. For example, the graph below shows the distribution for Egypt over the same time period. In that country, the UDP and TCP shares are significantly lower than the global level, while the DoT and DoH shares are significantly higher, suggesting that users there may be more concerned about the privacy of their DNS queries than the global average, or that there is a larger concentration of 1.1.1.1 users on Android devices who have set up 1.1.1.1 using DoT manually. (The 2024 Cloudflare Radar Year in Review found that Android had an 85% mobile device traffic share in Egypt, so mobile device usage in the country leans very heavily toward Android.)

RFC 1035 also defined a number of standard and Internet specific resource record types that return the associated information about the submitted query name. The most common record types are A and AAAA, which return the hostname’s IPv4 and IPv6 addresses respectively (assuming they exist). The DNS query type graph below shows that globally, these two record types comprise on the order of 80% of the queries received by 1.1.1.1. Among the others shown in the graph, HTTPS records can be used to signal HTTP/3 and HTTP/2 support, PTR records are used in reverse DNS records to look up a domain name based on a given IP address, and NS records indicate authoritative nameservers for a domain.

A response code is sent with each response from 1.1.1.1 to the client. Six possible values were originally defined in RFC 1035, with the list further extended in RFC 2136 and RFC 2671. NOERROR, as the name suggests, means that no error condition was encountered with the query. Others, such as NXDOMAIN, SERVFAIL, REFUSED, and NOTIMP define specific error conditions encountered when trying to resolve the requested query name. The response codes may be generated by 1.1.1.1 itself (like REFUSED) or may come from an upstream authoritative nameserver (like NXDOMAIN).

The DNS response code graph shown below highlights that the vast majority of queries seen globally do not encounter an error during the resolution process (NOERROR), and that when errors are encountered, most are NXDOMAIN (no such record). It is worth noting that NOERROR also includes empty responses, which occur when there are no records for the query name and query type, but there are records for the query name and some other query type.

With DNS being a first-step dependency for many other protocols, the amount of queries of particular types can be used to indirectly measure the adoption of those protocols. But to effectively measure adoption, we should also consider the fraction of those queries that are met with useful responses, which are represented with the DNS record adoption graphs.

The example below shows that queries for A records are met with a useful response nearly 88% of the time. As IPv4 is an established protocol, the remaining 12% are likely to be queries for valid hostnames that have no A records (e.g. email domains that only have MX records). But the same graph also shows that there’s still a significant adoption gap where IPv6 is concerned.

When Cloudflare’s DNS resolver gets a response back from an upstream authoritative nameserver, it caches it for a specified amount of time — more on that below. By caching these responses, it can more efficiently serve subsequent queries for the same name. The DNS cache hit ratio graph provides insight into how frequently responses are served from cache. At a global level, as seen below, over 80% of queries have a response that is already cached. These ratios will vary by location or ASN, as the query patterns differ across geographies and networks.

As noted in the preceding paragraph, when an authoritative nameserver sends a response back to 1.1.1.1, each record inside it includes information about how long it should be cached/considered valid for. This piece of information is known as the Time-To-Live (TTL) and, as a response may contain multiple records, the smallest of these TTLs (the “minimum” TTL) defines how long 1.1.1.1 can cache the entire response for. The TTLs on each response served from 1.1.1.1’s cache decrease towards zero as time passes, at which point 1.1.1.1 needs to go back to the authoritative nameserver. Hostnames with relatively low TTL values suggest that the records may be somewhat dynamic, possibly due to traffic management of the associated resources; longer TTL values suggest that the associated resources are more stable and expected to change infrequently.

The DNS minimum TTL graphs show the aggregate distribution of TTL values for five popular DNS record types, broken out across seven buckets ranging from under one minute to over one week. During the third week of February, for example, A and AAAA responses had a concentration of low TTLs, with over 80% below five minutes. In contrast, NS and MX responses were more concentrated across 15 minutes to one hour and one hour to one day. Because MX and NS records change infrequently, they are generally configured with higher TTLs. This allows them to be cached for longer periods in order to achieve faster DNS resolution.

DNS Security Extensions (DNSSEC) add an extra layer of authentication to DNS establishing the integrity and authenticity of a DNS response. This ensures subsequent HTTPS requests are not routed to a spoofed domain. When sending a query to 1.1.1.1, a DNS client can indicate that it is DNSSEC-aware by setting a specific flag (the “DO” bit) in the query, which lets our resolver know that it is OK to return DNSSEC data in the response. The DNSSEC client awareness graph breaks down the share of queries that 1.1.1.1 sees from clients that understand DNSSEC and can require validation of responses vs. those that don’t. (Note that by default, 1.1.1.1 tries to protect clients by always validating DNSSEC responses from authoritative nameservers and not forwarding invalid responses to clients, unless the client has explicitly told it not to by setting the “CD” (checking-disabled) bit in the query.)

Unfortunately, as the graph below shows, nearly 90% of the queries seen by Cloudflare’s resolver are made by clients that are not DNSSEC-aware. This broad lack of client awareness may be due to several factors. On the client side, DNSSEC is not enabled by default for most users, and enabling DNSSEC requires extra work, even for technically savvy and security conscious users. On the authoritative side, for domain owners, supporting DNSSEC requires extra operational maintenance and knowledge, and a mistake can cost your domain to disappear from the Internet, causing significant (including financial) issues.

The companion End-to-end security graph represents the fraction of DNS interactions that were protected from tampering, when considering the client’s DNSSEC capabilities and use of encryption (use of DoT or DoH). This shows an even greater imbalance at a global level, and highlights the importance of further adoption of encryption and DNSSEC.

For DNSSEC validation to occur, the query name being requested must be part of a DNSSEC-enabled domain, and the DNSSEC validation status graph represents the share of queries where that was the case under the Secure and Invalid labels. Queries for domains without DNSSEC are labeled as Insecure, and queries where DNSSEC validation was not applicable (such as various kinds of errors) fall under the Other label. Although nearly 93% of generic Top Level Domains (TLDs) and 65% of country code Top Level Domains (ccTLDs) are signed with DNSSEC (as of February 2025), the adoption rate across individual (child) domains lags significantly, as the graph below shows that over 80% of queries were labeled as Insecure.

DNS is a fundamental, foundational part of the Internet. While most Internet users don’t think of DNS beyond its role in translating easy-to-remember hostnames to IP addresses, there’s a lot going on to make even that happen, from privacy to performance to security. The new DNS page on Cloudflare Radar endeavors to provide visibility into what’s going on behind the scenes, at a global, national, and network level.

While the graphs shown above are taken from the DNS page, all the underlying data is available via the API and can be interactively explored in more detail across locations, networks, and time periods using Radar’s Data Explorer and AI Assistant. And as always, Radar and Data Assistant charts and graphs are downloadable for sharing, and embeddable for use in your own blog posts, websites, or dashboards.

If you share our DNS graphs on social media, be sure to tag us: @CloudflareRadar and @1111Resolver (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky). If you have questions or comments, you can reach out to us on social media, or contact us via email.

Cloudflare has been part of a multivendor, industry-wide effort to mitigate two critical DNSSEC vulnerabilities. These vulnerabilities exposed significant risks to critical infrastructures that provide DNS resolution services. Cloudflare provides DNS resolution for anyone to use for free with our public resolver 1.1.1.1 service. Mitigations for Cloudflare’s public resolver 1.1.1.1 service were applied before these vulnerabilities were disclosed publicly. Internal resolvers using unbound (open source software) were upgraded promptly after a new software version fixing these vulnerabilities was released.

All Cloudflare DNS infrastructure was protected from both of these vulnerabilities before they were disclosed and is safe today. These vulnerabilities do not affect our Authoritative DNS or DNS firewall products.

All major DNS software vendors have released new versions of their software. All other major DNS resolver providers have also applied appropriate mitigations. Please update your DNS resolver software immediately, if you haven’t done so already.

Domain name system (DNS) security extensions, commonly known as DNSSEC, are extensions to the DNS protocol that add authentication and integrity capabilities. DNSSEC uses cryptographic keys and signatures that allow DNS responses to be validated as authentic. DNSSEC protocol specifications have certain requirements that prioritize availability at the cost of increased complexity and computational cost for the validating DNS resolvers. The mitigations for the vulnerabilities discussed in this blog require local policies to be applied that relax these requirements in order to avoid exhausting the resources of validators.

The design of the DNS and DNSSEC protocols follows the Robustness principle: “be conservative in what you do, be liberal in what you accept from others”. There have been many vulnerabilities in the past that have taken advantage of protocol requirements following this principle. Malicious actors can exploit these vulnerabilities to attack DNS infrastructure, in this case by causing additional work for DNS resolvers by crafting DNSSEC responses with complex configurations. As is often the case, we find ourselves having to create a pragmatic balance between the flexibility that allows a protocol to adapt and evolve and the need to safeguard the stability and security of the services we operate.

Cloudflare’s public resolver 1.1.1.1 is a privacy-centric public resolver service. We have been using stricter validations and limits aimed at protecting our own infrastructure in addition to shielding authoritative DNS servers operated outside our network. As a result, we often receive complaints about resolution failures. Experience shows us that strict validations and limits can impact availability in some edge cases, especially when DNS domains are improperly configured. However, these strict validations and limits are necessary to improve the overall reliability and resilience of the DNS infrastructure.

The vulnerabilities and how we mitigated them are described below.

A DNSSEC signed zone can contain multiple keys (DNSKEY) to sign the contents of a DNS zone and a Resource Record Set (RRSET) in a DNS response can have multiple signatures (RRSIG). Multiple keys and signatures are required to support things like key rollover, algorithm rollover, and multi-signer DNSSEC. DNSSEC protocol specifications require a validating DNS resolver to try every possible combination of keys and signatures when validating a DNS response.

During validation, a resolver looks at the key tag of every signature and tries to find the associated key that was used to sign it. A key tag is an unsigned 16-bit number calculated as a checksum over the key’s resource data (RDATA). Key tags are intended to allow efficient pairing of a signature with the key which has supposedly created it.  However, key tags are not unique, and it is possible that multiple keys can have the same key tag. A malicious actor can easily craft a DNS response with multiple keys having the same key tag together with multiple signatures, none of which might validate. A validating resolver would have to try every combination (number of keys multiplied by number of signatures) when trying to validate this response. This increases the computational cost of the validating resolver many-fold, degrading performance for all its users. This is known as the Keytrap vulnerability.

Variations of this vulnerability include using multiple signatures with one key, using one signature with multiple keys having colliding key tags, and using multiple keys with corresponding hashes added to the parent delegation signer record.

We have limited the maximum number of keys we will accept at a zone cut. A zone cut is where a parent zone delegates to a child zone, e.g. where the .com zone delegates cloudflare.com to Cloudflare nameservers. Even with this limit already in place and various other protections built for our platform, we realized that it would still be computationally costly to process a malicious DNS answer from an authoritative DNS server.

To address and further mitigate this vulnerability, we added a signature validations limit per RRSET and a total signature validations limit per resolution task. One resolution task might include multiple recursive queries to external authoritative DNS servers in order to answer a single DNS question. Clients queries exceeding these limits will fail to resolve and will receive a response with an Extended DNS Error (EDE) code 0. Furthermore, we added metrics which will allow us to detect attacks attempting to exploit this vulnerability.

NSEC3 is an alternative approach for authenticated denial of existence. You can learn more about authenticated denial of existence here. NSEC3 uses a hash derived from DNS names instead of the DNS names directly in an attempt to prevent zone enumeration and the standard supports multiple iterations for hash calculations. However, because the full DNS name is used as input to the hash calculation, increasing hashing iterations beyond the initial doesn’t provide any additional value and is not recommended in RFC9276. This complication is further inflated while finding the closest enclosure proof. A malicious DNS response from an authoritative DNS server can set a high NSEC3 iteration count and long DNS names with multiple DNS labels to exhaust the computing resources of a validating resolver by making it do unnecessary hash computations.

For this vulnerability, we applied a similar mitigation technique as we did for Keytrap. We added a limit for total hash calculations per resolution task to answer a single DNS question. Similarly, clients queries exceeding this limit will fail to resolve and will receive a response with an EDE code 27. We also added metrics to track hash calculations allowing early detection of attacks attempting to exploit this vulnerability.

We would like to thank Elias Heftrig, Haya Schulmann, Niklas Vogel, Michael Waidner from the German National Research Center for Applied Cybersecurity ATHENE, for discovering the Keytrap vulnerability and doing a responsible disclosure.

We would like to thank Petr Špaček from Internet Systems Consortium (ISC) for discovering the NSEC3 iteration and closest encloser proof vulnerability and doing a responsible disclosure.

We would like to thank John Todd from Quad9  and the DNS Operations Analysis and Research Center (DNS-OARC) for facilitating coordination amongst various stakeholders.

And finally, we would like to thank the DNS-OARC community members, representing various DNS vendors and service providers, who all came together and worked tirelessly to fix these vulnerabilities, working towards a common goal of making the internet resilient and secure.

On Sunday, July 9, 2023, early morning UTC time, we observed a high number of DNS resolution failures — up to 7% of all DNS queries across the Asia Pacific region — caused by invalid DNSSEC signatures from Verisign .com and .net Top Level Domain (TLD) nameservers. This resulted in connection errors for visitors of Internet properties on Cloudflare in the region.

The local instances of Verisign’s nameservers started to respond with expired DNSSEC signatures in the Asia Pacific region. In order to remediate the impact, we have rerouted upstream DNS queries towards Verisign to locations on the US west coast which are returning valid signatures.

We have already reached out to Verisign to get more information on the root cause. Until their issues have been resolved, we will keep our DNS traffic to .com and .net TLD nameservers rerouted, which might cause slightly increased latency for the first visitor to domains under .com and .net in the region.

In order to proxy a domain’s traffic through Cloudflare’s network, there are two components involved with respect to the Domain Name System (DNS) from the perspective of a Cloudflare data center: external DNS resolution, and upstream or origin DNS resolution.

To understand this, let’s look at the domain name blog.cloudflare.com — which is proxied through Cloudflare’s network — and let’s assume it is configured to use origin.example as the origin server:

Here, the external DNS resolution is the part where DNS queries to blog.cloudflare.com sent by public resolvers like 1.1.1.1 or 8.8.8.8 on behalf of a visitor return a set of Cloudflare Anycast IP addresses. This ensures that the visitor’s browser knows where to send an HTTPS request to load the website. Under the hood your browser performs a DNS query that looks something like this (the trailing dot indicates the DNS root zone):

$ dig blog.cloudflare.com. +short
104.18.28.7
104.18.29.7

(Your computer doesn’t actually use the dig command internally; we’ve used it to illustrate the process) Then when the next closest Cloudflare data center receives the HTTPS request for blog.cloudflare.com, it needs to fetch the content from the origin server (assuming it is not cached).

There are two ways Cloudflare can reach the origin server. If the DNS settings in Cloudflare contain IP addresses then we can connect directly to the origin. In some cases, our customers use a CNAME which means Cloudflare has to perform another DNS query to get the IP addresses associated with the CNAME. In the example above, blog.cloudflare.com has a CNAME record instructing us to look at origin.example for IP addresses. During the incident, only customers with CNAME records like this going to .com and .net domain names may have been affected.

The domain origin.example needs to be resolved by Cloudflare as part of the upstream or origin DNS resolution. This time, the Cloudflare data center needs to perform an outbound DNS query that looks like this:

$ dig origin.example. +short
192.0.2.1

DNS is a hierarchical protocol, so the recursive DNS resolver, which usually handles DNS resolution for whoever wants to resolve a domain name, needs to talk to several involved nameservers until it finally gets an answer from the authoritative nameservers of the domain (assuming no DNS responses are cached). This is the same process during the external DNS resolution and the origin DNS resolution. Here is an example for the origin DNS resolution:

Inherently, DNS is a public system that was initially published without any means to validate the integrity of the DNS traffic. So in order to prevent someone from spoofing DNS responses, DNS Security Extensions (DNSSEC) were introduced as a means to authenticate that DNS responses really come from who they claim to come from. This is achieved by generating cryptographic signatures alongside existing DNS records like A, AAAA, MX, CNAME, etc. By validating a DNS record’s associated signature, it is possible to verify that a requested DNS record really comes from its authoritative nameserver and wasn’t altered en-route. If a signature cannot be validated successfully, recursive resolvers usually return an error indicating the invalid signature. This is exactly what happened on Sunday.

On Saturday, July 8, 2023, at 21:10 UTC our logs show the first instances of DNSSEC validation errors that happened during upstream DNS resolution from multiple Cloudflare data centers in the Asia Pacific region. It appeared DNS responses from the TLD nameservers of .com and .net of the type NSEC3 (a DNSSEC mechanism to prove non-existing DNS records) included invalid signatures. About an hour later at 22:16 UTC, the first internal alerts went off (since it required issues to be consistent over a certain period of time), but error rates were still at a level at around 0.5% of all upstream DNS queries.

After several hours, the error rate had increased to a point in which ~13% of our upstream DNS queries in affected locations were failing. This percentage continued to fluctuate over the duration of the incident between the ranges of 10-15% of upstream DNS queries, and roughly 5-7% of all DNS queries (external & upstream resolution) to affected Cloudflare data centers in the Asia Pacific region.

Initially it appeared as though only a single upstream nameserver was having issues with DNS resolution, however upon further investigation it was discovered that the issue was more widespread. Ultimately, we were able to verify that the Verisign nameservers for .com and .net were returning expired DNSSEC signatures on a portion of DNS queries in the Asia Pacific region. Based on our tests, other nameserver locations were correctly returning valid DNSSEC signatures.

In response, we rerouted our DNS traffic to the .com and .net TLD nameserver IP addresses to Verisign’s US west locations. After this change was propagated, the issue very quickly subsided and origin resolution error rates returned to normal levels.

All times are in UTC:

2023-07-08 21:10 First instances of DNSSEC validation errors appear in our logs for origin DNS resolution.

2023-07-08 22:16 First internal alerts for Asia Pacific data centers go off indicating origin DNS resolution error on our test domain. Very few errors for other domains at this point.

2023-07-09 02:58 Error rates have increased substantially since the first instance. An incident is declared.

2023-07-09 03:28 DNSSEC validation issues seem to be isolated to a single upstream provider. It is not abnormal that a single upstream has issues that propagate back to us, and in this case our logs were predominantly showing errors from domains that resolve to this specific upstream.

2023-07-09 04:52 We realize that DNSSEC validation issues are more widespread and affect multiple .com and .net domains. Validation issues continue to be isolated to the Asia Pacific region only, and appear to be intermittent.

2023-07-09 05:15 DNS queries via popular recursive resolvers like 8.8.8.8 and 1.1.1.1 do not return invalid DNSSEC signatures at this point. DNS queries using the local stub resolver continue to return DNSSEC errors.

2023-07-09 06:24 Responses from .com and .net Verisign nameservers in Singapore contain expired DNSSEC signatures, but responses from Verisign TLD nameservers in other locations are fine.

2023-07-09 06:41 We contact Verisign and notify them about expired DNSSEC signatures.

2023-07-09 06:50 In order to remediate the impact, we reroute DNS traffic via IPv4 for the .com and .net Verisign nameserver IPs to US west IPs instead. This immediately leads to a substantial drop in the error rate.

2023-07-09 07:06 We also reroute DNS traffic via IPv6 for the .com and .net Verisign nameserver IPs to US west IPs. This leads to the error rate going down to zero.

2023-07-10 09:23 The rerouting is still in place, but the underlying issue with expired signatures in the Asia Pacific region has still not been resolved.

2023-07-10 18:23 Verisign gets back to us confirming that they “were serving stale data” at their local site and have resolved the issues.

As mentioned in the introduction, the underlying cause for this incident was expired DNSSEC signatures for .net and .com zones. Expired DNSSEC signatures will cause a DNS response to be interpreted as invalid. There are two main scenarios in which this error was observed by a user:

1. CNAME flattening for external DNS resolution. This is when our authoritative nameservers attempt to return the IP address(es) that a CNAME record target resolves to rather than the CNAME record itself.

2. CNAME target lookup for origin DNS resolution. This is most commonly used when an HTTPS request is sent to a Cloudflare anycast IP address, and we need to determine what IP address to forward the request to. See How Cloudflare works for more details.

In both cases, behind the scenes the DNS query goes through an in-house recursive DNS resolver in order to lookup what a hostname resolves to. The purpose of this resolver is to cache queries, optimize future queries and provide DNSSEC validation. If the query from this resolver fails for whatever reason, our authoritative DNS system will not be able to perform the two scenarios outlined above.

During the incident, the recursive resolver was failing to validate the DNSSEC signatures in DNS responses for domains ending in .com and .net. These signatures are sent in the form of an RRSIG record alongside the other DNS records they cover. Together they form a Resource Record set (RRset). Each RRSIG has the corresponding fields:

As you can see, the main part of the payload is associated with the signature and its corresponding metadata. Each recursive resolver is responsible for not only checking the signature but also the expiration time of the signature. It is important to obey the expiration time in order to avoid returning responses for RRsets that have been signed by old keys, which could have potentially been compromised by that time.

During this incident, Verisign, the authoritative operator for the .com and .net TLD zones, was occasionally returning expired signatures in its DNS responses in the Asia Pacific region. As a result our recursive resolver was not able to validate the corresponding RRset. Ultimately this meant that a percentage of DNS queries would return SERVFAIL as response code to our authoritative nameserver. This in turn caused connection issues for users trying to connect to a domain on Cloudflare, because we weren't able to resolve the upstream target of affected domain names and thus didn’t know where to send proxied HTTPS requests to upstream servers.

Once we had identified the root cause we started to look at different ways to remedy the issue. We came to the conclusion that the fastest way to work around this very regionalized issue was to stop using the local route to Verisign's nameservers. This means that, at the time of writing this, our outgoing DNS traffic towards Verisign's nameservers in the Asia Pacific region now traverses the Pacific and ends up at the US west coast, rather than being served by closer nameservers. Due to the nature of DNS and the important role of DNS caching, this has less impact than you might initially expect. Frequently looked up names will be cached after the first request, and this only needs to happen once per data center, as we share and pool the local DNS recursor caches to maximize their efficiency.

Ideally, we would have been able to fix the issue right away as it potentially affected others in the region too, not just our customers. We will therefore work diligently to improve our incident communications channels with other providers in order to ensure that the DNS ecosystem remains robust against issues such as this. Being able to quickly get hold of other providers that can take action is vital when urgent issues like these arise.

This incident once again shows how impactful DNS failures are and how crucial this technology is for the Internet. We will investigate how we can improve our systems to detect and resolve issues like this more efficiently and quickly if they occur again in the future.

While Cloudflare was not the cause of this issue, and we are certain that we were not the only ones affected by this, we are still sorry for the disruption to our customers and to all the users who were unable to access Internet properties during this incident.

Update: On Tue Jul 11 22:24:21 UTC 2023_,_ Verisign posted an announcement, providing more details:

Last week, during a migration of one of our DNS resolution sites in Singapore, from one provider to another, we unexpectedly lost management access and the ability to deliver changes and DNS updates to the site. Following our standard procedure, we disabled all transit links to the affected site. Unfortunately, a peering router remained active, which was not immediately obvious to our teams due to the lack of connectivity there.

Over the weekend, this caused an issue that may have affected the ability of some internet users in the region to reach some .com and .net domains, as DNSSEC signatures on the site began expiring. The issue was resolved by powering off the site’s peering router, causing the anycast route announcement to be withdrawn and traffic to be directed to other sites.

We are updating our processes and procedures and will work to prevent such issues from recurring in the future.

The Singapore site is part of a highly redundant constellation of more than 200 sites that make up our global network. This issue had no effect on the core resolution of .com and .net resolution globally. We apologize to those who may have been affected.

On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.

The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.

Looking at Cloudflare’s 1.1.1.1 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.

This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar does not show any indication of an Internet disruption in Fiji coincident with the start of this problem.

It was suggested within the Hacker News comments that the problem could be DNSSEC related. Upon further investigation, it appears that may be the cause. In verifying the DNSSEC record for the .fj ccTLD, shown in the dig output below, we see that it states EDE: 9 (DNSKEY Missing): 'no SEP matching the DS found for fj.'

kdig fj. soa +dnssec @1.1.1.1 
;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 12710
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1
 
;; EDNS PSEUDOSECTION:
;; Version: 0; flags: do; UDP size: 1232 B; ext-rcode: NOERROR
;; EDE: 9 (DNSKEY Missing): 'no SEP matching the DS found for fj.'
 
;; QUESTION SECTION:
;; fj.                          IN      SOA
 
;; Received 73 B
;; Time 2022-03-08 08:57:41 EST
;; From 1.1.1.1@53(UDP) in 17.2 ms

Extended DNS Error 9 (EDE: 9) is defined as “A DS record existed at a parent, but no supported matching DNSKEY record could be found for the child.” The Cloudflare Learning Center article on DNSKEY and DS records explains this relationship:

The DS record is used to verify the authenticity of child zones of DNSSEC zones. The DS key record on a parent zone contains a hash of the KSK in a child zone. A DNSSEC resolver can therefore verify the authenticity of the child zone by hashing its KSK record, and comparing that to what is in the parent zone's DS record.

Ultimately, it appears that around midnight UTC, the .fj zone started to be signed with a key that was not in the root zone DS, possibly as the result of a scheduled rollover that happened without checking that the root zone was updated first by IANA, which updates the root zone. (IANA owns contact with the TLD operators, and instructs the Root Zone Publisher on the changes to make in the next version of the root zone.)

DNSSEC problems as the root cause of the observed issue align with the observation in the Hacker News comments that some were able to access .fj websites, while others were not. Users behind resolvers doing strict DNSSEC validation would have seen an error in their browser, while users behind less strict resolvers would have been able to access the sites without a problem.

Further analysis of Cloudflare resolver metrics indicates that the problem was resolved around 1400 UTC, when the DS was updated. When DNSSEC is improperly configured for a single domain name, it can cause problems accessing websites or applications in that zone. However, when the misconfiguration occurs at a ccTLD level, the impact is much more significant. Unfortunately, this seems to occur all too often.

(Thank you to Ólafur Guðmundsson for his DNSSEC expertise.)

The Internet is too vital to allow this known problem to continue any longer. It's time networks prevented leaks and hijacks from having any impact. It's time to make BGP safe. No more excuses.

Border Gateway Protocol (BGP), a protocol to exchange routes has existed and evolved since the 1980s. Over the years it has had security features. The most notable security addition is Resource Public Key Infrastructure (RPKI), a security framework for routing. It has been the subject of a few blog posts following our deployment in mid-2018.

Today, the industry considers RPKI mature enough for widespread use, with a sufficient ecosystem of software and tools, including tools we've written and open sourced. We have fully deployed Origin Validation on all our BGP sessions with our peers and signed our prefixes.

However, the Internet can only be safe if the major network operators deploy RPKI. Those networks have the ability to spread a leak or hijack far and wide and it's vital that they take a part in stamping out the scourge of BGP problems whether inadvertent or deliberate.

Many like AT&T and Telia pioneered global deployments of RPKI in 2019. They were successfully followed by Cogent and NTT in 2020. Hundreds networks of all sizes have done a tremendous job over the last few years but there is still work to be done.

If we observe the customer-cones of the networks that have deployed RPKI, we see around 50% of the Internet is more protected against route leaks. That's great, but it's nothing like enough.

Today, we are releasing isBGPSafeYet.com, a website to track deployments and filtering of invalid routes by the major networks.

We are hoping this will help the community and we will crowdsource the information on the website. The source code is available on GitHub, we welcome suggestions and contributions.

We expect this initiative will make RPKI more accessible to everyone and ultimately will reduce the impact of route leaks. Share the message with your Internet Service Providers (ISP), hosting providers, transit networks to build a safer Internet.

Additionally, to monitor and test deployments, we decided to announce two bad prefixes from our 200+ data centers and via the 233+ Internet Exchange Points (IXPs) we are connected to:

• 103.21.244.0/24

• 2606:4700:7000::/48

Both these prefixes should be considered invalid and should not be routed by your provider if RPKI is implemented within their network. This makes it easy to demonstrate how far a bad route can go, and test whether RPKI is working in the real world.

A Route Origin Authorization for 103.21.244.0/24 on rpki.cloudflare.com

In the test you can run on isBGPSafeYet.com, your browser will attempt to fetch two pages: the first one valid.rpki.cloudflare.com, is behind an RPKI-valid prefix and the second one, invalid.rpki.cloudflare.com, is behind the RPKI-invalid prefix.

The test has two outcomes:

• If both pages were correctly fetched, your ISP accepted the invalid route. It does not implement RPKI.

• If only valid.rpki.cloudflare.com was fetched, your ISP implements RPKI. You will be less sensitive to route-leaks.

a simple test of RPKI invalid reachability

We will be performing tests using those prefixes to check for propagation. Traceroutes and probing helped us in the past by creating visualizations of deployment.

A simple indicator is the number of networks sending the accepted route to their peers and collectors:

Routing status from online route collection tool RIPE Stat

In December 2019, we released a Hilbert curve map of the IPv4 address space. Every pixel represents a /20 prefix. If a dot is yellow, the prefix responded only to the probe from a RPKI-valid IP space. If it is blue, the prefix responded to probes from both RPKI valid and invalid IP space.

To summarize, the yellow areas are IP space behind networks that drop RPKI invalid prefixes. The Internet isn't safe until the blue becomes yellow.

Hilbert Curve Map of IP address space behind networks filtering RPKI invalid prefixes

Last but not least, we would like to thank every network that has already deployed RPKI and every developer that contributed to validator-software code bases. The last two years have shown that the Internet can become safer and we are looking forward to the day where we can call route leaks and hijacks an incident of the past.

But let's rewind a bit.

It all started in 2015, when we were looking at the code of our authoritative DNS server. The code flow was generally fine, but it was all peppered with naughty statements like this:

if qtype == "ANY" {
    // special case
}

This special code was ugly and error prone. This got us thinking: do we really need it? "ANY" is not a popular query type - no legitimate software uses it (with the notable exception of qmail).

Image by Christopher MichelCC BY 2.0

"ANY" queries, also called "* queries" in old RFCs, are supposed to return "all records" (citing RFC1035). There are two problems with this notion.

First, it assumes the server is able to retrieve "all records". In our implementation - we can't. Our DNS server, like many modern implementations, doesn't have a single "zone" file listing all properties of a DNS zone. This design allows us to respond fast and with information always up to date, but it makes it incredibly hard to retrieve "all records". Correct handling of "ANY" adds unreasonable code complexity for an obscure, rarely used query type.

Second, many of the DNS responses are generated on-demand. To mention just two use cases:

• Some of our DNS responses are based on location

• We are using black lies and DNS shotgun for DNSSEC

Storing data in modern databases and dynamically generating responses poses a fundamental problem to ANY.

Around the same time a catastrophe happened - Firefox started shipping with DNS code issuing "ANY" types. The intention was, as usual, benign. Firefox developers wanted to get the TTL value for A and AAAA queries.

To cite a DNS guru Andrew Sullivan:

In general, ANY is useful for troubleshooting but should never be usedfor regular operation. Its output is unpredictable given the effectsof caches. It can return enormous result sets.

In user code you can't rely on anything sane to come out of an "ANY" query. While an "ANY" query has somewhat defined semantics on the DNS authoritative side, it's undefined on the DNS resolver side. Such a query can confuse the resolver:

• Should it forward the "ANY" query to authoritative?

• Should it respond with any record that is already in cache?

• Should it do some a mixture of the above behaviors?

• Should it cache the result of "ANY" query and re-use the data for other queries?

Different implementations do different things. "ANY" does not mean "ALL", which is the main source of confusion. To our joy, Firefox quickly backpedaled on the change and stopped issuing ANY queries.

A typical 50Gbps DNS amplification targeting one of our customers. The attack lasted about 4 hours.

Furthermore, since the "ANY" query can generate a large response, they were often used for DNS reflection attacks. Authoritative providers receive a spoofed ANY query and send the large answer to a target, potentially causing DoS damage. We have blogged about that many times:

• The DDoS that knocked Spamhaus offline

• Deep inside a DNS amplification attack

• Reflections on reflections

• How the CPSC is inadvertently behind the largest attacks

The DoS problem with ANY is ancient. Here is a discussion about a patch to bind tweaking ANY from 2013.

There is also a second angle to the ANY DoS problem. Some reports suggested that performant DNS servers (authoritative or resolvers) can fill their outbound network capacity with numerous ANY responses.

The recommendation is simple - network operators must use "response rate limiting" when answering large DNS queries, otherwise they pose a DoS threat. The "ANY" query type just happens to often give such large responses, while providing little value to legitimate users.

In 2015 frustrated with the experience we announced we would like to stop giving responses to "ANY" queries and wrote a (controversial at a time) blog post:

• Deprecating DNS ANY meta-query type

A year later we followed up explaining possible solutions:

• What happened next - the deprecation of ANY

And here we come today! With RFC8482 we have an RFC proposed standard clarifying that controversial query.

ANY queries are a background noise. Under normal circumstances, we see a very small volume of ANY queries.

What precisely can be done about "ANY" queries? RFC8482 specifies that:

A DNS responder that receives an ANY query MAY decline to provide aconventional ANY response or MAY instead send a response with asingle RRset (or a larger subset of available RRsets) in the answersection.

This clearly defines the corner case - from now on the authoritative server may respond with, well, any query type to an "ANY" query. Sometimes simple stuff like this matters most.

This opens a gate for implementers - we can prepare a simple answer to these queries. As an implementer you may stick "A", or "AAAA" or anything else in the response if you wish. Furthermore, the spec recommends returning a special (and rarely used thus far) HINFO type. This is in fact what we do:

$ dig ANY cloudflare.com @ns3.cloudflare.com. 
;; ANSWER SECTION:
cloudflare.com.		3789	IN	HINFO	"ANY obsoleted" "See draft-ietf-dnsop-refuse-any"

Oh, we need to update the message to mention the fresh RFC number! NS1 agrees with our implementation:

$ dig ANY nsone.net @dns1.p01.nsone.net.
;; ANSWER SECTION:
nsone.net.		3600	IN	HINFO	"ANY not supported." "See draft-ietf-dnsop-refuse-any"

Our ultimate hero is wikipedia.org, which does exactly what the RFC recommends:

$ dig ANY wikipedia.org @ns0.wikimedia.org.
;; ANSWER SECTION:
wikipedia.org.		3600	IN	HINFO	"RFC8482" ""

On our resolver service we stop ANY queries with NOTIMP code. This makes us more confident the resolver isn't used to perform DNS reflections:

$ dig ANY cloudflare.com @1.1.1.1
;; ->>HEADER<<- opcode: QUERY, status: NOTIMP, id: 14151

On the client side, just don't use ANY DNS queries. On the DNS server side - you are allowed to rip out all the gory QTYPE::ANY handling code, and replace it with a top level HINFO message or first RRset found. Enjoy cleaning your codebase!

It took the DNS community some time to agree on the specifics, but here we are at the end. RFC8482 cleans up the last remaining DNS meta-qtype, and allows for simpler DNS authoritative and DNS resolver implementations. It finally clearly defines the semantics of ANY queries going through resolvers and reduces the DoS risk for the whole Internet.

Not all the effort must go to new shiny protocols and developments, sometimes, cleaning the bitrot is as important. Similar cleanups are being done in other areas. Keep up the good work!

We would like to thank the co-authors of RFC8482, and the community scrutiny and feedback. For us, RFC8482 is definitely a good thing, and allowed us to simplify our codebase and make the Internet safer.

Mission accomplished! One step at a time we can help make the Internet a better place.

We announced Cloudflare Registrar in September. We launched the product by making it available in waves to our existing customers. During that time we gathered feedback and continued making improvements to the product while also adding more TLDs.

Staring today, we’re excited to make Cloudflare Registrar available to all of our customers. Cloudflare Registrar only charges you what we pay to the registry for your domain and any user can now rely on that at-cost pricing to manage their domain. As part of this announcement, we’d like to share some insights and data about domain registration that we learned during the early access period.

When you launch your domain to the world, you rely on the Domain Name System (DNS) to direct your users to the address for your site. However, DNS cannot guarantee that your visitors reach your content because DNS, in its basic form, lacks authentication. If someone was able to poison the DNS responses for your site, they could hijack your visitors' DNS requests.

The Domain Name System Security Extensions (DNSSEC) can help prevent that type of attack by adding a chain of trust to DNS queries. When you enable DNSSEC for your site, you can ensure that the DNS response your users receive is the authentic IP address of your domain.

Across the industry, adoption of DNSSEC is abysmal. According to Verisign, 1% of .com domains use DNSSEC; less than 0.8% of .net domains do. Why is adoption so low? It’s inconvenient to enable DNSSEC for a site. Additionally, some registrars charge for the feature. APNIC observed that registrars who charge for DNSSEC see significantly lower adoption.

Cloudflare has made DNSSEC available for free for years, but we could not address the convenience factor until we launched our registrar. While we can create DS records, your registrar has to post them to the registry. Now that Cloudflare is a registrar, in addition to an authoritative DNS provider, we can make it one-click. We announced that feature in January. Since launching, 25% of domains on Cloudflare Registrar now use DNSSEC.

We’re going to keep working to make it even easier to enable for your domains. We want to help our customers reach 100% DNSSEC enablement by removing the need for even a single click.

When you begin a domain transfer to Cloudflare, we ask that you input an auth code that your current registrar provides and that is unique to each domain that you transfer. We use that auth code to send your request to the registry, who manages all domain names for given TLD. The registry confirms that the code is valid and then tells your current registrar to release the domain.

Once your current registrar receives that request, you have two options: manually approve the transfer or wait five days. If you wait five days and do nothing, the transfer will complete. While that might feel easier, we’ve been surprised to see that 62% of transfers were completed by manual approval.

Historically, domains used either country-code TLDs (ccTLDs) or generic TLDs (gTLDs). The generic ones include the 4 extensions behind the world’s most popular domains: .com, .net, .org and .info. In 2005, ICANN began considering adding new top-level domain extensions. In 2012, ICANN started accepting applications from registries, current and prospective, who wanted to manage TLDs. They received 1,930.

Of those 1,930 applications, 1,232 became supported extensions and were classified as new gTLDs (ngTLDs). Today, Cloudflare Registrar supports all 4 legacy gTLDs, 1 ccTLD and 241 ngTLDs. gTLDs continue to represent the vast majority of domains registered with Cloudflare. That distribution is consistent with trends in the domain name industry. We expect that to change a bit as we expand into more ccTLDs.

2,081 different TLDs are represented on Cloudflare and use our authoritative DNS. I imagine that number has grown in the time it took to publish this post. We support 246 TLDs on Registrar today. We know that many of you have domains you want to transfer that use TLDs we do not support currently, particularly amongst ccTLDs. From massive ccTLDs like .uk, to more obscure ngTLDs like .boutique, we’ve received a lot of requests to expand the list. For a reason I don’t understand yet, members of the Cloudflare engineering team own over 2% of all active .horse domains in the world and use them for internal testing projects. We’re working on that one, too, so we can make this page built on Workers return a Yes.

We’re working on it. Most ccTLDs require a unique accreditation and validation flow. We’re working every day to add to that list of supported TLDs, starting with the largest ones on Cloudflare.

Cloudflare Registrar is now available to all users. You can start transferring your domains by following this link here. Have questions? Instructions are available here.

When you launch your domain to the world, you rely on the Domain Name System (DNS) to direct your users to the address for your site. However, DNS cannot guarantee that your visitors reach your content because DNS, in its basic form, lacks authentication. If someone was able to poison the DNS responses for your site, they could hijack your visitors' requests.

The Domain Name System Security Extensions (DNSSEC) can help prevent that type of attack by adding a chain of trust to DNS queries. When you enable DNSSEC for your site, you can ensure that the DNS response your users receive is the authentic address of your site.

We launched support for DNSSEC in 2014. We made it free for all users, but we couldn’t make it easy to set up. Turning on DNSSEC for a domain was still a multistep, manual process. With the launch of Cloudflare Registrar, we can finish the work to make it simple to enable for your domain.

You can now enable DNSSEC with a single click if your domain is registered with Cloudflare Registrar. Visit the DNS tab in the Cloudflare dashboard, click "Enable DNSSEC", and we'll handle the rest. If you are not on Cloudflare Registrar, you can read more about transferring your domain here.

The Domain Name System (DNS) translates a site’s domain name, like cloudflare.com, to the address of the server hosting that site. When users request your website, their browser starts with a DNS query to find that IP address.

The query first asks the Internet root servers to locate the servers responsible for the top-level domain (TLD). In the case of .com, those servers are managed by the registry Verisign. Verisign then finds the authoritative nameservers for that particular domain and requests the IP from them. If you use Cloudflare for your site’s DNS, Cloudflare manages those nameservers and we respond with an anycast IP for your site, which is ultimately returned to your visitor.

DNS assumes each request in that chain can be trusted, but the protocol does not actually verify the response. That presumption leaves the series of requests vulnerable to attack. In that scenario, an attacker poisons the responses for your site with directions to a malicious one. Instead of arriving at your webpage, your visitors are directed to a site that can be used for phishing or other malicious purposes. To solve that problem, a layer is needed to verify that each response can be trusted.

DNSSEC builds that trust by adding cryptographic signatures to each handoff in the relay. Those signatures establish a chain of trust from the authoritative nameservers, through the TLD server, and all the way to the root servers of the Internet. Your visitors’ DNS resolver can validate that the IP address returned for your domain name was provided by the authentic source.

We began advocating for DNSSEC in 2014 and launched beta support in 2015. We’re committed to expanding its adoption on the internet. However, we’ve only been able to provide DNSSEC for your domain when you completed a series of manual actions. To make DNSSEC ubiquitous, we first have to make it easy to enable like we did for one-click SSL.

Historically, enabling DNSSEC required you to generate a DS record from a service like Cloudflare, copy it down, and then save it to your registrar so they could send it to your registry. That’s tedious. We can now remove those steps for you. When Cloudflare is your registrar, we can automatically apply DNSSEC through our support for CDS and CDNSKEY.

Instead of asking you to save the records yourself, Cloudflare Registrar automatically scans available DS records (and validates them) for domains that use our nameservers. When we notice that you have DNSSEC enabled, we grab the details and send it to the registry for you.

To turn on DNSSEC, navigate to the DNS tab for your domain in the Cloudflare dashboard. In the DNSSEC card, select “Enable” and that’s it. We’ll handle the rest. Your records will be set in the next 24-36 hours. It’s free, it’s one-click, and it helps secure your site.

If you have started transferring your domain to Cloudflare registrar, you can use the one-click DNSSEC feature as soon as the transfer completes. If you already have DS records for your domain, the domain transfer will protect the DS record and make sure it’s still current after the transfer.

While this feature removes some of the chore to enable DNSSEC, we’re committed to removing any hurdle to making the Internet safer. We’re working on supporting DNSSEC by default for sites on Cloudflare. We have some work to do to reach this goal, but we’re excited to help make DNSSEC the new normal.

Interested in helping us with that work? Visit the Cloudflare jobs page here to join our team.

Cloudflare first started talking about DNSSEC in 2014 and at the time, Nick Sullivan wrote: “DNSSEC is a valuable tool for improving the trust and integrity of DNS, the backbone of the modern Internet.”

Over the past four years, it has become an even more critical part of securing the internet. While HTTPS has gone a long way in preventing user sessions from being hijacked and maliciously (or innocuously) redirected, not all internet traffic is HTTPS. A safer Internet should secure every possible layer between a user and the origin they are intending to visit.

As a quick refresher, DNSSEC allows a user, application, or recursive resolver to trust that the answer to their DNS query is what the domain owner intends it to be. Put another way: DNSSEC proves authenticity and integrity (though not confidentiality) of a response from the authoritative nameserver. Doing so makes it much harder for a bad actor to inject malicious DNS records into the resolution path through BGP Leaks and cache poisoning. Trust in DNS matters even more when a domain is publishing record types that are used to declare trust for other systems. As a specific example, DNSSEC is helpful for preventing malicious actors from obtaining fraudulent certificates for a domain. Research has shown how DNS responses can be spoofed for domain validation.

This week we are announcing our full support for CDS and CDNSKEY from RFC 8078. Put plainly: this will allow for setting up of DNSSEC without requiring the user to login to their registrar to upload a DS record. Cloudflare customers on supported registries will be able to enable DNSSEC with the click of one button in the Cloudflare dashboard.

DNSSEC’s largest problem has been adoption. The number of DNS queries validated by recursive resolvers for DNSSEC has remained flat. Worldwide, less than 14% of DNS requests have DNSSEC validated by the resolver according to our friends at APNIC. The blame here falls on the shoulders of the default DNS providers that most devices and users receive from DHCP via their ISP or network provider. Data shows that some countries do considerably better: Sweden, for example, has over 80% of their requests validated, showing that the default DNS resolvers in those countries validate the responses as they should. APNIC also has a fun interactive map so you can see how well your country does.

So what can we do? To ensure your resolver supports DNSSEC, visit brokendnssec.net in your browser. If the page loads, you are not protected by a DNSSEC validating resolver and should switch your resolver. However, in order to really move the needle across the internet, Cloudflare encourages network providers to either turn on the validation of DNSSEC in their software or switch to publicly available resolvers that validate DNSSEC by default. Of course we have a favourite, but there are other fine choices as well.

Validation handles the user side, but another problem has been the signing of the zones themselves. Initially, there was concern about adoption at the TLD level given that TLD support is a requirement for DNSSEC to work. This is now largely a non-issue with over 90% of TLDs signed with DS records in the root zone, as of 2018-08-27.

It’s a different story when it comes to the individual domains themselves. Per NIST data, a woefully low 3% of the Fortune 1000 sign their primary domains. Some of this is due to apathy by the domain owners. However, some large DNS operators do not yet support the option at all, requiring domain owners who want to protect their users to move to another provider altogether. If you are on a service that does not support DNSSEC, we encourage you to switch to one that does and let them know that was the reason for the switch. Other large operators, such as GoDaddy, charge for DNSSEC. Our stance here is clear: DNSSEC should be available and included at all DNS operators for free.

In December of 2017, APNIC wrote about why DNSSEC deployment remains so low and that remains largely true today. One key point was that the number of domain owners who attempt DNSSEC activation but do not complete it is very high. Using Cloudflare as an example, APNIC measured that 40% of those who enabled DNSSEC in the Cloudflare Dash (evidenced by the presence of a DNSKEY record) were actually successful in serving a DS key from the registry. Current data over a recent 90 day period is slightly better: we are seeing just over 50% of all zones which attempted to enable DNSSEC were able to complete the process with the registry (Note: these domains still resolve, they are just still not secured). Of our most popular TLDs, .be and .nl have success rates of over 70%, but these numbers are still not where we would want them to be in an ideal world. The graph below shows the specific rates for the most popular TLDs (most popular from left to right).

This end result is likely not surprising to anyone who has tried to add a DS record to their registrar. Locating the part of the registrar UI that houses DNSSEC can be problematic, as can the UI of adding the record itself. Additional factors such as varying degrees of technical knowledge amongst users and simply having to manage multiple logins and roles can also explain the lack of completion in the process. Finally, varying levels of DNSSEC compatibility amongst registrars may prevent even knowledgeable users from creating DS records in the parent.

As an example, at Cloudflare, we took a minimalist UX approach for adding DS records for delegated child domains. A novice user may not understand the fields and requirements for the DS record:

CDS and CDNSKEY

As mentioned in the aforementioned APNIC blog, Cloudflare is supportive of RFC 8078 and the CDS and CDNSKEY records. This should come as no surprise given that our own Olafur Gudmundsson is a co-author of the RFC. CDS and CDNSKEY are records that mirror the DS and DNSKEY record types but are designated to signal the parent/registrar that the child domain wishes to enable DNSSEC and have a DS record presented by the registry. We have been pushing for automated solutions in this space for years and are encouraging the industry to move with us.

Today, we are announcing General Availability and full support for CDS and CDNSKEY records for all Cloudflare managed domains that enable DNSSEC in the Cloudflare dash.

Cloudflare will publish CDS and CDNSKEY records for all domains who enable DNSSEC. Parent registries should scan the nameservers of the  domains under their purview and check for these rrsets. The presence of a CDS key for a domain delegated to Cloudflare indicates that a verified Cloudflare user has enabled DNSSEC within their dash and that the parent operator (a registrar or the registry itself) should take the CDS record content and create the requisite DS record to start signing the domain. TLDs .ch and .cz already support this automated method through Cloudflare and any other DNS operators that choose to support RFC8078. The registrar Gandi and a number of TLDs have indicated support in the near future.

Cloudflare also supports CDS0 for the removal of the DS record in the case that the user transfers their domain off of Cloudflare or otherwise disables DNSSEC.

Below are a number of suggested procedures that parent registries may take to provide for the best experience for our users:

• Scan Selection - Parent Operators should only scan their child domains who have nameservers pointed at Cloudflare (or other DNS operators who adopt RFC8078). Cloudflare nameservers are indicated *.ns.cloudflare.com.

• Scan Regularly - Parent Operators should scan at regular intervals for the presence and change of CDS records. A scan every 12 hours should be sufficient, though faster is better.

• Notify Domain Contacts - Parent Operators should notify their designated contacts through known channels (such as email and/or SMS) for a given child domain upon detection of a new CDS record and an impending change of their DS record. The Parent Operator may also wish to provide a standard delay (24 hours) before changing the DS record to allow the domain contact to cancel or otherwise change the operation.

• Verify Success - Parent Operators must ensure that the domain continues to resolve after being signed. Should the domain fail to resolve immediately after changing the DS record, the Parent Operator must fall back to the previous functional state and should notify designated contacts.

For Cloudflare customers, this means an easier implementation of DNSSEC once your registry/registrar supports CDS and CDNSKEY. Customers can also enable DNSSEC for free on Cloudflare and manually enter the DS to the parent. To check your domain’s DNSSEC status, DNSViz (example cloudflare.com) has one of the most standards compliant tools online.

For registries and registrars, we are taking this step with the hope that more providers support RFC8078 and help increase the global adoption of technology that helps end users be less vulnerable to DNS attacks on the internet.

For other DNS operators, we encourage you to join us in supporting this method as the more major DNS operators that publish CDS and CDNSKEY, the more likely it will be that the registries will start looking for and use them.

Cloudflare will continue pushing down this path and has plans to create and open source additional tools to help registries and operators push and consume records. If this sounds interesting to you, we are hiring.

Subscribe to the blog for daily updates on our announcements.

By 'end-to-end security', I mean that neither the site owner nor users have to trust Cloudflare to serve the correct documents, like they do now. This is similar to how using HTTPS means you don't have to trust your ISP to not modify or inspect traffic.

The first step is to choose a domain name for your website. Websites should be given their own domain name, rather than served directly from the gateway by root hash, so that they are considered a distinct origin by the browser. This is primarily to prevent cache poisoning, but there are several functional advantages as well. It gives websites their own instance of localStorage and their own cookie jar which are sandboxed from inspection and manipulation by malicious third-party documents. It also lets them run Service Workers without conflict, and request special permissions of the user like access to the webcam or GPS. But most importantly, having a domain name makes a website easier to identify and remember.

Now that you've chosen a domain, rather than using it as-is, you’ll need to add "ipfs-sec" as the left-most subdomain. So for example, you'd use "ipfs-sec.example.com" instead of just "example.com". The ipfs-sec subdomain is special because it signals to the user and to their browser that your website is capable of being served with end-to-end integrity.

In addition to that, ipfs-sec domains require DNSSEC to be properly setup to prevent spoofing. Unlike with standard HTTPS, where DNS spoofing can't usually result in a on-path attacker attack, this is exactly what DNS spoofing does to IPFS because the root hash of the website is stored in DNS. Many registrars make enabling DNSSEC as easy as the push of a button, though some don't support it at all.

With the ipfs-sec domain, you can now follow the developer documentation on how to serve a generic static website from IPFS. Note that you'll need to use a CNAME setup and retain control of your DNS, rather than the easier method of just signing up for Cloudflare. This helps maintain a proper separation between the party managing the DNSSEC signing keys and the party serving content to end-users. Keep in mind that CNAME setups tend to be problematic and get into cases that are difficult to debug, which is why we reserve them for technically sophisticated customers.

You should now be able to access your website over HTTP and HTTPS, backed by our gateway.

HTTPS helps makes sure that nobody between the user and Cloudflare's edge network has tampered with the connection, but it does nothing to make sure that Cloudflare actually serves the content the user asked for. To solve this, we built two connected projects: a modified gateway service and a browser extension.

First, we forked the go-ipfs repository and gave it the ability to offer cryptographic proofs that it was serving content honestly, which it will do whenever it sees requests with the HTTP header X-Ipfs-Secure-Gateway: 1. The simplest case for this is when users request a single file from the gateway by its hash -- all the gateway has to do is serve the content and any metadata that might be necessary to re-compute the given hash.

A more complicated case is when users request a file from a directory. Luckily, directories in IPFS are just files containing a mapping from file name to the hash of the file, and very large directories can be transparently split up into several smaller files, structured into a search tree called a Hash Array Mapped Trie (HAMT). To convince the client that the gateway is serving the contents of the correct file, the gateway first serves the file corresponding to the directory, or every node in the search path if the directory is a HAMT. The client can hash this file (or search tree node), check that it equals the hash of the directory they asked for, and look up the hash of the file they want from within the directory's contents. The gateway then serves the contents of the requested file, which the client can now verify because it knows the expected hash.

Finally, the most complicated case by far is when the client wants to access content by domain name. It's complicated because the protocol for authenticating DNS, called DNSSEC, has very few client-side implementations. DNSSEC is also not widely deployed, even though some registrars make it even easier than setting up HTTPS. In the end, we ended up writing our own simple DNSSEC-validating resolver that's capable of outputting a cryptographically-convincing proof that it did some lookup correctly.

It works the same way as certificate validation in HTTPS: we start at the bottom, with a signature from some authority claiming to be example.com over the DNS records they want us to serve. We then lookup a delegation (DS record) from an authority claiming to be .com, that says "example.com is the authority with these public keys" which is in turn signed by the .com authority's private key. And finally, we lookup a delegation from the root authority, ICANN (whose public keys we already have), attesting to the public keys used by the .com authority. All of these lookups bundled together form an authenticated chain starting at ICANN and ending at the exact records we want to serve. These constitute the proof.

Chain of trust in DNSSEC.

The second project we built out was a browser extension that requests these proofs from IPFS gateways and ipfs-sec domains, and is capable of verifying them. The extension uses the webRequest API to sit between the browser's network stack and its rendering engine, preventing any unexpected data from being show to the user or unexpected code from being executed. The code for the browser extension is available on Github, and can be installed through Firefox's add-on store. We’re excited to add support for Chrome as well, but that can’t move forward until this ticket in their bug tracker is addressed.

On the other hand, if a user doesn't have the extension installed, the gateway won't see the X-Ipfs-Secure-Gateway header and will serve the page like a normal website, without any proofs. This provides a graceful upgrade path to using IPFS, either through our extension that uses a third-party gateway or perhaps another browser extension that runs a proper IPFS node in-browser.

My favorite website on IPFS so far has been the mirror of English Wikipedia put up by the IPFS team at Protocol Labs. It's fast, fun to play with, and above all has practical utility. One problem that stands out though, is that the mirror has no search feature so you either have to know the URL of the page you want to see or try to find it through Google. The Turkish-language mirror has in-app search but it requires a call to a dynamic API on the same host, and doesn't work through Cloudflare's gateway because we only serve static content.

I wanted to provide an example of the kinds of secure, performant applications that are possible with IPFS, and this made building a search engine seem like a prime candidate. Rather than steal Protocol Labs' idea of 'Wikipedia on IPFS', we decided to take the Kiwix archives of all the different StackExchange websites and build a distributed search engine on top of that. You can play with the finished product here: ipfs-sec.stackexchange.cloudflare-ipfs.com.

The way it's built is actually really simple, at least as far as search engines go: We build an inverted index and publish it with the rest of each StackExchange, along with a JavaScript client that can read the index and quickly identify documents that are relevant to a user's query. Building the index takes two passes over the data:

1. The first pass decides what words/tokens we want to allow users to search by. Tokens shouldn't be too popular (like the top 100 words in English), because then the list of all documents containing that token is going to be huge and it's not going to improve the search results anyways. They also shouldn't be too rare (like a timestamp with sub-second-precision), because then the index will be full of meaningless tokens that occur in only one document each. You can get a good estimate of the most frequent K tokens, using only constant-space, with the really simple space-saving algorithm from this paper.

2. Now that the first pass has given us the tokens we want to track, the second pass through the data actually builds the inverted index. That is, it builds a map from every token to the list of documents that contain that token, called a postings list. When a client wants to find only documents that contain some set of words/tokens, they download the list for each individual token and intersect them. It sounds less efficient than it is -- in reality, the postings lists are unnoticeably small (<30kb) even in the worst case. And the browser can 'pipeline' the requests for the postings lists (meaning, send them all off at once) which makes getting a response to several requests about as fast as getting a response to one.

We also store some simple statistics in each postings list to help rank the results. Essentially, documents that contain a query token more often are ranked higher than those that don't. And among the tokens in a query, those tokens that occur in fewer documents have a stronger effect on ranking than tokens that occur in many documents. That's why when I search for "AES SIV" the first result that comes back is:

• "Why is SIV a thing if MAC-and-encrypt is not the most secure way to go?"

while the following is the fourth result, even though it has a higher score and greater number of total hits than first result:

• "Why is AES-SIV not used, but AESKW, AKW1?"

(AES is a very popular and frequently discussed encryption algorithm, while SIV is a lesser-known way of using AES.)

But this is what really makes it special: because the search index is stored in IPFS, the user can convince themselves that no results have been modified, re-arranged, or omitted without having to download the entire corpus. There's one small trick to making this statement hold true: All requests made by the search client must succeed, and if they don't, it outputs an error and no search results.

To understand why this is necessary, think about the search client when it first gets the user's query. It has to tokenize the query and decide which postings lists to download, where not all words in the user's query may be indexed. A naive solution is to try to download the postings list for every word unconditionally, and interpret a non-200 HTTP status code as "this postings list must not exist". In this case, a network adversary could block the search client from being able to access postings lists that lead to undesirable results, causing the client to output misleading search results either through omission or re-arranging.

What we do instead is store the dictionary of every indexed token in a file in the root of the index. The client can download the dictionary once, cache it, and use it for every search afterwards. This way, the search client can consult the dictionary to find out which requests should succeed and only send those.

We were incredibly excited when we realized the new avenues and types of applications that combining IPFS with Cloudflare open up. Of course, our IPFS gateway and the browser extension we built will need time to mature into a secure and reliable platform. But the ability to securely deliver web pages through third-party hosting providers and CDNs is incredibly powerful, and its something cryptographers and internet security professionals have been working towards for a long time. As much fun as we had building it, we're even more excited to see what you build with it.

Subscribe to the blog for daily updates on our announcements.

Today we’re excited to introduce Cloudflare’s IPFS Gateway, an easy way to access content from the InterPlanetary File System (IPFS) that doesn’t require installing and running any special software on your computer. We hope that our gateway, hosted at cloudflare-ipfs.com, will serve as the platform for many new highly-reliable and security-enhanced web applications. The IPFS Gateway is the first product to be released as part of our Distributed Web Gateway project, which will eventually encompass all of our efforts to support new distributed web technologies.

This post will provide a brief introduction to IPFS. We’ve also written an accompanying blog post describing what we’ve built on top of our gateway, as well as documentation on how to serve your own content through our gateway with your own custom hostname.

Usually, when you access a website from your browser, your browser tracks down the origin server (or servers) that are the ultimate, centralized repository for the website’s content. It then sends a request from your computer to that origin server, wherever it is in the world, and that server sends the content back to your computer. This system has served the Internet well for decades, but there’s a pretty big downside: centralization makes it impossible to keep content online any longer than the origin servers that host it. If that origin server is hacked or taken out by a natural disaster, the content is unavailable. If the site owner decides to take it down, the content is gone. In short, mirroring is not a first-class concept in most platforms (Cloudflare’s Always Online is a notable exception).

The InterPlanetary File System aims to change that. IPFS is a peer-to-peer file system composed of thousands of computers around the world, each of which stores files on behalf of the network. These files can be anything: cat pictures, 3D models, or even entire websites. Over 5,000,000,000 files had been uploaded to IPFS already.

There are two key differences between IPFS and the web as we think of it today.

The first is that with IPFS anyone can cache and serve any content—for free. Right now, with the traditional web, most typically rely on big hosting providers in remote locations to store content and serve it to the rest of the web. If you want to set up a website, you have to pay one of these major services to do this for you. With IPFS, anyone can sign up their computer to be a node in the system and start serving data. It doesn’t matter if you’re working on a Raspberry Pi or running the world’s biggest server. You can still be a productive node in the system.

The second key difference is that data is content-addressed, rather than location-addressed. That’s a bit of a subtle difference, but the ramifications are substantial, so it’s worth breaking down.

Currently, when you open your browser and navigate to example.com, you’re telling the browser “fetch me the data stored at example.com’s IP address” (this happens to be 93.184.216.34). That IP address marks where the content you want is stored in the network. You then send a request to the server at that IP address for the “example.com” content and the server sends back the relevant information. So at the most basic level, you tell the network where to look and the network sends back what it found.

IPFS turns that on its head.

With IPFS, every single block of data stored in the system is addressed by a cryptographic hash of its contents, i.e., a long string of letters and numbers that is unique to that block. When you want a piece of data in IPFS, you request it by its hash. So rather than asking the network “get me the content stored at 93.184.216.34,” you ask “get me the content that has a hash value of QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy.” (QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy happens to be the hash of a .txt file containing the string “I’m trying out IPFS”).

Remember that with IPFS, you tell the network what to look for, and the network figures out where to look.

First off, it makes the network more resilient. The content with a hash of QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy could be stored on dozens of nodes, so if one node that was caching that content goes down, the network will just look for the content on another node.

Second, it introduces an automatic level of security. Let’s say you know the hash value of a file you want. So you ask the network, “get me the file with hash QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy” (the example.txt file from above). The network responds and sends the data. When you receive all the data, you can rehash it. If the data was changed at all in transit, the hash value you get will be different than the hash you asked for. You can think of the hash as like a unique fingerprint for the file. If you’re sent back a different file than you were expecting to receive, it’s going to have a different fingerprint. This means that the system has a built-in way of knowing whether or not content has been tampered with.

A Note on IPFS Addresses and Cryptographic Hashes

Since we’ve spent some time going over why this content-addressed system is so special, it’s worth talking a little bit about how the IPFS addresses are built. Every address in IPFS is a multihash, which means that the address combines information about both the hashing algorithm used and the hash output into one string. IPFS multihashes have three distinct parts: the first byte of the mulithash indicates which hashing algorithm has been used to produce the hash; the second byte indicates the length of the hash; and the remaining bytes are the value output by the hash function. By default, IPFS uses the SHA-256 algorithm, which produces a 32-byte hash. This is represented by the string “Qm” in Base58 (the default encoding for IPFS addresses), which is why all the example IPFS addresses in this post are of the form “Qm…”.

While SHA-256 is the standard algorithm used today, this multihash format allows the IPFS protocol to support addresses produced by other hashing algorithms. This allows the IPFS network to move to a different algorithm, should the world discover flaws with SHA-256 sometime in the future. If someone hashed a file with another algorithm, the address of that file would start some characters other than “Qm”.

The good news is that, at least for now, SHA-256 is believed to have a number of qualities that make it a strong cryptographic hashing algorithm. The most important of these is that SHA-256 is collision resistant. A collision occurs when there are two different files that produce the same hash when run through the SHA-256 algorithm. To understand why it’s important to prevent collisions, consider this short scenario. Imagine some IPFS user, Alice, uploads a file with some hash, and another user, Bob, uploads a different file that happens to produce the exact same hash. If this happened, there would be two different files in the network with the exact same address. So if some third person, Carol, sent out an IPFS request for the content at that address, she wouldn't necessarily know whether she was going to receive Bob’s file or Alice’s file.

SHA-256 makes collisions extremely unlikely. Because SHA-256 computes a 256-bit hash, there are 2^256 possible IPFS addresses that the algorithm could produce. Hence, the chance that there are two files in IPFS that produce a collision is low. Very low. If you’re interested in more details, the birthday attack Wikipedia page has a cool table showing exactly how unlikely collisions are, given a sufficiently strong hashing algorithm.

Now that we’ve walked through all the details of what IPFS is, you’re probably wondering how to use it. There are a number of ways to access content that’s been stored in the IPFS network, but we’re going to address two popular ones here. The first way is to download IPFS onto your computer. This turns your machine into a node of the IPFS network, and it’s the best way to interact with the network if you want to get down in the weeds. If you’re interested in playing around with IPFS, the Go implementation can be downloaded here.

But what if you want access to content that’s stored on IPFS without the hassle of operating a node locally on your machine? That’s where IPFS gateways come into play. IPFS gateways are third-party nodes that fetch content from the IPFS network and serve it to you over HTTPS. To use a gateway, you don’t need to download any software or type any code. You simply open up a browser and type in the gateway’s name and the hash of the content you’re looking for, and the gateway will serve the content in your browser.

Say you know you want to access the example.txt file from before, which has the hash QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy, and there’s a public gateway that is accessible at https://example-gateway.com

To access that content, all you need to do is open a browser and type

https://example-gateway.com/ipfs/QmXnnyufdzAWL5CqZ2RnSNgPbvCc1ALT73s6epPrRnZ1Xy

and you’ll get back the data stored at that hash. The combination of the /ipfs/ prefix and the hash is referred to as the file path. You always need to provide a full file path to access content stored in IPFS.

At the most basic level, you can access any of the billions of files stored on IPFS from your browser. But that’s not the only cool thing you can do. Using Cloudflare’s gateway, you can also build a website that’s hosted entirely on IPFS, but still available to your users at a custom domain name. Plus, we’ll issue any website connected to our gateway a free SSL certificate, ensuring that each website connected to Cloudflare's gateway is secure from snooping and manipulation. For more on that, check out the Distributed Web Gateway developer docs.

A fun example we’ve put together using the Kiwix archives of all the different StackExchange websites and build a distributed search engine on top of that using only IPFS. Check it out here.

IPFS is a peer-to-peer network, so there is the possibility of users sharing abusive content. This is not something we support or condone. However, just like how Cloudflare works with more traditional customers, Cloudflare’s IPFS gateway is simply a cache in front of IPFS. Cloudflare does not have the ability to modify or remove content from the IPFS network. If any abusive content is found that is served by the Cloudflare IPFS gateway, you can use the standard abuse reporting mechanism described here.

IPFS is only one of a family of technologies that are embracing a new, decentralized vision of the web. Cloudflare is excited about the possibilities introduced by these new technologies and we see our gateway as a tool to help bridge the gap between the traditional web and the new generation of distributed web technologies headlined by IPFS. By enabling everyday people to explore IPFS content in their browser, we make the ecosystem stronger and support its growth. Just like when Cloudflare launched back in 2010 and changed the game for web properties by providing the security, performance, and availability that was previously only available to the Internet giants, we think the IPFS gateway will provide the same boost to content on the distributed web.

Dieter Shirley, CTO of Dapper Labs and Co-founder of CryptoKitties said the following:

We’ve wanted to store CryptoKitty art on IPFS since we launched, but the tech just wasn’t ready yet. Cloudflare’s announcement turns IPFS from a promising experiment into a robust tool for commercial deployment. Great stuff!

The IPFS gateway is exciting, but it’s not the end of the road. There are other equally interesting distributed web technologies that could benefit from Cloudflare’s massive global network and we’re currently exploring these possibilities. If you’re interested in helping build a better internet with Cloudflare, we’re hiring!

Subscribe to the blog for daily updates on our announcements.

Photo by Mink Mingle / Unsplash

Cloudflare recently updated the authoritative DNS service to support nine new record types. Since these records are less commonly used than what we previously supported, we thought it would be a good idea to do a brief explanation of each record type and how it is used.

DNSKEY and DS work together to allow you to enable DNSSEC on a child zone (subdomain) that you have delegated to another Nameserver. DS is useful if you are delegating DNS (through an NS record) for a child to a separate system and want to keep using DNSSEC for that child zone; without a DS entry in the parent, the child data will not be validated. We’ve blogged about the details of Cloudflare’s DNSSEC implementation and why it is important in the past, and this new feature allows for more flexible adoption for customers who need to delegate subdomains.

Today, there is no way to restrict which TLS (SSL) certificates are trusted to be served for a host. For example if an attacker were able to maliciously generate an SSL certificate for a host, they could use a on-path attacker attack to appear as the original site. With SSHFP, TLSA, SMIMEA, and CERT, a website owner can configure the exact certificate public key that is allowed to be used on the domain, stored inside the DNS and secured with DNSSEC, reducing the risk of these kinds of attacks working.

It is critically important that if you rely on these types of records that you enable and configure DNSSEC for your domain.

This type of record is an answer to the question “When I’m connecting via SSH to this remote machine, it’s authenticating me, but how do I authenticate it?” If you’re the only person connecting to this machine, your SSH client will compare the fingerprint of the public host key to the one it kept in the known_hosts file during the first connection. However across multiple machines or multiple users from an organization, you need to verify this information against a common source of trust. In essence, you need the equivalent of the authentication that a certificate authority provides by signing an HTTPS certificate, but for SSH. Although it’s possible to set certificate authorities for SSH and to have them sign public host keys, another way is to publish the fingerprint of the keys in the domain via the SSHFP record type.

Again, for these fingerprints to be trustworthy it is important to enable DNSSEC on your zone.

The SSHFP record type is similar to TLSA record. You are specifying the algorithm type, the signature type, and then the signature itself within the record for a given hostname.

If the domain and remote server have SSHFP set and you are running an SSH client (such as OpenSSH 5.1+) that supports it, you can now verify the remote machine upon connection by adding the following parameters to your connection:

❯ ssh -o "VerifyHostKeyDNS=yes" -o "StrictHostKeyChecking=yes" [insertremoteserverhere]

TLSA records were designed to specify which keys are allowed to be used for a given domain when connecting via TLS. They were introduced in the DANE specification and allow domain owners to announce which certificate can and should be used for specific purposes for the domain. While most major browsers do not support TLSA, it may still be valuable for non browser specific applications and services.

For example, I’ve set a TLSA record for the domain hasvickygoneonholiday.com for TCP traffic over port 443. There are a number of ways to generate the record, but the easiest is likely through Shumon Huque’s tool.

For most of the examples in this post we will be using kdig rather than the ubiquitous dig. Generally preinstalled dig versions can be old and may not handle newer record types well. If your queries do not quite match up, you should either upgrade your version of dig or install knot.

;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 2218
;; Flags: qr rd ra ad; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 1

;; QUESTION SECTION:
;; _443._tcp.hasvickygoneonholiday.com. 	IN	TLSA

;; ANSWER SECTION:
_443._tcp.hasvickygoneonholiday.com. 300	IN	TLSA	3 1 1 4E48ED671DFCDF6CBF55E52DBC8B9C9CC21121BD149BC24849D1398DA56FB242
_443._tcp.hasvickygoneonholiday.com. 300	IN	RRSIG	TLSA 13 4 300 20180803233834 20180801213834 35273 hasvickygoneonholiday.com. JvC9mZLfuAyEHZUZdq4n8kyRbF09vwgx4c1fas24Ag925LILr1armjHbr7ZTp8ycS/Go3y3lgyYCuBeW/vT/3w==

;; Received 232 B
;; Time 2018-08-02 15:38:34 PDT
;; From 192.168.1.1@53(UDP) in 28.5 ms

From the above request and response, we can see that a) the response for the zone is secured and signed with DNSSEC (Flag: ad) and that I should be verifying a certificate with the public key (3 1 1) SHA256 hash (3 1 1) of 4E48ED671DFCDF6CBF55E52DBC8B9C9CC21121BD149BC24849D1398DA56FB242. We can use openssl (v1.1.x or higher) to verify the results:

❯ openssl s_client  -connect hasvickygoneonholiday.com:443 -dane_tlsa_domain "hasvickygoneonholiday.com" -dane_tlsa_rrdata "
3 1 1 4e48ed671dfcdf6cbf55e52dbc8b9c9cc21121bd149bc24849d1398da56fb242"
CONNECTED(00000003)
depth=0 C = US, ST = CA, L = San Francisco, O = "CloudFlare, Inc.", CN = hasvickygoneonholiday.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=hasvickygoneonholiday.com
   i:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=CloudFlare Inc ECC CA-2
 1 s:/C=US/ST=CA/L=San Francisco/O=CloudFlare, Inc./CN=CloudFlare Inc ECC CA-2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE7jCCBJSgAwIBAgIQB9z9WxnovNf/lt2Lkrfq+DAKBggqhkjOPQQDAjBvMQsw
...
---
SSL handshake has read 2666 bytes and written 295 bytes
Verification: OK
Verified peername: hasvickygoneonholiday.com
DANE TLSA 3 1 1 ...149bc24849d1398da56fb242 matched EE certificate at depth 0

SMIMEA records function similar to TLSA but are specific to email addresses. The domain for these records should be prefixed by “_smimecert.” and specific formatting is required to attach a SMIMEA record to an email address. The local-part (username) of the email address must be treated in a specific format and SHA-256 hashed as detailed in the RFC. From the RFC example: “ For example, to request an SMIMEA resource record for a user whose email address is "hugh@example.com", an SMIMEA query would be placed for the following QNAME: c93f1e400f26708f98cb19d936620da35eec8f72e57f9eec01c1afd6._smimecert.example.com

CERT records are used for generically storing certificates within DNS and are most commonly used by systems for email encryption. To create a CERT record, you must specify the certificate type, the key tag, the algorithm, and then the certificate, which is either the certificate itself, the CRL, a URL of the certificate, or fingerprint and a URL.

PTR (Pointer) records are pointers to canonical names. They are similar to CNAME in structure, meaning they only contain one FQDN (fully qualified domain name) but the RFC dictates that subsequent lookups are not done for PTR records, the value should just be returned back to the requestor. This is different to a CNAME where a recursive resolver would follow the target of the canonical name. The most common use of a PTR record is in reverse DNS, where you can look up which domains are meant to exist at a given IP address. These are useful for outbound mailservers as well as authoritative DNS servers.

It is only possible to delegate the authority for IP addresses that you own from your Regional Internet Registry (RIR). Creating reverse zones and PTR records for IPs that you can not (or do not) delegate does not serve any practical purpose.

For example, looking up the A record for marek.ns.cloudflare.com gives us the IP of 173.245.59.202.

❯ kdig a marek.ns.cloudflare.com +short
173.245.59.202

Now imagine we want to know if the owner of this IP ‘authorizes’ marek.ns.cloudflare.com to point to it. Reverse Zones are specifically crafted child zones within in-addr.arpa. (for IPv4) and ip6.arpa. (for IPv6) whom are delegated via the Regional Internet Registries to the owners of the IP address space. That is to say if you own a /24 from ARIN, ARIN will delegate the reverse zone space for your /24 to you to control. The IPv4 address is represented inverted as the subdomain in in-addr.arpa. Since Cloudflare owns the IP, we’ve delegated the reverse zone and created a PTR there.

❯ kdig -x 173.245.59.202
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 18658
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; 202.59.245.173.in-addr.arpa.		IN	PTR

;; ANSWER SECTION:
202.59.245.173.in-addr.arpa.	1222	IN	PTR	marek.ns.cloudflare.com.

For completeness, here is the +trace for the 202.59.245.173.in-addr.arpa zone. We can see that the /24 59.245.173.in-addr.arpa has been delegated to Cloudflare from ARIN:

❯ dig 202.59.245.173.in-addr.arpa +trace

; <<>> DiG 9.8.3-P1 <<>> 202.59.245.173.in-addr.arpa +trace
;; global options: +cmd
.			48419	IN	NS	a.root-servers.net.
.			48419	IN	NS	b.root-servers.net.
.			48419	IN	NS	c.root-servers.net.
.			48419	IN	NS	d.root-servers.net.
.			48419	IN	NS	e.root-servers.net.
.			48419	IN	NS	f.root-servers.net.
.			48419	IN	NS	g.root-servers.net.
.			48419	IN	NS	h.root-servers.net.
.			48419	IN	NS	i.root-servers.net.
.			48419	IN	NS	j.root-servers.net.
.			48419	IN	NS	k.root-servers.net.
.			48419	IN	NS	l.root-servers.net.
.			48419	IN	NS	m.root-servers.net.
;; Received 228 bytes from 2001:4860:4860::8888#53(2001:4860:4860::8888) in 25 ms

in-addr.arpa.		172800	IN	NS	e.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	d.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	b.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	f.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	c.in-addr-servers.arpa.
in-addr.arpa.		172800	IN	NS	a.in-addr-servers.arpa.
;; Received 421 bytes from 192.36.148.17#53(192.36.148.17) in 8 ms

173.in-addr.arpa.	86400	IN	NS	u.arin.net.
173.in-addr.arpa.	86400	IN	NS	arin.authdns.ripe.net.
173.in-addr.arpa.	86400	IN	NS	z.arin.net.
173.in-addr.arpa.	86400	IN	NS	r.arin.net.
173.in-addr.arpa.	86400	IN	NS	x.arin.net.
173.in-addr.arpa.	86400	IN	NS	y.arin.net.
;; Received 165 bytes from 199.180.182.53#53(199.180.182.53) in 300 ms

59.245.173.in-addr.arpa. 86400	IN	NS	ns1.cloudflare.com.
59.245.173.in-addr.arpa. 86400	IN	NS	ns2.cloudflare.com.
;; Received 95 bytes from 2001:500:13::63#53(2001:500:13::63) in 188 ms

Naming Authority Pointer Records are used in conjunction with SRV records, generally as a part of the SIP protocol. NAPTR records point to domains to specific services, if available for that domain. Anders Brownworth has an excellent description in detail on his blog. The start of his example, with his permission:

Let’s consider a call to 2125551212@example.com. Given only this address though, we don't know what IP address, port or protocol to send this call to. We don't even know if example.com supports SIP or some other VoIP protocol like H.323 or IAX2. I'm implying that we're interested in placing a call to this URL but if no VoIP service is supported, we could just as easily fall back to emailing this user instead. To find out, we start with a NAPTR record lookup for the domain we were given:

#host -t NAPTR example.com
example.com NAPTR 10 100 "S" "SIP+D2U" "" _sip._udp.example.com.
example.com NAPTR 20 100 "S" "SIP+D2T" "" _sip._tcp.example.com.
example.com NAPTR 30 100 "S" "E2U+email" "!^.*$!mailto:info@example.com!i" _sip._tcp.example.com.

Here we find that example.com gives us three ways to contact example.com, the first of which is "SIP+D2U" which would imply SIP over UDP at _sip._udp.example.com.

Uniform Resource Identifier records are commonly used as a compliment to NAPTR records and per the RFC, can be used to replace SRV records. As such, they contain a Weight and Priority field as well as Target, similar to SRV.

One use case is proposed by this draft RFC is to replace SRV records with URI records for discovering Kerberos key distribution centers (KDC). It minimizes the number of requests over SRV records and allows the domain owner to specify preference for TCP or UDP.

In the below example, it specifies that we should use a KDC on TCP at the default port and UDP on port 89 should the primary connection fail.

❯ kdig URI _kerberos.hasvickygoneonholiday.com
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 8450
;; Flags: qr rd ra; QUERY: 1; ANSWER: 2; AUTHORITY: 0; ADDITIONAL: 0

;; QUESTION SECTION:
;; _kerberos.hasvickygoneonholiday.com. 	IN	URI

;; ANSWER SECTION:
_kerberos.hasvickygoneonholiday.com. 283	IN	URI	1 10 "krb5srv:m:tcp:kdc.hasbickygoneonholiday.com"
_kerberos.hasvickygoneonholiday.com. 283	IN	URI	1 20 "krb5srv:m:udp:kdc.hasbickygoneonholiday.com:89"

Cloudflare now supports CERT, DNSKEY, DS, NAPTR, PTR, SMIMEA, SSHFP, and TLSA in our authoritative DNS products. We would love to hear if you have any interesting example use cases for the new record types and what other record types we should support in the future.

Our DNS engineering teams in London and San Francisco are both hiring if you would like to contribute to the fastest authoritative and recursive DNS services in the world.

Software Engineer

Photo by Niko Soikkeli / Unsplash

The root of the DNS tree has been using DNSSEC to protect the zone content since 2010. DNSSEC is simply a mechanism to provide cryptographic signatures alongside DNS records that can be validated, i.e. prove the answer is correct and has not been tampered with. To learn more about why DNSSEC is important, you can read our earlier blog post.

Today, the root zone is signed with a 2048 bit RSA “Trust Anchor” key. This key is used to sign further keys and is used to establish the Chain of trust that exists in the public DNS at the moment.

With access to this root Trust Anchor, it would be possible to re-sign the DNS tree and tamper with the content of DNS records on any domain, implementing an on-path DNS attack… without causing recursors and resolvers to consider the data invalid.

As explained in this blog the key is very well protected with eye scanners and fingerprint readers and fire-breathing dragons patrolling the gate (okay, maybe not dragons). Operationally though, the root zone uses two different keys, the mentioned Trust Anchor key (that is called the Key Signing Key or KSK for short) and the Zone Signing Key (ZSK).

The ZSK (Zone Signing Key) is used to generate signatures for all of the Resource Records (RRs) in a zone.

You can query for the DNSSEC signature (the RRSIG record) of “www.cloudflare.com” using your friendly dig command.

$ dig www.cloudflare.com +dnssec

;; QUESTION SECTION:
;www.cloudflare.com.		IN	A
;; ANSWER SECTION:
www.cloudflare.com.	4	IN	A	198.41.215.162
www.cloudflare.com.	4	IN	A	198.41.214.162
www.cloudflare.com.	4	IN	RRSIG	A 13 3 5 20180207170906 20180205150906 35273 cloudflare.com. 4W4mJXJRnd/wHnDyNo5minGvZY6hVNSXITnUI+pO6fzhnkpsEp1ko8K7 1PQ6r0s9SwLgrgfneqXyPs4b5X0YDw==

The two A records shown here can be cryptographically verified using the RRSIG and ZSK in the zone. The ZSK can itself be verified using the KSK, and so on… this continues upwards following the “chain of trust” until the root KSK is found.

The http://dnsviz.net/ tool can be used to help visualize how this verification can be done for any domain on the internet, for example here is the trust chain for “www.cloudflare.com”.

To verify the RRSIG on “www.cloudflare.com” we would need to cryptographically verify the signatures in reverse order on the diagram. First “cloudflare.com”, then “com”, and finally “.” – the root zone.

If you are able to access the secret key that’s used to sign the root, it’s possible to trick resolvers into verifying a "forged" answer.

While this DNSSEC signing has been deployed on the root zone, for over seven years, there is one operation that has never been attempted: rolling the Key Signing Key. This means to generate a new key and update every part of DNS infrastructure on the internet that needs it, retiring the old one completely.

The ZSK (Zone Signing Key) has been rolled religiously every quarter since 2010, however rolling the Key Signing Key is a much scarier operation. If it goes wrong it could leave the root zone signing invalid, meaning a large part of the internet would not trust any of the content, effectively knocking DNS offline for validating resolvers. After DNSSEC was designed, a mechanism was devised for rolling out a new Key Signing Key in RFC5011, this operation is commonly known as the 5011 roll-over.

All cryptographic keys have a life cycle that can represented by states:Generated == the key is created but only the “owner” knows of its properties.Published == the key has been made public either as a public key or a hash of it.Active == the key is in useRetired == the has been withdrawn from service but is still publishedRevoked == they key has been marked as not to be trusted ever again.Removed == taken out of publication

Different keys move through the states in different ways depending on the usage, for example some keys are never revoked, just removing them is sufficient, for example the root ZSK’s are never revoked. When rolled, the root KSK will pass through all states.

For most keys used in DNS the trust is derived by a relationship between the parent zone and the child zone. The parent publishes a special record, the DS (Delegation Signer), that contains cryptographically strong binding to the actual key, a hash. The child has a DNSKEY RRset at the top of its zone that has at least one key that matches one of the DS records in the parent. To complete the chain of trust the DNSKEY RRset MUST be signed by that key.

The root zone has no parent, thus trust cannot be derived in the same way. Instead, validating resolvers must be configured with the root Trust Anchor. This anchor must be refreshed during a key rollover or the validating resolver will not trust anything it sees in the root zone after the old KSK (from 2010) is retired from service. The Trust Anchors can be updated in a number of ways, such as a manual update, a software update, or an in-band update. The preferred update mechanism is the previously mentioned in-band update mechanism RFC5011-roll.

The process outlined in RFC 5011 relies on two factors, first that the new key is published in the DNSKEY RRset – which is signed by the old KSK, and is kept there for at least a hold-down period of 30 days. Validating resolvers that follow the procedure will check frequently to see if there is a new KSK in the DNSKEY set. The new key can be trusted because it has been signed with a key that is already in service. When there is new key, it is placed in PendingAddition state If at any point one of the key’s in PendingAddition is removed from the DNSKEY set, the resolver will forget about it. This means that if the key were to appear again, it would start a new 30 day hold-down period.

After the key has been in PendingAddtional for 30 consecutive days it is accepted into Active state and will be trusted to sign the DNSKEY set for the root. From this point onwards, the new key can be used to sign the Zone Signing Key, and in turn the root zone content itself.

There are two main reasons;

1. The community wants to be a sure that the RFC5011 mechanism works in practice. Knowing this makes future rollovers possible, and less risky. Regular rollovers are something to be done as a matter of good key hygiene, like changing your password regularly

2. Enables thinking about switching to different algorithms. RSA with a large key size is a strong algorithm, but using it causes DNS packets to be larger. There are other algorithms like the ones that Cloudflare uses that are based on elliptic curves have smaller keys but increased safety per bit. To switch to a new algorithm would require a new key.

Some people advocated rolling the key and changing the algorithm at the same time but that was deemed too risky. The right time to start talking about that is after the current roll concludes successfully.

ICANN started the rollover process last year. The new keys has been created and replicated to all the HSM’s (Hardware Security Modules) in the two facilities that ICANN operates. From now on we will use the terms KSK2010 (the old key) and KSK2017 (the new key).

Before starting the roll-over process, testing of RFC5011 implementations took place and most implementations reported success.

The new key was published in DNS on July 11th 2017, thus the DNSKEY set now contains two KSKs. At that point the new key/KSK2017 has entered Published state. It was scheduled to become Active on October 11th 2017. Any validating resolver that has been operating for at least 30 days during the July 11-October 11 window should have placed the new Trust Anchor in “Active” state before October 11th. But sometimes things do not go according to plan.

One of the things that was put in place before the rollover was a way for resolvers to signal to authoritative servers what trust anchor the resolver trusts RFC8145. RFC8145 was only published in April 2017, thus during the KSK2017 key publication phase, only the latest version of Bind-9 supported it by default.

The mechanism works by resolvers periodically sending a query to the root nodes, with a query name formatted like “_ta-4a5c” or “_ta-4a5c-4f66”. The name contains HEX encoded versions of the Trust Anchor identifiers, 19036 and 20326 respectively. This at least allows root operators to estimate the % of resolvers that have implemented RFC8145 AND are aware of each Trust Anchor.

On September 29 ICANN postponed the roll based on evidence from the resolvers that sent in reports.It was concerning that the latest and greatest version of Bind-9 in 4% of cases did not pick up the new Trust Anchor, this was explained in more detail in a DNS-OARC presentation. But this still leaves us with the question, why?

It is also important to note that although other implementations of RFC8145 did not enable it by default thus most of the reports were by Bind-9.

Rolling the KSK at this point would have resulted in the remaining resolvers not trusting the content of the root zone, ultimately breaking all DNS resolution through them.

At Cloudflare we operate validating resolvers in all of our >120 data centers, and we monitored the adoption of trust anchors on a weekly basis, expecting everything to work correctly. After 6 weeks we noticed that things were not going right, some of the resolvers had picked up the new trust anchor and others had not accepted the new trust anchor even though more than enough time had passed.

First let’s look at the assumptions that RFC5011 makes.

• The resolver is a long running process that understands time and and can keep state

• The resolver has access to persistent writeable storage that will work across reboots.

In the protocol community we had worried about the first one a lot, for the second one we had identified two failure cases: machine configured from old read-only medium, and new machine takes over. Both were considered rare enough and operators would know to deal with those exceptions.

Turns out the second assumption in RFC5011 had more failure modes than the community expected.

For example in Bind-9, it originally had a hardcoded list of “trusted-keys”. Later when RFC5011 support was added the configuration option “managed-keys” was added. It looks like some installations while religiously updating the software never changed from the fixed configuration to the RFC5011 managed one. In this case the only recovery is to change the configuration, and in some cases the operator selected this operating mode assuming he/she would distribute a new configuration file during rollover, but the person may have left or forgotten.

Software that uses managed-keys operations (Bind-9, Unbound, Knot-resolver) uses a file to maintain state between restarts. BUT it is possible that the file is read-only and in that case managed-keys works just like trusted-keys. Why anyone would have a configuration like that is a good question? The interesting obersevation is that unless the implementation complains loudly about the read-only state, the operator is not likely to notice. The only recovery option here is to change the configuration so the trust anchor file can be written.

Software upgrades are another possible reason for not picking up the new trust anchor, but only if the file containing the Trust Anchor state is overwritten or lost. This can happen if the resolver machine has a disk replacement/reformat etc. but in this case the net effect is only slowing down the acceptance of the new trust anchor. This failure is visible as as KSK2017 spends more than 30 days in state “PendingAddition” but that is only visible if someone is looking.

Modern operating practices use “containers” that are spun up and down, in those cases there is no “persistent” storage. To avoid validation errors in this case the software installed must know about the new key or perform a key discovery upon startup like the unbound-anchor program performs for Unbound.

There are probably few other reasons where operations may cause the errors seen by the Trust Anchor Signaling.

Back to what happened at Cloudflare? In our case the issue was a combination of upgrades and container issues. We were upgrading software on all our nodes and our resolver processes were allocated to different computers. Our fix was to quickly upgrade to a software version that knew about the new trust anchor, so future restarts/migrations would not cause loss in trust.

ICANN has just asked for comments on restarting the rollover process and perform the roll on October 11th 2018.

What can you do to prepare for the key rollover?If you operate a validating resolver, make sure you have the latest version of your vendors software, audit the configuration files and file permissions and check that your software supports both KSK2010 (key tag 19036) and KSK2017 (key tag 20326).

If you are a concerned end user right now there is nothing you can do, but the IETF is considering a proposal to allow remote trust anchor checking via queries. Hopefully this will be standardized soon and DNS resolver software vendors add support, but until then there is no testing possible by you.

If you speak languages other than English and you worry about your local operators should know about the DNSSEC Key Rollover failure modes, feel free to republish this blog or parts of it in your language.

Every quarter there is a new KSK signing ceremony where signatures for 3 months of use of the KSK are generated. February 6th 2018 is the next one and it will sign a DNSKEY set with both KSKs but only signed by KSK2010 . You can see the script for the ceremony here and you can even watch it online. But the fun part of this particular ceremony is the destruction of old HSM (Hardware Security Module), via some fancy contraption.

An HSM is a special kind of equipment that can store private keys and never leak them, and protects its secrets by erasing them when someone tries to access/tamper with the equipment. The secrets remain in the HSM as long as a non-replaceable battery lasts. The old KSK HSMs have a lifetime of 10 years and were made in late 2009 or early 2010 thus the battery is not designed to last much longer. Last year the private keys were safely and securly moved to newer models and the new machines have been in use for about a year. The final step of retiring the old machines is to destroy them during the ceremony, tune in to see how that is done.

Excited by working on cutting edge stuff? Or building systems at a scale where once-in-a-decade problems can get triggered every day? Then join our team.

CC BY 2.0 image by ajmexico, inspired by

This is an old question and the IPv4 RFCs answer it pretty clearly. The idea was to split the problem into two separate concerns:

• What is the maximum packet size that can be handled by operating systems on both ends?

• What is the maximum permitted datagram size that can be safely pushed through the physical connections between the hosts?

When a packet is too big for a physical link, an intermediate router might chop it into multiple smaller datagrams in order to make it fit. This process is called "forward" IP fragmentation and the smaller datagrams are called IP fragments[1].

Image by Geoff Huston, reproduced with permission

The IPv4 specification defines the minimal requirements. From the RFC791:

Every internet destination must be able to receive a datagram
of 576 octets either in one piece or in fragments to
be reassembled. [...]

Every internet module must be able to forward a datagram of 68
octets without further fragmentation. [...]

The first value - permitted reassembled packet size - is typically not problematic. IPv4 defines the minimum as 576 bytes, but popular operating systems can cope with very big packets, typically up to 65KiB.

The second one is more troublesome. All physical connections have inherent datagram size limits, depending on the specific medium they use. For example Frame Relay can send datagrams between 46 and 4,470 bytes. ATM uses fixed 53 bytes, classical Ethernet can do between 64 and 1500 bytes.

The spec defines the minimal requirement - each physical link must be able to transmit datagrams of at least 68 bytes. For IPv6 that minimal value has been bumped up to 1,280 bytes (see RFC2460).

On the other hand, the maximum datagram size that can be transmitted without fragmentation is not defined by any specification and varies by link type. This value is called the MTU (Maximum Transmission Unit)[2].

The MTU defines a maximum datagram size on a local physical link. The internet is created from non-homogeneous networks, and on the path between two hosts there might be links with shorter MTU values. The maximum packet size that can be transmitted without fragmentation between two remote hosts is called a Path MTU, and can potentially be different for every connection.

One might think that it's fine to build applications that transmit very big packets and rely on routers to perform the IP fragmentation. This is not a good idea. The problems with this approach were first discussed by Kent and Mogul in 1987. Here are a couple of highlights:

• To successfully reassemble a packet, all fragments must be delivered. No fragment can become corrupt or get lost in-flight. There simply is no way to notify the other party about missing fragments!

• The last fragment will almost never have the optimal size. For large transfers this means a significant part of the traffic will be composed of suboptimal short datagrams - a waste of precious router resources.

• Before the re-assembly a host must hold partial, fragment datagrams in memory. This opens an opportunity for memory exhaustion attacks.

• Subsequent fragments lack the higher-layer header. TCP or UDP header is only present in the first fragment. This makes it impossible for firewalls to filter fragment datagrams based on criteria like source or destination ports.

A more elaborate description of IP fragmentation problems can be found in these articles by Geoff Huston:

• Evaluating IPv4 and IPv6 packet fragmentation

• Fragmenting IPv6

Image by Geoff Huston, reproduced with permission

A solution to these problems was included in the IPv4 protocol. A sender can set the DF (Don't Fragment) flag in the IP header, asking intermediate routers never to perform fragmentation of a packet. Instead a router with a link having a smaller MTU will send an ICMP message "backward" and inform the sender to reduce the MTU for this connection.

The TCP protocol always sets the DF flag. The network stack looks carefully for incoming "Packet too big"[3] ICMP messages and keeps track of the "path MTU" characteristic for every connection[4]. This technique is called "path MTU discovery", and it is mostly commonly used for TCP, although it can also be applied to other IP-based protocols. Being able to deliver the ICMP "Packet too big" messages is critical in keeping the TCP stack working optimally.

In a perfect world, internet connected devices would cooperate and correctly handle fragment datagrams and the associated ICMP packets. In reality though, IP fragments and ICMP packets are very often filtered out.

This is because the modern internet is much more complex than anticipated 36 years ago. Today, basically nobody is plugged directly into the public internet.

Customer devices connect through home routers which do NAT (Network Address Translation) and usually enforce firewall rules. Increasingly often there is more than one NAT installation on the packet path (e.g. carrier-grade NAT). Then, the packets hit the ISP infrastructure where there are ISP "middle boxes". They perform all manner of weird things on the traffic: enforce plan caps, throttle connections, perform logging, hijack DNS requests, implement government-mandated web site bans, force transparent caching or arguably "optimize" the traffic in some other magical way. The middle boxes are used especially by mobile telcos.

Similarly, there are often multiple layers between a server and the public internet. Service providers sometimes use Anycast BGP routing. That is: they handle the same IP ranges from multiple physical locations around the world. Within a datacenter on the other hand it's increasingly popular to use ECMP Equal Cost Multi Path for load balancing.

Each of these layers between a client and server can cause a Path MTU problem. Allow me to illustrate this with four scenarios.

In the first scenario, a client uploads some data to the server using TCP so the DF flag is set on all of the packets. If the client fails to predict an appropriate MTU, an intermediate router will drop the big packets and send an ICMP “Packet too big” notification back to the client. These ICMP packets might get dropped by misconfigured customer NAT devices or ISP middle boxes.

According to the paper by Maikel de Boer and Jeffrey Bosma from 2012 around 5% of IPv4 and 1% of IPv6 hosts block inbound ICMP packets.

My experience confirms this. ICMP messages are indeed often dropped for perceived security advantages, but this is relatively easy to fix. A bigger issue is with certain mobile ISPs with weird middle boxes. These often completely ignore ICMP and perform very aggressive connection rewriting. For example Orange Polska not only ignores inbound "Packet too big" ICMP messages, but also rewrites the connection state and clamps the MSS to a non-negotiable 1344 bytes.

In next scenario, a client uploads some data with a protocol other than TCP, which has the DF flag cleared. For example, this might be a user playing a game using UDP, or having a voice call. The big outbound packets might get fragmented at some point in the path.

We can emulate this by launching ping with a large payload size:

$ ping -s 2048 facebook.com

This particular ping will fail with payloads bigger than 1472 bytes. Any larger size will get fragmented and won’t get delivered properly. There are multiple reasons why servers might mishandle fragments, but one of a popular problems is the use of ECMP load balancing. Due to the ECMP hashing, the first datagram containing a protocol header is likely to be load-balanced to a different server than the rest of the fragments, preventing the reassembly.

For a more detailed discussion of this issue, see:

• Our previous write up on ECMP.

• How Google attempts to solve ECMP fragmentation issues with Maglev L4 Load balancer.

Furthermore, server and router misconfiguration is a significant issue. According to RFC7852 between 30% and 55% of servers drop IPv6 datagrams containing fragmentation header.

The next scenario is about a client downloading some data over TCP. When the server fails to predict the correct MTU, it should receive an ICMP “Packet too big” message. Easy, right?

Sadly, it's not, again due to ECMP routing. The ICMP message will most likely get delivered to the wrong server - the 5-tuple hash of ICMP packet will not match the 5-tuple hash of the problematic connection. We wrote about this in the past, and developed a simple userspace daemon to solve it. It works by broadcasting the inbound ICMP “Packet too big” notification to all the ECMP servers, hoping that the one with the problematic connection will see it.

Additionally due to Anycast routing, the ICMP might be delivered to the wrong datacenter altogether! Internet routing is often asymmetric and the best path from an intermediate router might direct the ICMP packets to the wrong place.

Missing ICMP “Packet too big” notifications can result in connections stalling and timing out. This is often called a PMTU blackhole. To aid this pessimistic case Linux implements a workaround - MTU Probing RFC4821. MTU Probing tries to automatically identify packets dropped due to the wrong MTU, and uses heuristics to tune it. This feature is controlled via a sysctl:

$ echo 1 > /proc/sys/net/ipv4/tcp_mtu_probing

But MTU probing is not without its own issues. First, it tends to miscategorize congestion-related packet loss as MTU issues. Long running connections tend to end up with a reduced MTU. Secondly, Linux does not implement MTU Probing for IPv6.

Finally, there is a situation where the server sends big packets using a non-TCP protocol with the DF bit clear. In this scenario, the big packets will get fragmented on the path to the client. This situation is best illustrated with big DNS responses. Here are two DNS requests that will generate large responses and be delivered to the client as multiple IP fragments:

$ dig +notcp +dnssec DNSKEY org @199.19.56.1
$ dig +notcp +dnssec DNSKEY org @2001:500:f::1

These requests might fail due to already mentioned the misconfigured home router, broken NAT, broken ISP installations, or too restrictive firewall settings.

According to Boer and Bosma around 6% of IPv4 and 10% of IPv6 hosts block inbound fragment datagrams.

Here are some links with more information about the specific fragmentation issues affecting DNS:

• DNS-OARC Reply Size Test

• IPv6, Large UDP Packets and the DNS

With all these things going wrong, how does the internet still manage to work?

CC BY-SA 3.0, source: Wikipedia

This mainly due to the success of Ethernet. The great majority of the links in the public internet are Ethernet (or derived from it) and support the MTU of 1500 bytes.

If you blindly assume the MTU of 1500[5], you will be surprised how often it will work just fine. The internet keeps on working mostly because we are all using an MTU of 1500 and rarely need to do IP fragmentation and send ICMP messages.

This stops working on an unusual setup with links having a non-standard MTU. VPNs and other network tunnel software must be careful to ensure that the fragments and ICMP messages are working fine.

This is especially visible in the IPv6 world, where many users connect through tunnels. Having a healthy passage of ICMP in both ways is very important, especially since fragmentation in IPv6 basically doesn't work (we cited two sources claiming between 10% and 50% of IPv6 hosts block IPv6 Fragment header).

Since the Path MTU issues in IPv6 are so common, many IPv6 servers clamp down Path MTU to the protocol mandated minimum of 1280 bytes. This approach trades a bit of performance for best reliability.

To help explore and debug these issues, we built an online checker. You can find two versions of the test:

• IPv4 version: http://icmpcheck.popcount.org

• IPv6 version: http://icmpcheckv6.popcount.org

These sites launch two tests:

• The first test will deliver ICMP messages to your computer, with the intention of reducing the Path MTU to a laughingly small value.

• The second test will send fragment datagrams back to you.

Receiving a "pass" in both these tests should give you a reasonable assurance that the internet on your side of the cable is behaving well.

It's also easy to run the tests from command line, in case you want to run it on the server:

perl -e "print 'packettoolongyaithuji6reeNab4XahChaeRah1diej4' x 180" > payload.bin
curl -v -s http://icmpcheck.popcount.org/icmp --data @payload.bin
curl -v -s http://icmpcheckv6.popcount.org/icmp --data @payload.bin

This should reduce the path MTU to our server to 905 bytes. You can to verify this by looking into the routing cache table. On Linux you do this with:

ip route get `dig +short icmpcheck.popcount.org`

It's possible to clear the routing cache on Linux:

ip route flush cache to `dig +short icmpcheck.popcount.org`

The second test verifies if fragments are properly delivered to the client:

curl -v -s http://icmpcheck.popcount.org/frag -o /dev/null
curl -v -s http://icmpcheckv6.popcount.org/frag -o /dev/null

In this blog post we described the problems with detecting Path MTU values in the internet. ICMP and fragment datagrams are often blocked on both sides of the connections. Clients can encounter misconfigured firewalls, NAT devices or use ISPs which aggressively intercept connections. Clients also often use VPN's or IPv6 tunnels which, misconfigured, can cause path MTU issues.

Servers on the other hand increasingly often rely on Anycast or ECMP. Both of these things, as well as router and firewall misconfiguration are often a cause for ICMP and fragment datagrams being dropped.

Finally, we hope the online test is useful and can give you more insight into the inner workings of your networks. The test has useful examples of tcpdump syntax, useful to gain more insight. Happy network debugging!

Is fixing fragmentation issues for 10% of the internet exciting? We are hiring system engineers of all stripes, Golang programmers, C wranglers, and interns in multiple locations! Join us in San Francisco, London, Austin, Champaign and Warsaw.

1. In IPv6 the "forward" fragmentation works slightly differently than in IPv4. The intermediate routers are prohibited from fragmenting the packets, but the source can still do it. This is often confusing - a host might be asked to fragment a packet that it transmitted in the past. This makes little sense for stateless protocols like DNS. ↩︎

2. On a side note, there also exists a "minimum transmission unit"! In commonly used Ethernet framing, each transmitted datagram must have at least 64 bytes on Layer 2. This translates to 22 bytes on UDP and 10 bytes on TCP layer. Multiple implementations used to leak uninitialized memory on shorter packets! ↩︎

3. Strictly speaking in IPv4 the ICMP packet is named "Destination Unreachable, Fragmentation Needed and Don't Fragment was Set". But I find the IPv6 ICMP error description "Packet too big" much clearer. ↩︎

4. As a hint, TCP stack also include a maximum allowed "MSS" value in SYN packets (MSS is basically an MTU value reduced by size of IP and TCP headers). This allows the hosts to know what is the MTU on their links. Notice: this doesn't say what is the MTU on the dozens internet links between the two hosts! ↩︎

5. Let's err on the safe side. A better MTU is 1492, to accommodate for DSL and PPPoE connections. ↩︎

We’ve been working with registrars and registries in the IETF on making DNSSEC easier for domain owners, and over the next two weeks we’ll be starting out by enabling DNSSEC automatically for .dk domains.

Before we get into the details of how we've improved the DNSSEC experience, we should explain why DNSSEC is important and the function it plays in keeping the web safe.

DNSSEC’s role is to verify the integrity of DNS answers. When DNS was written in the early 1980’s, it was only a few researchers and academics on the internet. They all knew and trusted each other, and couldn’t imagine a world in which someone malicious would try to operate online. As a result, DNS relies on trust to operate. When a client asks for the address of a hostname like www.cloudflare.com, without DNSSEC it will trust basically any server that returns the response, even if it wasn’t the same server it originally asked. With DNSSEC, every DNS answer is signed so clients can verify answers haven’t been manipulated over transit.

If DNSSEC is so important, why do so few domains support it? First, for a domain to have the opportunity to enable DNSSEC, not only do its DNS provider, its registrar and its registry all have to support DNSSEC, all three of those parties have to also support the same encryption algorithms.

For domains that do have the ability to enable DNSSEC, DNSSEC is just not easy enough -- domain owners need to first enable DNSSEC with their DNS provider, and then copy and paste some values (called a DS record) from their DNS provider’s dashboard to their registrar’s dashboard, making sure not to miss any characters when copying and pasting, because that would cut off traffic to their whole domain. What we need here is automation.

It's been Cloudflare's long-standing statement that as the DNS operator, we would like to update the DS automatically for a user, but DNS operates on a legacy model where the registrar is able to talk directly to the registry, but the DNS operator (Cloudflare) is left completely out of that model.

Here at Cloudflare, we’re determined it’s time to change that outdated system. We have published an Internet Draft to propose a new model for how DNS operators, registries and registrars could operate and communicate to make specific user-authorized changes to domains. It’s important to point out that the IETF works on the principle of rough consensus and running code. Cloudflare, in conjunction with the .dk registry, has produced running code, and we’re very close to getting consensus. That Internet Draft is now making its way through the Standards Track within the IETF and on it’s way to becoming an fully-fledged RFC.

The ccTLD operator for Denmark (ie. the .dk domains) has also realized that the model is outdated. They provide their users (and the operators of nameservers associated with .dk domains) a programmatic way of installing and updating DS records. This is exactly what operators like Cloudflare need.

Cloudflare has been testing their API and is now ready to kick off an automated, clean, safe and reliable way of updating DS records for our .dk customers. Over the next two weeks we will enable DNSSEC for .dk domains that have started to in the past, but haven’t finished the process.

Of course, for Cloudflare, there’s no surprise that Denmark is the home to forward thinkers like this!

If you have a .dk domain on Cloudflare, you really don’t need to do anything except flip the switch enabling DNSSEC within the Cloudflare login console before we do the migration on Tuesday, April 18, 2017.

We are excited to work with the .dk registry on this first step to making DNSSEC automatic, and are looking for other TLD’s looking to make DNSSEC easy to use.

CC BY-SA 2.0 image by Chris Short

I will briefly explain a few concepts you need to know about DNSSEC and negative answers, and then we will dive into how CloudFlare saves on compute when asked for names that don’t exist.

Here’s a quick summary of DNSSEC:

This is an unsigned DNS answer (unsigned == no DNSSEC):

cloudflare.com.		299	IN	A	198.41.214.162
cloudflare.com.		299	IN	A	198.41.215.162

This is an answer with DNSSEC:

cloudflare.com.		299	IN	A	198.41.214.162
cloudflare.com.		299	IN	A	198.41.215.162
cloudflare.com.		299	IN	RRSIG	A 13 2 300 20160311145051 20160309125051 35273     cloudflare.com. RqRna0qkih8cuki++YbFOkJi0DGeNpCMYDzlBuG88LWqx+Aaq8x3kQZX TzMTpFRs6K0na9NCUg412bOD4LH3EQ==

Answers with DNSSEC contain a signature for every record type that is returned. (In this example, only A records are returned so there is only one signature.) The signatures allow DNS resolvers to validate the records returned and prevent on-path attackers from intercepting and changing the answers.

There are two types of negative answers. The first is NXDOMAIN, which means that the name asked for does not exist. An example of this is a query asking for missing.cloudflare.com. missing.cloudflare.com doesn’t exist at all.

The second type is NODATA, which means that the name does exist, just not in the requested type. An example of this would be asking for the MX record of blog.cloudflare.com. There are A records for blog.cloudflare.com but no MX records so the appropriate response is NODATA.

To see what gets returned in a negative NXDOMAIN answer, let’s look at the response for a query for bogus.ietf.org.

The first record that has to be returned in a negative answer with DNSSEC is an SOA, just like in an unsigned negative answer. The SOA contains some metadata about the zone and lets the recursor know how long to cache the negative answer for.

ietf.org.	1179	IN	SOA	ns0.amsl.com. glen.amsl.com. 1200000325 1800 1800 604800 1800

Because the domain is signed with DNSSEC, the signature for the SOA is also returned:

ietf.org.	1179	IN	RRSIG	SOA 5 2 1800 20170308083354 20160308073501 40452 ietf.org. S0gIjTnQGA6TyIBjCeBXL4ip8aEQEgg2y+kCQ3sLtFa3oNy9vj9kj4aP 8EVu4oIexr8X/i9L8Oj5ec4HOrQoYsMGObRUG0FGT0MEbxepi+wWrfed vD/3mq8KZg/pj6TQAKebeSQGkmb8y9eP0PdWdUi6EatH9ZY/tsoiKyqg U4vtq9sWZ/4mH3xfhK9RBI4M7XIXsPX+biZoik6aOt4zSWR5WDq27pXI 0l+BLzZb72C7McT4PlBiF+U86OngBlGxVBnILyW2aUisi2LY6KeO5AmK WNT0xHWe5+JtPD5PgmSm46YZ8jMP5mH4hSYr76jqwvlCtXvq8XgYQU/P QyuCpQ==

The next part of the negative answer in DNSSEC is a record type called NSEC. The NSEC record returns the previous and next name in the zone, which proves to the recursor that the queried name cannot possibly exist, because nothing exists between the two names listed in the NSEC record.

www.apps.ietf.org.	1062	IN	NSEC	cloudflare-verify.ietf.org. A RRSIG NSEC

This NSEC record above tells you that bogus.ietf.org does not exist because no names exist canonically between www.apps.ietf.org and cloudflare-verify.ietf.org. Of course, this record also has a signature contained in the answer:

www.apps.ietf.org.	1062	IN	RRSIG	NSEC 5 4 1800 20170308083322 20160308073501 40452 ietf.org. NxmjhCkTtoiolJUow/OreeBRxTtf2AnIPM/r2p7oS/hNeOdFI9tpgGQY g0lTOYjcNNoIoDB/r56Kd+5wtuaKT+xsYiZ4K413I+cmrNQ+6oLT+Mz6 Kfzvo/TcrJD99PVAYIN1MwzO42od/vi/juGkuKJVcCzrBKNHCZqu7clu mU3DEqbQQT2O8dYIUjLlfom1iYtZZrfuhB6FCYFTRd3h8OLfMhXtt8f5 8Q/XvjakiLqov1blZAK229I2qgUYEhd77n2pXV6SJuOKcSjZiQsGJeaM wIotSKa8EttJELkpNAUkN9uXfhU+WjouS1qzgyWwbf2hdgsBntKP9his 9MfJNA==

A second NSEC record is also returned to prove that there is no wildcard that would have covered bogus.ietf.org:

ietf.org.	1062	IN	NSEC	ietf1._domainkey.ietf.org. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF

This record above tells you that a wildcard (*. ietf.org) would have existed between those two names. Because there is no wildcard record at *.ietf.org, as proven by this NSEC record, the DNS resolver knows that really nothing should have been returned for bogus.ietf.org. This NSEC record also has a signature:

ietf.org.	1062	IN	RRSIG	NSEC 5 2 1800 20170308083303 20160308073501 40452 ietf.org. homg5NrZIKo0tR+aEp0MVYYjT7J/KGTKP46bJ8eeetbq4KqNvLKJ5Yig ve4RSWFYrSARAmbi3GIFW00P/dFCzDNVlMWYRbcFUt5NfYRJxg25jy95 yHNmInwDUnttmzKuBezdVVvRLJY3qSM7S3VfI/b7n6++ODUFcsL88uNB V6bRO6FOksgE1/jUrtz6/lEKmodWWI2goFPGgmgihqLR8ldv0Dv7k9vy Ao1uunP6kDQEj+omkICFHaT/DBSSYq59DVeMAAcfDq2ssbr4p8hUoXiB tNlJWEubMnHi7YmLSgby+m8b97+8b6qPe8W478gAiggsNjc2gQSKOOXH EejOSA==

All in all, the negative answer for bogus.ietf.org contains an SOA + SOA RRSIG + (2) NSEC + (2) NSEC RRSIG. It is 6 records in total, returning an answer that is 1095 bytes (this is a large DNS answer).

What you may have noticed is that because the negative answer returns the previous and next name, you can keep asking for next names and essentially “walk” the zone until you learn every single name contained in it.

For example, if you ask for the NSEC on ietf.org, you will get back the first name in the zone, ietf1._domainkey.ietf.org:

ietf.org.		1799	IN	NSEC 	ietf1._domainkey.ietf.org.  A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF

Then if you ask for the NSEC on ietf1._domainkey.ietf.org you will get the next name in the zone:

ietf1._domainkey.ietf.org. 1799	IN	NSEC 	apps.ietf.org. TXT RRSIG NSEC

And you can keep going until you get every name in the zone:

apps.ietf.org.		1799	IN	NSEC 	mail.apps.ietf.org. MX RRSIG NSEC

The root zone uses NSEC as well, so you can walk the root to see every TLD:

The root NSEC:

.			21599	IN	NSEC 	aaa. NS SOA RRSIG NSEC DNSKEY

.aaa NSEC:

aaa.			21599	IN	NSEC 	aarp. NS DS RRSIG NSEC

.aarp NSEC:

aarp.			21599	IN	NSEC	 abb. NS DS RRSIG NSEC

Zone walking was actually considered a feature of the original design:

The complete NXT chains specified in this document enable a resolver to obtain, by successive queries chaining through NXTs, all of the names in a zone. - RFC2535

(NXT is the original DNS record type that NSEC was based off of)

However, as you can imagine, this is a terrible idea for some zones. If you could walk the .gov zone, you could learn every US government agency and government agency portal. If you owned a real estate company where every realtor got their own subdomain, a competitor could walk through your zone and find out who all of your realtors are.

CC BY 2.0 image by KIUI

So the DNS community rallied together and found a solution. They would continue to return previous and next names, but they would hash the outputs. This was defined in an upgrade to NSEC called NSEC3.

6rmo7l6664ki2heho7jtih1lea9k6los.icann.org. 3599 IN NSEC3 1 0 5 2C21FAE313005174 6S2J9F2OI56GPVEIH3KBKJGGCL21SKKL A RRSIG

NSEC3 was a “close but no cigar” solution to the problem. While it’s true that it made zone walking harder, it did not make it impossible. Zone walking with NSEC3 is still possible with a dictionary attack. An attacker can use a list of the most common hostnames, hash them with the hashing algorithm used in the NSEC3 record (which is listed in the record itself) and see if there are any matches. Even if the domain owner uses a salt on the hash, the length of the salt is included in the NSEC3 record, so there are a finite number of salts to guess.

The Salt Length field defines the length of the salt in octets, ranging in value from 0 to 255.”

• RFC5155

If you recall from above, NODATA is the response from a server when it is asked for a name that exists, but not in the requested type (like an MX record for blog.cloudflare.com). NODATA is similar in output to NXDOMAIN. It still requires SOA, but it only takes one NSEC record to prove the next name, and to specify which types do exist on the queried name.

For example, if you look for a TXT record on apps.ietf.org, the NSEC record will tell you that while there is no TXT record on apps.ietf.org, there are MX, RRSIG and NSEC records.

apps.ietf.org.		1799	IN	NSEC 	mail.apps.ietf.org. MX RRSIG NSEC

There are two problems with negative answers:

The first is that the authoritative server needs to return the previous and next name. As you’ll see, this is computationally expensive for CloudFlare, and as you’ve already seen, it can leak information about a zone.

The second is that negative answers require two NSEC records and their two subsequent signatures (or three NSEC3 records and three NSEC3 signatures) to authenticate the nonexistence of one name. This means that answers are bigger than they need to be.

CloudFlare has a custom in house DNS server built in Go called RRDNS. What's unique about RRDNS is that unlike standard DNS servers, it does not have the concept of a zone file. Instead, it has a key value store that holds all of the DNS records of all of the domains. When it gets a query for a record, it can just pick out the record that it needs.

Another unique aspect of CloudFlare's DNS is that a lot of our business logic is handled in the DNS. We often dynamically generate DNS answers on the fly, so we don't always know what we will respond with before we are asked.

Traditional negative answers require the authoritative server to return the previous and next name of a missing name. Because CloudFlare does not have the full view of the zone file, we'd have to ask the database to do a sorted search just to figure out the previous and next names. Beyond that, because we generate answers on the fly, we don’t have a reliable way to know what might be the previous and next name, unless we were to precompute every possible option ahead of time.

One proposed solution to the previous and next name, and secrecy problems is RFC4470, dubbed 'White Lies'. This RFC proposes that DNS operators make up a previous and next name by randomly generating names that are canonically slightly before and after the requested name.

White lies is a great solution to block zone walking (and it helps us prevent unnecessary database lookups), but it still requires 2 NSEC records (one for previous and next name and another for the wildcard) to say one thing, so the answer is still bigger than it needs to be.

We decided to take lying in negative answers to its fullest extent. Instead of white lies, we do black lies.

For an NXDOMAIN, we always return \000.(the missing name) as the next name, and because we return an NSEC directly on the missing name, we do not have to return an additional NSEC for the wildcard. This way we only have to return SOA, SOA RRSIG, NSEC and NSEC RRSIG, and we do not need to search the database or precompute dynamic answers.

Our negative answers are usually around 300 bytes. For comparison, negative answers for ietf.org which uses NSEC and icann.org, which uses NSEC3 are both slightly over 1000 bytes, three times the size. The reason this matters so much is that the maximum size of an unsigned UDP packet is typically 512 octets. DNSSEC requires support for at least 1220 octets long messages over UDP, but above that limit, the client may need to upgrade to DNS over TCP. A good practice is to keep enough headroom in order to keep response sizes below fragmentation threshold during zone signing key rollover periods.

NSEC: 1096 bytes

ietf.org.		1799	IN	SOA	ns0.amsl.com. glen.amsl.com. 1200000317 1800 1800 604800 1800
ietf.org.		1799	IN	RRSIG	SOA 5 2 1800 20170213210533 20160214200831 40452 ietf.org. P8XoJx+SK5nUZAV/IqiJrsoKtP1c+GXmp3FvEOUZPFn1VwW33242LVrJ GMI5HHjMEX07EzOXZyLnQeEvlf2QLxRIQm1wAnE6W4SUp7TgKUZ7NJHP dgLr2gqKYim4CI7ikYj3vK7NgcaSE5jqIZUm7oFxxYO9/YPz4Mx7COw6 XBOMYS2v8VY3DICeJdZsHJnVKlgl8L7/yqrL8qhkSW1yDo3YtB9cZEjB OVk8uRDxK7aHkEnMRz0LODOJ10AngJpg9LrkZ1CO444RhZGgTbwzN9Vq rDyH47Cn3h8ofEOJtYCJvuX5CCzaZDInBsjq9wNAiNBgIQatPkNriR77 hCEHhQ==
ietf.org.		1799	IN	NSEC	ietf1._domainkey.ietf.org. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY SPF
ietf.org.		1799	IN	RRSIG	NSEC 5 2 1800 20170213210816 20160214200831 40452 ietf.org. B9z/JJs30tkn0DyxVz0zaRlm4HkeNY1TqYmr9rx8rH7kC32PWZ1Fooy6 16qmB33/cvD2wtOCKMnNQPdTG2qUs/RuVxqRPZaQojIVZsy/GYONmlap BptzgOJLP7/HOxgYFgMt5q/91JHfp6Mn0sd218/H86Aa98RCXwUOzZnW bdttjsmbAqONuPQURaGz8ZgGztFmQt5dNeNRaq5Uqdzw738vQjYwppfU 9GSLkT7RCh3kgbNcSaXeuWfFnxG1R2SdlRoDICos+RqdDM+23BHGYkYc /NEBLtjYGxPqYCMe/7lOtWQjtQOkqylAr1r7pSI2NOA9mexa7yTuXH+x o/rzRA==
www.apps.ietf.org.	1799	IN	NSEC	cloudflare-verify.ietf.org. A RRSIG NSEC
www.apps.ietf.org.	1799	IN	RRSIG	NSEC 5 4 1800 20170213210614 20160214200831 40452 ietf.org. U+hEHcTps2IC8VKS61rU3MDZq+U0KG4/oJjIHVYbrWufQ7NdMdnY6hCL OmQtsvuZVRQjWHmowRhMj83JMUagxoZuWTg6GuLPin3c7PkRimfBx7jI wjqORwcuvpBh92A/s/2HXBma3PtDZl2UDLy4z7wdO62rbxGU/LX1jTqY FoJJLJfJ/C+ngVMIE/QVneXSJkAjHV96FSEnreF81V62x9azv3AHo4tl qnoYvRDtK+cR072A5smtWMKDfcIr2fI11TAGIyhR55yAiollPDEz5koj BfMstC/JXVURJMM+1vCPjxvwYzTZN8iICf1AupyyR8BNWxgic5yh1ljH 1AuAVQ==

Black Lies: 357 bytes

cloudflare.com.		1799	IN	SOA	ns3.cloudflare.com. dns.cloudflare.com. 2020742566 10000 2400 604800 3600
blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. RRSIG NSEC
cloudflare.com.		1799	IN	RRSIG	SOA 13 2 86400 20160220230013 20160218210013 35273 cloudflare.com. kgjtJDuuNC/yX8yWQpol4ZUUr8s8yAXZi26KWBI6S3HDtry2t6LnP1ou QK10Ut7DXO/XhyZddRBVj3pIpWYdBQ==
blog.cloudflare.com.	3599	IN	RRSIG	NSEC 13 3 3600 20160220230013 20160218210013 35273 cloudflare.com. 8BKAAS8EXNJbm8DxEI1OOBba8KaiimIuB47mPlteiZf3sVLGN1edsrXE +q+pHaSHEfYG5mHfCBJrbi6b3EoXOw==

Our take on NODATA responses is also unique. Traditionally, NODATA responses contain one NSEC record to tell the resolver which types exist on the requested name. This is highly inefficient. To do this, we’d have to search the database for all the types that do exist, just to answer that the requested type does not exist. Remember that’s not even always possible because we have dynamic answers that are generated on the fly.

What we realized was that NSEC is a denial of existence. What matters in NSEC are the missing types, not the present ones. So what we do is we set all the types. We say, this name does exist, just not on the one type you asked for.

For example, if you asked for an TXT record of blog.cloudflare.com we would say, all the types exist, just not TXT.

blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. A WKS HINFO MX AAAA LOC SRV CERT SSHFP IPSECKEY RRSIG NSEC TLSA HIP OPENPGPKEY SPF

And then if you queried for a MX on blog.cloudflare.com, we would return saying we have every record type, even TXT, but just not MX.

blog.cloudflare.com.	3599	IN	NSEC	\000.blog.cloudflare.com. A WKS HINFO TXT AAAA LOC SRV CERT SSHFP IPSECKEY RRSIG NSEC TLSA HIP OPENPGPKEY SPF

This saves us a database lookup and from leaking any zone information in negative answers. We call this the DNS Shotgun.

We put a lot of care to ensure CloudFlare’s negative answers are standards compliant. We’re even pushing for them to become an Internet Standard by publishing an Internet Draft earlier this year.

RFC4470, White Lies, allows us to randomly generate next names in NSEC. Not setting the second NSEC for the wildcard subdomain is also allowed, so long as there exists an NSEC record on the actual queried name. And lastly, our lie of setting every record type in NSEC records for NODATA, is okay too –– after all, domains are constantly changing, it’s feasible that the zone file changed in between the time the NSEC record indicated to you there was no MX record on blog.cloudflare.com and when you queried successfully for MX on blog.cloudflare.com.

We’re proud of our negative answers. They help us keep packet size small, and CPU consumption low enough for us to provide DNSSEC for free for any domain. Let us know what you think, we’re looking forward to hearing from you.

Using the CloudFlare API via Python

Very early on in the company’s history we decided that everything that CloudFlare does on behalf of its customer-base should be controllable via an API. In fact, when you login to the CloudFlare control panel, you’re really just making API calls to our backend services. Over time that API has matured and improved. We are now on v4 of that API.

The current CloudFlare API is documented here and it’s used by both the CloudFlare control panel and directly by umpteen customers every minute of every day. The new API is designed with a clean naming structure and consistent data representation for data. It’s also extensible.

This blog entry introduces python-cloudflare, a Python wrapper providing full access to the CloudFlare v4 API.

Let’s get right into the thick-of-it with the simplest coding example available to show python-cloudflare in action. This example lists all your domains (zones) and also checks some basic features for each zone.

#!/usr/bin/env python
import CloudFlare
def main():
    cf = CloudFlare.CloudFlare()
    zones = cf.zones.get(params={'per_page':50})
    for zone in zones:
        zone_name = zone['name']
        zone_id = zone['id']
        settings_ipv6 = cf.zones.settings.ipv6.get(zone_id)
        ipv6_on = settings_ipv6['value']
        print zone_id, ipv6_on, zone_name
    exit(0)
if __name__ == '__main__':
    main()

The structure of the CloudFlare class matches the API documentation. The CloudFlare.zones.get() method returns information about the zones as per the list zones documentation. The same for CloudFlare.zones.settings.ipv6.get() and its documentation.

Data is passed into the methods via standard Python structures and they are returned in Python structures that match the API documentation. That means if you see an API call in the documentation, then you can translate it into the Python code in a one-to-one manner.

For example, take a look at the WAF list rules API call.

This codes into CloudFlare.zones.dns_records.post() method with the zone_id as the first argument, the package_id as the second argument and the optional parameters passed last (or if there aren’t any; then just drop the third argument for the call). Because this is a GET call there’s a .get() as part of the method name.

    r = cf.zones.firewall.waf.packages.rules.get(zone_id, package_id, params=params)

Here’s the much simpler Create DNS record API call.

This would be coded into the Python method CloudFlare.zones.dns_records.post() with the zone_id as the first argument and the required parameters passed as data. Because this is a POST call there’s a .post() as part of the method name.

    r = cf.zones.dns_records.post(zone_id, data=dns_record)

Here’s an example of that Create DNS record call in action. In this code, we add two records to an existing zone. We also show how the error is handled (in classic Python style).

    zone_name = 'example.com'
    try:
        r = cf.zones.get(params={'name': zone_name})
    except CloudFlare.CloudFlareAPIError as e:
        exit('/zones.get %s - %d %s' % (zone_name, e, e))
    except Exception as e:
        exit('/zones.get %s - %s' % (zone_name, e))
   zone_id = r['id']
   # DNS records to create
    dns_records = [
        {'name':'foo', 'type':'A',    'content':'192.168.100.100'}
        {'name':'foo', 'type':'AAAA', 'content':'2001:d8b::100:100'},
    ]
    for dns_record in dns_records:
        try:
            r = cf.zones.dns_records.post(zone_id, data=dns_record)
        except CloudFlare.CloudFlareAPIError as e:
            exit('/zones.dns_records.post %s - %d %s' % (record['name'], e, e))

There’s a whole folder of example code available on the GitHub repository.

As we stated above (and as CloudFlare has done many times before) we have placed this code up on GitHub for anyone to download and use. We welcome contributions and will review any pull requests. To install it, just clone it and follow the README instructions. For those that just want to get going right now; here’s the tl;dr install:

$ git clone https://github.com/cloudflare/python-cloudflare
$ cd python-cloudflare
$ ./setup.py build
$ sudo ./setup.py install

Not only do you get the python API calls, you also get a fully functioning CLI (command line interface) that allows quick creation of scripts that interface with CloudFlare.

From the CLI command you can call any of the CloudFlare API calls. The command responds with the returned JSON data. If you want to filter the results you possibly also want to install the highly-versatile jq command. Here’s a command to check the nameservers for a specific domain hosted on CloudFlare and then process it via jq.

$ cli4 name=example.com /zones | jq -c '.[]|{"name_servers":.name_servers}'
{
  "name_servers":[
    "alice.ns.cloudflare.com",
    "bob.ns.cloudflare.com"
  ]
}
$

The CLI command will convert on-the-fly zone names into zone identifiers. For example; if you want to check the DNSSEC status on a zone your operate on CloudFlare; then use this command.

$ cli4 /zones/:example.com/dnssec | jq '{"status":.status,"ds":.ds}'
{
  "status": "active",
  "ds": "example.com. 3600 IN DS 2371 13 2 00000000000000000000000000000 ..."
}
$

You can issue GET PUT POST PATCH or DELETE calls into the API. You can pass data into a CloudFlare API call with the CLI command. All documented via the README.md and wiki examples in GitHub.

Here’s a useful command for customers that need to flush their cache files.

$ cli4 --delete purge_everything=true /zones/:example.com/purge_cache
{
  "id":"d8afaec3dd2b7f8c1b470e594a21a01d"
}
$

See how the commands arguments match the purge_cache API documentation.

Finally, here’s an example of turning on DNSSEC via the API.

$ cli4 --patch status=active /zones/:example.com/dnssec | jq -c '{"status":.status}'
{"status":"pending"}
$

There are plenty more examples available within the GitHub repo.

Python isn’t the only language you can use to interact with CloudFlare’s API. If you’re a Go, or Node.js user, we also have client libraries you can use to interact with CloudFlare on our GitHub. Find them here Go client and here Node.js client. Want to write something in a different language? Feel free to do that. The API spec is online and ready for you to code up.

If you like what you read here today and are interested in joining one of CloudFlare’s software teams, then checkout our Join Our Team page.

1. The lack of legitimate ANY use.

2. The abundance of malicious ANY use.

3. The constant use of ANY queries in large DNS amplification DDoS attacks.

Additionally, we were about to launch Universal DNSSEC, and we could foresee the high cost of assembling ANY answers and providing DNSSEC-on-the-fly for those answers, especially when most of the time, those ANY answers were for malicious, illegitimate, clients.

Although we usually make a tremendous effort to maintain backwards compatibility across Internet protocols (recently, for example, continuing to support SHA-1-based SSL certificates), it was clear to us that the DNS ANY query was something that was better removed from the Internet than maintained for general use.

Our proposal at the time was to return an ERROR code to the querier telling them that ANY was not supported, and this sparked a robust discussion in the DNS protocol community. In this blog post, we’ll cover what has happened and what our final plan is.

Just before we published our blog a popular software started using ANY queries, to get all address records for a name -- something that ANY isn’t actually designed to do. The effect of this software was that our steady ANY query load had grown from few hundred per second to tens of thousands in a matter of days. Luckily, the software in question issued a revised version that did not use ANY, and our steady ANY query load returned to the old level.

CC BY-SA 2.0 image by Liam Quinn

As that was happening, a lively discussion started to form among those in the DNS community. The first fundamental question that had to be answered was: “Does ANY mean ALL?”. That is, was an ANY query meant to be a way to receive all of the records in a zone for the query name?

Different people had different interpretations of ANY depending on the kind of DNS service they were providing. For example, it is nice as an operator of a DNS resolver to be able to ask your own resolver, “what is stored in the resolver memory/cache for a particular name?”. An ANY query with the right query flags can help answer that. On the other hand, an operator of an authoritative server does not need that functionality, as AXFR provides another reliable way to get that information. In short, the ANY query is a nice tool for people that are trying to understand what is going on in DNS resolution. Some community members argued that ANY should be a restricted query to a privileged few that have the need to know.

While ANY can be a nice tool to debug and expose inconsistencies in resolver caches, it’s still not a great tool: with more and more Anycast resolver clusters, there is no guarantee that two subsequent queries will hit the same resolver.

Our in-house DNS server is optimized to provide dynamic answers to questions. For example, depending on how a CNAME is configured, our server may return the CNAME, fetch the real answer from the target of the CNAME (what we call CNAME Flattening), or provide CloudFlare addresses as answers. Thus for us to answer an ANY query, we need to compute all of the possible combinations just to know what to return.

Beyond that, our DNSSEC implementation signs answers on-the-fly. Thus, returning an answer with many different types of DNS records in the answer requires signing all of them at the edge. Thus providing support for ANY in the “traditional” sense had serious computational and response time implications. This is not unique to CloudFlare; we are not the only DNS implementation that has this high cost factor in answering ANY queries.

In a recent paper by Akamai the authors draw the conclusion that DNSSEC is the main cause of large answers used for DDoS attacks. But looking at the packet capture they included in the paper, it’s clear the real cause of the large answers is that the attackers use ANY queries to maximize the amplification factor.

CC BY-SA 2.0 image by Vladimer Shioshvili

We regularly see attacks that attempt to use our powerful DNS system as a source of reflection. We have in response to this created sophisticated systems to detect the attacks and mitigate them in a responsible manner. Our deprecation of ANY is a key part of those protections. One of our main mantra is “do not return larger than needed answers”, exactly to help protect others on the Internet from amplification attacks.

Soon after we announced our planned deprecation of ANY, we submitted an Internet Draft to the IETF proposing to restrict ANY queries to authorized parties only. In the resulting discussion, it became clear that what mattered here was how we deprecated ANY. To make sure no system was adversely affected, we had to take into account how various applications and DNS implementations were using ANY.

In short, there are two main uses of ANY queries:

1. Some programs use ANY as a probabilistic optimization attempt to get the answers they need.

2. Some use ANY to debug DNS resolvers when things go wrong.

However, neither of those use cases are satisfied by the current ANY landscape. There is a lack of common behavior among resolvers as to how they treat answers that are cached as a result of an ANY query. Some will return this data when a more specific query matches, while other resolvers will fetch the exact requested data, even though they already have it in cache from an ANY query. This is because resolvers interpret RFC2181 section 5.4.1 (Ranking Data) in differing ways, and some resolvers were written without applying the rules from RFC2181 at all. Some resolvers will not forward ANY queries to authoritative servers if there exists a single RRset for that name in the resolver’s cache, and others will forward it unless they have a prior ANY answer in the cache. Furthermore, some resolvers will only reuse the results of an ANY query to answer other ANY queries, queries for all other types will result in a direct query for the type requested.

In many cases, the ANY query results in an answer that is too large to fit in the UDP packet size requested, resulting in a truncated answer, leading to a follow up via TCP. For a while, the DNS community believed that returning truncated answers would stop attacks, but in reality, that will only mitigate simple attacks using forged packets. In attacks that are reflected via open resolvers, returning truncated packets will not work because the open resolvers are happy to fall back to TCP if the UDP answer sets the truncate bit.

So, there is no common understanding of how the ANY query should be treated to minimize its amplification potential. To be fair, the ANY query is a special type of query called a meta-query i.e. it is not an actual type. Nevertheless, the community was divided into two camps: “ANY == ALL” and “ANY != ALL”.

The community was also further divided into groups of “ANY is ok for everyone” and “ANY should be restricted to “good” clients”. Our position from the beginning was that “ANY != ALL” and we were looking for a way to help curb the number of large amplified attacks on the Internet that used ANY.

Over a few months, we engaged in a number of experiments to see how different DNS systems reacted to different non-ANY responses to the ANY query. After a fair amount of experimentation and discussions with colleagues around the world, we decided on an approach that is recursive resolver centric. What we wanted to do was to give answers that are friendly to recursive resolvers, i.e. we give them something small that they can cache and return to repeated ANY queries. Returning an error to a recursive resolver was not a good option, as the resolver will just ask the next authoritative server and visit all the authoritative servers before giving up.

We also wanted to avoid guessing the intention of the originator of the query, which is why we did not follow one proposal to give out the A+AAAA+MX records or a CNAME if one existed. We do not like that, as the answer is bigger than it has to be and there is more data in the answer than the originator wanted.

For example, consider an email server that wants an MX record if one exists, but will fallback to an IP address if the MX does not exist. Instead, we decided to return what we call a “harmless” answer––an answer that is not useful for any application on the Internet. We selected an old DNS record type that is not used much anymore, but has the nice property that all test tools display as text: HINFO. This approach is documented in the current Internet Draft Refuse ANY draft that was adopted by the DNSOP working group of the IETF that handles DNS protocol issues.

As you can see below, when asked for ANY, we only return one HINFO record and the optional RRSIG that is only needed when the zone is signed. This record can be cached, and has the added benefit of being small and therefore reducing the amplification factor the attacker expected.

; <<>> DiG 9.8.3-P1 <<>> @ns2.p31.dynect.net. amazon.com. any +dnssec +norec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29671
;; flags: qr aa; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;amazon.com.			IN	ANY

;; ANSWER SECTION:
amazon.com.		900	IN	SOA	dns-    external-master.amazon.com. root.amazon.com. 2010113317 180 60 3024000 60
amazon.com.		3600	IN	NS	pdns6.ultradns.co.uk.
amazon.com.		3600	IN	NS	pdns1.ultradns.net.
amazon.com.		3600	IN	NS	ns2.p31.dynect.net.
amazon.com.		3600	IN	NS	ns3.p31.dynect.net.
amazon.com.		3600	IN	NS	ns1.p31.dynect.net.
amazon.com.		3600	IN	NS	ns4.p31.dynect.net.
amazon.com.		60	IN	A	54.239.26.128
amazon.com.		60	IN	A	54.239.25.208
amazon.com.		60	IN	A	54.239.25.200
amazon.com.		60	IN	A	54.239.17.7
amazon.com.		60	IN	A	54.239.17.6
amazon.com.		60	IN	A	54.239.25.192
amazon.com.		900	IN	MX	5 amazon-smtp.amazon.com.
amazon.com.		900	IN	TXT	"spf2.0/pra include:spf1.amazon.com include:spf2.amazon.com include:amazonses.com -all"
amazon.com.		900	IN	TXT	"v=spf1 include:spf1.amazon.com include:spf2.amazon.com include:amazonses.com -all"

;; Query time: 30 msec
;; SERVER: 204.13.250.31#53(204.13.250.31)
;; WHEN: Wed Apr 13 09:57:59 2016
;; MSG SIZE  rcvd: 565

Unsigned answer from large internet company is 565 bytes long.

In contrast a signed answer from CloudFlare.com is only 224 bytes long or less than ½ the unsigned answer above.

; <<>> DiG 9.8.3-P1 <<>> cloudflare.com any @ns3.cloudflare.com +dnssec +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36238
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;cloudflare.com.			IN	ANY

;; ANSWER SECTION:
cloudflare.com.		3789	IN	HINFO	"Please stop asking for ANY" "See draft-ietf-dnsop-refuse-any"
cloudflare.com.		3789	IN	RRSIG	HINFO 13 2 3789 20160414100147 20160412080147 35273 cloudflare.com. lGyCY7IC5sgHBfE95IJXDUS4diFjE5kq4vNMhhqP6+2+NyTQh1zAh1qw 3C710mFvvuCWe4VyRiqlu1jUzMnuLg==

;; Query time: 80 msec
;; SERVER: 2400:cb00:2049:1::a29f:21#53(2400:cb00:2049:1::a29f:21)
;; WHEN: Wed Apr 13 10:01:47 2016
;; MSG SIZE  rcvd: 224

We have been returning the HINFO answer for ANY queries since October 2015, with very few reports of problems in the field, besides a single Twitter rant about us not understanding DNS.

A few other DNS server vendors and DNS operators have followed our lead or adopted a similar line of defense. The latest example is the University of Cambridge which modified their BIND implementation to only return a single RRset for an ANY query. A common DNS server (NSD) has also been limiting ANY responses to only A+AAAA+MX types and/or CNAME, and a patch to return an even smaller answer has been proposed.

The moral is that the ANY query is not a useful tool for most DNS operators, but it is a wonderful tool if one is in the business of generating attacks against anyone else on the Internet. Answering ANY queries with a giant answer is not helping mitigate the plague of DoS volume attacks.

CloudFlare has taken a step to make the Internet a less hostile place and is leading by example. We strongly urge others to follow in our steps and neuter the amplification factor that a single DNS query can achieve. We all want to build a safer Internet, and neutering ANY query is one small step that everyone can take.

CC BY 2.0 image by Robert Couse-Baker

Yesterday we wrote about the 400 gigabit per second attacks we see on our network.

One way that attackers DDoS websites is by repeatedly doing DNS lookups that have small queries, but large answers. The attackers spoof their IP address so that the DNS answers are sent to the server they are attacking, this is called a reflection attack.

Domains with DNSSEC, because of the size of some responses, are usually ripe for this type of abuse, and many DNS providers struggle to combat DNSSEC-based DDoS attacks. Just last month, Akamai published a report on attacks using DNS lookups against their DNSSEC-signed .gov domains to DDoS other domains. They say they have seen 400 of these attacks since November.

To prevent any domain on CloudFlare being abused for a DNS amplification attack in this way, we took precautions to make sure most DNS answers we send fit in a 512 byte UDP packet, even when the zone is signed with DNSSEC. To do this, we had to be creative in our DNSSEC implementation. We chose a rarely-used-for-DNSSEC signature algorithm and even deprecated a DNS record type along the way.

Dutch mathematician Arjen Lenstra famously talks about cryptography in terms of energy. (We’ve covered him once before on our blog). He takes the amount of energy required to break a cryptographic algorithm and compares that with how much water that energy could boil. To break a 228-bit RSA key requires less energy than it takes to boil a teaspoon of water. On the other hand, to break a 228-bit elliptic curve key requires the amount of energy needed to boil all the water on the earth.

With elliptic curve cryptography in the ECDSA signature algorithm, we can use smaller keys with the same level of security as a larger RSA key. Our elliptic curve keys are 256 bits long, equivalent in strength to a 3100 bit RSA key (most RSA keys are only 1024 or 2048 bits). You can compare below two signed DNSKEY sets, an RSA implementation against our ECDSA one. Ours is one quarter of the size of the matching RSA keys and signature.

As a side benefit, ECDSA is lightning fast, and our engineer Vlad Krasnov actually helped make it even faster. By implementing ECDSA natively in assembler, he was able to speed up signing by 21x. His optimizations are now part of the standard Go crypto library as of Go version 1.6. It now only takes us a split of a second, 0.0001 of a second, to sign records for a DNS answer.

In Akamai’s security report, the authors draw the conclusion that DNSSEC is the only cause of the large answers used for DDoS attacks, but the other cause of the large answers is that the attackers use ANY queries to maximize the amplification factor. ANY queries are a built-in debugging tool, meant to return every DNS record that exist for a name. Unfortunately, they are instead more often used for launching large DDoS attacks.

In September, we stopped answering ANY queries and published an Internet Draft to begin the process of making ANY deprecation an Internet standard. We did this carefully, and worked closely with the few remaining software vendors who use ANY to ensure that we wouldn’t affect their production systems.

An ANY query for DNSSEC-enabled cloudflare.com returns an answer that is 231 bytes. The alleged domain in Akamai’s paper, for comparison, returns an ANY query almost 18 times larger, at a whopping 4016 bytes.

By keeping our packet size small enough to fit in a 512 byte UDP packet, we keep the domains on us safe from being the amplification factor of a DDoS attack. If you are interested in using DNSSEC with CloudFlare, here are some easy steps to get you setup. If you are interested in working on technical challenges like these, we’d love to hear from you.

If you read this blog on a regular basis, you probably use the little tool called SSH, especially its ubiquitous and most popular implementation OpenSSH.

Maybe you’re savvy enough to only use it with public/private keys, and therefore protect yourself from dictionary attacks. If you do then you know that in order to configure access to a new host, you need to make a copy of a public key available to that host (usually by writing it to its disk). Managing keys can be painful if you have many hosts, especially when you need to renew one of the keys. What if DNSSEC could help?

CC BY 2.0 image by William Neuheisel

With version 6.2 of OpenSSH came a feature that allows the remote host to retrieve a public key in a customised way, instead of the typical authorized_keys file in the ~/.ssh/ directory. For example, you can gather the keys of a group of users that require access to a number of machines on a single server (for example, an LDAP server), and have all the hosts query that server when they need the public key of the user attempting to log in. This saves a lot of editing of authorized_keys files on each and every host. The downside is that it's necessary to trust the source these hosts retrieve public keys from. An LDAP server on a private network is probably trustworthy (when looked after properly) but for hosts running in the cloud, that’s not really practical.

DNSSEC is helpful here. That's right: now that we can verify responses from a DNS server, we can safely store public keys in DNS records!

So let's say we administer example.com and want to give Alice and Bob access to machines foo, bar and baz in that domain. We'll store their respective public keys in TXT[1] records named alice_pubkey.example.com and bob_pubkey.example.com. To be entirely accurate, it doesn’t really matter which zone these records belong to, but I’ll consider here that we only have one domain. The requirements are:

• the machines need to run OpenSSH server version 6.2 or later

• they also need a DNSSEC validating resolver (we'll use unbound-host)

• Alice and Bob's keys need to be less than 256 characters long (ECDSA or Ed25519 keys will work)

• DNSSEC needs to be correctly set up on the domain example.com (surprise!)

Alice and Bob generate keys like this:

foo:~$ ssh-keygen -t ecdsa

or like this:

foo:~$ ssh-keygen -t ed25519

and then follow the instructions. They will of course provide a non-empty passphrase. Then they send us (or whoever administers the zone file for example.com) the public key file, which may look like this:

ssh-ed25519 AAAAC3N...VY4A= alice@foo

We can strip the comment alice@foo out of that file, and use the rest as the value to create a TXT record with the name alice_pubkey in the domain example.com. Then, retrieving the key is as easy as this:

 foo:~$ unbound-host -t TXT alice_pubkey.example.com
 alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC3N…”

With -v, unbound-host will show us whether the signature has been verified

foo:~$ unbound-host -v -t TXT alice_pubkey.example.com
alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC…” (insecure)

With -D, it will actually check the signature:

foo:~$ unbound-host -D -v -t TXT alice_pubkey.example.com
alice_pubkey.example.com has TXT record “ssh-ed25519 AAAAC3N…” (secure)

If no record exists, it will show this:

foo:~$ unbound-host -D -v -t TXT charlie_pubkey.example.com
charlie_pubkey.example.com has no TXT record (secure)

Note that the absence of record is also labelled “secure”, thanks to NSEC.

Let’s prepare to parse that output. The sshd_config man page shows that sshd needs a specific user to run the program that will retrieve public keys. This is following the best practices of privilege separation. Let's call that user pubkeygrab and create an account on foo, bar and baz, giving it just the permissions it needs to work and nothing more:

foo:~$ useradd -m -d /var/empty -s /sbin/nologin pubkeygrab

Then create the script pubkeygrab.sh, and store it on each of the machines. Obviously, we'll make sure only root can edit it:

foo:~$ cat /usr/local/bin/pubkeygrab.sh
#!/bin/sh

USER=$1

/usr/sbin/unbound-host -v -D -t TXT ${USER}_pubkey.example.com \\
     | /usr/bin/grep -v "no TXT record" \\
     | /usr/bin/grep ' (secure)$' \\
     | /usr/bin/sed 's/.* "\(.*\)" (secure)$/\1/'

Now I'm certain that a lot of readers will have something to say about the style or the efficiency of this shell script, I just wrote it that way to highlight what steps need to be taken:

• it retrieves a TXT record, and doesn't output anything if the record doesn't exist

• if unbound-host has not confirmed that the record was correctly DNSSEC signed, it doesn't output anything

• if the above is successful, it filters out the text to return only the public key

• it doesn't try to do anything complex, because complexity is the enemy of security (or at least, that’s a point of view that I share with a few people)

• it works with multiple records

I'm sure you will write your own program to do the above. Just make sure it works only when you want it to. It is critical to ensure that it doesn't return anything at least when:

• a record for the corresponding user doesn't exist

• the records are not signed or not properly signed

• the local copy of the root key (/var/unbound/root.key, here) is corrupted.

Bonus points to you if you find more cases.

Now that you have read the warning above, add the following to /etc/ssh/sshd_config on foo, bar and baz:

AuthorizedKeysCommand /usr/local/bin/pubkeygrab.sh
AuthorizedKeysCommandUser pubkeygrab

and restart sshd. Check that the users alice and bob exist on each machine too. Note that the above change will also apply to all existing users. Now you can go to your CloudFlare account, select the domain example.com, and create the TXT records alice_pubkey and bob_pubkey. Paste their respective public keys in the value field. Soon after, Alice and Bob can log in. Ask Charlie to try too. If the above works for Alice and Bob, but fails for Charlie, congratulations, you have turned CloudFlare into a PKI for SSH.

If you remove the TXT records, Alice and Bob’s access should be revoked, and they should be unable to login, once the TTL of the TXT record is expired. However, note that when the output of pubkeygrab.sh is empty, sshd reverts to the usual AuthorizedKeysFile parameter to find a public key. If Alice and Bob are cheeky and want to keep their access after you removed their TXT records, they just need to copy their public key into that file any time before you ban them. If you don't want that, make sure the AuthorizedKeysFile parameter points to a place Alice and Bob can't write to.

I hope this is showing how interesting DNSSEC can be, and that we have more news on this topic soon.

1. Yes, it would be better to have a dedicated record, instead of overloading TXT records. ↩︎