Luckily, over the last year plus a lot of good work has happened in this arena. If you’ve been following the growth of RPKI’s validation data, then you’ll know that more and more networks are signing their routes and creating ROA’s or Route Origin Authorizations. These are cryptographically-signed assertions of the validity of an announced IP block and contribute to the further securing of the global routing table that makes for a safer Internet.

The protocol that we have not written much about is RTR. The Resource Public Key Infrastructure (RPKI) to Router Protocol - or RTR Protocol for short. Today we’re fixing that.

We have written a few times about RPKI (here and here). We have written about how Cloudflare both signs its announced routes and filters its routing inbound from other networks (both transits and peers) using RPKI data. We also added our efforts in the open-source software space with the release of the Cloudflare RPKI Toolkit.

The primary part of the RPKI (Resource Public Key Infrastructure) system is a cryptographically-signed database which is read and processed by a RPKI validator. The validator works with the published ROAs to build a list of validated routes. A ROA consists of an IP address block plus an ASN (Autonomous System Number) that together define who can announce which IP block.

After that step, it is then the job of that validator (or some associated software module) to communicate its list of valid routes to an Internet router. That’s where the RTR protocol (the RPKI to Router Protocol) comes in. Its job is to communicate between the validator and device in charge of allowing or rejecting routes in its table.

The IETF defines the RTR protocol in RFC 8210. This blog post focuses on version 1 and ignores previous versions.

In order to verifiably validate the origin Autonomous Systems and Autonomous System Paths of BGP announcements, routers need a simple but reliable mechanism to receive Resource Public Key Infrastructure (RFC 6480) prefix origin data and router keys from a trusted cache. This document describes a protocol to deliver them.

_This docume_nt describes version 1 of the RPKI-Router protocol.

The Internet’s routers are, to put it bluntly, not the best place to run a routing table’s cryptographic processing. The RTR protocol allows the heavy lifting to be done outside of the valuable processing modules that routers have. RTR is a very lightweight protocol with a low memory footprint. The router simply decides yay-or-nay when a route is received (called “announce” in BGP speak) and hence the router never needs to touch the complex cryptographic validation algorithms. In many cases, it also provides some isolation between the outside world, where certificates need to be fetched from across the globe and then stored, checked, processed, and databased locally. In many cases the control plane (where RTR communication happens) exists on private or protected networks. Separation is a good thing.

Cloudflare’s open-source software for RPKI validation also includes GoRTR, an implementation of the RTR protocol. As mentioned, in Cloudflare’s operational model, we separate the validation (done with OctoRPKI) from the RTR process.

RTR protocol implementations are also provided in other RPKI validation software packages. In fact, RPKI is unable to filter routes without the final step of running RTR (or something similar - should it exist). Here’s a current list of RPKI software packages that either validate or validate and run RTR.

• Cloudflare RPKI Validator Tools and Libraries (OctoRPKI & GoRTR).

• Dragon Research Labs RPKI Toolkit.

• NIC Mexico and LACNIC FORT project including the FORT validator.

• NLnet Labs Routinator 3000.

• RIPE NCC RPKI Validator version 2 (deprecated).

• RIPE NCC RPKI Validator version 3.

• rpki-client by OpenBSD

Each of these open source software packages has its own specific database model and operational methods. Because GoRTR reads a somewhat common JSON file format, you can mix and match between different validators and GoRTR’s code.

The protocol's core is all about synchronizing a database between a validator and a router. This is done using serial-numbers and session-ids.

It’s kicked off with a router setting up a TCP connection towards a backend RTR server followed by a series of serial-number exchanges and data records exchanges such that a cache on the validator (or RTR server) can be synced fully with a cache on the router. As mentioned, the lightweight protocol is void of all the cryptographic data that RPKI is built upon and simply deals with the validated routing list, which consists of CIDRs, ASNs and maybe a MaxLength parameter.

Here’s a simple Cisco configuration for enabling RTR on a router:

router bgp 65001
 rpki server 192.168.1.100
  transport tcp port 8282
 !
!

The configuration can take additional parameters in order to enable SSH or similar transport options. Other platforms (such as Juniper, Arista, Bird 2.0, etc) have their own specific configuration language.

The RTR protocol supports IPv4 and IPv6 routing information (as you would expect).

Being specified as a lightweight protocol, RTR allows the data to be transferred quickly. With a session-id created by the RTR cache server plus serial-numbers exchanged between cache servers and routers, there’s the solid ability for route authentication data on the router to be kept fresh with a minimum amount of actual data being transferred. Remember, as we said above, the router has much better things to do with its control plane processor like running the BGP convergence algorithm, or SRv6, or ISIS, or any of the protocols needed to manage routing tables.

All aspects of RPKI data processing are built around solid cryptographic principles. The five RIRs each hold a root key called a Trust Anchor (TA). Each publishes data fully signed up/down so that every piece of information can be proven to be correct and without tampering. A validators job is to do that processing and spit out (or store) a list of valid ROAs (Route Origin Authorizations) that are assertions traceable back to a known source. If you want to study this protocol, you can start with RFC6480 and work forward through all the other relevant RFCs (Hint: It’s at least thirty more RFCs from RFC6483 thru RFC8210 and counting).

However, RTR does not carry that trust through to the Internet router. All that complexity (and hence assertions) are stripped away before a router sees anything. It is 100% up to the network operator to build a reliable and secure path between validator or RTR cache and router so that this lightweight transfer is still trusted.

RTR helps somewhat in this space. It provides more than one way to communicate between cache server and router. The RFC defines various methods to communicate.

• A plain TCP connection (which is clearly insecure). In this case the RFC states: “the cache and routers MUST be on the same trusted and controlled network.”.

• A TCP connection with TCP-AO transport.

• A Secure Shell version 2 (SSHv2) transport.

• A TCP connection with TCP MD5 transport (which is already obsoleted by TCP-AO).

• A TCP connection over IPsec transport.

• Transport Layer Security (TLS) transport.

This plethora of options is all well and good; however, there’s no useful implementation of TCP-AO out in the production world and hence (ironically) a lot of early implementations are living with plain-text communications. SSH and TLS are much better options; however, this comes with classic operational problems to solve. For example, in SSH’s case, the RFC states:

It is assumed that the router and cache have exchanged keys out of band by some reasonably secured means.

For a TLS connection, there’s also some worthwhile security setup mentioned in the RFC. It starts off as follows:

Client routers using TLS transport MUST present client-side certificates to authenticate themselves to the cache in order to allow the cache to manage the load by rejecting connections from unauthorized routers.

Then the RFC continues with enough information to secure the connection fully. If implemented correctly, then security is correctly provided between RTR cache and router such that no MITM attack can take place.

Assuming that these operational issues are handled fully then the RTR protocol is a perfect protocol for operationally implementing RPKI’s final linkage into the routers.

A modern router software stack can be configured to run RTR against a cache. If you have a test lab (as most modern networks do); then you have all you need to see RPKI route filtering (and the dropping of invalid routes).

However, if you are without a router and want to see RTR in action, Cloudflare has just placed rpki-rtr-client on GitHub. This software, written in Python, performs the router portion of the RTR protocol and comes with enough debug output that it can also be used to help write new RTR caches, or test existing code bases. The code was written directly from the RFC and then tested against a public RTR cache that Cloudflare operates.

$ pip3 install netaddr
...
$ git clone https://github.com/cloudflare/rpki-rtr-client.git
...
$ cd rpki-rtr-client
$

Operating the client is easy (and doubly-so if you use the Cloudflare provided cache).

$ ./rtr_client.py -h rtr.rpki.cloudflare.com -p 8282
...
^C
$

As there is no router (and hence no dropping of invalids) this code simply creates data files for later review. See the README file for more information.

$ ls -lt data/2020-02
total 21592
-rw-r--r--  1 martin martin  5520676 Feb 16 18:22 2020-02-17-022209.routes.00000365.json
-rw-r--r--  1 martin martin  5520676 Feb 16 18:42 2020-02-17-024242.routes.00000838.json
-rw-r--r--  1 martin martin      412 Feb 16 19:56 2020-02-17-035645.routes.00000841.json
-rw-r--r--  1 martin martin      272 Feb 16 20:16 2020-02-17-041647.routes.00000842.json
-rw-r--r--  1 martin martin      643 Feb 16 20:36 2020-02-17-043649.routes.00000843.json
$

As the RTR protocol communicates and increments its serial-number, the rpki-rtr-client software writes the routing information is a fresh file for later review.

$ for f in data/2020-02/*.json ; do echo "$f `jq -r '.routes.announce[]|.ip' < $f | wc -l` `jq -r '.routes.withdraw[]|.ip' < $f | wc -l`" ; done
data/2020-02/2020-02-17-022209.routes.00000365.json   128483        0
data/2020-02/2020-02-17-024242.routes.00000838.json   128483        0
data/2020-02/2020-02-17-035645.routes.00000841.json        3        6
data/2020-02/2020-02-17-041647.routes.00000842.json        5        0
data/2020-02/2020-02-17-043649.routes.00000843.json        9        5
$

Valid ROAs are listed as follows:

$ jq -r '.routes.announce[]|.ip,.asn,.maxlen' data/2020-02/*0838.json | paste - - - | sort -V | head
1.0.0.0/24      13335   null
1.1.1.0/24      13335   null
1.9.0.0/16      4788    24
1.9.12.0/24     65037   null
1.9.21.0/24     24514   null
1.9.23.0/24     65120   null
1.9.31.0/24     65077   null
1.9.65.0/24     24514   null
1.34.0.0/15     3462    24
1.36.0.0/16     4760    null
$

The code can also dump the raw binary protocol and then replay that data to debug the protocol.

As the code is on GitHub, any protocol developer can feel free to expand on the code.

The present RFC defines version 1 of the protocol and it is expected that this lightweight protocol will progress to also include additional functions, but stay lightweight. RPKI is a Route Origin Validation protocol (i.e. mapping an IP route or CIDR to an ASN). It does not provide support for validating the AS-PATH. Neither does it provide any support for IRR databases (which are non-cryptographically-signed routing definitions). Presently IRR data is the primary method used for filtering routing on the global Internet. Today that is done by building massive filter lists within a router's configuration file and not via a lightweight protocol like RTR.

At the present time there’s an IETF proposal for RTR version 2. This is draft, work alongside the ASPA (Autonomous System Provider Authorization) draft and draft work. These draft documents from Alexander Azimov et al. define ASPA extending the RPKI data structures to handle BGP path information. The version 2 of RTR protocol should provide the required messaging in order to move ASPA data into the router.

Additionally, RPKI is going to potentially further expand, at some point, from today's singular data type (the ROA object). Just like with the ASPA draft, RTR will need to advance in lock-step. Hopefully the open-source code we have published will help this effort.

If RPKI is to become ubiquitous, then RTR support in all BGP speaking Internet routers is going to be required. Vendors need to complete their RTR software delivery and additionally support some of the more secure transport definitions from the RFC. Additionally, should the protocol advance, then timely support for the new version will be needed.

Cloudflare continues to be committed to a secure Internet; so should you also have the same thoughts and you like what you’ve read here or elsewhere on our blog; then please take a look at our jobs page. We have software and network engineering open roles in many of our offices around the world.

Courtesy of MIT Advanced Network Architecture Group 

So much has happened since that day (October 29’th to be exact) in 1969, in fact it’s an understatement to say “so much has happened”! It’s unclear that one blog article would ever be able to capture the full history of packets from then to now. Here at Cloudflare we say we are helping build a “better Internet”, so it would make perfect sense for us to honor the history of the Arpanet and its successor, the Internet, by focusing on some of the other folks that have helped build a better Internet.

Nothing takes away from what happened that October day. The move from a circuit-based networking mindset to a packet-based network is momentus. The phrase net-heads vs bell-heads was born that day - and it’s still alive today! The basics of why the Internet became a permissionless innovation was instantly created the moment that first packet traversed that network fifty years ago.

Professor Leonard (Len) Kleinrock continued to work on the very-basics of packet networking. The network used on that day expanded from two nodes to four nodes (in 1969, one IMP was delivered each month from BBN to various university sites) and created a network that spanned the USA from coast to coast and then beyond.

In the 1973 map there’s a series of boxes marked TIP. These are a version of the IMP that was used to connect computer terminals along with computers (hosts) to the ARPANET. Every IMP and TIP was managed by Bolt, Beranek and Newman (BBN), based in Cambridge Mass. This is vastly different from today’s Internet where every network is operated autonomously.

By 1977 the ARPANET had grown further with links from the United States mainland to Hawaii plus links to Norway and the United Kingdom.

ARPANET logical map 1977 via Wikipedia

Focusing back to that day in 1969, Steve Crocker (who was a graduate student at UCLA at that time) headed up the development of the NCP software. The Network Control Program (later remembered as Network Control Protocol) provided the host to host transmission control software stack. Early versions of telnet and FTP ran atop NCP.

During this journey both Len Kleinrock, Steve Crocker, and the other early packet pioneers have always been solid members of the Internet community and continue to deliver daily to a better Internet.

Steve Crocker and Bill Duvall have written a guest blog about that day fifty years ago. Please read it after you've finished reading this blog.

BTW: Today, on this 50th anniversary, UCLA is celebrating history via this symposium (see also https://samueli.ucla.edu/internet50/).

Their collective accomplishments are extensive and still relevant today.

In 1973 Vint Cerf was asked to work on a protocol to replace the original NCP protocol. The new protocol is now known as TCP/IP. Of course, everyone had to move from NCP to TCP and that was outlined in RFC801. At the time (1982 and 1983) there were around 200 to 250 hosts on the ARPANET, yet that transition was still a major undertaking.

Finally, on January 1st, 1983, fourteen years after that first packet flowed, the NCP protocol was retired and TCP/IP was enabled. The ARPANET got what would become the Internet’s first large scale addressing scheme (IPv4). This was better in so many ways; but in reality, this transition was just one more stepping stone towards our modern and better Internet.

Some people write code, some people write documents, some people organize documents, some people organize numbers. Jon Postel did all of these things. Jon was the first person to be in charge of allocating numbers (you know - IP addresses) back in the early 80’s. In a way it was a thankless job that no-one else wanted to do. Jon was also the keeper of the early documents (Request For Comment or RFCs) that provide us with how the packet network should operate. Everything was available so that anyone could write code and join the network. Everyone was also able to write a fresh document (or update an existing document) so that the ecosystem of the Arpanet could grow. Some of those documents are still in existence and referenced today. RFC791 defines the IP protocol and is dated 1981 - it’s still an active document in-use today! Those early days and Jon’s massive contributions have been well documented and acknowledged. A better Internet is impossible without these conceptual building blocks.

Jon passed away in 1998; however, his legacy and his thoughts are still in active use today. He once said within the TCP world: “Be conservative in what you send, be liberal in what you accept”. This is called the robustness principle and it’s still key to writing good network protocol software.

What’s the use of a protocol if you don’t have software to speak it. In the early 80’s there were many efforts to build both affordable and fast hardware, along with the software to speak to that hardware. At the University of California, Berkeley (UCB) there was a group of software developers tasked in 1980 by the Defense Advanced Research Projects Agency (DARPA) to implement the brand-new TCP/IP protocol stack on the VAX under Unix. They not-only solved that task; but they went a long way further than just that goal.

The folks at UCB (Bill Joy, Marshall Kirk McKusick, Keith Bostic, Michael Karels, and others) created an operating system called 4.2BSD (Berkeley Software Distribution) that came with TCP/IP ingrained in its core. It was based on the AT&T’s Unix v6 and Unix/32V; however it had significantly deviated in many ways. The networking code, or sockets as its interface is called, became the underlying building blocks of each and every piece of networking software in the modern world of the Internet. We at Cloudflare have written numerous times about networking kernel code and it all boils down to the code that was written back at UCB. Bill Joy went on to be a founder of Sun Microsystems (which commercialized 4.2BSD and much more). Others from UCB went on to help build other companies that still are relevant to the Internet today.

Fun fact: Berkeley’s Unix (or FreeBSD, OpenBSD, NetBSD as its variants are known) is now the basis of every iPhone, iPad and Mac laptops software in existence. Android’s and Chromebooks come from a different lineage; but still hold those BSD methodologies as the fundamental basis of all their networking software.

Do you believe that Al Gore invented the Internet? It’s actually doesn’t matter which side of this statement you want to argue; the simple fact is that the US Government funded the National Science Foundation (NSF) with the task of building an “information superhighway”. Al Gore himself said: “how do we create a nationwide network of information superhighways? Obviously, the private sector is going to do it, but the Federal government can catalyze and accelerate the process. '' He said that statement on September 19, 1994 and this blog post author knows that fact because I was there in the room when he said it!

The United States Federal Government help fund the growth of the Arpanet into the early version of the Internet. Without the government's efforts, we may not have been where we are today. Luckily, just a handful of years later, the NSF decided that in fact the commercial world could and should be the main building blocks for the Internet and instantly the Internet as we know it today was born. Packets that fly across commercial backbones are paid for via commercial contracts. The parts that are still funded by the government (any government) are normally only the parts used by universities, or military users.

But this author is still going to thank Al Gore for helping create a better Internet back in the early 90’s.

What can I say? In 1989 Tim Berners-Lee (who was later knighted and is now Sir Tim) invented the World Wide Web and we would not have billions of people using the Internet today without him. Period!

via Reddit

via Reddit

Yeah, let's clear up that subtle point. Sir Tim invented the World Wide Web (WWW) and Vint Cerf invented the Internet. When folks talk about using one or the other, it’s worth reminding then there is a difference. But I digress!

Sir Tim’s creation is what provides everyday folks with a window into information on the Internet. Before the WWW we had textual interfaces to information; but only if you knew where to look and what to type. We really need to remember every time we click on a link or press submit to buy something, that the only way that is usable is such mass and uniform form is because of Sir Tim’s creation.

Random Early Detection (RED) is an algorithm that saved the Internet back in the early 90’s. Built on earlier work by Van Jacobson, it defined a method to drop packets when a router was overloaded, or more importantly about to be overloaded. Packet network, before Van Jacobson’s or Sally Floyd’s work, would congest heavily and slow down. It seemed natural to never throw away data; but between the two inventors of RED, that all changed. Her follow-up work is described in an August 1993 paper.

Networks have become much more complex since August 1993, yet the RED code still exists and is used in nearly every Unix or Linux kernel today. See the tc-red(8) command and/or the Linux kernel code itself.

It’s with great sorrow that Sally Floyd passed away in late August. But, rest assured, her algorithm will possibly be used to help keep a better Internet flowing smoothly forever.

Remember that comment by Al Gore above saying that the private sector would build the Internet. Back in the late 90’s that’s exactly what happened. Telecom companies were selling capacity to fledgling ISPs. Nationwide IP backbones were being built by the likes of PSI, Netcom, UUnet, Digex, CAIS, ANS, etc. The telco’s themselves like MCI, Sprint, but interestingly not AT&T at the time, were getting into providing Internet access in a big way.

In the US everything was moving very fast. By the mid-90’s there was no way to get a connection anymore from a regional research network for your shiny new ISP. Everything had all gone commercial and the NSF funded parts of the Internet were not available for commercial packets.

The NSF, in it’s goal to allow commercial networks to build the Internet, had also specified that those networks should interconnect at four locations around the country. New Jersey, Chicago, Bay Area, California, and Washington DC area.

Network Access Point via Wikipedia

The NAP’s, as they were called, were to provide interconnection between networks and to provide the research networks a way to interconnect with commercial network along with themselves. The NAPs suddenly exploded in usage, near-instantly needing to be bigger, The buildings they were housed in ran out of space or power or both! Yet those networks needed homes, interconnections needed a better structure and the old buildings that were housing the Internet’s routers just didn’t cut it anymore.

Jay and Al had a vision. New massive datacenters that could securely house the growing need for the power-hungry Internet. But that’s only a small portion of the vision. They realized that if many networks all lived under the same roof then interconnecting them could indeed build a better Internet. They installed Internet Exchanges and a standardized way of cross-connecting from one network to another. They were carrier neutral, so that everyone was treated equal. It was, what became known as the “network effect” and it was a success. The more networks you had under one roof, the more that other networks would want to be housed within those same roofs. The company they created was (and still is) called Equinix. It wasn’t the first company to realize this; but it sure has become one of the biggest and most successful in this arena.

Today, a vast amount of the Internet uses Equinix datacenters, it’s IXs along with similar offerings from similar companies. Jay and Al’s vision absolutely paved the way to a better internet.

It turns out that people realized that the modern Internet is not all-commercial all-the-time. There is a need for other influences to be had. Civil society, governments, academics, along with those commercial entities should also have a say in how the Internet evolves. This brings into the conversation a myriad of people that have either been members of The Internet Society (ISOC) and/or have worked directly for ISOC over it’s 27+ years. This is the organization that manages and helps fund the IETF (where protocols are discussed and standardized). ISOC plays a decisive role at The Internet Governance Forum (IGF), and fosters a clear understanding of how the Internet should be used and protected to both the general public and regulators worldwide. ISOCs involvement with Internet Exchange development (vital as the Internet grows and connects users and content) has been a game changer for many-many countries, especially in Africa.

ISOC has an interesting funding mechanism centered around the dotORG domain. You may not have realized that you were helping the Internet grow when you registered and paid for your .org domain; however, you are!

Over the life of ISOC, the Internet has moved from being the domain of engineers and scientists into something used by nearly everyone; independent of technical skill or in-fact a full understanding of it’s inner workings. ISOC’s mission is "to promote the open development, evolution and use of the Internet for the benefit of all people throughout the world". It has been a solid part of that growth.

Giving voice to everyone on how the Internet could grow and how it should (or should not be) regulated, is front-and-center for every person involved with ISOC globally. Defining both an inclusive Internet and a better Internet is the everyday job for those people.

In the 1988, amongst other things, Professor Kanchana Kanchanasut registered and operated the country Top Level Domain .TH (which is the two-letter ISO 3166 code for Thailand). This was the first country to have a TLD; something all countries take for granted today.

Also in 1988, five Thai universities got dial-up connections to the Internet because of her work. However, the real breakthrough came when Prof. Kanchanasut’s efforts led to the first leased line interconnecting Thailand to the nascent Internet of the early 90’s. That was 1991 and since then Thailand’s connectivity has exploded. It’s an amazingly well connected country. Today it boasts a plethora of mobile operators, and international undersea and cross-border cables, along with Prof. Kanchanasut’s present-day work spearheading an independent and growing Internet Exchange within Thailand.

In 2013, the "Mother of the Internet in Thailand" as she is affectionately called, was inducted into the Internet Hall of Fame by the Internet Society. If you’re in Thailand, or South East Asia, then she’s the reason why you have a better Internet.

In the fifty years since that first packet there have been heros, both silent and profoundly vocal that have moved the Internet forward. There’s no was all could be named or called out; however, you will find many listed if you go look. Wander through the thousands of RFC’s, or check out the Internet Hall of Fame. The Internet today is a better Internet because anyone can be a contributor.

Cloudflare, or in fact any part of the Internet, would not be where it is today without the groundbreaking work of these people plus many others unnamed here. This fifty year effort has moved the needle in such a way that without all of them the runaway success of the Internet could not have been possible!

Cloudflare is just over nine years old (that’s only 18% of this fifty year period). Gazillions and gazillions of packets have flowed since Cloudflare started providing it services and we sincerely believe we have done our part with those services to build a better Internet.. Oh, and we haven’t finished our work, far from it! We still have a long way to go in helping build a better Internet. And we’re just getting started!

A letter from Matthew Prince (@eastdakota) and Michelle Zatlyn (@zatlyn) #BetterInternet $NET https://t.co/BHLI8MuuTS pic.twitter.com/Jirb0bPUzJ

— Cloudflare (@Cloudflare) September 13, 2019

If you’re interested in helping build a better Internet and want to join Cloudflare in our offices in San Francisco, Singapore, London, Austin, Sydney, Champaign, Munich, San Jose, New York or our new Lisbon Portugal offices, then buzz over to our jobs page and come join us! #betterInternet

A recap on what happened Monday

On Monday we wrote about a painful Internet wide route leak. We wrote that this should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. That blog entry came out around 19:58 UTC, just over seven hours after the route leak finished (which will we see below was around 12:39 UTC). Today we will dive into the archived routing data and analyze it. The format of the code below is meant to use simple shell commands so that any reader can follow along and, more importantly, do their own investigations on the routing tables.

This was a very public BGP route leak event. It was both reported online via many news outlets and the event’s BGP data was reported via social media as it was happening. Andree Toonk tweeted a quick list of 2,400 ASNs that were affected.

Quick dumps through the data, showing about 2400 ASns (networks) affected. Cloudflare being hit the hardest. Top 20 of affected ASns below pic.twitter.com/9J7uvyasw2

— Andree Toonk (@atoonk) June 24, 2019

This blog contains a large number of acronyms and those are explained at the end of the blog.

The RIPE NCC operates a very useful archive of BGP routing. It runs collectors globally and provides an API for querying the data. More can be seen at https://stat.ripe.net/. In the world of BGP all routing is public (within the ability of anyone collecting data to have enough collections points). The archived data is very valuable for research and that’s what we will do in this blog. The site can create some very useful data visualizations.

Presently, the RIPEstat data gets ingested around eight to twelve hours after real-time. It’s not meant to be a real-time service. The data can be queried in many ways, including a full web interface and an API. We are using the API to extract the data in a JSON format.

We are going to focus only on the Cloudflare routes that were leaked. Many other ASNs were leaked (see the tweet above); however, we want to deal with a finite data set and focus on what happened to Cloudflare’s routing. All the commands below can be run with ease on many systems. Both the scripts and the raw data file are now available on GitHub. The following was done on MacBook Pro running macOS Mojave.

First we collect 24 hours of route announcements and AS-PATH data that RIPEstat sees coming from AS13335 (Cloudflare).

$ # Collect 24 hours of data - more than enough
$ ASN="AS13335"
$ START="2019-06-24T00:00:00"
$ END="2019-06-25T00:00:00"
$ ARGS="resource=${ASN}&starttime=${START}&endtime=${END}"
$ URL="https://stat.ripe.net/data/bgp-updates/data.json?${ARGS}"
$ # Fetch the data from RIPEstat
$ curl -sS "${URL}" | jq . > 13335-routes.json
$ ls -l 13335-routes.json
-rw-r--r--  1 martin  staff  339363899 Jun 25 08:47 13335-routes.json
$

That’s 340MB of data - which seems like a lot, but it contains plenty of white space and plenty of data we just don’t need. Our second task is to reduce this raw data down to just the required data - that’s timestamps, actual routes, and AS-PATH. The third item will be very useful. Note we are using jq, which can be installed on macOS with the  brew package manager.

$ # Extract just the times, routes, and AS-PATH
$ jq -rc '.data.updates[]|.timestamp,.attrs.target_prefix,.attrs.path' < 13335-routes.json | paste - - - > 13335-listing-a.txt
$ wc -l 13335-listing-a.txt
691318 13335-listing-a.txt
$

We are down to just below seven hundred thousand routing events, however, that’s not a leak, that’s everything that includes Cloudflare’s ASN (the number 13335 above). For that we need to go back to Monday’s blog and realize it was AS396531 (Allegheny Technologies) that showed up with 701 (Verizon) in the leak. Now we reduce the data further:

$ # Extract the route leak 701,396531
$ # AS701 is Verizon and AS396531 is Allegheny Technologies
$ egrep '701,396531' < 13335-listing-a.txt > 13335-listing-b.txt
$ wc -l 13335-listing-b.txt
204568 13335-listing-b.txt
$

At 204 thousand data points, we are looking better. It’s still a lot of data because BGP can be very chatty if topology is changing. A route leak will cause exactly that. Now let’s see how many routes were affected:

$ # Extract the actual routes affected by the route leak
$ cut -f2 < 13335-listing-b.txt | sort -V -u > 13335-listing-c.txt
$ wc -l 13335-listing-c.txt
101 13335-listing-c.txt
$

It’s a much smaller number. We now have a listing of at least 101 routes that were leaked via Verizon. This may not be the full list because route collectors like RIPEstat don’t have direct feeds from Verizon, so this data is a blended view with Verizon’s path and other paths. We can see that if we look at the AS-PATH in the above files. Please note that I had a typo in this script when this blog was first published and only 20 routes showed up because the -n vs -V option was used on sort. Now the list is correct with 101 affected routes. Please see this short article from stackoverflow to see the issue.

Here’s a partial listing of affected routes.

$ cat 13335-listing-c.txt
8.39.214.0/24
8.42.245.0/24
8.44.58.0/24
...
104.16.80.0/21
104.17.168.0/21
104.18.32.0/21
104.19.168.0/21
104.20.64.0/21
104.22.8.0/21
104.23.128.0/21
104.24.112.0/21
104.25.144.0/21
104.26.0.0/21
104.27.160.0/21
104.28.16.0/21
104.31.0.0/21
141.101.120.0/23
162.159.224.0/21
172.68.60.0/22
172.69.116.0/22
...
$

This is an interesting list, as some of these routes do not originate from Cloudflare’s network, however, they show up with AS13335 (our ASN) as the originator. For example, the 104.26.0.0/21 route is not announced from our network, but we do announce 104.26.0.0/20 (which covers that route). More importantly, we have an IRR (Internet Routing Registries) route object plus an RPKI ROA for that block. Here’s the IRR object:

route:          104.26.0.0/20
origin:         AS13335
source:         ARIN

And here’s the RPKI ROA. This ROA has Max Length set to 20, so no smaller route should be accepted.

Prefix:       104.26.0.0/20
Max Length:   /20
ASN:          13335
Trust Anchor: ARIN
Validity:     Thu, 02 Aug 2018 04:00:00 GMT - Sat, 31 Jul 2027 04:00:00 GMT
Emitted:      Thu, 02 Aug 2018 21:45:37 GMT
Name:         535ad55d-dd30-40f9-8434-c17fc413aa99
Key:          4a75b5de16143adbeaa987d6d91e0519106d086e
Parent Key:   a6e7a6b44019cf4e388766d940677599d0c492dc
Path:         rsync://rpki.arin.net/repository/arin-rpki-ta/5e4a23ea-...

The Max Length field in an ROA says what the minimum size of an acceptable announcement is. The fact that this is a /20 route with a /20 Max Length says that a /21 (or /22 or /23 or /24) within this IP space isn’t allowed. Looking further at the route list above we get the following listing:

Route Seen            Cloudflare IRR & ROA    ROA Max Length
104.16.80.0/21    ->  104.16.80.0/20          /20
104.17.168.0/21   ->  104.17.160.0/20         /20
104.18.32.0/21    ->  104.18.32.0/20          /20
104.19.168.0/21   ->  104.19.160.0/20         /20
104.20.64.0/21    ->  104.20.64.0/20          /20
104.22.8.0/21     ->  104.22.0.0/20           /20
104.23.128.0/21   ->  104.23.128.0/20         /20
104.24.112.0/21   ->  104.24.112.0/20         /20
104.25.144.0/21   ->  104.25.144.0/20         /20
104.26.0.0/21     ->  104.26.0.0/20           /20
104.27.160.0/21   ->  104.27.160.0/20         /20
104.28.16.0/21    ->  104.28.16.0/20          /20
104.31.0.0/21     ->  104.31.0.0/20           /20

So how did all these /21’s show up? That’s where we dive into the world of BGP route optimization systems and their propensity to synthesize routes that should not exist. If those routes leak (and it’s very clear after this week that they can), all hell breaks loose. That can be compounded when not one, but two ISPs allow invalid routes to be propagated outside their autonomous network. We will explore the AS-PATH further down this blog.

More than 20 years ago, RFC1997 added the concept of communities to BGP. Communities are a way of tagging or grouping route advertisements. Communities are often used to label routes so that specific handling policies can be applied. RFC1997 includes a small number of universal well-known communities. One of these is the NO_EXPORT community, which has the following specification:

    All routes received carrying a communities attribute
    containing this value MUST NOT be advertised outside a BGP
    confederation boundary (a stand-alone autonomous system that
    is not part of a confederation should be considered a
    confederation itself).

The use of the NO_EXPORT community is very common within BGP enabled networks and is a community tag that would have helped alleviate this route leak immensely.

How BGP route optimization systems work (or don’t work in this case) can be a subject for a whole other blog entry.

As we saved away the timestamps in the JSON file and in the text files, we can confirm the time for every route in the route leak by looking at the first and the last timestamp of a route in the data. We saved data from 00:00:00 UTC until 00:00:00 the next day, so we know we have covered the period of the route leak. We write a script that checks the first and last entry for every route and report the information sorted by start time:

$ # Extract the timing of the route leak
$ while read cidr
do
  echo $cidr
  fgrep $cidr < 13335-listing-b.txt | head -1 | cut -f1
  fgrep $cidr < 13335-listing-b.txt | tail -1 | cut -f1
done < 13335-listing-c.txt |\
paste - - - | sort -k2,3 | column -t | sed -e 's/2019-06-24T//g'
...
104.25.144.0/21   10:34:25  12:38:54
104.22.8.0/21     10:34:27  12:29:39
104.20.64.0/21    10:34:27  12:30:00
104.23.128.0/21   10:34:27  12:30:34
141.101.120.0/23  10:34:27  12:30:39
162.159.224.0/21  10:34:27  12:30:39
104.18.32.0/21    10:34:29  12:30:34
104.24.112.0/21   10:34:29  12:30:34
104.27.160.0/21   10:34:29  12:30:34
104.28.16.0/21    10:34:29  12:30:34
104.31.0.0/21     10:34:29  12:30:34
8.39.214.0/24     10:34:31  12:19:24
104.26.0.0/21     10:34:36  12:29:53
172.68.60.0/22    10:34:38  12:19:24
172.69.116.0/22   10:34:38  12:19:24
8.44.58.0/24      10:34:38  12:19:24
8.42.245.0/24     11:52:49  11:53:19
104.17.168.0/21   12:00:13  12:29:34
104.16.80.0/21    12:00:13  12:30:00
104.19.168.0/21   12:09:39  12:29:34
...
$

Now we know the times. The route leak started at 10:34:25 UTC (just before lunchtime London time) on 2019-06-24 and ended at 12:38:54 UTC. That’s a hair over two hours. Here’s that same time data in a graphical form showing the near-instant start of the event and the duration of each route leaked:

We can also go back to RIPEstat and look at the activity graph for Cloudflare’s AS13335 network:

Clearly between 10:30 UTC and 12:40 UTC there’s a lot of route activity - far more than normal.

Note that as we mentioned above, RIPEstat doesn’t get a full view of Verizon’s network routing and hence some of the propagated routes won’t show up.

Having the routes is useful, but now we want to look at the paths of these leaked routes to see which ASNs are involved. We knew the offending ASNs during the route leak on Monday. Now we want to dig deeper using the archived data. This allows us to see the extent and reach of this route leak.

$ # Extract the AS-PATH of the route leak
$ # Use the list of routes to extract the full AS-PATH
$ # Merge the results together to show an amalgamation of paths.
$ # We know (luckily) the last few ASNs in the AS-PATH are consistent
$ cut -f3 < 13335-listing-b.txt | tr -d '[\[\]]' |\
awk '{
  n=split($0, a, ",");
  printf "%50s\n",
    a[n-5] "_" a[n-4] "_" a[n-3] "_" a[n-2] "_" a[n-1] "_" a[n];
}' | sort -u
   174_701_396531_33154_3356_13335
   2497_701_396531_33154_174_13335
   577_701_396531_33154_3356_13335
   6939_701_396531_33154_174_13335
  1239_701_396531_33154_3356_13335
  1273_701_396531_33154_3356_13335
  1280_701_396531_33154_3356_13335
  2497_701_396531_33154_3356_13335
  2516_701_396531_33154_3356_13335
  3320_701_396531_33154_3356_13335
  3491_701_396531_33154_3356_13335
  4134_701_396531_33154_3356_13335
  4637_701_396531_33154_3356_13335
  6453_701_396531_33154_3356_13335
  6461_701_396531_33154_3356_13335
  6762_701_396531_33154_3356_13335
  6830_701_396531_33154_3356_13335
  6939_701_396531_33154_3356_13335
  7738_701_396531_33154_3356_13335
 12956_701_396531_33154_3356_13335
 17639_701_396531_33154_3356_13335
 23148_701_396531_33154_3356_13335
$

This script clearly shows the AS-PATH of the leaked routes. It’s very consistent. Reading from the back of the line to the front, we have 13335 (Cloudflare), 3356 or 174 (Level3/CenturyLink or Cogent - both tier 1 transit providers for Cloudflare). So far, so good. Then we have 33154 (DQE Communications) and 396531 (Allegheny Technologies Inc) which is still technically not a leak, but trending that way. The reason why we can state this is still technically not a leak is because we don’t know the relationship between those two ASs. It’s possible they have a mutual transit agreement between them. That’s up to them.

Back to the AS-PATH’s. Still reading leftwards, we see 701 (Verizon), which is very very bad and clear evidence of a leak. It’s a leak for two reasons. First, this matches the path when a transit is leaking a non-customer route from a customer. This Verizon customer does not have 13335 (Cloudflare) listed as a customer. Second, the route contains within its path a tier 1 ASN. This is the point where a route leak should have been absolutely squashed by filtering on the customer BGP session. Beyond this point there be dragons.

And dragons there be! Everything above is about how Verizon filtered (or didn’t filter) its customer. What follows the 701 (i.e the number to the left of it) is the peers or customers of Verizon that have accepted these leaked routes. They are mainly other tier 1 networks of Verizon in this list: 174 (Cogent), 1239 (Sprint), 1273 (Vodafone), 3320 (DTAG), 3491 (PCCW), 6461 (Zayo), 6762 (Telecom Italia), etc.

What’s missing from that list are three networks worthy of mentioning - 1299 (Telia), 2914 (NTT), and 7018 (AT&T). All three implement a very simple AS-PATH filter which saved the day for their network. They do not allow one tier 1 ISP to send them a route which has another tier 1 further down the path. That’s because when that happens, it’s officially a leak as each tier 1 is fully connected to all other tier 1’s (which is part of the definition of a tier 1 network). The topology of the Internet’s global BGP routing tables simply states that if you see another tier 1 in the path, then it’s a bad route and it should be filtered away.

Additionally we know that 7018 (AT&T) operates a network which drops RPKI invalids. Because Cloudflare routes are RPKI signed, this also means that AT&T would have dropped these routes when it receives them from Verizon. This shows a clear win for RPKI (and for AT&T when you see their bandwidth graph below)!

BREAKING - AT&T / AS 7018 is now rejecting RPKI Invalid BGP announcements they receive from their peering partners. This is big news for routing security! If AT&T can do it - you can do it! :-) https://t.co/FhjmWByagO

— Job Snijders (@JobSnijders) February 11, 2019

That all said, keep in mind we are still talking about routes that Cloudflare didn't announce. They all came from the route optimizer.

This is a great question to ask. Normally we would look at the IRR (Internet Routing Registries) to see what policy an ASN wants for it’s routes.

$ whois -h whois.radb.net AS396531 ; the Verizon customer
%  No entries found for the selected source(s).
$ whois -h whois.radb.net AS33154  ; the downstream of that customer
%  No entries found for the selected source(s).
$ 

That’s enough to say that we should not be seeing this ASN anywhere on the Internet, however, we should go further into checking this. As we know the ASN of the network, we can search for any routes that are listed for that ASN. We find one:

$ whois -h whois.radb.net ' -i origin AS396531' | egrep '^route|^origin|^mnt-by|^source'
route:          192.92.159.0/24
origin:         AS396531
mnt-by:         MNT-DCNSL
source:         ARIN
$

More importantly, now we have a maintainer (the owner of the routing registry entries). We can see what else is there for that network and we are specifically looking for this:

$ whois -h whois.radb.net ' -i mnt-by -T as-set MNT-DCNSL' | egrep '^as-set|^members|^mnt-by|^source'
as-set:         AS-DQECUST
members:        AS4130, AS5050, AS11199, AS11360, AS12017, AS14088, AS14162,
                AS14740, AS15327, AS16821, AS18891, AS19749, AS20326,
                AS21764, AS26059, AS26257, AS26461, AS27223, AS30168,
                AS32634, AS33039, AS33154, AS33345, AS33358, AS33504,
                AS33726, AS40549, AS40794, AS54552, AS54559, AS54822,
                AS393456, AS395440, AS396531, AS15204, AS54119, AS62984,
                AS13659, AS54934, AS18572, AS397284
mnt-by:         MNT-DCNSL
source:         ARIN
$

This object is important. It lists all the downstream ASNs that this network is expected to announce to the world. It does not contain Cloudflare’s ASN (or any of the leaked ASNs). Clearly this as-set was not used for any BGP filtering.

Just for completeness the same exercise can be done for the other ASN (the downstream of the customer of Verizon). In this case, we just searched for the maintainer object (as there are plenty of route and route6 objects listed).

$ whois -h whois.radb.net ' -i origin AS33154' | egrep '^mnt-by' | sort -u
mnt-by:         MNT-DCNSL
mnt-by:     MAINT-AS3257
mnt-by:     MAINT-AS5050
$

None of these maintainers are directly related to 33154 (DQE Communications). They have been created by other parties and hence they become a dead-end in that search.

It’s worth doing a secondary search to see if any as-set object exists with 33154 or 396531 included. We turned to the most excellent IRR Explorer website run by NLNOG. It provides deep insight into the routing registry data. We did a simple search for 33154 using http://irrexplorer.nlnog.net/search/33154 and we found these as-set objects.

It’s interesting to see this ASN listed in other as-set’s but none are important or related to Monday’s route-leak. Next we looked at 396531

This shows that there’s nowhere else we need to check. AS-DQECUST is the as-set macro that controls (or should control) filtering for any transit provider of their network.

The summary of all the investigation is a solid statement that no Cloudflare routes or ASNs are listed anywhere within the routing registry data for the customer of Verizon. As there were 2,300 ASNs listed in the tweet above, we can conclusively state no filtering was in place and hence this route leak went on its way unabated.

In what could be considered the only plus from Monday’s route leak, we can confirm that there was no route leak within IPv6 space. Why?

It turns out that 396531 (Allegheny Technologies Inc) is a network without IPv6 enabled. Normally you would hear Cloudflare chastise anyone that’s yet to enable IPv6, however, in this case we are quite happy that one of the two protocol families survived. IPv6 was stable during this route leak, which now can be called an IPv4-only route leak.

Yet that’s not really the whole story. Let’s look at the percentage of traffic Cloudflare sends Verizon that’s IPv6 (vs IPv4). Normally the IPv4/IPv6 percentage holds steady.

This uptick in IPv6 traffic could be the direct result of Happy Eyeballs on mobile handsets picking a working IPv6 path into Cloudflare vs a non-working IPv4 path. Happy Eyeballs is meant to protect against IPv6 failure, however in this case it’s doing a wonderful job in protecting from an IPv4 failure. Yet we have to be careful with this graph because after further thought and investigation the percentage only increased because IPv4 reduces. Sometimes graphs can be misinterpreted, yet Happy Eyeballs still did a good job even as end users were being affected.

Happy Eyeballs, described in RFC8305, is a mechanism where a client (lets say a mobile device) tries to connect to a website both on IPv4 and IPv6 simultaneously. IPv6 is sometimes given a head-start. The theory is that, should a failure exist on one of the paths (sadly IPv6 is the norm), then IPv4 will save the day. Monday was a day of opposites for Verizon.

In fact, enabling IPv6 for mobile users is the one area where we can praise the Verizon network this week (or at least the Verizon mobile network), unlike the residential Verizon networks where IPv6 is almost non-existent.

As we have already stated,Verizon had impacted their own users/customers. Let’s start with their bandwidth graph:

The red line is 24 June 2019 (00:00 UTC to 00:00 UTC the next day). The gray lines are previous days for comparison. This graph includes both Verizon fixed-line services like FiOS along with mobile.

The AT&T graph is quite different.

There’s no perturbation. This, along with some direct confirmation, shows that 7018 (AT&T) was not affected. This is an important point.

Going back and looking at a third tier 1 network, we can see 6762 (Telecom Italia) affected by this route leak and yet Cloudflare has a direct interconnect with them.

We will be asking Telecom Italia to improve their route filtering as we now have this data.

The IETF is doing work in the area of BGP path protection within the Secure Inter-Domain Routing Operations Working Group (sidrops) area. The charter of this IETF group is:

The SIDR Operations Working Group (sidrops) develops guidelines for the operation of SIDR-aware networks, and provides operational guidance on how to deploy and operate SIDR technologies in existing and new networks.

One new effort from this group should be called out to show how important the issue of route leaks like today’s event is. The draft document from Alexander Azimov et al named draft-ietf-sidrops-aspa-profile (ASPA stands for Autonomous System Provider Authorization) extends the RPKI data structures to handle BGP path information. This in ongoing work and Cloudflare and other companies are clearly interested in seeing it progress further.

However, as we said in Monday’s blog and something we should reiterate again and again: Cloudflare encourages all network operators to deploy RPKI now!

• API - Application Programming Interface

• AS-PATH - The list of ASNs that a routes has traversed so far

• ASN - Autonomous System Number - A unique number assigned for each network on the Internet

• BGP - Border Gateway Protocol (version 4) - the core routing protocol for the Internet

• IETF - Internet Engineering Task Force - an open standards organization

• IPv4 - Internet Protocol version 4

• IPv6 - Internet Protocol version 6

• IRR - Internet Routing Registries - a database of Internet route objects

• ISP - Internet Service Provider

• JSON - JavaScript Object Notation - a lightweight data-interchange format

• RFC - Request For Comment - published by the IETF

• RIPE NCC - Réseaux IP Européens Network Coordination Centre - a regional Internet registry

• ROA - Route Origin Authorization - a cryptographically signed attestation of a BGP route announcement

• RPKI - Resource Public Key Infrastructure - a public key infrastructure framework for routing information

• Tier 1 - A network that has no default route and peers with all other tier 1's

• UTC - Coordinated Universal Time - a time standard for clocks and time

• "there be dragons" - a mistype, as it was meant to be "here be dragons" which means dangerous or unexplored territories

Today, Cloudflare can quantitatively confirm that Internet access has been shut down in the Democratic Republic of the Congo, information already reported by many press organisations. This shutdown occurred as the presidential election was taking place on December the 30th, and continues as the results are published.

Sadly, this act is far from unprecedented. We have published many posts about events like this in the past, including a different post about roughly three days of Internet disruption in the Democratic Republic of the Congo less than a year ago. A painfully familiar shape can be seen on our network monitoring platform, showing that the traffic in the country is barely reaching a quarter of its typical level:

Note that the graph is based on UTC and Democratic Republic of the Congo’s capital Kinshasa has the timezone of GMT+1.

The drop in bandwidth started just before midday on 31 December 2018 (around 10:30 UTC, 11:30 local time in Kinshasa). This can be clearly seen if we overlay each 24 hour day over each other:

The red line is 31 December, the gray lines the previous eight days. Looking at today’s overlay bandwidth graph, we can confirm this has continued and is an abnormal behaviour.

Other actors on the Internet have also been reporting similar figures. We hope that we can soon inform our readers the country is normally connected to the Internet again.

While 85 million people live in the country, very few people have internet access (6.21% according to Wikipedia’s List of countries by number of Internet users page). The country is also very large (2,344,858 square kms or 905,355 sq miles) and the 11th largest country in the world - around a quarter the landmass of the USA and nearly twice as big as South Africa. These facts play together and because of limited fiber deployment within the country; there are many places that still use very limited and expensive satellite Internet access. We can see in our bandwidth kgraphs that traffic to these satellite connected locations was not affected by this shutdown:

Note that the bandwidth levels are very low and represent a very small percentage of the overall traffic into Democratic Republic of the Congo.

Comparing that graph to the one from the largest mobile provider in the country; we clearly see the distinct cutoff.

15 months ago we wrote about an outage in Togo, were we noted that this adds Togo to the list of countries like Syria (twice), Iraq, Turkey, Libya, Tunisia, etc that have restricted or revoked Internet access. We have also written about unrest in Gabon (in 2016) and The Gambia (also in 2016). In Gambia’s case, the incumbent president lost the election! In fact we wrote “Rather than clamping down on the opposition by blocking the access to the Internet, it is quite possible that the blackout in Gambia may have infuriated voters and increased the vote against the president.”. Let’s see what happens in Democratic Republic of the Congo.

We'll update this blog once we see changes to these traffic levels. The Congolese government says they will restore internet access after election results are published on January 6th. That’s four days from now.

At Cloudflare, we’ll continue to do our part to try to ensure that vulnerable voices have access to the Internet. Cloudflare’s Project Galileo and Athenian Project help protect at risk websites -- such as those run by human rights organizations, journalists, and government entities reporting election results -- from being knocked offline by cyber attack.

We also support the principles for a Contract for the Web, which urge governments to commit to keeping all of the Internet available, all of the time, and Access Now’s #KeepitOn campaign. We can only hope that these efforts will yield more positive results in 2019.

According to a published report earlier this year, 84% of the Nigerian population own a mobile device (193 million population and 162 million mobile subscriptions). Again, that’s #1 for any country in Africa. But why so connected? Maybe because Nigeria (and Lagos specifically) is always on the move!

Lagos, as those that know the city say, never sleeps, it’s filled with color from the food to fashion to even the diverse people going about their business. The vibrancy of the city is like a hard slap to the face, no matter what you have been told, your first time here will still knock you out. In Lagos, anything is possible, from the sadness of poverty to the clearly visible upper class, the city sucks you in like a surfers dream wave. Visitor come into Lagos and leave feeling like they’ve been through a unique experience. The traffic is mind blowing and the same goes for the work pace.

Lagos, a city always on the move!

Cloudflare has now entered Lagos with a secure facility colocated and interconnected to one of the primary undersea cable operators along with a full connection to both the local Internet Exchange (IXPN) and the newly announced WAF-IX Lagos exchange. With every data center we add, local users not only connect faster and more reliably to sites proxied by Cloudflare, but our global DDoS mitigation becomes stronger and more robust.  Instead of serving Nigeria from Europe (London, Lisbon, etc), content can be delivered locally from Lagos.

If you follow the African telecom industry; then you'll know that West Africa’s economy is dominated by Nigeria; but few outside the region know that Nigeria is dominated by the phrase mobile-first. A country thats mobile-first means that mobile has eclipsed all other connectivity methods. Other countries with this status include Indonesia, China, Kenya, Philippines, Myanmar and many others. Being mobile-first doesn’t mean smartphone dominance (Myanmar qualifies because of the availability of $20 phones); but it does mean dependence on mobile infrastructure. In fact SMS or texting is still the number one common communications method for most of these places. But smartphone penetration in Nigeria is growing fast and sitting at around 21 million according to the recent report referenced above.

La la land has Hollywood, India has Bollywood and Nigeria has Nollywood. Yes, movie-making in Nigeria is a booming business with 1,500 or more movies produced a year and yet Nollywood is only twenty-five years old. That’s still less movies than Bollywood; but that doesn’t worry any of the Nigerian statisticians because they restate the revenue number based on a per-capita basis and that makes Nollywood bigger than Bollywood. But we digress.

The one thing that all that mobile penetration can provide is a distribution method for Nollywood’s movies and while this has nothing to do with Cloudflare’s commitment to Nigeria; it’s interesting to see that Netflix (the movie and tv streaming giant) has just invested heavily into Nollywood!

Even Hollywood has tapped Nigerians for major roles. David Oyelowo, the British actor of Nigerian descent, played Martin Luther King Jr. in the 2014 movie Selma. Danny Glover (of Lethal Weapon and The Color Purple fame) is of Nigerian descent and has acted in a Nigeria based movie called 93 Days. Then there’s Forest Whitaker (Last King of Scotland), John Boyega (Finn in Star Wars: The Force Awakens and The Last Jedi), and geek-favorite Richard Ayoade (Maurice Moss in the IT Crowd); all of Nigerian descent.

Lagos is well served by undersea cables that reach northwards to Europe and south towards South Africa. In fact every African west coast undersea cable lands in Lagos Nigeria.

• ACE (Africa Coast to Europe))

• Glo-1 (and Glo-2)

• MainOne

• NCSCS (Nigeria Cameroon Submarine Cable System)

• SAT-3/WASC

• WACS (West African Cable System)

While most of these cables are pumping bandwidth from North to South (i.e. Europe to Africa); some are now being used for inter-country connectivity at the IP backbone level. WACS has Nigeria, Angola, and South Africa connected. MainOne has Ghana and Nigeria connected, etc.

When we look at the Alexa-top-50 list for Nigerian users we find that 18 of those sites are Cloudflare customers. That means an instant win for both consumers (those smartphone users) and the website or app operators. BTW: of the remaining 32 sites, 15 are owned by massive content players (search, online video, social media, operating systems or phones) and of the remaining 17 sites (Wikipedia being one of highest visited on that list); they are mainly non-CDN’ed websites that are hosted outside the country (Europe or USA).

Cloudflare in West Africa far from finished! We still have places like Senegal, Côte d’Ivoire, and Ghana to do (just to name a few). As always, stay tuned as Cloudflare grows the network! Oh, and by now any regular reader of our blogs will know that we are growing, both in network deployment and in fantastic staff. So pop over to the jobs page and see what interests you!

Ulaanbaatar (or Ulan Bator; but shortened to UB by many) is the capital of Mongolia and located nearly a mile above sea level just outside the Gobi Desert - a desert that spans a good percentage of Central Asia’s Mongolia. (The rest of the Gobi Desert extends into China). The country is nestled squarely between Russia to the north and China to the south. It’s also home to some of the richest and ancient customs and festivals around. It’s those festivals that successfully draw in the tourists who want to experience something quite unique. Luckily, even with all the tourists, Mongolia has managed to keep its local customs; both in the cities and within its nomadic tribes.

via Wikipedia

History also has drawn explorers and conquerors to and from the region; but more on that later.

Any avid reader of our blogs will know that we frequently explain that the expansion of our network provides customers and end-users with both more capacity and less latency. That goal (covering 95% of the Internet population with 10 milliseconds or less of latency) means that Mongolia was seriously on our radar.

Now we have a data center in Ulaanbaatar, Mongolia, latency into that blue sky country is significantly reduced. Prior to this data center going live we were shipping bits into the country via Hong Kong, a whopping 1,800 miles away (or 50 milliseconds if we talk latency). That's far! We know this new data center is a win-win for both mobile and broadband customers within the country and for Cloudflare customers as a whole.

Ulaanbaatar is city number 154 on Cloudflare’s network. Our expansion into cities like Ulaanbaatar doesn’t just happen instantly; it takes many teams within Cloudflare in order to successfully deploy in a place like this.

However, before deploying, we need to negotiate a place to deploy into. A new city requires a secure data center for us to build into. A bandwidth partner is also required. We need to get access to the local networks and to also acquire cache-fill bandwidth in order to operate our CDN. Once we have those items negotiated, we can focus on the next steps. Any site we build has to match our own stringent security standards (we are PCI/DSS compliant – hence all our data centers need to also be PCI/DSS compliant). That’s a paperwork process, which surprisingly takes longer than most other stages (because we care about security).

Then logistics kicks in. A BOM (Bill of Materials) is created. Serial numbers recorded. Power plugs chosen. Fun fact: Cloudflare data centers are nearly all identical, except the power cables. While we live in a world where fiber optic cables and connectors are universal, independent of location (or speed in some cases), the power connections we receive for our data centers vary widely as we expand around the globe.

The actual shipping is possibly the more interesting part of the logistics process. Getting a few pallets of hardware strapped up and ready to move is only a small part of the process. Paperwork again becomes the all-powerful issue. Each country has its own customs and import process, each shipping company has its own twist on how to process things, and each shipment needs to be coordinated with a receiving party. Our logistics team pulls off this miracle for new sites, upgrades for existing sites, replacement parts for existing sites, all while sometimes calmly listening to mundane on-hold music from around the globe.

Then our hardware arrives! Seriously, this is a biggie and those around the office that follow these new city builds are always celebrating on those days. The logistics team has done their part; now it’s time for the deployment team to kick-in.

The deployment team’s main goal is to get hardware racked-and-stacked in a site where (in most cases) we are contracting out the remote-hands to do this work. Sometimes we send people to a site to build it; however, that’s not a scalable process and hence we use local remote-hands contractors to do this heavy-lifting and cabling. There are diagrams, there are instructions, there are color-coded cables (‘cause the right cable needs to go into the right socket). Depending on the size of the build; it can be just a days work or up-to a weeks worth of work. We vary our data center sizes based on the capacity needs for each city. Once racked-and-stacked there is one more job to get done within the Infrastructure team. They get the initial private network connection enabled and secured. That single connection provides us with the ability to continue to the next step. Actually setting up the network and servers at the new site.

Every new data center site is shipped with zero configuration loaded into network hardware and compute servers. They all ship raw with no Cloudflare knowledge embedded into them. This is by design. The network teams first goal is to configure the routers and switches, which is mainly a bootstrap process in order for the hardware to phone-home and securely request its full configuration setup. We have previously written about our extensive network automation methods. In the case of a new site, it’s not that different. Once the site can communicate back home, it’s clear to the software base that its configuration is out of date. Updates are pushed to the site and network monitoring is automatically enabled.

But let's not paint a perfect rosy picture. There can still be networking issues. Just one is worth pointing-out as it’s a recurring issue and one that plagues the industry globally. Fiber optic cables can sometimes be plugged in with their receive and transmit sides reversed. It’s a 50:50 chance of being right. Sometimes it just amazes us that we can’t get this fixed; but … a quick swap of the two connectors and we are back in business!

No conversation about Mongolia would be valid unless we discuss Genghis Khan. Back in the 13th century, Genghis Khan founded the Mongol Empire. He unified the tribes of what is now Mongolia (and beyond). He established the largest land empire in history and is well described both online and via various TV documentaries (The History Channel doesn’t skimp when it comes to covering him). Genghis Khan was a name of honor that he didn’t receive till 1206. Before that he was just named “Temujin”.

Photo by SarahTz CC by/2.0

Around 30 miles outside Ulaanbaatar is the equestrian statue of Genghis Khan on the banks of the Tuul River in Gorkhi Terelj National Park. Pictured above, this statue is 131 feet tall and built from 250 tons of steel.

We get to announce our new city (and country) data center during a very special time. The Golden Eagle Festival takes place during the first weekend of October (that’s October 6 and 7 this year). It’s a test of a hunter’s speed and agility. In this case the hunters (nomads of Mongolia) are practicing an ancient technique of using Golden Eagles to hunt. It takes place in the far west of the country in the Altai Mountains.

The most famous festival in Mongolia is the Naadam Festival in mid-July. So many things going on within that festival, all of which is a celebration of Mongolian and nomadic culture. The festival celebrates the local sports (wrestling, archery, and horse racing) along with plenty of arts. The opening ceremony can be quite elaborate.

When discussing Mongolia, travelers nearly always want to make sure their itineraries overlap at least one of these festivals!

Last week was Cloudflare’s Birthday Week and we announced many services and products. Rest-assured that everything we announced is instantly available in Ulaanbaatar, Mongolia (data center city 154) just like it’s available everywhere else on Cloudflare’s network.

One final trivia point regarding our Ulaanbaatar data center. With Ulaanbaatar live, we now have datacenters covering the letters A thru Z (from Amsterdam to Zurich), i.e. with U added, every letter is now fully represented.

If you like what you’ve read and want to come join our infrastructure team, our logistics team, our network team, our SRE team, or any of the teams that help with these global expansions, then we would love to see your resume or CV. Look here for job opening.

Today we need to talk about why existing operational practices for BGP routing and filtering have to significantly improve in order to finally stop route leaks and hijacks; which are sadly pervasive in today’s Internet routing world. In fact, the subtle art of running a BGP network and the various tools (both online and within your a networks subsystems) that are vital to making the Internet routing world a safe and reliable place to operate need to improve.

Internet routing and BGP and security along with its operational expertise must improve globally.

photo by Marco Verch by/2.0

Nothing specific triggered today’s writing except the fact that Cloudflare has decided that it's high-time we took a leadership role to finally secure BGP routing. We believe that each and every network needs to change its mindset towards BGP security both on a day-by-day and a long-term basis.

It's time to stop BGP route leaks and hijacks by deploying operationally-excellent RPKI!

Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a BGP route announcement with the correct originating AS number. RPKI is defined in RFC6480 (An Infrastructure to Support Secure Internet Routing). Cloudflare commits to RPKI.

Because any route can be originated and announced by any random network, independent of its rights to announce that route, there needs to be an out-of-band method to help BGP manage which network can announce which route. That system exists today. It's part of the IRR (Internet Routing Registry) system. Many registries exist, some run by networks, some by RIRs (Regional Internet Registries) and the grand daddy of IRRs, Merit's RADB service. This service provides a collective method to allow one network to filter another networks routes.

This works somewhat. An invalid announcement is normally squashed near-instantly as the route crosses an ASN boundary because one network is meant to filter the other network (based on rules created from the IRR database). This of course doesn’t happen perfectly - in fact, far from it. Route leaks or route hijacks happen more often than they should. A fact that is well documented. Here’s the highlights:

• 1997 - AS7007 mistakenly (re)announces 72,000+ routes (becomes the poster-child for route filtering).

• 2008 - ISP in Pakistan accidentally announces IP routes for YouTube by blackholing the video service internally to their network.

• 2017 - Russian ISP leaks 36 prefixes for payments services owned by Mastercard, Visa, and major banks.

• 2018 - BGP hijack of Amazon DNS to steal crypto currency.

That’s just a partial list! Each route leak or hijack exposes a lack of route filtering by the network that peers or transits the offending network.

RPKI comes into the picture because the existing IRR system lacks any form of cryptographic signing for its data. In fact, today the IRR databases contain plenty of invalid data (both stale data and typo’ed data). There's very little control over the creation of invalid data.

Implementing RPKI is just the first step in better BGP route security because RPKI only secures the route origin; it doesn't secure the path. (Sadly the same is true for IRR data). When we want to secure the path; we are going to need something else; but that comes later.

BGP routing isn’t secure. Its main hope, RPKI, uses a certificate system that’s akin to secure web browsing (or at-least its early days). While secure web browsing has moved on and is far more secure and is somewhat the default these days, the state of BGP route validation has not moved forward. To secure BGP routing, all networks would need to be embrace RPKI (and more). Cloudflare proposes to plot a course to improve BGP routing-security globally by setting an example and implementing best practices, installing operationally excellent software and promoting its RPKI effort worldwide. RPKI is one of our focuses in 2018 and beyond!

BGP isn’t simple. BGP on the global Internet doubly-so. This fact should not deter either the casual reader or the seasoned network engineer. What is important is to place the limit around what is worth knowing about and discarding all the minor items that make up the very complex world of BGP networking. In fact, to operate a BGP enabled network connected to a telco or ISP isn’t that complicated. It turns out that in the world of BGP, security is an afterthought.

Lets begin.

I’m going to pick a hypothetical example. The configuration of a single university within a country that operates an NREN (National Research & Education Network) for all it's universities. This is not uncommon. The university in this case is connected via a single telecommunications link and (using BGP terminology) has a single upstream. The NREN provides all the connectivity to the local and global Internet for its countries universities, along with connectivity to other NRENs in other countries.

We start with some basics. BGP is about numbers. First off is a unique number called the Autonomous System Number or ASN. This number comes from a range of numbers that are managed by the RIRs (Regional Internet Registries). For example, Cloudflare has the AS number 13335 allocated for its network. ASNs were just 16-bit numbers, but are now 32-bit numbers (because the internet grew to the point of running out of the 65,536 or 2^16 initial allocation). For our university, we will use 65099 as our example ASN. This is from the reserved block of ASNs and used here for documentation reasons only.

The second number is the IP addresses allocated to the university. Most reader are familiar with IP addresses; however in the BGP world we use IP blocks called CIDRs (Classless Inter-Domain Routing). This is a range of IP addresses that are sequential and bonded on binary boundaries. Within Cloudflare, we have quite a few IP blocks allocated by the RIRs. For our example, we will assume the university has two blocks allocated. 10.0.0.0/8 and 2001:db8::/32 . Both these are private or documentation addresses and later-on you’ll see these show up again in a different manner when we talk about filtering.

This is enough for us to get this university ready to connect to the NREN. Or maybe not.

Hold on a second - there’s paperwork to fill-in. Not actual paper; but close enough. While the internet is build on the concept of permissionless innovation, there’s still good practices that still need to be adhered too.

Before you can announce a route via your BGP speaking router, you need to setup either an IRR route object or an RPKI ROA (or both).

The IRR (Internet Routing Registries) is used to record a route that will be announced on the Internet and associate it with the ASN that will announce it. In this example we will use the private or documentation ranges of 10.0.0.0/8 and 2001:db8::/32 along with ASN 65099. The simplest IRR routing record looks like this:

route:	10.0.0.0/8
origin:	AS65099

In reality, we need a lot more to make it fully-functional and we need a place to upload this routing record. You could use your RIR to host your IRR data, or you could use global services like RADB or ALTDB. You can also use your transit provider in some cases. Once you have an account setup on one of these services, you will be ready to upload these routing record (how you upload it is very specific to the IRR chosen).

route:	10.0.0.0/8
descr:	University of Blogging
descr:	Anytown, USA
origin:	AS65099
mnt-by:	MNT-UNIVERSITY
notify:	person@example.com
changed:	person@example.com 20180101
source:	RADB

That last line reflects where you store your IRR routing record.

Just like your IP network blocks, its also good to place a record for your ASN in the IRR. When you networking gets more complex, this will be solidly needed. It doesn’t hurt to add it now.

aut-num:    AS65099
as-name:    UNIVERSITY-OF-BLOGGING-AS
descr:      University of Blogging
descr:      Anytown, USA
mnt-by:     MNT-UNIVERSITY
notify:     person@example.com
changed:    person@example.com 20180101
source:     RADB

You can check for their existence using the classic command line whois command (or the RADB website).

One last item needs to be completed; but not by you.

Your ASN needs to be placed in the as-set of your upstream ISP (or service provider). The entry in there will provide the rest of the global Internet an indication that your ASN is allowed to be routed via your upstream (the NREN in this case). If all goes well, something like this will show up in the IRRs.

as-set:     AS-NREN
descr:      NREN of country XX
members:    ...
members:    AS65099
members:    ...
mnt-by:     MNT-NREN
notify:     person@example.edu
changed:    person@example.edu 20180101
source:     RADB

The members area of this as-set provides a list of ASNs that are announced by the upstream (the ASN). We have not defined the upstreams ASN yet, so lets pretend they are ASN 65001 (this ASN is still from the documentation range).

BGP (like everything in networking) needs some configuration setup. This configuration would exist on a network router at the edge of your network, or whatever device is being used to connect the local network to the upstream (the NREN). We are using a very simple router config here to show the minimum configuration needed. Your configuration language could be different.

router bgp 65099
neighbor 192.168.0.2 remote-as 65001
neighbor 192.168.0.2 prefix-list as65001-listen in
neighbor 192.168.0.2 route-map as65001-listen in

This is a very trivial example (it’s missing a complete filter configuration that’s normally required). The key point is that the router doesn’t contain any code or language regarding the IRR entries shown above. That’s because the IRR entries are out-of-band. They exist outside of the BGP protocol. In other words, it takes more than just configuring a BGP session in order to actually connect to the global Internet.

The key filtering comes into play on the upstream (the NREN in this example). It’s the job of that network to confirm everything heard from its customer.

Two global databases are being discussed today. IRR & RPKI. While IRR is clearly in use today; it’s not the primary focus herein. However, it’s the de-facto bridging option for route filtering today.

As stated above, Internet Routing Registries (IRRs) have a very loose security model. This has been known for a long time. Records exist within IRRs that are both clearly wrong and/or are clearly missing. There’s no cryptographic signing of records. There are multiple suppliers of IRR data; some better than others. IRR still has some proponents that want to clean up its operational data (including the author of this blog). Efforts like IRRD4 (by Job Snijders @ NTT) could help clean-up IRR usage. IRR is not the main focus herein.

Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a route with an originating AS number. Presently the five RIRs (AFRINIC, APNIC, ARIN, LACNIC & RIPE) provide a method for members to take an IP/ASN pair and sign a ROA (Route Origin Authorization) record. The ROA record is what we need to focus on.

Once a route is signed; it can propagate to anyone that wants to use the data to filter routing or monitor this data as ROAs are public. A ROA is a digitally signed object that makes use of RFC3852 Cryptographic Message Syntax (CMS) as a standard encapsulation format. In fact ROAs are X.509 certificates as defined in RFC5280 (Internet X.509 Public Key Infrastructure Certificate) and RFC3779 (X.509 Extensions for IP Addresses and AS Identifiers).

As the ROA is a digitally signed object, it provides a means of verifying that an IP address block holder has authorized an AS (Autonomous System) to originate routes to that one or more prefixes within the address block. The RPKI system provides an attestation method for BGP routing.

define attestation: ... the action of bearing witness ... something which bears witness, confirms or authenticates

The existence of routing information (an IP block plus the matching ASN) within a valid certificate (i.e. something that can be validated against the RIRs authoritative data cryptographically) is the missing part of the BGP security system and something that the IRR system can't provide. You really know who should be doing what with a BGP route.

Good question. As we said above, the routing databases are outside of the BGP protocol. Both IRR and RPKI use a third-party entities to hold the database information. The difference is that with RPKI the same entity that allocated or assigned a numeric resource (like an IP address or ASN) also holds the CA (Certificate Authority) used to validate the ROAs record.

In the RPKI world; CAs are called TAs, or Trust Anchors. However, if you are familiar with the web security model, then you are familiar with what a TA is.

Today the five RIRs are the TAs for RPKI. This makes sense. Only the RIRs know who is an owner of IP space (and ASs). The present day RPKI systems operate in conjunction with existing RIR login credentials. Once you can login to a portal and control your IP allocations and ASN allocations; then you can also create, edit, modify, and delete RPKI data in the forms of ROAs. This is the basis of how RPKI separates itself from the IRR. You can only sign your own resources. You can’t just randomly create data. If you lose your RIR allocation, then you lose the RPKI data.From a policy point of view, there are some interesting issues that become apparent pretty quickly. First off, an ISP with an allocation needs to keep its RIR membership up to date (i.e. pay its dues). Second, it needs to be aware that the RIR and the ISP could be legal entities based in different countries and hence international law plays a role in any dispute between the ISP and RIR or in fact any third party that gets involved in an IP address dispute. This has been a concern within the RIPE (Europe, the Middle East and parts of Central Asia) region as RIPE is based in The Netherlands. Similarly, ARIN (North America and parts of the Caribbean) is a US entity.

Presently, because of the large amount of IP address transfers occurring between some RIR regions, the RIRs changed their TA root certificates so that each RIR includes every available IP address (0.0.0.0/0 & ::/0) and every available AS number (0-4,294,967,295). IP numeric space and ASN numeric space are well defined as follows:

IPv4: 0.0.0.0 - 255.255.255.255
IPv6: 0000:0000:0000:0000:0000:0000:0000:0000 - ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
ASN:  1 - 4,294,967,295 (AS 0 is unused)

IANA (Internet Assigned Numbers Authority) holds the master list for this space and divvies it up the five RIRs as allocations or assignments. The IPv4 and IPv6 assignments can be seen here and  here. ASNs can be found here. For example, here’s an abbreviated overview into how IPv6 space is allocated to various RIRs.

Prefix         Designation    Date          WHOIS         Status
2001:0000::/23 IANA        1999-07-01 whois.iana.org     ALLOCATED
2001:0200::/23 APNIC       1999-07-01 whois.apnic.net    ALLOCATED
2001:0400::/23 ARIN        1999-07-01 whois.arin.net     ALLOCATED
...
2001:1200::/23 LACNIC      2002-11-01 whois.lacnic.net   ALLOCATED
...
2001:4200::/23 AFRINIC     2004-06-01 whois.afrinic.net  ALLOCATED
...
2002:0000::/16 6to4        2001-02-01                    ALLOCATED
...
2a00:0000::/12 RIPE NCC    2006-10-03 whois.ripe.net     ALLOCATED
2c00:0000::/12 AFRINIC     2006-10-03 whois.afrinic.net  ALLOCATED
...

As stated above; each RIR holds a root key (a TA, or Trust Anchor) that provides them the ability to create signed records below their root. Below that TA there is a certificate that covers the exact space allocated or assigned to the specific RIR. This allows the TA to be somewhat static (or stable) and the RIR to update the underlying records as-needed.

Sadly not enough people or networks. While each RIR is supporting RPKI for its members; the toolset for successfully operating a network with RPKI enabled route filtering is still very limited.

It turns out that IXP (Internet Exchange Points) have started to realize that filtering using RPKI is a valid option for their route-servers.

In addition, a handful of networks are also participating in both signing IP routes and verifying IP routes via RPKI. This isn’t quite enough to secure the global Internet yet.

Then there's the Dutch!

In early September, the NLNOG technical meeting featured a non-trivial number of RPKI-related talks. It seems that local Dutch operators and software developers are taking RPKI seriously and it’s possible that The Netherlands may contain some of the more forward-thinking RPKI networks around. Read more here.

The Internet Society (Cloudflare is a strong supporter of this organization) has pushed an initiative called MANRS (Mutually Agreed Norms for Routing Security) in order to convince the network operator community to implement routing security. It focuses on Filtering, Anti-spoofing, Coordination, and Global Validation. The Internet Society is doing a good job in educating networks on the importance of better routing security. While they do educate networks about various aspects of running a healthy BGP environment; it's not an effort that creates any of the required new technologies. MANRS simply promotes best-practices, which is a good start and something Cloudflare can collaborate on. That all said, we think it’s simply too-polite an effort as it doesn’t have enough teeth to quickly change how networks behave.

Cloudflare also wants to move the BGP community further along the RPKI path. Our operational efforts can, and should, coexist with The Internet Society’s MANRS initiative; however, we're focusing on operationally viable solutions that help move the global network community much further along.

As network operators don’t want to run an cryptographic software on the control plane of a router (or even have RPKI data anywhere near the control plane), the normal deployment is to pair routers with a server.

The server runs all the RPKI code (including the crypto processing of the TA, the certificate tree, and the ROAs). When the router sees a new route, the router send a simple message across a communications path (that includes the origin AS plus the IP route). The server, running a validator, responds with a yes/no answer that drives the filtering of that BGP route. This lightweight protocol is defined in RFC6810, then updated later to include some BGPsec support in RFC8210 (The Resource Public Key Infrastructure (RPKI) to Router Protocol). This lightweight protocol is nicknamed “RTR”.

Present implementations include https://github.com/rtrlib/rtrlib (in ‘C’) and NIST’s package https://www.nist.gov/services-resources/software/bgp-secure-routing-extension-bgp-srx-prototype which is based on quagga; hence not usable in production.

Operationally, neither are fully usable within production environments.The RIPE validator https://github.com/RIPE-NCC/rpki-validator-3 (written in Java) can produce filter sets similar to IRR tools and seems to be the most prevalent tool for the limited number of RPKI setups found in networks today. There's recently a software release from NLnet Labs research group which is Rust-based. Their RPKI validator is called Routinator 3000.

The industry still needs some more operationally-focused software!

Yes. No. Maybe. Ask your lawyers.

For many years there’s been a solid discussion about the role of the RIRs as holders of the private key of the CA at the top of their tree. Five trees. IANA was meant to run a single root above them (similar to how DNSSEC works with one key held at the DNS root - or dot); but that didn’t happen for many reasons including the fact that IANA/ICANN was essentially reporting to the US government back when this was all being setup. The RIR setup has stuck and at this point no-one expects IANA to ever hold a single root certificate, plus it’s all historic at this point and not worth rehashing here.

This is not a major operational issue; however it does have some slight consequences. While having five roots could be considered a messy setup, it actually matches the web space CA model.

Some RIR regions have special issues. ARIN (in North America and portions of the Caribbean) has a TA and ROAs; but wants full indemnification should the data be wrong or used incorrectly. In the RIPE region (Europe, ME & Russia), the members voted down full support for RPKI because they didn’t want to have a Dutch entity (RIPE NCC) hold a certificate for a non-Dutch entity and have a Dutch LEA letter shutdown a network by forcing that certificate to be invalidated. Read their respective terms of service:

• ARIN at https://www.arin.net/resources/rpki/tal.html & https://www.arin.net/resources/rpki/rpa.pdf

• RIPE at https://www.ripe.net/manage-ips-and-asns/resource-management/certification/legal/ripe-ncc-certification-repository-terms-and-conditions

The legal issues aren’t the focus of this blog entry; but it will be obvious later when implementing RPKI as-to-why the legal issues become an impediment to successful global RPKI deployment.

Everyone assumes that IRR will ultimately go away; however, that’s a long long way out. There’s efforts underway to make IRR data cleaner, and in some cases, to (finally) link the underlying RPKI & IRR data together. They are very similar data; but with different security models.

This blog post was written with RPKI as the go-forward methodology in mind and hence does not need to address all the subtle issues around IRR brokenness. It would be a whole fresh blog post to address the legacy issues within IRR. That said, it’s clear that RPKI isn’t today a complete substitute for all the IRR data (and RPSL/RPSLng data) that exists today. The good news is that there’s work within the IETF and drafts in-flight to cover that. RPKI is a good protocol to base route filtering on and Cloudflare will be rolling out full support for RPKI enabled filtering and route announcements within its global network.

If you look back at the examples above of the university and its NREN, then realize that in the RPKI world the same information is being stored; however, the validity and attestation of the data increases n-fold once RPKI becomes the universal mechanism of choice.

Cloudflare wants to see this happen and will push for RPKI to become mainstream within the BGP world. We want to squash the existence of BGP route leaks and hijacks forever!

Read the RPKI and BGP: securing our part of the Internet blog entry to follow what we are doing on the technical side for Cloudflare’s RPKI implementation.

Subscribe to the blog for daily updates on our announcements.

We said that we would head to the mountains for Cloudflare’s 123rd data center, and mountains feature prominently as we talk about Kathmandu, Nepal, home of our newest deployment and our 42nd data center in Asia!

Five and three quarter key facts to get started:

• Nepal is home to the highest mountain in the world.

• Kathmandu has more UNESCO heritage sites in its immediate area than any other capital!

• The Nepalese flag isn’t a rectangle. It’s not even close!

• Nepal has never been conquered or ruled by another country.

• Kathmandu, Nepal is where Cloudflare has placed its 123rd data center.

• Nepal’s timezone is 5 hours 45 minutes ahead of GMT.

The mountainous nation of Nepal is home to Mount Everest, the highest mountain in the world, known in Nepali as Sagarmāthā. Most of us learn that at school; however there’s plenty of other mountains located in Nepal. Here’s the ones above 8,000 meters (extracted from the full list) to get you started:

• Mount Everest at 8,848 meters

• Kanchenjunga at 8,586 meters

• Lhotse at 8,516 meters

• Makalu at 8,463 meters

• Cho Oyu at 8,201 meters

• Dhaulagiri I at 8,167 meters

• Manaslu at 8,156 meters

• Annapurna I at 8,091 meters

Photo of Annapurna taken outside Pokhara by blog author

As we said, Kathmandu and Nepal are very mountainous region! Some of these mountains are shared with neighboring countries. In-fact, the whole Himalayan range stretches much further than just Nepal as it also encompasses neighboring countries such as Bhutan, China, India, and Pakistan.

The official flag of Nepal is not a rectangle, it’s a very unique shape. No other flag comes close to having this shape. At first viewing it looks simple, just two triangles (representing the Himalayas) placed vertically against the flagpole. Nope - the triangles have a very specific dimension.

The flags symbolism goes beyond the mountains. The two white symbols represent the calming moon and fierce sun or sometimes the cool weather in the Himalayas and warm lowlands.

But back to those two triangles. Ignoring the old adage ("It was my understanding that there would be no math."), let’s grab what Wikipedia says about the shape of this flag and see if we can follow along.

First off, let’s explain irrational vs rational numbers (or ratios or fractions). A rational number is a simple P/Q number like 1/2 or 3/4 or 1/5 or even 16/9 etc etc. The numerator and denominator must both be integers. Even 100003/999983 (using prime numbers) is a rational number. If the denominator is 0 then the number isn’t rational.

An irrational number is everything else. If it can’t be written as P/Q, then it’s irrational. This means a number that doesn’t terminate or doesn’t become periodic. For example, π or Pi (3.141592653589793238462…), e or Euler's Number (2.718281828459045235360…), the square root of 2 (1.414213562373095…), etc. These are all irrational numbers. Don’t be fooled by 4/3. While it’s impossible to full write out (1.33333… continuing forever), yet it’s actually a rational number.

(Read up about Hippasus, who’s credited with discovering the existence of irrational numbers, if you want an irrational murder story!)

That’s enough math theory; now back to the Nepali flag. Each red triangle has a rational ratio. The flag is started with simple 1:1 and 3:4 ratios and that’s the easy part. We are all capable of grabbing paper or cloth and make a rectangle that’s 3 inches by 4 inches, or 1.5 meters by 2 meters or any 3:4 ratio. It’s simple math. What gets complicated is adding the blue border. For that, we need to read-up what was said by the On-Line Encyclopedia of Integer Sequences (OEIS Foundation) and Wikipedia. They both go into great depth to describe the full mathematical dimensions of the flag. Let’s just paraphrase them slightly:

However the math (and geometry) award goes to the work done in order to produce "Calculation of the aspect ratio of the national flag of Nepal". The final geometric drawing is this:

Berechnung des Seitenverhältnisses der Nationalfahne von Nepal

Yeah - that’s going a bit too far for a Cloudflare blog! Let’s just say that Nepal’s flag is unique and quite interesting.

We are especially excited to announce our Kathmandu data center while attending APRICOT conference, being held in Nepal this year. The event, supported by APNIC, the local Regional Internet address Registry (RIR) for the Asia-Pacific region, attracts leaders from Internet industry technical, operational, and policy-making communities. Cloudflare's favorite part of APRICOT is the Peering Forum track on Monday.

Nepal is just one of eight countries making up the SAARC (South Asian Association for Regional Cooperation) organization. Headquartered in Kathmandu, it comprises Afghanistan, Bangladesh, Bhutan, India, Nepal, the Maldives, Pakistan and Sri Lanka.

Cloudflare has already deployed into neighboring India, and any astute reader of these blogs will know we are always working on adding sites where and when we can. The SAARC countries are in our focus.

For Cloudflare‘s next two data centers, we head to a different continent and way-south of the equator. Throughout 2018, we’ll announce a stream of deployments across many different cities that each improve the security and performance of millions of our customers.

If you’re motivated by the idea of helping build one of the world's largest networks, come join our team!

Five key facts to know about McAllen, Texas

• McAllen, Texas is on the southern tip of the Rio Grande Valley

• The city is named after John McAllen, who provided land in 1904 to bring the St. Louis, Brownsville & Mexico Railway railway into the area

• McAllen, Texas is named the City of Palms

• The border between Mexico and the USA is less than nine miles away from the data center

• McAllen, Texas is where Cloudflare has placed its 119th data center

While McAllen is close to the Mexican border, its importance goes well beyond that simple fact. The city is halfway between Dallas, Texas (where Cloudflare has an existing datacenter) and Mexico City, the center and capital of Mexico. This means that any Cloudflare traffic delivered into Mexico is better served from McAllen. Removing 500 miles from the latency equation is a good thing. 500 miles equates to around 12 milliseconds of round-trip latency and when a connection operates (as all connections should), as a secure connection, then there can be many round trip communications before the first page starts showing up. Improving latency is key, even if we have a 0-RTT environment.

Image courtesy of gcmap service

However, it gets better! A significant amount of Mexican Cloudflare traffic is delivered to ISPs and telcos that are just south of the Mexican border, hence McAllen improves their performance even more-so. Cloudflare chose the McAllen Data Center in order to provide those ISPs and telcos a local interconnect point.

As astute readers of the Cloudflare blog know, the Cloudflare network interconnects to a large number of Internet Exchanges globally. Why should that be any different in McAllen, Texas. It’s not. As of last week, there was a brand new Internet Exchange (IX) in McAllen, Texas.

Image, with permission, from Joel Pacheco’s Facebook page

MEX-IX is that new IX and it provides a whole new way to interconnect with Mexican carriers, many of which are present in McAllen already. Cloudflare will enable peering on that IX as quickly as we can.

Cloudflare has plenty of datacenter presence in South America, however Panama is the only datacenter we have operating within Central America. That means that we still have to work on Belize, Costa Rica, El Salvador, Guatemala, Honduras, and Nicaragua.

But there’s one more place we need to deploy into in order to move the Mexican story forward and that’s Mexico City. More about that in a later blog.

Cloudflare will continue to build new datacenters, including the ones south of the border, and ones around the globe. If you enjoy the idea of helping build one of the world's largest networks, come join our team!

Five fun facts:

• Cape Town is where the Atlantic Ocean and the Indian Ocean meet deep in the Southern Hemisphere.

• The city is the start of the Garden Route, a 185 mile (300 km) glorious coastal drive brimming with native flowers and stunning vistas.

• Cape Town is the gateway into the Stellenbosch and Franschhoek wine districts that make for very popular weekend excursions.

• The imposing Table Mountain can be seen from most of Cape Town. When you take the cable car to the top of the mountain, the view from up there is even more stunning.

• Cape Town is where Cloudflare has placed its 113th data center.

Back in December 2014, Cloudflare opened our first data center in Africa and our 30th datacenter globally. That was in Johannesburg, which has since seen over 10x growth in traffic delivered to South Africa and surrounding countries.

Now, we are expanding into our second city in South Africa — Cape Town, bringing us 870 miles (1,400km) closer to millions of Internet users. Only 15% smaller than Johannesburg by population, Cape Town commands a majority of the tourism business for the country.

For Cloudflare, our newest deployment reinforces our commitment to helping build a better Internet in South Africa and Southern Africa as a whole. Expanding on our existing peering at the JINX and NAPAfrica Internet exchanges in Johannesburg, we are now also a member at NAPAfrica Cape Town. As we enable the interconnections with in-country hosts and ISPs, we will distribute and optimize traffic delivery, while providing in-country redundancy for millions of Internet properties.

Cloudflare now operates six data centers across Africa. Cape Town joins existing facilities in Cairo (Egypt), Luanda (Angola), Mombasa (Kenya), Djibouti (Djibouti), and Johannesburg (South Africa).

In the graph below, Gauteng is the province containing Johannesburg and the nation's capital Pretoria, whereas the Western Cape province includes Cape Town and the aforementioned wine districts. With the second data center now operational in South Africa, we are seeing near equivalent performance over both major cities.

Latency (ms) decreases to Cloudflare customers. Source: Cedexis

Colombo and Cape Town are the first two of four data centers going live this week. Our next two deployments, yet to be revealed, will provide additional redundancy to existing facilities in North America. Can you guess the cities they are in?

CC-BY 2.0 image by Nelo Hotsuma

Curaçao is located in the Southern Caribbean Sea (just north of Venezuela) and has a strong Dutch heritage. Along with Aruba and Bonaire, Curaçao is part of the Lesser Antilles (they are called the ABC islands).

More importantly, Willemstad - the capital of Curaçao is where the Amsterdam Internet Exchange operates AMS-IX Caribbean. Why AMS-IX? Because of that Dutch relationship!

It’s AMS-IX’s goal (along with its local partners) to promote Curaçao as an interconnection location for the Caribbean. Cloudflare is there with all its services ready for that day!

Djibouti is a country of around 850,000 people with ~60% of the population living in the nation's capital, also called Djibouti. The country is about the same size as New Hampshire and its location (on the Horn of Africa) makes it a perfect location to bring ashore undersea fiber-optic cables.

Image of landing the EASSy cable, Djibouti, Feb. 2010 courtesy of WIOCC & EASSy

In fact, the number of undersea fiber-optic cables that have become active in the last five years along the coast of Africa is impressive. The African west coast has seen a boom and, the East African coast (where Djibouti is located) has seen even more of a boom with fiber-optic cables radiating outwards towards the Middle East, India, Asia, Southern Africa and Europe from Djibouti. The list of submarine fiber-optic cables that land at Djibouti seems to be increasing year-over-year. Recently the SeaMeWe-5 cable was brought into service. SeaMeWe stands for “South East Asia–Middle East–Western Europe”. It connects Marseille France (via a landing station in Toulon, France) through Egypt, into Djibouti, then across to Sri Lanka and finaly finishing up in Singapore. Interestingly, Cloudflare has data centers in four of those five locations.

SeaMeWe-5 courtesy of Submarine Cable Map from TeleGeography

Some of the existing undersea cables connecting Djibouti to the rest of the world include:

• Eastern Africa Submarine System (EASSy) connecting South Africa to Sudan via landing points including Djibouti.

• Europe India Gateway (EIG) connnecting India and UAE via Djibouti to the UK.

• Middle East North Africa (MENA) Cable System/Gulf Bridge International connecting Oman via Djibouti to Italy.

• SEACOM/Tata TGN-Eurasia connecting South Africa, India, Djibouti, Egypt and onto Europe.

• SeaMeWe-3 connecting Asia, India, Middle East and Europe via Djibouti.

• SeaMeWe-5 connecting Asia, Bangladesh, Sri Lanka, Middle East and Europe via Djibouti.

The undersea cable business is always busy. There’s the following undersea cables being built, planned (or just dreamed about):

• Asia Africa Europe-1 (AAE-1) RFS March 2017

• Australia West Express (AWE) RFS Q2 2018

• Djibouti Africa Regional Express (DARE) RFS May 2018

• Liquid Sea RFC 2018

Djibouti is also a vital communications gateway for landlocked countries like Ethiopia and South Sudan. Landlocked countries are highly dependent on their path towards an ocean for access to the global Internet. That makes Djibouti a perfect location to provide Cloudflare services for specific areas of Eastern or Central Africa.

Right next to the landing station in Djibouti is the Djibouti Internet Exchange (DjIX) which is located inside the Djibouti Data Center (DDC). They are separate buildings; however, highly interconnected. Cloudflare is a member of the DjIX and because of that internet exchange we expect to increase our connectivity up and down the East Coast of Africa.

If you’re interested to learn more about global undersea cables, look forward to a follow up deep dive (pun intended) blog post.

Back around the start of the 20th century, the French built a one-meter gauge railway between the Ethiopian capital of Addis Ababa and the deep-water port in Djibouti. This old railway was recently disbanded and finally last year it was replaced by the modern electrified Addis Ababa-Djibouti Railway. This standard gauge international railway connects Addis Ababa and the Port of Doraleh in Djibouti. It’s used for freight, passengers and communications. The two railways are somewhat parallel and are the perfect path for fiber-optic cables linking Addis Ababa and Djibouti and hence connecting Ethiopia to the rest of the world. It’s very common for railways to be associated with fiber-optic cables. Ethiopia's single telecom company (Ethio telecom) is connected via Djibouti’s single telecom company (Djibouti Telecom).

A long time ago the oceans were perceived as a barrier to efficient communications. Whilst they still represent significant distances and hence latency (the speed of light hasn’t changed as far as we can tell); they are not the economic barrier they once were. In fact, in some cases they can be very cost-effective at moving bits globally. Undersea cables have been pushing the barriers for Dense Wave Division Multiplexing (DWDM) and optical encoding for quite some time. A single fiber-optic cable can now encode data in the multi-Terabit range (16 or 32 Tbps is not complex to implement anymore). This eclipses anything that satellites can provide!

When an undersea cable surfaces (actually they are always buried, as they traverse the beaches on the ocean edges, before reaching the landing stations where the optical signal is converted into an electrical signal, as it ultimately reaches an Internet core router or Internet Exchange Point (IXP).

If you ever have the privilege of visiting a cable landing station, you’re in for a wonderful treat. Both the optical hardware and the power subsystems are normally some of the most interesting equipment in the Internet interconnection industry. Here’s Andrew Blum giving a TED talk about undersea cables from his book Tubes; a great introduction to undersea cables.

Earth (where Cloudflare is presently located) is tilted when compared to the Sun. This is why the sun rises and sets differently during the year. Two lines can be drawn around the globe because of this tilt. The Tropic of Cancer is in the Northern Hemisphere and it's counterpart, the Tropic of Capricorn in the Southern Hemisphere.

By Peter Mercator CC BY-SA 3.0

Both Curaçao and Djibouti are located between the Equator and the Tropic of Cancer; however, they aren't the only ones. Out of 109 datacenters so far, 20 are within these two lines. Here’s the list:

    Latitude     Longitude  IATA
  23°26'13"N   -----------   ---  Tropic of Cancer
  23°24'00"N   113°18'00"E   CAN  Guangzhou, China
  23°05'00"N   113°04'15"E   FUO  Foshan, China 
  22°38'00"N   113°49'00"E   SZX  Dongguan, China
  22°37'00"N   108°10'00"E   NNG  Nanning, China
  22°18'32"N   113°54'53"E   HKG  Hong Kong, Hong Kong
  19°05'19"N    72°52'05"E   BOM  Mumbai, India 
  14°30'31"N   121°01'10"E   MNL  Manila, Philippines
  13°40'52"N   100°44'50"E   BKK  Bangkok, Thailand
  12°59'40"N    80°10'50"E   MAA  Chennai, India
  12°11'20"N    68°57'35"W   CUR  Willemstad, Curaçao
  11°32'50"N    43°09'34"E   JIB  Djibouti, Djibouti
   9°04'17"N    79°23'00"W   PTY  Panama City, Panama
   6°09'52"N    75°25'23"W   MDE  Medellín, Colombia
   2°44'44"N   101°42'36"E   KUL  Kuala Lumpur, Malaysia
   1°21'33"N   103°59'22"E   SIN  Singapore, Singapore
   0°00'00"    -----------   ---  Equator
   0°06'48"S    78°21'31"W   UIO  Quito, Ecuador
   4°02'05"S    39°35'39"E   MBA  Mombasa, Kenya
   8°51'30"S    13°13'52"E   LAD  Luanda, Angola
  12°01'19"S    77°06'52"W   LIM  Lima, Peru    
  23°26'08"S    46°28'23"W   GRU  São Paulo, Brazil
  23°26'13"S   -----------   ---  Tropic of Capricorn

Just so, you the reader, are fully informed: our most northern data center is a close tie between Helsinki, Finland at 60°19'02"N and Oslo, Norway at 60°11'38"N coming in second. Our most southern data center is also somewhat close with Melbourne, Australia at 37°40'24"S and Auckland, New Zealand at 37°00'29"S followed by Buenos Aires, Argentina at 34°49'20"S coming in third.

Cloudflare has previously deployed into locations like Fujairah, UAE (just less than 100 miles from Dubai), Mombasa Kenya, and Marseille France, which are strongly associated with undersea cables. They all have multiple landing stations that are well-connected to data centers where Cloudflare (and Internet Exchange Points) are present. Even cities like Hong Kong, Singapore, Los Angeles, New York and Miami are associated with undersea landing cables. You can explore plenty of undersea cables using the Submarine Cable Map from TeleGeography.

According to Wikipedia, Binary-coded decimal (BCD) is a method of encodings decimal numbers where each decimal digit is represented by a fixed number of bits, usually four or eight.

This is used heavily by classic LED and LCD displays on alarm clocks, microwaves and other wonderful legacy electronics. In the days of TTL circuits, every self-respecting circuit designer would know how to build a BCD-based circuit. Here’s an example of BCD in action with a TTL circuit feeding a seven-segment LCD.

Image courtesy of and with permission of Nuts & Volts Magazine

Unrelated to anything binary, Cloudflare seems to like the letters B, C & D recently. In fact datacenter number 107 was Belgrade, number 108 is Curaçao and number 109 Djibouti. It seems we like the letters BCD so much that we have four more BCD data center cities presently being deployed. There’s one B, two Cs and one D within the upcoming weeks and months. Want to guess the cities? Here's a clue. Two north of the Equator, one close to the Equator and one quite south of the Equator. The Cloudflare deployment continues all over the globe!

If you’re interest in bringing the fast Internet infrastructure of Cloudflare into diverse locations like these, then check out the jobs we have listed online.

The North American Network Operators Group (NANOG) is the loci of modern Internet innovation and the day-to-day cumulative network-operational knowledge of thousands and thousands of network engineers. NANOG itself is a non-profit membership organization; but you don’t need to be a member in order to attend the conference or join the mailing list. That said, if you can become a member, then you’re helping a good cause.

The next NANOG conference starts in a few days (February 6-8 2017) in Washington, DC. Nearly 900 network professionals are converging on the city to discuss a variety of network-related issues, both big and small; but all related to running and improving the global Internet. For this upcoming meeting, Cloudflare has three network professionals in attendance. Two from the San Francisco office and one from the London office.

With the conference starting next week, it seemed a great opportunity to introduce readers of the blog as to why a NANOG conference is so worth attending.

While it seems obvious how to do some network tasks (you unpack the spiffy new wireless router from its box, you set up its security and plug it in); alas the global Internet is somewhat more complex. Even seasoned professionals could do with a recap on how traceroute actually works, or how DNSSEC operates, or this years subtle BGP complexities, or be enlightened about Optical Networking. All this can assist you with deployments within your networks or datacenter.

If there’s one thing that keeps the Internet (a network-of-networks) operating, it’s peering. Peering is the act of bringing together two or more networks to allow traffic (bits, bytes, packets, email messages, web pages, audio and video streams) to flow efficiently and cleanly between source and destination. The Internet is nothing more than a collection of individual networks. NANOG provides one of many forums for diverse network operators to meet face-to-face and negotiate and enable those interconnections.

While NANOG isn’t the only event that draws networks together to discuss interconnection, it’s one of the early forums to support these peering discussions.

In this day-and-age we are brutally aware that security is the number-one issue when using the Internet. This is something to think about when you choose your email password, lock screen password on your laptop, tablet or smartphone. Hint: you should always have a lock screen!

At NANOG the security discussion is focused on a much deeper part of the global Internet, in the very hardware and software practices, that operate and support the underlying networks we all use on a daily basis. An Internet backbone (rarely seen) is a network that moves traffic from one side of the globe to the other (or from one side of a city to the other). At NANOG we discuss how that underlying infrastructure can operate efficiently, securely, and be continually strengthened. The growth of the Internet over the last handful of decades has pushed the envelope when it comes to hardware deployments and network-complexity. Sometimes it only takes one compromised box to ruin your day. Discussions at conferences like NANOG are vital to the sharing of knowledge and collective improvement of everyone's networks.

Above the hardware layer (from a network stack point of view) is the Domain Name System (DNS). DNS has always been a major subject of discussion within the NANOG community. It’s very much up to the operational community to make sure that when you type a website name into web browser or type in someone’s email address into your email program that there’s a highly efficient process to convert from names to numbers (numbers, or IP address, are the address book and routing method of the Internet). DNS has had its fair share of focus in the security arena and it comes down to network operators (and their system administrator colleagues) to protect DNS infrastructure.

Nearly everyone knows that bad news sells. It’s a fact. To be honest, the same is the case in the network operator community. However, within NANOG, those stories of disasters are nearly always told from a learning and improvement point of view. There’s simply no need to repeat a failure, no-one enjoys it a second time around. Notable stories have included subjects like route-leaks, BGP protocol hiccups, peering points, and plenty more.

We simply can’t rule out failures within portions of the network; hence NANOG has spent plenty of time discussing redundancy. The internet operates using routing protocols that explicitly allow for redundancy in the paths that traffic travels. Should a failure occur (a hardware failure, or a fiber cut), the theory is that the traffic will be routed around that failure. This is a recurring topic for NANOG meetings. Subsea cables (and their occasional cuts) always make for good talks.

While we learned twenty or more years ago how to type into Internet routers on the command line, those days are quickly becoming history. We simply can’t scale if network operational engineers have to type the same commands into hundreds (or thousands?) of boxes around the globe. We need automation. This is where NANOG has been a leader in this space. Cloudflare has been active in this arena and Mircea Ulinic presented our experience for Network Automation with Salt and NAPALM at the previous NANOG meeting. Mircea (and Jérôme Fleury) will be giving a follow-up in-depth tutorial on the subject at next week’s meeting.

The first NANOG conference was held June 1994 in Ann Arbor, Michigan and the conference has grown significantly since then. While it’s fun to follow the history, it’s maybe more important to realize that NANOG has covered a multitude of subjects since that start. Go scan the archives at nanog.org and/or watch some of the online videos.

Let’s not forget the advantages of spending time with other operators within a relaxed setting. After all, sometimes the big conversations happen when spending time over a beer discussing common issues. NANOG has long understood this and it’s clear that the Tuesday evening Beer ’n Gear social is set up specifically to let network geeks both grab a drink (soft drinks included) and poke around with the latest and greatest network hardware on show. The social is as much about blinking lights on shiny network boxes as it is about tracking down that network buddy.

Oh; there’s a fair number of vendor giveaways (so far there’s 15 hardware and software vendors signed up for next week’s event). After all, who doesn’t need a new t-shirt?

But there’s more to the downtime and casual hallway conversations. For myself (the author of this blog), I know that sometimes the most important work is done within the hallways during breaks in the meeting vs. standing in front of the microphone presenting at the podium. The industry has long recognized this and the NANOG organizers were one of the early pioneers in providing full-time coffee and snacks that cover the full conference agenda times. Why? Because sometimes you have to step out of the regular presentations to meet and discuss with someone from another network. NANOG knows its audience!

NANOG isn’t the only forum to discuss network operational issues, however it’s arguably the largest. It started off as a “North American” entity; however, in the same way that the Internet doesn’t have country barriers, NANOG meetings (which take place in the US, Canada and at least once in the Caribbean) have fostered an online community that has grown into a global resource. The mailing list (well worth reviewing) is a bastion of networking discussions.

In a different realm, the Internet Engineering Task Force (IETF) focuses on protocol standards. Its existence is why diverse entities can communicate. Operators participate in IETF meetings; however, it’s a meeting focused outside of the core operational mindset.

Central to the Internet’s existence is ICANN. Meeting three times a year at locations around the globe, it focused on the governance arena and in domain names and related items. Within the meetings there’s an excellent Tech Day.

In the numbers arena ARIN is an example of a regional routing registry (an RIR) that runs members meeting. An RIR deals with allocating resources like IP addresses and AS numbers. ARIN focuses on the North American area and sometimes holds its meetings alongside NANOG meetings.

ARIN’s counterparts in other parts of the world also hold meetings. Sometimes they simply focus on resource policy and sometimes they also focus on network operational issues. For example RIPE (in Europe, Central Asia and the Middle East) runs a five-day meeting that covers operational and policy issues. APNIC (Asia Pacific), AFRINIC (Africa), LACNIC (Latin America & Caribbean) all do similar variations. There isn’t one absolute method and that's a good thing. It’s worth pointing out that APNIC holds it’s members meetings once a year in conjunction with APRICOT which is the primary operations meeting in the Asia Pacific region.

While NANOG is somewhat focused on North America, there are also the regional NOGs. These regional NOGs are vital to help the education of network operators globally. Japan has JANOG, Southern Africa has SAFNOG, MENOG in the Middle East, AUSNOG & NZNOG in Australia & New Zealand, DENOG in Germany, PHNOG in the Philippines, and just to be different, the UK has UKNOF (“Forum” vs. “Group”). It would be hard to list them all; but each is a worthwhile forum for operational discussions.

Peering specific meetings also exist. Global Peering Forum, European Peering Forum, and Peering Forum de LACNOG for example. Those focus on bilateral meetings within a group of network operators or administrators and specifically focus on interconnect agreements.

In the commercial realms there’s plenty of other meetings that are attended by networks like Cloudflare. PTC and International Telecoms Week (ITW) are global telecom meetings specifically designed to host one-to-one (bilateral) meetings. They are very commercial in nature and less operational in focus.

As you would guess, you will find our network team at RIR meetings, sometimes at IETF meetings, sometimes at ICANN meetings, often at various regional NOG meetings (like SANOG in South East Asia, NoNOG in Norway, RONOG in Romania, AUSNOG/NZNOG in Australia/New Zealand and many other NOGs). We get around; however, we also run a global network and we need to interact with many many networks around the globe. These meetings provide an ideal opportunity for one-to-one discussions.

If you've heard something you like from Cloudflare at one of these operational-focused conferences, then check out our jobs listings (in various North American cities, London, Singapore, and beyond!)

Image courtesy TeleGeography Submarine Cable Map

These things happen and that’s not a good thing. The cut was reported on the venerable BBC news website. For the telecom operators in Jersey (JT Global) this wasn’t good news. However looking at the traffic from Cloudflare’s point of view; we can see that while the cable cut removed the direct path from London to Jersey, it was replaced by the backup path from Paris to Jersey. The move was 100% under the control of the BGP routing protocol. It's a relief that there's a fallback for when these unpredictable events happen.

Here's a look at one network on the island.

The red traffic is being served from our London data center (the normal location for all Jersey traffic) and the blue traffic is coming from our Paris data center. The step could well be caused by either a delayed break in one of the cables or the careful rerouting of traffic by JT Global network engineers.

CC-BY-SA-3.0 Wikipedia Jersey Map

Cloudflare is still serving traffic cleanly into Jersey. We will continue to monitor the traffic and services into the island.

The top IPv6 networks are shown here.

The chart shows the percentage of IPv6 within a specific network vs. the relative bandwidth of that network. We will talk about specific networks below.

IPv6 is faster for two reasons. The first is that many major operating systems and browsers like iOS, macOS, Chrome and Firefox impose anywhere from a 25ms to 300ms artificial delay on connections made over IPv4. The second is that some mobile networks won’t need to perform extra v4 → v6 and v6 → v4 translations to connect visitors to IPv6 enabled sites if the phone is only assigned an IPv6 address. (IPv6-only phones are becoming very common. If you have a phone on T-Mobile, Telstra, SK Telecom, Orange, or EE UK, to name a few, it’s likely you’re v6-only.)

How much faster is IPv6? Our data shows that visitors connecting over IPv6 were able to connect and load pages in 27% less time than visitors connecting over IPv4. LinkedIn found an even more dramatic effect, with up to a 40% performance boost on mobile connections over IPv6. Facebook also found a significant performance increase, around 10-15% on IPv6.

IPv6 is clearly important for driving a faster, better internet, so who is driving IPv6 adoption?

In terms of countries, Belgium leads by a mile. Over the last 30 days, 56.47% of traffic (in bytes) to Belgians on Cloudflare has been over IPv6. This is largely due to Telenet, an ISP in Belgium, doing almost 96.8% of their traffic over IPv6!

1. Belgium, 56.47%

2. Ireland, 31.75%*

3. Greece, 20.79%

4. Germany, 15.87%

5. Ecuador, 15.62%

6. Luxembourg, 15.51%

7. Portugal, 14.07%

8. Estonia, 13.75%

9. India, 11.84%

10. Peru, 10.57%

*The Irish numbers are artificially high due to several of Facebook’s IPv6 ranges being registered in Ireland. Facebook does an enormous amount of traffic over IPv6 –– 81% of Facebook’s traffic through Cloudflare is IPv6. In fact, Facebook’s crawling over IPv6 actually accounts for 6.9% of Cloudflare’s total outbound IPv6 traffic.

If you look just 6 months ago, less than 1% of India’s traffic was over IPv6. India has experienced an amazing rise in IPv6 connectivity, especially over the last month and a half.

The US is ranked 17th on the global list at 8.78%. And since we have a Canadian co-founder (and three Canadian PoPs), it hasn’t gone unnoticed at Cloudflare that Canada is just beating the US in 16th place at 9.14%.

A handful of networks are responsible for the majority of IPv6 traffic through Cloudflare. In fact, the top 10 networks by IPv6 traffic to and from Cloudflare are responsible for over half (55.4%) of Cloudflare’s IPv6 traffic. Those networks are:

The above chart shows which networks account for the highest percentage of IPv6 traffic through Cloudflare. In terms of which networks are promoting IPv6 on their own network traffic, these are the networks Cloudflare sees with the highest ratios of IPv6 to IPv4 traffic:

In terms of devices, mobile traffic is 50% more likely to use IPv6 than desktop traffic (21.4% of mobile traffic uses IPv6 traffic, whereas only 13.6% of desktop traffic is over IPv6).

In mobile, iOS sends slightly more traffic than Android over IPv6, with iOS sending about a quarter of its traffic (23.5%) and Android sending about one fifth (18.7%).

For Windows, the newer the OS, the more traffic it sends over IPv6, with XP (2001) sending just 1.1% of requests over IPv6 and Windows 10 (2015) sending 18.7%, almost a fifth of all requests over IPv6.

Here’s the full breakdown of browsers and operating systems to explain the effect of OS and browser on the percentage of IPv6 traffic. In it, you can see some interesting things, for example, iOS and ChromeOS apps tend to use IPv6 a lot more than other operating systems. Also, Chrome on mobile sends about twice as much IPv6 traffic than Chrome on desktop.

Below you can see the net effect of using a specific browser or operating system on IPv6 traffic:

Even more interesting is that while only 10.97% of our total DNS traffic is over IPv6, 22.03% of our DNS packet floods are IPv6 as well as IPv4 (we see no IPv6-only attacks, all the attacks we see on IPv6 are also seen on IPv4).

At Cloudflare, we weren’t happy with only hundreds of thousands of IPv6-enabled websites. We wanted the millions and millions of Cloudflare customers to have IPv6. Over the last few months, we have been carefully enabling IPv6 for around 100,000 sites a day, all the time monitoring how our systems operated and how our traffic behaved. We also monitored customer support and of course, social media.

People noticed. People were happy. People posted what they saw!

Some with a long range view:

there is rapid growth in number of AAAA websites from 76K (08/2016) to 109K (10/2016) (source @dan_wing dataset: https://t.co/CRzN5TweKz ) pic.twitter.com/0KNhqBMFsS

— Vaibhav Bajpai (@bajpaivaibhav) October 26, 2016

Some zoomed into the last few months:

there is rapid growth in number of AAAA websites from 76K (08/2016) to 134K (11/2016) (source @dan_wing dataset: https://t.co/CRzN5TweKz) pic.twitter.com/ygFEVSNSps

— Vaibhav Bajpai (@bajpaivaibhav) November 15, 2016

In both of the above graphs, you can really see the impact of Cloudflare enabling IPv6 for all of our customers’ sites, starting in August and wrapping up this week. Below is one more public measurement from Eric Vyncke where you can see the effect on world IPv6 measurements:

As of a late 2016, the IETF’s Internet Architecture Board has taken one more step in its path towards v6 adoption. It's taken the bold step of saying that new protocols don't need to be v4 backward compatible. This can usher-in some great new protocol designs and solutions that simply don't need to live in v4's legacy world. At Cloudflare, we've always been aggressive with our desire to provide our customers cutting edge solutions. We can't wait to see what's cooking!

Along with the increased bandwidth observed, at Cloudflare, we are seeing a solid increase in unique IPv6 addresses seen.

In June number of IPv6 addresses (/64 masked) crossed IPv4 addresses connecting to @CloudFlare for the first time. pic.twitter.com/eTPFNoueHQ

— Matthew Prince (@eastdakota) August 17, 2016

Cloudflare is committed to bringing the most up to date, fastest and most secure technologies to our customers. We are committed to IPv6, TLS 1.3 and ubiquitous DNSSEC. Just like we'd never go back to an unencrypted life, there's no going back to a v4-only world.

PS - Want to live an IPv6 life? We’re hiring.

Big thanks to Dani Grant, Igor Postelnik, Lukasz Mierzwa and Marty Strong.

Using the CloudFlare API via Python

Very early on in the company’s history we decided that everything that CloudFlare does on behalf of its customer-base should be controllable via an API. In fact, when you login to the CloudFlare control panel, you’re really just making API calls to our backend services. Over time that API has matured and improved. We are now on v4 of that API.

The current CloudFlare API is documented here and it’s used by both the CloudFlare control panel and directly by umpteen customers every minute of every day. The new API is designed with a clean naming structure and consistent data representation for data. It’s also extensible.

This blog entry introduces python-cloudflare, a Python wrapper providing full access to the CloudFlare v4 API.

Let’s get right into the thick-of-it with the simplest coding example available to show python-cloudflare in action. This example lists all your domains (zones) and also checks some basic features for each zone.

#!/usr/bin/env python
import CloudFlare
def main():
    cf = CloudFlare.CloudFlare()
    zones = cf.zones.get(params={'per_page':50})
    for zone in zones:
        zone_name = zone['name']
        zone_id = zone['id']
        settings_ipv6 = cf.zones.settings.ipv6.get(zone_id)
        ipv6_on = settings_ipv6['value']
        print zone_id, ipv6_on, zone_name
    exit(0)
if __name__ == '__main__':
    main()

The structure of the CloudFlare class matches the API documentation. The CloudFlare.zones.get() method returns information about the zones as per the list zones documentation. The same for CloudFlare.zones.settings.ipv6.get() and its documentation.

Data is passed into the methods via standard Python structures and they are returned in Python structures that match the API documentation. That means if you see an API call in the documentation, then you can translate it into the Python code in a one-to-one manner.

For example, take a look at the WAF list rules API call.

This codes into CloudFlare.zones.dns_records.post() method with the zone_id as the first argument, the package_id as the second argument and the optional parameters passed last (or if there aren’t any; then just drop the third argument for the call). Because this is a GET call there’s a .get() as part of the method name.

    r = cf.zones.firewall.waf.packages.rules.get(zone_id, package_id, params=params)

Here’s the much simpler Create DNS record API call.

This would be coded into the Python method CloudFlare.zones.dns_records.post() with the zone_id as the first argument and the required parameters passed as data. Because this is a POST call there’s a .post() as part of the method name.

    r = cf.zones.dns_records.post(zone_id, data=dns_record)

Here’s an example of that Create DNS record call in action. In this code, we add two records to an existing zone. We also show how the error is handled (in classic Python style).

    zone_name = 'example.com'
    try:
        r = cf.zones.get(params={'name': zone_name})
    except CloudFlare.CloudFlareAPIError as e:
        exit('/zones.get %s - %d %s' % (zone_name, e, e))
    except Exception as e:
        exit('/zones.get %s - %s' % (zone_name, e))
   zone_id = r['id']
   # DNS records to create
    dns_records = [
        {'name':'foo', 'type':'A',    'content':'192.168.100.100'}
        {'name':'foo', 'type':'AAAA', 'content':'2001:d8b::100:100'},
    ]
    for dns_record in dns_records:
        try:
            r = cf.zones.dns_records.post(zone_id, data=dns_record)
        except CloudFlare.CloudFlareAPIError as e:
            exit('/zones.dns_records.post %s - %d %s' % (record['name'], e, e))

There’s a whole folder of example code available on the GitHub repository.

As we stated above (and as CloudFlare has done many times before) we have placed this code up on GitHub for anyone to download and use. We welcome contributions and will review any pull requests. To install it, just clone it and follow the README instructions. For those that just want to get going right now; here’s the tl;dr install:

$ git clone https://github.com/cloudflare/python-cloudflare
$ cd python-cloudflare
$ ./setup.py build
$ sudo ./setup.py install

Not only do you get the python API calls, you also get a fully functioning CLI (command line interface) that allows quick creation of scripts that interface with CloudFlare.

From the CLI command you can call any of the CloudFlare API calls. The command responds with the returned JSON data. If you want to filter the results you possibly also want to install the highly-versatile jq command. Here’s a command to check the nameservers for a specific domain hosted on CloudFlare and then process it via jq.

$ cli4 name=example.com /zones | jq -c '.[]|{"name_servers":.name_servers}'
{
  "name_servers":[
    "alice.ns.cloudflare.com",
    "bob.ns.cloudflare.com"
  ]
}
$

The CLI command will convert on-the-fly zone names into zone identifiers. For example; if you want to check the DNSSEC status on a zone your operate on CloudFlare; then use this command.

$ cli4 /zones/:example.com/dnssec | jq '{"status":.status,"ds":.ds}'
{
  "status": "active",
  "ds": "example.com. 3600 IN DS 2371 13 2 00000000000000000000000000000 ..."
}
$

You can issue GET PUT POST PATCH or DELETE calls into the API. You can pass data into a CloudFlare API call with the CLI command. All documented via the README.md and wiki examples in GitHub.

Here’s a useful command for customers that need to flush their cache files.

$ cli4 --delete purge_everything=true /zones/:example.com/purge_cache
{
  "id":"d8afaec3dd2b7f8c1b470e594a21a01d"
}
$

See how the commands arguments match the purge_cache API documentation.

Finally, here’s an example of turning on DNSSEC via the API.

$ cli4 --patch status=active /zones/:example.com/dnssec | jq -c '{"status":.status}'
{"status":"pending"}
$

There are plenty more examples available within the GitHub repo.

Python isn’t the only language you can use to interact with CloudFlare’s API. If you’re a Go, or Node.js user, we also have client libraries you can use to interact with CloudFlare on our GitHub. Find them here Go client and here Node.js client. Want to write something in a different language? Feel free to do that. The API spec is online and ready for you to code up.

If you like what you read here today and are interested in joining one of CloudFlare’s software teams, then checkout our Join Our Team page.

In Oslo, we have now built our third data center in Scandinavia. This joins our existing facilities in Stockholm and Copenhagen. With a data center in Norway, we recognize an important country that stands above others with a staggering 95.05% of the population having Internet connectivity. This Internet penetration rate is the fourth best in the world. For reference, the Internet penetration rate in the US is 84%, the UK is 90% and Egypt, where we deployed our last data center it is only 50%

At 59.9500° N, Oslo is also the “northernmost” CloudFlare data center on our network map.

Oslo, according to the Norwegian Sagas is over 1,000 years old. CloudFlare has built itself into a facility just a handful of years old and while we respect all the wonderful history and tradition associated with Norway, we hope the locals appreciate our 21st century choice.

Norway has a very important position within the history of the Internet (well the ARPANET actually). In June 1973, the Royal Radar Establishment in Norway became one of the first international connections to the ARPANET. It seems CloudFlare is just catching up with that history.

Norwegian domains end in .NO and Norway’s country TLD operator, NORID, recently upgraded their support to allow CloudFlare users to correctly setup their domains with full DNSSEC protection. NORID added support for ECDSA Algorithm-13, which means that all .NO domains can now use DNSSEC with CloudFlare. We are grateful for their quick actions in this area and hope that returning the favor of adding Oslo to our map continues this road to better protection of DNS traffic.

In Oslo we have connected to the Norwegian Internet Exchange NIX and the Free Internet Exchange Oslo FIXO. Both will provide us with excellent coverage of the local Norwegian networks.

Norway’s excellence in Internet penetration is not the only area they excel in. Norway is also a large oil producer. In typical Scandinavian fashion the country has collected the revenue carefully and thought about its impact on current and future generations. We admire that the Norwegians have been able to save enough to provide for future generations via the "oil fund". CloudFlare, via its careful fiscal mindset, also expects to continue serving Internet customers, well into future generations.

We are excited about placing Oslo on the CloudFlare map as our 75th data center. The studious followers of CloudFlare’s expansion plans will realize we still have more Nordic countries to add to that ever-growing map. Stay tuned!

Map courtesy of gcmap.com

And yet, as we pointed out above, around 4,000 miles to the west, CloudFlare has followed the path of nearly a million Norwegians and decided to also put down roots in Minneapolis. The Twin Cities of Minneapolis and St. Paul are on the map as data center number 76.

Minnesota has around 16.5% of its population with an ancestry claimed from Norway. They migrated over the last two centuries and brought with them some of their culinary customs. (e.g. lutefisk and lefse). Lefse is a hand-rolled potato-based dough that produces a soft flatbread unique to Norway. It’s yummy. As for the gelatinous lutefisk (it’s an aged and dried cod) that’s served on top of that flatbread, it’s absolutely an acquired taste (and smell). A quick review shows there’s some CloudFlare employees that love it; some that don’t. We won’t share the exact results of that poll. You, the reader, can make your own decision.

Minnesota includes some pretty famous people with Norwegian descent. Garrison Keillor, of A Prairie Home Companion fame, comes to mind. Or is that just the personas depicted within the character's in Lake Wobegon that come off as Norwegian descendants? All we know is that there’s a statue of the Unknown Norwegian in Lake Wobegon.

Gene Amdahl is one amazing computer guy with parents of Norwegian descent. He’s one of the greatest contributors to large scale computers there ever was. First at IBM and then his own mainframe company, Amdahl Corporation. He was born in South Dakota; but we mention him here because North and South Dakota will be well-served by our Minnesota-based Minneapolis data center.

Yes, our Minneapolis data center will be able to serve more than just Minnesota. With the Midwest Internet Cooperative Exchange MICE, a local peering point, having CloudFlare as a new participant, we know we will be able to provide our customers with excellent coverage of this area.

Its exciting to announce these two data centers with their wonderful historical linking; but like we always say: stay tuned to hear about the next one!

Image by Andrew D. Ferguson

The story of IPv6 support for customers of CloudFlare is about as long as the story of CloudFlare itself. June 6th, 2011 (four years ago) was the original World IPv6 Day, and CloudFlare participated. Each year since, the global Internet community has pushed forward with additional IPv6 deployment. Now, four years later, CloudFlare is celebrating June 6th knowing that our customers are being provided with a solid IPv6 offering that requires zero configuration to enable. CloudFlare is the only global CDN that provides IPv4/IPv6 delivery of content by default and at scale.

IPv6 has been featured in our blog various times over the last four years. We have provided support for legacy logging systems to handle IPv6 addresses, provided DDoS protection on IPv6 alongside classic IPv4 address space, and provided the open source community with software to handle ECMP load balancing correctly.

Today CloudFlare is celebrating four years of World IPv6 Day with a few new graphs:

First, let's measure IPv6 addresses (broken down by /64 blocks).

While CloudFlare operates the CDN portion of web traffic delivery to end-users, we don’t control the end-user's deployment of IPv6. We do, however, see IP addresses when they are used, and we clearly see an uptick in deployed IPv6 at the end-user sites. Measuring unique IP addresses is a good indicator of IPv6 deployment.

Next we can look at how end users, using IPv6, access CloudFlare customers' websites.

With nearly 25% of our IPv6 traffic being delivered to mobile devices as of today, CloudFlare is happy to help demonstrate that mobile operators have embraced IPv6 with gusto.

The third graph looks at traffic based on destination country and is a measure of end-users (eyeballs as they are called in the industry) that are enabled for IPv6. The graph shows the top countries that CloudFlare delivers IPv6 web traffic to.

On the Y Axis, we have the percentage of traffic delivered via IPv6. On the X Axis we have each of the last eight months. Let's just have a shoutout to Latin America. In Latin America CloudFlare operates five data centers: Argentina, Brazil, Chile, Columbia, and Peru. CloudFlare would like to highlight our Peru datacenter because it has the highest percentage of traffic being delivered over IPv6 in Latin America. Others agree.

The real question is: What's next for users stuck with only IPv4? Because the RIRs have depleted their IPv4 address space pools, ISPs are simply out of options regarding the need to embrace IPv6. Basically, there are no more IPv4 addresses available.

Supporting IPv6 is even more important today than it was four years ago and CloudFlare is happy that it provides this automatic IPv6 solution for all its customers. Come try it out and don’t let other web properties languish in the non-IPv6 world by telling a friend about our automatic IPv6 support.

Starting three months ago CloudFlare moved to make IPv6 enabled the default setting for all new accounts and for existing accounts that had simply sat on the default settings mode. This resulted in an uptick in the IPv6 traffic stats from our network and was noticed globally. Our support issues have been near zero; which makes sense as the previous three years has been a fantastic proving ground for IPv6 networking.

IPv6 is here today and it works.

But it's still not widely used. It's time for that to change.

Looking at the top one million web sites tracked by Alexa we see that around 7% are accessible using IPv6. Of that 7% a full 20% are accessible using IPv6 because they are CloudFlare customers with IPv6 enabled.

We hope and expect that both those percentages will grow over time.

However some complex sites, that use the origin server IP address within their codebase, have had issues enabling IPv6.

Many sites log the IP address of the visitor when a web page is accessed. This has caused some interesting headaches when migrating to IPv6. Why? Because when software developers hardcode a 4-byte IPv4-address into a database they will have a near-impossible time dealing with at 16-byteIPv6 address. Take a look at this IETF style graphic showing the overall size of the two addressing schemes.

IPv4 (32 bits) +----+----+ |####|####| +----+----+

IPv6 (128 bits) +----+----+----+----+----+----+----+----+ |####|####|####|####|####|####|####|####| +----+----+----+----+----+----+----+----+

When a developer uses a string (or character) area to store the IP address there’s a string length issue.

In IPv4, the maximum length could be represented by the IP address 255.255.255.255 (which is 15 bytes long). For IPv6 the address could be FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF (which is 39 bytes long). Some programming languages and databases can be string length agnostic; however most aren’t.

As a stopgap measure CloudFlare has introduced a pseudo IPv4 address that is mapped from the real IPv6 address and can be used in place of the IPv6. It's not a perfect solution and software and databases are going to have to be upgraded to support IPv6, but in the meantime this pseudo IPv4 address can be used where an IPv4 format address is expected.

This workaround is now available for every CloudFlare customer and we promise to persuade as many of our users who have chosen to disable IPv6 to enable IPv6 fully. Those few remaining sites will want to be part of the new Internet.

Happy third anniversary World IPv6 Day!

I'm the new Network Strategy guy at CloudFlare and my first day was Monday, March 10th, I arrived at the offices as instructed, not knowing what to expect. After all, this was a startup! My desk; my computer; my t-shirt; my miscellaneous office items were all laid-out ready for me to sit down. You can't ask for anything more. This is the type of welcome that simply makes you know you've arrived. I had arrived.

Right away, without skipping a beat, I was being walked around the offices and introduced to the San Francisco-based people at CloudFlare. The company has nearly 80 employees. It was a little daunting at first; but well worth every minute. I was delighted to meet all the people I was going to spend copious amounts of time interacting with. By the end of the hour I had met with accounting and administration, then sales, some smart marketing folks, the heads-down operations team, a cool support team and of course the network engineering group that I will work with some day. After all, I am a network guy!

I've spent a lot of years working on IP networking. I started writing network software in the 80s. Next, I spent some years building networks and datacenters. Other careers included global internet routing and internet peering. In the past 10+ years I've been a strong proponent of IPv6--more of this later.

What about that 5,000 number I mentioned? That was the other addition to CloudFlare on Monday. I walked in the door to that carefully laid-out desk and 5,000 customers signed up to have their web properties accelerated and protected by CloudFlare. Yes, 5,000! That's not a daily sign-up record, however it's nice to know that I'm not the only person that believes in CloudFlare! I think those 5,000 web properties and myself are all in very good company. I know I'll do my part to help protect those 5,000 customers plus all the other CloudFlare customers.

I'm an IPv6 guy. I fully believe in IPv6. I've lived and breathed it for many-many years. People that know me know that IPv6 have been my passion for a long time. Luckily CloudFlare also shares that passion.

The Internet is running out of addresses (think area codes and phone number in the telephone world). IPv6 fixes that issue. The IPv6 protocol was defined by the IETF back in the late 90's and has spent the last decade minimally deployed. It is sadly overlooked by many network operators. It took the in-your-face realization that IPv4 address space (the IP addressing scheme defined at the birth of the Internet) is extremely close to being fully allocated and that no more space is available before IPv6 started being taken seriously. Events like World IPv6 Day and World IPv6 Launch organized by Internet Society (ISOC) helped to finally drive home the message that IPv6 really worked and that it must be deployed if we want the Internet to continue to grow. For the last three years, CloudFlare has been running an IPv4/IPv6 gateway for its customers. What that means it any CloudFlare customer’s web property is available over both IPv4 and IPv6. It is already deployed, available, working and being used!

But that's not the full story because one of the configuration options that is build into the CloudFlare customer control panel is a switch that says "IPv6 off" or "IPv6 on." (the default is "IPv6 off").

Many customers have flipped the switch to enable IPv6. That's good; but it's time to make the default setting "IPv6 on." In this day and age this is a very safe thing to do. Over the next few weeks CloudFlare is going to make the default for new customer be "IPv6 on." No need to flip that switch to be enabled for the whole Internet (that's IPv4 and IPv6).

In the upcoming weeks CloudFlare will enable IPv6 for existing customers in a staggered release. CloudFlare takes the delivery of each and every bit very serious and you can be assured that every person at the company is involved in making this operation is successful. Yes there will be the option to turn off IPv6; but we strongly believe that at this point there's little need for that option to be exercised. By the time we finish this process, there will be more IPv6-enabled web presence than all CDNs, combined.

Finally, let's go back and talk about that first day. I hadn't completed all the new employee paperwork yet I left the offices feeling excited and happy and slightly drained. CloudFlare is not a sleepy-quiet-backwater-feet-up-on-the-desk company. It's a high-paced, innovative, going-places, dynamic place to work and it's filled with dedicated people ready to build the next best thing. I'm honored to have joined the company. It's going to be a wild ride! If you want to join me (along with those 5,000 other Monday signups) check out the careers page. We are hiring!