We are excited to announce OPKSSH (OpenPubkey SSH) has been open-sourced under the umbrella of the OpenPubkey project. While the underlying protocol OpenPubkey became an open source Linux foundation project in 2023, OPKSSH was closed source and owned by BastionZero (now Cloudflare). Cloudflare has gifted this code to the OpenPubkey project, making it open source.

In this post, we describe what OPKSSH is, how it simplifies SSH management, and what OPKSSH being open source means for you.

A cornerstone of modern access control is single sign-on (SSO), where a user authenticates to an identity provider (IdP), and in response the IdP issues the user a token. The user can present this token to prove their identity, such as “Google says I am Alice”. SSO is the rare security technology that both increases convenience — users only need to sign in once to get access to many different systems — and increases security.

OpenID Connect (OIDC) is the main protocol used for SSO. As shown below, in OIDC the IdP, called an OpenID Provider (OP), issues the user an ID Token which contains identity claims about the user, such as “email is alice@example.com”. These claims are digitally signed by the OP, so anyone who receives the ID Token can check that it really was issued by the OP.

Unfortunately, while ID Tokens do include identity claims like name, organization, and email address, they do not include the user’s public key. This prevents them from being used to directly secure protocols like SSH or End-to-End Encrypted messaging.

Note that throughout this post we use the term OpenID Provider (OP) rather than IdP, as OP specifies the exact type of IdP we are using, i.e., an OpenID IdP. We use Google as an example OP, but OpenID Connect works with Google, Azure, Okta, etc.

^{Shows a user Alice signing in to Google using OpenID Connect and receiving an ID Token}

OpenPubkey, shown below, adds public keys to ID Tokens. This enables ID Tokens to be used like certificates, e.g. “Google says alice@example.com is using public key 0x123.” We call an ID token that contains a public key a PK Token. The beauty of OpenPubkey is that, unlike other approaches, OpenPubkey does not require any changes to existing SSO protocols and supports any OpenID Connect compliant OP.

^{Shows a user Alice signing in to Google using OpenID Connect/OpenPubkey and then producing a PK Token} While OpenPubkey enables ID Tokens to be used as certificates, OPKSSH extends this functionality so that these ID Tokens can be used as SSH keys in the SSH protocol. This adds SSO authentication to SSH without requiring changes to the SSH protocol.

OPKSSH frees users and administrators from the need to manage long-lived SSH keys, making SSH more secure and more convenient.

“In many organizations – even very security-conscious organizations – there are many times more obsolete authorized keys than they have employees. Worse, authorized keys generally grant command-line shell access, which in itself is often considered privileged. We have found that in many organizations about 10% of the authorized keys grant root or administrator access. SSH keys never expire.”  - Challenges in Managing SSH Keys – and a Call for Solutions by Tatu Ylonen (Inventor of SSH)

In SSH, users generate a long-lived SSH public key and SSH private key. To enable a user to access a server, the user or the administrator of that server configures that server to trust that user’s public key. Users must protect the file containing their SSH private key. If the user loses this file, they are locked out. If they copy their SSH private key to multiple computers or back up the key, they increase the risk that the key will be compromised. When a private key is compromised or a user no longer needs access, the user or administrator must remove that public key from any servers it currently trusts. All of these problems create headaches for users and administrators.

OPKSSH overcomes these issues:

Improved security: OPKSSH replaces long-lived SSH keys with ephemeral SSH keys that are created on-demand by OPKSSH and expire when they are no longer needed. This reduces the risk a private key is compromised, and limits the time period where an attacker can use a compromised private key. By default, these OPKSSH public keys expire every 24 hours, but the expiration policy can be set in a configuration file.

Improved usability: Creating an SSH key is as easy as signing in to an OP. This means that a user can SSH from any computer with opkssh installed, even if they haven’t copied their SSH private key to that computer.

To generate their SSH key, the user simply runs opkssh login, and they can use ssh as they typically do.

Improved visibility: OPKSSH moves SSH from authorization by public key to authorization by identity. If Alice wants to give Bob access to a server, she doesn’t need to ask for his public key, she can just add Bob’s email address bob@example.com to the OPKSSH authorized users file, and he can sign in. This makes tracking who has access much easier, since administrators can see the email addresses of the authorized users.

OPKSSH does not require any code changes to the SSH server or client. The only change needed to SSH on the SSH server is to add two lines to the SSH config file. For convenience, we provide an installation script that does this automatically, as seen in the video below.

^{Shows a user Alice SSHing into a server with her PK Token inside her SSH public key. The server then verifies her SSH public key using the OpenPubkey verifier.}

Let’s look at an example of Alice (alice@example.com) using OPKSSH to SSH into a server:

• Alice runs opkssh login. This command automatically generates an ephemeral public key and private key for Alice. Then it runs the OpenPubkey protocol by opening a browser window and having Alice log in through their SSO provider, e.g., Google. 

• If Alice SSOs successfully, OPKSSH will now have a PK Token that commits to Alice’s ephemeral public key and Alice’s identity. Essentially, this PK Token says “alice@example.com authenticated her identity and her public key is 0x123…”.

• OPKSSH then saves to Alice’s .ssh directory:

  • an SSH public key file that contains Alice’s PK Token 

  • and an SSH private key set to Alice’s ephemeral private key.

• When Alice attempts to SSH into a server, the SSH client will find the SSH public key file containing the PK Token in Alice’s .ssh directory, and it will send it to the SSH server to authenticate.

• The SSH server forwards the received SSH public key to the OpenPubkey verifier installed on the SSH server. This is because the SSH server has been configured to use the OpenPubkey verifier via the AuthorizedKeysCommand.

• The OpenPubkey verifier receives the SSH public key file and extracts the PK Token from it. It then verifies that the PK Token is unexpired, valid, signed by the OP and that the public key in the PK Token matches the public key field in the SSH public key file. Finally, it extracts the email address from the PK Token and checks if alice@example.com is allowed to SSH into this server.

Consider the problems we face in getting OpenPubkey to work with SSH without requiring any changes to the SSH protocol or software:

How do we get the PK Token from the user’s machine to the SSH server inside the SSH protocol? We use the fact that SSH public keys can be SSH certificates, and that SSH certificates have an extension field that allows arbitrary data to be included in the certificate. Thus, we package the PK Token into an SSH certificate extension so that the PK Token will be transmitted inside the SSH public key as a normal part of the SSH protocol. This enables us to send the PK Token to the SSH server as additional data in the SSH certificate, and allows OPKSSH to work without any changes to the SSH client.

How do we check that the PK Token is valid once it arrives at the SSH server? SSH servers support a configuration parameter called the AuthorizedKeysCommand that allows us to use a custom program to determine if an SSH public key is authorized or not. Thus, we change the SSH server’s config file to use the OpenPubkey verifier instead of the SSH verifier by making the following two line change to sshd_config:

AuthorizedKeysCommand /usr/local/bin/opkssh verify %u %k %t
AuthorizedKeysCommandUser root

The OpenPubkey verifier will check that the PK Token is unexpired, valid and signed by the OP. It checks the user’s email address in the PK Token to determine if the user is authorized to access the server.

How do we ensure that the public key in the PK Token is actually the public key that secures the SSH session? The OpenPubkey verifier also checks that the public key in the public key field in the SSH public key matches the user’s public key inside the PK Token. This works because the public key field in the SSH public key is the actual public key that secures the SSH session.

We have open sourced OPKSSH under the Apache 2.0 license, and released it as openpubkey/opkssh on GitHub. While the OpenPubkey project has had code for using SSH with OpenPubkey since the early days of the project, this code was intended as a prototype and was missing many important features. With OPKSSH, SSH support in OpenPubkey is no longer a prototype and is now a complete feature. Cloudflare is not endorsing OPKSSH, but simply donating code to OPKSSH.

OPKSSH provides the following improvements to OpenPubkey:

• Production ready SSH in OpenPubkey

• Automated installation

• Better configuration tools

See the OPKSSH readme for documentation on how to install and connect using OPKSSH.

There are a number of ways to get involved in OpenPubkey or OPKSSH. The project is organized through the OPKSSH GitHub. We are building an open and friendly community and welcome pull requests from anyone. If you are interested in contributing, see our contribution guide.

We run a community meeting every month which is open to everyone, and you can also find us over on the OpenSSF Slack in the #openpubkey channel.

Organizations have embraced Zero Trust initiatives that modernize secure access to web applications and networks, but often the strategies they use to manage privileged access to their infrastructure can be siloed, overcomplicated, or ineffective. When we speak to customers about their infrastructure access solution, we see common themes and pain points:

• Too risky: Long-lived credentials and shared keys get passed around and inflate the risk of compromise, excessive permissions, and lateral movement

• Too clunky: Manual credential rotations and poor visibility into infrastructure access slow down incident response and compliance efforts

Some organizations have dealt with the problem of privileged access to their infrastructure by purchasing a Privileged Access Management (PAM) solution or by building a homegrown key management tool. Traditional PAM solutions introduce audit logging and session recording features that capture user interactions with their servers and other infrastructure and/or centralized vaults that rotate keys and passwords for infrastructure every time a key is used. But this centralization can introduce performance bottlenecks, harm usability, and come with a significant price tag. Meanwhile, homegrown solutions are built from primitives provided by cloud providers or custom infrastructure-as-code solutions, and can be costly and tiresome to build out and maintain. 

We believe that organizations should apply Zero Trust principles to their most sensitive corporate resources, which naturally includes their infrastructure. That’s why we’re augmenting Cloudflare’s Zero Trust Network Access (ZTNA) service with Access to Infrastructure to support privileged access to sensitive infrastructure, and offering features that will look somewhat similar to those found in a PAM solution:

• Access: Connect remote users to infrastructure targets via Cloudflare’s global network.

• Authentication: Eliminate the management of credentials for servers, containers, clusters, and databases and replace them with SSO, MFA, and device posture. 

• Authorization: Use policy-based access control to determine who can access what target, when, and under what role. 

• Auditing: Provide command logs and session recordings to allow administrators to audit and replay their developers’ interactions with the organization’s infrastructure.

At Cloudflare, we are big believers that unified experiences produce the best security outcomes, and because of that, we are natively rebuilding each BastionZero feature into Cloudflare’s ZTNA service. Today, we will cover the recently-released feature for short-lived SSH access.

SSH (Secure Shell) is a protocol that is commonly used by developers or system administrators to secure the connections used to remotely administer and manage (usually Linux/Unix) servers. SSH access to a server often comes with elevated privileges, including the ability to delete or exfiltrate data or to install or remove applications on the server. 

Modern enterprises can have tens, hundreds, or even thousands of SSH targets. Servers accessible via SSH can be targeted in cryptojacking or proxyjacking attacks. Manually tracking, rotating, and validating SSH credentials that grant access is a chore that is often left undone, which creates risks that these long-lived credentials could be compromised. There’s nothing stopping users from copying SSH credentials and sharing them with other users or transferring them to unauthorized devices.

Although many organizations will gate access to their servers to users that are inside their corporate network, this is no longer enough to protect against modern attackers. Today, the principles of Zero Trust demand that an organization also tracks who exactly is accessing their servers with SSH, and what commands they are running on those servers once they have access. In fact, the elevated privileges that come along with SSH access mean that compliance frameworks like SOC2, ISO27001, FedRAMP and others have criteria that require monitoring who has access with SSH and what exactly they are doing with that access. 

We’ve introduced SSH with Access for Infrastructure to provide customers with granular control over privileged access to servers via SSH. The feature provides improved visibility into who accessed what service and what they did during their SSH session, while also eliminating the risk and overhead associated with managing SSH credentials. Specifically, this feature enables organizations to:

• Eliminate security risk and overhead of managing SSH keys and instead use short-lived SSH certificates issued by a Cloudflare-managed certificate authority (CA).

• Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as.

• Monitor infrastructure access with Access and SSH command logs, supporting regulatory compliance and providing visibility in case of security breach.

• Avoid changing end-user workflows. SSH with Access for Infrastructure supports whatever native SSH clients end users happen to be using. 

SSH with Access for Infrastructure is supported through one of the most common deployment models of Cloudflare One customers. Users can connect using our device client (WARP), and targets are made accessible using Cloudflare Tunnel (cloudflared or the WARP connector). This architecture allows customers with existing Cloudflare One deployments to enable this feature with little to no effort. The only additional setup will be configuring your target server to accept a Cloudflare SSH certificate.

Cloudflare One already offers multiple ways to secure organizations' SSH traffic through network controls. This new SSH with Access for Infrastructure aims to incorporate the strengths of those existing solutions together with additional controls to authorize ports, protocols, and specific users as well as a much improved deployment workflow and audit logging capabilities.

How does Access for Infrastructure eliminate your SSH credentials? This is done by replacing SSH password and SSH keys with an SSH Certificate Authority (CA) that is managed by Cloudflare. Generally speaking, a CA’s job is to issue certificates that bind an entity to an entity’s public key. Cloudflare’s SSH CA has a secret key that is used to sign certificates that authorize access to a target (server) via SSH, and a public key that is used by the target (server) to cryptographically validate these certificates. The public key for the SSH CA can be obtained by querying the Cloudflare API. And the secret key for the SSH CA is kept secure by Cloudflare and never exposed to anyone. 

To use SSH with Access for Infrastructure to grant access via SSH to a set of targets (i.e. servers), you need to instruct those servers to trust the Cloudflare SSH CA. Those servers will then grant access via SSH whenever they are presented with an SSH certificate that is validly signed by the Cloudflare SSH CA.

The same Cloudflare SSH CA is used to support SSH access for all of your developers and engineers to all your target servers. This greatly simplifies key management. You no longer need to manage long-lived SSH keys and passwords for individual end users, because access to targets with SSH is granted via certificates that are dynamically issued by the Cloudflare SSH CA. And, because the Cloudflare SSH CA issued short-lived SSH certificates that expire after 3 minutes, you also don’t have to worry about creating or managing long-lived SSH credentials that could be stolen by attackers. 

The 3-minute time window on the SSH certificate only applies to the time window during which the user has to authenticate to the target server; it does not apply to the length of the SSH session, which can be arbitrarily longer than 3 minutes. This 3-minute window was chosen because it was short enough to reduce the risk of security compromise and long enough to ensure that we don’t miss the time window of the user’s authentication to the server, especially if the user is on a slow connection.

One of the problems with traditional SSH is that once a user has an SSH key or password installed on a server, they will have access to that server forever — unless an administrator somehow remembers to remove their SSH key or password from the server in question. This leads to privilege creep, where too many people have standing access to too many servers, creating a security risk if an SSH key or password is ever stolen or leaked.

Instead, SSH with Access for Infrastructure allows you to centrally write policies in the Cloudflare dashboard specifying exactly what (set of) users has access to what (set of) servers. Users may be authenticated by SSO, MFA, device posture, location, and more, which provides better security than just authenticating them via long-lived SSH keys or passwords that could be stolen by attackers.

Moreover, the SSH certificates issued by the Cloudflare CA include a field called valid_principals which indicates the specific Linux user (e.g. root, read-only, ubuntu, ec2-user) that can be assumed by the SSH connection. As such, you can write policies that specify the (set of) Linux users that a given (set of) end users may access on a given (set of) servers, as shown in the figure below. This allows you to centrally control the privileges that a given end user has when accessing a given target server. (The one caveat here is that the server must also be pre-configured to already know about the specific Linux user (e.g. root) that is specified in the policies and presented in the SSH certificate. Cloudflare is NOT managing the Linux users on your Linux servers.)

As shown below, you could write a policy that says users in Canada, the UK, and Australia that are authenticated with MFA and face recognition can access the root and ec2-user Linux users on a given set of servers in AWS.

Cloudflare captures SSH command logs because we built an SSH proxy that intercepts the SSH connections. The SSH proxy establishes one SSH connection between itself and the end user’s SSH client, and another SSH connection between itself and the target (server). The SSH proxy can therefore inspect the SSH commands and log them. 

SSH commands are encrypted at rest using a public key that the customer uploads via the Cloudflare API. Cloudflare cannot read SSH command logs at rest, but they can be extracted (in encrypted form) from the Cloudflare API and decrypted by the customer (who holds the corresponding private key). Instructions for uploading the encryption public key are available in our developer documentation.

To understand how Cloudflare’s SSH proxy works, we first must review how a generic SSH connection is established.

First off, SSH runs over TCP, so to establish an SSH connection, we first need to complete a TCP handshake. Then, once the TCP handshake is complete, an SSH key exchange is needed to establish an ephemeral symmetric key between the client and the server that will be used to encrypt and authenticate their SSH traffic. The SSH key exchange is based on the server public key, also known as the hostkey. If you’ve ever used SSH, you’ve probably seen this message — that is the SSH server telling your SSH client to trust this hostkey for all future SSH interactions. (This is also known as TOFU or Trust On First Use.)

Finally, the client needs to authenticate itself to the server. This can be done using SSH passwords, SSH keys, or SSH certificates (as described above). SSH also has a mode called none, which means that the client does NOT need to authenticate itself to the server at all.

To understand this, we note that whenever you set up SSH with Access for Infrastructure in the Cloudflare dashboard, you first need to create the set of targets (i.e. servers) that you want to make accessible via SSH. Targets can be defined by IP address or hostname. You then create an Access for Infrastructure application that captures the TCP ports (e.g. port 22) that SSH runs over for those targets, and write policies for those SSH connections, as we already described above and in our developer documentation.

This setup allows Cloudflare to know the set of IP addresses and ports for which it must intercept SSH traffic. Thus, whenever Cloudflare sees a TCP handshake with an IP address and port that must be intercepted, it sends traffic for that TCP connection to the SSH proxy. 

The SSH proxy leverages the client’s already authenticated identity from the WARP client, and enforces the configured Access for Infrastructure policies against it. If the policies do not allow the identity to connect to the target under the requested Linux user (e.g. root), the SSH proxy will reject the connection and log an Access denied event to the Access logs. Otherwise, if policies do allow the identity to connect, the the SSH proxy will establish the following two SSH connections: 

1. SSH connection from SSH proxy to target

2. SSH connection from end user’s SSH client (via Cloudflare’s WARP client) to SSH proxy

Let’s take a look at each of these SSH connections, and the cryptographic material used to set them up. 

To establish the SSH connection from SSH proxy to the target, the SSH proxy acts as a client in the SSH key exchange between itself and the target server. The handshake uses the target server’s hostkey to establish an ephemeral symmetric key between the client and the server that will encrypt and authenticate their SSH traffic. Next, the SSH proxy must authenticate itself to the target server. This is done by presenting the server with a short-lived SSH certificate, issued by the Cloudflare SSH CA, for the specified Linux user that is requested for this connection as we already described above. Because the target server has been configured to trust the Cloudflare SSH CA, the target server will be able to successfully validate the certificate and the SSH connection will be established.

To establish the SSH connection from the end-user's SSH client to SSH proxy, the SSH proxy acts as a server in the SSH key exchange between itself and the end-user’s SSH client. 

To do this, the SSH proxy needs to inform the end user’s SSH client about the hostkey that will be used to establish this connection. But what hostkey should be used? We cannot use the same hostkey used by the target server, because that hostkey is the public key that corresponds to a private key that is known only to the target server, and not known to the SSH proxy. So, Cloudflare’s SSH proxy needs to generate its own hostkey. We don’t want the end user to randomly see warnings like the one shown below, so the SSH proxy should provide the same hostkey each time the user wants to access a given target server. But, if something does change with the hostkey of the target server, we do want the warning below to be shown. 

To achieve the desired behavior, the SSH proxy generates a hostkey and its corresponding private key by hashing together (a) a fixed secret value valid that associated with the customer account, along with (b) the hostkey that was provided by this target server (in the connection from SSH proxy to target server). This part of the design ensures that the end user only needs to see the TOFU notification the very first time it connects to the target server via WARP, because the same hostkey is used for all future connections to that target. And, if the hostkey of the target server does change as a result of a Monster-In-The-Middle attack, the warning above will be shown to the user.

Finally, during the SSH key exchange handshake from WARP client to SSH proxy, the SSH proxy informs that end user’s native SSH client that it is using none for client authentication. This means that the SSH client does NOT need to authenticate itself to the server at all. This part of the design ensures that the user need not enter any SSH passwords or store any SSH keys in its SSH configuration in order to connect to the target server via WARP. Also, this does not compromise security, because the SSH proxy has already authenticated the end user via Cloudflare’s WARP client and thus does not need to use the native SSH client authentication in the native SSH client.

Put this all together, and we have accomplished our goal of having end users authenticate to target servers without any SSH keys or passwords, using Cloudflare’s SSH CA instead. Moreover, we also preserve the desired behaviors of the TOFU notifications and warnings built into native SSH clients!

Before we wrap up, let’s review the cryptographic keys you need in order to deploy SSH with Access for Infrastructure. There are two keys:

1. Public key of the SSH CA. The private key of the SSH CA is only known to Cloudflare and not shared with anyone. The public key of the SSH CA is obtained from the Cloudflare API and must be installed on all your target servers. The same public key is used for all of your targets. This public key does not need to be kept secret.

2. Private key for SSH command log encryption. To obtain logs of SSH commands, you need to generate a public-private key pair, and upload the public key to Cloudflare. The public key will be used to encrypt your SSH commands logs at REST. You need to keep the private key secret, and you can use it to decrypt your SSH command logs. 

That’s it! No other keys, passwords, or credentials to manage!

At Cloudflare, we are committed to providing the most comprehensive solution for ZTNA, which now also includes privileged access to sensitive infrastructure like servers accessed over SSH.

Organizations can now treat SSH like any other Access application and enforce strong MFA, device context, and policy-based access prior to granting user access. This allows organizations to consolidate their infrastructure access policies into their broader SSE or SASE architecture.

You can try out Access for Infrastructure today by following these instructions in our developer documentation. Access for Infrastructure is currently available free to teams of under 50 users, and at no extra cost to existing pay-as-you-go and Contract plan customers through an Access or Zero Trust subscription. Expect to hear about a lot more features from us as we continue to natively rebuild BastionZero’s technology into Cloudflare’s Access for Infrastructure service!

SSH (Secure Shell Protocol) is an important protocol for managing remote machines. It provides a way for infrastructure teams to remotely and securely manage their fleet of machines. SSH was a step-up in security from other protocols like telnet. It ensures encrypted traffic and enforces per user controls over access to a particular machine. However, it can still introduce a significant security risk. SSH, especially root access, is destructive in the wrong hands (think rm -r *) and can be difficult to track. Logging and securing user actions via SSH typically requires custom development or restrictive software deployments. We’re excited to announce SSH command logging as part of Cloudflare Zero Trust.

Security teams put significant effort into securing SSH across their organization because of the negative impact it can have in the wrong hands. Traditional SSH security consists of strong authentication, like certificate based authentication, and tight controls on who has “root” access. Additionally, VPNs and IP allow lists are used to further protect a machine from being publicly accessible to the Internet. The security challenges that remain are visibility and potential for lateral movement.

SSH commands to a remote machine are end-to-end encrypted, which means that it is impossible to see what is being run by a particular user on a specific machine. Typically, logs can only be captured on the machine itself in log files, and a malicious user can delete log files as easily as any other command they’re running to cover their tracks. Solutions exist to send these logs to an external logging service, but this requires installing additional software on every machine that can be accessed using SSH. ProxyJump, a common way to deploy a JumpHost model, further compounds this problem because a user can easily traverse a local network of machines once they gain access to a machine in the network.

We built SSH command logging into Cloudflare Zero Trust to provide SSH visibility at a network layer instead of relying on software on individual machines. Our first customer for this capability is the Cloudflare security team. SSH command logging provides a full replay of all commands run during an SSH session, including across multiple jump-hosts or bastions. This allows for a clear picture of what happened in the event of an accident, suspected breach, or attack.

SSH command logging was built as an extension of Cloudflare’s Secure Web Gateway. The Secure Web Gateway already performs secure TLS inspection of all traffic from a user device. Now, it also supports SSH inspection by bootstrapping a proxy server upon new connections. Administrators are able to configure network policies to allow SSH access and audit the specific commands run.

This then exposes that machine for SSH access and proxies all SSH commands via Cloudflare’s global edge network. All commands are automatically captured without complex logging software installed on the machine and across all hosts. TTY traffic can also be recorded for a later full session replay.

As an added measure of security, all logs captured by Cloudflare are immediately encrypted via a public key provided by each customer, to ensure that only authorized security users can inspect SSH commands. Furthermore, we are launching this feature with an opt-in FIPS 140-2 mode to support FedRAMP compliant users.

All user authentication is performed via Cloudflare Short-Lived Certificates. Once the client certificate is loaded onto a user machine, their SSH setup is complete and secure. This eliminates the need for tedious and tricky SSH key-pair management. This means no more key management on end user devices, all the need is the Cloudflare root CA, and they can access any machine they are entitled to, including using ProxyJump.

This is only the beginning for SSH security in Cloudflare Zero Trust. In the future, we will integrate with popular SIEM tools and provide alerting for specific commands and risky behavior.

SSH command logging is in closed beta, and we will be opening this feature up to users in the coming weeks. Stay tuned for more updates when we announce general availability!

The Pi 400 has been sitting gathering dust when Cloudflare launched Auditable Terminal giving me the perfect excuse to get out the Pi 400 and hook it up.

Auditable Terminal gives you a fully featured SSH client in your browser. You authenticate using Cloudflare Access and can log into a computer from anywhere just using the browser and get a terminal. And using Cloudflare Tunnel you can securely connect a computer to Cloudflare without punching holes in a firewall. And you end up with a consistent terminal experience across devices: 256 colours, Unicode support and the same fonts everywhere.

This is ideal for my use case: set up the Pi 400 on my home network, use Cloudflare Tunnel to connect it to the Cloudflare network, use Auditable Terminal to connect to the Pi 400 via Cloudflare and the tunnel using nothing more than a browser.

Here’s what it looks like:

Before getting into the details of how I set that up, it’s worth stopping and appreciating how cool this is. I am logging into the Pi 400 via SSH but inside the browser window just by visiting a URL in a browser and authenticating using Cloudflare for Teams. And since it’s a URL that SSH session can just be a browser bookmark!

There’s another reason I wanted to do this: I like to use Cloudflare’s own products. I have a number of Cloudflare accounts that I pay for so that I see the true customer experience. I see the emails we send out and what it’s like to navigate our UI for real use. The Pi 400 gave me a chance to experience with Cloudflare Tunnel and Auditable Terminal.

Also this is really cool.

So… if you want to do this yourself, follow along as I take you through the steps I went through to hook a brand new Pi 400 up to Cloudflare and access it from anywhere.

I plugged the Pi 400 into my TV via HDMI, to the Internet via Ethernet and booted it. It comes with the OS pre-installed on an SD card so it only took seconds to be at a welcome screen:

Inevitably with a brand new machine there were a lot of updates to install so I let the machine run through that before logging in for the first time.

The Pi 400 doesn’t come with the SSH server enabled, so it’s necessary to run the raspi-config program from the command line (sudo raspi-config). The SSH server is under option “3 Interface Options”:

It’s option “P2 SSH” and when turned on will allow SSH access to the machine. By default this will be using SSH with password authentication and so it’s pretty important to change the default pi/raspberry combination (and to go much further and switch to using certificates).

But for the rest of this run through I am going to stick with password authentication in SSH. We’ll see later that Cloudflare has a clever solution to setting up SSH security with certificates.

With SSH access setup and the Pi 400 fully updated, the second step is to set up Cloudflare Tunnel.

Even with SSH setup on the Pi 400 it’s only accessible from inside my home network and I want to get access to it from anywhere. I could open a port in my home firewall, but I hate that idea.

The answer is Cloudflare Tunnel. It’s a (free!) small daemon (called cloudflared) that will connect from the Pi 400 to multiple Cloudflare data centers and maintain connections to them. It’ll be authenticated to my Cloudflare account and once setup I’ll be able to use Cloudflare for Teams to connect to it via the web.

Setting it up is a doddle. I already had a domain name setup on Cloudflare, if I hadn’t I could have set one up quickly. The Cloudflare Tunnel documentation takes you through installing it.

The only thing I didn’t know was the architecture of the Pi 400 (32 bit? 64 bit?); so I ran lscpu which tells me that it’s armv7l (which is 32-bit). How did I know that armv7l was 32-bit? I didn’t. I googled it.

You can build cloudflared from source if you wish as it’s an open source project. But I chose to go the speedy route and wget it.

With cloudflared installed on the Pi 400 it was time to authenticate it to my account. Authentication is clearly documented and I started by typing cloudflared tunnel login. On a machine with a UI and browser this would have opened the browser ready for me to log into my Cloudflare account and set up cloudflared.

Since I was doing all this while SSH’d into the Pi I had to copy and paste the URL to my browser and then choose which of my domains I would use. (I did this before we changed the name from Argo Tunnel to Cloudflare Tunnel… it’s the same thing).

Once authorized, the cloudflared command running on the Pi 400 automatically sets up the tunnel. In the dashboard I received confirmation of this:

And back on the Pi 400 there was confirmation that cloudflared had been authorized and that a cert.pem file was created.

Now that cloudflared was authorized it was time to go ahead and set up an actual tunnel that can be used for the Auditable Terminal. I chose the incredibly original name ‘terminal’ for this.

At this point the tunnel is set up but not running yet. So the Pi 400 is not yet connected to Cloudflare. I’ll come back to that. But the next step is setting up Cloudflare for Teams so that users can authenticate to the other end of the tunnel and get access.

I didn’t have Cloudflare for Teams setup so the first step was to visit https://dash.teams.cloudflare.com/ and following the setup guide. I chose the unoriginal team name jgctesting and entered it:

And it’s here that things got complicated. I ended up writing this blog post to give anyone a step-by-step guide to getting Auditable Terminal working, but also so that we can learn from my confusion. (The team has put up a complete tutorial on this here.)

If you follow the Cloudflare for Teams documentation you have three choices.

It wasn’t 100% obvious to me that what I needed to do was add an application (as opposed to a location). The answer is… add an application. The process for doing so is documented here.

In particular, I’m going to add a Self-Hosted Application (rather than SaaS) since I’m self-hosting it; it’s an SSH server on a Raspberry Pi 400 hidden behind the sofa after all.

The next step is to give the application a name and specify the URL it will be accessible via. For me that’s “Raspberry Pi 400” (yeah, I’m great with original names) and I’m making it available via pi400.jgc.org.

Cloudflare for Teams provides a lot of different authentication and identity options, but for this demo I am going to use the simplest. When I want to connect to this application (which will be Auditable Terminal ultimately) I’m going to have Cloudflare email me a one-time PIN.

Next up is adding a rule which will let Cloudflare for Teams know who should be allowed access to this application. I am allowing anyone with an @jgc.org email address to get a one-time PIN. Rules can be much more complex than that!

And finally I need to enable browser rendering (which will enable the SSH access and Auditable Terminal).

At this stage I had cloudflared setup and authorized on the Pi 400, Cloudflare for Teams setup so that anyone with an @jgc.org email could get a PIN to access Auditable Terminal on pi400.jgc.org (I’ve since reconfigured everything so going to that URL doesn’t do anything today).

But all these things need connecting together. That means: editing a config file on the Pi 400 and telling Cloudflare to route to the tunnel for pi400.jgc.org.

I started with the config file which was /home/pi/.cloudflared/config.yml. The first step was to get the tunnel ID.

And then edit the config file. Three things needed changing from the defaults: the tunnel (which is just the ID from cloudflared tunnel list), the credentials-file (which is a JSON file in the same location as the config file and with the same name as the tunnel ID) and the hostname (in my case pi400.jgc.org)

With the tunnel installed, authorized and configured I was able to finally run it and make sure that it can connect to Cloudflare. It worked! And created four connections to Cloudflare. Two to Lisbon and two to Amsterdam. Cloudflare Tunnel manages these connections to ensure that my Pi 400 is reachable across the Cloudflare network.

And finally… I just needed to tell cloudflared to route pi400.jgc.org to the tunnel (once again using that tunnel ID).

And then it was time to try it. I went back to my browser and browsed to pi400.jgc.org and was greeted with a login screen. Since I’d authorized any @jgc.org address I entered an @jgc.org email.

The next step was to enter the one-time PIN that was emailed to me.

And, since I am using user/password for SSH I was asked for the user name followed by password.

And, wow, there I was looking at a command prompt on the Pi 400.

Of course, just doing on my laptop wasn’t enough, so I went ahead and logged in the same way on my phone.

The absolute last step was to make sure that cloudflared runs automatically which was as simple as typing sudo cloudflared --config .cloudflared/config.yml service install.

This was pretty fast (especially factoring in my ignorance about all the details of configuring Cloudflare for Teams and Cloudflare Tunnel). Now I can log into that Pi 400 from anywhere in just a browser. So powerful.

But it’s still using username/password for SSH and that can be fixed with short-lived certificates. And my authentication to access is using one-time PINs; for a real application it would be better to use SSO.

Cloudflare for Teams gives organizations of any size the ability to add Zero Trust controls to resources and data while also improving performance with Cloudflare’s network. Starting today, your team can use that same platform to seamlessly connect to non-HTTP resources from inside of a browser with the same level of audit control available in web applications.

Cloudflare’s browser-based terminal renders a fully functional console that a user can launch with a single click. Users authenticate with their organization’s SSO and Cloudflare’s edge checks that they meet the team’s Zero Trust rules for the resource being accessed.

Once approved, users can run commands over SSH as if they were using their native command line without any client side configuration or agent. Cloudflare’s network will accelerate their connection, apply rules about what data transfers can take place, and record the session for administrators to audit as needed.

We built Cloudflare’s browser-based terminal based on conversations with customers who are struggling to secure and deliver applications that live outside of the browser. We heard from developers who had to deal with using, and supporting, existing workflows to connect over SSH into machines or extend legacy applications to large, remote workforces.

We’re starting with a terminal for SSH use cases, but Cloudflare’s platform will provide a browser-based interface for nearly any application that your team needs. Your security team can create Zero Trust rules to determine who can reach those resources and how — while logging every connection. You'll be able to add advanced security features to record sessions and inspect and filter data to stop incidents from starting in non-HTTP connections like SSH and soon RDP.

The platform also makes applications faster for your end users. Cloudflare’s network accelerates connections from your services to your team in any region. Existing team members can migrate to a Zero Trust model without any client-side configuration. New hires can find every resource they need, not just web applications, in a single location and launch them with a single click.

The work for an IT department to add Zero Trust controls to a web application is made easier thanks to web browsers, reverse proxies, and browser cookies. Web applications that used to live on a private network can be deployed behind a reverse proxy, like Cloudflare, and users can visit a public DNS address in any web browser while the reverse proxy checks for identity. Cloudflare Access builds on these tools to give your team the ability to add Zero Trust rules to any web application in less than 10 minutes.

Non-web applications introduce challenges. Most traditional applications that require a thick client rely on private networks. The client software expects to reach a private IP, over a specific protocol, and making that IP public is a non-starter for almost all organizations because of the risk of data loss. Even if it were public, end users would still need to run client software on their device.

Authentication to those applications also relies on legacy approaches. Developers hold long-lived SSH keys to reach machines and business users keep usernames and passwords on sticky notes for RDP sessions. These types of resources make it difficult or impossible to integrate with your SSO provider and other controls like device posture.

You can also use Cloudflare to log every authentication event and HTTP request and response without any server-side code changes. Teams can deploy a comprehensive logging layer for any web application alongside Zero Trust controls without any server-side code changes.

We’ve heard from our customers that a data control and logging gap remains in “every other application” outside of the browser. While teams invest in significant improvements in web applications, anything outside of the browser becomes a blind spot.

Web browsers make almost any SaaS application accessible to any user on any device. A user can pick up any laptop from any manufacturer and edit an Excel spreadsheet in Microsoft 365 or update a customer record in Salesforce.

That ease-of-use begins to break down for any other combination that includes a mobile device or an application that does not run in the browser. Some organizations ship dedicated hardware with a specific OS to certain team members that need it. More teams rely on expensive virtualization platforms that slow down user workflows.

Regardless of client-side approach, the connectivity between the user and the resource also suffers from the same problems of any traditional private network. Traffic is backhauled through centralized appliances and users suffer through slow performance to complete mission-critical workflows like managing production machines or enterprise resource planning.

Migrating to a Zero Trust model can become a chore when end users have to change their local configuration for non-web applications. When our own team first deleted our VPN, the most popular chat room became a thread where engineers would share SSH configuration files and answer questions about which environment variables to set to reach Kubernetes workloads.

Application discovery also becomes a problem. Organizations have to update wiki pages with inventories of IP addresses and ports for commonly used services. End users have to ask other team members for help connecting to a specific resource.

We’re excited to help your team address all four of these challenges with today’s announcement. Like web application flows, the solution takes minutes to deploy, requires no end-user configuration, and consists of just three components:

1. Your service running in an on-premise environment or public cloud

2. A secure connection from that service to Cloudflare’s edge using a lightweight daemon called cloudflared

3. A user’s browser, where Cloudflare renders the SSH session

We’ve talked to enterprises who have compliance requirements to add second factor authentication to all of their self-hosted applications, but they estimated that doing so would take months of development time. With Cloudflare Access, your team can use the second factor authentication of your identity provider as a requirement to reach applications of any type in a matter of minutes.

You can layer these types of identity-based rules with other signals, like the country where the user sits or the health of the device using integrations with Tanium, Carbon Black, Crowdstrike, and other providers. Organizations can require that users only connect from corporate devices or build login flows that support enterprise providers like Okta and Azure AD alongside public authentication options like GitHub and Google.

Cloudflare’s Zero Trust platform also helps your team get rid of outdated authentication processes like long-lived SSH security keys. The solution takes the JSON Web Token issued during the login and converts it to short-lived certificates that authorize the user’s session on a machine.

Cloudflare Zero Trust Apps platform gives your team the same level of control over files, data, and even commands that you have today in Cloudflare Gateway and applies it to any supported application type in your enterprise.

First, your team can now build rules that control who in your organization can transfer data to or away from a machine over an SSH connection or to a remote desktop over RDP. Build rules by machine, user and group identity, or country and device. Keep data on the machines or desktops in your environments and off of the roaming devices outside of your organization.

Coming soon, deploy a high visibility solution with low effort by enabling session recording for any connection type. Cloudflare Zero Trust Apps will record the screen of any session, batch the recordings in intervals, and send them to a storage location you have configured. We’ll be adding structured command logging and keyboard input to this flow as well.

Today’s launch not only improves security for any application in your organization, it also makes life easier for all of your users.

The browser-based interface of Cloudflare Zero Trust Apps can be launched from a single dashboard that is tailored to the permissions of each end user. Users login to a home page that your organization controls and Cloudflare displays each application they can reach — web, SSH, RDP, and others.

Users can click on any tile in their view to launch the interface for a given application without leaving their browser. Cloudflare’s Zero Trust login flow authorizes them to the session and they can begin doing their work without modifying SSH configuration files or editing RDP clients locally.

Mobile also just works. Cloudflare can render the session in any common browser on tablets and phones, making it possible for technicians on a job site or users away from their desk to reach any service as seamlessly as they can connect to a web application.

Like any product in Cloudflare One, this solution does not force your team to pick between security and performance. Cloudflare Zero Trust Apps makes the applications that your team needs faster.

The approach starts by building on the remote browser isolation technology that powers Cloudflare Browser. Cloudflare Zero Trust Apps renders the application in the browser as if it were a native application. Users can highlight, copy-and-paste, and use shortcuts.

Next, the solution uses Cloudflare’s network to accelerate traffic from the server to your end user. Cloudflare determines the fastest path across our global backbone and delivers the experience to your team from a data center nearby in more than 200 cities in over 100 countries.

Today’s launch begins with support for SSH. We plan to continue to add support for additional application types over the next few months, in addition to structured command logging and filtering for SSH. Does your team have a resource that has been painful to use? Let us know as we prioritize the expansion.

If your team uses Cloudflare Access for SSH flows, you can begin using Zero Trust Apps immediately with a single configuration change. To get started, follow the instructions here.

As part of Cloudflare for Teams, your organization can start using Cloudflare Zero Trust Apps at no cost for up to 50 users as part of the Cloudflare for Teams free plan. Advanced security features, like session recording, will be available on the Cloudflare for Teams Standard plan.

Almost exactly two years ago, we launched Cloudflare Spectrum for our Enterprise customers. Today, we’re thrilled to extend DDoS protection and traffic acceleration with Spectrum for SSH, RDP, and Minecraft to our Pro and Business plan customers.

When we think of Cloudflare, a lot of the time we think about protecting and improving the performance of websites. But the Internet is so much more, ranging from gaming, to managing servers, to cryptocurrencies. How do we make sure these applications are secure and performant?

With Spectrum, you can put Cloudflare in front of your SSH, RDP and Minecraft services, protecting them from DDoS attacks and improving network performance. This allows you to protect the management of your servers, not just your website. Better yet, by leveraging the Cloudflare network you also get increased reliability and increased performance: lower latency!

While access to websites from home is incredibly important, being able to remotely manage your servers can be equally critical. Losing access to your infrastructure can be disastrous: people need to know their infrastructure is safe and connectivity is good and performant. Usually, server management is done through SSH (Linux or Unix based servers) and RDP (Windows based servers). With these protocols, performance and reliability are key: you need to know you can always reliably manage your servers and that the bad people are kept out. What's more, low latency is really important. Every time you type a key in an SSH terminal or click a button in a remote desktop session, that key press or button click has to traverse the Internet to your origin before the server can process the input and send feedback. While increasing bandwidth can help, lowering latency can help even more in getting your sessions to feel like you're working on a local machine and not one half-way across the globe.

While we stay at home, many of us are also looking to play and not only work. Video games in particular have seen a huge increase in popularity. As personal interaction becomes more difficult to come by, Minecraft has become a popular social outlet. Many of us at Cloudflare are using it to stay in touch and have fun with friends and family in the current age of quarantine. And it’s not just employees at Cloudflare that feel this way, we’ve seen a big increase in Minecraft traffic flowing through our network. Traffic per week had remained steady for a while but has more than tripled since many countries have put their citizens in lockdown:

Minecraft is a particularly popular target for DDoS attacks: it's not uncommon for people to develop feuds whilst playing the game. When they do, some of the more tech-savvy players of this game opt to take matters into their own hands and launch a (D)DoS attack, rendering it unusable for the duration of the attacks. Our friends at Hypixel and Nodecraft have known this for many years, which is why they’ve chosen to protect their servers using Spectrum.

While we love recommending their services, we realize some of you prefer to run your own Minecraft server on a VPS (virtual private server like a DigitalOcean droplet) that you maintain. To help you protect your Minecraft server, we're providing Spectrum for Minecraft as well, available on Pro and Business plans. You'll be able to use the entire Cloudflare network to protect your server and increase network performance.

Configuring Spectrum is easy, just log into your dashboard and head on over to the Spectrum tab. From there you can choose a protocol and configure the IP of your server:

After that all you have to do is use the subdomain you configured to connect instead of your IP. Traffic will be proxied using Spectrum on the Cloudflare network, keeping the bad people out and your services safe.

So how much does this cost? We're happy to announce that all paid plans will get access to Spectrum for free, with a generous free data allowance. Pro plans will be able to use SSH and Minecraft, up to 5 gigabytes for free each month. Biz plans can go up to 10 gigabytes for free and also get access to RDP. After the free cap you will be billed on a per gigabyte basis.

Spectrum is complementary to Access: it offers DDoS protection and improved network performance as a 'drop-in' product, no configuration necessary on your origins. If you want more control over who has access to which services, we highly recommend taking a look at Cloudflare for Teams.

We're very excited to extend Cloudflare's services to not just HTTP traffic, allowing you to protect your core management services and Minecraft gaming servers. In the future, we'll add support for more protocols. If you have a suggestion, let us know! In the meantime, if you have a Pro or Business account, head on over to the dashboard and enable Spectrum today!