For many organizations, it is becoming increasingly challenging and inefficient to adapt to risks across their growing attack surface. In particular, security teams struggle with multiple siloed tools that fail to share risk data effectively with each other, leading to excessive manual effort to extract signals from the noise. To address this complexity, Cloudflare launched risk posture management capabilities earlier this year to make it easier for organizations to accomplish three key jobs on one platform:

1. Evaluating risk posed by people by using first-party user entity and behavior analytics (UEBA) models

2. Exchanging risk telemetry with best-in-class security tools, and

3. Enforcing risk controls based on those dynamic first- and third-party risk scores.

Today’s announcement builds on these capabilities (particularly job #2) and our partnership with Okta by enabling organizations to share Cloudflare’s real-time user risk scores with Okta, which can then automatically enforce policies based on that user’s risk. In this way, organizations can adapt to evolving risks in less time with less manual effort.

Introduced earlier this year, Cloudflare’s user risk scoring analyzes real-time telemetry of user activities and behaviors and assigns a risk score of high, medium, or low. For example, if Cloudflare detects risky or suspicious activity from a user — such as impossible travel, where a user logs in from multiple geographically dispersed locations within a short time frame, data loss prevention (DLP) detections, or endpoint detections suggesting that the device is infected — the user’s risk score will increase. The activity leading to that scoring is logged for analysis.

Cloudflare includes predefined risk behaviors to help you get started. Administrators can create policies based on specific risk behaviors and adjust the risk level for each behavior based on their company’s tolerance.

Customers that opt in to this new integration will be able to share continually updated Cloudflare user risk scores with Identity Threat Protection with Okta AI. If a user is deemed too risky, Okta will automatically take action to mitigate the risk, such as enforcing multi-factor authentication (MFA) verification or universally logging the user out from all applications. 

For example, a user has a low risk score from Cloudflare that was shared with Okta, but after exhibiting “impossible travel” behavior, the user’s risk level is raised to high. Cloudflare sends the updated score to Okta, which triggers a Universal Logout and an MFA challenge if the user attempts to log in again. Access to sensitive systems may be revoked completely until the user is verified. 

^{Figure 1. Diagram showing risky behavior by a user, resulting in sign-out.}

We begin by detecting risky behavior from a user (such as an “impossible travel” event between two geographic locations). Instances of risky behavior are called Risk Events. We perform two actions when we observe a Risk Event: logging the event and evaluating whether further action is required. For customers that have enabled Risk Score Sharing with Okta, any change in Risk Score is transmitted to Okta’s Identity Threat Protection (ITP).

Upon receiving a new event, Okta evaluates the change in user risk against the organization's policies. These policies may include actions such as re-authenticating the user if they become high risk.

When we design new features, we aim for them to be extensible across the industry. For this reason, we chose the OpenID Shared Signals Framework Specification (SSF) to be the foundation of our transmission format. By doing this, we are able to leverage current and future providers that support the standard. The core functionality of SSF revolves around sharing Security Event Tokens (SETs), a specialized version of a JSON Web Token (JWT). Providers can produce and consume Security Event Tokens, forming a “network” of shared user risk information between providers.

^{Figure 2. Diagram showing a Security Event Token being transmitted from Cloudflare to Okta.}

The diagram above (Figure 2) details the process of sharing risk. When sharing Risk Score changes with Okta, we bundle metadata about the risk event and user into the body of a Security Event Token. Following this, the JWT/SET is signed using our private key. This is an important step, as the signature is used to verify the sender's identity (cryptographic authenticity) and that the payload body has not been tampered with (cryptographic integrity). In plain terms, this signature is used by Okta to verify that the event is unaltered and was sent by Cloudflare.

Once Okta has verified the authenticity and integrity of the SET token, they may use the risk metadata within the body to execute Identity Threat Protection policies defined by the customer. These policies could include actions such as “if a high risk score is received from Cloudflare, sign out the offending user”.

Learn more about the Shared Signals Framework and CAEP in Okta’s announcement blog post.

Cloudflare customers can easily enable risk score sharing from the Cloudflare One SSO setup page. This is available to customers whether you’ve already integrated with Okta or are setting up the integration for the first time. You will also be able to confirm that the feature was enabled in your audit logs.

If you’ve already integrated Okta within your Cloudflare One dashboard:

1. As an admin, navigate to Settings > Authentication and select the Okta login method.

2. Select “send risk score to Okta.”

If you haven’t yet integrated Okta within your Cloudflare One dashboard:

1. As an admin, navigate to Settings > Authentication and select a new login method.

2. Follow the instructions to add Okta as an SSO.

3. Select “send risk score to Okta.”

Now, whenever a user’s risk score changes within the organization, information is sent to Okta automatically and an audit log is documented.

In conclusion, the ability to incorporate rich context is essential for making accurate and informed access decisions. With vast amounts of data — including user logins, logouts, websites visited, and emails sent — human analysts would struggle to keep pace with modern security challenges. Cloudflare provides context in the form of a risk score, enabling Okta’s risk engine to make more informed policy decisions about users. This sharing of information powers the continuous evaluation required to enforce Zero Trust policies within your organization, ultimately strengthening your organization’s security posture.

Not yet a Cloudflare One customer? Reach out for a consultation or contact your account manager.

On Wednesday, October 18, 2023, we discovered attacks on our system that we were able to trace back to Okta – threat actors were able to leverage an authentication token compromised at Okta to pivot into Cloudflare’s Okta instance. While this was a troubling security incident, our Security Incident Response Team’s (SIRT) real-time detection and prompt response enabled containment and minimized the impact to Cloudflare systems and data. We have verified that no Cloudflare customer information or systems were impacted by this event because of our rapid response. Okta has now released a public statement about this incident.

This is the second time Cloudflare has been impacted by a breach of Okta’s systems. In March 2022, we blogged about our investigation on how a breach of Okta affected Cloudflare. In that incident, we concluded that there was no access from the threat actor to any of our systems or data – Cloudflare’s use of hard keys for multi-factor authentication stopped this attack.  

The key to mitigating this week’s incident was our team’s early detection and immediate response. In fact, we contacted Okta about the breach of their systems before they had notified us. The attacker used an open session from Okta, with Administrative privileges, and accessed our Okta instance. We were able to use our Cloudflare Zero Trust Access, Gateway, and Data Loss Prevention and our Cloudforce One threat research to validate the scope of the incident and contain it before the attacker could gain access to customer data, customer systems, or our production network. With this confidence, we were able to quickly mitigate the incident before the threat-actors were able to establish persistence.

According to Okta’s statement, the threat-actor accessed Okta’s customer support system and viewed files uploaded by certain Okta customers as part of recent support cases. It appears that in our case, the threat-actor was able to hijack a session token from a support ticket which was created by a Cloudflare employee. Using the token extracted from Okta, the threat-actor accessed Cloudflare systems on October 18. In this sophisticated attack, we observed that threat-actors compromised two separate Cloudflare employee accounts within the Okta platform. We detected this activity internally more than 24 hours before we were notified of the breach by Okta. Upon detection, our SIRT was able to engage quickly to identify the complete scope of compromise and contain the security incident. Cloudflare’s Zero Trust architecture protects our production environment, which helped prevent any impact to our customers.

We urge Okta to consider implementing the following best practices, including:

• Take any report of compromise seriously and act immediately to limit damage; in this case Okta was first notified on October 2, 2023 by BeyondTrust but the attacker still had access to their support systems at least until October 18, 2023.

• Provide timely, responsible disclosures to your customers when you identify that a breach of your systems has affected them.

• Require hardware keys to protect all systems, including third-party support providers.

For a critical security service provider like Okta, we believe following these best practices is table stakes.

If you are an Okta customer, we recommend that you reach out to them for further information regarding potential impact to your organization. We also advise the following actions:

• Enable Hardware MFA for all user accounts. Passwords alone do not offer the necessary level of protection against attacks. We strongly recommend the usage of hardware keys, as other methods of MFA can be vulnerable to phishing attacks.

• Investigate and respond to:

  • All unexpected password and MFA changes for your Okta instances.

  • Suspicious support-initiated events.

  • Ensure all password resets are valid and force a password reset for any under suspicion.

  • Any suspicious MFA-related events, ensuring only valid MFA keys are present in the user's account configuration.

• Monitor for:

  • New Okta users created.

  • Reactivation of Okta users.

  • All sessions have proper authentication associated with it.

  • All Okta account and permission changes.

  • MFA policy overrides, MFA changes, and MFA removal.

  • Delegation of sensitive applications.

  • Supply chain providers accessing your tenants.

• Review session expiration policies to limit session hijack attacks.

• Utilize tools to validate devices connected to your critical systems, such as Cloudflare Access Device Posture Check.

• Practice defense in depth for your detection and monitoring strategies.

Cloudflare’s Security and IT teams continue to remain vigilant after this compromise. If further information is disclosed by Okta or discovered through additional log analysis, we will publish an update to this post.

Cloudflare's Security Incident Response Team is hiring.

Our understanding is that during January 2022, hackers outside Okta had access to an Okta support employee’s account and were able to take actions as if they were that employee. In a screenshot shared on social media, a Cloudflare employee’s email address was visible, along with a popup indicating the hacker was posing as an Okta employee and could have initiated a password reset.

We learnt of this incident via Cloudflare’s internal SIRT. SIRT is our Security Incident Response Team and any employee at Cloudflare can alert SIRT to a potential problem. At exactly 03:30 UTC, a Cloudflare employee emailed SIRT with a link to a tweet that had been sent at 03:22 UTC. The tweet indicated that Okta had potentially been breached. Multiple other Cloudflare employees contacted SIRT over the following two hours.

The following timeline outlines the major steps we took following that initial 03:30 UTC email to SIRT.

03:30 - SIRT receives the first warning of the existence of the tweets.

03:38 - SIRT sees that the tweets contain information about Cloudflare (logo, user information).

03:41 - SIRT creates an incident room to start the investigation and starts gathering the necessary people.

03:50 - SIRT concludes that there were no relevant audit log events (such as password changes) for the user that appears in the screenshot mentioned above.

04:13 - Reached out to Okta directly asking for detailed information to help our investigation.

04:23 - All Okta logs that we ingest into our Security Information and Event Management (SIEM) system are reviewed for potential suspicious activities, including password resets over the past three months.

05:03 - SIRT suspends accounts of users that could have been affected.

We temporarily suspended access for the Cloudflare employee whose email address appeared in the hacker’s screenshots.

05:06 - SIRT starts an investigation of access logs (IPs, locations, multifactor methods) for the affected users.

05:38 - First tweet from Matthew Prince acknowledging the issue.

Because it appeared that an Okta support employee with access to do things like force a password reset on an Okta customer account had been compromised, we decided to look at every employee who had reset their password or modified their Multi-Factor Authentication (MFA) in any way since December 1 up until today. Since Dec. 1, 2021, 144 Cloudflare employees had reset their password or modified their MFA. We forced a password reset for them all and let them know of the change.

05:44 - A list of all users that changed their password in the last three months is finalized. All accounts were required to go through a password reset.

06:40 - Tweet from Matthew Prince about the password reset.

07:57 - We received confirmation from Okta that there were no relevant events that may indicate malicious activity in their support console for Cloudflare instances.

Cloudflare uses Okta internally as our identity provider, integrated with Cloudflare Access to guarantee that our users can safely access internal resources. In previous blog posts, we described how we use Access to protect internal resources and how we integrated hardware tokens to make our user authentication process more resilient and prevent account takeovers.

In the case of the Okta compromise, it would not suffice to just change a user's password. The attacker would also need to change the hardware (FIDO) token configured for the same user. As a result it would be easy to spot compromised accounts based on the associated hardware keys.

Even though logs are available in the Okta console, we also store them in our own systems. This adds an extra layer of security as we are able to store logs longer than what is available in the Okta console. That also ensures that a compromise in the Okta platform cannot alter evidence we have already collected and stored.

Okta is not used for customer authentication on our systems, and we do not store any customer data in Okta. It is only used for managing the accounts of our employees.

The main actions we took during this incident were:

1. Reach out to Okta to gather more information on what is known about the attack.

2. Suspend the one Cloudflare account visible in the screenshots.

3. Search the Okta System logs for any signs of compromise (password changes, hardware token changes, etc.). Cloudflare reads the system Okta logs every five minutes and stores these in our SIEM so that if we were to experience an incident such as this one, we can look back further than the 90 days provided in the Okta dashboard. Some event types within Okta that we searched for are: user.account.reset_password, user.mfa.factor.update, system.mfa.factor.deactivate, user.mfa.attempt_bypass, and user.session.impersonation.initiate. It’s unclear from communications we’ve received from Okta so far who we would expect the System Log Actor to be from the compromise of an Okta support employee.

4. Search Google Workplace email logs to view password resets. We confirmed password resets matched the Okta System logs using a separate source from Okta considering they were breached, and we were not sure how reliable their logging would be.

5. Compile a list of Cloudflare employee accounts that changed their passwords in the last three months and require a new password reset for all of them. As part of their account recovery, each user will join a video call with the Cloudflare IT team to verify their identity prior to having their account re-enabled.

If you are also an Okta customer, you should reach out to them for further information. We advise the following actions:

1. Enable MFA for all user accounts. Passwords alone do not offer the necessary level of protection against attacks. We strongly recommend the usage of hard keys, as other methods of MFA can be vulnerable to phishing attacks.

2. Investigate and respond:a. Check all password and MFA changes for your Okta instances.b. Pay special attention to support initiated events.c. Make sure all password resets are valid or just assume they are all under suspicion and force a new password reset.d. If you find any suspicious MFA-related events, make sure only valid MFA keys are present in the user's account configuration.

3. Make sure you have other security layers to provide extra security in case one of them fails.

Cloudflare’s Security and IT teams are continuing to work on this compromise. If further information comes to light that indicates compromise beyond the January timeline we will publish further posts detailing our findings and actions.

We are also in contact with Okta with a number of requests for additional logs and information. If anything comes to light that alters our assessment of the situation we will update the blog or write further posts.