A case study of Twilio’s “non-malicious” breach

First, let me be clear that I have no insider knowledge. This is my best guess at what occurred, based on publicly available information here.

Yet, despite the warning, previous incidents and effective controls, exposed S3 buckets containing either highly sensitive data, cached entities, SDKs, static assets are still being discovered on a daily basis.

Background

Twilio is not only one of the most well respected cloud communications platform globally for their business success, but they’ve also been a leader in enabling software developers to programmatically make and receive phone calls, send and receive text messages, and perform other communication functions using its web service APIs.

As per the article posted on TheRegister, the attacker was able to get into Twilio’s Amazon Web Services S3 bucket, which was left unprotected and writable, and alter their TaskRouter v1.20 SDK to include “non-malicious” code that appeared designed primarily to track whether or not the modification worked.

Let me draw your attention to an important excerpt from this article published on TheRegister

The JavaScript SDK is Twilio’s recommended method for linking your business events, such as incoming phone calls from customers and alerts from monitoring systems, to its TaskRouter platform, which routes calls and jobs to your staff. For instance, if someone who prefers to speak Spanish hits the “call me, I need help” button on your website, your web app uses the TaskRouter SDK to create a task, in this case “call this customer now,” which is routed via a queue to a staffer who can speak Spanish and handle the call.

Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request.

The team was able to detect, resolve and implement measures to protect from this incident within 12 hours.

Twilio’s engineering, operations, and security teams have been leaders in cloud computing, and what happened to them could have happened to any one of our infrastructures. How and why?

Non conformance to a regular AWS S3 exfiltration pattern

A typical run-of-the-mill exfiltration attack pattern via AWS CLI would look like this

1. Discover exposed endpoint
2. Conduct reconnaissance to identify authorization scheme
3. Execute listing of bucket to identify content type

aws s3 ls EXPOSED_BUCKET_NAME --no-sign-request

After inspecting the content in step (3), the attacker possibly decided to deviate from standard pattern and move from step (4) to step (5)

4. Exfiltrate data

aws s3 sync s3://EXPOSED_BUCKET_NAME LOCAL_DIR --no-sign-request

5. The attacker possibly implanted a “non-malicious” JavaScript tracking rootkit to modify the behavior of a JavaScript environment and escape detection by overriding the native JavaScript objects. Thereafter the modified files were uploaded to writable s3 bucket

aws s3 sync LOCAL_DIR/[REVISED_FILE].js s3://EXPOSED_BUCKET_NAME --no-sign-request

Rather than exfiltrating data (which seemingly wasn’t sensitive), the attacker implanted a “non-malicious” tracking rootkit to further observe and understand the exponential attack surface. Why?

Why Infiltrate with “non-malicious” code?

Based on the article it is speculated that the tracker appears to be an attempt to install a payment-card skimmer. However, this could be a potential reconnoissance technique to observe and understand the scope of TaskRouter SDK. Based on information gathered, future attacks could have been potentially planned and orchestrated.

• Is the SDK installed as a part of a central service across all tenants?
• Is the SDK installed as a part of a service per tenant ?
• Is the task router platform a central platform (common across tenants) or a per tenant deployment ?
• Is the SDK included as a part of the central twilio infrastructure or partner infrastructure ?
• Is this endpoint a content distribution network ? If so, based on the tracker, can one assess spread over a period of time ?
• Given that the “non-malicious” code was injected directly into the storage unit, this tracking would have provided a sense of CI/CD release cadence i.e. if new code is deployed at a regular cadence it would have overwritten the tracking logic.
• Is AWS CloudTrail logging enabled for bucket level actions and object level actions or both?

Evading Detection

Amazon S3 is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon S3. If you don’t configure a trail, you can still view the most recent events in the CloudTrail console in Event history. Using the information collected by CloudTrail, you can determine the request that was made to Amazon S3, the IP address from which the request was made, who made the request, when it was made, and additional details.

This deviant access pattern would have (or not have) raised an alert depending on how event tracking was possibly configured (bucket or object level actions).

It seems like the code injection process was fairly straightforward (the modification added code to the end of the TaskRouter.js v1.20 SDK). However an adversary could have employed a plethora of techniques to evade detection. The browser’s or server side security model does not erect trust boundaries within a single JavaScript environment. Therefore, the attacker can employ a number of techniques to disrupt the integrity or confidentiality of trusted JavaScript running in an untrusted JavaScript environment. Not all of these techniques work in all browsers, but majority of the still relevant techniques work in many of the major browsers.

• Shadowing: An attacker can replace native JavaScript objects in the global scope by declaring global variables with the same name as the native objects. Browsers let web pages override native objects to help with compatibility. For example, not too long ago a pre-release version of Firefox introduced an immutable JSON object into the global scope, but this change broke a number of prominent web sites that implemented their own JSON object. To avoid breaking these sites, Firefox changed the JSON object to be mutable (e.g., over-writable by attackers).
• Prototype poisoning: An attacker can also alter the behavior of native objects, such as strings, functions, and regular expressions, by manipulating the prototype of these objects.
• Reflection: Any inspection logic in SDK could hope to detect prototype poisoning using JavaScript’s reflection APIs. For example, most JavaScript implementations provide a native method called __lookupGetter__ that can be used to determine whether a property has been replaced by a getter. Similarly, the inspection logic could hope to use a function’s toString property to determine if a function has been replaced by the attacker. Unfortunately, the attacker can defeat even this carefully crafted defense by emulating the reflection API itself. In particular, the attacker can replace __lookupGetter__ by altering Object.prototype. Similarly, the attacker can alter the toString method of every function by poisoning the Function.prototype object.
• String Obfuscation: Injecting code provides the attacker with a myriad of techniques to hide the payload, making detection much more challenging for security products. Some of the obfuscation techniques are
  - Encoding : unescape(“%48%6…..”)
  - Unicode encoding : \u0048\u0065\u006C\…
  - Character substitution : “P976e246..”.replace(/[0­9]/g,””)
  - String splitting and character encoding
  - XOR
  - Base64
• CSP Header based protection: (As recommended by my colleague Prabhu Subramanian) Twilio can recommend its clients to use CSP (Content Security Policy) headers with hash to ensure that it’s supply chain (OSS libraries, SDKs) are not infiltrated. The recommended hash could be computed in the CI/CD pipeline, published with the distribution and stored as a S3 Object metadata as well. Random verification of hash can assist to detect such rootkits.

Activate defenses

1. Click on the permissions for your buckets to make sure you’re not allowing List, Upload/Delete, Write objects and read or write permissions for “Everyone”, “Any AWS User”, “Any Authenticated User”.
2. Check your bucket policy to make sure you’re not using “Principal”: “*” in your policy’s Allow statements.
3. Enable Cloud Trail Logging for S3 bucket-level actions and S3 object-level actions and configure to detect anomalies (past and present egress communication).
4. Employ code verification techniques (not just a part of CI/CD workflow) in or order to detect for injection patterns described in the evading detection section above.
5. A dated yet still relevant post by Scott Piper that advises sprinkling honey tokens throughout your cloud environments in order to trigger detection when those honey tokens are used.