A case study of Twilio’s “non-malicious” breach
First, let me be clear that I have no insider knowledge. This is my best guess at what occurred, based on publicly available information here.
Yet, despite the warning, previous incidents and effective controls, exposed S3 buckets containing either highly sensitive data, cached entities, SDKs, static assets are still being discovered on a daily basis.
Twilio is not only one of the most well respected cloud communications platform globally for their business success, but they’ve also been a leader in enabling software developers to programmatically make and receive phone calls, send and receive text messages, and perform other communication functions using its web service APIs.
As per the article posted on TheRegister, the attacker was able to get into Twilio’s Amazon Web Services S3 bucket, which was left unprotected and writable, and alter their TaskRouter v1.20 SDK to include “non-malicious” code that appeared designed primarily to track whether or not the modification worked.
Let me draw your attention to an important excerpt from this article published on TheRegister
Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to
hxxps://gold.platinumus.top/track/awswrite?q=dmnand followed the URL returned in the HTML by that request.
The team was able to detect, resolve and implement measures to protect from this incident within 12 hours.
Twilio’s engineering, operations, and security teams have been leaders in cloud computing, and what happened to them could have happened to any one of our infrastructures. How and why?
Non conformance to a regular AWS S3 exfiltration pattern
A typical run-of-the-mill exfiltration attack pattern via AWS CLI would look like this
- Discover exposed endpoint
- Conduct reconnaissance to identify authorization scheme
- Execute listing of bucket to identify content type
aws s3 ls EXPOSED_BUCKET_NAME --no-sign-request
After inspecting the content in step (3), the attacker possibly decided to deviate from standard pattern and move from step (4) to step (5)
4. Exfiltrate data
aws s3 sync s3://EXPOSED_BUCKET_NAME LOCAL_DIR --no-sign-request
aws s3 sync LOCAL_DIR/[REVISED_FILE].js s3://EXPOSED_BUCKET_NAME --no-sign-request
Rather than exfiltrating data (which seemingly wasn’t sensitive), the attacker implanted a “non-malicious” tracking rootkit to further observe and understand the exponential attack surface. Why?
Why Infiltrate with “non-malicious” code?
Based on the article it is speculated that the tracker appears to be an attempt to install a payment-card skimmer. However, this could be a potential reconnoissance technique to observe and understand the scope of TaskRouter SDK. Based on information gathered, future attacks could have been potentially planned and orchestrated.
- Is the SDK installed as a part of a central service across all tenants?
- Is the SDK installed as a part of a service per tenant ?
- Is the task router platform a central platform (common across tenants) or a per tenant deployment ?
- Is the SDK included as a part of the central twilio infrastructure or partner infrastructure ?
- Is this endpoint a content distribution network ? If so, based on the tracker, can one assess spread over a period of time ?
- Given that the “non-malicious” code was injected directly into the storage unit, this tracking would have provided a sense of CI/CD release cadence i.e. if new code is deployed at a regular cadence it would have overwritten the tracking logic.
- Is AWS CloudTrail logging enabled for bucket level actions and object level actions or both?
Amazon S3 is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon S3. If you don’t configure a trail, you can still view the most recent events in the CloudTrail console in Event history. Using the information collected by CloudTrail, you can determine the request that was made to Amazon S3, the IP address from which the request was made, who made the request, when it was made, and additional details.
This deviant access pattern would have (or not have) raised an alert depending on how event tracking was possibly configured (bucket or object level actions).
- Prototype poisoning: An attacker can also alter the behavior of native objects, such as strings, functions, and regular expressions, by manipulating the prototype of these objects.
__lookupGetter__that can be used to determine whether a property has been replaced by a getter. Similarly, the inspection logic could hope to use a function’s toString property to determine if a function has been replaced by the attacker. Unfortunately, the attacker can defeat even this carefully crafted defense by emulating the reflection API itself. In particular, the attacker can replace
Object.prototype. Similarly, the attacker can alter the toString method of every function by poisoning the
- String Obfuscation: Injecting code provides the attacker with a myriad of techniques to hide the payload, making detection much more challenging for security products. Some of the obfuscation techniques are
- Encoding :
- Unicode encoding :
- Character substitution :
- String splitting and character encoding
- CSP Header based protection: (As recommended by my colleague Prabhu Subramanian) Twilio can recommend its clients to use CSP (Content Security Policy) headers with hash to ensure that it’s supply chain (OSS libraries, SDKs) are not infiltrated. The recommended hash could be computed in the CI/CD pipeline, published with the distribution and stored as a S3 Object metadata as well. Random verification of hash can assist to detect such rootkits.
- Click on the permissions for your buckets to make sure you’re not allowing List, Upload/Delete, Write objects and read or write permissions for “Everyone”, “Any AWS User”, “Any Authenticated User”.
- Check your bucket policy to make sure you’re not using
“Principal”: “*”in your policy’s Allow statements.
- Enable Cloud Trail Logging for S3 bucket-level actions and S3 object-level actions and configure to detect anomalies (past and present egress communication).
- Employ code verification techniques (not just a part of CI/CD workflow) in or order to detect for injection patterns described in the evading detection section above.
- A dated yet still relevant post by Scott Piper that advises sprinkling honey tokens throughout your cloud environments in order to trigger detection when those honey tokens are used.