Monthly Archives: March 2019

Responding to AWS Abuse Alerts

If you have never received the dreaded AWS Abuse notifications on your cloud instances then you need not read the rest of the article 🙂

However, if you recently adopted AWS and received such a notice, then the following tips might come in handy.

  • Do not panic
  • To ensure you are NOT falling for any phishing campaign, please pay a close watch on the sending email addresses. AWS sends the abuse notices from the following email addresses:

Amazon EC2 Abuse <ec2-abuse@amazon.com>

  • The abuse notices are usually sent to the email addresses associated with Root account and to the email addresses that are explicitly added in alternate contacts in the settings page. Please ensure emails going to these addresses are monitored. If you have a multiple people that needs to be alerted, create a Distribution List and add them in alternate contacts of AWS Settings. The abuse reports clearly mentions the following details:
  • Instance IDs
  • Public IP address of instances involved in malicious behaviour
  • Ports and Protocols
  • Destination IP, Ports and URLs
  • Start Time and End Time (if it has indeed ended)
  • Type of malicious activity like port scan, crypto mining, bot behaviour
  • Read through the notice carefully, based on the notice you can figure out the type of the incident your instances are involved in
  • Note that AWS does not provide any technical support on these issues
  • First thing is to block inbound and outbound public connections to the reported host. You can use Security Group of the host for this. There would definitely be downtime of this host, so please know the repercussions
  • You might want to leave the access open to only whitelisted IP addresses to examine the host
  • If multiple hosts are involved, better to use Network ACL as its applied at the subnet level
  • Go through the logs, bash history, running services and processes. You can use netstat or lsof commands
  • You might have to preserve the instance for forensics purposes. Use shutdown rather than terminate. This article has good points with regards to forensics on AWS
  • Once you have taken the necessary remediation steps, do not forget to reply to the AWS team with the action you have taken (this is important)
  • A very important prerequisite for any Incident Response is solid logging setup. Make sure you have it set up for all the hosts. Cloudtrail provides valuable insight on API calls made.
  • Some challenges exist in a microservice architecture where hosts are transient, make sure all the nodes are configured properly for logging.
  • Leverage Open Source or commercial software for logging.
  • Highly recommended to run Network or Host based IDS. There are lot of open source solutions that you can deploy if cost is a concern. Ex: OSSEC, Wazuh (OSSEC fork),

I have collated some of the notices that I handled first-hand as part of my consulting and some sourced through friends. Typical notices are as follows:

Example 1:

“Dear Amazon EC2 Customer,

We’ve received a report that your instance(s):

Instance Id: xxxxx

IP Address: xx.xx.xx.xx

has been placing spam (unsolicited messages, typically advertisements) on websites hosting online discussions, such as Internet forums; check the information provided below by the abuse reporter.

This is specifically forbidden in our User Agreement: http://aws.amazon.com/agreement/

Please confirm that all necessary steps to cease this activity have been taken on your side and reply this email to send your reply of action to the original abuse reporter. This will activate a flag in our ticketing system, letting us know that you have acknowledged receipt of this email.

It’s possible that your environment has been compromised by an external attacker. It remains your responsibility to ensure that your instances and all applications are secured. The link http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1233

provides some suggestions for securing your instances.

Case number: xxxxxxxxxxxx

Additional abuse report information provided by original abuse reporter:

* Destination IPs:

* Destination Ports:

* Destination URLs:

* Abuse Time: Wed Sep 30 14:39:00 UTC

* Log Extract:

<<<

Reported-From:

Category: abuse

Report-Type: regbot

Service: regbot

>>>

* Comments:

Probable Cause & Response:

Improper configuration of Apache web server. Someone forgot to turn off the Proxy Requests ON causing random people to connect and send spam. Required us to dig through the logs, check running services and processes.

Remember it only takes a small window of time for a malicious user to discover and exploit vulnerabilities. Strongly recommend having a Host Based IDS like OSSEC on each of the servers even if they are not publicly exposed

Example 2:

“Hello,

We’ve received a report(s) that your AWS resource(s)

EC2 Instance Id: i-0xxxxxxxxx

[IP Address]

has been implicated in activity which resembles scanning remote hosts on the internet for security vulnerabilities. Activity of this nature is forbidden in the AWS Acceptable Use Policy (https://aws.amazon.com/aup/). We’ve included the original report below for your review.

Please take action to stop the reported activity and reply directly to this email with details of the corrective actions you have taken. If you do not consider the activity described in these reports to be abusive, please reply to this email with details of your use case.

If you’re unaware of this activity, it’s possible that your environment has been compromised by an external attacker, or a vulnerability is allowing your machine to be used in a way that it was not intended.

We are unable to assist you with troubleshooting or technical inquiries. However, for guidance on securing your instance, we recommend reviewing the following resources:

Please remember that you are responsible for ensuring that your instances and all applications are properly secured. If you require any further information to assist you in identifying or rectifying this issue, please let us know in a direct reply to this message.

Regards,

AWS Abuse

Abuse Case Number: xxxxxxxx

— -Beginning of forwarded report(s) — -

* Log Extract:

<<<

— — — — — — — — — — — — — –

AWS Account: xxxxxxxxx

Report begin time: 29-Apr-2016 08:11:37 UTC

Report end time: 29-Apr-2016 08:12:35 UTC

Protocol: TCP

Remote IP: xx.xxx.xx.xx

Remote port(s): multiple ports (1505 ports in total)

Total bytes sent: 61200

Total packets sent: 1530

Total bytes received: 0

Total packets received: 0

>>>

* Comments:

<<<

>>>

Probable Cause & Response:

In this case, someone indeed forgot to take prior approval while carrying out port scanning. This could also occur if a malicious user or malware was trying to scan public servers as part of reconnaissance

NOTE: A few days back AWS waived off prior requirement of taking approval from AWS for pentesting

Example 3:

This was not AWS Abuse report 🙂 but I just added to to create some awareness

A friend of mine called up stating that his company’s AWS account was hacked. Though he mentioned that they had changed the root account password, there was a spike in the number of instances being spun/spinned up and his boss’s credit card was charged for 1.5 lakhs INR (about 2000$).

Apparently they were using the root account for operations and had not created different user accounts, forget about IAM roles!

As soon as I instructed them to revoke the AWS API Keys, the spike in usage dropped and meanwhile they were busy talking to the credit card company for a refund/chargeback. Ofcourse it was a costly lesson for them for not using the best practices of MFA, using separate accounts, roles and etc

A good practice would be to list down all the assets based on region in a simple spreadsheet. This can come in handy when responding to notices especially if you are using different regions in AWS. There are some good open source as well as commercial tools that will help you effectively manage security posture of your AWS environment. I will probably write a detailed article on using some of these but if you can check out following tools:

  • Scout Suite
  • Trusted Advisor Reports (You get exhaustive report with Business Support)
  • Nessus — Cloud configuration audits

A couple of months back AWS also released steps on automating responses to abuse alerts. I haven’t tried this yet and will share my experience once done. You can read about it here:

Found this useful? buymeacoffee.com/prax