Category Archives: security

Responding to AWS Abuse Alerts

If you have never received the dreaded AWS Abuse notifications on your cloud instances then you need not read the rest of the article 🙂

However, if you recently adopted AWS and received such a notice, then the following tips might come in handy.

  • Do not panic
  • To ensure you are NOT falling for any phishing campaign, please pay a close watch on the sending email addresses. AWS sends the abuse notices from the following email addresses:

Amazon EC2 Abuse <ec2-abuse@amazon.com>

  • The abuse notices are usually sent to the email addresses associated with Root account and to the email addresses that are explicitly added in alternate contacts in the settings page. Please ensure emails going to these addresses are monitored. If you have a multiple people that needs to be alerted, create a Distribution List and add them in alternate contacts of AWS Settings. The abuse reports clearly mentions the following details:
  • Instance IDs
  • Public IP address of instances involved in malicious behaviour
  • Ports and Protocols
  • Destination IP, Ports and URLs
  • Start Time and End Time (if it has indeed ended)
  • Type of malicious activity like port scan, crypto mining, bot behaviour
  • Read through the notice carefully, based on the notice you can figure out the type of the incident your instances are involved in
  • Note that AWS does not provide any technical support on these issues
  • First thing is to block inbound and outbound public connections to the reported host. You can use Security Group of the host for this. There would definitely be downtime of this host, so please know the repercussions
  • You might want to leave the access open to only whitelisted IP addresses to examine the host
  • If multiple hosts are involved, better to use Network ACL as its applied at the subnet level
  • Go through the logs, bash history, running services and processes. You can use netstat or lsof commands
  • You might have to preserve the instance for forensics purposes. Use shutdown rather than terminate. This article has good points with regards to forensics on AWS
  • Once you have taken the necessary remediation steps, do not forget to reply to the AWS team with the action you have taken (this is important)
  • A very important prerequisite for any Incident Response is solid logging setup. Make sure you have it set up for all the hosts. Cloudtrail provides valuable insight on API calls made.
  • Some challenges exist in a microservice architecture where hosts are transient, make sure all the nodes are configured properly for logging.
  • Leverage Open Source or commercial software for logging.
  • Highly recommended to run Network or Host based IDS. There are lot of open source solutions that you can deploy if cost is a concern. Ex: OSSEC, Wazuh (OSSEC fork),

I have collated some of the notices that I handled first-hand as part of my consulting and some sourced through friends. Typical notices are as follows:

Example 1:

“Dear Amazon EC2 Customer,

We’ve received a report that your instance(s):

Instance Id: xxxxx

IP Address: xx.xx.xx.xx

has been placing spam (unsolicited messages, typically advertisements) on websites hosting online discussions, such as Internet forums; check the information provided below by the abuse reporter.

This is specifically forbidden in our User Agreement: http://aws.amazon.com/agreement/

Please confirm that all necessary steps to cease this activity have been taken on your side and reply this email to send your reply of action to the original abuse reporter. This will activate a flag in our ticketing system, letting us know that you have acknowledged receipt of this email.

It’s possible that your environment has been compromised by an external attacker. It remains your responsibility to ensure that your instances and all applications are secured. The link http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1233

provides some suggestions for securing your instances.

Case number: xxxxxxxxxxxx

Additional abuse report information provided by original abuse reporter:

* Destination IPs:

* Destination Ports:

* Destination URLs:

* Abuse Time: Wed Sep 30 14:39:00 UTC

* Log Extract:

<<<

Reported-From:

Category: abuse

Report-Type: regbot

Service: regbot

>>>

* Comments:

Probable Cause & Response:

Improper configuration of Apache web server. Someone forgot to turn off the Proxy Requests ON causing random people to connect and send spam. Required us to dig through the logs, check running services and processes.

Remember it only takes a small window of time for a malicious user to discover and exploit vulnerabilities. Strongly recommend having a Host Based IDS like OSSEC on each of the servers even if they are not publicly exposed

Example 2:

“Hello,

We’ve received a report(s) that your AWS resource(s)

EC2 Instance Id: i-0xxxxxxxxx

[IP Address]

has been implicated in activity which resembles scanning remote hosts on the internet for security vulnerabilities. Activity of this nature is forbidden in the AWS Acceptable Use Policy (https://aws.amazon.com/aup/). We’ve included the original report below for your review.

Please take action to stop the reported activity and reply directly to this email with details of the corrective actions you have taken. If you do not consider the activity described in these reports to be abusive, please reply to this email with details of your use case.

If you’re unaware of this activity, it’s possible that your environment has been compromised by an external attacker, or a vulnerability is allowing your machine to be used in a way that it was not intended.

We are unable to assist you with troubleshooting or technical inquiries. However, for guidance on securing your instance, we recommend reviewing the following resources:

Please remember that you are responsible for ensuring that your instances and all applications are properly secured. If you require any further information to assist you in identifying or rectifying this issue, please let us know in a direct reply to this message.

Regards,

AWS Abuse

Abuse Case Number: xxxxxxxx

— -Beginning of forwarded report(s) — -

* Log Extract:

<<<

— — — — — — — — — — — — — –

AWS Account: xxxxxxxxx

Report begin time: 29-Apr-2016 08:11:37 UTC

Report end time: 29-Apr-2016 08:12:35 UTC

Protocol: TCP

Remote IP: xx.xxx.xx.xx

Remote port(s): multiple ports (1505 ports in total)

Total bytes sent: 61200

Total packets sent: 1530

Total bytes received: 0

Total packets received: 0

>>>

* Comments:

<<<

>>>

Probable Cause & Response:

In this case, someone indeed forgot to take prior approval while carrying out port scanning. This could also occur if a malicious user or malware was trying to scan public servers as part of reconnaissance

NOTE: A few days back AWS waived off prior requirement of taking approval from AWS for pentesting

Example 3:

This was not AWS Abuse report 🙂 but I just added to to create some awareness

A friend of mine called up stating that his company’s AWS account was hacked. Though he mentioned that they had changed the root account password, there was a spike in the number of instances being spun/spinned up and his boss’s credit card was charged for 1.5 lakhs INR (about 2000$).

Apparently they were using the root account for operations and had not created different user accounts, forget about IAM roles!

As soon as I instructed them to revoke the AWS API Keys, the spike in usage dropped and meanwhile they were busy talking to the credit card company for a refund/chargeback. Ofcourse it was a costly lesson for them for not using the best practices of MFA, using separate accounts, roles and etc

A good practice would be to list down all the assets based on region in a simple spreadsheet. This can come in handy when responding to notices especially if you are using different regions in AWS. There are some good open source as well as commercial tools that will help you effectively manage security posture of your AWS environment. I will probably write a detailed article on using some of these but if you can check out following tools:

  • Scout Suite
  • Trusted Advisor Reports (You get exhaustive report with Business Support)
  • Nessus — Cloud configuration audits

A couple of months back AWS also released steps on automating responses to abuse alerts. I haven’t tried this yet and will share my experience once done. You can read about it here:

Found this useful? buymeacoffee.com/prax

Tuning OSSEC Email Notifications

One of the common complaints you will encounter while working with Intrusion Detection Systems (IDSs) are about false positives and continuos notifications. OSSEC is no different, despite a global upper rule for email notifications, it continues to bombard emails for events with lower severity ratings.

Although the documentation of OSSEC states this explicitly , it does not mention which exact rules can trigger these email notifications:

“Some rules have an option set to force OSSEC into sending an alert email. This option is <options>alert_by_email</options>. One of these rules is 1002. To ignore these rules you will have to create a rule to specifically ignore it, or overwrite the rule without the alert_by_emailoption.

I sifted through the OSSEC configurations to list out the rules generating email notifications lower than the threshold limit(which is level 7 by default):

If you want to disable the email alerts, you would need to edit one or more of the above rules. I have provided the line numbers so that you can quickly refer them. I had difficulty in embedding the table above so its a screenshot (I did try airtable, but didn’t quite like it to be using here). You can download the .csv files for reference from here —

https://github.com/gravityfuel/ossec-tuning

Let’s take syslog_rules.xml as an example. As per the configuration, this is a level 2 rule and implying that it should not trigger any email notifications for the events if the global configuration for email notifications is set to Level 7 and above. But in reality, this particular rule triggers a lot of notifications if the server is public facing and the corresponding syslog entry contains any of the preconfigured key words:

Rule:

<rule id=”1002″ level=”2″>

<match>$BAD_WORDS</match>

<options>alert_by_email</options>

<description>Unknown problem somewhere in the system.</description>

</rule>

Trigger Condition:

<var name=”BAD_WORDS”>core_dumped|failure|error|attack|bad |illegal |denied|refused|unauthorized|fatal|failed|Segmentation Fault|Corrupted</var>

Based on your environment, you can tune this further by either deleting some of the bad words or prevent this rule from triggering email alert by adding no_email_alert options

<options>no_email_alert</options>

or by commenting the default rule

<! — <options>alert_by_email</options> →

Once modified, the rule would look something like this:

<rule id=”1002″ level=”2″>

<match>$BAD_WORDS</match>

<options>no_email_alert</options>

<description>Unknown problem somewhere in the system.</description>

Ofcourse, above is only one of the many ways of minimizing notifications. Do comment on how you deal with incessant and false positive notifications.

Open Port 445, WannaCry,India, — A report

When WannaCry wrecked havoc last week there were widespread concerns that a lot of systems in India had fallen to this malware. However there were conflicting reports about the infection rate in India. Some reports cited that it was not as bad as expected while others differed.

With this in mind, I just wanted to know how vulnerable we were for this malware.

As per documents WannaCry malware tries to spread by infecting hosts that has port 445 open with Server Message Block (SMB) version 1 and running an unpatched version of Windows. Firstly, there is absolutely no need for systems to expose port 445 to internet, but why they are open could be topic for another blog post.

I looked up Shodan to identify the number of systems in India running with port 445 open to Internet. The result was a whopping 25000+ systems.

There is an interesting aspect to this. The numbers varied to a large degree over a period of week. Peeking on Weekdays and tapering on weekends. The following chart gives an overview:

On the weekdays the number of machines online with port 445 open were around 25342 at the highest while on weekends it lowered to 17000! Thats a different of nearly 8000, so looks like lot of systems are shutdown at the weekends. Do note that the Shodan crawlers work 24/7 and update the results realtime.

Though Shodan resulted around 25,000+ systems with port 445 open, not all of these machines were running vulnerable versions of Windows Operating Systems. Taking one of the days when the number was at its peak, following is the breakup:

There is however a large chunk of machines running older versions of Windows. Windows 2003/XP/2008/7/Vista accounted for nearly 65% of the servers scanned. Following chart gives the breakup of Windows Machines (Servers and Desktops editions included):

As you can see, lot of machines in India are running out of date operating systems. Note that machines like Windows Server 2003, Windows XP, Windows 7 have met their End Of Life. Microsoft went beyond its policy of not releasing security updates to their EOL products by actually releasing security updates for discontinued Operating Systems like XP, 2003 and others. If you are still running those Operating Systems please patch, better, Upgrade. Check this Microsoft page for more information — Customer Guidance for Wannacrypt Attacks

SMB Version Spread:

smb version spread in India

Note that SMB v.1 is vulnerable to WannaCry Attack

SMB Authentication Status.

For the exploit to work, SMB Authentication should be in a disabled state. Fortunately a good percentage of scanned machines seemed to have authentication enabled. Also note that exposing network shares to public is a security nightmare as it can leak data meant for internal users. At the least, enable password protection.

Top Cities in India with Port 445 Open:

Metros seemed to top the list of cities where machines were running with port 445 open. There is no reason for anyone to expose port 445 on internet. At the most they are needed for file / printer sharing and it should be limited to Local network. One should configure their firewalls to block port 445

ISPs with subscribers having Port 445 publicly exposed.

BSNL seems to operate a network with maximum number of machines having port 445 open. I am not surprised to see port 445 open is such large numbers on BSNL network. I suspect UPnP (Universal Plug n Play) feature is enabled by default in the Routers provided by BSNL and this in turn is causing the port 445 on internal hosts to be exposed! (people using BSNL/ISP Provided routers please comment)

There is an nmap script to check if the machine is vulnerable or not. To correlate the above findings, I randomly checked few machines in the list and couple of them turned out to be vulnerable, while others did not. Based on this, following seems to fit the bill for WannaCry infection:

  • Running SMB version 1.0
  • Running an unpatched version of Windows 7 / xp/ 2012/ 8 / 2003
  • Port 445 exposed to internet
  • Authentication disabled

If you are running a version of Windows and would like to check if the required patch is installed or not, please refer the article below:

https://support.microsoft.com/en-us/help/4023262/how-to-verify-that-ms17-010-is-installed

Further reading: