On 29/06/2015 the server was detected to be sending spam. Even though we have been take measures from updating our server security link. We don’t like spam and we don’t like sending spam. Our provider blocked the port 25 to prevent further spam being sent. For us, whilst we don’t like to be blocked, we don’t want to be sending spam either. In this case our provider only blocked the port rather than the entire server, this means that our website would continue to operate and allowing us to check through the server logs to resolve the issue.
This website only has the following ports open prior to the port 25 block:
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp open http
443/tcp open httpsThis leads to the question, how can a server be sending out spam if the mail port of 25 isn’t even open in the fist place? Logically for spam to be sent and to be detected by our un-managed provider, they would notice this from the network level as they don’t have root access to the server. This means our first point of reference would be the network logs to see its current activity, in addition to this we have been provided with some logs on the spam itself from our provider.
When we checked the network information, we noticed some unusual activity:
netstat -tanpu
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 1 5.135.149.57:32998 98.136.217.202:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:60185 98.136.216.26:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:46344 98.138.112.32:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:42946 98.138.112.33:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:49332 188.125.69.79:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:36927 66.196.118.37:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:57274 207.46.8.199:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:33001 98.136.217.202:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:59820 98.136.217.203:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:56587 98.136.216.25:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:33848 64.233.184.27:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:33854 64.233.184.27:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:33012 98.136.217.202:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:36932 66.196.118.37:25 SYN_SENT 6514/apache2
tcp 0 1 5.135.149.57:56595 98.136.216.25:25 SYN_SENT 6514/apache2
The apache2 service should be running on port 80 for HTTP or port 443 for HTTPS. We can see that on the logs it suggest that the program is running with the process ID of 6514, we then looked into the process ID to see the list of open files associated to this ID.
lsof -p 6514
apache2 6514 www-data cwd DIR 8,1 4096 214405 /var/www/wp-content/themes/[THEME NAME]/css
The output was quiet extensive, however the most interesting line is the one above, where by it shows the theme CSS directory is currently running a process. We looked into this directory and found php script inside a CSS directory! Afterwards we checked the creation date of this php script and compared it against the server apache2 logs. We was able to identify the source of the issue and cleaned up the server.
Spam statistics
Whilst we have resolved the spam issue, we decided to look into the spam emails to see how many spam emails there was and to see their contents.
Total number of spam emails
exim -bpc
124262
We can see that the server tried to send a total of 124,262 email emails.
Message information
exim -bp
89m 1.6K 1ZBIi4-0006pW-Bo <> *** frozen ***
sherri_odom@example.com
The message tried to be sent 89 minutes ago with the message size of 1.6k. The message ID is: 1ZBIi4-0006pW-Bo and it attempted to be sent from sherri_odom@example.com, this particular email account doesn’t actually exist either.
The “frozen” state means the server tried to send an email address but it was unsuccessful. After a few attempts the emails are set in the frozen state an added to the mail queue.
Viewing message header
exim -Mvh 1ZBIi4-0006pW-Bo
Debian-exim 101 103
<>
1435998124 0
-ident Debian-exim
-received_protocol local
-body_linecount 32
-max_received_linelength 103
-allow_unqualified_recipient
-allow_unqualified_sender
-frozen 1435998124
-localerror
XX
1
sherri_odom@example.com
159P Received: from Debian-exim by example.com with local (Exim 4.80)
id 1ZBIi4-0006pW-Bo
for sherri_odom@example.com; Sat, 04 Jul 2015 09:22:04 +0100
046 X-Failed-Recipients: your_email_address@example.com
029 Auto-Submitted: auto-replied
060F From: Mail Delivery System <Mailer-Daemon@example.com>
034T To: sherri_odom@example.com
059 Subject: Mail delivery failed: returning message to sender
049I Message-Id: <E1ZBIi4-0006pW-Bo@example.com>
038 Date: Sat, 04 Jul 2015 09:22:04 +0100
Here we can see the header of the email which tried to be sent.
Message content
exim -Mvb 1ZBIi4-0006pW-Bo
1ZBIi4-0006pW-Bo-D
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
your_email_address@example.com
Mailing to remote domains not supported
------ This is a copy of the message, including all the headers. ------
Return-path: <sherri_odom@example.com>
Received: from www-data by example.com with local (Exim 4.80)
(envelope-from <sherri_odom@example.com>)
id 1ZBIi4-0006pL-AF
for your_email_address@example.com; Sat, 04 Jul 2015 09:22:04 +0100
To: your_email_address@example.com
Subject: Best Popular Vagra web store
X-PHP-Originating-Script: 33:dir.php
From: "Sherri Odom" <sherri_odom@example.com>
Reply-To:"Sherri Odom" <sherri_odom@example.com>
X-Priority: 3 (Normal)
MIME-Version: 1.0
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit
Message-Id: <E1ZBIi4-0006pL-AF@example.com>
Date: Sat, 04 Jul 2015 09:22:04 +0100
<div>
Best Popular Vagra web store – <a href="http://SpamWebsite.com/">check it out</a>
</div>
As we can see in the email, the contents are definitely from a spamming nature. This is also the typical email in which you may receive within your inbox from time to time. In this case, the emails was blocked from being sent in the first place, this means this spam never reached your inbox.
Removing the emails from the system
exim -bp | exiqgrep -i | xargs exim -Mrm
Message 1ZBIi4-0006pW-Bo has been removed
The issue for the email spam has been resolved, however we still have the issue where by the server currently still holds 124,262. We will look to clear the mail queue.
