Spam

On 29/06/2015 the server was detected to be sending spam. Even though we have been take measures from updating our server security link. We don’t like spam and we don’t like sending spam. Our provider blocked the port 25 to prevent further spam being sent. For us, whilst we don’t like to be blocked, we don’t want to be sending spam either. In this case our provider only blocked the port rather than the entire server, this means that our website would continue to operate and allowing us to check through the server logs to resolve the issue.

This website only has the following ports open prior to the port 25 block:

PORT    STATE SERVICE
21/tcp  open  ftp
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

This leads to the question, how can a server be sending out spam if the mail port of 25 isn’t even open in the fist place? Logically for spam to be sent and to be detected by our un-managed provider, they would notice this from the network level as they don’t have root access to the server. This means our first point of reference would be the network logs to see its current activity, in addition to this we have been provided with some logs on the spam itself from our provider.

spam_status

When we checked the network information, we noticed some unusual activity:

netstat -tanpu

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      1 5.135.149.57:32998      98.136.217.202:25       SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:60185      98.136.216.26:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:46344      98.138.112.32:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:42946      98.138.112.33:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:49332      188.125.69.79:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:36927      66.196.118.37:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:57274      207.46.8.199:25         SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:33001      98.136.217.202:25       SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:59820      98.136.217.203:25       SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:56587      98.136.216.25:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:33848      64.233.184.27:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:33854      64.233.184.27:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:33012      98.136.217.202:25       SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:36932      66.196.118.37:25        SYN_SENT    6514/apache2    
tcp        0      1 5.135.149.57:56595      98.136.216.25:25        SYN_SENT    6514/apache2

The apache2 service should be running on port 80 for HTTP or port 443 for HTTPS. We can see that on the logs it suggest that the program is running with the process ID of 6514, we then looked into the process ID to see the list of open files associated to this ID.

lsof -p 6514

apache2 6514 www-data cwd DIR 8,1 4096 214405 /var/www/wp-content/themes/[THEME NAME]/css

The output was quiet extensive, however the most interesting line is the one above, where by it shows the theme CSS directory is currently running a process. We looked into this directory and found php script inside a CSS directory! Afterwards we checked the creation date of this php script and compared it against the server apache2 logs. We was able to identify the source of the issue and cleaned up the server.


Spam statistics

Whilst we have resolved the spam issue, we decided to look into the spam emails to see how many spam emails there was and to see their contents.

Total number of spam emails

exim -bpc
 124262

We can see that the server tried to send a total of 124,262 email emails.

Message information

exim -bp

89m  1.6K 1ZBIi4-0006pW-Bo <> *** frozen ***
          sherri_odom@example.com

The message tried to be sent 89 minutes ago with the message size of 1.6k. The message ID is: 1ZBIi4-0006pW-Bo and it attempted to be sent from sherri_odom@example.com, this particular email account doesn’t actually exist either.

The “frozen” state means the server tried to send an email address but it was unsuccessful. After a few attempts the emails are set in the frozen state an added to the mail queue.

Viewing message header

exim -Mvh 1ZBIi4-0006pW-Bo

Debian-exim 101 103
<>
1435998124 0
-ident Debian-exim
-received_protocol local
-body_linecount 32
-max_received_linelength 103
-allow_unqualified_recipient
-allow_unqualified_sender
-frozen 1435998124
-localerror
XX
1
sherri_odom@example.com

159P Received: from Debian-exim by example.com with local (Exim 4.80)
	id 1ZBIi4-0006pW-Bo
	for sherri_odom@example.com; Sat, 04 Jul 2015 09:22:04 +0100
046  X-Failed-Recipients: your_email_address@example.com
029  Auto-Submitted: auto-replied
060F From: Mail Delivery System <Mailer-Daemon@example.com>
034T To: sherri_odom@example.com
059  Subject: Mail delivery failed: returning message to sender
049I Message-Id: <E1ZBIi4-0006pW-Bo@example.com>
038  Date: Sat, 04 Jul 2015 09:22:04 +0100

Here we can see the header of the email which tried to be sent.

Message content

exim -Mvb 1ZBIi4-0006pW-Bo

1ZBIi4-0006pW-Bo-D
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  your_email_address@example.com
    Mailing to remote domains not supported

------ This is a copy of the message, including all the headers. ------

Return-path: <sherri_odom@example.com>
Received: from www-data by example.com with local (Exim 4.80)
	(envelope-from <sherri_odom@example.com>)
	id 1ZBIi4-0006pL-AF
	for your_email_address@example.com; Sat, 04 Jul 2015 09:22:04 +0100
To: your_email_address@example.com
Subject:   Best Popular Vagra web store
X-PHP-Originating-Script: 33:dir.php
From: "Sherri Odom" <sherri_odom@example.com>
Reply-To:"Sherri Odom" <sherri_odom@example.com>
X-Priority: 3 (Normal)
MIME-Version: 1.0
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit
Message-Id: <E1ZBIi4-0006pL-AF@example.com>
Date: Sat, 04 Jul 2015 09:22:04 +0100


<div>
 Best Popular Vagra web store – <a href="http://SpamWebsite.com/">check it out</a>
</div>

As we can see in the email, the contents are definitely from a spamming nature. This is also the typical email in which you may receive within your inbox from time to time. In this case, the emails was blocked from being sent in the first place, this means this spam never reached your inbox.

Removing the emails from the system

exim -bp | exiqgrep -i | xargs exim -Mrm

Message 1ZBIi4-0006pW-Bo has been removed

The issue for the email spam has been resolved, however we still have the issue where by the server currently still holds 124,262. We will look to clear the mail queue.