Securing email server in a Nutshell

Securing email server in a Nutshell

November 9, 2018
Profile Icon

Jason Franscisco

Email stands out as one of the most used means of communications we owe in this digital world. When information is involved in such digital communications, security should be considered as the highest priority and this article helps you to understand how an email security mechanism works and how it is implemented.

Before diving into the security of email servers let's take a brief look on the protocols involved in building mail servers. Basically, there are three types of protocols which played a key role in introducing the E-mail system to the world.

  • POP(110)
  • IMAP(143)
  • SMTP(Port 25)

POP stands for Post Office Protocol in which all the mails stored on a mail server gets deleted as soon as they are delivered to the client device. The backup is not available in this scenario if the mails get deleted on the client's device also.

IMAP stands for Internet Message Access Protocol. In this protocol there is a feature to view, organize and delete your emails from the server. Until you decide to delete the mails they will be there.

SMTP stands for Simple Mail Transfer Protocol. Here, we can store all the mails in the server and can sync them to any of the devices we want. SMTP works along with POP and IMAP.

All these protocols generally run in unsecure connection but when they are configured and changed to secure mode then the default port that they will run on will also change to secure mode automatically.

SMTP , POP and IMAP are the core protocols on which email will work but there are many other protocols launched by vendors like MAPI(which is API), Messaging Application Program Interface by Microsoft which is used by Outlook and EAS stands for Exchange Active Sync which is used to sync mails, contacts, events, calendars to connected devices.

Architecture of Email server.

Email server architecture is simple to understand and easy to implement any changes. In Mail server architecture, either POP or IMAP will be working within the server and SMTP will be working along the server.

Architecture of Email Server

SMTP is just a management protocol, which helps to manage the email protocols. It is not involved in sending and receiving emails. All the work will be done by POP/IMAP as per configured. SMTP just manages the emails they receive and send.

When it comes to security, everything we communicate, manage, handle has to be concealed and protected.

Email Server Security Checklist:

  • Closing ports
  • Malware scan of attachments
  • Spam control
  • File types filtering
  • Disable Relaying
  • Enable SPF, DKIM, DMARC
  • Securing DNS
  • Enable reverse DNS

Closing ports:

Usually every protocol runs on a port that is specified to and every device has 65535 ports, for an email server we might be using two ports for secured communication and unsecure communication.

Tit is always advisable to open only the ports you are meant to open rather than opening all the ports available. Always filter the traffic using firewall.

Malware Scan of attachments:

There are many advanced techniques to hide some data in some file formats and one of the basic techniques is steganography. Steganography is used to hide data in the images like jpeg, png. We can hide any data we desire and extract the data using various tools. Similarly, we have to scan compressed files and should observe for any abnormal behavior of files.

Spam control: Spam mails are one of the most challenging problems internet is facing today. Spamming is always ahead in its way whereas the Counter spam mechanisms are lagging behind.

Uncontrolled spam results DoS attack. The memory will be occupied to the most when there are more spam mails bouncing into your inbox. So, we can expect a situation where the server runs out of space and that effects availability of the server which is nothing but termed as Denial of Service(DoS). 

File types filtering: There are so many file types out there like a heap but most of the communications in the mails include word, excel sheet exchanging and some basic formats which can be counted by finger tips. Limiting file types to share in the emails will help to tighten security.

There might be users that will be looking for sharing executable files or some binary files without any intention to harm. It is better to stay away from such file sharings and there are many platforms out there to share the files. We can use integrations in mail service to make things handy.

Another way is to add some extension like salt to password for all the attachments before saving in the server and removing that extension while serving to user. This helps a lot in improving the security because an attacker knows he uploaded executable file, when we change the extension, attacker might not know where the file is and as the extension is changed he cannot filter it even if he is authenticated.

Disable Relaying:

Relaying is a process where anyone can connect to any mail server configured with open relay and send email. This will help spammer to show their true potential.

Email Server Security Relaying Process

Throttling limit:

There is a possibility to set limit for number of emails sent by individual in an interval of time which will helps to find out if someone is spamming around and can block that user.



SPF stands for Sender Policy Framework in which a receiver server only accepts the emails from the domain that are allowed.

Sender Policy Framework
  1. An SPF enabled email server receives an email from somebody@example.com
  2. The email server looks up example.com and reads the SPF TXT record in DNS.
  3. If the originating server of the email matches one of the allowed servers in the SPF record, the message is accepted.


Domain Keys Identified Mail (DKIM) adds an encrypted signature on every message that can be validated by a remote server against a DNS TXT record.

Domain Keys Identified Mail


The Domain-based Message Authentication, Reporting and Conformance (DMARC) protocol builds on SPF and DKIM to handle verification of sender domains. DMARC also provides reporting feature which grants the organizations to access to their email policy. Additionally, DMARC specifies what to do with a message if the SPF and DKIM authentication mechanisms fail. The combination of SPF, DKIM and DMARC creates a trustworthy email environment. All three rely on DNS TXT records to work.

Domain-based Message Authentication, Reporting and Conformance

Securing DNS:

DNS is where the web browser picks IP address of the website from. If DNS is tampered, then our entire online activity like what we watch and what we communicate will be tracked. So, it is highly recommended to use some trusted DNS or build your own.

Enable reverse DNS:

When reverse DNS is activated, SMTP verifies that the senders IP address matches both the host and domain names that were submitted by the SMTP client in the EHLO/HELO command. This is helpful in blocking the messages which fail in address matching test.

These are all the different techniques, which can help you to secure your email server without any hassles. Please let us know in the comment section if we miss any.

Credit: Manindra - Security Researcher

Explore Cybersecurity Platforms

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros.

Learn more
white arrow pointing top right

About Loginsoft

For over 16 years, leading companies in Telecom, Cybersecurity, Healthcare, Banking, New Media and more have come to rely on Loginsoft as a trusted resource for technology talent. Whether Onsite, Offsite, or Offshore, we deliver.

Loginsoft is a leading Cybersecurity services company providing Security Advisory Research to generate metadata for vulnerabilities in Open source components, Discovering ZeroDay Vulnerabilities, Developing Vulnerability Detection signatures using MITRE OVAL Language.

Expertise in Integrations with Threat Intelligence and Security Products, integrated more than 200+ integrations with leading TIP, SIEM, SOAR and Ticketing Platforms such as Cortex XSOAR, Anomali, ThreatQ, Splunk, IBM QRadar, IBM Resilient, Microsoft Azure Sentinel, ServiceNow, Swimlane, Siemplify, MISP, Maltego, Cryptocurrency APIs with Digital Exchange Platforms, CISCO, Datadog, Symantec, Carbonblack, F5, Fortinet and so on.

Interested to learn more? Let’s start a conversation.

Book a meeting


Latest Articles

Get practical solutions to real-world challenges, straight from experts who conquered them.

View all our articles

Sign up to our Newsletter