Comparison: 20 email service providers put to the deliverability test

The results at a glance

note: We are a permanent service partner of Emarsys, but we also provide services on the platforms of other providers examined here. We made sure to make the comparison on other systems regardless of our collaboration with Emarsys and our services. The email sample sets, on the basis of which we checked the listed deliverability criteria, did not receive any emails from companies that we handle in email marketing ourselves.

Explanations of the comparative result

In the email samples we checked, Emarsys, Teradata eCircle, and Mailingwork meet CSA requirements almost seamless, followed by Artegic, Optivo, Responsys and Salesforce subsidiary ExactTarget.

A bit less complete In the implementation of the CSA technical criteria, the comparison shows Selligent, Inxmail, Experian (the platform of the company acquired in 2009 United Mail Solutions (UMS), not the US counterpart Cheetahmail), Evalanche and XQueue. They are followed by CleverReach, SmartFocus (formerly Emailvision) and Promio.

Serious gaps When implementing the technical CSA requirements, our sample includes Agnitas, Kajomi, Rapidmail and Silverpop. The thinnest criteria coverage was shown in our review of Flatrate Newsletters.

Potential and implementation

Beyond simply documenting implemented parameters, we were able to determine that there are significant qualitative fluctuations with regard to the consistency with which individual technical configurations were implemented. We have marked this in the graphic with the detailed comparison result with appropriate colors.

Green and gray represent consistency

Some shipping service providers were very consistent in the implementation — and also in the non-implementation — of technical configurations in the sample. When presenting the comparison result, we marked this with the colors green and gray: In these cases, a technical criterion was either implemented technically correctly and in accordance with CSA requirements for all evaluated emails — or the criterion was consistently implemented in none of the evaluated emails.

Light blue stands for inconsistency

For some shipping service providers, the comparison has shown that one or more technical criteria across various mail servers, IP addresses and domains were partly correct, partly incorrectly and partly not implemented at all. In the comparative presentation, we identified this with the color light blue: In this case, some of the emails evaluated met the criteria examined, others did not. Light blue therefore characterizes shipping platforms that actually have the potential to fully meet a specific criterion — but do not always do so consistently.

Dark blue stands for incomplete fulfilment of a criterion

In dark blue, we marked the cases in which, in our comparison, a technical criterion — such as the SPF entry or the list unsubscribe link — was implemented in such a way that it worked technically, but was implemented in a way that did not meet the CSA requirements. The color dark blue also indicates platforms that could potentially meet more CSA requirements, but do not (consistently) do so in practice.

Implementation ethics: Our Must Haves, Good to Haves & Should Haves

If you look at the comparison result more closely, one more point catches the eye: While 77 percent of the entries across all platforms and criteria in the “Our Must Haves” category are green and thus indicate complete compliance with the respective criterion and provider for all emails examined, the figure in the “Our Good to Haves” category is only 45 percent. In the “Our Should Haves” category, “green” coverage drops to just 40 percent.

“Implementation morale” therefore decreases as the importance of technical deliverability criteria decreases. This in and of itself may not seem surprising at first. However, this finding becomes somewhat controversial when you consider that the compared shipping service providers are all CSA certified and should therefore (with a few exceptions set by the CSA) fully meet all criteria. And this also includes more “bureaucratic” criteria, such as the implementation of the “X-CSA Complaints” header.

To put it casually, it's surprising that the result is that not everywhere that says CSA is actually in it. In other words, not all parameters specified by the CSA are always implemented. For email marketers, this means that a critical second look at deliverability is recommended when choosing a platform as well as when setting up — even if a shipping service provider can come up with CSA certification.

Read more in the Publicare blog

” How much is a CSA certificate really worth?

Explanations of the comparison

1. General

This comparison relates exclusively to various technical configurations of the compared shipping platforms in terms of meeting the CSA admission criteria. This comparison makes no statement about the many other platform features and features that should also play an important role in choosing the appropriate email marketing delivery solution for companies.

2. Empirical basis

For this settlement, Publicare signed up for a variety of newsletters and for commercial communication from various companies. From the volume of all email communication that we received in this context within the last six months, we created sample sets of three emails per sending platform for evaluation. Each set of 3 contains emails from three different companies, which act as senders in the email header.

The sets of 3 were randomly selected by us. No shipping service provider was given preferential treatment or discrimination when selecting and evaluating the data.

However, we are aware that a set of three emails per delivery platform is a relatively small sample. Occasional “lucky moves” or vice versa “bad luck” when choosing the sets of 3 cannot be absolutely ruled out — even though we were committed to maximum fairness at all times. However, since our test criteria do not relate to email content, which is often added individually, but to 'technical' aspects, such as certain header fields, which are each generated automatically by the shipping system, we believe that our samples show a trend that is suitable as a basis for a comparison. We see this assessment confirmed by the many cases in which criteria were consistently met well or poorly across all three emails in a sample set.

3. Analysis criteria

We compared each email according to eight technical criteria, which are included in the CSA's acceptance criteria for certified senders in the 'Technical Configurations' section and are largely among the best practices of the major Internet service providers for mass senders.

We are aware that there are a number of other technical criteria for optimal deliverability rates that could not be taken into account as part of this comparison — such as the technical protection of sending mail servers against external attacks, including

4. Color markings and weight of criteria

a. color scheme

In the presentation of the comparison results, we have color-coded the implementation of the CSA criteria. We did this as follows:

Green indicates that a comparison criterion for all emails in a set of 3 was technically correct and optimally implemented in accordance with the CSA criteria.
We marked dark blue when a comparison criterion for all emails in a set of 3 was technically correct but not implemented in accordance with CSA requirements.
Light blue indicates that a comparison criterion was correctly implemented only for some emails from a set of 3.
A field appears in gray if a comparison criterion was not implemented for any of the emails from a set of 3 (regardless of whether incorrect or not at all).

b. Weight of criteria

In our opinion, the individual criteria for good deliverability are of varying importance. In order to make this clear in the overall analysis, we have decided to categorize the criteria. For this purpose, we have divided the eight analysis criteria into three categories.

We have made the category a comparison criterion assigned to depends on how important its implementation — in our opinion — is for the deliverability of marketing emails. Estimates of how important it may be to implement a comparison criterion from other aspects were not taken into account. In assessing the importance, we have based our many years of experience, the guidelines and best practices of major Internet service providers, and the requirements of the CSA.

The analysis categories in detail

Category 1: “Our must haves”: These criteria are essential for good deliverability

We have assigned criteria that we consider essential in connection with the deliverability of marketing emails to the first category. These are basic sender authentication measures, which are required by most major Internet Service Providers (ISPs) and are often taken into account as mail filter criteria.

1. Reverse DNS record

What is it about?
A reverse DNS record assigns a server's IP address to a domain. In this way, it can be checked whether the IP address used to send an email can actually be assigned to the sender domain specified by the sending server.

What did we specifically check?
Is it possible to reverse resolve the IP address of the sending mail server via PTR? Does the PTR resource record take the form of a fully qualified domain name (FDQN)? Does this FDQN exactly match the domain name that the sending mail server transmits in the SMTP dialog in the HELO command?

Why is that important?
A reverse DNS entry is required by almost all major Internet service providers (ISPs) or recommended as a best practice. For CSA-certified broadcasters, a correct PTR entry is mandatory (see point l in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• We marked shipping platforms in green where every email evaluated met the CSA requirements.
• Sending platforms where only some of the evaluated emails met CSA requirements are marked light blue.
• Sending platforms where none of the evaluated emails met the criteria examined are marked gray.
  

2. SPF record for the envelope sender domain (MFROM)

What is it about?
The Sender Policy Framework (SPF) is a widely used method for sender authentication. The SPF process can be used to determine whether an IP address is allowed to send emails in the name of the sender domain.

What did we specifically check?
Is there a correct SPF record for the envelope sender domain of the email (MAIL-FROM)? Does this SPF record end in “-all” or “~all”?

The SPF record for the exact envelope from domain (FDQN) was checked. If there was no SPF record for the subdomain used for sending, but only for the corresponding main domain, we did not consider this criterion to be met. The same applies to SPF records that contained incorrectly encoded characters. Sender ID records were not included in our comparison.

Why is that important?
SPF is considered as a filter criterion by almost all major ISPs. Such an SPF entry is mandatory by the CSA for certified broadcasters (see point r in the 'Technical Configurations' section of the CSA admission criteria for mass senders).

How did we mark?

• We marked sending platforms in green where each evaluated email had a correct SPF record for the envelope sender, which ended in “-all” or “~all” and thus meets the CSA requirement for technical configurations, point r: “For the sender address (envelope-mfrom) specified in SMTP communication between e-mail servers, an SPF-MFROM record must be entered, which is SPF-MFROM allows systems on the recipient side to carry out an SPF test. The SPF record must end with “~all” or “-all.” In this regard, please also note the conditions under the recommendations for SenderID entries under (3.g) . *”
• Sending platforms are marked in dark blue where there was a correct SPF record for the envelope sender for every evaluated email, but this did not end in “-all” or “~all” (but, for example, in “? all”).
• Light blue indicates sending platforms where only some of the evaluated emails had a correct SPF record for the envelope sender.
• Grey marks sending platforms where there was no correct SPF record for the envelope sender for any of the evaluated emails.

3. DomainKeys Identified Mail (DKIM)

What is it about?
In addition to SPF, DKIM is the most commonly used method for sender authentication. Unlike SPF, authentication works via a cryptographic process. To do this, a hash value must be inserted as a DKIM signature in the header of the sent email. The receiving mail server compares this value with a publicly stored key and checks whether the two match.

What did we specifically check?
Does the header of the sent email contain a DKIM signature? Is there a DKIM key for the domain that was specified in the signature, with the selector that was mentioned in the signature?

Why is that important?
In addition to SPF, the DKIM process is considered as a filter criterion by almost all major ISPs. Microsoft is an exception. With exceptions, the use of DKIM is required for CSA-certified broadcasters (see point u in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• Green marks delivery platforms where every email evaluated met the requirements.
  
• Sending platforms where only some of the evaluated emails met the requirements are marked light blue.
• Sending platforms where none of the evaluated emails met the requirements are marked gray.
  

Category 2: “Our Good to Haves”: These criteria are recommended for optimal deliverability

The second category includes criteria that major Internet service providers generally do not use as general filter criteria when delivering emails. However, their fulfillment ensures a smoother flow of opt-out and complaint processes. In this way, high spam complaint rates and problems with individual ISPs are avoided in the long term.

1. Whois entry for the sending IP address

What is it about?
Especially in the event of problems, it is important that the receiving Internet service providers can quickly and easily determine which shipping service provider an IP address is assigned to. It should therefore be possible to clearly identify a shipping service provider via a Whois query. A contact or abuse email address should be included in the Whois entry.

What did we specifically check?
Is there a Whois entry for the IP address from which an email was sent? Does this Whois entry point directly to the shipping service provider — or 'just' to a host through which the shipping service provider must be requested in turn? Does the Whois record contain an abuse email address?

Why is that important?
It can be assumed that Whois entries are usually not checked and evaluated by default for every incoming email. However, in the event of problems — if an ISP suspects that your email address is being misused to send spam, etc. — it is necessary for the ISP to be able to contact the sender quickly and easily. For CSA-certified senders, a correct Whois entry is mandatory (see point m in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• Green marks shipping platforms where every email evaluated had a Whois entry for the sender IP, which firstly contained an abuse address and secondly directly named the shipping service provider. This complies with the CSA requirements for technical configurations, point m: “The Whois details of the IP address or the associated FQDNS must clearly refer to the certified sender. In addition, the IP address and FQDN must include reachable contact email addresses as well as easy-to-find, general abuse email addresses in WHOIS (see RFC2142, RFC3013 Section 2.5 and current guidelines/best practices of RIRs and registries).”
• Dark blue indicates delivery platforms where every email evaluated had a Whois entry for the sender IP, but all or some of which did not contain an abuse address or did not directly name the shipping service provider.
• Sending platforms where only some of the evaluated emails had a Whois entry for the sender IP are marked with light blue. (This case did not occur.)
• Grey stands for sending platforms where there was no Whois entry for the sender IP for any of the evaluated emails. (This case did not occur.)
  

2. List unsubscribe

What is it about?
The list-unsubscribe link in the email header allows the recipient of an email to easily and conveniently unsubscribe from a mailing list.

What did we specifically check?
Does the email header include a list-unsubscribe link? Is it a mailto link or an HTTP link?

Why is that important?
The automatic integration of a list-unsubscribe link in the header of every email by the sending system guarantees that every recipient can unsubscribe from a company's commercial communication at any time. Some ISPs, such as Google (Gmail), automatically read the list-unsubscribe link and use it for unsubscribe links or unsubscribe buttons in the webmail interface. For CSA-certified broadcasters, a correct list unsubscribe link is mandatory (see point v in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• Sending platforms where every evaluated email had a list-unsubscribe header with a link in HTTP format are marked green. PHP links were treated as HTTP links. This complies with the CSA requirements for technical configurations, point v: “A “list-unsubscribe” link (see RFC23693) must be inserted in the email header. The links provided must make it easier to unsubscribe at least at list level. The list-unsubscribe header contains an HTTP link. The sender can send the user a confirmation email that a cancellation has been made. Exceptions to this obligation may be claimed if, for reasons relating to the design of the service and the associated sending of automated emails, it is not necessary or possible to unsubscribe as described above. The appeal office shall decide on such an exception. (Mandatory from 1.7.2011)”.
• Sending platforms are marked in dark blue, where every evaluated email had a list-unsubscribe header, which, however, only contained a mailto link.
• Light blue shows delivery platforms where only some of the evaluated emails could provide a list-unsubscribe link.
• Gray stands for delivery platforms where none of the evaluated emails had a list-unsubscribe link.
  

Category 3: “Our Should Haves”: These criteria are mandatory for CSA certified senders

The following criteria, which are mandatory for CSA-certified senders, make sense for deliverability:

1. SPF for the HELO identity of the sending mail server

What is it about?
SPF can also be used to determine which IP addresses can be used to send emails on behalf of the sending server's HELO identity. Checking the SPF HELO record is only recommended for receiving mail servers in RFC 4408 — it is not mandatory for the receiving mail servers.

What did we specifically check?
Is there a correct SPF record for the HELO identity of the sending mail server for every email? Does this SPF record end in “-all” or “~all”?

The SPF record for the exact envelope from domain (FDQN) was checked. If there was no SPF record for the subdomain mentioned in the HELO identity, but only for the corresponding main domain, this fact was assessed as an SPF record that did not exist. SPF records that contained incorrectly encoded characters were also treated as non-existent SPF records. Sender ID records were not included in our comparison.

Why is that important?
An SPF HELO check can be helpful for the receiving mail servers in special cases. For certified senders, the CSA requires an SPF HELO entry for the FQDN of the sending email server (see point s in the 'Technical Configurations” section of the CSA acceptance criteria for mass senders).

How did we mark?

• Sending platforms that had a correct SPF record for the HELO identity for each evaluated email, which ended in “-all” or “~all”, are marked green.
• Sending platforms marked in dark blue were there a correct SPF record for the HELO identity for each evaluated email, but this did not end in “-all” or “~all” (but, for example, in “? all”).
• Light blue stands for sending platforms where only some of the evaluated emails had a correct SPF record for the HELO identity.
• Sending platforms that did not have a valid SPF record for the HELO identity for any of the evaluated emails are marked gray.

2nd list ID

What is it about?
The “List-Id” field in the email header is used to identify recipient lists for mass mailings.

What did we specifically check?
Does the email header contain a populated “List-Id” field?

Sender-specific X-headers, which could also contain a campaign or list ID, were not included in this analysis. All investigated senders are CSA-certified and should therefore use the “List-Id” field — even if they also use their own X-headers with campaign or list IDs.

Why is that important?
If there are problems or complaints about an email campaign, the sender can use the “List-Id” field to quickly find out exactly which recipient group has been contacted. For CSA-certified senders, this information is mandatory. However, it can be assumed that the “List-Id” field is not checked by default by the receiving mailbox providers. A missing “List-Id” field causes problems, especially in the event of a complaint. For CSA-certified broadcasters, a correct “List-Id” header is mandatory (see point w in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• ‍ Green stands for delivery platforms where every evaluated email had a filled “List-Id” field.
• Light blue shows sending platforms where only some of the evaluated emails had a filled “List ID” field.
• Sending platforms where none of the evaluated emails had a filled “List ID” field are marked in gray.
  

3. X-CSA complaints

What is it about?
The “X-CSA Complaints” field in the email header is a special header field for CSA-certified senders that contains a complaint email address. This provides a barrier-free option for recipients of mass emails to contact the CSA or the Association of the German Internet Industry e.V. (eco) directly in the event of a complaint regarding a CSA-certified sender.

What did we specifically check?
Does the header of the reviewed email contain an “X-CSA-Complaints” field? Does the x-csa complaints field contain an eco-complaint address?

Shipper-specific X-headers, which could also contain a complaint address — e.g. the responsible complaint email address of the shipping service provider — do not meet CSA requirements. All investigated senders are CSA certified and should therefore use the “X-CSA complaints” field, regardless of whether they use another X-header field for a complaint email address.

Why is that important?
The “X-CSA Complaints” header provides a simple and easy-to-find complaint option for email recipients. This minimizes the risk that recipients of an email will click on the spam button or even request a blacklisting for an IP in the event of a complaint due to a lack of visible alternatives. For CSA-certified senders, a correct “X-CSA complaints” header is mandatory (see point x in the 'Technical Configurations' section of the CSA acceptance criteria for mass senders).

How did we mark?

• Green stands for delivery platforms where every email evaluated had an “X-CSA complaints” header with an eco-complaints address.
• Sending platforms where only some of the evaluated emails had an “X-CSA complaints” header with an eco-complaints address are marked in light blue.
• Sending platforms where none of the evaluated emails had an “X-CSA-Complaints” header with an eco-complaints address are marked gray.