No newsletter for René Fön
UTF8 has long made it possible to display umlauts and special characters in mail content. But what does it look like in the address line? Our test revealed: Freemailers and email marketing platforms remain conservative when processing email addresses.
René Fön sets about his first task. As a new (fictitious) employee, he not only received a lanyard for his office keys on his first day of work, but also an email address: réne.fön@publicare.de. Fully equipped, he should now test whether a B2C mailing campaign is also correctly displayed on free mailers. “The first thing you do is create an account with common freemail providers,” says the email from his team leader.
As soon as the young employee has started to register, he gets a cold sweat on his forehead. No matter how often René enters his preferred address “rené.fön”, GMX and Web.de turn it into “rene.foen” every time.
Modified variants of the name René Fön are also not accepted: None of the freemail providers from T-Online to Gmail to Mail.de allows “rene.foen”, “rené.fon” or “rené.foen” as part of the email address. René has no choice but to register with the free providers with a username without special characters.
Only 7-bit ASCII in the local part
A single name or a combination of last name and first name with a period or hyphen in the middle is commonly known as a username. From the IETF's point of view, this is the “local part,” whose handling is defined in RFC 6531 and RFC 5321.
Are UTF8-encoded special characters really not allowed in the local part? The RFC 6531 published in 2012 is vague here. In principle, non-ASCII characters, i.e. German and all other special characters, are allowed in the entire mail header. However, the parsing of the local part currently remains unchanged.
“Although the characters in the are permitted to contain non-ASCII characters, the actual parsing of the and the delimiters used are unchanged from the base email specification [RFC5321]. Any domain name to be looked up in the DNS MUST conform to and be processed as specified for Internationalizing Domain Names in Applications (IDNA) [RFC5890].”
Section 4.1.3 of RFC 5321 explains that non-ASCII characters are simply not allowed in mailbox names and mail commands. This means: In principle, UTF8 is allowed in the local part, but every mail system is free to decide whether and when it supports more than the character size defined in RFC 5321.
Nevertheless, there is at least one mail server on the market that parses SMTPUTF8 in the complete recipient address: Postfix from version 3.0.0.
Newsletter subscription with umlauts
Now René is curious and wants to know whether it's because of the Freemail providers that he can't use special characters in his username. He knows that umlauts have been allowed in domain names for a long time. The first German umlaut domain “Öko.de” was registered in 2004. So he creates his own umlaut email domain and takes his test to the extreme by also using an umlaut in the domain part of the email address. René sets up the following three addresses:
1.rené.fön@begrüßung.de
2. rene.fon@begrüßung.de
3. xn--ren-dma.xn--fn-fka@begrüßung.de
With these three addresses, René now wants to sign up for various newsletters in order to find out how various major email marketing service providers handle the circulating addresses.
Can he register as a subscriber using the addresses of major German email service providers? And if so, are emails also delivered to these addresses?
As can be deduced from the results in the first section, the first email address has little chance of success. However, variants 2 and 3 should be accepted and processed.
Address variant 3 is Punycode. The coding process makes Unicode strings compatible with ASCII again. It was designed back in 2003 to handle internationalized domain names and should be generally known by now.
Only Agnitas and Inxmail accepted René Fön's email address. The others reject all three address variants due to “invalid characters.”
Punycode: Developers on duty
The fact that even Punycode often fails validation can be explained with the double hyphen. The central element of Punycode is the string xn--, which is virtually absent in real names and words. Unfortunately, the two consecutive hyphens are so obscure that, for example, Web.de does not allow the string in user names.
Especially when building extensive systems, developers often rely on utilities to validate email addresses. However, these libraries don't spend long on compatibility issues. The email address validation of the PHP filter_var function is even notorious for its mysteriousness.
Instead of rejecting special characters and Punycode, it seems more sustainable if the processing applications always store every email address encoded as Punycode and deliver it decoded.
Conclusion: Be safe with special characters
With René Fön, the joy of successfully subscribing to the Agnitas and Inxmail newsletters only lasted until the next date the newsletter was sent. Because although his e-mail addresses passed the registration process without an error message, René Fön did not receive a single newsletter.
Browsers, email clients and mail servers are still in operation that do not pay attention to internationalized email addresses or reject them in principle. Email marketing service providers that use such conservative systems minimize risks, but at the same time avoid customers from Germany and abroad.
In the end, René Fön has decided to only use ASCII-compliant addresses until the handling of internationalized email addresses becomes established. However, he did not delete his addresses, but set up redirects. Who knows, he might still receive one of the subscribed newsletters at some point.
