SPF PermError: Too Many DNS Lookups

Home SPF PermError: Too Many DNS Lookups - When SPF Record Exceeds 10-DNS-Lookup Limit

SPF PermError: Too Many DNS Lookups - When SPF Record Exceeds 10-DNS-Lookup Limit

20th Feb 2024 SPF SPF PermError Too Many DNS Lookups

"SPF PermError: too many DNS lookups" is a common error seen in many SPF (Sender Policy Framework) implementations. When an often overlooked SPF 10-DNS-lookup limit is exceeded, an SPF PermError, aka SPF permanent error, is returned. SPF PermError's can affect your email deliverability.

This article explains what the SPF 10-DNS-lookup limit is, what the consequences are when an SPF record falls foul of it, and how to fix this issue using DMARCLY's Safe SPF feature.

When you set up SPF on a domain, sometimes you run into some SPF permanent error along the lines of "SPF PermError: too many DNS lookups". This can be seen on an email server with compliant SPF support, or from an online SPF record checker.

How is "SPF PermError: too many DNS lookups" interpreted by DMARC?

When "SPF PermError: too many DNS lookups" is returned during an SPF check, DMARC treats that as fail since it's a permanent error, and all SPF permanent errors are interpreted as fail by DMARC.

What is SPF DNS lookup limit?

According to the official RFC specification document RFC7208:

SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. If this number is exceeded during a check, a PermError MUST be returned. The "include", "a", "mx", "ptr", and "exists" mechanisms as well as the "redirect" modifier do count against this limit. The "all", "ip4", and "ip6" mechanisms do not require DNS lookups and therefore do not count against this limit.

In other words, the SPF specification requires that the number of mechanisms and modifiers that do DNS lookups must not exceed 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. Otherwise, an SPF PermError, more specifically "SPF PermError: too many DNS lookups", is returned.

This limit is imposed on the receiving email server side. Here are a few popular SPF software packages that implement this limit:

Why impose the SPF DNS lookup limit?

Why this seemingly artificial limit? Well, as it turns out, the 10-DNS-lookup limit is implemented to thwart Denial-of-Service (DoS) attacks. Consider such a scenario:

a malicious user creates an SPF record on domain malicious.com, with references to another domain victim.com;
he then sends a lot of emails from malicious.com to mailboxes hosted by different email service providers (ESP) with SPF implemented;
upon receiving such an email, the ESP queries the DNS for victim.com;
since many ESP's are involved, they amplify this traffic; this effectively turns into a DoS attack at victim.com;
what's more, the true source of the attack is hidden.

As you can see, a pretty innocent email authentication mechanism can be exploited for malicious use, if no care has been taken! While the consequences can be severe, the solution to this problem is simple: putting a limit on the max number of DNS lookups per check on the ESP side can drastically mitigate it, since the amplification is limited to 10, instead of potentially much larger.

Does my SPF record exceed the SPF DNS lookup limit?

You can use our SPF record lookup tool to check your SPF DNS lookup count. In addition to the basic information about your SPF setup on your domain, it also shows the number of DNS-querying mechanisms/modifiers. Here is the result of an SPF check on microsoft.com, which has exactly an SPF DNS lookup count of 10:

DMARCLY SPF record DNS queries

I suggest that you run a similar check on your domain, and see what the number looks like.

What happens if it exceeds the SPF DNS lookup limit?

When the SPF implementation on the receiving email server encounters more than 10 DNS-querying mechanisms/modifiers in the sender's domain's SPF record, it returns "SPF PermError: too many DNS lookups". As mentioned above, an SPF PermError is interpreted by DMARC as fail, and consequently, the email might not land in the inbox, depending on the email server's settings.

Therefore, your best bet is to keep the DNS-querying mechanisms/modifiers in your SPF record <= 10.

But I can't. I have so much stuff in my SPF record!

I understand - nowadays almost every company outsources essential services to 3rd-party service providers, like email delivery, marketing, and more. Putting an include for each of the services in the record counts 1 against the limit. And if they further contain DNS-querying mechanisms/modifiers, it reaches/exceeds the limit fairly quickly.

While many email service providers (ESPs) like Gmail send unauthenticated emails to spam by default, Microsoft Office 365 takes a step even further: they block email sender domains automatically if they fail email authentication, including SPF authentication.

The Antispam policy allows administrators to “Allow” domains regardless of the reputation of the domain. We’re changing our policies to not honor Allow rules when the domain fails authentication.

— Microsoft Office 365, April 2020

Learn more about this on Microsoft Office 365's roadmap.

Enter SPF record flattening

There is one simple solution to this problem though. Through "flattening" an SPF record, one can reduce the number of DNS-querying mechanisms/modifiers so that it's smaller than 10.

This is how "SPF record flattening" works: for each of the DNS-querying mechanisms/modifiers, query the DNS to get the IP addresses, then replace the original mechanism/modifier with the IP addresses. Each time a mechanism or a modifier is replaced, the total DNS lookup count is decremented by 1.

Using this SPF record flattening technique, you can turn a very complex SPF record containing well over 10 DNS-querying mechanisms/modifiers into a "flat" IP address list, staying comfortably in the "safe zone".

Let's take a look at what a flattened SPF record looks like. Here are the IP addresses by flattening the SPF record on microsoft.com:

DMARCLY flattened SPF record

As you can see, this flattened SPF record contains the same IP addresses as those in the original SPF record on microsoft.com, and yet it has no DNS-querying mechanism/modifier in itself!

Problem solved? Well, not quite yet.

What if the IP addresses underlying one of the include mechanisms are changed? That means the flattened SPF record now goes out of synchronization on these IP addresses, which will produce incorrect results in SPF authentication.

Of course, you can manually flatten the SPF record again, and update it in the DNS. Needless to say, this is terribly tedious and error-prone, not to mention you will have to monitor it all the time.

Good news is, DMARCLY has a feature called "Safe SPF", which is exactly purpose-built to save your sanity.

Safe SPF to the rescue

Kill two birds with one stone with Safe SPF: always keep your SPF record's DNS-querying mechanisms/modifiers below 10, without having to worry about manually flattening the SPF record and updating it in the DNS at all!

Follow the steps here to set up Safe SPF on your domain:

Generate Safe SPF record

In dashboard->DNS Records->Safe SPF, choose the domain you want to set up Safe SPF on, then click the Generate Safe SPF Record button, as shown below:
Publish the Safe SPF record

Now that the Safe SPF record is generated, you need to publish it on your domain in the DNS. Publish it as you would a regular SPF record. Keep in mind: a Safe SPF record is an SPF record.

Here is how to publish an SPF record.
Verify the Safe SPF record

Next you need to verify the Safe SPF record is published correctly and accessible to all. Click the Verify Safe SPF button now:
Save Safe SPF

Finally, you need to save Safe SPF so that your SPF record is always up to date:

Here is what it looks like:

As can be seen from above, Safe SPF is activated on the specified domain.

Once Safe SPF is activated on a domain:

the Safe SPF record contains the same IP addresses as those in the original SPF record;
the Safe SPF record has no more DNS-querying mechanisms/modifiers than 10;
it is always updated when the underlying IP addresses change;
there is zero maintenance on your side.

How to update an existing Safe SPF record (Method 1)

After you generate/publish a Safe SPF record, you might want to update the original SPF record at a later time.

For example, you might want to:

add a new mechanism (include, ip4, mx, a, etc.) to the SPF record;
update an existing mechanism in the SPF record;
remove an existing mechanism from the SPF record.

You can use the same Safe SPF process in the last section for this purpose. One thing to note here is that you need to update your original SPF record with your intended changes before you plug it into the Safe SPF process.

Let's walk through a practical example to show exactly how to do this.

Say your domain is: yourdomain.com, and the original SPF record on the domain looked like this:

v=spf1 mx include:someservice.com -all

you created a Safe SPF record for it in the past:

v=spf1 include:_u.yourdomain.com._spf.dmarcly.com -all

and your organization plans to integrate a new email service called anotherservice. Now you need to include it in your SPF record, so that the emails sent from that service's hosts pass SPF authentication.

Let's assume anotherservice's SPF include is:

include:anotherservice.com

you need to update your original SPF record to include this service, so that it looks like:

v=spf1 mx include:someservice.com include:anotherservice.com -all

Next, you need to go through the whole Safe SPF process with the updated original SPF record:

v=spf1 mx include:someservice.com include:anotherservice.com -all

Specifically, use the above value for the Original SPF Record field in the "Generate Safe SPF record" step in the "Safe SPF to the rescue" section. Then go through all the remaining steps in the Safe SPF process.

Once you are done, your new Safe SPF record on yourdomain.com contains all the IP addresses in include:anotherservice.com, in addition to the existing IP addresses. And if there is any underlying change in include:anotherservice.com, your Safe SPF record will pick it up automatically.

This approach applies to all scenarios including adding, replacing, and removing.

For example, if you want to replace someservice.com with anotherservice.com in your SPF record, just update it to:

v=spf1 mx include:anotherservice.com -all

then run it through the whole Safe SPF process.

In another example where you want to remove the mx mechanism from the SPF record, simply update it to:

v=spf1 include:someservice.com -all

then run it through the whole Safe SPF process.

How to update an existing Safe SPF record (Method 2)

Another way to update your existing Safe SPF record is to add the new mechanism directly to your published Safe SPF record.

Note that this approach only applies to adding an additional mechanism, rather than replacing or removing an existing one. If you want to replace or remove an existing mechanism, please use Method 1 described above.

Let's say you've published a Safe SPF record on your domain:

v=spf1 include:_u.yourdomain.com._spf.dmarcly.com -all

This record contains all the IP addresses resulted from all the mechanisms in your original SPF record.

Then you need to add a new service include:newservice.com, you can simply update the SPF record on your domain to:

v=spf1 include:_u.yourdomain.com._spf.dmarcly.com include:newservice.com -all

Now the SPF record on your domain contains all the IP addresses resulted from all the mechanisms in your original SPF record, as well as those in newservice.com. In other words, the emails sent from newservice's hosts will pass SPF authentication.

Partial Safe SPF

If some of the external includes in your SPF record are not flattenable, for example, Office 365 requires include:spf.protection.outlook.com in the top-level SPF record, you can use partial Safe SPF to solve the problem.

Basically, it's nothing more than leaving the non-flattenable SPF include out, running the rest of your original SPF record through Safe SPF, then adding that SPF include back along with the generated Safe SPF record, while publishing the SPF record on your domain.

Learn more about partial Safe SPF.