What Is Automatic/Dynamic SPF Record Flattening and How It Fixes the SPF PermError: Too Many DNS Lookups Issue

Home What Is Automatic/Dynamic SPF Record Flattening and How It Fixes the SPF PermError: Too Many DNS Lookups Issue

What Is Automatic/Dynamic SPF Record Flattening and How It Fixes the SPF PermError: Too Many DNS Lookups Issue

When the SPF PermError: Too Many DNS Lookups issue strikes, your email deliverability can take a bad hit due to SPF fail. We will explain how automatic/dynamic SPF record flattening can solve this problem below.

Nowadays, more and more services are necessary to run online operations on a day-to-day basis: marketing, sales, customer support, human resources, etc. As a result, you need to include such services as SendGrid, Mailchimp, Zendesk, and more into your SPF record.

Sooner or later though, you might find that you run into the SPF PermError: Too Many DNS Lookups issue, because you've included too many services and your SPF record now consumes more than 10 DNS lookups combined.

SPF records are hierarchical

To authorize a 3rd-party service to send emails on your organization's behalf in the SPF terms, simply "include" that service in your SPF record. For example, if you use Google to deliver your emails, add Google to your SPF record:

v=spf1 include:_spf.google.com -all

From now on, your emails sent from Google's hosts will pass SPF validation, even if Google makes changes to _spf.google.com, e.g., adding more IP addresses. Such changes are transparent to you as a customer.

In other words, SPF abstracts away the underlying changes with the include mechanism.

This hierarchical nature of SPF records allows you to use a 3rd-party service by simply including that service's SPF record, without having to update your SPF record for any underlying changes in that 3rd-party SPF record.

It comes with caveats though. For those who are not too familiar with SPF, adding a seemingly simple service like below seems innocent enough:

v=spf1 include:bluehost.com -all

It looks like a perfectly valid SPF record, and your emails should gleefully pass SPF validation, right?

However, a quick check on bluehost.com reveals that this service actually contains 11 DNS lookups.

Bluehost SPF record contains 11 DNS lookups

In other words, simply including this service in your SPF record violates the SPF rule that an SPF record can contain no more than 10 DNS lookups!

This happens because the bluehost service includes other services like qualtrics.com, google.com, salesforce.com, sparkpostmail.com, and mailjet.com; and these services in turn can include even other services, pulling more DNS lookups into your SPF record.

It's a "heavy" SPF record, for sure. But in reality, it's not uncommon to come across various SPF records that contain other 3rd-party services like bluehost.com.

SPF validation works by walking through the hierarchical "tree" represented by an SPF record, and checking if any node on the tree matches the incoming IP address. Evaluating a node on the tree adds to the DNS lookup count if it's one of: a, mx, include, ptr, exists, and redirect mechanisms.

During this validation process, if the DNS lookup count goes above 10, SPF cops out and returns SPF PermError: Too Many DNS Lookups.

SPF record flattening

SPF record flattening, as the name indicates, is a process to sort of "flatten" the hierarchy in an SPF record so that the flattened record contains fewer than 10 DNS lookups. This process is sometimes called SPF record compression or SPF record optimization.

It works by walking through the SPF tree of the record, translating every tree node into one ore more IP addresses, then creating a new SPF record with these IP addresses.

For example, if an a mechanism is encountered in the record, SPF record flattening will look up the IP address of the specified host, and add the IP address to the array later used to construct the flattened SPF record.

If an mx mechanism is seen, SPF record flattening will collect all the mail servers' IP addresses of the specified host, and add them for processing later.

If an include mechanism is found, SPF record flattening will call a recursive function to flatten the child SPF record specified by the include mechanism.

Mechanism like ip4 and ip6 are added verbatim, as these don't consume any DNS lookups.

Automatic/dynamic SPF record flattening

SPF record flattening is great. However, it's static by default, meaning each time any of the 3rd-party services is changed, you'd have to run through the same SPF record flattening process and update it in the DNS. Manually. This happens all the time, and manual SPF record flattening is very tedious and DNS updating is error-prone.

Fortunately, SPF record flattening can be automated. The automated SPF record flattening process is often called automatic SPF record flattening or dynamic SPF record flattening. The "dynamic" in the name reflect the fact that the SPF record is dynamic: any change in the 3rd-party services will make it to the final SPF record.

DMARCLY's Safe SPF feature implements automatic/dynamic SPF record flattening, and it allows anyone to set this up within a couple of minutes. Once set up, you would never need to worry about violating the SPF 10+ DNS lookup issue or updating the DNS settings manually anymore!

To use Safe SPF, log in to the DMARCLY dashboard, then go to DNS Records -> Safe SPF, then follow the Safe SPF guide to set up automatic/dynamic SPF record flattening on your domain.

If you don't have a DMARCLY account yet, sign up for a 14-day free service trial.

Use Safe SPF to fix the SPF PermError: Too Many DNS Lookups issue.

How automatic/dynamic SPF record flattening fixes the SPF PermError Too Many DNS Lookups issue

Consider how SPF validation works with the new Safe SPF record. When a new email comes in on the receiving server, the server checks the return-path of the email and the return-path field is company.com.

Next the server fetches the SPF record on company.com, which is both:

has all the IP addresses from the original SPF record;
consumes fewer than 10 DNS lookups.

Now for each IP address in the Safe SPF record, SPF checks if the connecting host's IP address matches. If so, SPF validation passes. If no matching IP address is found, SPF validation fails.

To sum up, SPF validation works the same with a Safe SPF record as it does with a "vanilla" SPF record. A Safe SPF record is an SPF record after all.

Note that automatic/dynamic SPF record flattening keeps the best of both worlds: the flattened SPF record still contain the IP addresses in the original SPF tree, while keeping a DNS lookup count under 10.

Automatic/dynamic SPF record flattening FAQs

Why am I seeing Google's IPs in my flattened SPF record even I didn't include Google's SPF record?

As SPF records are hierarchical, some services in your original SPF record can contain Google's SPF record. During the SPF record flattening process, Google's SPF record was found, flattened, and added to the IP list.

For example, if your original SPF record includes bluehost (bluehost.com), your flattened SPF record will contain Google's IP addresses since bluehost.com's SPF record includes Google's.

With automatic/dynamic SPF flattening, do I need to manually update the SPF record whenever there is any underlying change?

No. Since automatic/dynamic SPF flattening happens every few minutes, any underlying change will make it to the Safe SPF record.

How do I add new services to my Safe SPF record?

Refer to this article.

Why am I seeing "Failed to get A record for domain: some-domain.com"?

If you are seeing an error like "Failed to get A record for domain: some-domain.com" while generating a Safe SPF record, chances are you have an a mechanism in your original SPF record and the domain is some-domain.com. In this case, the error message means DMARCLY was unable to resolve the hostname some-domain.com into an IPv4 address. Therefore, you can either remove the a mechanism from your original SPF record, or create an A record on some-domain.com in the DNS.

You can also run into this issue if you have an a:some-domain.com mechanism in your original SPF record. Similar to above, to fix the issue, you can either remove the a:some-domain.com mechanism from your original SPF record, or create an A record on some-domain.com in the DNS.

If you specify an mx mechanism in your original SPF record, and it resolves to some hostname like mail.some-domain.com, make sure that mail.some-domain.com resolves to an IPv4 address for the Safe SPF record to be generated successfully.

Why am I seeing "Failed to get MX record for domain: some-domain.com"?

If you are seeing an error like "Failed to get MX record for domain: some-domain.com" while generating a Safe SPF record, it means DMARCLY wasn't able to find an MX record for some-domain.com. To fix it, you can either remove the mx mechanism from your original SPF record, or create at least one MX record on some-domain.com in the DNS.