IPv6 Issues
Incident Report for Abusix Intelligence
Resolved
Summary:

Between 2020-01-22 16:00 GMT and 2020-01-23 14:20 GMT we were returning incorrect results for some IPv6 lookups made to our live query service due to a bug in rbldnsd. Rsync was unaffected.

The issue:

Yesterday afternoon around 1600 GMT we made a configuration update to our rbldnsd clusters to put several new datasets live and to put some others into test. This inadvertently caused a newer version of rbldnsd to be deployed at the same time.

We provide all of our lists free of charge to many entities that provide blacklist lookup and monitoring services, such as multirbl.valli.org, MXToolbox and DNSWL to name a few, some of these entities load our data locally via rsync and others do live queries to our rbldnsd clusters, DNSWL does the latter.

Around 1320 GMT today, we started getting support tickets and live chat requests from users that had received notifications from DNSWL that their IPv6 address was listed in our Policy blacklist (which is IPv4 only).

The solution:

We immediately began to roll back our containers to the previous version which was done within the hour and we were answering correctly by 1420 GMT.

Upon investigation, we found that there was a bug in the Trie handling in the version of rbldnsd that was deployed which caused IPv6 addresses to be returned as listed when they were not - they simply had the same bit prefix in the Trie as some IPv4 addresses that were listed.

This bug has already been fixed and is in the process of being tested by our engineers.

Lesson learned:

As a result of this, we’ve already made changes to ensure that configuration updates do not cause a different version of rbldnsd to be deployed when new containers are built.

Sorry for any inconvenience and confusion that this may have caused.

We would like to thank everyone who contacted us reporting this issue. We work hard to make sure everything runs smoothly but problems do happen and we rely on our users and community to inform us of this kind of events so that we can react as fast as possible.

We would like to reinforce our commitment to deliver the best and most reliable solutions, so please get in touch at support@abusix.ai or via the Intercom live chat at www.abusix.ai if you ever see something strange.

Kind regards,
Steve.

--
Steve Freegard
Senior Product Owner
Abusix Intelligence
Posted Jan 23, 2020 - 18:00 UTC
This incident affected: Cloud DNS.