Search engine optimization (SEO) is a marketing channel many businesses invest in every year. One of the core foundations of SEO is making sure a website remains easily accessible by search engines like Google, allowing them to assess website content and display its pages in their search results for relevant user searches.

Occasionally, website owners and web developers use certain backend rules to block search engines from accessing their websites, but it’s a dangerous game as a simple mistake using these rules can quickly destroy hard earned work climbing the ranks of Google.

A simple file called the robots.txt, present on most websites, is like the gatekeeper for search engine crawlers. Depending on how the robots.txt file is configured, these crawlers are allowed through, or blocked at the door.

Oftentimes webmasters or developers will block off search engines when they are working on a new design, as they do not want these “draft websites” showing up in the likes of Google. Where people get caught out is when they publish their new designs and forget to turn off the blocking rules in the robots.txt file (or put simply they forget to reopen the door).

We just so happened to notice a rather large organization who has fallen victim to this exact issue, amidst an incredibly important time no less. The National Party is a key political party in New Zealand and currently leading the polls to secure government in the upcoming election. But around September 1st/2nd they uploaded a new design of their website and a new robot.txt file along with it. You guessed it, this robots.txt file contains the rules required to deindex a website. You can even check it out here and see the “Disallow” rule for yourself (though if this article gets around perhaps they’ll fix it, so here’s a screenshot too).

The rule tells every search engine to ignore the site.

If you want to see the changes they made when refreshing their design, you can check it out below which is a snapshot of the National website on WayBackMachine, which takes “interactive pictures” of websites allowing users to visit old content.

Old designs from August 2023:

New Designs (current):

Also captured was the exact day the new robots.txt file was uploaded (sorry whoever gets in trouble for this).

Old robots.txt file from August 30, 2023:

New robots.txt file on September 1st, 2023.

So it’s been over a month since this rule has been live on their website, but what’s the damage? We had a look at one of our SEO tools SEMrush and there is already a marked decline estimation of their traffic from search engines, from August to September once these new designs were added. This was after they had a nice increase in the lead up to August, too.

In addition, National have started to lose rankings for important keywords on Google like “nz policies” which is searched by kiwis an estimated 1,000 times per month. Previously National’s website held the number 2 position on Google (the second result when you search “nz policies” on Google). Now as you can see below they have dropped to position 7, below NZ first, ACT and the Opportunities Party, with their page unable to display information due to the blocking rule.

You can see here the gravity of such a mistake, how many people might have navigated to National’s website first had they held the number 2 spot? These users who might be on the fence with their votes, are more likely to be visiting the New Zealand First website instead. This is exactly why SEO is such an important tool for businesses. If National do not fix this issue soon it’s likely they will keep dropping in the ranks on Google at a critical time.

So how should kiwi business owners ensure they don’t fall victim to this same issue? We can’t always trust web developers will follow protocol as demonstrated by whoever is in charge of Nationals most recent design update. Luckily there are a few simple services available to mitigate this risk such as robotsalert.com. Services like this will crawl your website and your robots.txt file daily and send you a reminder if there’s ever an issue.

Ultimately, this case serves as a compelling example of why businesses, including major political organizations, need to prioritize technical SEO. And perhaps why web developers need to take the indexability of their clients websites a bit more seriously. Ask any SEO, this really does happen a lot.