Case Study · SEO Audit · Server Diagnosis · NDA

// case_001 · three_crawl_verification Drei Crawls. Ein curl-Test. Drei Jahre Sichtbarkeit, die niemand bemerkt hat.

I found a server bug that only affects Googlebot, not humans. Three standard SEO audits had previously flagged it as “138 dead URLs” and recommended 301 redirects. We reproduced it, cross-checked it with curl, and identified the true cause: a server concurrency issue that has been quietly eating away at rankings for three years.

3Crawls

more than 8 days for reproduction

138 → 0

5xx error during curl cross-check

Hypotheses, prioritized

30Min

Developer Time Quick Fix

Background

Eine Sichtbarkeit, die seit Jahren sinkt. Eine Website, die einwandfrei aussieht.

The client agency was facing a long-standing problem: an international B2B corporation had been steadily losing visibility for three years. Three previous external audits had provided content recommendations, reviewed backlinks, and measured performance. Yet visibility continued to decline.

The Glitch in the Plan

The first Screaming Frog crawl reported 138 URLs with a 502 (Bad Gateway) status. In any standard audit workflow, that’s a clear sign: these URLs are dead—set up 301 redirects, and you’re done. Three providers before us had recommended exactly that. No one had ever checked whether the URLs were actually dead.

What we found

A technically well-maintained corporate website with over 5,000 URLs, built to be multilingual, and featuring clean internal linking. Hosted by an established provider, with monitoring in good standing. This is exactly the kind of situation where standard audits fall short, because the issues only arise under crawl load.

Three paths, one decision

301 setzen, eskalieren oder noch einmal hingucken.

There were three options on the table. We didn’t decide on them for the agency, but rather together with the agency, based on risk, cost, and long-term impact.

Option A · Apply

Set up 301 redirects

Standard procedure for all three pre-service providers: mark 138 URLs as dead, create a redirect list, and bill the client.

Risk The cause remains unknown

Time 2 days

Costs low

Episode Visibility continues to decline

Option B · Escalate

Call your hosting provider right away

Assumption: "The server is having an issue." Open a ticket; the hosting provider asks for detailed steps to reproduce the issue. Without a diagnosis, it turns into a long back-and-forth.

Risk The host doesn't see anything

Time weeks

Costs high (escalation loops)

Episode The blame game

Option C · Diagnosis

Three-Crawl Verification

Three crawls over 8 days using different configurations, plus a manual curl cross-check on the same list of URLs. Diagnose first, then take action.

Risk Performed well in testing

Time 8-day diagnosis

Costs Part of Option B

Episode Cause + Fix: Path is in use

8 days · Step by step

Was zwischen „138 Tote" und „Concurrency-Bug" passiert ist.

Four phases, documented and reproducible. From the initial standard crawl to the validated hypothesis, every step can be traced individually in the repository.

Phase 01

Standard Crawl & Findings

First Screaming Frog crawl using the default configuration (5 threads). Result: 138 URLs with a 502 status. URL list exported.

Day 1

Phase 02

Reproduction + Curl-Cross Test

Identical crawl 8 days later: again 138 × 502, identical URL list. Parallel curl loop in sequence: 0 × 502, all 200/302/404. URLs are not dead.

Day 8

Phase 03

Concurrency Verification

Third crawl with an expanded set (417 URLs instead of 360): 40 × 502, same pattern, significantly fewer in absolute terms. Errors correlate with the number of parallel requests.

Late on Day 8

Phase 04

Hypotheses + Presentation

7 technical hypotheses were formulated and prioritized. Top hypothesis: PHP-FPM worker pool exhaustion. The findings were handed over to the agency along with a section on server diagnostics in the audit report.

Day 9

Stack & Tools

Was im Werkzeugkasten lag.

Crawler

Screaming Frog SEO Spider

3 Crawls über 8 Tage
5 Threads default vs. 2 Threads Smoke
360 / 360 / 417 URLs Set-Größe
Status-Code-Vergleich pro URL-Set

Cross-check

curl + bash

Sequenzielle Loops, 1 Req/sec
Identische URL-Liste wie Crawl
HTTP-Status-Code-Logging

Database

GSC + Server Logs

Crawl-Frequenz über 90 Tage
5xx-Korrelation mit Bot-Traffic
Cross-Reference URL-Set

What customers say

A voice from the constituency.

Three audits earlier had led us to believe that the 138 URLs were dead. It wasn’t until we performed the three-crawl verification that we realized the real problem wasn’t the URLs, but our hosting configuration. Armed with this insight, we were finally able to approach the host with a clear plan of action, rather than groping around in the dark.

White-Label-Auftraggeber Advertising Agency · Fair Use · Privacy Policy in Effect

Covered by an NDA

Three patterns that make this case reproducible.

This diagnosis isn't an isolated case. It's a pattern you can apply to any website whose visibility is declining for no apparent reason.

01

Reproduce the issue before redirecting. A single crawl is never enough to diagnose a 5xx error.

02

Compare sequential and parallel testing. A curl loop with one request per second provides the most accurate differential diagnosis.

03

List hypotheses, don't guess. If you list all possible causes and rule them out one by one, you present a finding rather than a guess.

Frequently Asked Questions

Visibility dropping for no apparent reason? Let's monitor it for 30 minutes.

If your reports show 138 supposedly dead URLs or your crawl frequency drops for no apparent reason, a three-crawl verification over 8 days can provide clarity. The first phone call is free, and an NDA is standard.

Request an audit White-label mandate

// further reading

Marketing Architecture

SEO

CMS

Du brauchst eine neue Website?

White Label SEO

Du brauchst eine neue Website?

// case_001 · three_crawl_verification Drei Crawls. Ein curl-Test. Drei Jahre Sichtbarkeit, die niemand bemerkt hat.