Technical SEO

By the end of this level you will be able to take any URL that is missing from Google and say exactly which link in the chain broke. That chain has three links, and most people treat them as one: Googlebot has to be able to crawl the URL, Google has to decide to index it, and only then can the page rank. A failure at the first link is fixed on the server, in robots.txt or in your internal linking. A failure at the second one is almost never fixed by configuration.

That conflation is the single thing most people get wrong about technical SEO. Technical work gets treated as a checklist you complete once (sitemap submitted, robots.txt present, HTTPS on) when it is really a diagnostic discipline. The useful question is never “do I have a sitemap?” but “which URLs is Googlebot fetching this week, which ones has Google declined to index, and what do those have in common?”. Someone who can answer that fixes real problems. Someone ticking boxes fixes another site’s problems.

The second common mistake is assuming a technical problem announces itself with an error. It does not. The expensive failures in technical SEO are silent: a redirect chain that works perfectly for human visitors, a canonical tag pointing at the wrong URL that empties an entire section within weeks, a filter parameter generating hundreds of thousands of URLs nobody asked for. The browser never complains. Google Search Console reports it late. A crawl of your own site and your server log files are what catch it early.

This level is built to be executed, not just read. Every lesson ends in something you do: a crawl in Screaming Frog, a query against a log file, a specific check in Google Search Console, a checklist you walk before anyone touches a server.

What this level covers

• Write and audit a robots.txt that controls Googlebot and the AI crawlers without accidentally blocking resources Google needs to render your pages.
• Separate a crawling problem from an indexing problem from a quality problem, and name the Google Search Console report that proves which one you have.
• Read a full Screaming Frog crawl: status codes, canonical tags, click depth, duplicate content and redirect chains.
• Interpret Core Web Vitals against the thresholds Google documents, and decide whether speed work is worth funding on your site.
• Diagnose a JavaScript site: what Googlebot sees after rendering, and what never makes it into the served HTML.
• Plan a site migration or a URL structure change without losing rankings, using a pre-launch checklist and a post-launch verification pass.

Crawling, rendering and indexing are three separate decisions

Google makes three separate decisions about every URL, and collapsing them into one is the source of most bad diagnoses. The first is crawling: Googlebot requests the URL and the server either responds or does not. The second is rendering: for pages that depend on JavaScript, Google executes the code in a Chromium instance and produces the final HTML. The third is indexing: Google decides whether the resulting page earns a place in the index.

Each decision fails for its own reasons. Crawling fails because of robots.txt, 5xx responses, slow server response times, or because the URL is simply not linked from anywhere. Rendering fails when content depends on a request Googlebot does not execute, when hydration takes too long, or when internal links are not real anchor elements with an href attribute. Indexing fails when Google has seen the page perfectly well and concluded that it adds nothing.

That last case causes the most frustration. The “Crawled, currently not indexed” state in Google Search Console is not a technical error you clear by resubmitting a sitemap. It is a verdict. Google spent resources fetching the page and decided it does not justify the storage. Resubmitting, requesting indexing or bolting on structured data does not change that verdict. Changing it requires the page to say something a better-ranked page does not already say.

Core Web Vitals and what speed is actually worth

Speed is a modest ranking factor and a large business factor, and it is healthier to treat it that way. Google documents three Core Web Vitals with public thresholds: LCP (Largest Contentful Paint) at 2.5 seconds or less, INP (Interaction to Next Paint) at 200 milliseconds or less, and CLS (Cumulative Layout Shift) at 0.1 or less. Google assesses those values at the 75th percentile of real visits, not in an isolated lab test.

The practical consequence of that 75th percentile is that a site can feel instant on a developer’s laptop and still fail. Field data comes from real users, on real phones, on real connections. A speed audit that only looks at a local Lighthouse run is therefore incomplete: it produces a lab number, and the lab number is not the number Google uses.

The second consequence is about sequencing. If a page is not indexed, its LCP is irrelevant. If an entire section is blocked in robots.txt, its CLS is irrelevant. Speed gets optimised once crawling and indexing already work, not before. This level places performance exactly there: after the crawl and index lessons, and before any cosmetic polish.

The changes that break silently

Redirects, canonical tags and site migrations share a dangerous property: when they are done badly, the site keeps working for people. A visitor who lands on a three-hop redirect chain sees the final page and notices nothing at all. Googlebot notices, because every hop costs a request and a long chain eventually gets treated as a dead end.

Canonical tags carry the same risk profile. A template-generated canonical tag that points every product page at its category page removes those product pages from the index without producing a single browser error. The symptom arrives weeks later as a traffic drop nobody connects to last month’s deployment. The defence is a scheduled Screaming Frog crawl that compares each crawled URL against the canonical URL it declares.

Site migrations concentrate all of these risks at once, which is why they get a dedicated lesson. A change of domain, CMS or URL structure without a complete and tested redirect map is the fastest known way to lose visibility. What saves a migration is not heroics during launch weekend. It is the checklist prepared beforehand and the systematic verification afterwards.

Facets, pagination and crawl budget in ecommerce

An ecommerce site generates URLs far faster than Googlebot can crawl them, and that is the root of every crawl budget problem. A catalogue of a thousand products with five combinable filters easily produces hundreds of thousands of URL combinations, nearly all with near-identical content. Googlebot discovers them, crawls them, and spends on them the requests it is not spending on the pages that actually sell.

Crawl budget is not a universal problem, though. A two hundred page site does not have a crawl budget issue no matter how often the phrase appears in blog posts: Googlebot covers it comfortably. Crawl budget starts to matter when the number of crawlable URLs clearly exceeds the number of URLs that deserve to exist, which happens mostly in ecommerce, in classifieds portals and on any site with an indexable internal search.

The fix combines several tools from this level, and none of them works alone. Internal linking decides what gets discovered first, robots.txt decides what never gets crawled, canonical tags consolidate the duplicates, and log file analysis is the only thing that confirms whether the change worked. Server logs show exactly what Googlebot requested and when, with no sampling and no estimation.

Triage order: symptom, likely cause, where to look

A technical diagnosis runs bottom up: access first, then indexing, then quality, and speed last. This table lists the most common symptoms in the order you should rule them out, with the most likely cause and the place where you confirm it.

Order	Symptom	Likely cause	Where to look
1	URL missing from Google even with the site: operator	robots.txt block, noindex tag, or a 4xx/5xx response	URL Inspection in Google Search Console plus a Screaming Frog crawl
2	“Discovered, currently not indexed”	Googlebot knows the URL but has not prioritised it: weak internal linking or exhausted crawl budget	Page indexing report and server log files
3	“Crawled, currently not indexed”	A quality verdict from Google, duplicate or near-duplicate content	Side-by-side comparison with URLs that do get indexed
4	“Alternate page with proper canonical tag” on pages that should be indexed	Template-generated canonical tag pointing at the wrong URL	Canonical column in the Screaming Frog crawl
5	Sharp traffic drop right after a deployment	Lost redirects, changed URL structure, or a staging robots.txt pushed to production	Before and after crawl comparison, plus robots.txt in production
6	Google indexes an incomplete version of the page	Content injected by JavaScript that Googlebot does not get when rendering	Rich Results Test and the rendered HTML in URL Inspection
7	Thousands of parameter URLs in the index	Crawlable facets and an indexable internal search	Page indexing report and a list mode crawl
8	Page ranks fine but converts badly on mobile	Core Web Vitals outside the threshold in field data	Core Web Vitals report in Google Search Console

The 14 lessons in this level

1. robots.txt in 2026: The Definitive Guide (AI Bots Included)
2. XML Sitemaps: When They Matter and When They Don’t
3. Crawl Budget: Who Actually Needs to Care
4. Indexing: Why ‘Crawled — Currently Not Indexed’ Is a Quality Verdict
5. Canonicals, noindex and the Art of Not Contradicting Yourself
6. JavaScript SEO: Rendering, Hydration and What Googlebot Sees
7. Core Web Vitals 2026: LCP, INP, CLS and What They’re Really Worth
8. Redirects: 301, 302, Chains and Silent Losses
9. HTTP Status Codes for SEOs
10. Pagination, Facets and Ecommerce Crawl Hell
11. Log File Analysis: The Most Underrated Instrument You Own
12. Hreflang and Internationalization Without Breaking Things
13. Site Migrations: The Checklist That Prevents Disaster
14. Crawling Like a Professional with Screaming Frog

How to work through this level

The lesson order in this level is a diagnostic order rather than a thematic one, so follow it. The first four lessons build the mental model of crawling and indexing, and without that model the canonical and JavaScript lessons read as disconnected tricks.

• Before you start, verify your site in Google Search Console and export the page indexing report. That report is the working material for nearly every lesson here.
• Install the free version of Screaming Frog and run a full crawl of your site today. Keep that crawl as the baseline you compare against once you start making changes.
• Do one lesson per day and apply it to your own site before moving on. Two lessons read back to back with nothing executed leave nothing behind.
• Ask your hosting provider for access to raw server log files as soon as you begin the level. It usually takes a few days and you will need them for the log analysis lesson.
• Save the migration lesson for a real migration, and come back to it then with the checklist open in front of you.

Once these fourteen lessons are done, your infrastructure stops being the bottleneck and you can spend the rest of your time on content and links, which is where rankings are actually won. To see how this level fits with the others, open the full free SEO course index, where the earlier and later levels are laid out in the recommended order.