The robots.txt file is the first resource most crawlers request and tells them which paths on your site they may or may not access. Fixing this issue means publishing a valid robots.txt at the site root so search engines, training crawlers, and answer-engine bots all have an explicit policy to follow. Without it, every crawler defaults to its own behavior, which can over-expose private sections or waste crawl budget on low-value URLs.

Example

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /

Sitemap: https://example.com/sitemap.xml

Where to make the change

• Raw HTML or static site: create a plain text file named robots.txt and upload it to the document root so it is served at https://example.com/robots.txt.
• WordPress: WordPress generates a virtual robots.txt by default. Override it with a physical file in the site root, or use your SEO plugin's robots.txt editor.
• Shopify: Shopify auto-generates robots.txt and exposes a robots.txt.liquid template you can edit from the theme code editor to add or remove rules.
• Wix or Squarespace: both platforms generate robots.txt automatically and offer limited overrides through the SEO settings panel.
• Headless or framework sites: add robots.txt to the project's public or static directory so it ships with the build.

Common causes and how to resolve them

• File is missing entirely: create one. Even an empty file with User-agent: * and no rules is preferable to a 404.
• File returns a 5xx error: some crawlers stop crawling the entire site when robots.txt is unreachable. Investigate server or CDN issues that block the path.
• Accidentally blocking the whole site: a stray Disallow: / under User-agent: * hides every URL from search. Audit the file before deploying changes from staging.
• Wrong location: robots.txt must sit at the site root, not in a subdirectory. Crawlers do not look elsewhere.

Best practices

• Reference the sitemap: add a Sitemap: line so any crawler that fetches robots.txt also discovers your XML sitemap.
• Keep it simple: a short, well-commented file is easier to audit than a sprawling one full of legacy rules. Remove directives whose original purpose is no longer relevant.
• Do not rely on robots.txt for security: the file is public, so listing private paths in it advertises them. Block sensitive paths with authentication, not just Disallow.
• Manage AI crawlers explicitly: if you want to allow or block bots like GPTBot, ClaudeBot, or PerplexityBot, add named User-agent sections for each. Defaulting to allow keeps your content eligible for citation in AI answer engines.

An XML sitemap gives search engines and AI crawlers an explicit, machine-readable list of the URLs you want indexed, along with metadata such as last-modified dates. Fixing this issue means generating a valid sitemap, publishing it at a stable URL, and pointing crawlers to it via robots.txt and Search Console. This is especially important for large sites, sites with weak internal linking, or sites where new content needs to be discovered quickly.

Example

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-04-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/getting-started</loc>
    <lastmod>2026-03-22</lastmod>
  </url>
</urlset>

Where to make the change

• Raw HTML or static site: generate the XML at build time and publish it at /sitemap.xml. Many static-site generators have a built-in sitemap option.
• WordPress: a dedicated SEO plugin will generate and update the sitemap automatically, typically at /sitemap_index.xml.
• Shopify: Shopify generates /sitemap.xml automatically and updates it as products and collections change.
• Wix or Squarespace: both platforms generate the sitemap automatically and serve it at the standard root path.
• Headless or framework sites: use the framework's built-in sitemap module (for example next-sitemap for Next.js or the @astrojs/sitemap integration) so the file is regenerated on every build.

Common causes and how to resolve them

• No sitemap published: generate one and publish it at the site root. Reference it from robots.txt with a Sitemap: line.
• Sitemap lists URLs that 404 or redirect: stale entries waste crawl budget and trigger Search Console warnings. Regenerate the sitemap from current published URLs only.
• Mixed canonical and non-canonical URLs: the sitemap should list canonical URLs only. Listing both http and https, or trailing-slash variants, splits crawl signals.
• Sitemap exceeds size limits: a single sitemap may contain at most 50,000 URLs and 50 MB uncompressed. Split larger sites into multiple sitemaps and reference them from a sitemap index file.

Best practices

• Submit to Search Console: add the sitemap URL in Google Search Console (and Bing Webmaster Tools) so the platform tracks coverage and surfaces errors.
• Keep lastmod accurate: Google uses lastmod as a hint when prioritizing crawls. Update it whenever the page content meaningfully changes; do not bump it on every build.
• Exclude noindex and disallowed URLs: the sitemap should describe what you want indexed. Listing pages that are blocked from crawling sends mixed signals.
• Use sitemap index files for multi-sitemap setups: a sitemap index aggregates child sitemaps (for example one per content type) and is the single URL you submit to search engines.

Alt attributes provide a text alternative for images, used by screen readers when a user cannot see the image and by search engines and multimodal AI models to understand what the image depicts. Fixing this issue means writing concise, descriptive alt text for every meaningful image on the page, and using empty alt for purely decorative images so assistive technology can skip them. The same alt text helps multimodal LLMs and image search tools surface and cite your visuals.

Example

<img src="/images/marathon-shoes.jpg"
     alt="Side view of a lightweight neon-green marathon running shoe">

<img src="/images/divider.svg" alt="">

Where to make the change

• Raw HTML: add the alt attribute to every <img> tag. For decorative images, use alt="" so screen readers skip them.
• WordPress: the media library has an Alt text field on every uploaded image. Fill it once and the value is reused everywhere the image is inserted.
• Shopify: in the product or page editor, click an image and use the Edit alt text option. Theme code that renders product images should pull image.alt.
• Wix or Squarespace: click the image in the editor and use the Alt text or Image description field in the side panel.
• Headless or framework sites: require an alt prop on every image component and lint for missing values during build.

Common causes and how to resolve them

• Attribute missing: add alt="" at minimum so the page validates. Replace with descriptive text on any image that conveys information.
• Alt text is the filename: "IMG_4837.jpg" tells the user nothing. Replace with a description of what is in the image.
• Alt text duplicates the visible caption: screen readers will read both, doubling the user's listening time. Differentiate the alt from the caption or empty the alt.
• Decorative images get descriptive alt text: background patterns and dividers should use alt="", otherwise screen readers narrate noise.
• Bulk imports left alt blank: a one-off audit of the media library or a CMS-side script can fill missing alt values from product names, captions, or asset descriptions.

Best practices

• Describe the function, not just the appearance: for a chart, describe what the chart shows. For a button image, describe what the button does.
• Keep it concise: aim for under 125 characters. Screen readers do not pause mid-sentence, so long alt text is exhausting to listen to.
• Skip "image of" or "picture of": the screen reader already announces it as an image.
• Include the focus keyword when natural: alt text is a minor on-page signal, but stuffing keywords in unnatural alt text harms both accessibility and ranking.

Google Analytics 4 is a free measurement platform that records visitor behavior, traffic sources, and conversion data. Installing it does not directly affect rankings, but the data it produces is essential for measuring SEO performance, identifying high-value pages, and prioritizing future optimization work. Fixing this issue means installing the GA4 tag site-wide and confirming events appear in your reports.

Example

<!-- Place inside <head> on every page -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-XXXXXXXXXX"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
  gtag('config', 'G-XXXXXXXXXX');
</script>

Where to make the change

• Raw HTML: paste the GA4 snippet into the <head> of every page. A shared layout file is the cleanest place.
• WordPress: use a site-kit or analytics plugin, or paste the snippet into your theme's header file. Avoid duplicating the tag across multiple plugins.
• Shopify: add the GA4 measurement ID under Online Store, Preferences, Google Analytics, or install the official Google channel app.
• Wix or Squarespace: both platforms offer a GA4 measurement ID field in their analytics settings panel.
• Headless or framework sites: install via Google Tag Manager or use the framework's analytics integration so the tag fires on every route change.

Common causes and how to resolve them

• Tag not installed at all: install GA4 using the snippet above or the platform-specific path.
• Universal Analytics tag still in place: Universal Analytics stopped processing data on July 1, 2024. Replace any UA tag (UA-...) with a GA4 measurement ID (G-...).
• Tag fires only on the homepage: ensure the snippet is in a shared template loaded by every page, not pasted into one post.
• Cookie banner blocks the tag before consent: integrate Consent Mode v2 so GA4 receives anonymized signals before consent and full data after.
• Tag duplicated: two snippets on the same page double-count sessions. Audit with the Tag Assistant browser extension and remove duplicates.

Best practices

• Verify with real-time reports: after installing, open GA4's Realtime view and load the site to confirm events arrive.
• Use Google Tag Manager: for any site beyond the simplest, GTM gives you a single place to manage analytics, conversion tags, and third-party scripts without code changes.
• Configure key events: mark conversions (purchase, lead, signup) as key events so reports highlight what matters for SEO ROI.
• Link to Search Console: connecting GA4 to Search Console surfaces organic queries alongside on-site behavior in the same reports.

This test fails when image responses lack caching headers that allow browsers to store them for repeat visits. With proper caching directives, returning visitors retrieve images from the local cache instead of refetching them, producing noticeably faster subsequent page loads. Fixing this issue is a server or CDN configuration change.

Where to make the change

• Server configuration: add Cache-Control headers to image responses. A common setting is Cache-Control: public, max-age=31536000, immutable for versioned assets.
• CDN configuration: Cloudflare, Fastly, CloudFront, and similar offer rules to set long cache lifetimes for image MIME types.
• WordPress and plugins: a caching plugin or a CDN integration handles cache headers without server changes.
• Shopify: Shopify serves images with appropriate cache headers automatically; failures usually point to externally hosted images.

Common causes and how to resolve them

• No Cache-Control or Expires header: add one. Browsers will revalidate every request without it.
• Cache-Control: no-cache or no-store: these are correct for HTML responses but wrong for static images. Set a long max-age instead.
• Short max-age (under a day): images change rarely; a max-age of months or a year is typical.
• Cookie-bearing image requests: if the image URL sets cookies or varies by user, browsers cannot cache effectively. Serve images from a cookie-free subdomain or your CDN.

Best practices

• Use versioned filenames: renaming the file when the content changes lets you set near-permanent cache lifetimes without ever serving stale files.
• Mark immutable assets as such: the immutable directive tells browsers to skip even revalidation requests.
• Serve from a CDN: CDNs add caching, compression, and edge delivery in one step.

This test fails when external JavaScript files are served without caching headers that allow browsers to store and reuse them. Cached scripts skip the network on repeat visits, reducing load time and bandwidth for returning users. Fixing this issue is a server or CDN configuration change.

Where to make the change

• Server configuration: add Cache-Control: public, max-age=31536000, immutable to JavaScript responses, paired with versioned filenames.
• CDN configuration: set a default cache rule for *.js assets at the edge.
• Build pipeline: emit hashed filenames (for example app.7f3c1e.js) so cache lifetimes can be long without risking stale code.
• WordPress and plugins: caching and CDN plugins handle JS cache headers automatically.

Common causes and how to resolve them

• No Cache-Control header: add one. Without it, browsers revalidate on every page load.
• Short max-age: bump to a year for versioned files. Use a shorter lifetime only for unversioned scripts that may change in place.
• no-cache or no-store on static JS: remove these directives; they belong on dynamic responses, not static assets.
• Cookies on script requests: serve scripts from a cookie-free subdomain or a CDN to enable shared caching.

Best practices

• Versioned filenames: hashed file names plus immutable caching is the gold standard for cache effectiveness.
• Self-host critical scripts: hosting third-party scripts yourself lets you control cache headers and avoid vendor outages.
• Audit with DevTools: the Network panel's "Disk cache" column shows whether each script is genuinely served from cache on repeat visits.

This test fails when external CSS files are served without caching headers that allow browsers to store and reuse them. Cached stylesheets skip the network on repeat visits, reducing load time for returning users while still allowing updates through versioned filenames. Fixing this issue is a server or CDN configuration change.

Where to make the change

• Server configuration: add Cache-Control: public, max-age=31536000, immutable to CSS responses, paired with versioned filenames.
• CDN configuration: set a default cache rule for *.css assets at the edge.
• Build pipeline: emit hashed filenames so cache lifetimes can be long without risking stale styles.
• WordPress and plugins: caching plugins handle CSS cache headers automatically.

Common causes and how to resolve them

• No Cache-Control header: add one. Without it, browsers revalidate on every page load.
• Short max-age: bump to a year for versioned files; keep it shorter only for unversioned files that may change in place.
• Stylesheets behind authentication: private styles cannot use public caching. Move shared styles to a public path.

Best practices

• Versioned filenames: hashed names plus immutable caching is the gold standard.
• Inline critical CSS: the styles needed for above-the-fold rendering can be inlined while the rest is cached and loaded asynchronously.
• Audit with DevTools: confirm that on repeat visits, the stylesheet is served from disk cache rather than the network.

This test fails when different URL variants of the site (apex versus www, HTTP versus HTTPS, with or without trailing slash) do not all resolve consistently to a single canonical version. When the same content is reachable at multiple URLs, search engines may split ranking signals across the variants. Fixing this issue means picking one canonical form for the site and 301-redirecting every other variant to it. See the Google Search Central canonicalization guide for the full specification.

Where to make the change

• Server configuration: add 301 redirects in your web server config (Nginx, Apache, IIS) or via _redirects on Netlify, vercel.json on Vercel, or Cloudflare Page Rules.
• CDN: most CDNs offer redirect rules that fire before the origin is hit, which is the cheapest way to enforce canonicalization.
• Application code: as a backstop, applications can detect the wrong host or scheme and emit a 301 redirect to the canonical URL.

Common causes and how to resolve them

• Both apex and www serve the same content: pick one (most sites prefer apex today) and 301-redirect the other to it permanently.
• HTTP and HTTPS both serve content: redirect all HTTP requests to HTTPS, then add HSTS so browsers stop trying HTTP at all.
• Trailing slash inconsistency: pick one form (with or without) and redirect the other.
• Uppercase or mixed-case URLs: redirect uppercase paths to their lowercase canonical form.
• Index page accessible at multiple paths: for example /index.html, /index.php, and / all serving the same content. Redirect everything to /.

Best practices

• Use 301 (permanent) redirects: they pass the strongest ranking signal and are cached aggressively by browsers.
• Pair with <link rel="canonical">: the canonical tag is a backup signal for cases the redirect cannot cover (parameterized URLs, syndicated content).
• Avoid chains: redirect directly to the final URL in one hop.
• Update internal links to canonical URLs: linking to non-canonical variants triggers unnecessary redirects on every visit.

This test fails when the site does not serve a branded, helpful page for missing URLs. A custom 404 keeps users on the site by offering navigation, search, and suggested links rather than dropping them at the browser's generic error screen. Fixing this issue means designing a 404 template, ensuring it returns the correct HTTP 404 status, and pointing your server at it.

Where to make the change

• Server configuration: point your web server at a custom 404 page. In Nginx, use error_page 404 /404.html;. In Apache, use ErrorDocument 404 /404.html.
• WordPress: create a 404.php template in your theme. WordPress invokes it automatically for missing URLs.
• Shopify: edit the 404.liquid template in the theme code editor.
• Wix or Squarespace: use the platform's built-in 404 page editor in site settings.
• Headless or framework sites: Next.js uses not-found.js, Astro uses 404.astro, and similar frameworks all have a convention for the missing-route page.

Common causes and how to resolve them

• No custom 404 at all: create one. The minimum is the site header, navigation, a clear "page not found" message, and links to popular sections.
• 404 page returns HTTP 200: a "soft 404" confuses search engines. Verify the response status with DevTools or curl, and configure the server to return 404 for missing routes.
• 404 page redirects to homepage: redirecting users to the homepage hides broken URLs from analytics and search engines. Show a real 404 instead.
• Generic browser error visible: the server is not invoking the custom template. Check the configuration directive.

Best practices

• Keep the site chrome: show the header, footer, and navigation so users do not feel lost.
• Offer next steps: a search box, popular links, or category navigation help users get back on track.
• Track 404s: log missing-URL hits in analytics so you can spot broken inbound links worth fixing or redirecting.
• Be clear about the error: "Sorry, that page does not exist" is better than vague messaging that confuses users.