Understanding HTTP 404 Not Found: Why It Happens and What to Do About It

404 Not Found is the only HTTP status code most non-engineers can name. It's the broken-link punchline, the "this page doesn't exist" tombstone, the URL that ate your weekend. But for all its fame, 404 is full of small traps: subtle SEO mistakes, soft-404 anti-patterns, and confusion with its quieter cousins 410, 403, and 451. Let's go through what 404 actually means, when to return it, and how to ship a 404 experience that doesn't lose users (or search rankings).

What Is a 404?

A 404 means the server understood your request perfectly — it just can't find anything to serve at the URL you asked for.

The 404 (Not Found) status code indicates that the origin server did not find a current representation for the target resource or is not willing to disclose that one exists. — RFC 9110, Section 15.5.5

In plain English: either the resource genuinely doesn't exist at this URL, or the server is pretending it doesn't exist (typically for privacy or security reasons — more on that below). The RFC deliberately leaves the door open to both interpretations, which is why "404 vs 403" has been a design debate since 1996.

A 404 Is a Successful Response

This trips people up constantly. A 404 is not a network failure. It's a server response. DNS resolved. TCP connected. TLS handshook. The server received your request, parsed it, ran its routing logic, and concluded: "no such resource."

Contrast with the things that look superficially similar but aren't 404s:

If you're debugging "my site is down" and you see 404s, the site is up. The 404 is doing exactly what it was designed to do.

Common Causes

In practice, 404s come from a small set of repeat offenders:

1. Typos in the URL — /abuot instead of /about. Case sensitivity matters on most servers (/About ≠ /about).
2. Deleted pages — The content was real once; now it's gone. If the removal is permanent and you want to make that explicit, you should be returning 410 instead.
3. Stale inbound links — A high-traffic external site links to a URL you renamed three years ago. The 404 isn't your bug, but it is your problem.
4. Client-side routing mismatches — SPAs that hand every path to the bundle but quietly 200 on routes the bundle doesn't know about. Bad for users, terrible for SEO.
5. Trailing-slash differences — /blog/post and /blog/post/ are two URLs. Pick one and 301 the other.
6. Build-output drift — A deploy renames an asset hash but the HTML still references the old one. Cascading 404s on CSS/JS look like a "broken site," not a missing page.
7. Authorization confusion — A logged-out user hits /admin/users/42. Returning 404 to hide existence is sometimes the right move; defaulting to 404 because you forgot to check auth is not.

404 vs 410 vs 403 vs 451

The most common mistake with 404 is reaching for it when one of these four would be more accurate. Choose deliberately:

A useful heuristic: if the resource never existed, return 404. If it existed and is permanently gone, return 410. If it exists but the requester can't see it, return 403. If a lawyer is involved, return 451.

The privacy edge case is the wrinkle: returning 403 on /admin/users/42 confirms that user 42 exists. If that's a leak, return 404 instead — the RFC explicitly permits this. Just be consistent: returning 403 for users you have permission for and 404 for users you don't will leak existence anyway.

Returning 404 Correctly Across Platforms

Express / Node.js

app.get("/posts/:slug", async (req, res) => {
  const post = await db.posts.findBySlug(req.params.slug);
  if (!post) {
    return res.status(404).json({ error: "Post not found" });
  }
  res.json(post);
});
 
// Catch-all for unknown routes — must be registered last
app.use((req, res) => {
  res.status(404).send("Not found");
});

The catch-all is important. Without it, Express returns its default HTML error page, which leaks framework details and looks broken.

Next.js App Router

Next.js gives you two complementary primitives. notFound() throws an error that the router converts into a 404 response, and the nearest not-found.tsx boundary renders the UI:

// app/posts/[slug]/page.tsx
import { notFound } from "next/navigation";
 
export default async function PostPage({ params }) {
  const post = await getPost(params.slug);
  if (!post) notFound();
  return <article>{post.body}</article>;
}
 
// app/not-found.tsx
export default function NotFound() {
  return (
    <main>
      <h1>Page not found</h1>
      <p>The page you were looking for doesn't exist.</p>
    </main>
  );
}

This pattern preserves your layout (header, footer, search) on the 404 page automatically, and Next.js sends the correct 404 status to clients and crawlers — not a soft 404.

NGINX

server {
  error_page 404 /404.html;
 
  location = /404.html {
    root /var/www/static;
    internal;
  }
}

internal ensures users can't request /404.html directly and get a 200. The page is only served as part of an error response.

REST API JSON body

For machine consumers, return a structured body so clients can branch on a stable code:

HTTP/1.1 404 Not Found
Content-Type: application/json
 
{
  "error": "post_not_found",
  "message": "No post exists with slug 'how-to-404'.",
  "documentation": "https://api.example.com/docs/errors#post_not_found"
}

Don't return 200 with { "error": "..." }. Any decent HTTP client treats 2xx as success, and you'll have to special-case every call site forever.

404 and SEO

This is where 404 stops being a code and starts being a strategy.

Soft 404s are the cardinal sin. A "soft 404" is when your server returns 200 OK with content that says "Page not found." Google's crawlers detect this pattern and treat the URL as a 404 anyway — but they also lose trust in your status codes everywhere else, which can hurt indexing of legitimate pages. Always return an actual 404 status.

When to 301 instead of 404. If you renamed or migrated a page, 301 redirect the old URL to the new one. You keep the link equity. 404 throws it away.

When to 410 instead of 404. If a page is permanently gone and you want Google to drop it from the index fast, 410 is the right answer. Crawlers retire 410s sooner than 404s.

Keep 404s out of your sitemap. Sitemaps are a list of URLs you affirm exist. Listing dead URLs there is a credibility hit.

Don't blanket-redirect all 404s to the homepage. It looks helpful but Google explicitly calls this out as a soft-404 pattern. It also destroys your analytics — every broken link looks like homepage traffic.

For the migration story specifically, our understanding 301 redirects post goes deep on choosing between 301, 302, 404, and 410 during content moves.

Designing a 404 Page People Don't Bounce From

The default behavior of a 404 page is to lose the user. A few cheap moves dramatically improve that:

• Keep the chrome. Show your normal header, navigation, and footer. A blank "404" page feels like the whole site died.
• Offer a search box. Most lost users are looking for something — give them a way to find it without going back to Google.
• Link to the popular pages. Even three or four prominent links recover a meaningful share of visitors who'd otherwise bounce.
• Log every 404 with referrer and path. A weekly report of top 404s with non-empty referrers is the cheapest broken-link-finder in existence — most of the entries will be your own old URLs.
• Make it on-brand. The fact that a user landed on a 404 doesn't mean they want a generic apology. A page that matches your voice keeps engagement up.

Monitoring 404 Rate

Your 404 rate is a useful health signal. Normal background noise is steady; sudden spikes are almost always actionable.

Watch for:

• A deployment that just shipped. Number-one cause of acute 404 spikes is a build that renamed asset hashes without updating the references. Look at which paths spiked.
• A single new top-404 path. Means an external site or social card just linked to something that doesn't exist. Either create the page (if it should exist) or 301 to the right one.
• Bot traffic. Vulnerability scanners hit /.env, /wp-admin, and /phpmyadmin constantly. These 404s are healthy — they mean your server is correctly saying "no" to nonsense requests. Don't chase them; do filter them out of your dashboards so they don't drown out real signal.

A modest setup: log 404 rate per route, alert if any single path crosses 100 hits/hour after being near zero. That alarm will fire approximately every time you break something.

Common Pitfalls

1. Returning 200 from SPA fallback routes. If you serve index.html for every path so client-side routing works, you also need to detect unknown routes on the client and return a 404 to crawlers (Next.js's notFound() handles this; pure-SPA setups need server-side rendering or a prerender step).
2. Using 404 to hide private resources inconsistently. Either return 404 for everything the user can't see, or return 403 for everything. Mixing them leaks which IDs exist.
3. Cascading asset 404s. A 404 on /main.abc123.js looks like a single error in your logs but breaks the entire page in the browser. Treat asset 404s as severity-1 alerts.
4. Redirecting all 404s to /. Hurts SEO, breaks analytics, frustrates users who almost had the right URL.
5. Not setting Cache-Control on 404 responses. A poorly-cached 404 can survive on your CDN long after you've created the page that should live there. Either set a short max-age or invalidate aggressively.
6. Forgetting that HEAD should match GET. A HEAD /missing should also return 404. Some custom routers only register GET handlers and accidentally 405 the HEAD request.

Wrapping Up

404 is the most-shipped status code on the web, but it's underused as a design decision. The rules of thumb:

• Return real 404s — never soft 404s with a 200 body
• Use 410 for permanent deletions, 301 for moves, 403 for permission denials, 451 for legal blocks
• Keep 404 pages on-brand, with navigation and a search box
• Log 404 paths with referrers — it's the cheapest broken-link detector you'll ever own
• Treat asset 404s as urgent; treat scanner 404s as background noise
• Alert on per-path 404 spikes immediately after deploys

For more, see our pages on 404 Not Found, 410 Gone, 403 Forbidden, and 451 Unavailable for Legal Reasons. And if your 404s are showing up because of a content migration, the understanding 301 redirects post walks through how to preserve link equity instead of bleeding it away.