Next.js 网站页面未在 Google 上建立索引

Question

我有一个 next.js 网站。我想为 SEO 索引网站，这样实际网站就可以在谷歌上看到。当我在谷歌控制台上请求索引时，它说由于未经授权的请求（401），该页面无法索引。

我在公共文件夹中有一个 robots.txt 文件和 sitemap.xml。

这是我的 robots.txt：

User-agent: *
Disallow:

Sitemap: https://cardcounter21.com/sitemap.xml

网站地图：

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <!-- Home Page -->
   <url>
      <loc>https://cardcounter21.com/home</loc>
      <lastmod>2024-09-08</lastmod>
      <changefreq>monthly</changefreq>
      <priority>1.0</priority>
   </url>
   
   <!-- Learn How Page -->
   <url>
      <loc>https://cardcounter21.com/learnhow</loc>
      <lastmod>2024-09-08</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.9</priority>
   </url>

   <!-- Handbook Page -->
   <url>
      <loc>https://cardcounter21.com/handbook</loc>
      <lastmod>2024-09-08</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
   </url>

   <!-- Premium Handbook Page -->
   <url>
      <loc>https://cardcounter21.com/premiumhandbook</loc>
      <lastmod>2024-09-08</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
   </url>

   <!-- Card Counter Upsell Page -->
   <url>
      <loc>https://cardcounter21.com/cardcounterupsell</loc>
      <lastmod>2024-09-08</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.7</priority>
   </url>
</urlset>

这是我的中间件.ts：

import { authMiddleware } from "@clerk/nextjs";
import { NextResponse, NextRequest } from "next/server";

const isBot = (req: NextRequest) => { 
  const userAgent = req.headers.get('user-agent') || '';
  const botUserAgents = [
    'Googlebot',
    'Bingbot',
    'Yahoo! Slurp',
    'DuckDuckBot',
    'Baiduspider',
    'YandexBot',
    'Sogou',
    'Exabot',
    'facebot',
    'ia_archiver'
  ];

  return botUserAgents.some((bot) => userAgent.includes(bot));
};

export default authMiddleware({
  afterAuth(auth, req, evt) {
    if (isBot(req)) {
      return NextResponse.next();
    }

    if (!auth.userId && !auth.isPublicRoute) {
      const home = new URL("/home", req.url);
      return NextResponse.redirect(home);
    }
  },
  publicRoutes: [
    "/sitemap",
    "/home", 
    "/learnhow",
    "/handbook", 
    "/premiumHandbook", 
    "/CardCounterUpsell",
    "/sitemap.xml", 
    "/robots.txt", 
    "/api/simulation", 
    "/Images/Logo", 
    "/test",
    "/api/webhook",
    "/api/welcomeemails",
    "/api/handbookEmails"
  ],
});

export const config = {
  matcher: ["/((?!.+\\.[\\w]+$|_next).*)", "/", "/(api|trpc)(.*)"],
};

这是错误的照片：

任何帮助将不胜感激！

谢谢！

Answer 1

robots.txt 示例：

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

参考https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt