Skip to content

Crawler does not crawl absolute URIs even if they are on prerender.origin #12271

@LorisSigrist

Description

@LorisSigrist

Describe the bug

During Prerendering the Crawler is supposed to discover all internal links & follow them. However, if an internal link is on an absolute URI (https://my-site.com) or a Network-Path-refernece (//my-site.com) it will not be crawled, even if the origin is the same as prerender.origin.

I would expect these links to be crawled as well, assuming they are on the base path.

Example:

  • prerender.origin = "https://my-page.com"
  • base = "/base"
  • href = "https://my-page.com/base/some-page" (should cause /some-page to be crawled)

Reproduction

https://github.com/LorisSigrist/sveltekit-absolute-uri-crawling-repro (instructions in README)

Logs

No response

System Info

System:
    OS: macOS 14.4.1
    CPU: (14) arm64 Apple M3 Max
    Memory: 1.02 GB / 36.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 21.7.3 - /opt/homebrew/bin/node
    npm: 10.5.0 - /opt/homebrew/bin/npm
    pnpm: 9.0.6 - /opt/homebrew/bin/pnpm
    bun: 1.1.8 - /opt/homebrew/bin/bun
  Browsers:
    Safari: 17.4.1
  npmPackages:
    @sveltejs/adapter-static: ^3.0.0 => 3.0.1 
    @sveltejs/kit: ^2.4.3 => 2.5.7 
    @sveltejs/vite-plugin-svelte: ^3.0.2 => 3.1.0 
    svelte: ^4.2.7 => 4.2.15 
    vite: ^5.2.6 => 4.5.2

Severity

serious, but I can work around it

Additional Information

<link rel="alternate" links MUST be absolute URIs according to the spec.

This issue makes it impossible for the crawler to discover alternate language versions of pages if they are only linked via alternate tags.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions