Error occur when build with next-mdx-remote and next.js standalone option

[EiffelFly]

Context

Hi!! Thank you for building and maintaining such a great package. I am encountering some issues when using codehike + next-mdx-remote + Nextjs with the standalone option.

During development, the code goes smoothly. But after building it with the Next.js standalone option, the app can't find some file that codehike relied on and throw Internal Error.

error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/app/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'] {
code-hike-test-2 |   errno: -2,
code-hike-test-2 |   code: 'ENOENT',
code-hike-test-2 |   syscall: 'open',
code-hike-test-2 |   path: '/app/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
code-hike-test-2 | }

The way to reproduce

1. Please clone this repo https://github.com/EiffelFly/code-hike-remote-docker
2. install everything and use the script npm run build
3. Activate the standalone server with node ./.next/standalone/server.js
4. goto http://localhost:3000

I have set up a docker image if this could help.

1. npm run docker:app:develop
2. goto http://localhost:3000

[thecuvii]

same with MDX-Bundler.

RequestId: eeabce19-9e2c-4811-a253-f8e2bb47d004 Error: Runtime exited with error: exit status 1
Runtime.ExitError
2022-10-02T15:50:03.592Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
}
✘ [ERROR] [plugin @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0:
0 │ # GFM
╵ ^
2022-10-02T15:50:03.651Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0: ERROR: [plugin: @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
    at failureErrorWithLog (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: [Error],
      id: '',
      location: [Object],
      notes: [],
      pluginName: '@mdx-js/esbuild',
      text: "Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'"
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: ceffc43d-627d-4def-b9ef-37d29e9be86d Error: Runtime exited with error: exit status 1
Runtime.ExitError

[pomber]

Both are using pnpm? I'll try to test it tomorrow

[thecuvii]

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

[thecuvii]

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

shikijs/shiki#138 (comment)

[thecuvii]

Above solution not works for mdx-bundler.

Basically same error with kentcdodds/mdx-bundler#158

[EiffelFly]

Both are using pnpm? I'll try to test it tomorrow

Yes i am using pnpm!

[EiffelFly]

Ok, that's a problem with shiki and next.js.
Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.
You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

shikijs/shiki#138 (comment)

This is very interesting, thanks for your help! Let me test it further

[pomber]

Above solution not works for mdx-bundler.

Basically same error with kentcdodds/mdx-bundler#158

@xWil what's the error with mdx-bundler? it may be a different issue

[thecuvii]

kentcdodds/mdx-bundler#158

✘ [ERROR] Could not resolve "@code-hike/mdx/dist/components.cjs.js"
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109:
3 │ ...eSlot, Scrollycoding} from "@code-hike/mdx/dist/components.cjs.js";
╵                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can mark the path "@code-hike/mdx/dist/components.cjs.js" as external to exclude it from the bundle, which will remove this error.
2022-10-08T12:44:21.409Z	6601f87c-e5b2-4983-8777-41a6e196398a	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109: ERROR: Could not resolve "@code-hike/mdx/dist/components.cjs.js"
    at failureErrorWithLog (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: undefined,
      id: '',
      location: [Object],
      notes: [Array],
      pluginName: '',
      text: 'Could not resolve "@code-hike/mdx/dist/components.cjs.js"'
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: 6601f87c-e5b2-4983-8777-41a6e196398a Error: Runtime exited with error: exit status 1
Runtime.ExitError

Local-dev works great, the error above was came from next.js server rendering.

[pomber]

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

[thecuvii]

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

Yeah, all pages are rendered inside getStaticProps, but not all of them are rendered at build stage.

https://nextjs.org/docs/api-reference/data-fetching/get-static-paths

[manduks]

this fixed the issue for me on getServerSideProps

// in your next.js page 
export const config = {
  unstable_includeFiles: ['node_modules/**/shiki/**/*.json'],
};