Issue with backend-side ES6 imports with "type":"module" with an express/nextjs setup

[alexey-dc]

What version of Next.js are you using?

10.1.3

What version of Node.js are you using?

15.9.0

What browser are you using?

Chrome

What operating system are you using?

MacOS

How are you deploying your application?

Running locally via express

Describe the Bug

I have an Express.js server that sets up Next.js, and I want to use ES6 modules with my backend.

My package.json has the line "type": "module", which enables ES6 module support in Node.js. Everything is imported fine, but when I try to load a page, I get the following exception:

error - Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/alexey/work/alexey/temp/.next/server/pages/_document.js
require() of ES modules is not supported.
require() of /Users/alexey/work/alexey/temp/.next/server/pages/_document.js from /Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/require.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename _document.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /Users/alexey/work/alexey/temp/package.json.

    at new NodeError (node:internal/errors:329:5)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1108:13)
    at Module.load (node:internal/modules/cjs/loader:971:32)
    at Function.Module._load (node:internal/modules/cjs/loader:812:14)
    at Module.require (node:internal/modules/cjs/loader:995:19)
    at require (node:internal/modules/cjs/helpers:92:18)
    at requirePage (/Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/require.js:1:1184)
    at loadComponents (/Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/load-components.js:1:795)
    at DevServer.findPageComponents (/Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/next-server.js:77:296)
    at DevServer.renderErrorToHTML (/Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/next-server.js:139:29)
    at DevServer.renderErrorToHTML (/Users/alexey/work/alexey/temp/node_modules/next/dist/server/next-dev-server.js:35:1392)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:94:5)
    at async DevServer.renderError (/Users/alexey/work/alexey/temp/node_modules/next/dist/next-server/server/next-server.js:138:1659) {
  code: 'ERR_REQUIRE_ESM'
}

Indeed, looking at .next/server/pages/_document.js, it has a var installedModules = require('../ssr-module-cache.js'); directive, which is against the rules for "type": "module".

This seems to imply that I can not use Next.js with ES6 syntax in Node - which is too bad!

Expected Behavior

I think what I would expect is that Next.js would compile in a way that's compatible with ES6 modules - i.e. when "type": "module" is enabled, it relies on import, not require

To Reproduce

I've created a minimal setup where I'm able to get this to reproduce:

package.json

{
  "scripts": {
    "start": "node index.js"
  },
  "name": "es6_import_issue",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "author": "Alexey Chernikov",
  "dependencies": {
    "express": "^4.17.1",
    "next": "^10.1.3",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  },
  "type": "module"
}

index.js

/*
  This style works if I don't do "type": "module"
  in package.json - main.jsx loads fine!
*/
/*
const express = require('express');
const next = require('next');
const http = require('http');
*/
/*
  With this style and "type": "module" in package.json
  I get the error described
*/
import express from 'express';
import next from 'next';
import http from 'http';

class Server {
  constructor(port) {
    this.port = port;
    this.express = express();
    this.next = next({ dev: process.env.NODE_ENV !== 'production' });
  }

  async start() {
    await this.next.prepare();
    this.express.get('/', (req, res) => {
      return this.next.render(req, res, `/main`, req.query);
    })

    this.express.get('*', (req, res) => {
      return this.next.render(req, res, `/${req.path}`, req.query);
    })

    this.server = http.createServer(this.express);
    this.server.listen(this.port);
  }
}

const begin = async () => {
  const port = 3000;
  new Server(port).start();
  console.log(`Server running on port ${port}`);
};

begin();

And also in pages/main.jsx:

const hello = () => {
  return <div>
    Hello world
  </div>
}

export default hello

With this setup, after a yarn install and yarn start, I see the error above. I left the require style that works fine w/o a "type": "module" directive in comments so it's quick to test that this Express+Next.js setup is in fact functional.

[JacobLey]

So the TLDR is that NextJS does not really support native ESM right now (they should... but not yet). It is possible, but you have to hack around it in a few places.

The main issue is that NextJS compiles everything into CJS and builds each page separately. When the code is expecting ESM things start to break...

I managed to get ESM running on my custom server, so here is my checklist of things to fix to support ESM:

1. In your next.config.js manually write a CJS package.json for NextJS, see example Allow top-level ESM usage with Next projects #23277 (comment)
2. Anywhere you are importing node modules with a default export, you might have to wrap any default imports where modules have improperly defined a default export. See Issue/PR for NextJS specific case Webpack 5 does not correctly transpile .js file with "type":"module" #23029 Export defaults as named exports #23341
3. In your next.config.js you have to tweak the webpack resolver: ESM in .mjs files cause a dev mode runtime error #17806 (comment)
4. Use webpack 5 https://nextjs.org/docs/messages/webpack5
5. Use webpack externals to provide duplicates of code for every page. So nextJS will only transpile the original page, but every other file is left untouched in its native ESM form. See code snippet at end. Keep in mind I am using a typescript /dist out dir, so any relative paths might need some tweaking depending on your use case...
6. You still need to transpile your JSX, so do that ahead of time (I personally use typescript to achieve this) with jsx: react-jsx. Annoyingly React does not yet support ESM importing of this file (waiting for this to be released https://github.com/facebook/react/pull/20304/files) so you'll need to provide your own wrapper for that with jsxImportSource
7. If you use server-side styling of things like styled-components, you also need a transpiling step here. Again I use typescript plugins https://www.npmjs.com/package/typescript-plugin-styled-components with https://www.npmjs.com/package/ttypescript to work around these issues.
8. Your next.config.js cannot be ESM yet (pending Load next.config.mjs as ESM #22153) so make sure that file has a top-level package.json declaring CJS. Again I use a lerna monorepo, so my top level code is CJS, and all the packages are ESM.
9. I highly recommend enabling top-level await so pages are still ESM compatible https://webpack.js.org/configuration/experiments/

Code snippet to add to your next.config.js to prevent ESM duplicates/issues

defaultConfig.module.rules.push({
            test: {
                and: [
                    /\.js$/,
                    path => !path.includes('node_modules'),
                ],
            },
            include: ['/'],
            use: opts.defaultLoaders.babel,
        });
if (opts.isServer) {
            // By default NextJS will completely transpile Server code, but that is
            // unnecessary (NodeJS handles native ESM syntax, and Typescript handles any transforms).
            // Excluding these files results in better performance
            // (no duplicate files and less transpiling at build time)
            // and allows for usage of "unique" values like `Symbol`.

            const [, major, minor] = /^v(\d+).(\d+).\d+$/.exec(process.version);
            defaultConfig.target = `node${major}.${minor}`;
            defaultConfig.output.environment = {
                arrowFunction: true,
                bigIntLiteral: true,
                const: true,
                destructuring: true,
                dynamicImport: true,
                forOf: true,
                module: true,
            };
            defaultConfig.externals = async ({ request, contextInfo }) => {
                if (contextInfo.issuer === '') {
                    // Original set of pages, need to include these.
                    return;
                }
                if (request.startsWith('@<custom-package>/')) {
                    // custom package, must use import
                    return `import ${request}`;
                }
                if (request.startsWith('.')) {
                    if (request.startsWith('../next-server/')) {
                        // Pages that were not explicitly defined (e.g. errors)
                        // so coming from node_modules.
                        return `commonjs ${Path.join('next/dist/pages', request)}`;
                    }
                    if (/.[cm]?js$/.test(request)) {
                        // Local JS file
                        if (opts.dev) {
                            // In development bundle "local" files to best support hot reload.
                            return;
                        }
                        if (request.startsWith('..')) {
                            // Transpiled version gets placed two directories deeper
                            return `import ../../${request}`;
                        }
                        return `import ${request}`;
                    }
                    // Custom NextJS file (e.g. `.css`)
                    return;
                }
                // Some other package, safer to use require
                return `commonjs ${request}`;
            };

So NextJS with native ESM is possible! But it is a lot and probably overwhelming. So proceed at your own risk until NextJS does a better job of supporting it by default.

[brianjenkins94]

@JacobLey Is there a simpler answer now that experimental.esmExternals exists? Do you know how much of this is still necessary?

[JacobLey]

Hmm I admittedly haven't fully tested the capabilities but I have been following these features. In general I think NextJS supports ESM in node_modules, I'm not sure how much it supports your "local" files as ESM by default.

I think I mentioned it above, but it is worth noting a general caveat to my instructions is my goal is to write code that runs automatically.
For example NextJS is perfectly capable of taking typescript code .ts(x), and starting a NextJS server. However since I was using a custom server, I wanted code that was completely valid javascript .js so I had a step where I perform all transpiling myself. The result is NextJS loses some of its magic, but gained much more configurability and re-usability with my codebase, much of which is not NextJS-centric.
I don't use NextJS for linting or typescript, I simply use it to render React code on both the client and server. I have my own separately maintained linter/typescript rules that are applied to my whole monorepo. My method works, but it is heavily opinionated.

If you are someone that is planning to deploy your NextJS in more basic ways like via Vercel or next start, a "pure" NextJS app, then this level of detail is probably unnecessary for your use case. Just write Typescript with import/export syntax and don't worry about how it actually gets executed.

Anyways, in order of my points above:

1. I believe this is still necessary, as they don't expect your code to be ESM.
2. This is most likely still a problem. I know libraries like styled-components still have wonky default import behavior. Really depends on the implementing library.
3. Not sure, try omitting it and see what happens?
4. resolved by default
5. I suspect this is still an issue. My understanding of esmExternals is that it supports import of third-party (e.g. from node_modules) packages. I think NextJS still tries to bundle the "local" files and I'm not sure how well that is supported when files are ESM. I think NextJS has also tweaked the default Webpack config to better re-use bundles of code so the duplicate-code issues may not be such an issue.
6. Still necessary for react-jsx, at least if you are doing your own transpiling.
7. Yes still recommended. Again in my setup the code should already be execution ready, so webpack can't be re-writing those files with compressed/configured styled-components.
8. This should be resolved, as next.config.mjs (esm syntax) is natively supported
9. I think this might be enabled by default now?

I'll try upgrading my local version and see what changes are still necessary, and report back

[brianjenkins94]

This is exceptional, thank you for your effort!

I seem to have this mostly working but I had to remove "type": "module" from my package.json (which is frustrating but I realized from your original comment you may be doing the same (#8)) and I can't seem to get top-level await to work, but I can live without that for now.

[JacobLey]

So I upgraded to NextJS 12.0.4

I think my assumptions above are generally correct, ESM "locally" isn't natively supported, but modules are.

• I still had to use next.config.js. I tried switching to mjs or writing an ESM js and had some issues. Might be fixable but that wasn't really my focus. May investigate further.
• No need to specify esmExternals or webpack5. These are all handled natively. topLevelAwait was still required.

My "redacted" webpack config in next.config.js (takes into account notes I made above):

'use strict';

const fs = require('fs');
const Path = require('path');

let wrotePackageJson = false;

module.exports = {
    pageExtensions: ['js'],
    webpack: (defaultConfig, opts) => {

        if (!wrotePackageJson) {
            fs.mkdirSync(Path.join(opts.dir, '.next'), { recursive: true });
            fs.writeFileSync(Path.join(opts.dir, '.next/package.json'), '{ "type": "commonjs" } ');
            wrotePackageJson = true;
        }

        defaultConfig.experiments = {
            ...defaultConfig.experiments,
            topLevelAwait: true,
        };

        if (opts.dev && !opts.isServer) {
            defaultConfig.module.rules.push({
                test: /\.js$/,
                type: 'javascript/auto',
                resolve: {
                    fullySpecified: false,
                },
            });
        }

        if (opts.isServer) {

            const [, major, minor] = /^v(\d+).(\d+).\d+$/.exec(process.version);
            defaultConfig.target = `node${major}.${minor}`;
            defaultConfig.output.environment = {
                arrowFunction: true,
                bigIntLiteral: true,
                const: true,
                destructuring: true,
                dynamicImport: true,
                forOf: true,
                module: true,
            };
            defaultConfig.externals.unshift(async ({ request }) => {
                if (request.startsWith('@<custom-package>/')) {
                    // Local package (does not resolve to node_modules/**), must use import
                    return `import ${request}`;
                }
                if (request.startsWith('.') && /.[cm]?js$/.test(request)) {
                    // Local JS file
                    if (opts.dev) {
                        // In development bundle "local" files to best support hot reload.
                        return;
                    }
                    if (request.startsWith('..')) {
                        // Transpiled version gets placed two directories deeper
                        return `import ../../${request}`;
                    }
                }
            });
        }

        return defaultConfig;
    },
};

Yes I had to remove "type": "module" from my root package.json.

My directory setup looks something like:

my-app/
├─ package.json ("type": "commonjs")
├─ next.config.js (CJS syntax)
├─ my-nextjs-app/
│  ├─ package.json ("type": "module")
│  ├─ pages/
│  │  ├─ my-next-page.tsx (Typescript, ESM syntax)
│  ├─ tsconfig.json
│  ├─ dist/
│  │  ├─ pages/
│  │  │  ├─ my-next-page.js (Native ESM JS, genreated by tsc outside of NextJS)
├─ reusable-library/
│  ├─ package.json ("type": "module")
│  ├─ utils.js (ESM JS, used by Nextjs and other custom backend scripts/apps. @my-app/reusable-library/utils)

[kachkaev]

Fix: #33637 (still work-in-progress at the time of this comment)