Plugin not respecting `include` / `exclude` when generating declarations

[aapoalas]

What happens and why it is wrong

I have a directory within which various entry files are sought based on their file paths. These entry files are then one by one (not as an array of input files) built using Rollup. As some of these files may be TS files or may import TS files, I wanted to use rollup-plugin-typescript2 to type check and build these files.

However, after adding the plugin to the rollup config with only .ts and .tsx files included, build times jump massively. At the same time JavaScript files which import eg. lodash with the intention that Rollup will warn me of these modules being taken as external since I'm not using rollup-plugin-node-resolve (intentional, again) start appearing in the bundling and give errors for being CJS and thus not exporting anything. That is to say that clearly this plugin is resolving those imports even in normal JavaScript files and happily transpiling or at least analyzing these JavaScript files for types. This despite me explicitly only including TS files and explicitly excluding non-TS files.

Versions

• typescript: 3.4.1
• rollup: 1.15.1
• rollup-plugin-typescript2: 0.21.2

rollup.config.js

const EXTERNALS = ["known-external-module", "another-known-external-module"];
const inputOptions = {
    external: moduleId =>
        moduleId.startsWith("custom-loader!") ||
        (isTestFile &&
            !(
                moduleId.startsWith("./") || moduleId.startsWith("../") // Test files are expected to contain external modules that will later be resolved by a webpack test build
            )) ||
        EXTERNALS.includes(moduleId),
    input, // input file path
    onwarn: (warning, warn) => {
        warning.message = `${input}: ${warning.message}`;
        warn(warning);
    },
    plugins: [
        rollupPluginReplace({
            delimiters: ["<%= ", " %>"],
            values, // Doing some string replacement
        }),
        rollupPluginTypescript2({
            include: ["*.ts", "**/*.ts", "*.tsx", "**/*.tsx"], // Only run the plugin on TS files
            exclude: ["!*.ts", "!**/*.ts", "!*.tsx", "!**/*.tsx"], // Seriously, don't run it for any other files
        }),
        rollupPluginBabel({
            include: ["*.jsx", "**/*.jsx", "*.tsx", "**/*.tsx"],
            plugins: [[babelPluginJSX, { useBuiltIns: true }]], // Just transpiling JSX and TSX
        }),
        customImportCSSPlugin(), // Include CSS files as text
        rollupPluginHTML(), // Include HTML files as text
        rollupPluginJSON(), // Include JSON files as JS objects
        PRODUCTION && // Minify if necessary
            terser({
                sourcemap: false,
            }),
    ],
};
const outputOptions = {
    amd: {
        id: dataObject[input],
    },
    esModule: false,
    file: output.replace(/\.(jsx|tsx?)$/, ".js"), // Write output as normal JS
    format: isTestFile ? "esm" : "amd", // Yes yes, I know, AMD *sigh*
    interop: false,
    preferConst: true,
};

tsconfig.json

{
    "compilerOptions": {
        "baseUrl": ".",
        "sourceMap": true, // allow sourcemap support
        "strictNullChecks": true, // enable strict null checks as a best practice
        "module": "esnext",
        "target": "es6", // specify ECMAScript target version
        "allowJs": true, // allow a partial TypeScript and JavaScript codebase
        "moduleResolution": "node",
        "traceResolution": false,
        "jsx": "react",
        "lib": ["dom", "es2017", "esnext"],
        "resolveJsonModule": true,
        "paths": {
            "*": ["./node_modules/@types/*", "*"]
        }
    },
    "exclude": ["bin", "build", "node_modules"]
}

plugin output with verbosity 3

log:

rpt2: built-in options overrides: {
    "noEmitHelpers": false,
    "importHelpers": true,
    "noResolve": false,
    "noEmit": false,
    "inlineSourceMap": false,
    "outDir": "/path/.rpt2_cache/placeholder",
    "moduleResolution": 2,
    "allowNonTsExtensions": true
}
rpt2: parsed tsconfig: {
    "options": {
        "baseUrl": "/path",
        "sourceMap": true,
        "strictNullChecks": true,
        "module": 6,
        "target": 2,
        "allowJs": true,
        "moduleResolution": 2,
        "traceResolution": false,
        "jsx": 2,
        "lib": [
            "lib.dom.d.ts",
            "lib.es2017.d.ts",
            "lib.esnext.d.ts"
        ],
        "resolveJsonModule": true,
        "paths": {
            "*": [
                "./node_modules/@types/*",
                "*"
            ]
        },
        "configFilePath": "/path/tsconfig.json",
        "noEmitHelpers": false,
        "importHelpers": true,
        "noResolve": false,
        "noEmit": false,
        "inlineSourceMap": false,
        "outDir": "/path/.rpt2_cache/placeholder",
        "allowNonTsExtensions": true
    },
    "fileNames": [
        ...
    ],
    "typeAcquisition": {
        "enable": false,
        "include": [],
        "exclude": []
    },
    "raw": {
        "compilerOptions": {
            "baseUrl": ".",
            "sourceMap": true,
            "strictNullChecks": true,
            "module": "esnext",
            "target": "es6",
            "allowJs": true,
            "moduleResolution": "node",
            "traceResolution": false,
            "jsx": "react",
            "lib": [
                "dom",
                "es2017",
                "esnext"
            ],
            "resolveJsonModule": true,
            "paths": {
                "*": [
                    "./node_modules/@types/*",
                    "*"
                ]
            }
        },
        "exclude": [
            "bin",
            "build",
            "node_modules"
        ],
        "compileOnSave": false
    },
    "errors": [],
    "wildcardDirectories": {
        "/path": 1
    },
    "compileOnSave": false,
    "configFileSpecs": {
        "includeSpecs": [
            "**/*"
        ],
        "excludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "validatedIncludeSpecs": [
            "**/*"
        ],
        "validatedExcludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "wildcardDirectories": {
            "/path": 1
        }
    }
}

[ezolenko]

On a run with verbosity > 1, do you see plugin printing rpt2: transpiling <file> entries for the files you don't want processed?

Those entries are for all root files that are getting transpiled. They are controlled by both include/exclude in plugin options and include/exclude in tsconfig.

Build process looks something like that:

• rollup starts with input file and adds it to its module list
• it then passes all modules in the list through all plugins
• rpt2 uses filter based on include/exclude in rollup config and transpiles modules that pass it
• rpt2 feeds transpiled js back to rollup
• rollup looks at the transpiled js for more imports to add to the module list and starts the process all over again
• at the end, rpt2 looks at all files that it transpiled and that was found by typescript based on tsconfig and generates declaration for them (if declaration: true is set in tsconfig)

Files found through tsconfig should show up in fileNames array in rpt2: parsed tsconfig: part of output (you removed them in the log above)

    "fileNames": [
        ...
    ],

Since you have allowJs: true in tsconfig, typescript will attempt to import and process any js imports starting from any of those root files or their dependencies. If you disable that you'll get less processing.

You are also likely being hit by #148 (Compilation very slow with TS v3.4)

[aapoalas]

Hmm, okay so seems likely that some combination of tsconfig and rpt2 handling is causing the extra work. The fileNames array contains, for me, every single javascript file in the repository, not counting the actual build directory where the build is happening but that is understandable since there is the single "*" path mentioned. That is some 352 JS files.

verbosity === 2 gives me the following kind of log:

rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: ambient types changed, redoing all semantic diagnostics
rpt2: rolling caches

ad nauseum. Interspeded there is a single "transpiling" message which is expected since a single TS entry file is included in this repository.

So it seems that the problem is that rpt2 finds all of my JavaScript based on the "*" rule and then proceeds to create declarations for every single file on every single build. Or at least it does something quite time consuming on every single build.

I guess this sort of begs the question if declaration generation for all files from tsconfig is necessary? Or should the paths only be used to guide imports found from transpiled / processed modules?

EDIT: Removing the "*" rule did not change the slowness so I guess the fileNames is just based on include / exclude in tsconfig.

[aapoalas]

Switching to allowJS: false does speed up the buillds but not by a whole lot. And even then, each and every build combs through the include / non-exclude paths to find files.

I'm relatively sure this is not really how the plugin should work. If this were TypeScript working inside eg. Visual Studio Code then I'd expect that every file gets cataloged and their declaration generated (although even that would probably be done at a lower priority if they're not currently open / imported by an open file). However, with Rollup I don't think TS (or the plugin, mostly) should care about every file that is available in the system. It should only concern itself with those files that it receives from Rollup as well as modules imported from that file. And of course here the tsconfig includes / excludes and paths should come into play.

Or at the very least I don't think that the plugin should be combing through the available modules if it is not going to do anything anyway.

[ezolenko]

After you set allowJS: false, do you still see all those js files in your fileNames array in parsed tsconfig?

Plugin will print 'transpiling' entry for every file it actually transpiles, and it will print 'writing declarations` entry for every declaration it generates (you will have to up verbosity to 3 to see those though).

If it doesn't print anything then it doesn't process that file directly. Typescript might be looking at those files internally if they get imported, but that's out of my control (but will be minimized by allowJs: false.

[ezolenko]

Are you running it in watch mode btw? Why does it do 8 builds in that log you posted above?

[aapoalas]

allowJS: false did indeed remove the files from the fileNames array and speeds up the builds. It comes, however, at the quite debilitating cost of not being able to write partially TS code bases.

Watch mode is not used. The reason for the multitude of builds is mentioned in the original comment: The build system in question consists of tens of individual files which are built one by one using rollup. The 8 outputs you see are each one of those build steps.

Here is an example of what I believe to be just a single file being built:

log:

rpt2: built-in options overrides: {
    "noEmitHelpers": false,
    "importHelpers": true,
    "noResolve": false,
    "noEmit": false,
    "inlineSourceMap": false,
    "outDir": "/my/build/path/.rpt2_cache/placeholder",
    "moduleResolution": 2,
    "allowNonTsExtensions": true
}
rpt2: parsed tsconfig: {
    "options": {
        "baseUrl": "/my/build/path",
        "sourceMap": true,
        "strictNullChecks": true,
        "module": 6,
        "target": 7,
        "allowJs": true,
        "moduleResolution": 2,
        "traceResolution": false,
        "jsx": 2,
        "lib": [
            "lib.dom.d.ts",
            "lib.es2017.d.ts",
            "lib.esnext.d.ts"
        ],
        "resolveJsonModule": true,
        "paths": {
            "tests/*": [
                "tests/*"
            ],
            "library/*": [
                "node_modules/@my-company/library/*"
            ],
            "some/library/path": [
                "node_modules/@my-company/library/real/library/path"
            ],
            "*": [
                "./node_modules/@types/*",
                "*"
            ]
        },
        "configFilePath": "/my/build/path/tsconfig.json",
        "noEmitHelpers": false,
        "importHelpers": true,
        "noResolve": false,
        "noEmit": false,
        "inlineSourceMap": false,
        "outDir": "/my/build/path/.rpt2_cache/placeholder",
        "allowNonTsExtensions": true
    },
    "fileNames": [
        ...300ish files...
    ],
    "typeAcquisition": {
        "enable": false,
        "include": [],
        "exclude": []
    },
    "raw": {
        "compilerOptions": {
            "baseUrl": ".",
            "sourceMap": true,
            "strictNullChecks": true,
            "module": "esnext",
            "target": "esnext",
            "allowJs": true,
            "moduleResolution": "node",
            "traceResolution": false,
            "jsx": "react",
            "lib": [
                "dom",
                "es2017",
                "esnext"
            ],
            "resolveJsonModule": true,
            "paths": {
                "tests/*": [
                    "tests/*"
                ],
                "library/*": [
                    "node_modules/@my-company/library/*"
                ],
                "some/library/path": [
                    "node_modules/@my-company/library/real/library/path"
                ],
                "*": [
                    "./node_modules/@types/*",
                    "*"
                ]
            }
        },
        "exclude": [
            "bin",
            "build",
            "node_modules"
        ],
        "compileOnSave": false
    },
    "errors": [],
    "wildcardDirectories": {
        "/my/build/path": 1
    },
    "compileOnSave": false,
    "configFileSpecs": {
        "includeSpecs": [
            "**/*"
        ],
        "excludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "compileOnSave": false
    },
    "errors": [],
    "wildcardDirectories": {
        "/my/build/path": 1
    },
    "compileOnSave": false,
    "configFileSpecs": {
        "includeSpecs": [
            "**/*"
        ],
        "excludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "validatedIncludeSpecs": [
            "**/*"
        ],
        "validatedExcludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "wildcardDirectories": {
            "/my/build/path": 1
        }
    }
}
rpt2: typescript version: 3.4.1
rpt2: tslib version: 1.9.3
rpt2: rollup version: 1.15.1
rpt2: rollup-plugin-typescript2 version: 0.21.2
rpt2: plugin options:
{
    "verbosity": 3,
    "check": true,
    "clean": false,
    "cacheRoot": "/my/build/path/.rpt2_cache",
    "include": [
        "*.ts+(|x)",
        "**/*.ts+(|x)"
    ],
    "exclude": [
        "*.d.ts",
        "**/*.d.ts"
    ],
    "abortOnError": true,
    "rollupCommonJSResolveHack": false,
    "useTsconfigDeclarationDir": false,
    "tsconfigOverride": {},
    "transformers": [],
    "tsconfigDefaults": {},
    "objectHashIgnoreUnknownHack": false,
    "typescript": "version 3.4.1"
}
rpt2: rollup config:
{
    "chunkGroupingSize": 5000,
    "experimentalCacheExpiry": 10,
    "inlineDynamicImports": false,
    "input": "module/path/entry/index.js",
    "perf": false,
    "plugins": [
        {
            "name": "replace"
        },
        {
            "name": "rpt2"
        },
        {
            "name": "babel"
        },
        {
            "name": "inject-css"
        },
        {
            "name": "html"
        },
        {
            "name": "json"
        }
    ]
}
rpt2: tsconfig path: /my/build/path/tsconfig.json
rpt2: included:
[
    "*.ts+(|x)",
    "**/*.ts+(|x)"
]
rpt2: excluded:
[
    "*.d.ts",
    "**/*.d.ts"
]
rpt2: resolving '../src/file1'
rpt2:     to '/my/build/path/module/path/src/file1.js'
rpt2: resolving '../src/file2'
rpt2:     to '/my/build/path/module/path/src/file2.js'
rpt2: resolving '../src/file3'
rpt2:     to '/my/build/path/module/path/src/file3.js'
rpt2: resolving '../src/file4'
rpt2:     to '/my/build/path/module/path/src/file4.js'
rpt2: resolving '../src/file5'
rpt2:     to '/my/build/path/module/path/src/file5.js'
rpt2: generating target 1
rpt2: Ambient types:
rpt2:     /my/build/path/node_modules/@types/events/index.d.ts
rpt2:     /my/build/path/node_modules/@types/glob/index.d.ts
rpt2:     /my/build/path/node_modules/@types/minimatch/index.d.ts
rpt2:     /my/build/path/node_modules/@types/node/ts3.2/index.d.ts
rpt2: rolling caches
rpt2: built-in options overrides: {
    "noEmitHelpers": false,
    "importHelpers": true,
    "noResolve": false,
    "noEmit": false,
    "inlineSourceMap": false,
    "outDir": "/my/build/path/.rpt2_cache/placeholder",
    "moduleResolution": 2,
    "allowNonTsExtensions": true
}
rpt2: parsed tsconfig: {
    "options": {
        "baseUrl": "/my/build/path",
        "sourceMap": true,
        "strictNullChecks": true,
        "module": 6,
        "target": 7,
        "allowJs": true,
        "moduleResolution": 2,
        "traceResolution": false,
        "jsx": 2,
        "lib": [
            "lib.dom.d.ts",
            "lib.es2017.d.ts",
            "lib.esnext.d.ts"
        ],
"resolveJsonModule": true,
        "paths": {
            "tests/*": [
                "tests/*"
            ],
            "library/*": [
                "node_modules/@my-company/library/*"
            ],
            "some/library/path": [
                "node_modules/@my-company/library/real/library/path"
            ],
            "*": [
                "./node_modules/@types/*",
                "*"
            ]
        },
        "configFilePath": "/my/build/path/tsconfig.json",
        "noEmitHelpers": false,
        "importHelpers": true,
        "noResolve": false,
        "noEmit": false,
        "inlineSourceMap": false,
        "outDir": "/my/build/path/.rpt2_cache/placeholder",
        "allowNonTsExtensions": true
    },
    "fileNames": [
        ...here again...
        ],
    "typeAcquisition": {
        "enable": false,
        "include": [],
        "exclude": []
    },
    "raw": {
        "compilerOptions": {
            "baseUrl": ".",
            "sourceMap": true,
            "strictNullChecks": true,
            "module": "esnext",
            "target": "esnext",
            "allowJs": true,
            "moduleResolution": "node",
            "traceResolution": false,
            "jsx": "react",
            "lib": [
                "dom",
                "es2017",
                "esnext"
            ],
            "resolveJsonModule": true,
            "paths": {
                "tests/*": [
                    "tests/*"
                ],
                "library/*": [
                    "node_modules/@my-company/library/*"
                ],
                "some/library/path": [
                    "node_modules/@my-company/library/real/library/path"
                ],
                "*": [
                    "./node_modules/@types/*",
                    "*"
                ]
            }
        },
        "exclude": [
            "bin",
            "build",
            "node_modules"
        ],
        "compileOnSave": false
    },
    "errors": [],
    "wildcardDirectories": {
        "/my/build/path": 1
    },
"compileOnSave": false,
    "configFileSpecs": {
        "includeSpecs": [
            "**/*"
        ],
        "excludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "validatedIncludeSpecs": [
            "**/*"
        ],
        "validatedExcludeSpecs": [
            "bin",
            "build",
            "node_modules"
        ],
        "wildcardDirectories": {
            "/my/build/path": 1
        }
    }
}
rpt2: included:
[
    "*.ts+(|x)",
    "**/*.ts+(|x)"
]
rpt2: excluded:
[
    "*.d.ts",
    "**/*.d.ts"
]
rpt2: resolving '../src/file3'
rpt2:     to '/my/build/path/module/path/src/file3.js'
rpt2: generating target 1
rpt2: rolling caches

Why I think this is one file (and not two files) is that the file3.js there is in the "src" folder which I've expressly blacklisted from appearing as entry files. So I'm not, myself, ever calling rollup with that file as the input file.

The output of the plugin always hangs before the rpt2: resolving 'x' so I'm guessing that's the real bottleneck. Checking with allowJS: false all the files are still being run through the plugin, each Rollup build also seems to have rpt2 outputting the last two lines from above, "generating target 1" and "rolling caches".

I'm definitely not an expert and know unfortunately little about how Rollup plugins truly work. That said, it does seem excessive to me that rpt2 is running on and even resolving seemingly all imports from at least the given entry file, even though that entry file is .js and thus should not fall within the include rules.

Or is this simply that the plugin is starting up on build start regardless of the entry file type?

[ezolenko]

resolving part is rollup asking the plugin to resolve an import from one of the modules rollup ends up processing (it asks all plugins and I think takes first valid reply in the chain). Plugin then uses typescript resolver to resolve the module, which ends up being a js file. That path is sent back to rollup, which loads it and adds it to the module list and runs it through the plugin chain. rpt2 will not try to transpile it because it is not ts/tsx, but babel and the rest will do what they do. Rollup then takes fully processed source, does its magic and injects it into the bundle.

Looks like in this run you didn't have any typescript files, so nothing was transpiled (unless there were more lines in that log).

We can't avoid resolving js files imported from ts files because we need to resolve it to know it is a js file. But I suppose rpt2 shouldn't be trying to resolve imports for modules it didn't transpile itself in the first place.

That might kill another use case of importing ts files directly from js files and having them transpiled on the fly, I'll have to check if that is even possible.

[aapoalas]

Yeah, in this case there actually were no TS files at all. Which is pretty much why I believed something to be amiss in the first place: No TS files are present, the plugin is set in the plugin chain just in case there are. Yet, the build times absolutely exploded. (Although, there still are some hundred files that I cut from the log. I figured you don't really want to see that :) )

This is definitely a bit of a convoluted problem. Personally, in my use case, I dislike rpt2 doing any of the resolving so far as it does not pertain to TS files importing other files, or JS files importing TS files. Even TS files importing eg. "lodash" to node modules on its own is something I do not like since, in my case, I don't want to allow these sorts of imports. Yet, I am using rpt2 with the idea that if someone has imported "lodash" and "@types/lodash" is installed, then the type checking should work (although the person who did this should be given a talking to)... So yeah, I'm sorry. I have a really dumb system on my hands :D

So far as I am concerned, importing TS files from JS files should work. TypeScript, I believe, does support this with allowJS: true. That said, I'm unsure what would happen if rpt2 stopped trying to resolve all imports... I think that it should be possible to just rely on the load hook for the plugin; check if the ID ends with .ts and then run the plugin based on that... There would then either need to be a plugin such as rollup-plugin-extensions in before rpt2 in the plugin chain or rpt2 would need to define its own extension checker for that to work.

Which, in the end, is what you're already doing. I presume that the problem is just that you're using the TS resolver, especially on imports from files that are not actually TypeScript. The TS resolver seems to take an unacceptably long time. So far as it comes to resolving logic within a given Rollup build, that including rpt2 changes the resolving logic from standard Rollup (only relative paths are resolved, .js extension can be left out) to node resolving is definitely a surprise.

When rpt2 is type checking a TS file, then I would expect the TS resolver to be used since ambient types as well as types inferred from JS files are relevant to the type validity of the file. But even then actual transpiling of the file I would expect to just consist of removing the TS and returning plain JS.

Sorry, this is probably getting way out of scope.

[ezolenko]

Try with current master, plugin should not be resolving random stuff now.

[ezolenko]

In 0.23.0 now

[aapoalas]

Awesome. I'll try and get around to testing this one of these days.