`bundledDependencies` attempted to be fetched from npm instead

[obermillerk]

Do you want to request a feature or report a bug?

What is the current behavior?
A package being installed from the NPM registry with bundled dependencies will look for the bundled dependencies on NPM registry instead of using the bundled versions. If modules don't exist on the npm registry, the install fails saying

error Couldn't find package "bundled-package@*" required by "installing-package@x.x.x" on the "npm" registry.

If the current behavior is a bug, please provide the steps to reproduce.

yarn add win-bt

This is my module that I've published to npm, it uses bundled dependencies for some windows specific features, so unfortunately this will only work on windows. There are some other prerequisites that are necessary for a full successful install, but it doesn't get to any of those before this bundled dependencies error so it shouldn't be an issue. Alternately, you could potentially use some other module that has bundled dependencies on npm, I just don't know of any specifically that do. Regardless, all you have to do is yarn add like so and the error shows up.

What is the expected behavior?
Expected to fetch the module and use the dependencies bundled therein instead of looking on NPM for them.

Please mention your node.js, yarn and operating system version.
node:8.9.1 yarn:1.5.1 and 1.7.0 os: Windows 10 v1803 (build 17134.48)

[rally25rs]

we are probably iterating over the dependencies without removing bundledDependencies from the list.

Likely here: https://github.com/yarnpkg/yarn/blob/master/src/package-request.js#L284

[obermillerk]

Any progress on this? Thanks

[nbeerbower]

Hi all, just wanted to chime in.

win-wifi's package.json's dependencies section currently looks like this (which works with NPM):

"dependencies": {
    "deasync": "file:deasync-0.1.10-custom.tgz",
    "windows.devices.enumeration": "file:windows.devices.enumeration-0.1.0.tgz",
    "windows.devices.wifi": "file:windows.devices.wifi-0.1.0.tgz",
    "windows.networking.connectivity": "file:windows.networking.connectivity-0.1.0.tgz",
    "windows.security.credentials": "file:windows.security.credentials-0.1.0.tgz"
  }

I was able to get the installation to work with yarn by using relative file paths:

"dependencies": {
    "deasync": "file:./deasync-0.1.10-custom.tgz",
    "windows.devices.enumeration": "file:./windows.devices.enumeration-0.1.0.tgz",
    "windows.devices.wifi": "file:./windows.devices.wifi-0.1.0.tgz",
    "windows.networking.connectivity": "file:./windows.networking.connectivity-0.1.0.tgz",
    "windows.security.credentials": "file:./windows.security.credentials-0.1.0.tgz"
  }

Using Yarn 1.7.0

EDIT: sorry this is win-wifi, same issue though

[obermillerk]

I don't believe this is actually the issue since the win-bt library does not have the file dependencies listed as the wifi library does.

[sambostock]

For what it's worth, this bug seems to be the direct cause of twilio/flex-plugin-builder#2. @twilio/flex-ui bundles a dependency on @twilio/frame-ui, which Yarn fails to handle. It goes looking for it on NPM, where it is a private package, which causes Yarn to abort the install.

Similar to the described scenario, only Yarn encounters an authentication error, not a not found error.

[feelepxyz]

👋 from Dependabot and wanted to chime in with another perspective that this bug is currently allowing us to update vulnerable bundled sub-dependencies in yarn (something we can't do in npm).

For example, installing fsevents@1.2.4 with npm installs a vulnerable version of tar@4.4.1, yarn resolves the patched version tar@4.4.10, this is resolved from the bundled parents requirement (node-pre-gyp requires tar@^4). If a vulnerable version is already in the lockfile, removing the entry and running yarn install` will resolve the latest version based on the parent requirement (this is how Dependabot updates sub-dependenices in yarn).

FWIW fsevents@2 also moved away from using bundleDependencies so seems there's some movement away from this pattern. I also don't see the benefit of using this to bundle private dependencies (as done in @twilio/flex-ui) because the bundled dependency is essentially public but through another package. I might be missing something here but are there any legit use cases where libraries should use bundledDependencies?

I'm in favour of leaving this bug unfixed because of the security benefit of being able to update vulnerable sub-dependencies 🤠

[mysticflute]

I don't think it's good for public libraries to use bundledDependencies normally... In the example you gave not using bundledDependencies at all would also solve it right?

I think the more appropriate use case is for applications that won't be published on npm or features that are still in development but can't be published yet. In these cases you may want to share it with other people or enable CI integrations and use bundledDependencies to handle dependencies that are also unpublished. A private registry is another option but bundledDependencies is a lot simpler in many cases.

[isaacs]

In tap, I bundle the treport and ink deps, because ink has a peerDependency on a specific version of react, and this interacts badly with projects that are using a different react version. (In ink's case, it really doesn't need to be a peerDependency, since it's used more like a regular dependency, but that's another issue.)

Simple reproduction case:

$ mkdir yarn-gh-5998

$ cd yarn-gh-5998/
/Users/isaacs/dev/js/yarn-gh-5998

$ yarn init --yes
yarn init v1.16.0
warning The yes flag has been set. This will automatically answer yes to all questions, which may have security implications.
success Saved package.json
✨  Done in 0.04s.

$ yarn add tap
yarn add v1.16.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 271 new dependencies.
info Direct dependencies
└─ tap@14.4.1
info All dependencies
├─ @babel/generator@7.5.0
├─ @babel/helper-function-name@7.1.0
├─ @babel/helper-get-function-arity@7.0.0
├─ @babel/helper-split-export-declaration@7.4.4
├─ @babel/highlight@7.5.0
├─ @babel/parser@7.5.0
├─ @babel/runtime@7.5.4
├─ @babel/template@7.4.4
├─ @babel/traverse@7.5.0
├─ @types/prop-types@15.7.1
├─ @types/react@16.8.23
├─ ajv@6.10.1
├─ ansi-escapes@3.2.0
├─ ansi-styles@3.2.1
├─ ansicolors@0.3.2
├─ anymatch@3.0.3
├─ append-transform@1.0.0
├─ arg@4.1.0
├─ argparse@1.0.10
├─ arrify@1.0.1
├─ asn1@0.2.4
├─ astral-regex@1.0.0
├─ async-hook-domain@1.1.0
├─ asynckit@0.4.0
├─ auto-bind@2.1.0
├─ aws-sign2@0.7.0
├─ aws4@1.8.0
├─ babel-core@6.26.3
├─ babel-generator@6.26.1
├─ babel-helper-builder-react-jsx@6.26.0
├─ babel-helpers@6.24.1
├─ babel-plugin-syntax-jsx@6.18.0
├─ babel-plugin-syntax-object-rest-spread@6.13.0
├─ babel-plugin-transform-es2015-destructuring@6.23.0
├─ babel-plugin-transform-object-rest-spread@6.26.0
├─ babel-plugin-transform-react-jsx@6.24.1
├─ babel-register@6.26.0
├─ babel-template@6.26.0
├─ balanced-match@1.0.0
├─ bcrypt-pbkdf@1.0.2
├─ binary-extensions@2.0.0
├─ bind-obj-methods@2.0.0
├─ brace-expansion@1.1.11
├─ braces@3.0.2
├─ browser-process-hrtime@1.0.0
├─ buffer-from@1.1.1
├─ caching-transform@3.0.2
├─ caller-callsite@2.0.0
├─ caller-path@2.0.0
├─ callsites@2.0.0
├─ camelcase@5.3.1
├─ capture-stack-trace@1.0.1
├─ cardinal@2.1.1
├─ caseless@0.12.0
├─ chalk@2.4.2
├─ chokidar@3.0.2
├─ ci-info@2.0.0
├─ cli-truncate@1.1.0
├─ cliui@4.1.0
├─ code-point-at@1.1.0
├─ color-convert@1.9.3
├─ color-name@1.1.3
├─ combined-stream@1.0.8
├─ commander@2.20.0
├─ commondir@1.0.1
├─ concat-map@0.0.1
├─ convert-source-map@1.6.0
├─ core-js@2.6.9
├─ core-util-is@1.0.2
├─ coveralls@3.0.4
├─ cp-file@6.2.0
├─ cross-spawn@6.0.5
├─ csstype@2.6.6
├─ dashdash@1.14.1
├─ debug@2.6.9
├─ decamelize@1.2.0
├─ default-require-extensions@2.0.0
├─ delayed-stream@1.0.0
├─ detect-indent@4.0.0
├─ ecc-jsbn@0.1.2
├─ emoji-regex@7.0.3
├─ end-of-stream@1.4.1
├─ error-ex@1.3.2
├─ es6-error@4.1.1
├─ escape-string-regexp@1.0.5
├─ esm@3.2.25
├─ esprima@4.0.1
├─ execa@1.0.0
├─ extend@3.0.2
├─ extsprintf@1.3.0
├─ fast-deep-equal@2.0.1
├─ fast-json-stable-stringify@2.0.0
├─ fill-range@7.0.1
├─ find-cache-dir@2.1.0
├─ findit@2.0.0
├─ flow-parser@0.101.1
├─ flow-remove-types@2.101.0
├─ forever-agent@0.6.1
├─ form-data@2.3.3
├─ fs-exists-cached@1.0.0
├─ fs.realpath@1.0.0
├─ fsevents@2.0.7
├─ function-loop@1.0.2
├─ get-caller-file@2.0.5
├─ get-stream@4.1.0
├─ getpass@0.1.7
├─ glob-parent@5.0.0
├─ globals@11.12.0
├─ growl@1.10.5
├─ handlebars@4.1.2
├─ har-schema@2.0.0
├─ har-validator@5.1.3
├─ has-ansi@2.0.0
├─ home-or-tmp@2.0.0
├─ hosted-git-info@2.7.1
├─ http-signature@1.2.0
├─ inflight@1.0.6
├─ inherits@2.0.4
├─ ink@2.3.0
├─ invariant@2.2.4
├─ invert-kv@2.0.0
├─ is-arrayish@0.2.1
├─ is-binary-path@2.1.0
├─ is-ci@2.0.0
├─ is-extglob@2.1.1
├─ is-finite@1.0.2
├─ is-number@7.0.0
├─ is-stream@1.1.0
├─ isarray@1.0.0
├─ isstream@0.1.2
├─ istanbul-lib-hook@2.0.7
├─ istanbul-lib-instrument@3.3.0
├─ istanbul-lib-processinfo@1.0.0
├─ istanbul-lib-report@2.0.8
├─ istanbul-lib-source-maps@3.0.6
├─ istanbul-reports@2.2.6
├─ jackspeak@1.4.0
├─ js-tokens@4.0.0
├─ js-yaml@3.13.1
├─ jsesc@1.3.0
├─ json-parse-better-errors@1.0.2
├─ json-schema-traverse@0.4.1
├─ json-schema@0.2.3
├─ json-stringify-safe@5.0.1
├─ json5@0.5.1
├─ jsprim@1.4.1
├─ lcid@2.0.0
├─ lcov-parse@0.0.10
├─ load-json-file@4.0.0
├─ locate-path@3.0.0
├─ lodash.flattendeep@4.4.0
├─ lodash.throttle@4.1.1
├─ log-driver@1.2.7
├─ log-update@3.2.0
├─ loose-envify@1.4.0
├─ lru-cache@4.1.5
├─ make-error@1.3.5
├─ map-age-cleaner@0.1.3
├─ mem@4.3.0
├─ merge-source-map@1.1.0
├─ mime-db@1.40.0
├─ mime-types@2.1.24
├─ mimic-fn@1.2.0
├─ minimist@0.0.8
├─ neo-async@2.6.1
├─ nested-error-stacks@2.1.0
├─ nice-try@1.0.5
├─ node-modules-regexp@1.0.0
├─ normalize-package-data@2.5.0
├─ npm-run-path@2.0.2
├─ nyc@14.1.1
├─ oauth-sign@0.9.0
├─ once@1.4.0
├─ onetime@2.0.1
├─ opener@1.5.1
├─ optimist@0.6.1
├─ os-homedir@1.0.2
├─ os-locale@3.1.0
├─ os-tmpdir@1.0.2
├─ own-or-env@1.0.1
├─ p-defer@1.0.0
├─ p-finally@1.0.0
├─ p-is-promise@2.1.0
├─ p-limit@2.2.0
├─ p-locate@3.0.0
├─ p-try@2.2.0
├─ package-hash@3.0.0
├─ parse-json@4.0.0
├─ path-exists@3.0.0
├─ path-is-absolute@1.0.1
├─ path-key@2.0.1
├─ path-parse@1.0.6
├─ path-type@3.0.0
├─ performance-now@2.1.0
├─ pirates@3.0.2
├─ pkg-dir@3.0.0
├─ private@0.1.8
├─ process-nextick-args@2.0.1
├─ pseudomap@1.0.2
├─ psl@1.2.0
├─ pump@3.0.0
├─ punycode@1.4.1
├─ qs@6.5.2
├─ react-is@16.8.6
├─ react-reconciler@0.20.4
├─ read-pkg-up@4.0.0
├─ read-pkg@3.0.0
├─ readable-stream@2.3.6
├─ readdirp@3.1.1
├─ redeyed@2.1.1
├─ regenerator-runtime@0.13.2
├─ release-zalgo@1.0.0
├─ repeating@2.0.1
├─ request@2.88.0
├─ require-directory@2.1.1
├─ resolve-from@3.0.0
├─ resolve@1.11.1
├─ restore-cursor@2.0.0
├─ rimraf@2.6.3
├─ safer-buffer@2.1.2
├─ semver@5.7.0
├─ set-blocking@2.0.0
├─ shebang-command@1.2.0
├─ shebang-regex@1.0.0
├─ slash@1.0.0
├─ source-map-support@0.5.12
├─ source-map@0.6.1
├─ spawn-wrap@1.4.2
├─ spdx-correct@3.1.0
├─ spdx-exceptions@2.2.0
├─ sprintf-js@1.0.3
├─ sshpk@1.16.1
├─ stack-utils@1.0.2
├─ string_decoder@1.1.1
├─ strip-eof@1.0.0
├─ supports-color@5.5.0
├─ tap-mocha-reporter@4.0.1
├─ tap@14.4.1
├─ tcompare@2.3.0
├─ test-exclude@5.2.3
├─ to-fast-properties@1.0.3
├─ to-regex-range@5.0.1
├─ tough-cookie@2.4.3
├─ treport@0.4.0
├─ trivial-deferred@1.0.1
├─ ts-node@8.3.0
├─ tunnel-agent@0.6.0
├─ tweetnacl@0.14.5
├─ typedarray-to-buffer@3.1.5
├─ typescript@3.5.3
├─ uglify-js@3.6.0
├─ unicode-length@1.0.3
├─ uri-js@4.2.2
├─ util-deprecate@1.0.2
├─ validate-npm-package-license@3.0.4
├─ verror@1.10.0
├─ vlq@0.2.3
├─ which-module@2.0.0
├─ which@1.3.1
├─ widest-line@2.0.1
├─ wordwrap@0.0.3
├─ wrap-ansi@5.1.0
├─ write-file-atomic@3.0.0
├─ y18n@4.0.0
├─ yallist@3.0.3
├─ yaml@1.6.0
├─ yapool@1.0.0
├─ yargs-parser@13.1.1
├─ yargs@13.2.4
├─ yn@3.1.0
└─ yoga-layout-prebuilt@1.9.3
✨  Done in 7.29s.

$ find . -name treport
./node_modules/tap/node_modules/treport
./node_modules/treport

$ find . -name ink
./node_modules/ink
./node_modules/tap/node_modules/ink

This is an extraneous dependency, because nothing is depending on ink at the top level. Additionally, it means that yarn.lock and yarn list may contain incorrect information, since the version actually in place might not match what the lockfile and list output report.

bundleDependencies is often a bit of a hack, but it's a hack that provides a much-needed escape valve in a lot of otherwise sticky edge cases. It'd be good if yarn didn't double-install them.

[chucknelson]

Want to chime in on this as well, as I really think it should be fixed / is a bug. As of this comment I'm using yarn@1.19.1.

Similar to @sambostock's scenario above with @twilio/flex-ui bundling a private dependency of @twilio/frame-ui, I also have a project that includes packages that include private bundled dependencies. The behavior is exactly the same - yarn attempts to fetch these bundled dependencies from the registry, but they won't be there (by design).

Considering these are officially documented dependency types for both npm and yarn, I would think these "private bundled dependency" scenarios would be supported.

For any yarn core members that may read this, would a PR be considered here, or is more discussion required?

[chucknelson]

So I think I might attempt a PR for this soon (unless someone else gets to it first). Unfortunately it's a little more complicated than just "if bundled, skip!" when resolving and installing dependencies.

For the first pass I think the following might work:

1. Identify/label a dependency as "bundled" with its parent.
   Similar to optional dependencies, may need to pass around information that a dependency is "bundled" with its parent for use in other parts of the yarn lifecycle.
2. If bundled, don't attempt to download/fetch/request the dependency.
   Not sure if there needs to be some "validation" of the bundled version, or if that's assumed to be taken care of when the parent dependency was packaged.
3. Continue to try and resolve and fetch the bundled dependency's dependencies.

If anyone has more insight or feedback please let me know - thanks!

I could also probably look at how npm implements bundled dependency resolution...but I think my head will explode if I try to go look through that code right now 😞

FYI also made sure this wasn't already fixed in yarn v2/"berry", and doesn't seem to be:

➤ YN0001: │ HTTPError: @twilio/frame-ui@npm:^0.36.1: Response code 404 (Not Found)
    at EventEmitter.<anonymous> (/development/yarn/.yarn/releases/yarn-berry.js:360:59866)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)