Add @source not support (#17255)

This PR adds a new source detection feature: `@source not "…"`. It can
be used to exclude files specifically from your source configuration
without having to think about creating a rule that matches all but the
requested file:

```css
@import "tailwindcss";
@source not "../src/my-tailwind-js-plugin.js";
```

While working on this feature, we noticed that there are multiple places
with different heuristics we used to scan the file system. These are:

- Auto source detection (so the default configuration or an `@source
"./my-dir"`)
- Custom sources ( e.g. `@source "./**/*.bin"` — these contain file
extensions)
- The code to detect updates on the file system

Because of the different heuristics, we were able to construct failing
cases (e.g. when you create a new file into `my-dir` that would be
thrown out by auto-source detection, it'd would actually be scanned). We
were also leaving a lot of performance on the table as the file system
is traversed multiple times for certain problems.

To resolve these issues, we're now unifying all of these systems into
one `ignore` crate walker setup. We also implemented features like
auto-source-detection and the `not` flag as additional _gitignore_ rules
only, avoid the need for a lot of custom code needed to make decisions.

High level, this is what happens after the now:

- We collect all non-negative `@source` rules into a list of _roots_
(that is the source directory for this rule) and optional _globs_ (that
is the actual rules for files in this file). For custom sources (i.e
with a custom `glob`), we add an allowlist rule to the gitignore setup,
so that we can be sure these files are always included.
- For every negative `@source` rule, we create respective ignore rules.
- Furthermore we have a custom filter that ensures files are only read
if they have been changed since the last time they were read.

So, consider the following setup:

```css
/* packages/web/src/index.css */
@import "tailwindcss";
@source "../../lib/ui/**/*.bin";
@source not "../../lib/ui/expensive.bin";
```

This creates a git ignore file that (simplified) looks like this:

```gitignore
# Auto-source rules
*.{exe,node,bin,…}
*.{css,scss,sass,…}
{node_modules,git}/

# Custom sources can overwrite auto-source rules
!lib/ui/**/*.bin

# Negative rules
lib/ui/expensive.bin
```

We then use this information _on top of your existing `.gitignore`
setup_ to resolve files (i.e so if your `.gitignore` contains rules e.g.
`dist/` this line is going to be added _before_ any of the rules lined
out in the example above. This allows negative rules to allow-list your
`.gitignore` rules.

To implement this, we're rely on the `ignore` crate but we had to make
various changes, very specific, to it so we decided to fork the crate.
All changes are prefixed with a `// CHANGED:` block but here are the
most-important ones:

- We added a way to add custom ignore rules that _extend_ (rather than
overwrite) your existing `.gitignore` rules
- We updated the order in which files are resolved and made it so that
more-specific files can allow-list more generic ignore rules.
- We resolved various issues related to adding more than one base path
to the traversal and ensured it works consistent for Linux, macOS, and
Windows.

## Behavioral changes

1. Any custom glob defined via `@source` now wins over your `.gitignore`
file and the auto-content rules.
   - Resolves #16920
3. The `node_modules` and `.git` folders as well as the `.gitignore`
file are now ignored by default (but can be overridden by an explicit
`@source` rule).
   - Resolves #17318
   - Resolves #15882
4. Source paths into ignored-by-default folders (like `node_modules`)
now also win over your `.gitignore` configuration and auto-content
rules.
    -  Resolves #16669
5. Introduced `@source not "…"` to negate any previous rules.
   - Resolves #17058
6. Negative `content` rules in your legacy JavaScript configuration
(e.g. `content: ['!./src']`) now work with v4.
   - Resolves #15943 
7. The order of `@source` definitions matter now, because you can
technically include or negate previous rules. This is similar to your
`.gitingore` file.
9. Rebuilds in watch mode now take the `@source` configuration into
account
   - Resolves #15684

## Combining with other features

Note that the `not` flag is also already compatible with [`@source
inline(…)`](#17147)
added in an earlier commit:

```css
@import "tailwindcss";
@source not inline("container");
```

## Test plan

- We added a bunch of oxide unit tests to ensure that the right files
are scanned
- We updated the existing integration tests with new `@source not "…"`
specific examples and updated the existing tests to match the subtle
behavior changes
- We also added a new special tag `[ci-all]` that, when added to the
description of a PR, causes the PR to run unit and integration tests on
all operating systems.

[ci-all]

---------

Co-authored-by: Philipp Spiess <[email protected]>

Author: RobinMalfait

.github/workflows/ci.yml

@@ -25,13 +25,13 @@ jobs:
             os: macos-14
 
         # Exclude windows and macos from being built on feature branches
-        on-main-branch:
-          - ${{ github.ref == 'refs/heads/main' }}
+        run-all:
+          - ${{ github.ref == 'refs/heads/main' || contains(github.event.pull_request.body, '[ci-all]') }}
         exclude:
-          - on-main-branch: false
+          - run-all: false
             runner:
               name: Windows
-          - on-main-branch: false
+          - run-all: false
             runner:
               name: macOS
 

.github/workflows/integration-tests.yml

@@ -33,13 +33,13 @@ jobs:
           - workers
 
         # Exclude windows and macos from being built on feature branches
-        on-main-branch:
-          - ${{ github.ref == 'refs/heads/main' }}
+        run-all:
+          - ${{ github.ref == 'refs/heads/main' || contains(github.event.pull_request.body, '[ci-all]') }}
         exclude:
-          - on-main-branch: false
+          - run-all: false
             runner:
               name: Windows
-          - on-main-branch: false
+          - run-all: false
             runner:
               name: macOS
 

Cargo.lock

@@ -0,0 +1,3 @@
+This project is dual-licensed under the Unlicense and MIT licenses.
+
+You may use this code under the terms of either license.

crates/ignore/COPYING

@@ -0,0 +1,45 @@
+[package]
+name = "ignore"
+version = "0.4.23"  #:version
+authors = ["Andrew Gallant <[email protected]>"]
+description = """
+A fast library for efficiently matching ignore files such as `.gitignore`
+against file paths.
+"""
+documentation = "https://docs.rs/ignore"
+homepage = "https://github.com/BurntSushi/ripgrep/tree/master/crates/ignore"
+repository = "https://github.com/BurntSushi/ripgrep/tree/master/crates/ignore"
+readme = "README.md"
+keywords = ["glob", "ignore", "gitignore", "pattern", "file"]
+license = "Unlicense OR MIT"
+edition = "2021"
+
+[lib]
+name = "ignore"
+bench = false
+
+[dependencies]
+crossbeam-deque = "0.8.3"
+globset = "0.4.16"
+log = "0.4.20"
+memchr = "2.6.3"
+same-file = "1.0.6"
+walkdir = "2.4.0"
+dunce = "1.0.5"
+
+[dependencies.regex-automata]
+version = "0.4.0"
+default-features = false
+features = ["std", "perf", "syntax", "meta", "nfa", "hybrid", "dfa-onepass"]
+
+[target.'cfg(windows)'.dependencies.winapi-util]
+version = "0.1.2"
+
+[dev-dependencies]
+bstr = { version = "1.6.2", default-features = false, features = ["std"] }
+crossbeam-channel = "0.5.8"
+
+[features]
+# DEPRECATED. It is a no-op. SIMD is done automatically through runtime
+# dispatch.
+simd-accel = []

crates/ignore/Cargo.toml

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Andrew Gallant
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

crates/ignore/LICENSE-MIT

@@ -0,0 +1,58 @@
+# ignore
+
+The ignore crate provides a fast recursive directory iterator that respects
+various filters such as globs, file types and `.gitignore` files. This crate
+also provides lower level direct access to gitignore and file type matchers.
+
+[![Build status](https://github.com/BurntSushi/ripgrep/workflows/ci/badge.svg)](https://github.com/BurntSushi/ripgrep/actions)
+[![](https://img.shields.io/crates/v/ignore.svg)](https://crates.io/crates/ignore)
+
+Dual-licensed under MIT or the [UNLICENSE](https://unlicense.org/).
+
+### Documentation
+
+[https://docs.rs/ignore](https://docs.rs/ignore)
+
+### Usage
+
+Add this to your `Cargo.toml`:
+
+```toml
+[dependencies]
+ignore = "0.4"
+```
+
+### Example
+
+This example shows the most basic usage of this crate. This code will
+recursively traverse the current directory while automatically filtering out
+files and directories according to ignore globs found in files like
+`.ignore` and `.gitignore`:
+
+```rust,no_run
+use ignore::Walk;
+
+for result in Walk::new("./") {
+    // Each item yielded by the iterator is either a directory entry or an
+    // error, so either print the path or the error.
+    match result {
+        Ok(entry) => println!("{}", entry.path().display()),
+        Err(err) => println!("ERROR: {}", err),
+    }
+}
+```
+
+### Example: advanced
+
+By default, the recursive directory iterator will ignore hidden files and
+directories. This can be disabled by building the iterator with `WalkBuilder`:
+
+```rust,no_run
+use ignore::WalkBuilder;
+
+for result in WalkBuilder::new("./").hidden(false).build() {
+    println!("{:?}", result);
+}
+```
+
+See the documentation for `WalkBuilder` for many other options.

crates/ignore/README.md

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org/>

crates/ignore/UNLICENSE

@@ -0,0 +1,64 @@
+use std::{env, io::Write, path::Path};
+
+use {bstr::ByteVec, ignore::WalkBuilder, walkdir::WalkDir};
+
+fn main() {
+    let mut path = env::args().nth(1).unwrap();
+    let mut parallel = false;
+    let mut simple = false;
+    let (tx, rx) = crossbeam_channel::bounded::<DirEntry>(100);
+    if path == "parallel" {
+        path = env::args().nth(2).unwrap();
+        parallel = true;
+    } else if path == "walkdir" {
+        path = env::args().nth(2).unwrap();
+        simple = true;
+    }
+
+    let stdout_thread = std::thread::spawn(move || {
+        let mut stdout = std::io::BufWriter::new(std::io::stdout());
+        for dent in rx {
+            stdout.write(&*Vec::from_path_lossy(dent.path())).unwrap();
+            stdout.write(b"\n").unwrap();
+        }
+    });
+
+    if parallel {
+        let walker = WalkBuilder::new(path).threads(6).build_parallel();
+        walker.run(|| {
+            let tx = tx.clone();
+            Box::new(move |result| {
+                use ignore::WalkState::*;
+
+                tx.send(DirEntry::Y(result.unwrap())).unwrap();
+                Continue
+            })
+        });
+    } else if simple {
+        let walker = WalkDir::new(path);
+        for result in walker {
+            tx.send(DirEntry::X(result.unwrap())).unwrap();
+        }
+    } else {
+        let walker = WalkBuilder::new(path).build();
+        for result in walker {
+            tx.send(DirEntry::Y(result.unwrap())).unwrap();
+        }
+    }
+    drop(tx);
+    stdout_thread.join().unwrap();
+}
+
+enum DirEntry {
+    X(walkdir::DirEntry),
+    Y(ignore::DirEntry),
+}
+
+impl DirEntry {
+    fn path(&self) -> &Path {
+        match *self {
+            DirEntry::X(ref x) => x.path(),
+            DirEntry::Y(ref y) => y.path(),
+        }
+    }
+}