Feature: Ignore files not under version control

[mdeweerd]

Context:
I created a script to add lines to the file ignore list that I can run if there are only false codespell positives.

I want to exclude files that are not under git version control as these do not need to be added as exceptions.

It seems appropriate and useful that 'codespell' has an option to do so.

For example, the following shell command filters to only files that are under version control:
git ls-files $(ls -p | grep -v -E '/$').

I could apply something like that to the script by using the codespell output two times or running it a second time on the filtered files. Done.

However, excluding files that are not under version controls seems like a good filter for codespell.

EDIT: updating the ignored lines file could even be a feature in codespell.

[DimitriPapadopoulos]

Currently codespell is a simple piece of software focused on processing the arguments it is given, without filters — except for hidden files. I am not certain adding filters is worth the extra complexity. You could instead modify the arguments passed to codespell. Any way, I'll let maintainers decide.

[mdeweerd]

The use case is that when running it from the CLI with local files used for development, these files are codespell'ed as well and when running the script to ignore line that are misspelled, lines from files not under version control also got added.
I updated my script to avoid that, but a CLI codespell run still shows these "false positives".

[yarikoptic]

this might also be a very nice alternative to needing to explicit list to ignore all of those other folders (.git/ , .mypy_cache/) etc. And it would also help to avoid side-effects of fixing some files not under git, while committing only the ones under git. So overall I would have liked to have such mode and making it configurable via config.

edits:

• it would also help to avoid dealing with Behavior of --skip with folder names #1915

[DimitriPapadopoulos]

OK, you convinced me. Which version control though? Not everyone uses git.

[mdeweerd]

I vote for git - I moved almost all my subversion projects to git, rcs/cvs has been long gone, and it's what I encounter the most.

[mdeweerd]

Note that git can do this:

git ls-files -- ':*php'
git ls-files -- . ':!*sql' ':!*bin'

So the file glob patterns could all be provided to git ls-files and then codespell just needs to iterate over the result.

[yarikoptic]

well, could be a setting ignore_vcs = git and then puke NotImplementedError for anything but git ATM so keeping it open for others to add andling for what they care. Minor note: for the test, make sure you have some files with obscure filenames [*] and use git ls-files -z and separate based on \000 not newline , and also I think that would avoid you need to deal with quotes:

use without -z

❯ mkdir /tmp/test; cd /tmp/test; git init; echo 1 > "new
line"; touch empty; touch '"quoted"'; git add *; git commit -m "added files"; git ls-files
Initialized empty Git repository in /tmp/test/.git/
[master (root-commit) 8bd959f] added files
 3 files changed, 1 insertion(+)
 create mode 100644 "\"quoted\""
 create mode 100644 empty
 create mode 100644 "new\nline"
"\"quoted\""
empty
"new\nline"

with -z but then translating \000 into a | for visualization

❯ rm -rf /tmp/test; mkdir /tmp/test; cd /tmp/test; git init; echo 1 > "new
line"; touch empty; touch '"quoted"'; git add *; git commit -m "added files"; git ls-files -z | tr '\000' '|'
Initialized empty Git repository in /tmp/test/.git/
[master (root-commit) 28da778] added files
 3 files changed, 1 insertion(+)
 create mode 100644 "\"quoted\""
 create mode 100644 empty
 create mode 100644 "new\nline"
"quoted"|empty|new
line|%

[*] In DataLad we programmatically compose "most obscure" (although still without new line IIRC) filename for a filesystem at hands during testing: https://github.com/datalad/datalad/blob/maint/datalad/tests/utils_pytest.py#L1453 . So on my laptop it is :

❯ python -c 'from datalad.tests.utils_pytest import OBSCURE_FILENAME; print(repr(OBSCURE_FILENAME))'
' |;&%b5{}\'"<>ΔЙקم๗あ .datc '

NB if you use debian package -- there would be no unicode part.

so note spaces leading/trailing it to ensure we do not strip them anywhere...

[spetrosi]

As a workaround you can use https://github.com/marketplace/actions/get-all-changed-files to get changed files and run codespell against them

[cvvergara]

In file: untracked.sh

csv_string=$(git ls-files . --exclude-standard --others --directory --ignored)
csv_string=${csv_string//$'\n'/,}   # to have a comma separated string
csv_string=${csv_string//$'/'/''}   # Remove the directory slash
echo "$csv_string"

Do not check on untracked files and on locale:

codespell -S "$(bash untracked.sh),locale"

[mdeweerd]

Using the SKIP argument is a nice alternative which I'ld optimise to:

codespell -S "$(git ls-files . --exclude-standard --others --directory --ignored --deduplicate | tr '\n' ',' | sed 's/,$//')"

This does slow down things a lot (and it's not the git command).
On a bit repo with some big local log files:
0m58.092s with the SKIP argument,
0m9.218s without.

So 6 times slower with SKIP!!

[afranke]

Duplicate of #1887

[mdeweerd]

#1887 is related, but not exactly the same - could be "merged" with this one.

#1887 proposes to reuse '.gitignore' . Not all .gitignore exclusions are exempt from version control and not all other files are under version control.

IHMO it is more relevant to check only files under version control rather than checking all files not covered by .gitignore.

However, #1897 is a match.

So I conclude that a 4 year old request that has not made it as a function yet will likely never make it.

I am closing this in favor of the older request feature request.