Datasource git on local file system fails

[thefriendlynet]

Deployment Type

Self-hosted

NetBox Version

v4.0.2

Python Version

3.11

Steps to Reproduce

1. create Datasource type git, URL e.g. ///home/datasources
2. try sync

Expected Behavior

files synced from local file system

Observed Behavior

SyncError("Fetching remote data failed (TypeError): LocalGitClient.__init__() got an unexpected keyword argument 'quiet'")

netbox/netbox/core/data_backends.py

        clone_args = {
            "branch": self.params.get('branch'),
            "config": self.config,
            "depth": 1,
            "errstream": porcelain.NoneStream(),
            "quiet": True,
        }

"quiet" and "depth" aren't supported by dulwich for local filesystem git repo's

[thefriendlynet]

This might be an issue in the dulwich LocalGitClient code, don't seem to be able to fix it from Netbox.

EDIT: see below for why it seemed my changes to the Netbox code didn't work.
(removing the dulwich code changes to avoid any further confusion)

[jeffgdotorg]

Thanks for reporting a problem you've encountered in NetBox.

I've managed to reproduce the problem you describe in a fresh NetBox 4.0.6 installation. For the benefit of future workers of this issue, here's what I did to set up the bare local Git repo and otherwise prepare the system.

ubuntu@nb406-repro-16760:~$ sudo mkdir /home/datasources
ubuntu@nb406-repro-16760:~$ sudo chown netbox:netbox /home/datasources
ubuntu@nb406-repro-16760:~$ sudo -u netbox git init /home/datasources
hint: Using 'master' as the name for the initial branch. This default branch name
...
Initialized empty Git repository in /home/datasources/.git/
ubuntu@nb406-repro-16760:~$ sudo -u netbox /bin/bash
netbox@nb406-repro-16760:/home/ubuntu$ cd /home/datasources/
netbox@nb406-repro-16760:/home/datasources$ git status
On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)
netbox@nb406-repro-16760:/home/datasources$ cat > README.txt
Hello world, this is an attempt to reproduce NetBox bug issue #16760.

Good luck!
netbox@nb406-repro-16760:/home/datasources$ git add .
netbox@nb406-repro-16760:/home/datasources$ git config --global user.email '***@netboxlabs.com'
netbox@nb406-repro-16760:/home/datasources$ git config --global user.name "Jeff Gehlbach"
netbox@nb406-repro-16760:/home/datasources$ git commit -m "Added a file"
[master (root-commit) f88787d] Added a file
 1 file changed, 3 insertions(+)
 create mode 100644 README.txt
ubuntu@nb406-repro-16760:~$ sudo -i
root@nb406-repro-16760:~# source /opt/netbox/venv/bin/activate
(venv) root@nb406-repro-16760:~# pip3 install dulwich
Collecting dulwich
  Downloading dulwich-0.22.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB)
Requirement already satisfied: urllib3>=1.25 in /opt/netbox/venv/lib/python3.10/site-packages (from dulwich) (2.2.2)
Downloading dulwich-0.22.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (979 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 979.1/979.1 kB 13.9 MB/s eta 0:00:00
Installing collected packages: dulwich
Successfully installed dulwich-0.22.1
(venv) root@nb406-repro-16760:~# systemctl restart netbox netbox-rq

I'm moving your issue along to needs owner status. If you would like to work it through to a PR, please say so and a maintainer will assign it to you.

[thefriendlynet]

Thanks for taking the time to add these steps.

EDIT: I didn't restart the Netbox worker, only the Netbox process, that's why my changes to the Netbox code didn't work, duh

This fixes the issue, but is it acceptable if I rework it into a PR ?

--- a/core/data_backends.py
+++ b/core/data_backends.py
@@ -133,6 +133,10 @@
                         "password": self.params.get('password'),
                     }
                 )
+        elif not self.url_scheme:
+            clone_args["depth"] = None
+            del(clone_args["quiet"])
 
         logger.debug(f"Cloning git repo: {self.url}")
         try:

[thefriendlynet]

If you would like to work it through to a PR, please say so and a maintainer will assign it to you.

I'd like to submit a PR based on the diff I posted.

[jeremystretch]

@thefriendlynet thanks, I've assigned this to you.