Support uploading over http(s)://

[kousu]

This is a follow up to #3: filling in the blanks from https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/, I thought that only git annex get worked with http remotes. But now I think that was premature.

Investigate and make git-annex uploading work over http remotes.

Specifically this means these commands should work:

• git annex sync --content
• git annex copy --to origin

[kousu]

Using a test repo at https://data.praxisinstitute.org.dev.neuropoly.org/uofc/CANDICE-fMRI-, which is currently running the patch for #3, download anonymously:

git clone ... && git annex get

[kousu@nigiri down]$ git clone https://data.praxisinstitute.org.dev.neuropoly.org/UofC/CANDICE-fMRI-.git
Cloning into 'CANDICE-fMRI-'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 71 (delta 21), reused 0 (delta 0)
Receiving objects: 100% (71/71), 9.72 KiB | 4.86 MiB/s, done.
Resolving deltas: 100% (21/21), done.
[kousu@nigiri down]$ cd CANDICE-fMRI-/
[kousu@nigiri CANDICE-fMRI-]$ git annex get
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (from origin...) 
47%   2.87 MiB        895 KiB/s 3s    
ok                                    
get sub-BAN02/anat/sub-BAN02_T1w.nii.gz (from origin...) 
52%   3.31 MiB          2 MiB/s 1s
ok                                
get sub-BAN03/anat/sub-BAN03_T1w.nii.gz (from origin...) 
ok                                
(recording state in git...)

then construct an new annexed file:

cp -r sub-BAN02/ sub-BAN04
dd if=/dev/urandom of=sub-BAN04/anat/sub-BAN02_T1w.nii.gz count=512
git add sub-BAN04/
git commit -m "Subject 04"

prove it's distinct (it's the last one, the one with 1 copy):

git annex whereis

[kousu@nigiri CANDICE-fMRI-]$ git annex whereis
whereis sub-BAN01/anat/sub-BAN01_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN02/anat/sub-BAN02_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN03/anat/sub-BAN03_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN04/anat/sub-BAN02_T1w.nii.gz (1 copy) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
ok

and now see what happens if we try to upload it; with debugging turned on:

git annex copy --to origin

[kousu@nigiri CANDICE-fMRI-]$ git config annex.debug true
[kousu@nigiri CANDICE-fMRI-]$ git annex copy --to origin
[2022-05-08 02:52:23.217458919] (Utility.Process) process [28036] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2022-05-08 02:52:23.219000083] (Utility.Process) process [28036] done ExitSuccess
[2022-05-08 02:52:23.219516464] (Utility.Process) process [28037] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2022-05-08 02:52:23.221135085] (Utility.Process) process [28037] done ExitSuccess
[2022-05-08 02:52:23.22179462] (Utility.Process) process [28038] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..6c3b23ed5b89e73fdb170e4dfb3dc4f7324acd87","--pretty=%H","-n1"]
[2022-05-08 02:52:23.224259745] (Utility.Process) process [28038] done ExitSuccess
[2022-05-08 02:52:23.225976282] (Utility.Process) process [28039] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2022-05-08 02:52:23.228332152] (Utility.Process) process [28040] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--error-unmatch","--"]
[2022-05-08 02:52:23.229159541] (Utility.Process) process [28041] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.22958377] (Utility.Process) process [28042] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.230213916] (Utility.Process) process [28039] done ExitSuccess
[2022-05-08 02:52:23.231313697] (Utility.Process) process [28043] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
copy sub-BAN01/anat/sub-BAN01_T1w.nii.gz [2022-05-08 02:52:23.269482287] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:23.715269089] (Utility.Process) process [28045] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:33.439560188] (Utility.Process) process [28045] done ExitSuccess
[2022-05-08 02:52:33.439918575] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.531585766] (Utility.Process) process [28081] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:33.533291906] (Utility.Process) process [28081] done ExitSuccess
[2022-05-08 02:52:33.53378081] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.622360074] (Utility.Process) process [28082] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:38.54396353] (Utility.Process) process [28082] done ExitSuccess
[2022-05-08 02:52:38.544127686] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.629070613] (Utility.Process) process [28086] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:38.630412108] (Utility.Process) process [28086] done ExitSuccess
(not found) failed
copy sub-BAN02/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:38.63124213] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.717445234] (Utility.Process) process [28087] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:43.084445916] (Utility.Process) process [28087] done ExitSuccess
[2022-05-08 02:52:43.084688615] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.170468809] (Utility.Process) process [28088] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:43.172434906] (Utility.Process) process [28088] done ExitSuccess
[2022-05-08 02:52:43.172943361] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.258412042] (Utility.Process) process [28089] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:46.248660233] (Utility.Process) process [28089] done ExitSuccess
[2022-05-08 02:52:46.249043634] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.335692255] (Utility.Process) process [28090] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:46.337206776] (Utility.Process) process [28090] done ExitSuccess
(not found) failed
copy sub-BAN03/anat/sub-BAN03_T1w.nii.gz [2022-05-08 02:52:46.337740833] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.423388602] (Utility.Process) process [28091] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:49.262936219] (Utility.Process) process [28091] done ExitSuccess
[2022-05-08 02:52:49.263330345] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.349186911] (Utility.Process) process [28092] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:49.350967438] (Utility.Process) process [28092] done ExitSuccess
[2022-05-08 02:52:49.35138871] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.435139518] (Utility.Process) process [28093] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:52.986881594] (Utility.Process) process [28093] done ExitSuccess
[2022-05-08 02:52:52.987023514] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.076025925] (Utility.Process) process [28096] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:53.077770146] (Utility.Process) process [28096] done ExitSuccess
(not found) failed
copy sub-BAN04/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:53.078448893] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.16577961] (Utility.Process) process [28097] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:56.868778349] (Utility.Process) process [28097] done ExitSuccess
[2022-05-08 02:52:56.869119387] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:56.954725712] (Utility.Process) process [28098] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:56.956656836] (Utility.Process) process [28098] done ExitSuccess
[2022-05-08 02:52:56.957140664] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:57.041148655] (Utility.Process) process [28099] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:53:00.095495453] (Utility.Process) process [28099] done ExitSuccess
[2022-05-08 02:53:00.095774849] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:53:00.183376947] (Utility.Process) process [28100] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:53:00.18501078] (Utility.Process) process [28100] done ExitSuccess
(not found) failed
[2022-05-08 02:53:00.185253039] (Utility.Process) process [28043] done ExitSuccess
[2022-05-08 02:53:00.185356286] (Utility.Process) process [28042] done ExitSuccess
[2022-05-08 02:53:00.185460462] (Utility.Process) process [28041] done ExitSuccess
[2022-05-08 02:53:00.185552806] (Utility.Process) process [28040] done ExitSuccess
copy: 4 failed

Interestingly, git-annex is sending an Authorization header, and it looks like it's leaning on git credential so so it's git doing the prompting, which is nice and elegant. So maybe uploading over HTTP does work afterall.

But this part of the protocol isn't documented on the wiki, or at least I don't want to waste my time trying to find my way around that place. But I bet we have enough here to reverse engineer it; it looks like the first thing it needs is a response to a HEAD request, presumably checking if the file already exists or not; so we'll have to add another route for that, and once we do that maybe it'll send a POST or a PUT request to actually upload the file, and we'll have to add a route for that too. And then it should be done 🤞

[kousu]

I tried again to get this working for a few hours but no matter what I tried I couldn't convince git-annex to do it.

I learnt it issues a HEAD request for each file. If that returns 200, and the Content-Length matches the expected size, it says "ok". Otherwise, it says "not found". It never tries to issue a PUT or a POST. So I doubt this is supported.

I suspect it's a result of there being no server in the world that it could possibly be tested against. GitHub doesn't support it, Apache doesn't support PUTs, GitLab probably doesn't either. If we added annex HTTP uploading support we'd be the first, probably.

I'll put in a feature request and see what joeyh says.

[kousu]

Feature request/request for clarification: https://git-annex.branchable.com/forum/HTTP_uploads/?updated

[kousu]

This is a wontfix:

There is no "one true way" to upload a file to a HTTP server.

I don't want to add some additional git-annex specific HTTP-based protocol.

(I lightly disagree: the One True Way is PUT + an Authorization: header, which is just what S3 uses, and what Gitea could easily support. But if the code doesn't exist it doesn't exist.)