Error installing `tar.gz` with python 3.13.4 onwards when `%TMP%` folder is in an ImDisk created RAMDISK volume.

[barkoder]

Disclosure: I'm on Windows 7. I don't know whether or not this issue exists in the official release.

Error installing tar.gz with python 3.14

Free threaded 3.14

(venv) Y:\venv\scripts>python3.14t.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz
     / 995.6 kB 387.3 kB/s 0:00:02
ERROR: Could not install packages due to an OSError: [WinError 1] Incorrect function: 'X:\\tmp\\pip-req-build-4_c7a8_i'

Regular 3.14

(venv) Y:\venv\scripts>python.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz
     / 995.6 kB 411.5 kB/s 0:00:02
ERROR: Could not install packages due to an OSError: [WinError 1] Incorrect function: 'X:\\tmp\\pip-req-build-2ni5ww57'

No issue installing .zip with python 3.14

(venv) Y:\venv\scripts>python3.14t.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.zip
Collecting https://github.com/mikf/gallery-dl/archive/master.zip
  Downloading https://github.com/mikf/gallery-dl/archive/master.zip
     \ 1.4 MB 219.2 kB/s 0:00:06
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: gallery_dl
  Building wheel for gallery_dl (pyproject.toml) ... done
  Created wheel for gallery_dl: filename=gallery_dl-1.31.0.dev0-py3-none-any.whl size=701611 sha256=f8e8c41bf2d36da608b86c2552a3f9718f8a78ae9fe7a85618f972401734334c
  Stored in directory: X:\tmp\pip-ephem-wheel-cache-__tzil9b\wheels\56\1f\b8\27cad7dbd6d2f8f9e3d9f2592394436080c4e8ce200d08b77b
Successfully built gallery_dl
Installing collected packages: gallery_dl
  Attempting uninstall: gallery_dl
    Found existing installation: gallery_dl 1.30.9
    Uninstalling gallery_dl-1.30.9:
      Successfully uninstalled gallery_dl-1.30.9
Successfully installed gallery_dl-1.30.9

This issue does not exist in 3.13.8. Installing .tar.gz or .zip works fine. EDIT: This issue does exist in 3.13.8. The last working version is 3.13.3. See below comment.

Thank you so much @adang1345 for your unofficial python interpreters!
Seriously, you are awesome for making these!

[adang1345]

I gave this a try with Python 3.14.0 on my Windows 7 machine, and it worked without issue.

(venv) C:\Users\adang>python -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz
     / 996.7 kB 3.5 MB/s 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: gallery_dl
  Building wheel for gallery_dl (pyproject.toml) ... done
  Created wheel for gallery_dl: filename=gallery_dl-1.31.0.dev0-py3-none-any.whl
 size=702138 sha256=8c1b0e1b07614c45c3ba931b1c680870637784e774ad333ba228d714cb3e4d79
  Stored in directory: C:\Users\adang\AppData\Local\Temp\pip-ephem-wheel-cache-v7jlneo9\wheels\12\63\b4\7ed405fe9ccea94b80d90d87e76ce66e0fecd9b9e29d803719
Successfully built gallery_dl
Installing collected packages: gallery_dl
Successfully installed gallery_dl-1.31.0.dev0

Try adding the -v argument to your pip invocation and see whether that reveals anything useful.

[barkoder]

I tried your pip -v suggestion but that didn't help unfortunately.
But I did figure out why it's happening.
python versions 3.13.4 - 3.14.0 don't like it when %TMP% folder is in ImDisk created RAMDISK volumes.

I've confirmed Python 3.13.3 works when %TMP% folder is in ImDisk created RAMDISKs.

To reproduce.
Run Windows 7 x64 SP1 Ultimate
Install https://sourceforge.net/projects/imdisk-toolkit/files/20220301/ (20220301 is the version I'm using. There are newer versions as well but I've not reproduced the bug with those versions).
Create an NTFS or exFAT RAMDISK of 1024MB. Say drive letter X:

First to confirm it used to work with 3.13.3

Install https://github.com/adang1345/PythonVista/raw/master/3.13.3/python-3.13.3-amd64-full.exe
Run cmd.

X:
md tmp
set TMP=X:\tmp
set TEMP=X:\tmp
python --version && python.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Python 3.13.3
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz (998 kB)
     ---------------------------------------- 998.5/998.5 kB 1.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: gallery_dl
  Building wheel for gallery_dl (pyproject.toml) ... done
  Created wheel for gallery_dl: filename=gallery_dl-1.31.0.dev0-py3-none-any.whl size=703184 sha256=cdac923250f61b1c563328449657c7f642c9536a179900695c6c85d8baa8f549
  Stored in directory: X:\tmp\pip-ephem-wheel-cache-77tc2m6j\wheels\25\ff\1b\fd8a6b035f92eb9808314dfe7832003d04eaf0c61ba8e4c900
Successfully built gallery_dl
Installing collected packages: gallery_dl
Successfully installed gallery_dl

[notice] A new release of pip is available: 25.0.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip

Now to confirm it doesn't work with 3.13.4

Install https://github.com/adang1345/PythonVista/raw/master/3.13.4/python-3.13.4-amd64-full.exe
Run cmd.

X:
md tmp
set TMP=X:\tmp
set TEMP=X:\tmp
python --version && python.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Python 3.13.4
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz
     - 998.5 kB 1.0 MB/s 0:00:01
ERROR: Could not install packages due to an OSError: [WinError 1] Incorrect function: 'X:\\tmp\\pip-req-build-5rbqqbqd'


[notice] A new release of pip is available: 25.1.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip

I tested every version 3.13.4 onwards. They all fail.
Something changed between 3.13.3 and 3.13.4 that is causing this issue.

The reason I use RAMDISKs is to minimize wear on my disks. So I would appreciate any help. Thanks!

[adang1345]

I ran the same test on Windows 11 with the official Python 3.13.3 and 3.13.4, and I get the same results. Here is the stack trace for the 3.13.4 failure:

X:\>python.exe -m pip -v --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Using pip 25.2 from C:\Program Files\Python313\Lib\site-packages\pip (python 3.13)
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz (998 kB)
     ---------------------------------------- 998.5/998.5 kB 28.2 MB/s  0:00:00
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\commands\install.py", line 393, in run
    requirement_set = resolver.resolve(
        reqs, check_supported_wheels=not options.target_dir
    )
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 79, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 538, in collect_root_requirements
    reqs = list(
        self._make_requirements_from_install_req(
    ...<2 lines>...
        )
    )
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 494, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
        ireq.link,
    ...<2 lines>...
        version=None,
    )
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 226, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ~~~~~~~~~~~~~^
        link,
        ^^^^^
    ...<3 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 309, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        link=link,
        ^^^^^^^^^^
    ...<4 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 162, in __init__
    self.dist = self._prepare()
                ~~~~~~~~~~~~~^^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 239, in _prepare
    dist = self._prepare_distribution()
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 320, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 537, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 608, in _prepare_linked_requirement
    local_file = unpack_url(
        link,
    ...<4 lines>...
        hashes,
    )
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\operations\prepare.py", line 190, in unpack_url
    unpack_file(file.path, location, file.content_type)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\utils\unpacking.py", line 326, in unpack_file
    untar_file(filename, location)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\utils\unpacking.py", line 245, in untar_file
    tar.extractall(location, filter=pip_filter)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tarfile.py", line 2343, in extractall
    tarinfo, unfiltered = self._get_extract_tarinfo(
                          ~~~~~~~~~~~~~~~~~~~~~~~~~^
        member, filter_function, path)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tarfile.py", line 2432, in _get_extract_tarinfo
    self._handle_fatal_error(e)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "C:\Program Files\Python313\Lib\tarfile.py", line 2430, in _get_extract_tarinfo
    filtered = filter_function(unfiltered, path)
  File "C:\Program Files\Python313\Lib\site-packages\pip\_internal\utils\unpacking.py", line 210, in pip_filter
    member = data_filter(member, location)
  File "C:\Program Files\Python313\Lib\tarfile.py", line 848, in data_filter
    new_attrs = _get_filtered_attrs(member, dest_path, True)
  File "C:\Program Files\Python313\Lib\tarfile.py", line 773, in _get_filtered_attrs
    dest_path = os.path.realpath(dest_path, strict=os.path.ALLOW_MISSING)
  File "<frozen ntpath>", line 738, in realpath
OSError: [WinError 1] Incorrect function: 'X:\\tmp\\pip-req-build-9eqcc3an'

I looked at the changelog for Python 3.13.4, and I suspect that the fix for python/cpython#135034 is causing your issue.

[adang1345]

I also did some testing with different versions of pip, and it seems that the error occurs when you have pip 24.1 or higher and the Python instance contains python/cpython#135034. It's not clear to me whether this is a bug in Python, pip, or ImDisk.

[barkoder]

I tried to see if the same bug is reproducible in SoftPerfect RAM Disk 3.4.8 (Freeware, Not FOSS). Mounted an NTFS partition and set TEMP and TMP to a folder inside its drive letter.
I used python3.14t.exe(with its default pip 25.2) to install https://github.com/mikf/gallery-dl/archive/master.tar.gz
And it works perfectly.
I only tested 3.14.0, but I'm pretty sure all versions work fine with Softperfect's RAMdisk. Problem is Softperfect's RAMdisk is an inferior solution to ImDisk due to non-auto-shrinking Volume Sizes.
I think the way ImDisk's RAMDISK volume maps itself on Windows is different from the way Softperfect's RAMDISK volume maps itself. I just don't know how exactly, yet.
If this was a bug in ImDisk, Py3.13.3 and Py3.14.0(with pip 24.0) also shouldn't work right? But 3.13.3(default) and 3.14.0(pip==24.0) works with ImDisk perfectly? So this has to be a bug with python(>=3.13.4) or pip(>=24.1). I'm guessing it's pip. But it can't be a bug with ImDisk.
For your testing - SoftPerfect RAM Disk 3.4.8 Final Freeware version