There's no readline module on Windows Python (cmd.Cmd)

[keeely]

BPO	45870
Nosy	@gvanrossum, @rhettinger, @terryjreedy, @eryksun, @zooba, @keeely, @E3V3A
Files	cmd_shell_example.py: Script to demo use of readline module with cmd

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-22.13:51:22.349>
labels = ['type-bug', 'library', '3.10']
title = "There's no readline module on Windows Python (cmd.Cmd)"
updated_at = <Date 2022-02-02.16:23:35.559>
user = 'https://github.com/keeely'

bugs.python.org fields:

activity = <Date 2022-02-02.16:23:35.559>
actor = 'E3V3A'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-11-22.13:51:22.349>
creator = 'keeely'
dependencies = []
files = ['50461']
hgrepos = []
issue_num = 45870
keywords = []
message_count = 16.0
messages = ['406778', '406800', '407118', '407162', '407164', '407168', '407169', '407172', '407174', '407176', '407177', '407178', '407180', '407258', '407261', '412372']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'rhettinger', 'terry.reedy', 'eryksun', 'steve.dower', 'keeely', 'E3V3A']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45870'
versions = ['Python 3.10']

[keeely]

In the past this was worked around by installing a third-party module:
https://github.com/pyreadline/pyreadline

For Python 3.10, however this module doesn't work. And pyreadline doesn't seem to be maintained anymore, so it's possible it won't get fixed.
pyreadline/pyreadline#73

Consider the following code:

import cmd
import readline

open("history.txt", "w").write("first line\nsecond line\n")

readline.clear_history()
readline.read_history_file("history.txt")

class MyShell(cmd.Cmd):
    def __init__(self):
        super().__init__()

shell = MyShell()
shell.cmdloop()

This works fine on MacOs Python, also on Linux and on Windows prior to 3.10 with the readline module added. It won't work going forward.

The Windows cmd implementation clearly uses readline or some compatible lib under the hood, so Python should make it available to developers.

[eryksun]

The Windows cmd implementation clearly uses readline or
some compatible lib under the hood

The REPL shell and input() call PyOS_Readline(). If this call isn't hooked (e.g. by the readline module), and stdin is a console file, then it reads a line from the console via ReadConsoleW(). If the console's input stream is configured in line-input mode, which we assume it is, this function provides basic readline-ish support, including a line editor that supports input history and aliases. The history is stored in an in-memory buffer in the console, which doesn't get save and reused across console sessions. There is no public API to get the history contents, and there is no support at all to set it.

The console/terminal team at Microsoft apparently don't want to do anything with the builtin readline support, which is seen as a legacy feature. To the contrary, I've even seen them suggest that they're evaluating the addition of a new client-side readline library in the native API, which I assume would be similar to PowerShell's PSReadLine (e.g. based on ReadConsoleInputW(), and supporting a classic console mode in addition to emacs and vi modes).

For now, I suggest patching pyreadline for your own use. In the reported issue, I see it's trying to use collections.Callable. Instead it should import collections.abc and use collections.abc.Callable.

[terryjreedy]

Thank you Eryk for the info.

As a bug report, this should be closed as '3rd party'. As an enhancement request, it needs to be specified more and should perhaps be discussed on python-ideas.

[keeely]

You can take the view that it's not a bug (with some justification), but a few lines in the cmd docs would make all the difference in terms of wasted time.

I have now abandoned my Windows port and suggested users install WSL2 instead which is the easiest way forward for me, but it'd be nice to have known from the start that portions of cmd functionality are not available for Win32 instead of the indirect references via readline. You could throw us a bone here.

[terryjreedy]

What specific sentences would you like where in which doc. (Please link as 'cmd doc' is too vague.)

[terryjreedy]

Sorry, you obviously mean
https://docs.python.org/3/library/cmd.html#module-cmd
What to add where still applies.

[terryjreedy]

Guido and Raymond, you are the two active coredevs that have contributed the most lines to cmd module. What do either of you think?

[eryksun]

You can take the view that it's not a bug (with some justification),
but a few lines in the cmd docs would make all the difference in
terms of wasted time.

If anything, I think the readline documentation should have a note explaining the situation in Windows. The documentation of the cmd module already makes the readline dependency clear:

If the readline module is loaded, input will automatically inherit 
bash-like history-list editing (e.g. Control-P scrolls back to the
last command, Control-N forward to the next one, Control-F moves the 
cursor to the right non-destructively, Control-B moves the cursor to 
the left non-destructively, etc.).

[gvanrossum]

AFAIK the reason command history works in cmd.py on Windows is that it's built into the terminal program. Or maybe into the operating system.

Thus, the user can use line editing and history, but there is no API (in Python) to interact with these.

I'm sure Steve Dower can explain the exact situation -- it may depend on which Windows version and which terminal program you use (my only recent experience is with winterm on Windows 10).

I agree that (once we sort out what works in what versions of Windows and which terminal programs) we should clarify this in the docs.