Terminals served by tornado websockets

I've opened this issue to collect any discussion around what would be involved in supporting Windows. This issue can then hopefully provide a technical resource for any sufficiently motivated individual to take a crack at it!

Relies on https://github.com/spyder-ide/pywinpty/pull/67 and a release.

Note this is for Python 3 only because pywinpty does not support Python 2.

Here's a screenshot with export SHELL=`which cmd`:

Here it is with export SHELL=`which bash`:

Passing build on https://ci.appveyor.com/project/blink1073/terminado/build/1.0.7

I can take care of any Windows related maintenance issues moving forward.

When I leave a Terminal open in the IPython Notebook for a long period of time, it eventually stops responding and shows a traceback on the server side:

[E 2015-04-06 18:40:47.096 ahenriks ioloop:588] Exception in callback (36, <function wrap.<locals>.null_wrapper at 0x7f8869f13f28>)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.4/dist-packages/tornado/ioloop.py", line 840, in start
        handler_func(fd_obj, events)
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
        return fn(*args, **kwargs)
      File "/usr/local/lib/python3.4/dist-packages/terminado/management.py", line 187, in pty_read
        s = ptywclients.ptyproc.read(65536)
      File "/usr/local/lib/python3.4/dist-packages/ptyprocess/ptyprocess.py", line 803, in read
        b = super(PtyProcessUnicode, self).read(size)
      File "/usr/local/lib/python3.4/dist-packages/ptyprocess/ptyprocess.py", line 507, in read
        s = self.fileobj.read(size)
    OSError: [Errno 9] Bad file descriptor

The odd thing is that these are raised at the rate of a hundreds/thousands a minute. Not sure if this is an issue with terminado or ptyprocess.

I tried pip installing

tornado
tornado_xstatic
xstatic

But it still doesn't seem to be enough, on the website I get a 500 error, python tells me:

$ python demos/single.py 
ERROR:tornado.application:Uncaught exception GET / (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8765', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Host': 'localhost:8765', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Connection': 'keep-alive'})
Traceback (most recent call last):
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado_xstatic.py", line 36, in _get_xstatic_data_dir
    return cls._cached_xstatic_data_dirs[mod_name]
KeyError: 'termjs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado/web.py", line 1443, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "demos/single.py", line 14, in get
    ws_url_path="/websocket")
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado/web.py", line 699, in render
    html = self.render_string(template_name, **kwargs)
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado/web.py", line 806, in render_string
    return t.generate(**namespace)
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado/template.py", line 345, in generate
    return execute()
  File "termpage_html.generated.py", line 5, in _tt_execute
    _tt_tmp = xstatic('termjs', 'term.js')  # termpage.html:22
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado_xstatic.py", line 62, in make_url
    version_bit = "?v=" + XStaticFileHandler.get_version({'static_path': ''}, fs_style_path)
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado/web.py", line 2642, in get_version
    abs_path = cls.get_absolute_path(settings['static_path'], path)
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado_xstatic.py", line 49, in get_absolute_path
    root = cls._get_xstatic_data_dir(mod_name)
  File "/home/stu/.virtualenvs/scratch/lib/python3.5/site-packages/tornado_xstatic.py", line 38, in _get_xstatic_data_dir
    xsmod = getattr(__import__('xstatic.pkg', fromlist=[mod_name]), mod_name)
AttributeError: module 'xstatic.pkg' has no attribute 'termjs'
ERROR:tornado.access:500 GET / (127.0.0.1) 6.68ms

A fix for #183

This includes switching to using a dedicated executor for blocking IO, specifically wrapping calls to ptyprocess.write which, can hang when the PTY buffer (the size of which is set by the OS) is exceeded. TermSocket users can provide their own thread pool by using the named blocking_io_executor parameter on initialization, but by default a shared, process-wide single threaded executor will be used.

This also includes a regression test which will fail on Debian and OSX (but may be able to pass on some OSs due to the variance of the PTY buffer's size).

This also switched the default filetype of terminado/tests/basic_test.py from dos to unix to match with the rest of the files in this package and prevent some editors from auto-inserting carriage returns on save.

We are dealing with an issue where large copy/pastes into Terminado (through open terminals created in the Jupyter UI) cause the Jupyter server to hang.

We diagnosed this issue and it seems to be one where Terminado is using PtyProcess.write on the entirety of stdin message, which for copy/pasted text is the entirety of the pasted text. PtyProcess is using blocking writes to a PTY file descriptor. When the size of the write exceeds the max PTY buffer size (which I believe is set at the OS level) the write blocks indefinitely. Because this is happening inside of the Tornado event loop, it hangs the server.

It's not clear to me if this should be considered a bug in Terminado or PtyProcess. I don't see PtyProcess offering a non-blocking write mechanism, so I'm not sure what the best immediate solution is. It seems like PtyProcess could offer non-blocking writes which could be leveraged by Terminado, or Terminado could split input into some reasonably small chunks before passing them through.

Hi, it's exciting to see terminal works on windows now. But I am experiencing constant cpu spike.

After started Jupyter-lab with jupyter lab --core-mode 4 new process spawned showed up in process explorer as usual

    jupyter.exe
    \-python.exe
       |-jupyter-lab.exe
         |-python.exe         with listening port

after opening a terminal in the jupyter lab web portal. one of the jupyter-lab python process is giving constant full cpu utilization, though the terminal is totally idle.

    jupyter.exe
    \-python.exe
       |-jupyter-lab.exe
         |-python.exe   <---- constant 33% cpu usage. 
           |- winpty-agent.exe
                 |- conhost.exe
                 |- powershell.exe

Is there anything I can try to solve this ? BTW, the box is a Windows2012 R2 (VMWare VM), with CPU 3 core. I have cygwin installed. Can I configure jupyter terminal to use Cygwin's bash instead ?

Alleviates https://github.com/jupyterlab/jupyterlab/issues/10633. A full solution should take the users choice of scrollback history length into consideration rather than setting a constant value of "maxlen" on line 44.

Hi there! I've got a reproducible issue (on my environment, at least) that surrounds installing terminado via pip.

Background: I've got a process that we use to build a python environment that is relocatable between machines. It builds python in a subdirectory of the user doing the build, not in the OS installed location. I'm not sure if this is relevant or not, but seems possible.

When I run this:

.../native-env-builder/target/native-dev62-SNAPSHOT/bin/pip install --no-use-wheel -b pip-build terminado==0.7

I get this error:

DEPRECATION: --no-use-wheel is deprecated and will be removed in the future. Please use --no-binary :all: instead. Collecting terminado==0.7 Using cached terminado-0.7.tar.gz Requirement already satisfied: ptyprocess in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from terminado==0.7) Requirement already satisfied: tornado>=4 in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from terminado==0.7) Requirement already satisfied: singledispatch in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.7) Requirement already satisfied: certifi in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.7) Requirement already satisfied: backports_abc>=0.4 in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.7) Requirement already satisfied: six in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from singledispatch->tornado>=4->terminado==0.7) Installing collected packages: terminado Found existing installation: terminado 0.6 Uninstalling terminado-0.6: Successfully uninstalled terminado-0.6 Running setup.py install for terminado ... error Complete output from command /Users/ssuchter/src/pepperdata11/native-env-builder/target/native-dev62-SNAPSHOT/bin/python -u -c "import setuptools, tokenize;file='/Users/ssuchter/src/pepperdata11/native-env-builder/target/pip-build/terminado/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/y6/6ccct9kd5ms1080vt5nv4jm00000gn/T/pip-PNF25V-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build/lib creating build/lib/terminado copying terminado/init.py -> build/lib/terminado copying terminado/management.py -> build/lib/terminado copying terminado/uimodule.py -> build/lib/terminado copying terminado/websocket.py -> build/lib/terminado creating build/lib/terminado/tests copying terminado/tests/init.py -> build/lib/terminado/tests copying terminado/tests/basic_test.py -> build/lib/terminado/tests error: can't copy 'terminado/_static': doesn't exist or not a regular file

----------------------------------------

Rolling back uninstall of terminado Command "/Users/ssuchter/src/pepperdata11/native-env-builder/target/native-dev62-SNAPSHOT/bin/python -u -c "import setuptools, tokenize;file='/Users/ssuchter/src/pepperdata11/native-env-builder/target/pip-build/terminado/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/y6/6ccct9kd5ms1080vt5nv4jm00000gn/T/pip-PNF25V-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /Users/ssuchter/src/pepperdata11/native-env-builder/target/pip-build/terminado/

But when I run this:

.../native-env-builder/target/native-dev62-SNAPSHOT/bin/pip install --no-use-wheel -b pip-build terminado==0.6

everything works:

DEPRECATION: --no-use-wheel is deprecated and will be removed in the future. Please use --no-binary :all: instead. Collecting terminado==0.6 Using cached terminado-0.6.tar.gz Requirement already satisfied: ptyprocess in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from terminado==0.6) Requirement already satisfied: tornado>=4 in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from terminado==0.6) Requirement already satisfied: singledispatch in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.6) Requirement already satisfied: certifi in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.6) Requirement already satisfied: backports_abc>=0.4 in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from tornado>=4->terminado==0.6) Requirement already satisfied: six in ./native-dev62-SNAPSHOT/lib/python2.7/site-packages (from singledispatch->tornado>=4->terminado==0.6) Installing collected packages: terminado Running setup.py install for terminado ... done Successfully installed terminado-0.6

For now, I'm going to use 0.6, but I thought I'd like to let you know about the issue.

Hi, I am trying to fix this issue jupyterlab/jupyterlab#1366 in jupyterlab so that terminals open in the current working directory.

In order to do this, we need to be able to pass key word arguments to the new_terminal call in new_named_terminal so that the jupyter server can create terminals with different working directories.

https://github.com/jupyter/terminado/blob/b0b32f3efdde4252466f4e57b98773d9e86951ec/terminado/management.py#L317-L324

Jupyter brings in latest tornado version 6.0.1 and it breaks Jupyter Terminal due to removing of gen.Task in earlier version, https://www.tornadoweb.org/en/stable/releases/v6.0.0.html

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-terminado-0.17.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-terminado-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/terminado-0.17.1, configfile: pyproject.toml, testpaths: terminado/tests/
collected 9 items

terminado/tests/basic_test.py .......F.                                                                                                                              [100%]

================================================================================= FAILURES =================================================================================
____________________________________________________________________ UniqueTermTests.test_max_terminals ____________________________________________________________________

self = <terminado.tests.basic_test.UniqueTermTests testMethod=test_max_terminals>

    @tornado.testing.gen_test
    @pytest.mark.skipif("linux" not in platform, reason="It only works on Linux")
    async def test_max_terminals(self):
        tms = await self.get_term_clients(["/unique"] * MAX_TERMS)
        pids = await self.get_pids(tms)
        self.assertEqual(len(set(pids)), MAX_TERMS)  # All PIDs unique

        # MAX_TERMS+1 should fail
        tm = await self.get_term_client("/unique")
        msg = await tm.read_msg()
        self.assertEqual(msg, None)  # Connection closed

        # Close one
        tms[0].close()
        msg = await tms[0].read_msg()  # Closed
        self.assertEqual(msg, None)

        # Should be able to open back up to MAX_TERMS
        tm = await self.get_term_client("/unique")
        msg = await tm.read_msg()
>       self.assertEqual(msg[0], "setup")
E       TypeError: 'NoneType' object is not subscriptable

terminado/tests/basic_test.py:307: TypeError
---------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------
WARNING  tornado.general:autoreload.py:128 tornado.autoreload started more than once in the same process
ERROR    tornado.application:web.py:1798 Uncaught exception GET /unique (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:43321', method='GET', uri='/unique', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/lib64/python3.8/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/home/tkloczko/rpmbuild/BUILD/terminado-0.17.1/terminado/websocket.py", line 56, in open
    self.terminal = self.term_manager.get_terminal(url_component)
  File "/home/tkloczko/rpmbuild/BUILD/terminado-0.17.1/terminado/management.py", line 323, in get_terminal
    raise MaxTerminalsReached(self.max_terminals)
terminado.management.MaxTerminalsReached: Cannot create more than 3 terminals
ERROR    tornado.application:web.py:1798 Uncaught exception GET /unique (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:43321', method='GET', uri='/unique', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/lib64/python3.8/site-packages/tornado/websocket.py", line 942, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/home/tkloczko/rpmbuild/BUILD/terminado-0.17.1/terminado/websocket.py", line 56, in open
    self.terminal = self.term_manager.get_terminal(url_component)
  File "/home/tkloczko/rpmbuild/BUILD/terminado-0.17.1/terminado/management.py", line 323, in get_terminal
    raise MaxTerminalsReached(self.max_terminals)
terminado.management.MaxTerminalsReached: Cannot create more than 3 terminals
============================================================================= warnings summary =============================================================================
../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1294
  /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1294: PytestConfigWarning: Unknown config option: timeout

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

terminado/tests/basic_test.py:310
  /home/tkloczko/rpmbuild/BUILD/terminado-0.17.1/terminado/tests/basic_test.py:310: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.timeout(timeout=ASYNC_TEST_TIMEOUT, method="thread")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================== slowest 10 durations ===========================================================================
8.42s call     terminado/tests/basic_test.py::NamedTermTests::test_namespace
6.90s call     terminado/tests/basic_test.py::CommonTests::test_basic_command
6.54s call     terminado/tests/basic_test.py::NamedTermTests::test_max_terminals
6.54s call     terminado/tests/basic_test.py::UniqueTermTests::test_max_terminals
4.39s call     terminado/tests/basic_test.py::UniqueTermTests::test_unique_processes
4.30s call     terminado/tests/basic_test.py::SingleTermTests::test_single_process
2.59s call     terminado/tests/basic_test.py::UniqueTermTests::test_large_io_doesnt_hang
0.80s call     terminado/tests/basic_test.py::CommonTests::test_basic
0.13s call     terminado/tests/basic_test.py::NamedTermTests::test_new

(1 durations < 0.005s hidden.  Use -vv to show these durations.)
========================================================================= short test summary info ==========================================================================
FAILED terminado/tests/basic_test.py::UniqueTermTests::test_max_terminals - TypeError: 'NoneType' object is not subscriptable
================================================================= 1 failed, 8 passed, 2 warnings in 40.89s =================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
attrs                         22.1.0
Babel                         2.11.0
Brlapi                        0.8.3
build                         0.9.0
charset-normalizer            3.0.1
cssselect                     1.1.0
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
editables                     0.3
exceptiongroup                1.0.0
extras                        1.0.0
fixtures                      4.0.0
gpg                           1.17.1-unknown
hatchling                     1.11.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
iniconfig                     1.1.1
Jinja2                        3.1.2
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
markdown-it-py                2.1.0
MarkupSafe                    2.1.1
mdit-py-plugins               0.3.3
mdurl                         0.1.2
myst-parser                   0.18.1
numpy                         1.23.1
packaging                     21.3
pathspec                      0.10.2
pbr                           5.9.0
pep517                        0.13.0
pip                           22.3.1
pluggy                        1.0.0
ptyprocess                    0.7.0
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pytest                        7.2.0
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
requests                      2.28.1
rpm                           4.17.0
scour                         0.38.2
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
tomli                         2.0.1
tornado                       6.2
typing_extensions             4.4.0
urllib3                       1.26.12
wheel                         0.38.4
zipp                          3.11.0

Hello,

Thank you for an amazing terminal emulator!

Can anyone please provide an example of how Terminado could be used with Unix socket file instead of TCP port, in familiar example as in demos/unique.py? Is this even possible?

Seeing some test fails on https://github.com/conda-forge/terminado-feedstock/pull/38, looking into it:

        if future_cell[0].cancelled() or not future_cell[0].done():
>           raise TimeoutError("Operation timed out after %s seconds" % timeout)
E           asyncio.exceptions.TimeoutError: Operation timed out after 30.0 seconds

..\_test_env\lib\site-packages\tornado\ioloop.py:528: TimeoutError
------------------------------ Captured log call ------------------------------
WARNING  tornado.general:autoreload.py:128 tornado.autoreload started more than once in the same process

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=575370&view=logs&j=7df5a329-fcdb-5857-5720-a5dae94c0641&t=428acfd7-2921-5219-037f-9169d136e406&l=494

Maybe i missed some test dep changes? Might be the weekend before i can investigate further...

The font size feels like a bit too small at times. Ability to pass font size parameter through make_terminal function in template html file to terminado.js file

In terminado/_static/terminado.js I dont see any handling of backpressure in case too much data floods the terminal. This is important to not screw up terminal state by fast producers, or even segfault at any stage in between.

For further info, see https://xtermjs.org/docs/guides/flowcontrol/.

The confusion stems from the fact that during my use of jupyterlab's terminal function, when I refresh the web page in the terminal screen, I only get the messages I didn't receive again, but I lose the history.

Then I tried to find the answer by looking at the code, but I found that we used preopen_buffer but not read_buffer, and I wanted to know the reason for this

After I replaced the preopen_buffer with a read_buffer, not only can I achieve history passing, but also the function of fetching messages when reopening, and I also think there is no need to keep too large a buffer, the current length-independent preopen_buffer may cause some other problems

after F5:

using read_buffer

No change after F5, more friendly