[Tilix] Can't paste multiline commands

[KaKi87]

GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu) [Debian GNU/Linux 12 (bookworm)]
ble.sh, version 0.4.0-nightly+546499b (noarch) [git 2.47.1, GNU Make 4.3, GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)]
bash-completion, version 2.11 (hash:aefcfdd4b924e8b03db0d49be8bdb5881ad7d0cd, 77084 bytes) (noarch)
zoxide, version 0.9.6 (/home/linuxbrew/.linuxbrew/bin/zoxide)
locale: LANG=en_US.UTF-8
terminal: TERM=xterm-256color wcwidth=14.0-west/16.0-2+ri, vte:6800 (65;6800;1)
options: +extglob +histappend -hostcomplete +inherit_errexit +login_shell

Hello,

Trying to paste multiline commands results in multiple errors that don't occur when ble.sh is disabled :

Thanks

[akinomyoga]

Which terminal do you use? This seems like a terminal's quirk that sends the raw pasted data without converting the newlines to RET properly.

[KaKi87]

This is Tilix, a popular terminal emulator for GNOME.

But you're right, the issue doesn't occur on Konsole, the popular terminal emulator for KDE.

[akinomyoga]

It seems to call vte.pasteClipboard, but I don't find this function either in the Tilix codebase and in VTE. Maybe this is defined in another library or in an old version of VTE. I'll investigate them later.

[akinomyoga]

It seems to be processed in this function call chain.

• Terminal::copyClipboard - /generated/vte/vte/Terminal.d - GtkD
  • vte_terminal_copy_clipboard - /src/vtegtk.cc- vte
    • Terminal::emit_copy_clipboard - /src/vte.cc - vte
      • signals[SIGNAL_COPY_CLIPBOARD] - /src/vtegtk.cc - vte
        • VteTerminalClass::copy_clipboard - /src/vtegtk.cc - vte
          • vte_terminal_real_copy_clipboard - /src/vtegtk.cc - vte
            • Widget::copy - /src/widget.hh - vte
              • Terminal::widget_copy - /src/vte.cc - vte

However, it seems to copy the current selected terminal content to the clipboard. What is relevant now is Terminal::widget_paste in the same file as Terminal::widget_copy:

• Terminal::widget_paste - /src/vte.cc - vte
  • vte::terminal::pastify_string - /src/pastify.cc - vte

The conversion from LF (\n, \x0a) to CR (\r, \x0d) is properly performed on this line. I thought the possibility that this conversion is added recently, so I traced back the change history in the vte codebase:

• f93a093 2021-11-28
• 7daaabb 2021-11-28
• 235eb0c 2015-12-01
• 05c680a 2015-10-13
• 0501b07 2002-08-07
• a062c63 2002-08-03 introduced

However, this conversion was introduced in 22 years ago. It is unlikely that your Tilix uses a 23-year-old version of vte.

[KaKi87]

What does this mean then ?

[akinomyoga]

There are three possibilities currently in my mind.

• Tilix doesn't use vte's pasting utility, but it instead implements the pasting in its own code, where the conversion is missing.
• Tilix or some other library performs the conversion of the opposite direction after vte performs it.
• Tilix configures the pseudo-terminal handler to perform the opposite conversion from CR to LF.

[KaKi87]

Okay, what's the next step ?

[akinomyoga]

There is actually a suspicious piece of code:

https://github.com/steveniemitz/tilix/blob/d317dc8137a1eaa702e34a3e9aab63a2541d607b/source/gx/tilix/terminal/advpaste.d#L122-L133

I'm going to check how this function is used.

[akinomyoga]

@KaKi87 I guess SETTINGS_ADVANCED_PASTE_REPLACE_CRLF_KEY implies that Tilix has a setting of "Convert CRLF and CR to LF" in its configuration. Can you find it? If so, could you try turning it off and see if the behavior changes?

edit: If the configuration setting could be searched by a string, the key would probably be "advanced-paste-replace-crlf".

[KaKi87]

Here's what I found :

This option was enabled, and the issue no longer occurs after disabling it.

That said, it would be nice for ble.sh to work regardless as normal bash does.

Thank you

[akinomyoga]

• /source/gx/tilix/terminal/terminal.d:1441
  • /source/gx/tilix/terminal/advpaste.d:145
    • /source/gx/tilix/terminal/advpaste.d:127
• /source/gx/tilix/terminal/terminal.d:1449
  • /src/vte.cc:4858

So this bypasses the conversion by vte.

[akinomyoga]

This option was enabled, and the issue no longer occurs after disabling it.

Thanks for the confirmation.

That said, it would be nice for ble.sh to work regardless as normal bash does.

This is a bug of Tilix, which fails to implement the correct terminal protocol for pasting. When the option is enabled, it should convert CRLF and LF to CR for the terminal protocol (instead of converting them into LF as it does now).

The behavioral difference between Readline and ble.sh is due to the support for the multiline editing mode in ble.sh. Readline/Bash doesn't offer a fully-capable multiline editing mode (though one could insert a newline character by C-v C-j or C-q C-j). Readline/Bash instead provides a way to delimit the multiline editing mode by its parser state. On the other hand, ble.sh allows users to control and edit multiline commands, where C-j is used to execute the command. This follows the behavior of the Lisp interaction mode in Emacs.

Here, of course, ble.sh behaves differently from Readline. As far as you request exactly the same behavior of the line editor for the multiline editing mode, we cannot support any multiline mode. In particular, it is unreasonable to break or limit the multiline editing mode just to reproduce the Readline behavior for broken data fed by an unusual terminal protocol.

[KaKi87]

I'm sorry I don't understand where the difficulty is : if bash can consider all of CR/LF/CRLF as newlines, why couldn't ble.sh ?

If I was gonna do this in JS for example, I would just replace split(/\r+/g) with split(/[\r\n]+/g).

Thanks

[akinomyoga]

In short, ble.sh assigns a different feature to C-j for the multiline editing mode. We cannot assign two functions to a single key.

[KaKi87]

But I'm not using this shortcut, I'm just pasting something from my browser.

[akinomyoga]

But I'm not using this shortcut, I'm just pasting something from my browser.

Yes, then the necessary conversion should have been made by Tilix.

[KaKi87]

But that's where my question is : why couldn't ble.sh accept all of CR/LF/CRLF on paste as bash does ?

[akinomyoga]

Actually, you can configure ble.sh so that it treats the raw LF (the terminal representation of C-j) the same as CR (the terminal representation of C-m and RET) by the following setting:

ble-bind -f 'C-j' 'accept-single-line-or-newline'

However, this conflicts with the multiline editing; this overwrites the keybinding to execute a command from the multiline editing mode. You need to bind another key to accept-line so that you can run a command from the multiline editing mode.

However, I'm not going to make it the default.

[akinomyoga]

This seems to be already reported in Tilix at gnunn1/tilix#1959.

[KaKi87]

I see, so all programs rely on terminal emulators to send only one type of newline format.

[akinomyoga]

I see, so all programs rely on terminal emulators to send only one type of newline format.

Yes, or the terminal application needs to use a special terminal mode called the bracketed paste mode. The communication between the terminal and the terminal application is not just plain text, but various types of key combinations, application messages, events, etc. are also transmitted there. Then, everything needs to be encoded properly.

There are two ways to send the text containing newlines: one is to convert everything to CR, or the other is to quote the entire text with the bracketed-paste-mode begin/end markers (when the terminal detects the support by the application). However, the Tilix sends the raw plain text directly to the terminal application. It is like putting {data: Hello, terminal world!} in a JSON file (instead of the correctly quoted {"data": "Hello, terminal world!"}).

[akinomyoga]

@KaKi87 I'm now trying to debug Tilix. I could build and launch Tilix from the source on my machine after solving several problems. However, I have trouble reproducing the reported behavior. How can I paste a text into the terminal so that the problem reproduces?

I've found that the context menu, which pops up when I right-click the terminal display area, has an item "Paste". However, this correctly pastes the text; it quotes the pasted text within the bracketed paste markers, or when the bracketed paste mode is disabled, it correctly converts all the newline characters to CR as far as I observe. This is probably the pasting feature vte provides.

From what I observe in the Tilix source code, I guess Tilix has a special UI for pasting text with a dialog or a pane with a textbox, buttons, etc. However, I so far cannot identify how to call the UI. Do you have an idea?

[akinomyoga]

OK, I could reproduce the problem. I needed to turn on the option of the advanced pasting dialog in the settings.

[akinomyoga]

To paste any texts, we should call vte's vte_terminal_paste_text. The D binding (GtkD) offers the interface vte.pasteText, but this interface doesn't seem to have been released, so Tilix cannot call this vte interface. The interface was added in GtkD in commit gtkd-developers/GtkD@43060ab about 2.5 years ago, but the last release of GtkD, 3.10.0, is about three-year-old, and Tilix references this 3.10.0. I decided to add a code to manually replace LF with CR, with code comments describing the situation with the GtkD version.

I submitted a PR at gnunn1/tilix#2248. However, it seems the pull requests in Tilix are not processed at all recently. I anticipate it would take time for Tilix to be finally fixed.

@KaKi87 If gnunn1/tilix#2248 is not merged soon (and that's likely), one option is to pull and compile the fixed version of Tilix, akinomyoga:stop-pasting-raw-data, by your self. To compile Tilix, one needs to first install the D building system, dub. One also needs to install a D compiler dmd (Digital Mars D) or ldc (LLVM D compiler). According to README, gdc (GNU Compiler Collection - D support) is unsupported by Tilix. Then, you can clone the repository [email protected]:akinomyoga/tilix.git and check out the PR branch stop-pasting-raw-data. You can run dub build --build=release inside the directory. If it is successful, a binary tilix should be created.

In my case, it didn't compile with ldc (in Fedora). I needed to fix dub.json in the Tilix repository as follows:

diff --git a/dub.json b/dub.json
index 45f6e0c6..905d9d6c 100644
--- a/dub.json
+++ b/dub.json
@@ -4,7 +4,7 @@
     "copyright": "Copyright 息 2016, Gerald Nunn",
     "authors": ["Gerald Nunn"],
     "mainSourceFile": "source/app.d",
-    "dflags-ldc": ["-disable-linker-strip-dead","-link-defaultlib-shared=false"],
+    "dflags-ldc": ["-disable-linker-strip-dead","-link-defaultlib-shared=true"],
     "dependencies": {
         "gtk-d:gtkd": {
             "version": "3.10.0"

[KaKi87]

Oh wow, I didn't expect you to invest so much !

In fact, I was planning to investigate switching to another terminal emulator on Pop OS, like Ptyxis. 😅