Try to parse and run ble.sh

[andychu]

Why?

• it's a very substantial program using bash as a programming language, but it's not a batch program
• it's probably one of the hardest scripts to run (probably won't get it to run in totality, but maybe some parts will)
• It could be faster. maybe that will help optimize Oil.
  • I want Oil to be fast at char-by-char processing
• Maybe it will help us figure out what kind of API we need to let other people build other interactive UIs
  • $LINES and $COLUMNS, etc.

https://github.com/akinomyoga/ble.sh

[andychu]

#620 was the original issue, already addressed return 2>&1 etc.

[andychu]

Two issues blocking parsing:

• several cases of arith assignment where var name is dynamic doesn't work #640
• the \$file one

~/git/wild/ble.sh$ for name in */*.sh; do echo $name; ~/git/oilshell/oil/bin/osh -n $name | wc -l; done

...

memo/D1259.readbind.sh
84
out/ble.sh
            ((_ble_hook_c_$name=0))
                               ^
out/ble.sh:1644: Left-hand side of this assignment is invalid
0
src/benchmark.sh
2342
src/canvas.sh
    ((${_prefix}x=_pos[0]))
                 ^
src/canvas.sh:1521: Left-hand side of this assignment is invalid
0
src/color.sh
    (($_var=_ret))
           ^
src/color.sh:670: Left-hand side of this assignment is invalid
0
src/decode.sh
22515
src/def.sh
133
src/edit.sh
45409
src/history.sh
        ((_ble_builtin_history_rskip_dict[\$file]+=$2))
                                          ^
src/history.sh:701: Unexpected token in arithmetic context
0
src/util.sh
            ((_ble_hook_c_$name=0))
                               ^
src/util.sh:917: Left-hand side of this assignment is invalid
0

[andychu]

Note: can parse lib/core-syntax.sh (6651 lines, one of the biggest) by commenting out 5 instances of dynamic LHS in arith

• src/history.sh (1671 lines) parses with a few edits

[andychu]

• src/util.sh needs many patches (or relaxing parsing). This pattern is used all over the place.
  • there is at least one instance of it in our WILD corpus too
  • might want to take another look at arith assignment where var name is dynamic doesn't work #640 , or just allow parsing as a half-measure, and do
    evaluation later

Rules:

• arith_expr.Word:

  • statically a var name -> treat as VarRef -- but do this at evaluation time?
  • otherwise evaluate as word, and then treat that as an integer constant

• or just add Word on LHS as valid? And that must be an integer constant on RHS.

  • or on LHS it can be a var name?

[andychu]

@akinomygoa

On #640 I added a temporary hack shopt -s parse_unimplemented that parses dynamic arithmetic LHS but doesn't implement it. So at least this idiom won't prevent you from using osh -n. Now I can do:

for name in */*.sh; do echo $name; ~/git/oilshell/oil/bin/osh -O parse_unimplemented -n $name | wc -l; done

and there's only one remaining error:

src/history.sh
        ((_ble_builtin_history_rskip_dict[\$file]+=$2))
                                          ^
src/history.sh:701: Unexpected token in arithmetic context

If you want to use osh -n, maybe rewrite this without (( )) or with eval? I think this is particularly confusing because as we discussed \$file in this context does not mean '$file' -- it means something closer to "$file", correct?

---

And it should take 30 seconds to build osh from head on most Linux machines, although you can also wait until the next release:

https://github.com/oilshell/oil/wiki/Contributing

[akinomyoga]

Oh, I didn't notice your comment (but my user name is akinomyoga). I tried the latest commit which worked nicely! In testing I replaced the \$file line with eval '((...))', but I think I'll leave it as it is in the master branch for now because ble.sh is currently unlikely to run on Oil, so ble.sh is still only targeting Bash. Also, probably I'm too used to Bash syntaxes, but the \$file trick is natural to me. Nevertheless, I'll add some code comments above the line.

[akinomyoga]

I'm currently trying to source ble.sh in osh. ble.sh can be loaded without activation by passing the option --attach=none. Even if it doesn't fully work, I think it may be possible to just source ble.sh with minimal initialization. I'm currently using the following script to load ble.sh (with some modifications such as \$file)

# ble.osh
shopt -s parse_unimplemented
BASH_VERSION='5.0.11(1)-release'
BASH_VERSINFO=(5 0 11 1 release x86_64-redhat-linux-gnu)
set -o emacs
source "$(dirname "${BASH_SOURCE[0]}")/ble.sh" --attach=none

1. No line-editing mode by default ([[ -o emacs/vi ]])

The first one that I encountered is the line-editing mode detection. In Bash, the line editing mode can be tested with [[ -o emacs ]] and [[ -o vi ]]. When both of the tests fail, it implies that the line editing is turned off in Bash. ble.sh rejects the session with the line-editing turned off. I tested Oil and found that it implements the option, but both emacs and vi option is unset by default even though the twos cannot be unset at the same time by set -o/+o emacs/vi. In Oil, set +o vi implies set -o emacs, and set +o emacs implies set -o vi, and vice versa. What is the conceptual model for the state of the line-editing mode in Oil? I created a test for this.

test1.sh

#!/usr/bin/env bash

function print-line-editing-mode {
  local emacs=disabled vi=disabled
  [[ -o emacs ]] && emacs=enabled
  [[ -o vi ]] && vi=enabled
  echo "emacs $emacs, vi $vi"
}

echo -n "[default] "
print-line-editing-mode

echo -n "[set +o emacs] "
set +o emacs
print-line-editing-mode

echo -n "[set -o emacs] "
set -o emacs
print-line-editing-mode

echo -n "[set -o vi] "
set -o vi
print-line-editing-mode

echo -n "[set +o vi] "
set +o vi
print-line-editing-mode

echo -n "[set +o emacs] "
set +o emacs
print-line-editing-mode

set -o emacs

bash$ source test1.sh
[default] emacs enabled, vi disabled
[set +o emacs] emacs disabled, vi disabled
[set -o emacs] emacs enabled, vi disabled
[set -o vi] emacs disabled, vi enabled
[set +o vi] emacs disabled, vi disabled
[set +o emacs] emacs disabled, vi disabled

osh$ source test1.sh
[default] emacs disabled, vi disabled
[set +o emacs] emacs disabled, vi enabled
[set -o emacs] emacs enabled, vi disabled
[set -o vi] emacs disabled, vi enabled
[set +o vi] emacs enabled, vi disabled
[set +o emacs] emacs disabled, vi enabled

2. Comment: Array/scalar $FUNCNAME

The second one was $FUNCNAME which is actually the first array element ${FUNCNAME[0]} intentionally unsupported in Oil. I think the current Oil design is reasonable, but let me make a comment. The variable FUNCNAME was initially a scalar variable but extended to be an array in Bash 3.0, so it is a scalar variable and an array at the same time in Bash. This is the historical reason why the name is not FUNCNAMES but FUNCNAME even though it is an array. [ Note: Even for pure array variables, ble.sh uses this kind of parameter expansions in many places where the first element of an array contains the main result and the others contain additional information. ]

3. BUG: Start of ${@:offset} and ${*:offset} is shifted

The third one is that the results of ${@:offset} and ${*:offset} is shifted in Oil. I think this is a bug.

test2.sh

#!/usr/bin/env bash

set X Y Z W
echo "\$1=$1 \$2=$2 \$3=$3 \$4=$4"
echo "\${*:2}=(${*:2}) \${@:2}=(${@:2})"

osh$ source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Z W) ${@:2}=(Z W)

bash$ source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

zsh% source test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

ksh$ source ./test2.sh
$1=X $2=Y $3=Z $4=W
${*:2}=(Y Z W) ${@:2}=(Y Z W)

4. BUG: ${!prefix_@} in a function fails in the second invocation

See the following test case. Even though it succeeds for the first time, it fails for the second time.

test3.sh

#!/usr/bin/env bash

abc_hello=1
function hello1 { echo "${!abc_@}"; }
hello1
hello1

$ osh test3.sh
abc_hello
  function hello1 { echo "${!abc_@}"; }
                             ^~~~
test3.sh:4: fatal: Bad indirect expansion: ''

[andychu]

This is great, thanks for all the patches! About the first one, I just changed something there, and I agree the behavior is odd, but I see bash and OSH agree (at HEAD):

$ bin/osh --rcfile /dev/null -c 'test -o emacs; echo $?; test -o vi; echo $?'
1
1
$ bash  --rcfile /dev/null -c 'test -o emacs; echo $?; test -o vi; echo $?'
1
1

$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)

Do you get a different behavior?

I agree they are mutually exclusive but they both start off OFF. bash has a weird conceptual model but I think OSH copies it.

[akinomyoga]

Thank you for your comment!

I agree they are mutually exclusive but they both start off OFF. bash has a weird conceptual model but I think OSH copies it.

The default value in Bash depends on whether Bash is started in a non-interactive session or an interactive session. When you use the form sh -c '...', the shell starts in a non-interactive session while ble.sh works in an interactive session. In non-interactive sessions, test -o emacs/vi neither succeeds in Bash, which reflects the fact that the line editing is not activated in non-interactive sessions. This is reasonable. While, in interactive sessions, the emacs line-editing is activated by default, so it is reasonable that test -o emacs succeeds.

On the other hand, test -o emacs/vi neither succeeds in Oil by default even in the interactive session where the line-editing is enabled. The behavior of Oil in which test -o emacs/vi fails in both interactive and non-interactive sessions is strange.

By the way, the line-editing can be turned off in interactive sessions of Bash. The setting is set +o emacs. This is the reason why set +o emacs doesn't imply set -o vi in Bash unlike in Oil.

[andychu]

Ah OK I reproduced this with -i, see #659. If this is blocking more testing let me know

[akinomyoga]

Good news. I think I could finally source ble.sh without fatal errors (with many fixes). But of course, I still get many warning messages. Also, what has been done so far is just source without starting the ble.sh session, so ble.sh is still unlikely to run on Oil, but now it is possible to test some parts of ble.sh. I list up the problems that I encountered so far as follows:

5. INCONSISTENCY: Cannot create an empty associative array

Oil rejects declare -A dict=() with an error while it accepts declare -A dict; dict=(), which appears to be inconsistent. Note that declare -A dict and declare -A dict=() have different results in Bash. We want to have consistency. It is inconvenient that one cannot initialize an empty associative array

$ osh -c 'declare -A dict; dict=()'
$ osh -c 'declare -A dict=()'
  declare -A dict=()
             ^~~~~
[ -c flag ]:1: Got -A but RHS isn't an associative array

6. NYI (not yet implemented): exec <> a.txt fails without error messages

Even if it is not implemented, we want at least one error message (including the line number, etc.) instead of a stack trace. ble.sh uses this on named pipes (e.g. mkfifo a.fifo; exec <> a.fifo).

$ osh -c 'exec <> a.txt'
Traceback (most recent call last):
  File "/home/murase/prog/ext-github/oilshell.oil/bin/oil.py", line 962, in <module>
    sys.exit(main(sys.argv))
  File "/home/murase/prog/ext-github/oilshell.oil/bin/oil.py", line 908, in main
    return AppBundleMain(argv)
  File "/home/murase/prog/ext-github/oilshell.oil/bin/oil.py", line 881, in AppBundleMain
    status = ShellMain('osh', argv0, main_argv, login_shell)
  File "/home/murase/prog/ext-github/oilshell.oil/bin/oil.py", line 697, in ShellMain
    status = main_loop.Batch(ex, c_parser, arena, nodes_out=nodes_out)
  File "/home/murase/prog/ext-github/oilshell.oil/core/main_loop.py", line 166, in Batch
    is_return, is_fatal = ex.ExecuteAndCatch(node)
  File "/home/murase/prog/ext-github/oilshell.oil/osh/cmd_exec.py", line 1851, in ExecuteAndCatch
    status = self._Execute(node, fork_external=fork_external)
  File "/home/murase/prog/ext-github/oilshell.oil/osh/cmd_exec.py", line 1786, in _Execute
    if self.fd_state.Push(redirects, self.waiter):
  File "/home/murase/prog/ext-github/oilshell.oil/core/process.py", line 372, in Push
    if not self._ApplyRedirect(r, waiter):
  File "/home/murase/prog/ext-github/oilshell.oil/core/process.py", line 255, in _ApplyRedirect
    raise NotImplementedError(r.op_id)
NotImplementedError: 159

7. NYI: Missing feature exec {fd}> a.txt

This is a Bash-4.1 feature. ble.sh uses this for Bash 4.1 and later.

$ bash -c 'exec {fd}> a.txt; echo hello >&$fd; cat a.txt'
hello
$ osh -c 'exec {fd}> a.txt; echo hello >&$fd; cat a.txt'
  exec {fd}> a.txt; echo hello >&$fd; cat a.txt
       ^
[ -c flag ]:1: exec: '{fd}' not found

8. NYI: Missing feature closing file descriptors exec >&-

Without this, one cannot close file descriptors.

$ osh -c 'exec 5> a.txt; exec 5>&-'
  exec 5> a.txt; exec 5>&-
                         ^
[ -c flag ]:1: Redirect descriptor should look like an integer, got (value.Str s:-)

9. NYI: Missing feature of read delimiters read -d ''

$ osh -c 'read -r -d "" < a.txt; read -r -d "," < a.txt'
  read -r -d "" < a.txt; read -r -d "," < a.txt
          ^~
[ -c flag ]:1: 'read' doesn't accept flag -d
  read -r -d "" < a.txt; read -r -d "," < a.txt
                                 ^~
[ -c flag ]:1: 'read' doesn't accept flag -d

10. Error message of builtin read is not friendly

It says 'builtin' doesn't accept instead of 'read' doesn't accept.

$ osh -c 'builtin read -d ""'
  builtin read -d ""
               ^~
[ -c flag ]:1: 'builtin' doesn't accept flag -d

11. BUG: trap doesn't ignore --

I can find similar behavior for eval though I haven't checked the other builtins. In Bash, when one needs to pass an arbitrary value to trap / eval, one needs to specify -- because, for example, command=--help; eval "$command" will not be interpreted as expected.

$ osh -c 'trap "echo hello" INT'
$ osh -c 'trap -- "echo hello" INT'
  trap -- "echo hello" INT
          ^
[ -c flag ]:1: Invalid signal or hook 'echo hello'
$ osh -c 'eval -- echo hello'
  -- echo hello
  ^~
[ eval at line 1 of -c flag ]:1: '--' not found

12. NYI: Missing support of %()T in printf (unimplemented)

$ osh -c 'printf "%(%s)T\n"'
  %(%s)T\n
   ^
(source.ArgvWord word_spid:2):1: Invalid printf format character

13. BUG: ${var=x} is local

Even though an assignment of a shell variable var=x in a function scope takes an effect in the outer scope variable, the effect of ${var=x} is restricted in local. ble.sh uses this in some functions. Or do you have a rationale for this? Though it can be easily worked around, I don't see the reason for the design.

$ osh -c 'f() { : "${hello:=x}"; echo $hello; }; f; echo $hello'
x

$ osh -c 'f() { hello=x; }; f; echo $hello'
x

14. Bash/Oil differences

There are other warnings related to the features which I think are out of the scope of Oil.

14.1 Recursive arithmetic evaluation

The most warnings are caused by missing recursive evaluation described in #648.

14.2 Empty variables in arithmetic expressions

The second most type of warnings is this.

$ osh -c 'a=; echo $((a))'
[??? no location ???] warning: Invalid integer constant ''

14.3 Missing support of bind

It is used for setting up readline variables.

      ((_ble_bash>=40100)) && builtin bind 'set skip-completed-text on'
                                      ^~~~
./ble.sh:16300: 'bind' isn't a shell builtin
      ((_ble_bash>=40300)) && builtin bind 'set colored-stats on'
                                      ^~~~
./ble.sh:16301: 'bind' isn't a shell builtin
      ((_ble_bash>=40400)) && builtin bind 'set colored-completion-prefix on'
                                      ^~~~
./ble.sh:16302: 'bind' isn't a shell builtin

14.4 Output format of declare -p

This problem described in #647 already causes many warnings and strange behavior while source ble.sh --attach=none.

[andychu]

Thanks this is very useful. I just fixed point 13 -- it was an unintentional divergence.

read -d is #356

I'll respond to the rest a bit later. More bugs/testing is welcome!

[andychu]

Also what's the difference between declare -A dict and declare -A dict=() in bash?

By the way I will merge failing spec tests even if there's no fix yet. Showing the difference is often 50% of the work.

The way I find the relevant tests is to do grep foo spec/*.test.sh