Failure to launch external commands due to race condition in 'posix_spawn' usage

[aweeraman]

There has been a bug report of an exec format error when executing "cat TODO | while read line; do ls; done".

Ref:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965072
• https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863

I was able to reproduce this issue on 93u+m 2020-07-14 and it appears to be an issue at least since 93u+ 2012-08-01.

$ cat TODO
1
2
$ cat TODO | while read line; do ls; done
ls: ls: cannot execute [Exec format error]
ls: ls: cannot execute [Exec format error]

Tagging @chrisbertin and @mirabilos - reporters of the issue.

[posguy99]

Can't reproduce it here on macOS 10.14.6..

[02:28 PM][ttys001][~/ksh-test]
[34] iMac $ echo $KSH_VERSION
Version AJM 93u+m 2020-07-16
[02:29 PM][ttys001][~/ksh-test]
[35] iMac $ cat TODO
1
2
[02:29 PM][ttys001][~/ksh-test]
[36] iMac $ cat TODO | while read line; do ls; done
TODO
TODO
[02:29 PM][ttys001][~/ksh-test]
[37] iMac $ /bin/ksh
[02:29 PM][ttys001][~/ksh-test]
[38] iMac $ echo $KSH_VERSION
Version AJM 93u+ 2012-08-01
[02:29 PM][ttys001][~/ksh-test]
[39] iMac $ cat TODO
1
2
[02:29 PM][ttys001][~/ksh-test]
[40] iMac $ cat TODO | while read line; do ls; done
TODO
TODO
[02:29 PM][ttys001][~/ksh-test]
[41] iMac $ 

The slightly different test in the Ubuntu bug doesn't fail either.

[mirabilos]

Marc Wilson dixit:

Can't reproduce it here on macOS 10.14.6..

According to the reporter, they get a working ksh93 if they disable posix_spawn configure option so it may be a, very widespread, bug in GNU libc (Linux/kFreeBSD/Hurd-only, I presume).

[JohnoKing]

According to the reporter, they get a working ksh93 if they disable posix_spawn configure option so it may be a, very widespread, bug in GNU libc (Linux/kFreeBSD/Hurd-only, I presume).

I can confirm that disabling posix_spawn causes this bug to go away. OpenSUSE disables it in their build of ksh with a patch titled ksh-qemu.patch, which implies there is another bug related to posix_spawn that affects QEMU.

[McDutchie]

On macOS, the compiler flags used by the Apple version (which I've incorporated into src/cmd/INIT/cc.darwin) disable SHOPT_SPAWN. I can confirm that, if I change that flag to -DSHOPT_SPAWN=1 and recompile from scratch, I can reproduce this bug on the Mac.

I can also reproduce this bug on FreeBSD, on which SHOPT_SPAWN is not disabled by default.

Neither of these two systems use GNU libc, so it seems likely that this is a bug in ksh and not in any C library. Does anyone here have an idea how to fix it?

Alternatively, I wonder if there would be any disadvantage to simply removing the use of posix_spawn altogether.

[McDutchie]

Minimal reproducer for regression test:

printf '%s\n' 1 2 3 | while read line; do ls /dev/null; done

Normal output:

/dev/null
/dev/null
/dev/null

Bug output:

/dev/null
ls: ls: cannot execute [Exec format error]
ls: ls: cannot execute [Exec format error]

[McDutchie]

Well, I think the advantage of using posix_spawn(2) is made clear by the following:

$ arch/*/bin/ksh -c 'time for ((i=1; i<=10000; i++)); do /usr/bin/true; done'

real	0m22,02s
user	0m07,42s
sys	0m11,15s
$ ./ksh_SHOPT_SPAWN -c 'time for ((i=1; i<=10000; i++)); do /usr/bin/true; done'

real	0m13,66s
user	0m04,43s
sys	0m07,66s

So it would be really nice if someone could figure out how to fix ksh's use of posix_spawn.

[chrisbertin]

A few observations: If input is redirected, instead of piped, no errors. The invalid behavior is not 100% consistent. Running a failing command in a sub-shell works (which is the workaround I have been using). Using built-in commands work. sixt $ cat > /tmp/list 1 2 3 sixt $ while read f; do ls -ld /; done < /tmp/xx drwxr-xr-x 30 root root 4096 Jul 14 08:08 / drwxr-xr-x 30 root root 4096 Jul 14 08:08 / drwxr-xr-x 30 root root 4096 Jul 14 08:08 / sixt $ cat /tmp/list | while read f; do ls -ld /; done drwxr-xr-x 30 root root 4096 Jul 14 08:08 / /bin/ls: /bin/ls: cannot execute [Exec format error] /bin/ls: /bin/ls: cannot execute [Exec format error] sixt $ cat /tmp/list | while read f; do ls -ld /; done /bin/ls: /bin/ls: cannot execute [Exec format error] /bin/ls: /bin/ls: cannot execute [Exec format error] /bin/ls: /bin/ls: cannot execute [Exec format error] sixt $ ksh -c 'cat /tmp/xx | while read f; do ls -ld /; done' drwxr-xr-x 30 root root 4096 Jul 14 08:08 / drwxr-xr-x 30 root root 4096 Jul 14 08:08 / drwxr-xr-x 30 root root 4096 Jul 14 08:08 / sixt $ cat /tmp/list | while read f; do echo $f /; done 1 / 2 / 3 / sixt $ Chris (chris.bertin@gmail.com or chris.bertin@protonmail.com)

…

On Fri, Jul 17, 2020 at 2:24 PM Anuradha Weeraman ***@***.***> wrote: There has been a bug report of an exec format error when executing "cat TODO | while read line; do ls; done". Ref: - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965072 - https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863 I was able to reproduce this issue on 93u+m 2020-07-14 and it appears to be an issue at least since 93u+ 2012-08-01. $ cat TODO 1 2 $ cat TODO | while read line; do ls; done ls: ls: cannot execute [Exec format error] ls: ls: cannot execute [Exec format error] Tagging @chrisbertin <https://github.com/chrisbertin> and @mirabilos <https://github.com/mirabilos> - reporters of the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#79>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQKGJUJOFTOVXOBFI6ZRDYDR4C6RPANCNFSM4O65E6KA> .

[posguy99]

Is this 2020 bug relevant?

att#470
which leads to
att#468

and the original ast ML discussion:
https://www.mail-archive.com/ast-developers@research.att.com/msg00717.html

[McDutchie]

Thanks @posguy99, yes, that is very relevant indeed.

So that is another problem with using posix_spawn(2): it "doesn't have a way to set the terminal group before the exec()". That might be fine for many use cases, but clearly that's unacceptable behaviour for a shell!

The RATIONALE section of the POSIX spec for posix_spawn(2) also makes for interesting reading. Note that the spec title itself is marked "ADVANCED REALTIME", indicating that this is intended for real-time applications (which a shell very much is not, no matter how fast it is). Some quotes:

[…] although they may be an efficient replacement for many fork()/ exec pairs, their goal is to provide useful process creation primitives for systems that have difficulty with fork(), not to provide drop-in replacements for fork()/ exec.
[…] We can identify several problems with posix_spawn() and posix_spawnp(), but there does not appear to be a solution that introduces fewer problems. Environment modification for child process attributes not specifiable via the attrp or file_actions arguments must be done in the parent process, and since the parent generally wants to save its context, it is more costly than similar functionality with fork()/exec.
[…] The fork() function is a wonderfully powerful operation. We do not expect to duplicate its functionality in a simple, fast function with no special hardware requirements. It is worth noting that posix_spawn() and posix_spawnp() are very similar to the process creation operations on many operating systems that are not UNIX systems.

So, while running /usr/bin/true in a loop might be quite a bit faster, it looks clear that it's not worth the cost in bugs and undefined behaviour. And if you run a command that actually does something, that performance difference quickly approaches insignificance. So, this is another instance where the ksh2020 developers made the right call. We must get rid.

On the plus side, that makes fixing this bug quite simple.

[posguy99]

Pointing out, tho, that if you follow those 2020 bugs all the way through, they lead you back to the performance degradation that was an large part of the original ksh2020 kerfuffle.

[McDutchie]

Agreed. That's not going to happen though.