The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.
It seems like this 2019 paper covers this point, and the content in the gist? I was expecting to see a reference to it
A fork() in the road
https://dl.acm.org/doi/abs/10.1145/3317550.3321435
Discussed at the time: https://news.ycombinator.com/item?id=19621799
Although it does say that vfork() is difficult to use safely, while the gist recommends it? I think there is still some clarity needed around the use cases.
Fork today is a convenient API for a single-threaded process with a small memory footprint and simple memory layout that requires fine-grained control over the execution environment of its children but does not need to be strongly isolated from them. In other words, a shell. It’s no surprise that the Unix shell was the first program to fork [69], nor that defenders of fork point to shells as the prime example of its elegance [4, 7]. However, most modern programs are not shells. Is it still a good idea to optimise the OS API for the shell’s convenience?
I too could use some more clarity around the use cases
https://news.ycombinator.com/item?id=30510519
The gist seems to be from 2017 so it wouldn't have been able to reference that paper.
As u/amaranth pointed out, my gist predates the MSFT paper, which mostly explains why I didn't reference. Though, to be fair, I saw that paper posted here back in 2019, and I commented on it plenty (13 comments) then. I could have edited my gist to reference it, and, really, probably should have. Sometime this week I will add a reference to it, as well as this and that HN post, since they are clearly germane and useful threads.
I vehemently disagree with those who say that vfork() is much more difficult to use correctly than fork(). Neither is particularly easy to use though. Both have issues to do with, e.g., signals. posix_spawn() is not exactly trivial to use, but it is easier to use it correctly than fork() or vfork(). And posix_spawn() is extensible -- it is not a dead end.
My main points are that vfork() has been unjustly vilified, fork() is really not good, vfork() is better than fork(), and we can do better than vfork(). That said, posix_spawn() is the better answer whenever it's applicable.
Note that the MSFT paper uncritically accepts the idea that vfork() is dangerous. I suspect that is because their focus was on the fork-is-terrible side of things. Their preference seems to be for spawn-type APIs, which is reasonable enough, so why bother with vfork() anyways, right? But here's the thing: Windows WSL can probably get a vfork() added easily enough, and replacing fork() with vfork() will generally be a much simpler change than replacing fork() with posix_spawn(), so I think there is value in vfork() for Microsoft.
Use cases for vfork() or afork()? Wherever you're using fork() today to then exec, vfork() will make that code more performant and it generally won't take too much effort to replace the call to fork() with vfork(). afork() is for apps that need to spawn lots of processes quickly -- these are rare apps, but uses for them do arise from time to time. But also, afork() should be easier to use safely than vfork(). And, again, for Microsoft there is value in vfork() as a smaller change to Linux apps so they can run well in WSL.
BTW, see @famzah's popen-noshell issue #11 [0] for a high-perf spawn use case. I linked it from my gist, and, in fact, the discussion there led directly to my writing that gist.
If you are going to edit, the google query links with the #q=xyz format no longer seem to work, so maybe update them to the ?q=xyz format which still works.
(Also this article and discussions on it now take up many of the top spots, which I guess is the disadvantage to linking to google for a topic)
I've updated the gist to include that, this, and many other links.
> "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells."
Yes, but why is this characterized as something negative?
Isn't that the entire point? Operating systems are there to serve user requests, and shells are an interface between user and OS.
Shells simply developed features that users required of them.
Shells haven't been the primary interface between the user and the OS for decades.
"The primary interface between the user and the OS" is the definition of "shell". That's why the Microsoft Windows process that draws the Start button and filesystem windows is called "the Windows shell".
I don't think OP meant shell as in the Windows shell, or Linux DEs. I mean, how many of those use fork() even on Linux, or would be easier to implement if they did?
Linux desktop environments do use fork(), and the Microsoft shell doesn't use fork() because Microsoft Windows doesn't have it.
In the Linux context, the fact that random things inherit stdout appending to .xsession-errors and inheriting environment variables is often useful. fork() also makes it fairly straightforward to do things like set a VM size limit or change an environment variable for a newly launched program, which is often useful when you're launching a program from just about anything. I don't know whether rearchitecting Microsoft Windows to work that way would have made the Windows Shell easier to write.
However, and this is the crucial point, fork() was impossible to support on Win16, because segment register values can be stashed anywhere in your 8086 program's memory, and they're just literally added to the offset address with a 4-bit shift, so there's no reliable way to make a copy of a running process elsewhere in memory that doesn't accidentally share segments with the original. You'd have to do what monocasa was saying old Unix did and checkpoint the process to disk. (I suspect Unix never did that, but it's similar to what PDP-11 Unix did do.)
Which Linux DEs use fork without exec?
Inheriting stdout etc does not require fork. It requires a spawn API that has a flag to inherit stdout, such as e.g. Win32 CreateProcess. Inheriting handles by default, on the other hand, is a recipe for hard-to-debug bugs.
Oh, I didn't mean without exec, but there are some programs like gnome-terminal that do that too. I just meant that forking, doing process configuration with system calls to open and close files and whatnot, and then running exec, is maybe a more convenient way to launch a program in a modified environment, than having a CreateProcess system call with fifty zillion flags.
Everything in Unix is a recipe for hard-to-debug bugs.
Shells have relatively simple operational models, so _any_ API would probably be workable for shells.
Meanwhile, programs with more complex requirements have to work around these APIs. And many programs call other programs, or otherwise have to do tricky process lifecycle management.
The lowest-level APIs should, in theory, cater to the most complex cases, not to the simplest ones. This doesn't prevent a simpler API from existing, but catering to a simple use case in the primitives does hinder more complex needs.
(I think the more nuanced point is that the OS itself might not have a much better design available in any case. Unixes have a lot of neat stuff, but it's a lot of "design by user feature request", and "standardize 4 slightly different ways of doing things", so there is a lot of weirdness and it's hard to have The Perfect API in that case)
> Shells have relatively simple operational models, so _any_ API would probably be workable for shells.
You'd think that, but implementing the UNIX shell and all of its semantics (piping, redirection, waiting, child reaping, jobs, foreground/background, prompting etc) using fork/clone + exec* is way more simple than, say, on Windows. Some API designs are better for that specific task
> Shells have relatively simple operational models, so _any_ API would probably be workable for shells.
True. Today anyways. Back in the 70s though, there was a lot of innovation going on around process spawning, and fork+exec almost certainly made it easy to play with those ideas. I'm referring to job control, for example. But also things like the parent-child relationships between the shell and all the processes in a pipeline -- not all shells have set those up the same way.
So, yeah, maybe we need not just posix_spawn() but posix_pipeline_spawn(), why not. Make it even easier to write a shell. After all, plumbing a complex pipeline with posix_spawn() requires a fair bit of code.
Will any API do? Yes, provided it covers all the things Unix shells do nowadays. It's still easiest to get all the functionality (that a shell dev might want to build) with fork+exec though, especially since the shell author gets a great deal of control that way, though they get that at the price of having to know a great deal of stuff intimately. Arguably, anyone wishing to implement a posix_pipeline_spawn() would be like a shell developer.
The thing is that there are many other programs which require process control, which are not shells. Orders and orders of magnitudes of programs which are not shells. So we can optimize an API for building shells, but it's not going to make writing those other programs easier.
Shells are cool and good, and I don't want to discount fork too much, just saying that the API design space isn't _only_ for shells.
> Isn't that the entire point?
The exokernel people would disagree.
You see, an operating system as commonly conceived has at least two major jobs:
- abstract away underlying hardware
- safely multiplex resources
And do the above with as little overhead as possible.
Now the thing is: whenever you have multiple goals, you need to make trade-offs, and you aren't as good at any one goal as you could be.
So the exokernel folks made a suggestion in the 90s: let the OS concentrate on safely multiplexing resources, and do all the abstracting in user level libraries.
See eg https://www.classes.cs.uchicago.edu/archive/2019/winter/3310... or https://people.eecs.berkeley.edu/~kubitron/cs262/handouts/pa...
Normal application programming would mostly look the same as before, your libraries just do more of the heavy lifting. But it's much easier to swap out different libraries than it is to swap out kernel-level functionality.
That vision never caught on with mainstream OSes. But: widespread virtualisation made it possible. You can see hypervisors like Xen as exokernel OSes that do the bare minimum required to safely multiplex, but don't provide (many) abstractions.
> Yes, but why is this characterized as something negative?
Unfortunately, the text does not provide sufficient context. Shell are not properly supported in any OS (probably except plan9), since 1. the OS provides no enforcement or convention of CLI API interface (there is no enforced encoding standard or checkable stuff), 2. the OS provides no rules for file names to be shell-friendly and 3. there are no dedicated communication channels towards shells or in between programs and shells.
So all in all, shells remain a hack around the system that is "simple to implement the initials" and is annoying to use and write at many corner cases.
> Shells simply developed features that users required of them.
Cross out "simply" and call it convenience+arbitrary complex scripting glue for 4 main goals: 1. piping 2. basic text processing 3. basic job control 4. path hackery
That is the most glorious ** that i've read all day.
Larry Wall, creator of Perl, famously wrote that "It is easier to port a shell than a shell script."
https://en.wikipedia.org/wiki/Shell_script
So we can write operating systems easily if it's just an infinite superloop?