Strong resonance with the famous essay "The Rise of Worse is Better" [1], which contrasted the (better) "MIT/Stanford style of design" with the (worse) "New Jersey approach".
MIT/Stanford:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the interface to be simple than the implementation.
New Jersey:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.
TFA maps "simplicity" to "MIT/Stanford simplicity" (simplicity for the user) and "smallness" to "New Jersey simplicity" (simplicity for the developer).
I wonder if the root of the tension between the two schools comes down to the ambiguity of the user/developer distinction. Developers are also users. Simplicity of implementation is helpful to developers when they are working directly on implementation, while simplicity of interface is helpful to developers when they are using other developers' work.
I think you're right on that point of contention, it's too far to assert a universal and clear good/bad dynamic here because that line between developer/user is contextual and fuzzy.
Another interesting irony I'll note, Lisp is the "New Jersey approach" towards symbolic AI. Americans clinging to their Lisp systems were deeply entrenched in a "worse-is-better" mindset. Your interface, the computational model, didn't need to be designed for logic programming, that was wholly secondary. Do everything as much as possible in Lisp, and then offload the relational description to a small (not simple) library. American knowledge engineers were looked at as overpaid procedural hackers with zero mathematical elegance and very little credibility. More or less the same perception these self-same Lisp-machine users had for Unix and C programmers.
It's all about perspective, at the end of the day. Where we draw the line in the sand on these categories is free-to-choose, yet it also determines everything. We're always someone else's villain under different semantics.
I appreciate the New Jersey simplicity as a user (with development skills) too, though. It's usually just a matter of time before I have to dive into the program/library/whatever internals to fix a bug.
Pretty much anything. I mean, even consider a text editor, monospace font, no syntax highlighting. It's already pretty daunting. Doable, certainly, but a big task. We're lucky people already made some, and we can copy their designs even when we choose not to copy their exact code.
From the authors, Charles Simonyi went to Microsoft as one of the original Word implementers, while Excel was born on Mac OS and only later ported to Windows.
> There's no native Unix equivalent to frequencies, this sort | uniq -c is the closest we can get. Not only is it less performant (it has to collect the full input into memory before continuing), but it ties aggregation to ordering.
One of the core features of the Unix command-line is that it is user-extensible. If there's no "native" command equivalent to frequencies, you can write your own, and it will be given the same first-class treatment as any other binary in your PATH. This is entirely in keeping with the Unix philosophy of simple implementations.
Yeah, that point was weak. The argument went from "Unix pipelines are not simple" to "unix based operating systems don't have an equivalent to Clojure's frequency function by default so it's worse"
OK but how would it look like if you had such a program? Shells are not known for having the extensive set of functions that real programming languages have.
I think the point was more that it's hard to compose shell utilities to make something new that's not supported out of the box. Instead you do have to write an entirely new utility and use that.
That said having key/value semantics and not just stream of bytes would make the shell much more versatile, at the cost of making it bigger.
That's not what the linked article is saying. The point made in the article is that the 'uniq' utility couples together two unrelated concepts--sorting and frequency counting--and this coupling forces you to write more complex code, compared to the version where frequency counting is decoupled from sorting.
All of this is completely correct, but it has nothing to do with bash vs. Clojure. The same decoupling can easily be achieved in either language.
I think the article is a bit weak when it's making this point, because it's not about Unix pipelines. It's about POSIX shell utilities being too bare-bones.
But Unix pipelines are not simple too. They have a couple of nitty-gritty details that often come out and bite you.
1. They can only stream raw bytes, so all the programs that deal with lists like sort and uniq have to separate items using a delimiter (usually newline). If you want to process data with that delimiter in it, you're in for a ride. And if you want to write a custom tool, you have to do all the splitting yourselves (luckily it's so common most programming language will provide a ready-made facility for you to do that). This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".
2. Error are hidden by default in shell. Nowadays you can explicitly change this behavior, but you have to remember to do `set -o pipefail` and I don't think it was always there.
3. There is no data typing at all. Everything is binary or text. Nowadays a lot of programs just output JSON, and the users (if they even stay inside the shell) almost always reach for jq to parse it. But jq is not a Unix philosophy program: it's an entire streaming functional programing that can do quite a lot. But even jq gets hairy when you have to do a bigger query or transformation. In that case users often reach out to Python or another language and just move the business logic there.
I think this fits well with what the article is trying to say: Unix pipes are pretty easy (small) to implement on your own (compare that to something like Nushell's pipes). But the moment you to do something that's a little different than the happy path it was built for, you need to go for another tool (jq) that has its own built-in pipe and small programming language, because Unix pipes won't cut it. And for more complex (hehe) things, you'll have to reach for a larger (and simpler) tool: a full-fledged programming language.
> This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".
I'm not sure if you're actually trying to argue for a specific position here, but you highlight several negative results of one "philosophy" of development, with the implication that the alternative wouldn't have those negatives.
This is a tricky point to refute because you're right, these are flaws and there could be a system that doesn't have them.
So why do these flawed systems exist and proliferate?
Because the real life result isn't actually a choice between "sloppy but quick to develop" and "elegant well engineered but slow".
The choice is actually between "sloppy but exists" and, well, nothing, because the other version never actually materializes.
(And of course, I feel compelled to point out that bash is a user interface not a programming language. Any attempt to replace it or improve it without focusing on that main point is doomed to failure, which is why you see so many people who apparently think that what bash really needs is strict type checking or something and end up creating a completely awful user experience)
This was my thought, too. Either you can extend your toolkit—make a utility like dedup (https://codeberg.org/napcakes/dedup) and put in on your path (yay! so easy)—or you can't and thus must define the boundary of the allowed toolkit. In this case, it seems overly disingenuous to not allow awk, which will do the thing for you just fine, no Clojure needed at all. awk is hashmaps galore.
Honestly, so much of our conceptions of "what's wrong" is more to do with lack of familiarity with history than any actually unsolved problem.
Great piece, very straightforward examples, although I did have to squint for quite a while to grok the Closure portion.
I am currently building a piece of very modular software and it has been the hardest-to-design project of my entire career. I would never be allotted this amount of time-effort at any job I have held to make something this robust and clearly defined. Many aspects of this project have taken 3-5 rounds trying-and-trashing to get an abstraction that is uncomplicated.
This is precisely why vibe coding is so successful for building tiny isolated scripts, and so disastrous for anything else. It's just really dang hard to build something large and simple.
Personally, I'm big fan of those pipes mentioned. And the tooling that can be created around those _simple_ "primitives". Comparison to Google drive is obscure, in a way that it compares one gigantic piece of software into these small and simple.
Bottom line is probably true, but if you are an open-source maintainer mentioned, and you have only so few hours to spend, you just cannot create those gigantic softwares either. You need to choose from the cards on your hand.
> This is precisely why vibe coding is so successful for building tiny isolated scripts, and so disastrous for anything else. It's just really dang hard to build something large and simple.
Brooks, in “the Mythical Man-Month”:
“One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported
for industrial teams.
Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.
In the upper left of Fig. 1.1 is a program. It is complete in itself, ready to be run by the author on the system on which it was developed. That is the thing commonly produced in garages, and that is the object the individual programmer uses in estimating productivity.
There are two ways a program can be converted into a more useful, but more costly, object. These two ways are represented by the boundaries in the diagram.
Moving down across the horizontal boundary, a program becomes a programming product. This is a program that can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data. […] As a rule of thumb, I estimate that a programming product costs at least three times as much as a debugged program with the same function.
Moving across the vertical boundary, a program becomes a component in a programming system. This is a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks. To become a programming system component, a program must be written so that every input and output conforms in syntax and semantics with precisely defined interfaces. The program must also be designed so that it uses only a prescribed budget of resources — memory space, input-output devices, computer time. Finally, the program must be tested with other system components,
in all expected combinations. […] A programming system component costs at least three times as much as a stand-alone program of the same function. The cost may be greater if the system has many components.
In the lower right-hand corner of Fig. 1.1 stands the programming systems product. This differs from the simple program in all of the above ways. It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts.”</i>
> The reason for this is that in Rust, a struct couples type-checking to a fixed data representation. You can't get one without the other.
> Clojure decouples data representations from type checking.
This is funny to me because seen from the other side, (this) Clojure couples runtime type information to data structures: you're no longer allowed to define a data structure that doesn't have some runtime type information attached. A fixed static structure is just the consequence of not adding dynamic type information.
Meanwhile in Rust you can get type-checking ‘without’ a fixed structure by using trait objects.
I think all of TFA was flimsy. This coupling of which TFA is bad, somehow? I don't even see a good definition of "coupled", nor a good argument of why/how "uncoupling" makes for simple and small.
Regardless of whether the type information is static or dynamic, you're still coupling some type to some data. The type is still implicit even after compilation; there still exists a structure to the data, even if that structure isn't easily discerned without the source. Or to put it another way: just because there's no runtime type information, doesn't mean that the data now is entirely decoupled from the type.
Using "length of the correctness statement + length of its proof" works quite well as proxy for complexity of a component (the longer, the more complex).
Copy pasted functions with subtle changes mean you cannot reuse the proof (DRY). Giant functions with lots of if/else statements however might cause a branch explosion in the proof. The right abstraction removes lots of assumptions that a proof could depend on, limiting the search space and often forcing elegance (this also applies to math, eg. when reasoning with abstract groups instead of integers).
The wrong abstraction might force case distinctions on consumers of the abstraction.
> Using "length of the correctness statement + length of its proof" works quite well as proxy for complexity of a component (the longer, the more complex).
It sounds like a reasonable concept, but then Principia Mathematica takes 300 pages to prove that 1+1 is 2.
Important to note as well, is that "simple" isn't "lesser" or "dumber", it can be "more" and "wider", yet still "simpler".
Expectedly, Rich Hickey explains it best, watch the "Simple Made Easy" talk if you haven't before, one of the few talks I probably watch bi-yearly: https://www.youtube.com/watch?v=SxdOUGdseq4
Few things, concepts and ideas have changed as much of my programming mind as Hickey's talk and ultimately Clojure have done over the years.
Wish we had new amazing Hickey talks to link to, maybe it seems he's about the hang up the hammock perhaps?
My reaction to the Unix pipeline was that, the reason it exploded in complexity is because the pieces were too simple. They were insufficiently expressive.
But the word is used in a different way here, and I'll have to watch the talk to understand what exactly is meant. (Something like orthogonality?)
Agreed. There are multiple senses for "simple" and on of those senses is "small". It is actually a very useful sense too, as Solomonov/Levin/Kolmogorov/Chaitin-style inductive reasoning has shown.
And it is important not to just make snappy quips by equivocating.
I don’t know. This reads more like a “Clojure is great” post, and Clojure is great. But the author takes a swing and a miss on the Rich Hickey magic. The UNIX example is contrived (the number of occurrences in the order they occur?), and trying to redefine simple in a way that excludes UNIX pipelines doesn’t work.
>the reason it exploded in complexity is because the pieces were too simple
the reason it exploded in complexity is because the complexity is in the interaction of the parts, not the properties of the parts. It's not so much about expressiveness but about the fact that the Unix philosophy has a willful disregard for systems thinking.
Russ Ackhoff has the analogy of the house. How do you design rooms in a house? Only ever with the house in mind. You can't just hope the rooms work, you always need to check whether a new room improves the house, if somethings wrong you fix the room, not the house.
Unix tools have a disregard for the house. Clojure works well because every part of Clojure is designed not just to improve itself but to improve Clojure. Emacs is similar in that regard, you can see that when people make Emacs packages. The good ones always have Emacs in mind, not just their own functionality.
The author uses the term “coupling” to describe a simple program from the code’s perspective - “you need to have a good mental model of your program”. I agree, but I’d argue that more importantly you need to have a good domain model. Good design communicates accurately and a high Gulf of Evaluation/Execution are actually what makes programs feel “complex” to a user.
These concepts simplify for the user these questions: “I know what I want - how do I make the program do it?” (Execution) and “The program did something — what state is it actually in?” (Evaluation)
I believe the author was getting at these concepts, especially in their Google Drive example - how the large program has a “small” UX. Understanding the domain model provides a much stronger basis for designing user interfaces, and understanding the Gulf of Evaluation/Execution allows you to build incredibly complex-looking, large UX’s without confusing or overwhelming the user.
Honestly, having watched people argue about what simple is for about the last 10 years, I've pretty much settled on it not being a well-defined term. We know complex when we see it for sure, at least when it is present in quantity, but simplicity is not just the absense of complexity. There's at least three concepts we're all trying to stuff into the same word, and they are not only not "orthogonal" they are often in conflict with each other. I don't even think it can be rehabilitated, it can only really be abandoned, to clear the way to trying to characterize the multiple concepts we're trying to stuff into this one word.
It is especially dangerous when something is "good" and people try to appropriate the term to appropriate the goodness of the term, as if goodness flows from a term to the thing it is attached to rather than the other way around. "Simple" is good so my good thing must be "simple" to be "good". But it doesn't. Simple can even be bad, in the wrong place or in the wrong sort of "simple" for a given job.
"Few tokens" - perhaps the most literal simplicity, literally, it doesn't use many tokens to do the job. But as the article points out, that doesn't necessarily fit with...
"Easy to reuse" - This is that simplicity that functional programming aspires to, where you craft some precise abstraction that somehow captures something like "monad". Haskell is full of this sort of simplicity, oozing out of every pore, but people generally think of it as a very complex and hard langauge, contrasting...
"Easy to understand" - As in, not cognitively complex. It is amazing how quickly things that I would otherwise describe as very simple still blow out our little minds. Consider the first time you saw quicksort... or even how it feels now. It's not a lot of tokens, but it's twisty and recursive and especially if you're not mathematically trained and in practice it's easy to call it more "complicated" than a CRUD form that takes in and validates 10 parameters in a straightforward way, even though in terms of what is actually happening the CRUD form may be doing vastly more than the little quicksort algorithm. It just isn't being twisty, recursive, and subtle in how it does it.
And I'm just filling out the first three that come to mind. Note these are not always in conflict by any means... but they certainly aren't always in harmony with each other either.
(One might argue "easy to reuse" is more about the complexity of the code doing the reusing, but I feel like this is definitely something people mean when they talk about the simplicity of code.)
Reducing entropy or cross entropy fits intuition and theory and it's what LLMs target during training in order to model languages, to model the world and to enable calibrated reasoning across uncertainty and gray areas. I think you can use entropy reduction as a good definition of simplicity even though it's not always "simple" to understand what that entails.
That is an understatement! And not only that, "simple" is relative. The common saying that gets push back is "Do the simplest thing possible." People always seem to ignore the "possible" part of that phrase. Forgive me for being a little cute here, but a complex solution is simpler than a really complex solution.
Depends on what you mean by "simple". In my head, specifically with software, simple is defined as "does less", which is definitely somewhat easier to make, than a complex program that "does more".
Complexity (the opposite of simplicity) has nothing to do with the size of a program, but usually there is a high chance that a large program is more complex than smaller one, purely because the complexity multiplies, not just adds up.
A single regex line could be far more complex than a 100-line java program.
This is beside the point, and I'm being pedantic, but the unix pipeline and the clojure expression don't quite do the same thing.
The clojure expression reads the entire file into memory first, and then operates on that memory representation.
The pipeline processes the input in chunks. The two `sort`s might read the entire contents into memory, but an implementation like GNU sort will instead, for large inputs, create temporary files for sections of the input and then merge sort them to the output.
You could make clojure do the same thing, but it might not be as "simple" as the pipeline.
> Now, let's say we want to make a small change: show the output in the original file order. In Clojure, this is fairly straightforward: store an ordered sequence of the words in word_seq, store a map from each word to its frequency in freq_map, iterate over the sequence, and look up each word in the map:
Emphasis on just "storing" something. The author seems to not understand that this change makes the solutions categorically different.
It's a little like critiquing the efficiency of moving a piano up a stairwell by saying why didn't they just use a crane via the window? Using a crane isn't a better way of getting a piano up the stairwell.
> What might not be clear about this is that a struct loses runtime information. If you want to iterate a map, that's easy: call for (key, val) in map { .... If you want to iterate a struct ... get fucked? write a proc-macro?
I don't see how the example would benefit at all from having that information at run-time when it exists at compile-time. The problem with Rust specifically I think is that it's a relative PITA to get to the information that exists at compile-time in this case. In Zig, you'd just iterate over the fields using the same language you use for run-time code.
const fields = std.meta.fields(@TypeOf(instance));
inline for (fields) |field| {
const name = field.name;
const value = @field(instance, name);
std.debug.print("{s}: {any}", .{name, value};
}
The combination of high dynamicity at compile-time with low-effort switching between compile-time and run-time contexts makes this not too bad.
I've been thinking about a new AI based dimension to this. If your program is split into smaller decoupled "modules" then all the code for each "module" can fit into an AI context window. In this way the AI can have all the context to edit a "module" by just loading all the code for that "module". You would not need things like vector search as much. If we assume 20 tokens per line of code and the AI context window is 100k to 1M tokens, then that would argue for having "modules" between 5,000 and 50,000 loc, depending on which AI model you are using.
> then that would argue for having "modules" between 5,000 and 50,000 loc, depending on which AI model you are using.
FWIW, I set hard limits to 200 LOC for every single source code file in any AI-related projects, also with restrictions on "formatting hacks" and other golf-like stuff.
I think beyond 5000 LOC in a single file and all available models already get lost frequently, even if the "context limit" theoretically is way above that. Maybe aim for like 1K LOC at max unless you want to have lots of misunderstandings.
The general point is true, but the shell pipeline gets a lot more elegant if you use the sort-and-accumulate paradigm that the classic shell utilities were written for (which uses O(1) memory, by sorting on disk). Using mostly the author's own code, and adding --count to uniq:
(Where the final awk papers over the fact that we're mixing tabs and spaces here; obviously, awk is also good at doing the accumulation step, but uniq --count suffices here.)
(I originally posted the above as a comment on lobste.rs, on this same article.)
Very interesting solution, and in the spirit of the original article. If I understood the snippet right, you are sorting the input sequence on the first column (the words) and then on the second one (the frequencies)
It is nevertheless "complecting": the uniq assumes the data is sorted and the columns of your data structure move together. Maybe this algorithm is already complex regardless of the implementation.
the point is to do a stable sort on (word, line number) lexicographically, then when we do "uniq" we can take the first line number.
In contrast to the "we need a frequency table" idea in the article, this solution trades off memory by transferring all the line numbers in the stream. This is very much in the spirit of the infamous McIlroy/Knuth "bakeoff"[1] -- tradeoff some efficiency (via extra book-keeping or sorts) in return for composability.
The sort is on line numbers, as aozgaa said. Again, the paradigm here - and this is designed for a different time - is that your data most definitely does not fit in RAM, so you use sort(1) to sort on disk and run your software using only constant memory. (In modern software, databases can and do sort on disk, but few programs do.)
In detail, for input "foo bar FOO qux FOO foo", we convert to
[1 foo, 2 bar, 3 foo, 4 qux, 5 foo, 6 foo]
(with newlines instead of commas, obviously), then sort by word (then line number) to
[2 bar, 1 foo, 3 foo, 5 foo, 6 foo, 4 qux]
at which point the uniq invocation gives <count> <first_line> <word>, i.e.
[1 2 bar, 4 1 foo, 1 4 qux]
albeit with an ugly mix of tabs and spaces. One final sort by <first_line> gives us
[4 1 foo, 1 2 bar, 1 4 qux]
and then it's just a matter of formatting the output:
[foo 4, bar 1, qux 1]
The generally-useful point is that the classic shell utilities really do work pretty well if you're operating within their paradigm, which isn't "throw everything in a hash table". (That's the paradigm of later scripting languages.)
This argument is just based on "I wish the things I need to do were baked into the language." It's nice when that happens, but once programs get sufficiently large and complex, it stops mattering--you're dealing with domain-specific concepts that have zero built-in helpers and you're just building everything yourself regardless.
The examples in the essay are perhaps not specific enough - they do gesture in the direction of the author’s point, but they also admit other valid interpretations like your own. I think the Rich Hickey talk “Simple Made Easy”, which this post is based on, makes the point more clearly and precisely.
(For what it’s worth, the point of both this essay and the aforementioned talk is that programs do not have to get complex, even when they get large, even when it gets hard because there are no more easy / close-at-hand / familiar helpers in the language to tackle the domain specifics. In support of this point I will note that Rich Hickey is the creator of Clojure, a language in which “building domain-specific helpers yourself” is very nearly idiomatic.)
Maybe, but there are a couple of different kinds of "things" here: the actual primitive operations, and the techniques available for combining and coordinating them. For example, the `sort` program can only be told to sort based on certain specific analyses of the line contents (like extracting a certain "field" and converting that string to integer); it can't be supplied with a higher-order function. While things like `tee` exist, it isn't really practical to treat part of the pipeline as a reusable abstraction, except perhaps by writing a file (which leads to the author's approach with temporary files and `join`). And of course you're stuck with constant serialization and deserialization between processes.
https://news.ycombinator.com/item?id=49600275 is a great illustration of how to use Bash's "means of combination" more effectively than what OP thought of doing. But notice that it still relies on decorate-sort-undecorate for keyed sorting, and not only that but the undecoration has to be deferred until after `uniq` has made use of it, in a way that magically collapses the first decoration into something that can be used for a second sort, while adding a second decoration (the count). And then it still needs postprocessing because you can't express control over the structure of the decoration (there isn't really any, it's just string concatenation, but I mean you don't even control the order in which the pieces are concatenated).
I was surprised that the author didn’t translate the piped representation of the first program to a procedural program with explicit calls to subprocedures. It’s bigger, but it’s dead-simple and easy to slip instructions in the middle of.
The first example feels like too much of a straw man, and I'm not sure how I feel about the definition of simple (and yes I've seen Hickey's talk which I very much do agree with). Obviously a cohesive general purpose programming language like clojure is going to do better on a problem with abitrary sub-structure, especially when you want to rethink that substructure. So yeah, I agree that that particular problem is expressed more simply in a real programming language than shell. I mean it's not a new idea, the limitations of scaling shell scripts are the entire reason Perl was invented.
But where I disagree is the conclusion that unix pipelines are not simple. IMHO unix pipelines as a platform are incredibly simple and powerful, allowing for solving a massive range of small problems much more elegantly than any general purpose programming language. Obviously the constraints that enable this simplicity at the low-end, are real tradeoffs that prevent simplicity at the high-end. But one of the core principles of effective engineering is do the minimum to solve the problem at hand, no more, no less.
I think that you can elegantly implement the second problem in bash:
grep -Eo '[A-Za-z]+' README.md \
| tr A-Z a-z \
| {
distinct=()
declare -A freqs
while read word; do
[[ -v freqs["$word"] ]] || distinct+=("$word")
((freqs["$word"]++))
done
for word in "${distinct[@]}"; do
echo "${freqs["$word"]}" "$word"
done
}
Used grep here because the original pipeline erroneously introduces an empty line when the file starts with a character that doesn't match '[:alpha:]'.
Another solution to the pipeline example, this time making use of a subprogram for the frequency/accumulation:
< README.md \
tr -c '[:alpha:]' '\n' \
| tr '[:upper:]' '[:lower:]' \
| awk '
NF {
if (!($0 in count)) order[++n] = $0
count[$0]++
}
END {
for (i = 1; i <= n; i++) {
print count[order[i]], order[i]
}
}
'
If you don't allow `awk` in your "pure bash" then ofc this is not satisfactory. But it has the upside that the associative arrays are pretty explicit data structures (for the ordering and counts, respectively).
Nice to see more people getting nerdsniped by the sh code. ;-)
Yes, associative arrays work well. I think it should even be possible to use bash associative arrays. But at that point you're no longer doing classic sh - awk is basically halfway to Perl. (And pretty awesome.)
Yeah, insisting on "only" shell is weird, shell is at heart a process orchestrater, and denying it it's processes is rejecting most of it's functionality. It is equivalent to saying "do this in python, but you are not allowed to use any modules"
Without processes shell is just a weird sad little language, with them it turns into this epic data flow language. With some real design stinkers, Most of these are due to it's interactive first focus, The features desirable for interactive use, often start to stink for stored program use, I will note that having the same language for interactive and scripting is pretty kick ass.
On the subject of dataflow languages has there been any research in this area? Something that can stitch together processes as well or better than shell? Perl may work in this role but I have to admit I really dislike it's syntax, and as such I never learned Perl enough to love it. and while most other scripting languages can technicaly create pipelines, it is very awkward compared to shell.
> On the subject of dataflow languages has there been any research in this area? Something that can stitch together processes as well or better than shell? Perl may work in this role but I have to admit I really dislike it's syntax, and as such I never learned Perl enough to love it. and while most other scripting languages can technicaly create pipelines, it is very awkward compared to shell.
I've been working on a design for a Python library to facilitate this sort of thing.
You can skip `tr` as well - works on BSD awk and GNU awk.
{
$0 = tolower($0)
gsub(/[^[:alpha:]]/, "\n")
for (i = 1; i <= NF; i++) {
if (!($i in freq)) order[++n] = $i
freq[$i]++
}
} END {
for (i = 1; i <= n; i++)
printf "%d %s\n", freq[order[i]], order[i]
}
UNIX Pipes do not mandate having to use tons of different programs
with stupid commandline options. I simulate them in ruby itself;
method chaining works in a very similar way, but I built a pseudo
pipe around it. The idea was more to have an object oriented shell,
e. g. combine good ideas from UNIX pipes and the MS powershell.
They are simple if you design them well and have them be flexible
too. The reason UNIX pipes were awkward is because they delegated
onto many different programs such as awk or sed with their own
strange rules. Nowhere does it say you HAVE to use such awkward
tools. Use better tools and the idea of piping becomes simple,
similar to (a more flexible variant of) method chaining. Just
without being tied down to a specific object per se (I do use
the pipe-handler master object to handle the pipe instructions;
each pipe instruct I call cmdlet, e. g. shorter for commandlet,
as this is how I like to think about this in terms. This also
combines e. g. virtualdub + avisynth ideas. I loved them when I
used windows. The idea behind avisynth is great - not necessarily
all of the syntax, but the idea that all multimedia audio can be
operated on at all times in flexible ways.)
Unix pipes are easy, and they are a very good abstraction, and their choice of abstraction boundaries is superlative, but they are not simple. (I have more than once seen a colleague implementing a ring buffer for feeding data into a Unix pipe!)
- waitpid for all process in the other process chain?
- propagate/handle signals, like for example SIGINT/SIGTSTP/SIGPIPE/SIGHUP forward and back signals?
- change stdio buffering mode?
- remember to count how many bytes actually were written by write, and blocking if not, before clobbing the 64kb of the pipe buffer size with another write?
- flush, then close all file descriptors left behind by the pipes when it ends?
It's for reasons like that, that I don't trust anything non-trivial, not-shell to use pipelines correctly.
Maybe part of it is that fully functional notation necessarily preserves certain properties of the computation, while imperative style doesn't have to. For a complex piece of code, all of the hairiness is manifest in a functional style, but can be made implicit in an imperative style. So there's an economy to the imperative approach as things get complex, but in a sense we're just laundering the complexity from understanding the machinations of the code to understanding the correctness of the code.
Hence, objects; keep alternating between functional (stateless) and imperative (stateful) as you ascend the abstraction tower, so that the hairiness does not leak between layers and hence accumulate.
1. "You need to have taste (so: experience) to do DRY right". Same chunk of code more than once, so natural/expected behavior is to export to a helper... Aaaand the now introduced coupling is (too) often ignored, as in "no thought given whatsoever".
2. "Sometimes it's better to just leave it as is". 95% to 98% of "same chunk of code" in a number of places. The temptation to DRY is strong, yet the "numerically mere 2 to 5 pp" make the extracted helper an exercise in all manners of gymnastics. The only correct answer is: do not start.
(Own experience; your mileage may vary - if it does, feel free to comment back!)
I don't think it's possible to have a universal "colloquial" definition of "simple" that is also precise. It's all relative.
This is why I think the formalists studying complexity back in the 50s had the right approach. You can only give "simple" precise meaning within some kind of formal system with a shared set of initial axioms or assumptions. From that point you can define it quantitatively over some set of objects (relations, programs).
Funnily enough, this approach also touches on Hickey's etymological derivation. The root words also fundamentally have to do with the quantity of relationships.
I mean I am not part of the same company, so I am just talking out of my butt, but I cannot imagine a dev worth their money taking more than two weeks, to fix a bug, especially today with AI assistance, most bugs are found the same day, the gnarly ones maybe take two days and in my lifetime as an engineer (10 years), I have not yet seen a bug that took me more than a week.
Taking 9 months to fix a bug sounds alarming to me.
Strong resonance with the famous essay "The Rise of Worse is Better" [1], which contrasted the (better) "MIT/Stanford style of design" with the (worse) "New Jersey approach".
MIT/Stanford:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the interface to be simple than the implementation.
New Jersey:
> Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.
TFA maps "simplicity" to "MIT/Stanford simplicity" (simplicity for the user) and "smallness" to "New Jersey simplicity" (simplicity for the developer).
I wonder if the root of the tension between the two schools comes down to the ambiguity of the user/developer distinction. Developers are also users. Simplicity of implementation is helpful to developers when they are working directly on implementation, while simplicity of interface is helpful to developers when they are using other developers' work.
[1] https://dreamsongs.com/RiseOfWorseIsBetter.html
I think you're right on that point of contention, it's too far to assert a universal and clear good/bad dynamic here because that line between developer/user is contextual and fuzzy.
Another interesting irony I'll note, Lisp is the "New Jersey approach" towards symbolic AI. Americans clinging to their Lisp systems were deeply entrenched in a "worse-is-better" mindset. Your interface, the computational model, didn't need to be designed for logic programming, that was wholly secondary. Do everything as much as possible in Lisp, and then offload the relational description to a small (not simple) library. American knowledge engineers were looked at as overpaid procedural hackers with zero mathematical elegance and very little credibility. More or less the same perception these self-same Lisp-machine users had for Unix and C programmers.
It's all about perspective, at the end of the day. Where we draw the line in the sand on these categories is free-to-choose, yet it also determines everything. We're always someone else's villain under different semantics.
I appreciate the New Jersey simplicity as a user (with development skills) too, though. It's usually just a matter of time before I have to dive into the program/library/whatever internals to fix a bug.
It's why only nerds use Linux (simple for the developer) but everyone uses Office (simple for the user).
Sorry, I meant Microsoft Copilot 365.
I think Excel is a good example that everyone can build a clear mental model on how to use it, but implementing one is a daunting task.
Pretty much anything. I mean, even consider a text editor, monospace font, no syntax highlighting. It's already pretty daunting. Doable, certainly, but a big task. We're lucky people already made some, and we can copy their designs even when we choose not to copy their exact code.
Now consider Microsoft Word.
Are you nuts?
Excel is great, but it's an arcane beast rivaled only by Emacs configs. People pass down Excel formulas by word of mouth like they're magic spells.
Which traces back to Xerox Bravo text editor.
From the authors, Charles Simonyi went to Microsoft as one of the original Word implementers, while Excel was born on Mac OS and only later ported to Windows.
> There's no native Unix equivalent to frequencies, this sort | uniq -c is the closest we can get. Not only is it less performant (it has to collect the full input into memory before continuing), but it ties aggregation to ordering.
One of the core features of the Unix command-line is that it is user-extensible. If there's no "native" command equivalent to frequencies, you can write your own, and it will be given the same first-class treatment as any other binary in your PATH. This is entirely in keeping with the Unix philosophy of simple implementations.
Yeah, that point was weak. The argument went from "Unix pipelines are not simple" to "unix based operating systems don't have an equivalent to Clojure's frequency function by default so it's worse"
OK but how would it look like if you had such a program? Shells are not known for having the extensive set of functions that real programming languages have.
I think the point was more that it's hard to compose shell utilities to make something new that's not supported out of the box. Instead you do have to write an entirely new utility and use that.
That said having key/value semantics and not just stream of bytes would make the shell much more versatile, at the cost of making it bigger.
That's not what the linked article is saying. The point made in the article is that the 'uniq' utility couples together two unrelated concepts--sorting and frequency counting--and this coupling forces you to write more complex code, compared to the version where frequency counting is decoupled from sorting.
All of this is completely correct, but it has nothing to do with bash vs. Clojure. The same decoupling can easily be achieved in either language.
I think the article is a bit weak when it's making this point, because it's not about Unix pipelines. It's about POSIX shell utilities being too bare-bones.
But Unix pipelines are not simple too. They have a couple of nitty-gritty details that often come out and bite you.
1. They can only stream raw bytes, so all the programs that deal with lists like sort and uniq have to separate items using a delimiter (usually newline). If you want to process data with that delimiter in it, you're in for a ride. And if you want to write a custom tool, you have to do all the splitting yourselves (luckily it's so common most programming language will provide a ready-made facility for you to do that). This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".
2. Error are hidden by default in shell. Nowadays you can explicitly change this behavior, but you have to remember to do `set -o pipefail` and I don't think it was always there.
3. There is no data typing at all. Everything is binary or text. Nowadays a lot of programs just output JSON, and the users (if they even stay inside the shell) almost always reach for jq to parse it. But jq is not a Unix philosophy program: it's an entire streaming functional programing that can do quite a lot. But even jq gets hairy when you have to do a bigger query or transformation. In that case users often reach out to Python or another language and just move the business logic there.
I think this fits well with what the article is trying to say: Unix pipes are pretty easy (small) to implement on your own (compare that to something like Nushell's pipes). But the moment you to do something that's a little different than the happy path it was built for, you need to go for another tool (jq) that has its own built-in pipe and small programming language, because Unix pipes won't cut it. And for more complex (hehe) things, you'll have to reach for a larger (and simpler) tool: a full-fledged programming language.
> This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".
I'm not sure if you're actually trying to argue for a specific position here, but you highlight several negative results of one "philosophy" of development, with the implication that the alternative wouldn't have those negatives.
This is a tricky point to refute because you're right, these are flaws and there could be a system that doesn't have them.
So why do these flawed systems exist and proliferate?
Because the real life result isn't actually a choice between "sloppy but quick to develop" and "elegant well engineered but slow".
The choice is actually between "sloppy but exists" and, well, nothing, because the other version never actually materializes.
(And of course, I feel compelled to point out that bash is a user interface not a programming language. Any attempt to replace it or improve it without focusing on that main point is doomed to failure, which is why you see so many people who apparently think that what bash really needs is strict type checking or something and end up creating a completely awful user experience)
I imagine just ask would be enough for most usecases:
awk '{ freq[$0] += 1 } END { for (n in freq ){ print freq[n] " " n } }' < file.txt
This was my thought, too. Either you can extend your toolkit—make a utility like dedup (https://codeberg.org/napcakes/dedup) and put in on your path (yay! so easy)—or you can't and thus must define the boundary of the allowed toolkit. In this case, it seems overly disingenuous to not allow awk, which will do the thing for you just fine, no Clojure needed at all. awk is hashmaps galore.
Honestly, so much of our conceptions of "what's wrong" is more to do with lack of familiarity with history than any actually unsolved problem.
That is nothing special about UNIX, every other OS with command line experience, or REPL windows, has similar capabilities.
Great piece, very straightforward examples, although I did have to squint for quite a while to grok the Closure portion.
I am currently building a piece of very modular software and it has been the hardest-to-design project of my entire career. I would never be allotted this amount of time-effort at any job I have held to make something this robust and clearly defined. Many aspects of this project have taken 3-5 rounds trying-and-trashing to get an abstraction that is uncomplicated.
This is precisely why vibe coding is so successful for building tiny isolated scripts, and so disastrous for anything else. It's just really dang hard to build something large and simple.
Personally, I'm big fan of those pipes mentioned. And the tooling that can be created around those _simple_ "primitives". Comparison to Google drive is obscure, in a way that it compares one gigantic piece of software into these small and simple.
Bottom line is probably true, but if you are an open-source maintainer mentioned, and you have only so few hours to spend, you just cannot create those gigantic softwares either. You need to choose from the cards on your hand.
> This is precisely why vibe coding is so successful for building tiny isolated scripts, and so disastrous for anything else. It's just really dang hard to build something large and simple.
Brooks, in “the Mythical Man-Month”:
“One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams.
Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.
In the upper left of Fig. 1.1 is a program. It is complete in itself, ready to be run by the author on the system on which it was developed. That is the thing commonly produced in garages, and that is the object the individual programmer uses in estimating productivity.
There are two ways a program can be converted into a more useful, but more costly, object. These two ways are represented by the boundaries in the diagram.
Moving down across the horizontal boundary, a program becomes a programming product. This is a program that can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data. […] As a rule of thumb, I estimate that a programming product costs at least three times as much as a debugged program with the same function.
Moving across the vertical boundary, a program becomes a component in a programming system. This is a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks. To become a programming system component, a program must be written so that every input and output conforms in syntax and semantics with precisely defined interfaces. The program must also be designed so that it uses only a prescribed budget of resources — memory space, input-output devices, computer time. Finally, the program must be tested with other system components, in all expected combinations. […] A programming system component costs at least three times as much as a stand-alone program of the same function. The cost may be greater if the system has many components.
In the lower right-hand corner of Fig. 1.1 stands the programming systems product. This differs from the simple program in all of the above ways. It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts.”</i>
> The reason for this is that in Rust, a struct couples type-checking to a fixed data representation. You can't get one without the other.
> Clojure decouples data representations from type checking.
This is funny to me because seen from the other side, (this) Clojure couples runtime type information to data structures: you're no longer allowed to define a data structure that doesn't have some runtime type information attached. A fixed static structure is just the consequence of not adding dynamic type information.
Meanwhile in Rust you can get type-checking ‘without’ a fixed structure by using trait objects.
Yeah that example was pretty flimsy and contrived.
I think all of TFA was flimsy. This coupling of which TFA is bad, somehow? I don't even see a good definition of "coupled", nor a good argument of why/how "uncoupling" makes for simple and small.
Regardless of whether the type information is static or dynamic, you're still coupling some type to some data. The type is still implicit even after compilation; there still exists a structure to the data, even if that structure isn't easily discerned without the source. Or to put it another way: just because there's no runtime type information, doesn't mean that the data now is entirely decoupled from the type.
Using "length of the correctness statement + length of its proof" works quite well as proxy for complexity of a component (the longer, the more complex).
Copy pasted functions with subtle changes mean you cannot reuse the proof (DRY). Giant functions with lots of if/else statements however might cause a branch explosion in the proof. The right abstraction removes lots of assumptions that a proof could depend on, limiting the search space and often forcing elegance (this also applies to math, eg. when reasoning with abstract groups instead of integers). The wrong abstraction might force case distinctions on consumers of the abstraction.
> Using "length of the correctness statement + length of its proof" works quite well as proxy for complexity of a component (the longer, the more complex).
It sounds like a reasonable concept, but then Principia Mathematica takes 300 pages to prove that 1+1 is 2.
Important to note as well, is that "simple" isn't "lesser" or "dumber", it can be "more" and "wider", yet still "simpler".
Expectedly, Rich Hickey explains it best, watch the "Simple Made Easy" talk if you haven't before, one of the few talks I probably watch bi-yearly: https://www.youtube.com/watch?v=SxdOUGdseq4
Few things, concepts and ideas have changed as much of my programming mind as Hickey's talk and ultimately Clojure have done over the years.
Wish we had new amazing Hickey talks to link to, maybe it seems he's about the hang up the hammock perhaps?
He's speaking at the coming Clojure Conj, so there will be a new talk soon.
The word simple is used here a way I'm having trouble wrapping my head around.
This specific usage appears to come from this linked talk, Simple Made Easy:
https://www.youtube.com/watch?v=SxdOUGdseq4
My reaction to the Unix pipeline was that, the reason it exploded in complexity is because the pieces were too simple. They were insufficiently expressive.
But the word is used in a different way here, and I'll have to watch the talk to understand what exactly is meant. (Something like orthogonality?)
Agreed. There are multiple senses for "simple" and on of those senses is "small". It is actually a very useful sense too, as Solomonov/Levin/Kolmogorov/Chaitin-style inductive reasoning has shown.
And it is important not to just make snappy quips by equivocating.
I don’t know. This reads more like a “Clojure is great” post, and Clojure is great. But the author takes a swing and a miss on the Rich Hickey magic. The UNIX example is contrived (the number of occurrences in the order they occur?), and trying to redefine simple in a way that excludes UNIX pipelines doesn’t work.
The example is even more contrived than that, since he could have just used awk
>the reason it exploded in complexity is because the pieces were too simple
the reason it exploded in complexity is because the complexity is in the interaction of the parts, not the properties of the parts. It's not so much about expressiveness but about the fact that the Unix philosophy has a willful disregard for systems thinking.
Russ Ackhoff has the analogy of the house. How do you design rooms in a house? Only ever with the house in mind. You can't just hope the rooms work, you always need to check whether a new room improves the house, if somethings wrong you fix the room, not the house.
Unix tools have a disregard for the house. Clojure works well because every part of Clojure is designed not just to improve itself but to improve Clojure. Emacs is similar in that regard, you can see that when people make Emacs packages. The good ones always have Emacs in mind, not just their own functionality.
The author uses the term “coupling” to describe a simple program from the code’s perspective - “you need to have a good mental model of your program”. I agree, but I’d argue that more importantly you need to have a good domain model. Good design communicates accurately and a high Gulf of Evaluation/Execution are actually what makes programs feel “complex” to a user.
These concepts simplify for the user these questions: “I know what I want - how do I make the program do it?” (Execution) and “The program did something — what state is it actually in?” (Evaluation)
I believe the author was getting at these concepts, especially in their Google Drive example - how the large program has a “small” UX. Understanding the domain model provides a much stronger basis for designing user interfaces, and understanding the Gulf of Evaluation/Execution allows you to build incredibly complex-looking, large UX’s without confusing or overwhelming the user.
Honestly, having watched people argue about what simple is for about the last 10 years, I've pretty much settled on it not being a well-defined term. We know complex when we see it for sure, at least when it is present in quantity, but simplicity is not just the absense of complexity. There's at least three concepts we're all trying to stuff into the same word, and they are not only not "orthogonal" they are often in conflict with each other. I don't even think it can be rehabilitated, it can only really be abandoned, to clear the way to trying to characterize the multiple concepts we're trying to stuff into this one word.
It is especially dangerous when something is "good" and people try to appropriate the term to appropriate the goodness of the term, as if goodness flows from a term to the thing it is attached to rather than the other way around. "Simple" is good so my good thing must be "simple" to be "good". But it doesn't. Simple can even be bad, in the wrong place or in the wrong sort of "simple" for a given job.
> We know complex when we see it for sure, at least when it is present in quantity
Because familiarity is a confound for intuition about complexity, even this is not always true.
Maxwell's equations will look complex to the uninitiated, and can represent the pinnacle of simplicity to those who already understand them.
Could you try to define these three separate concept?
I see at least:
"Few tokens" - perhaps the most literal simplicity, literally, it doesn't use many tokens to do the job. But as the article points out, that doesn't necessarily fit with...
"Easy to reuse" - This is that simplicity that functional programming aspires to, where you craft some precise abstraction that somehow captures something like "monad". Haskell is full of this sort of simplicity, oozing out of every pore, but people generally think of it as a very complex and hard langauge, contrasting...
"Easy to understand" - As in, not cognitively complex. It is amazing how quickly things that I would otherwise describe as very simple still blow out our little minds. Consider the first time you saw quicksort... or even how it feels now. It's not a lot of tokens, but it's twisty and recursive and especially if you're not mathematically trained and in practice it's easy to call it more "complicated" than a CRUD form that takes in and validates 10 parameters in a straightforward way, even though in terms of what is actually happening the CRUD form may be doing vastly more than the little quicksort algorithm. It just isn't being twisty, recursive, and subtle in how it does it.
And I'm just filling out the first three that come to mind. Note these are not always in conflict by any means... but they certainly aren't always in harmony with each other either.
(One might argue "easy to reuse" is more about the complexity of the code doing the reusing, but I feel like this is definitely something people mean when they talk about the simplicity of code.)
Reducing entropy or cross entropy fits intuition and theory and it's what LLMs target during training in order to model languages, to model the world and to enable calibrated reasoning across uncertainty and gray areas. I think you can use entropy reduction as a good definition of simplicity even though it's not always "simple" to understand what that entails.
https://benoitessiambre.com/entropy.html
Simple never means it is easy and it is always the biggest misconception in software.
That is an understatement! And not only that, "simple" is relative. The common saying that gets push back is "Do the simplest thing possible." People always seem to ignore the "possible" part of that phrase. Forgive me for being a little cute here, but a complex solution is simpler than a really complex solution.
Depends on what you mean by "simple". In my head, specifically with software, simple is defined as "does less", which is definitely somewhat easier to make, than a complex program that "does more".
Complexity (the opposite of simplicity) has nothing to do with the size of a program, but usually there is a high chance that a large program is more complex than smaller one, purely because the complexity multiplies, not just adds up.
A single regex line could be far more complex than a 100-line java program.
This is beside the point, and I'm being pedantic, but the unix pipeline and the clojure expression don't quite do the same thing.
The clojure expression reads the entire file into memory first, and then operates on that memory representation.
The pipeline processes the input in chunks. The two `sort`s might read the entire contents into memory, but an implementation like GNU sort will instead, for large inputs, create temporary files for sections of the input and then merge sort them to the output.
You could make clojure do the same thing, but it might not be as "simple" as the pipeline.
This bothered me too. The author even said:
> Now, let's say we want to make a small change: show the output in the original file order. In Clojure, this is fairly straightforward: store an ordered sequence of the words in word_seq, store a map from each word to its frequency in freq_map, iterate over the sequence, and look up each word in the map:
Emphasis on just "storing" something. The author seems to not understand that this change makes the solutions categorically different.
It's a little like critiquing the efficiency of moving a piano up a stairwell by saying why didn't they just use a crane via the window? Using a crane isn't a better way of getting a piano up the stairwell.
I think the assumption here is that the set of distinct words (hence, also the keys of the frequency map) will be small compared to the overall file.
> What might not be clear about this is that a struct loses runtime information. If you want to iterate a map, that's easy: call for (key, val) in map { .... If you want to iterate a struct ... get fucked? write a proc-macro?
I don't see how the example would benefit at all from having that information at run-time when it exists at compile-time. The problem with Rust specifically I think is that it's a relative PITA to get to the information that exists at compile-time in this case. In Zig, you'd just iterate over the fields using the same language you use for run-time code.
The combination of high dynamicity at compile-time with low-effort switching between compile-time and run-time contexts makes this not too bad.
I've been thinking about a new AI based dimension to this. If your program is split into smaller decoupled "modules" then all the code for each "module" can fit into an AI context window. In this way the AI can have all the context to edit a "module" by just loading all the code for that "module". You would not need things like vector search as much. If we assume 20 tokens per line of code and the AI context window is 100k to 1M tokens, then that would argue for having "modules" between 5,000 and 50,000 loc, depending on which AI model you are using.
> then that would argue for having "modules" between 5,000 and 50,000 loc, depending on which AI model you are using.
FWIW, I set hard limits to 200 LOC for every single source code file in any AI-related projects, also with restrictions on "formatting hacks" and other golf-like stuff.
I think beyond 5000 LOC in a single file and all available models already get lost frequently, even if the "context limit" theoretically is way above that. Maybe aim for like 1K LOC at max unless you want to have lots of misunderstandings.
Right; the context window also has to fit chain-of-thought about the code that has been read in, and so on and so forth.
I already preferred to keep source code files under about 200 LOC, so.
The general point is true, but the shell pipeline gets a lot more elegant if you use the sort-and-accumulate paradigm that the classic shell utilities were written for (which uses O(1) memory, by sorting on disk). Using mostly the author's own code, and adding --count to uniq:
(Where the final awk papers over the fact that we're mixing tabs and spaces here; obviously, awk is also good at doing the accumulation step, but uniq --count suffices here.)
(I originally posted the above as a comment on lobste.rs, on this same article.)
Very interesting solution, and in the spirit of the original article. If I understood the snippet right, you are sorting the input sequence on the first column (the words) and then on the second one (the frequencies)
It is nevertheless "complecting": the uniq assumes the data is sorted and the columns of your data structure move together. Maybe this algorithm is already complex regardless of the implementation.
btw, this paradigm reminds me of APL
the point is to do a stable sort on (word, line number) lexicographically, then when we do "uniq" we can take the first line number.
In contrast to the "we need a frequency table" idea in the article, this solution trades off memory by transferring all the line numbers in the stream. This is very much in the spirit of the infamous McIlroy/Knuth "bakeoff"[1] -- tradeoff some efficiency (via extra book-keeping or sorts) in return for composability.
Agreed, very neat.
[1] https://homepages.cwi.nl/~storm/teaching/reader/BentleyEtAl8...
The sort is on line numbers, as aozgaa said. Again, the paradigm here - and this is designed for a different time - is that your data most definitely does not fit in RAM, so you use sort(1) to sort on disk and run your software using only constant memory. (In modern software, databases can and do sort on disk, but few programs do.)
In detail, for input "foo bar FOO qux FOO foo", we convert to
[1 foo, 2 bar, 3 foo, 4 qux, 5 foo, 6 foo]
(with newlines instead of commas, obviously), then sort by word (then line number) to
[2 bar, 1 foo, 3 foo, 5 foo, 6 foo, 4 qux]
at which point the uniq invocation gives <count> <first_line> <word>, i.e.
[1 2 bar, 4 1 foo, 1 4 qux]
albeit with an ugly mix of tabs and spaces. One final sort by <first_line> gives us
[4 1 foo, 1 2 bar, 1 4 qux]
and then it's just a matter of formatting the output:
[foo 4, bar 1, qux 1]
The generally-useful point is that the classic shell utilities really do work pretty well if you're operating within their paradigm, which isn't "throw everything in a hash table". (That's the paradigm of later scripting languages.)
It's a nice dissection of the topic. But I would like to reframe it as "Simple Is Not 'Always' Small."
Logic that the blog presents is sound; but sometimes small works as a good-enough proxy to get to the simplicity that we find elegant.
This argument is just based on "I wish the things I need to do were baked into the language." It's nice when that happens, but once programs get sufficiently large and complex, it stops mattering--you're dealing with domain-specific concepts that have zero built-in helpers and you're just building everything yourself regardless.
The examples in the essay are perhaps not specific enough - they do gesture in the direction of the author’s point, but they also admit other valid interpretations like your own. I think the Rich Hickey talk “Simple Made Easy”, which this post is based on, makes the point more clearly and precisely.
(For what it’s worth, the point of both this essay and the aforementioned talk is that programs do not have to get complex, even when they get large, even when it gets hard because there are no more easy / close-at-hand / familiar helpers in the language to tackle the domain specifics. In support of this point I will note that Rich Hickey is the creator of Clojure, a language in which “building domain-specific helpers yourself” is very nearly idiomatic.)
Maybe, but there are a couple of different kinds of "things" here: the actual primitive operations, and the techniques available for combining and coordinating them. For example, the `sort` program can only be told to sort based on certain specific analyses of the line contents (like extracting a certain "field" and converting that string to integer); it can't be supplied with a higher-order function. While things like `tee` exist, it isn't really practical to treat part of the pipeline as a reusable abstraction, except perhaps by writing a file (which leads to the author's approach with temporary files and `join`). And of course you're stuck with constant serialization and deserialization between processes.
https://news.ycombinator.com/item?id=49600275 is a great illustration of how to use Bash's "means of combination" more effectively than what OP thought of doing. But notice that it still relies on decorate-sort-undecorate for keyed sorting, and not only that but the undecoration has to be deferred until after `uniq` has made use of it, in a way that magically collapses the first decoration into something that can be used for a second sort, while adding a second decoration (the count). And then it still needs postprocessing because you can't express control over the structure of the decoration (there isn't really any, it's just string concatenation, but I mean you don't even control the order in which the pieces are concatenated).
“When is it useful to be small?”
I like this question. Some projects will sacrifice usefulness in the name of simplicity.
I was surprised that the author didn’t translate the piped representation of the first program to a procedural program with explicit calls to subprocedures. It’s bigger, but it’s dead-simple and easy to slip instructions in the middle of.
How is the clojure program given not that?
The first example feels like too much of a straw man, and I'm not sure how I feel about the definition of simple (and yes I've seen Hickey's talk which I very much do agree with). Obviously a cohesive general purpose programming language like clojure is going to do better on a problem with abitrary sub-structure, especially when you want to rethink that substructure. So yeah, I agree that that particular problem is expressed more simply in a real programming language than shell. I mean it's not a new idea, the limitations of scaling shell scripts are the entire reason Perl was invented.
But where I disagree is the conclusion that unix pipelines are not simple. IMHO unix pipelines as a platform are incredibly simple and powerful, allowing for solving a massive range of small problems much more elegantly than any general purpose programming language. Obviously the constraints that enable this simplicity at the low-end, are real tradeoffs that prevent simplicity at the high-end. But one of the core principles of effective engineering is do the minimum to solve the problem at hand, no more, no less.
I think that you can elegantly implement the second problem in bash:
Used grep here because the original pipeline erroneously introduces an empty line when the file starts with a character that doesn't match '[:alpha:]'.
Another solution to the pipeline example, this time making use of a subprogram for the frequency/accumulation:
If you don't allow `awk` in your "pure bash" then ofc this is not satisfactory. But it has the upside that the associative arrays are pretty explicit data structures (for the ordering and counts, respectively).
Nice to see more people getting nerdsniped by the sh code. ;-)
Yes, associative arrays work well. I think it should even be possible to use bash associative arrays. But at that point you're no longer doing classic sh - awk is basically halfway to Perl. (And pretty awesome.)
Yeah, insisting on "only" shell is weird, shell is at heart a process orchestrater, and denying it it's processes is rejecting most of it's functionality. It is equivalent to saying "do this in python, but you are not allowed to use any modules"
Without processes shell is just a weird sad little language, with them it turns into this epic data flow language. With some real design stinkers, Most of these are due to it's interactive first focus, The features desirable for interactive use, often start to stink for stored program use, I will note that having the same language for interactive and scripting is pretty kick ass.
On the subject of dataflow languages has there been any research in this area? Something that can stitch together processes as well or better than shell? Perl may work in this role but I have to admit I really dislike it's syntax, and as such I never learned Perl enough to love it. and while most other scripting languages can technicaly create pipelines, it is very awkward compared to shell.
> On the subject of dataflow languages has there been any research in this area? Something that can stitch together processes as well or better than shell? Perl may work in this role but I have to admit I really dislike it's syntax, and as such I never learned Perl enough to love it. and while most other scripting languages can technicaly create pipelines, it is very awkward compared to shell.
I've been working on a design for a Python library to facilitate this sort of thing.
You can skip `tr` as well - works on BSD awk and GNU awk.
> Unix pipelines are not simple
But they are.
UNIX Pipes do not mandate having to use tons of different programs with stupid commandline options. I simulate them in ruby itself; method chaining works in a very similar way, but I built a pseudo pipe around it. The idea was more to have an object oriented shell, e. g. combine good ideas from UNIX pipes and the MS powershell.
They are simple if you design them well and have them be flexible too. The reason UNIX pipes were awkward is because they delegated onto many different programs such as awk or sed with their own strange rules. Nowhere does it say you HAVE to use such awkward tools. Use better tools and the idea of piping becomes simple, similar to (a more flexible variant of) method chaining. Just without being tied down to a specific object per se (I do use the pipe-handler master object to handle the pipe instructions; each pipe instruct I call cmdlet, e. g. shorter for commandlet, as this is how I like to think about this in terms. This also combines e. g. virtualdub + avisynth ideas. I loved them when I used windows. The idea behind avisynth is great - not necessarily all of the syntax, but the idea that all multimedia audio can be operated on at all times in flexible ways.)
Unix pipes are easy, and they are a very good abstraction, and their choice of abstraction boundaries is superlative, but they are not simple. (I have more than once seen a colleague implementing a ring buffer for feeding data into a Unix pipe!)
> But they are.
Did you remember to:
- check for the other spawned process exit code?
- waitpid for all process in the other process chain?
- propagate/handle signals, like for example SIGINT/SIGTSTP/SIGPIPE/SIGHUP forward and back signals?
- change stdio buffering mode?
- remember to count how many bytes actually were written by write, and blocking if not, before clobbing the 64kb of the pipe buffer size with another write?
- flush, then close all file descriptors left behind by the pipes when it ends?
It's for reasons like that, that I don't trust anything non-trivial, not-shell to use pipelines correctly.
"In Clojure this is fairly straightforward"
Somehow when things get complex, I could never find fully functional style to be more understandable than imperative
Maybe part of it is that fully functional notation necessarily preserves certain properties of the computation, while imperative style doesn't have to. For a complex piece of code, all of the hairiness is manifest in a functional style, but can be made implicit in an imperative style. So there's an economy to the imperative approach as things get complex, but in a sense we're just laundering the complexity from understanding the machinations of the code to understanding the correctness of the code.
Hence, objects; keep alternating between functional (stateless) and imperative (stateful) as you ascend the abstraction tower, so that the hairiness does not leak between layers and hence accumulate.
Strikes a practical chord or two:
1. "You need to have taste (so: experience) to do DRY right". Same chunk of code more than once, so natural/expected behavior is to export to a helper... Aaaand the now introduced coupling is (too) often ignored, as in "no thought given whatsoever".
2. "Sometimes it's better to just leave it as is". 95% to 98% of "same chunk of code" in a number of places. The temptation to DRY is strong, yet the "numerically mere 2 to 5 pp" make the extracted helper an exercise in all manners of gymnastics. The only correct answer is: do not start.
(Own experience; your mileage may vary - if it does, feel free to comment back!)
> In Bash you need a bunch of temp files and ugly opaque regexes, sorts, and joins:
> That's because our original program was small but not simple.
I would say no - because it was inflexible.
I don't think it's possible to have a universal "colloquial" definition of "simple" that is also precise. It's all relative.
This is why I think the formalists studying complexity back in the 50s had the right approach. You can only give "simple" precise meaning within some kind of formal system with a shared set of initial axioms or assumptions. From that point you can define it quantitatively over some set of objects (relations, programs).
Funnily enough, this approach also touches on Hickey's etymological derivation. The root words also fundamentally have to do with the quantity of relationships.
I mean I am not part of the same company, so I am just talking out of my butt, but I cannot imagine a dev worth their money taking more than two weeks, to fix a bug, especially today with AI assistance, most bugs are found the same day, the gnarly ones maybe take two days and in my lifetime as an engineer (10 years), I have not yet seen a bug that took me more than a week.
Taking 9 months to fix a bug sounds alarming to me.
simple = maintainable
"In Clojure this is straightforward"