Years ago (2010 iirc), I reported to the Postgres list, with a patch, that (one of) the reason Postgres “didn’t work on NFS” was because it couldn’t deal with short writes. I got told the usual “you’re holding it wrong” instead of an acknowledgement of PG not sticking to the spec.
I patched my own systems (wrap the calls in a loop as per standard practice) and then proceeded to run literally hundreds of thousands of PostgreSQL instances on NFS for many more years with no problems.
The patch was eventually integrated it seems but I never found out why because I lost interest in trying to work with the community after that experience.
This saddens me.
Looking at this thread - the patch is obviously right. It doesn't matter about NFS, Linux, or any host of random crap people brought up. (I unfortunately suspect if you hadn't mentioned any of that, and just said "write doesn't guarantee full writes, this handles that" it may have gone better).
The documentation (literally on every system) is quite clear:
"The return value is the number of bytes actually written. This may be size, but can always be smaller. Your program should always call write in a loop, iterating until all the data is written."
(see https://www.gnu.org/software/libc/manual/html_node/I_002fO-P...)
Literally all documentation you can find about write on all systems that implement it clearly mentions it doesn't have to write all the bytes you ask it to.
Instead of saying "yeah, that's right", people go off in every random direction, with people arguing about your NFS mount options, what things can cause write to do that and whether they are possible[1] or they should care about them.
Sigh.
[1] spoiler alert: it doesn't matter. The API allows for this, you need to handle it.
I'm a bit confused by this perception of the thread. There was one person doubtful the patch is the right approach (Tom), but still offered concrete review comments (of actual issues that'd at the very least need to be documented in code comments). Others agreed that it's something that needs to be done.
> Instead of saying "yeah, that's right", people go off in every random direction, with people arguing about your NFS mount options, what things can cause write to do that and whether they are possible[1] or they should care about them.
Given that changing the mount option solved the immediate problem for the OP, I fail to see how it's random.
> [1] spoiler alert: it doesn't matter. The API allows for this, you need to handle it.
Sure, it's not like the return value was ignored before though. I agree retrying is the better response, but erroring out is a form of handling (and will also lead to retries in several places, just with a loop that's not tightly around the write()).
You had me until you tried to justify bubbling the error instead of handling it at its source while possible.
Hm? I explicitly said that retries would be better?
That can be a correct response, for instance, if you have some sort of async i/o in place so you don't want to wait for a write that's not ready because you'll block.
They were not ewview comments. They were arguments about where it ends. He's totalyl unwilling, and says you shouldn't run of of NFS. But then other people are, and are talking about mount options, when the problem clearly lies in the code not following the specs.
Why are:
> 1. If writes need to be retried, why not reads? (No, that's not an invitation to expand the scope of the patch; it's a question about NFS implementation.)
and
> 4. As coded, the patch behaves incorrectly if you get a zero return on a retry. If we were going to do this, I think we'd need to absorb the errno-munging currently done by callers into the writeAll function.
not review comments?
> when the problem clearly lies in the code not following the specs.
There were plenty of questions about how exactly the fix should look like below
https://www.postgresql.org/message-id/AD4A13A5-8778-4D94-BBB...
How's that not review?
> Given that changing the mount option solved the immediate problem for the OP, I fail to see how it's random.
It's random because pretty much anyone could volunteer that opinion. By diagnosing the problem, making a patch and reaching out the author has already made their intention of fixing the problem clear. Unless it has been established that the problem isn't valid a workaround isn't really relevant.
Effective communication and exchange of ideas needs to have a good ratio between work and value. I used to frequent meetups where people would present projects with thousands of hours or work and priorities behind them. There would almost always be someone with less experience stating their "ideas" on what should be done instead. Eventually those people would end up talking among themselves where their ideas could flow freely without any restriction of actual work being done.
Experienced people provide value. They try to understand the problem, add their own experience to it and validate the work that has already been done. They make the problem smaller and closer to a solution. They don't, or shouldn't, casually increase the scope for little reason.
> It's random because pretty much anyone could volunteer that opinion. By diagnosing the problem, making a patch and reaching out the author has already made their intention of fixing the problem clear. Unless it has been established that the problem isn't valid a workaround isn't really relevant.
I'm baffled by this. Even if the fix had been immediately committed, the workaround of using nointr still would have been valuable, because a fixed version of postgres wouldn't immediately have been released.
You seem to argue in a way that entirely counteract your own later comments.
If someone spends x amount of work hours on something, that is what they want feedback on. They aren't looking for quick suggestions on other paths to explore. It isn't a brain storming session. It is work done being represented by an e-mail, code or a product. You are being presented with their theory for a solution. At some point something else might be relevant, but that isn't something to assume. The assumption should be that the person presenting have made their choices based on their situation.
It is often the same with software. Good feedback on software isn't random ideas, suggestions or feature requests that adds hundreds of hours of work on a whim. It is feedback that considers the work that has already been done. Anyone can come up with something else, especially in theory and with a blank slate. It doesn't really require anything other than an opinion. Hacker News certainly is proof of that.
Sure. Postgres misses a bug for years because they can't give even the most basic attention to someone on a mailing list, you know were you supposedly talk about things like bugs, and then defend the whole thing with excuses, rhetoric and further useless assumptions but I am the bad guy. This is exactly why people don't bother. Experience is just a liability these days in the "community". Companies maintain their own patches and conversations happens privately. Just sad to see such potential for accessibility wasted on arrogance. I guess it isn't really surprising though when everyone different already left.
You're basically just making shit up at this point. The most critical message in that thread contained pieces like
https://news.ycombinator.com/item?id=19128154
> You're basically just making shit up at this point.
You keep making my point. All you have is defensive, deflecting and rude comments. If you disagree with something it isn't hard to say "I disagree that x is y because z", like anyone interested in a discussion would. I will keep this in mind when interacting with the community, commissioning work or buying services related to Postgres comes across my desk.
Turns out people don't like one baseless accusation after another. I provided you with concrete references where review comments where made, where people agreed it was necessary. Yet you deny that happened, without refuting anything concrete.
Note that you're the one hiding behind at least the third sockpuppet account.
There is really no point in carrying on this conversation. I have made a number of argument and you are the one who have accused me of things. First I was contradicting myself, then making shit up and now of hiding behind sockpuppets. I haven't denied anything, you singled out a single sentence of my comments where you could snipe at me. The overall point, the discussion, and the repeated arguments I have made still stands.
I am not hiding anymore than any other anonymous account and for no other reason than time management and privacy. I shouldn't frequent hacker news for reasons previously stated, but sometimes I tell myself that the decent thing is to reply. Especially as many people won't. Now it is clear that this discussion isn't going anywhere so I won't be creating any other accounts or comments. To prevent any ambiguity this won't change.
No one suggested having an anonymous account is an issue. The issue is the account being a sockpuppet one.
> I agree retrying is the better response, but erroring out is a form of handling
In this case it is “a form” but that specific handling is provably wrong. That is important: everybody could have tried and proved again. Therefore just dismissing the correct handling and keeping the wrong one is, let me repeat, also provably wrong.
If “write” expects you to always call it in a loop, I wonder why it wasn’t simply implemented so that it calls itself in a loop in the first place. Or at least provide a “writeAll” wrapper function.
What if the disk is literally unplugged after a partial write? (or the network cable unplugged, etc.)
Callers would still need to be informed that a partial write occurred and would need to do something about it.
This is literally the case study from worse is better :) https://www.jwz.org/doc/worse-is-better.html
When I call write(), it is typically already inside a bigger loop. If the system can't write my whole buffer right now, then I am happy to get the CPU back so I can do more work before trying again (and now I might have more data for it too).
Actually as great as Postgres is and as generally approachable the community is - my experience was the same a few times and I read it on the mailing list happening to others:
Someone comes along with a patch or idea. Bunch of big Postgres people come knock it and it dies right there.
Happened to me when I suggested a more multi-tenant approach back around 2010 and today we have Citus. I was told that (paraphrased) no users were asking for that sort of thing.
I see it kind of happening with the Foreign Key Array patch that the author asked for help to rebase and no one bothered to reply.
Someone suggested replacing the Postmaster with a threaded approach so it could scale better (and showed benchmarks of their implementation handling more connections). Community response was there were already 3rd party connection pools that do the job. An outsider looking in considers this nuts - most people would not run additional components if they did not need to!
Another example: NAMEDATALEN restricts every identifier to 63 bytes - a limit that causes frequent issues (more so now that we have partitioning) and also a problem for multi-lingual table names. It’s been proposed to increase this limit a few times. Every time the answer has been: abbreviate your table names or recompile your own version of Postgres.
Could name a few other examples too I’ve noticed over the years and just sighed at. I don’t expect every idea to be accepted or even welcomed - but there is that sense of bias against change.
Yea, that's definitely something that happens. Partially due to some conservatism, partially due to having way more patches than review bandwidth, ...
> Another example: NAMEDATALEN restricts every identifier to 63 bytes - a limit that causes frequent issues (more so now that we have partitioning) and also a problem for multi-lingual table names. It’s been proposed to increase this limit a few times. Every time the answer has been: abbreviate your table names or recompile your own version of Postgres.
I think if it were easy, it'd immediately be fixed, but there's a fair bit of complexity in fixing it nicely. For partially historical and partially good reasons object names are allocated with NAMEDATALEN space, even for shorter names. So just increasing it would further waste memory... We're going to have to fix this properly one of these days.
Re NAMEDATALEN specifically - you at least acknowledge it eventually needs fixing. On the mailing list there were a lot of important Postgres people who don’t seem to agree.
Agreeing that there is something worthy of fixing is a first step. It should have happened with this NFS patch and imo some other stuff. The considerations for how and when should be dear with separately.
Well, that's the thing - changing NAMEDATALEN is a seemingly small change, but it'll require much more work than just increasing the value. Increasing the value does not seem like a great option, because (a) how long before people start complaining about the new one and (b) it wastes even more memory. So I assume we'd switch to a variable-length strings, which however affects memory management, changes a lot of other stuff from fixed-length to variable-length, etc. So testing / benchmarking needed and all of that.
Which is why people are not enthusiastic about changing it, when there are fairly simple workarounds (assuming keeping the names short is considered to be a workaround).
> (a) how long before people start complaining about the new one
Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones.
How much memory are we talking about?
> Very likely many years, or even never. People don't use large names because they like it, they always prefer small ones.
Well, we don't have exactly a barrage of complaints about the current limit either.
> How much memory are we talking about?
Good question.
The thing is - it's not just about table names. NameData is used for any object name, so it affects pretty much any system catalog storing name. A simple grep on the repo says it affects about 40 catalogs (out of 80), including pg_attribute, pg_class, pg_operator, pg_proc, pg_type (which tend to be fairly large).
So the amount of additional memory may be quite significant, because all of this is cached in various places.
Yea, I think pg_attribute is likely to be the main issue here. For one, it obviously exists many times per table, and there are workloads with a lot of tables. But also importantly it's included in all tuple descriptors, which in turn get created during query execution in a fair number of places. It's currently ~140 bytes, with ~64bytes of that being the column name - just doubling that would increase the overhead noticeably, and we already have plenty of complaints about pg_attribute. I think it'd be fairly useless to just choose another fixed size, we really ought to make it variable length.
Is it ~140 bytes? pahole says it's 112 (without CATALOG_VARLEN).
The impact of doubling NameData size would be quite a bit worse, though, thanks to doubling of chunk-size in allocset. At the moment it fits into a 128B chunk (so just ~16B wasted), but by doubling NameData to 128B the struct would suddenly be 176B, which requires 256B chunk (so 80B wasted). Yuck.
> Is it ~140 bytes? pahole says it's 112 (without CATALOG_VARLEN).
Well, but on-disk varlena data is included. pg_column_size() averages 144 bytes for pg_attribute on my system.
> The impact of doubling NameData size would be quite a bit worse, though, thanks to doubling of chunk-size in allocset. At the moment it fits into a 128B chunk (so just ~16B wasted), but by doubling NameData to 128B the struct would suddenly be 176B, which requires 256B chunk (so 80B wasted). Yuck.
I'm not sure that actually matters that much. Most attributes are allocated as part of TupleDescData, but that allocates all attributes together.
> Well, but on-disk varlena data is included. pg_column_size() averages 144 bytes for pg_attribute on my system.
Sure, but I thought we're talking about in-memory stuff as you've been talking about tuple descriptors. I don't think the on-disk size matters all that much, TBH, it's likely just a tiny fraction of data stored in the cluster.
> I'm not sure that actually matters that much. Most attributes are allocated as part of TupleDescData, but that allocates all attributes together.
Ah. Good point.
> I don't think the on-disk size matters all that much, TBH, it's likely just a tiny fraction of data stored in the cluster.
I've seen pg_attribute take up very significant fractions of the database numerous times, so I do think the on-disk size can matter. And there's plenty places, e.g. catcache, where we store the full on-disk tuple (rather than just the fixed-length prefix); so the on-disk size is actually quite relevant for the in-memory bit too.
> Agreeing that there is something worthy of fixing is a first step. It should have happened with this NFS patch and imo some other stuff. The considerations for how and when should be dear with separately.
But there were like multiple people agreeing that it needs to be changed. Including the first two responses that the thread got. And there were legitimate questions around how interrupts need to be handled, about errors ought to be signaled if writes partially succeed and everything. Do you really expect us to integrate patches without thinking about that? And then the author vanished...
> Do you really expect us to integrate patches without thinking about that?
Whoa! You made huge leap there. At what point was it suggested that patches be recklessly applied?
That didn't happen. Your quote actually suggests a reasonable progression and at no point is there any suggestion, implied or otherwise, that changes be integrated without due consideration.
Not irrationally dismissing criticism != abandoning sound design and development.
Multiple people on this thread have expressed the opinion that the patch was "too perfect" or that the only reason that the patch wasn't simply accepted was that the maintainers "feel weird when there's nothing to criticize".
"Multiple people" were left with nothing better to which to attribute maintainer decisions. See the dysfunction there? Don't address the problem, or the proposed solution; just leave it to fester and create discontent.
That's not how well run projects function.
Ugh. I don't know a single PostgreSQL committer with that attitude. Maybe my patches are too crap to be in that situation, not sure.
> Someone comes along with a patch or idea. Bunch of big Postgres people come knock it and it dies right there.
This is not unique to Postgres. I've seen this behavior on many development mailing lists (e.g. Mutt-dev).
While this can and does suck when it happens to you, this is exactly what it takes to keep products focused so they don't die death by a thousand cuts (or feature requests). For every awesome feature embarked upon, there's an opportunity cost of bug fixes, stability updates, tech debt reductions, and other inglorious but necessary work. Aggressively de-scoping is the difficult but necessary work of keeping a product alive in a competitive marketplace. And yes, it's a marketplace even if the product is open source.
Yep, I think it’s an unfortunate side effect of dealing with an onslaught of bug reports, many of which are user error or someone else’s problem. It’s common in any kind of user support, you start seeing the same report over and over.
I even saw something similar when I went to the ER recently. Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else.
"Even doctors will pickup on one thing you say and draw conclusions from that and dismiss everything else."
This pattern seems really common, and is what scares me about the future in general. The 'experts' concentrate on the stuff they understand / are best to the detriment of where the actual focus needs to be. In a lot of cases, this is despite the insistence of the supposed non-expert who is the one suffering as a result.
Some of the worst cases are as a child, where you get in trouble twice. First for wasting adults time because you didn't tell them properly, and then again for pointing out that you did.
I've had this experience with doctors so often it is chilling.
Elderly relatives writhing in pain, only to have doctors say it's indigestion (it was a perforated ulcer and an uncle had previously died from a wrongly diagnosed ulcer perforating). My partner was misdiagnosed with flu when it was pneumonia which then developed into pleurisy (I'd never seen either of the latter, but was telling the doctor that's what the symptoms looked like - 15 years later he still suffers pain from the pleurisy). I had an arm paralyzed through severe pain and the consultant doctor planned an operation "to cut the nerve" - I said I thought it was a frozen shoulder and that such a procedure was unnecessary; 6 months later the paralysis began to subside and the consultant agreed it was a frozen shoulder). Another relative died of bowel cancer that was said to be back pain (she died in the hospital where she worked). I know of several people who were telling the doctor they had cancer, only to have the doctors dismiss it as trivial, with most of these people dying because of their untreated cancer. As a child I had joint pains for years that were diagnosed as "growing pains" but turned out to a hip disease (younger cousins ended up with the same condition and because I'd already had it, they were more readily diagnosed by family members).
In both directions (treating trivial as serious and treating serious as trivial) I've seen so many mistakes. I'd be much happier to see a doctor google the symptoms rather than jump to a conclusion about what is wrong.
There's a famous anecdote where junior doctors are taught the importance of observation, by senior doctors tricking them into tasting urine. It doesn't seem to be a lesson they learn. Even when their own objective test results are contra-indicative of their pre-judgement I've seen doctors scratch their heads but stick with their incorrect pre-judgement.
When doctors I know have a family member go into hospital, you should see how attentive my doctor friends get concerning what is being said and done to their relatives. Some doctors will not even allow relatives to go into hospital for non-emergency treatment at certain times of year (because of timetabling there can be very inexperienced doctors on duty at certain times of the year).
>frozen shoulder
As someone who suffered through that on _both_ shoulders I can sympathize. For me, the doctor missed it. The chiropractor I was sent to, took one look and said it was 'frozen shoulder'. I have never even heard of such a thing before. It took nearly two years to get full movement on my right shoulder. Then the left froze :-(
> coolestuk
I'm just curious. Are these experiences in the UK?
I have seen and heard of some similar things, but my experience is only with the US healthcare system.
I’m a complete outsider to PostgreSQL specifics, but: if that is the reason, then this is a case for improving messaging.
If I’d been in the patch submitter’s shoes, I wouldn’t have thought twice about writing off that community.
If I had gotten your reply, instead: 100% fair play, thank you for your consideration, and send my love to the dev team.
Messaging matters.
I somehow responded with this to the wrong post earlier:
Honestly, if someone's spending time working on a high value open source project (which PG absolutely is), I'd rather they spend less time (than I do) crafting their internet comments to sound nice and more time contributing to society. And I hope people who actually use the product feel the same way, understand why every single use case can't be carefully considered every time it comes up, and don't take it personally.
But messaging influences public perception influences inclination of future potential contributors to participate influences quality of the software. As we see here.
People like to think they can escape politics. They can’t. Any group of >1 humans will involve politics.
Learning how to be respectful and polite is like learning how to touch type: it’s a small, painful price to pay once, for a lifetime of copious reward.
The less value you provide, the more important politics are. If you aren't doing much, then you damn well better have great messaging. If you're doing a lot, and people are banging on your doors to get what you're selling or giving away, then don't waste your time being polite. Nobody cares. They just want you to do what you do.
> Nobody cares.
Except georgebarnett and jarym.
Or perhaps that was the point, that PostgreSQL isn’t doing much! :)
> I patched my own systems (wrap the calls in a loop as per standard practice) and then proceeded to run literally hundreds of thousands of PostgreSQL instances on NFS for many more years with no problems. -georgebarnett
> Actually as great as Postgres is and as generally approachable the community is - my experience was the same a few times and I read it on the mailing list happening to others: - jarym
It seems that despite things not being perfect (and they never are), these people are still having positive experiences with the product and the community overall. So, to restate, some people care enough to post about it on HN, but nobody (who continues to use PG, and there are a lot of us) cares all that much.
I disagree that this is required. You can see from Linus Torvald's backtracking on decades of abrasive behavior that it was never an important part of Linux after all, so an abrasive experience for people trying to help other open source projects is probably going to be superfluous too. You can still reject ideas without disregarding them or the person.
I’m not sure if that logic holds. Who’s to say Linux would not have been more or less successful if Linus had behaved differently? For all we know, Linux may have succeeded despite his behavior, rather than because of it.
That said, I feel that a strong and positive community around a project is always an asset. I’ve seen many more projects fail due to community interaction being bad than I have from it being good.
Are you sure you're disagreeing with the post you responded to? It sounds like you entirely agree with them, but you open as though you disagree...
I think he just got the last two phrases backwards :)
"We don't support running Postgres on NFS" isn't the same thing as "fuck you Intel ACPI team; you're dumber than a chimpanzee". Equating disagreement and criticism with Linus-isms is why the relationship between users and developers is such a mess to begin with. Being a maintainer requires you to say "no" sometimes, but it doesn't require you to be a jerk.
No, it's worse.
Linus was trying to make things work, with profanity. Postgres couldn't be bothered.
Sure, performative profanity isn't everyone's cup of tea, but milquetoast passive-aggressive dismissals of people like OP who ARE TOTALLY RIGHT aren't actually nice.
Please cite which parts of the thread you're referring to. Again, several people +1'd the idea, and reviewed the code.
> You can see from Linus Torvald's backtracking on decades of abrasive behavior that it was never an important part of Linux after all
It's ten years too soon to conclude that Torvalds backing away from abrasive behaviour didn't kill Linux.
The patch author is doing the work. Telling them “no” isn’t going to make them focus instead on the project leadership’s other priorities like it would in a corporate team.
It's not at simple as that.
The patch author does the "initial" work. But then it's up to the team to learn the patch, understand it and keep maintaining it.
Every line of code is baggage.
If there is no demand for something at the time, it makes sense for maintainers to reject that. It's up to them to maintain that patch from now on.
There is another scenario, I submitted a Pull Request to a OSS project, the authors discussed it and rejected it and then implemented it in the exact same way as I did. That was hurtful.
I can't upvote your comment for some reason, but exactly this
Honestly, if someone's spending time working on a high value open source project (which PG absolutely is), I'd rather they spend less time (than I do) crafting their internet comments to sound nice and more time contributing to society. And I hope people who actually use the product feel the same way, understand why every single use case can't be carefully considered every time it comes up, and don't take it personally.
> crafting their internet comments to sound nice
It's not hard.
One thought. Perhaps it comes easier to you than it does to other folks. We all have different skills, and sometimes what looks like malice is really just incompetence. I find that assuming positive intent adds more value to my life than doing the opposite.
A second, separate thought. Perhaps it's not quite as easy as you think it is, and perhaps you're not as good at it as you think you are. Case in point, the comment you just made was not exactly the nicest I've read today.
IMHO it's way more complicated than it seems, because of a mix of technical, cultural and personal reasons.
- technical: E-mail is not a particularly good medium to convey emotions (in either way). For example someone with a naturally terse communication style may be perceived as harsh, while in person he's actually a very friendly bloke. And there's no way to communicate this impression back.
- cultural: Often what is quite polite in one culture may be seen as quite impolite or even rude in another. It's not just country vs. country, but even region vs. region (like for example East Coast vs. West Coast).
- personal: People in the community may know each other pretty well, in which case the communication style may be quite a bit more direct. But others may lack the context.
It's almost impossible to get it right all the time, without resorting to entirely mechanical corporate communication style. Which is not fut at all.
Of course, this does not mean there are no truly harsh / rude / WTF posts (even on PostgreSQL lists). But I'd say most of the time it's not meant that way.
>Telling them “no” isn’t going to make them focus instead on the project leadership’s other priorities like it would in a corporate team.
No one is asking them to. An open source project is not a corporation, it has no shareholders who require growth at all costs. So someone doesn't contribute to the project, if there's enough other contributors to keep it healthy then who care? No need to try and get every single potential contributor to contribute code to the project.
You can’t pay an opportunity cost on an opportunity you don’t have. Whatever the reason for rejecting the patch in this case, it is not a missed opportunity to work on bugs and tech debt as suggested by the parent.
Well, just proposing things and waiting for the team to implement them is easy.
The core team has also to prioritize, deal with already planned features, and shoot down tons of people with inane ideas as well (not just good ones).
> Could name a few other examples too I’ve noticed over the years and just sighed at. I don’t expect every idea to be accepted or even welcomed - but there is that sense of bias against change.
not just bias against change. while there are some very talented and friendly people in the pg community, there are a few bad actors that are openly hostile and aggressive, that feel that they must be part of every discussion. it gets worse when it is a change that they made that is causing an issue, as ego takes a front seat.
unfortunately, these few bad actors make dealing with the pg mailing lists in general very unpleasant, and have made myself (a popular extension maintainer) and others try to keep interaction to an absolute minimum. that's not good for the community.
I'm honestly curious who are those bad actors, and examples of such behavior. I'm not trolling you, I really am curious - because that does not match my experience with the community at all.
I'm sure there were cases of inappropriate / rude communication, but AFAICS those were rare one-off incidents. While you're apparently talking about long-term and consistent behavior. So I wonder who you're talking about? I can't think of anyone who'd consistently behave like that - particularly not among senior community members.
I know that you read the mailing lists, just spending a week on bugs, or following any major or minor discussions on hackers should be enough for you to figure a couple of them out, but I'm not going to personally call anyone out by name on hacker news.
my comment, which is based on the multiple interactions I've had with the community, stands as is: some fantastic people, and a few aggressive bad actors that spoil things.
I kind of wish the pg community came and read this. I guess this is one of the reason why MySQL were much more dominant in the early days and pg had relatively small usage.
IMHO there were other / more important reasons why MySQL was initially more successful.
Yes 63 Identifier limit is a real pain, especially for constraint names, where a long name can convey a lot of valuable info. NOT IN FKs ie opposite of current FK’s for IS IN, would be useful for XOR constraints I would guess they could be implemented quite easily using existing FK infrastructure.
Not the worst thing in the world. I have trouble reading code with very long identifiers. Conventionally, a line of code should not exceed 80 characters or so. That's pretty hard if identifers are 60+ characters by themselves. If your project has standard abbreviations for things, "63 characters ought to be enough for anybody"
I had a project last year where DB names were autogenerated per tenant, with the tenant ID being a UUID, so you're left with 63-36=27 characters. Starts to feel narrow.
(Putting tenant_id as a column was not an option because for each tenant, a third-party software was started that wanted to have its own DB.)
I don't think this is really limited to Postgres. Any relatively big OS project the core committers they tend to do what they do. I submitted a patch to a pretty big OS project and pinged the person assigned to the area. That person pretty much wrote his own version of my patch. I guess it got fixed so that was good. I guess the good thing about open source is if you want you can just build your own version with whatever you want. However smaller projects are more receptive since they want all the attention and help they can get.
I think the "big project" angle is that maintainability is of higher priority than for smaller project. So more minor things need to to be addressed than in smaller projects. And that then either leads to being very nitpicky in reviews (which costs both sides cycles), or polishing patches pretty liberally (which costs the submitter experience, but reduces the time required for both sides).
> and today we have Citus.
After reading this, I have been wondering if the other requests/ideas are not startup ideas
I'm sure some of them are. And Citus is not the only startup pushing PostgreSQL in a way the community did not want to.
If you look at https://wiki.postgresql.org/wiki/PostgreSQL_derived_database..., it's a pretty long list - some of the product are successful, some are dead. And then there are products adding some extra sauce on top of PostgreSQL (e.g. timescaleDB).
The relevant thread: https://www.postgresql.org/message-id/46ED7555-9383-45D2-A5C...
There's literally one person saying that NFS is crapshot anyway, and several +1s for retries. And the former was accompanied with several questions about the concrete implementation.
One the messages told me to go lobby NFS vendors.
Many others were as you note bikeshedding the commit.
Only one, if I recall, questioned why it was a controversial patch at all.
If you check the patch that was committed eventually, IIRC it’s identical to what I proposed.
As I said, I don’t look back particularly favourably on my interactions with the community.
Looking at your submission, it was very good: a clean way to reproduce, reference to the specification that tells users of a syscall how to use it. I'm perplexed that you got so much pushback on this when it seemed pretty straight-forward. I'd guess that this was your first patch to postgres and the default reaction is to be defensive and decline the patch.
Thinking about the social structure for a minute, I honestly think you might have done better to leave out some of the content of your patch submission, let people push back on the more obvious stuff, and have ready answers. There's a phenomena where curators feel they must push back, and they feel weird when there's nothing to criticize - you can get around this by giving them something to criticize, with a ready-at-hand response!
Sorry this happened.
I know that as ‘Take out the duck’.
http://mamchenkov.net/wordpress/2012/07/23/a-duck/
Somewhat OT, but that is generally referred to as a duck[1], and it _definitely_ has its uses.
> The artist working on the queen animations for Battle Chess... did the animations for the queen the way that he felt would be best, with one addition: he gave the queen a pet duck. He animated this duck through all of the queen’s animations, had it flapping around the corners. He also took great care to make sure that it never overlapped the “actual” animation.
> As expected, he was asked to remove the duck, which he did, without altering the real Queen animation.
[1]: https://softwareengineering.stackexchange.com/a/122148
Haha cool, I've been around a while but I've never heard of this. I wonder if this concept can be combined with confessional debugging, aka rubber-ducking? :)
It’s sill wrong to suggest to every contributor to put “the duck” in his contribution, when the fact is then that it’s the people who handle the contribution who are the problem.
I had an American friend who lived in Germany for many years who had a similar approach to dealing with immigration officials. He learned quickly that no matter how thorough he was when applying to extend his stay, his packet would always be "missing" some form. So he started leaving a few forms out of his packet, that way what they asked for was something he already had.
Ooh, you know it seems like you could use that method to misdirect the official from noticing a forged document.
In fact in open source I can imagine a strategy to get an underhanded bug accepted by pairing it with an obvious, trivial bug. The obvious one gets caught and submitter apologizes and resubmits. Then the dev merges the underhanded bug, comforted that a fairly common iteration cycle has been completed with a newbie.
I've seen references to something similar in home remodeling - required paint and trim are stored in a closet during the work, but that closet itself never gets repainted. When the homeowner says "hey, you missed this!" they get the option of a discount and doing it themselves or the contractor sending someone to use the materials already on site. People who complete it themselves have a higher feeling of accomplishment because they were involved rather than simply hiring it done.
This is good advice and something I’ve learned over the past years.
One of the interesting points I’ve relfected on over time has been how _my_ issue was solved with the patch, however for others it wasn’t.
When I think about this I of course recognise that Postgres owes me nothing, however we both had similarly aligned objectives (fix bugs, do good), but because of our inability to successfully communicate, we didn’t get to a productive outcome and so a fixable bug sat in the release for some time.
I do wonder how we go about improving that kind of circumstance.
It's hard. I'm pretty sure that in the general case, the problem isn't solvable.
In Morgan LLywelyn's Finn Mac Cool ( https://www.amazon.com/dp/0312877374/ ), Finn is a man with no family from an inferior tribe, and he's a troop leader in a band of scummy soldiers of no social standing whatever. On the other hand, his potential is obvious -- he will eventually work his way up to self-made king.
While stationed in the capital, he falls in love with a respectable, middle-class blacksmith's daughter. She won't give him the time of day because of the difference in social class, but over time, as his respectability steadily climbs, she warms up to him.
They have a failed sexual encounter, and everything falls apart. She feels too awkward to approach him anymore. He believes (incorrectly) that they've become married at a level inappropriate to her class, and eventually comes to her home to propose a much higher grade of marriage. But she doesn't know how to accept without -- in her own eyes -- suffering a loss of dignity. A painful scene follows in which it's obvious that he wants to marry her, she wants to marry him, her mother wants her to marry him, but somehow none of them can see how to actually get to that point.
People really don't go for innovation in social protocols, even when the protocols they know are failing them.
> you can get around this by giving them something to criticize, with a ready-at-hand response!
So you suggest that the patches should be obviously “worse” in order to have better chance to be accepted?
He is (so much was plain). If those sub-optimal patches are accepted against expectations, a follow up patch with improvements can, in due time, be submitted.
Then please explain how would you create a patch where the only new content is a loop around the existing call to write but “sell” it “with a duck“ easier?
Try with a never-ending loop? That surely gets rejected ;-)
Which of the messages on the thread do you consider as bikeshedding? I personally don't see any - that's not to say I agree with everything said on the thread, but overall it seems like a reasonable discussion.
IMHO it's a bit strange to assume everyone will agree with your patch from the very beginning. People may lack proper understanding of the issue, or they may see it from a different angle, and so on. Convincing others that your solution is the right one is an important part of getting patch done. I don't see anything wrong with that.
I feel this is a little unfair to the community and I encourage others to read the thread before forming a negative opinion.
It seems like a fairly good example of an engineering discussion. Even if the patch is correct and conforming and solves a problem, that doesn't mean there is no use for further discussion, and the surrounding discussion does seem to have merit.
Things like:
* Are there closely-related problems still left unsolved (e.g. retrying reads)?
* Is something not configured according to best practices known by other community members?
* Expressing an opinion that the use case is dangerous, so that nobody (including other people reading the thread in the future) will take it as an endorsement for running postgres on NFS.
* Some legitimate-sounding questions about the specifics of the patch (around zero returns, what writeAll handles versus its callers, etc.).
* At least one person agreed with you that it's a reasonable thing to do.
I don't think HN is really the right place to assess the technical merits of a postgres patch, but the discussion itself seems well within reasonable bounds.
This reply from Tom Lane reminded me of the PC losering issue from the “worse is better” essay [1].
> 2. What is the rationale for supposing that a retry a nanosecond later will help? If it will help, why didn't the kernel just do that?
[1] https://www.jwz.org/doc/worse-is-better.html
I have found the Postgres community extremely helpful and friendly. Perhaps this part of the interaction explains why they dropped the issue for a while?
https://www.postgresql.org/message-id/223BF4DE-B274-4428-832...
Do you have some examples where you disagreed with them or pointed out their mistakes? Obviously everyone is "extremely helpful and friendly" when you agree with them or don't point out their mistakes.
The "asshole reviewer" is definitely a thing.
I don't think I've encountered "asshole reviewer" in postgres community, but maybe my "pain threshold" is just higher than yours. More often than not it's a valid disagreement about technical matters.
Review in the PostgreSQL community does tend to be on the "harsh" side.
I've only submitted one patch (\pset linestyle unicode for psql), and with a few rounds of review and revision it made it in. Overall I found this process nitpicky but productive. However, there is a point at which the harshly critical review can be detrimental, and from the other comments mentioned here, it sounds like this has been the case in the past.
With regard to the semantics of write(2) and fwrite(3) these are clearly documented in SUS and other standards, and the "valid disagreement" in this case may have been very counter-productive if it killed off proper review and acceptance of a genuine bug. There are, of course, problems with fsync(2) which have had widespread discussion for years.
> Review in the PostgreSQL community does tend to be on the "harsh" side.
Can you share an example of a review that you consider harsh? (feel free to share privately, I'm simply interested in what you consider harsh)
I admit some of the reviews may be a bit more direct, particularly between senior hackers who know each other, and from time to time there are arguments. It's not just rainbows and unicorns all the time, but in general I find the discussion extremely civilized (particularly for submissions from new contributors). But maybe that's just survivor bias, and the experience is much worse for others ...
> I've only submitted one patch (\pset linestyle unicode for psql), and with a few rounds of review and revision it made it in. Overall I found this process nitpicky but productive. However, there is a point at which the harshly critical review can be detrimental, and from the other comments mentioned here, it sounds like this has been the case in the past.
Oh, 2009 - good old days ;-) Thanks for the patch, BTW.
There's a fine line between nitpicking and attention to detail. We do want to accept patches, but OTOH we don't want to make the code worse (not just by introducing bugs, even code style matters too). At some point the committer may decide the patch is in "good enough" shape and polish it before the commit, but most of that should happen during the review.
> > Review in the PostgreSQL community does tend to be on the "harsh" side.
> Can you share an example of a review that you consider harsh? (feel free to share privately, I'm simply interested in what you consider harsh)
I wouldn't use the word harsh, but I do think there's plenty reviews where too much bikeshedding happens.
As the other responder noted, "harsh" was probably not the best adjective to use here, so I retract that.
Harsh? All reviews I have got have been nice. The only annoying thing is how some issues can be bikeshedded to death.
I think you patch was too perfect. Always leave some small thing for the gatekeepers to latch on, then be all "oh, yes, totally overlooked this, thank you, I'll fix it ASAP".
I think one of the Dilbert books had this concept. Make one bullet point ridiculous so your boss can have some input and ask you to remove the “Be involved in a land war in Asia” step in your ten point plan.
This is the first I'm hearing of the "take out the duck" concept. I've run software development teams for quite some time now and I have zero problem taking patches that are perfect without nitpicking them.
What the hell is wrong with people?
It's something that emerges in big open source projects. Not really related to software development teams that all answer to the same boss.
It's by far not limited to only open source projects. There are many situations where some fault, no matter how small or irrelevant, has to be found in order to sate the ego of the person doing the review or consideration of a problem or piece of work.
I don't think it's limited to open source project either. But open source projects are unusual in that often even when commit rights etc. are controlled, being able to opine on submitted patches is open, and a way of building social status within the project. And while that often works well, it also does attract people who may even mean well, but will be overly critical because it's their way of feeling they're contributing.
In most corporate projects, odds are you're dealing with a much smaller pool of reviewers.
We’re taking barely-evolved apes whose brains are made for running down antelope and picking bugs out of each other’s hair, and putting them in charge of incredibly complex machines that have only existed for a short time. There are bound to be some problems.
People have insecurities and egos. Sometimes you come across someone who need to feel useful, or need to feel important, or have a need to assert their power. Or that simply don't trust you, and assumes that if they don't see a problem, something is lurking under the surface.
With a lot of people you will never need this. But especially in larger projects where any number of people might latch on and review submissions, the chances for someone to find something to complain about goes up rapidly.
And when one person has found something to complain about, all kinds of other social dysfunction quickly becomes an issue too.
Did you actually read the thread and conclude it to be unreasonable gatekeeping, or are you assuming that it was?
Claiming that there is no problem is in some ways better than gatekeeping and in some ways worse. But you're right, I should not have called this gatekeeping, since the motivations are muddled.
That's an excellent patch. Partial writes (and reads) is a known behavior which is often overlooked. It rarely (if ever) occurs on local file systems, and even the network ones manifest it only from time to time, usually when there is some kind of congestion/segmentation is going on.
It's a pity a lot developers out there do not have an ingrained awareness about this.
Everyone in that thread knows what a partial write is. That's why the patch author saw this error message:
If they didn't know what a partial write was, they wouldn't have identified it specifically as an error case with its own informative error message.
The proposed patch retries rather than throwing an error.
By "short writes" you mean partial writes?
Yes, thank you.
You can't tell people anything.
http://habitatchronicles.com/2004/04/you-cant-tell-people-an...
The same pattern emerges in big commercial projects too. Why do you think it took 20 years to fix console resizing in Windows?
Ballmer and his ilk wouldn't allow it.
In any large organizations, fiefdoms are bound to develop. If you don't have principles for dealing with it, what ends up happening is that new ideas will be crowded out and claimed by one of the leaders of the fiefdom. The Linux model has formalized this process. I think more large open source projects should use this model.
this is why we have facebook
Try committing to a Microsoft open source project or working with a lot of ex-Microsoft employees. You’ll want to kill yourself.
<off-topic>I committed to a Microsoft open source project and worked with (ex-)Microsoft people. The experience was great :)</off-topic>
I've had this experience when I've tried to contribute to ASP.NET Core and .NET Core - in the case of ASP.NET Core, they weren't in the slightest bit interested in fixing something that's obviously broken. In the case of .NET Core, it was made clear that adding something new to the encryption library was going to take a long time, possibly years, even though there was demand for it.
I've given up on both :/
It sounds like you have an interesting anecdote to share.
If you can just give us the anecdote rather than making a vague, sweeping critique of MS OSS projects, you might avoid the downvotes.