I completely agree with one of the comments from there:
> But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better.
It really does not make any sense to use memory-mapped files when writing logs.
Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files.
Only when reading logs, to search for problems, accessing them as read-only memory-mapped files is OK.
Actually not only for logs, but almost always, read-write memory-mapped files are either inefficient or too complex to use (i.e. to avoid problems you must carefully use msync and/or madvise, which eliminates the simplicity that makes memory-mapped files preferable to using pread/pwrite). It is better to use memory-mapped files only for read-only accesses, using the appropriate option flags in open and mmap.
I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...
additionally and ironically in a case like this AI would have been probably more efficient and already resolved the problem with a PR cycle instead of human histeric gate keeping.
This is really astonishing. Are there no checks and balances in place for design decision in such a critical system component? What were the thoughts of all the major distros when they decided to go with systemd then?
systemd was funded and implemented by the biggest commercial distro, Red Hat, first. Other distros were influenced by its decision (RH has always had a lot of influence on distro direction generally) and followed suit.
I’m not sure Red Hat ever had a lot of database design expertise internally and that probably explains the design and outcome. As I said earlier, the design was not subject to public scrutiny before it was implemented.
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):
"""
The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.
If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.
Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.
Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.
[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.
[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...
sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)
That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.
With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.
journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.
> complaints about logs being lost during a power failure or kernel panic
I recall reading somewhere about a thing which persists data in a particular region of RAM that is guaranteed to be left alone by the kernel, and therefore will persist across a reboot. Buffering in such a region could persist data when the kernel panics but it obviously wouldn't survive loss of power.
Wouldn't opening the file O_APPEND (maybe O_DIRECT also?) and using fdatasync be better? That way we've basically implemented a WAL and skipped all the other database parts we don't need or care about.
I would argue the described format is exactly the origin of the problem.
It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.
In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.
The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions.
This optimally solves the problem of the space taken by logs on disk.
The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.
You can also write directly compressed and flush (without resetting state) after every line. Let the compression state reset on reboot, it's not that important to preserve it.
journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.
It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:
[Service]
LogFilterPatterns=~I am a completely useless log entry
LogFilterPatterns=~I am another useless log entry
But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.
rsyslog is an incredible piece of software. Every time I'm looking for something to do with logs, opening the docs or googling finds the feature for me and myriads alternatives. I know it still exists and use it heavily on any system I'm in charge of, but there's some regret at having a dual system with journalctl...
I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful.
all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them.
syslog experience? very efficient also on heavy load production instances, never lost a log, pipe grep and jq and you have the info you need.
so what I experienced is that a default linux install was shipping a rock solid logging system by default, reliable and usable and everybody knew what was where and you will find it. now i just have fancy stuff, units etc and lost all of that.
no I dont need to tune config parameters on a default install to have working basic logging tnx.
WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.
The problem with systemd is the scale. Some of it is fine, some of it is actually quite good. But other areas are just replacing existing systems with things worse for the majority of traditional users. Logging, time and dns come to mind.
> syslog experience? very efficient also on heavy load production instances, never lost a log,
You’re lucky. The original syslog protocol was fire-and-forget UDP (which I believe is still the default, though it’s been ages and I could be wrong) and the daemon was single threaded. I/O or CPU starvation could easily lead to dropped logs.
IIRC those are "dumb" rate limits though, aren't they? I.e. if 1000 of the same dumb message comes across from the service then you'll rate limit out the 3 useful messages occurring at the same time.
I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.
Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.
The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
If you have systemd>=253 you can make use of LogFilterPatterns[0] (in .service files), but it's really unpredictable, cumbersome to work with, and does not work with user services or non-service log sources.
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.
Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.
If you want to store logs in a database, just use standard rsyslog. It has supported database backends pretty much since its inception at the dawn of the century. No need to reinvent anything.
Well I think we found the reason for journald's complexity right there, systemd devs and reinventing the wheel (plus breaking backwards compat in the process) is a match made in heaven
While the rest of systemd is actually pretty modular, journald is unfortunately the only other required component. You can not run systemd without journald running in some way; closest you can get is setting Storage=none and forwarding the logs elsewhere.
journald is in a weird state where its "good enough" and mandatory that most people forget how bad it is until something like this pops up.
Normally I'm pretty happy with systemd and its many components; I even willingly run systemd-resolved, which is probably the other most hated component. But journald makes a lot of weird choices and if I could drop it I would in a heartbeat.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
- workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717
- kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030
- kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805
- bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192
- firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe
- ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316
- mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405
Yeah docker does a ton of network stuff when you start/stop containers, depending on your configuration. It's extra fun because it can drop existing connections when that happens.
Had a process quietly in a crash loop for a solid month on my workstation until I figured out what was causing my random network outages.
> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes
Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
You lose rollbacks, superior syncing, redundancy and data integrity assurance and add complexity for what deceased ssd wear which is hardly an actual problem.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
I wish Qubes Domain-0 was a customized Gentoo with OpenRC. Fedora with systemd was a poor choice to base off. Nobody should have let Poettering have the influence he was given over userland, systemd is an almost irrevocable mistake.
My hunch having looked at the journald code as an amateur is that this write amplification is coming from scattering, with a few possible sources:
1. Writes try to compress away duplicate metadata at the application layer, which causes them to issue scattered writes when new metadata shows up.
2. Indexing is also surprisingly log-line/application-layer aware, such that index writes might also be scattering.
3. The indexes themselves seem like they could benefit from an append-mostly write model with periodic compaction rather than a mutate-in-place model.
4. I was surprised that the journal’s “WAL” doesn’t seem to be a major concern of a lot of the code. For a database, supporting reads “through” the WAL with periodic application back to the data files (“checkpoints” in RDBMS) seems like something I’d expect to see more of here. But I don’t really have deep understanding of the code, so I may be missing that it’s doing that already.
The choice of mmap instead of regular file writes here isn’t, as others have proposed, a design flaw. I think that makes sense given what journald is (a database) and how significant its durability concerns are. And it looks like the code does spend a lot of time trying to be careful about which blocks/pages are dirtied. But this is a famously hard-to-get-write (ha!) area so perhaps defects are present at that layer.
The systemd developers are talented in their area; I am not a systemd hater. However, “talented at low-level OS design” is not the same as “talented at building a database from scratch”, and I think that shows here.
I strongly feel like this system could be a wrapper around SQLite, which is definitely something that could be integrated everywhere journald is used (license-wise and compatibility-wise). I’m puzzled as to why that wasn’t chosen as an approach: a SQLite vfs implementation that handled compression and online rotation seems like it would have resulted in a design that’s both more interoperable and less prone to flaws like this one.
I also think that a per-log-emitter setting that doesn’t eagerly persist to disk (wait for page cache flush) would be very useful to have available—perhaps even as a default—for user-level/init6 level logs that are OK with a potential for data loss on kernel panic.
My caveman understanding is that mmap writes are bad for transactional accesses because you have little to no control over sync. The OS can decide to commit changes to disk anytime, in any order which is the opposite of what you want for anything ressembling a database.
By default yes, that’s true. But while there isn’t a reliable don’t-flush-this-page system, there definitely are ways to force the flush of specific ranges in an mmapped file.
But you’re generally right. I think that’s why most databases have the notion of a WAL, which is carefully append-only. But the non-WAL data files in most DBs I’ve used are accessed via mmap.
A long time ago I had this over-optimistic idea: x86 hardware (and probably most other hardware) has these cool hardware-managed dirty page bits. So you would write to a mapped page, not even take a page fault, and the hardware would record that it's dirty. Later on the kernel would notice and flush. Excellent performance.
Hahaha. It's much much much more complex. For various reasons (maybe good, maybe bad -- see below), Linux barely uses the real hardware dirty bit. Instead, when you map a file as shared-writable, at first it might not really be mapped at all. If you read it, it gets faulted in and becomes readable. When you first try to write to it, a page fault is generated, and, on non-FRED x86, the page fault itself is very slow. The kernel will do things, including calling into the FS and updating atime [0], to make the page logically writable. It updates the page tables so that the CPU knows it's writable, and it sets the dirty bit right then (after all, this is a bit faster than letting the CPU set it immediately thereafter when you retry the faulting write).
Okay, now it's writable. Writes are essentially free until the kernel decides to write the data back to the disk. The kernel will mark the page non-writable (because it wants to get notified the next time you try to write to it) and flush the TLB (which is extremely expensive, especially on x86 systems that aren't the latest AMD CPUs). And it will write the page back, more or less as if you had used normal syscalls to write it.
There's more fun, though. Some filesystems and/or backing stores need "stable pages" -- they need the page cache pages that are being written to not be modified while being written back. btrfs, for example, wants to checksum the data and then write the data and the checksum out consistently, and if something changes the data while it's being DMAed, then this can't happen. So special locks might be taken to delay future writes to the page until writeback is done, and that includes blocking the "make writable" page fault handler. Oops, there goes performance.
Could the kernel do better? Probably. Will it? Unlikely in the near future. I've contemplated a special mechanism to map a "fast write" window onto a file that would be permanently writable and use the hardware dirty bit to tell the kernel when to transfer the data out. Even if anyone ever implemented this, it would be a very specialized thing, it would incur polling overhead, and it would be utterly silly to use it for something like syslog.
Just use pwrite or io_uring unless you have actual evidence that mmap is better.
mmap read is a different story, of course.
[0] I think that updating atime at make-writable time instead of at writeback time is both non-performant and semantically incorrect. I've never convinced the maintainers well enough, though.
I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.
I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.
So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.
(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.
> When did this become a discussion limited to journald on btrfs?
btrfs is in the HN thread title.
> and that seems like something btrfs should fix at some point
Amazing. The Linux kernel should change to work around journald's inflexibility?
What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.
Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
Thank you for your work. ISTM the workload is naturally LSM-shaped.
> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.
> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).
If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.
Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.
If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.
In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.
Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone.
Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log aggregation layer.
My personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure
The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.
SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work.
An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.)
SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.
Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.
I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.
How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?
Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?
On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.
I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right?
On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process.
Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.
Libtorrent 2.0 switched exclusively to mmaped read/writes for torrent downloads, which resulted in various performance and especially memory consumption issues on ALL platforms.
For some reason Windows handled increased memory consumption the least gracefully.
Many people continued to use v1.2 which use regular files.
V2.1 ended up using pread/pwrite nowz it's fine now.
The principled excuse for mmap is when you're reading all over a file at high performance and you want to avoid either excessive syscalls or double caching. Which sounds like what a torrent program does but evidently it doesn't even work well for them.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
Enterprise users increase the logging and I've never heard of premature SSD failure due to this. The event log is capped in size (adjustable). It's nominally < 100MB.
Your games continually dumping GBs of data into local cache on the other hand...
syslog-ng FTW! Been using it since like 2003. Great router tools in the conf, network support, crazy regex rules for when you're trying to tune your syslog collector system after a few beers. Even has this awesome (footgun?) feature that lets me pipe matching lines to other tools I wrote (also beer influenced).
Nice in theory in practice you always retain them locally as well, just in case the network connection goes down.
If network egress fails and logs are pushed in real time over a connection with no local backing, you face an ugly tradeoff: either drop log data silently (loss of visibility during the very network partition you need to debug) or apply backpressure to services (potentially hanging applications when logging buffers saturate).
Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd things being horse shit as usual because it was vibecoded even before LLM existed.
And there are still people that said that systemd and tools are awesome because they never encountered any of the countless ridicule bugs.
Systemd should just use DuckDB. It's perfect for this job.
"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"
Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.
BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.
Seems like there are a lot of mistaken impressions about DB storage engines out there.
Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.
You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?
What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.
I’ve met people that think the log should be remotely stored and not written locally, since it’ll be shipped to splunk or whatever anyway.
Those people change their minds the first time a machine has intermittent network issues, and the logs needed to debug it are lost (or worse, the log buffer fills, then stdout fills, which backpressures the application, creating an outage while simultaneously eating the logs).
Agreed. And in my experience , most large cloud providers’ Linux systems I’ve worked on (either their VMs as a tenant or their underlying hardware as an employee) log locally and ship additionally.
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
> syslog doesn't have nearly the functionality that journald + journalctl does.
A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.
Also, in regards to your original comment:
> This issue report feels like it ought to be accompanied by a fix.
This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!
So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.
[0] AIUI, it came out of Google, but my understanding might be incorrect.
[1] ...regardless of whether or not that effect is intentional...
Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.
(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)
If you think you can do better than journald's existing format, propose a new one with tests to prove it.
The fact that you're personally powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation.
> I'm not entirely sure this is a bug so much as an inefficiency report.
Performance bugs absolutely are bugs... especially when they're in a long-running corporation-backed project that presents itself as the project atop which all Linux systems should be built.
Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make.
Per our Guidelines:
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.
I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not.
However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthesized one?
> Way to double down on the “needlessly picky and combative” angle, dude.
Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading.
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Indeed. It also means "don't come to me with problems alone." Yes, one can come with a problem, but the exhortation is to come with possible solutions as well and seek guidance on which one is best.
Yes, a lot of awful practices have always been a thing.
> e.g. hook in a senior engineer that you know is intimate with the system.
unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use
> unless, of course, you don't know said engineer because you don't even work in the same company...
Or they work in a different part of the fairly-large company that you both work for.
I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relevant to the stuff that the company works on.
I would argue the difference is that upstream is well known for being defensive about their ideas and pushing back.
On LKML you might get cursed out, but if your fix is solid fix, it has high chances of getting through. Regardless of how true it would be in reality, the atmosphere created by upstream is that I do not expect the same with journald unless you convince redhat management
What is the real-world use for these features? Who is this built for?
Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files.
We have some services at work that log to text files and some to journald.
The log volume to file is >> the log volume to journald.
Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and better than something like `journalctl -u myservice.service --since '2026-08-01' --until '2026-08-02' -g 'query'`. (The tab completion and discoverability is also better, I guess)
I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020.
I'm no fan of journald, but I have some methodological issues with this test.
The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either.
Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test.
I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.
Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.
Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
In this case, it's also a very ego-driven project.
Honestly, the few times I went into systemd source (to deal with how they didn't document some critical information without which I couldn't ensure coexistence of other software, software needed for functionality systemd didn't expose), I found it a total mess - combined with very loud and explicit ways the decision of the "stewards" were defended by the team, I would be frankly wary of trying to contribute anything non-trivial.
I completely agree with one of the comments from there:
> But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better.
It really does not make any sense to use memory-mapped files when writing logs.
Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files.
Only when reading logs, to search for problems, accessing them as read-only memory-mapped files is OK.
Actually not only for logs, but almost always, read-write memory-mapped files are either inefficient or too complex to use (i.e. to avoid problems you must carefully use msync and/or madvise, which eliminates the simplicity that makes memory-mapped files preferable to using pread/pwrite). It is better to use memory-mapped files only for read-only accesses, using the appropriate option flags in open and mmap.
great and clear summary thank you!
I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...
additionally and ironically in a case like this AI would have been probably more efficient and already resolved the problem with a PR cycle instead of human histeric gate keeping.
This is really astonishing. Are there no checks and balances in place for design decision in such a critical system component? What were the thoughts of all the major distros when they decided to go with systemd then?
There is a certain consistency to IBM-dominated projects that suggests that there are checks and balances and that they are working as intended.
This design predates the IBM acquisition by seven years.
And Poettering worked at Microsoft until recently.
systemd was funded and implemented by the biggest commercial distro, Red Hat, first. Other distros were influenced by its decision (RH has always had a lot of influence on distro direction generally) and followed suit.
I’m not sure Red Hat ever had a lot of database design expertise internally and that probably explains the design and outcome. As I said earlier, the design was not subject to public scrutiny before it was implemented.
I wonder if something like https://github.com/open-telemetry/otel-arrow would be a better fit instead
Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):
"""
The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.
"""
See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.
Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.
[0] <https://news.ycombinator.com/item?id=49291376>
I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc.
Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.
[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.
[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...
sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)
Is that true even with PRAGMA journal_mode = WAL?
https://www.sqlite.org/wal.html
That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.
With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.
journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.
That would make for a fun experiment for a syslog service. I encourage anyone who wants to give it a shot!
Clickhouse seems to be a popular logstore these days. And is column-oriented.
Yes, WAL by definition means it writes the data at least twice.
Yes it's true even with journal_mode=WAL. The same pages are touched (minimum 1 full page plus one per index), only the atomicity layer is different.
> complaints about logs being lost during a power failure or kernel panic
I recall reading somewhere about a thing which persists data in a particular region of RAM that is guaranteed to be left alone by the kernel, and therefore will persist across a reboot. Buffering in such a region could persist data when the kernel panics but it obviously wouldn't survive loss of power.
Wouldn't opening the file O_APPEND (maybe O_DIRECT also?) and using fdatasync be better? That way we've basically implemented a WAL and skipped all the other database parts we don't need or care about.
I would argue the described format is exactly the origin of the problem.
It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.
In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.
The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions.
This optimally solves the problem of the space taken by logs on disk.
The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.
Witty very large files using some form of indexing (graylog etc) is sensible.
If decompression is a pain though, change your logrotate so it doesn’t compress. Obviously costs more in disk space and less in compute.
You can also write directly compressed and flush (without resetting state) after every line. Let the compression state reset on reboot, it's not that important to preserve it.
journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.
It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:
[Service]
LogFilterPatterns=~I am a completely useless log entry
LogFilterPatterns=~I am another useless log entry
But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.
That was the task for years for syslog services that dealt with it without issue.
rsyslog is an incredible piece of software. Every time I'm looking for something to do with logs, opening the docs or googling finds the feature for me and myriads alternatives. I know it still exists and use it heavily on any system I'm in charge of, but there's some regret at having a dual system with journalctl...
I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful.
all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them.
syslog experience? very efficient also on heavy load production instances, never lost a log, pipe grep and jq and you have the info you need.
so what I experienced is that a default linux install was shipping a rock solid logging system by default, reliable and usable and everybody knew what was where and you will find it. now i just have fancy stuff, units etc and lost all of that.
no I dont need to tune config parameters on a default install to have working basic logging tnx.
WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.
https://www.devuan.org/
The problem with systemd is the scale. Some of it is fine, some of it is actually quite good. But other areas are just replacing existing systems with things worse for the majority of traditional users. Logging, time and dns come to mind.
Gentoo also has a systemd-free option (and even if it didn't, you could make one because Gentoo is basically LinuxFromScratch)
WHAT R UR CFLAGS??!?
> syslog experience? very efficient also on heavy load production instances, never lost a log,
You’re lucky. The original syslog protocol was fire-and-forget UDP (which I believe is still the default, though it’s been ages and I could be wrong) and the daemon was single threaded. I/O or CPU starvation could easily lead to dropped logs.
And still does. I generally replace journald with rsyslogd on systemd setups.
The syslog architecture never had a filter component in the middle that could drop logs before they reached syslogd.
> I'd consider any btrfs install that does not have nocow on...to be defective.
You're getting COW on the extents if you're snapshotting anyway.
systemd-journald also has rate limits that you can configure ;-)
IIRC those are "dumb" rate limits though, aren't they? I.e. if 1000 of the same dumb message comes across from the service then you'll rate limit out the 3 useful messages occurring at the same time.
I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.
Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.
The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.
Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.
I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
If you have systemd>=253 you can make use of LogFilterPatterns[0] (in .service files), but it's really unpredictable, cumbersome to work with, and does not work with user services or non-service log sources.
[0]: https://www.freedesktop.org/software/systemd/man/latest/syst...
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.
Text or text-like (e.g. text content with simple control char delimiters for metadata) would be far superior than the slow-down from Sqlite's safety mechanisms. Optimising logs for read, at the expense of write, is a bad pattern to me.
Read optimized? That is funny because reading logs from journald is dog slow compared to, you know, log files.
I was talking about alternatives like Sqlite. It might optimise complex querying, but writing to it is slower than simple appends.
There's an open source project which is a syslog daemon that stores logs in DuckDB. You can configure journald to forward logs to it.
https://github.com/phare/sloggo
If you want to store logs in a database, just use standard rsyslog. It has supported database backends pretty much since its inception at the dawn of the century. No need to reinvent anything.
> No need to reinvent anything.
Well I think we found the reason for journald's complexity right there, systemd devs and reinventing the wheel (plus breaking backwards compat in the process) is a match made in heaven
Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald.
Why hasn't this been done if it's that terrible?
You can trivially configure it to forward logs to another service to manage them.
that's not the point, why ship most common linux distros with an unreliable logging solution by default?
because its everything or nothing with systemd its a monolith wearing a trenchcoat
Because it's a big fat lie
While the rest of systemd is actually pretty modular, journald is unfortunately the only other required component. You can not run systemd without journald running in some way; closest you can get is setting Storage=none and forwarding the logs elsewhere.
journald is in a weird state where its "good enough" and mandatory that most people forget how bad it is until something like this pops up.
Normally I'm pretty happy with systemd and its many components; I even willingly run systemd-resolved, which is probably the other most hated component. But journald makes a lot of weird choices and if I could drop it I would in a heartbeat.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom
Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
The two I most often see in Ubuntu's dmesg are:
audit - appears to be some sort of AppArmor logging?
br[] - bridge interface docker uses consistently rebuilds itself? May be related to docker compose networking.
Yeah docker does a ton of network stuff when you start/stop containers, depending on your configuration. It's extra fun because it can drop existing connections when that happens.
Had a process quietly in a crash loop for a solid month on my workstation until I figured out what was causing my random network outages.
I noticed plasmashell is write heavy and logs to journald a lot so I have just switched to XFCE partly for that reason.
Unfortunately it's not always easy to move away from systemd. I still run void on one of my machines, but aur make things so much easier.
> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes
Have you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
I don't have experience with xfs or zfs, and I don't have a free stand for experiments right now unfortunately.
You lose rollbacks, superior syncing, redundancy and data integrity assurance and add complexity for what deceased ssd wear which is hardly an actual problem.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
I’ve been using devuan more or less since day one. I highly recommend it.
FreeBSD isn’t too shabby these days either.
I wish Qubes Domain-0 was a customized Gentoo with OpenRC. Fedora with systemd was a poor choice to base off. Nobody should have let Poettering have the influence he was given over userland, systemd is an almost irrevocable mistake.
https://github.com/systemd/systemd/issues/40262#issuecomment...
Thanks for that comment. Out of curiosity how did you come up with that setup? For me most of that test suite sounds alien
If it weren't mmaped files, I would use strace/gdb, or even fuse proxy file system.
But these are mmaped, I don't know any easy debugging or monitoring solution besides writing kprobes/systemtap hooks.
How would you debug it?
My hunch having looked at the journald code as an amateur is that this write amplification is coming from scattering, with a few possible sources:
1. Writes try to compress away duplicate metadata at the application layer, which causes them to issue scattered writes when new metadata shows up.
2. Indexing is also surprisingly log-line/application-layer aware, such that index writes might also be scattering.
3. The indexes themselves seem like they could benefit from an append-mostly write model with periodic compaction rather than a mutate-in-place model.
4. I was surprised that the journal’s “WAL” doesn’t seem to be a major concern of a lot of the code. For a database, supporting reads “through” the WAL with periodic application back to the data files (“checkpoints” in RDBMS) seems like something I’d expect to see more of here. But I don’t really have deep understanding of the code, so I may be missing that it’s doing that already.
The choice of mmap instead of regular file writes here isn’t, as others have proposed, a design flaw. I think that makes sense given what journald is (a database) and how significant its durability concerns are. And it looks like the code does spend a lot of time trying to be careful about which blocks/pages are dirtied. But this is a famously hard-to-get-write (ha!) area so perhaps defects are present at that layer.
The systemd developers are talented in their area; I am not a systemd hater. However, “talented at low-level OS design” is not the same as “talented at building a database from scratch”, and I think that shows here.
I strongly feel like this system could be a wrapper around SQLite, which is definitely something that could be integrated everywhere journald is used (license-wise and compatibility-wise). I’m puzzled as to why that wasn’t chosen as an approach: a SQLite vfs implementation that handled compression and online rotation seems like it would have resulted in a design that’s both more interoperable and less prone to flaws like this one.
I also think that a per-log-emitter setting that doesn’t eagerly persist to disk (wait for page cache flush) would be very useful to have available—perhaps even as a default—for user-level/init6 level logs that are OK with a potential for data loss on kernel panic.
Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
Say more? Sounds like a good story
My caveman understanding is that mmap writes are bad for transactional accesses because you have little to no control over sync. The OS can decide to commit changes to disk anytime, in any order which is the opposite of what you want for anything ressembling a database.
By default yes, that’s true. But while there isn’t a reliable don’t-flush-this-page system, there definitely are ways to force the flush of specific ranges in an mmapped file.
But you’re generally right. I think that’s why most databases have the notion of a WAL, which is carefully append-only. But the non-WAL data files in most DBs I’ve used are accessed via mmap.
The problems are much more than just sync.
A long time ago I had this over-optimistic idea: x86 hardware (and probably most other hardware) has these cool hardware-managed dirty page bits. So you would write to a mapped page, not even take a page fault, and the hardware would record that it's dirty. Later on the kernel would notice and flush. Excellent performance.
Hahaha. It's much much much more complex. For various reasons (maybe good, maybe bad -- see below), Linux barely uses the real hardware dirty bit. Instead, when you map a file as shared-writable, at first it might not really be mapped at all. If you read it, it gets faulted in and becomes readable. When you first try to write to it, a page fault is generated, and, on non-FRED x86, the page fault itself is very slow. The kernel will do things, including calling into the FS and updating atime [0], to make the page logically writable. It updates the page tables so that the CPU knows it's writable, and it sets the dirty bit right then (after all, this is a bit faster than letting the CPU set it immediately thereafter when you retry the faulting write).
Okay, now it's writable. Writes are essentially free until the kernel decides to write the data back to the disk. The kernel will mark the page non-writable (because it wants to get notified the next time you try to write to it) and flush the TLB (which is extremely expensive, especially on x86 systems that aren't the latest AMD CPUs). And it will write the page back, more or less as if you had used normal syscalls to write it.
There's more fun, though. Some filesystems and/or backing stores need "stable pages" -- they need the page cache pages that are being written to not be modified while being written back. btrfs, for example, wants to checksum the data and then write the data and the checksum out consistently, and if something changes the data while it's being DMAed, then this can't happen. So special locks might be taken to delay future writes to the page until writeback is done, and that includes blocking the "make writable" page fault handler. Oops, there goes performance.
Could the kernel do better? Probably. Will it? Unlikely in the near future. I've contemplated a special mechanism to map a "fast write" window onto a file that would be permanently writable and use the hardware dirty bit to tell the kernel when to transfer the data out. Even if anyone ever implemented this, it would be a very specialized thing, it would incur polling overhead, and it would be utterly silly to use it for something like syslog.
Just use pwrite or io_uring unless you have actual evidence that mmap is better.
mmap read is a different story, of course.
[0] I think that updating atime at make-writable time instead of at writeback time is both non-performant and semantically incorrect. I've never convinced the maintainers well enough, though.
What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment...) comment mean? Who are the "large folio people"?
https://news.ycombinator.com/item?id=44113020
So is this a regression or a bug in systemd triggered by this change?
Featured on:
https://northeasttimes.com/2026/08/14/a-single-log-line-eats...
Someone should implement a new operating system that can efficiently handle text processing.
It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.
We could call it UNIX.
I'm probably the main person responsible for making journald usable at all.
But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.
Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.
My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.
Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet.
https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...
The format is documented
https://github.com/systemd/systemd/blob/main/docs/JOURNAL_FI...
Why not just have a SQLite file and call it a day?
Also, why mmaped file?
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic.
There was mailing list discussion at the time journald was conceived though, you can find it if you look.
https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
Well there was an ambition, apparently.
> Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance
> Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/
It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
... Sounds like a signature on a patch that triggers an epic Linus rant on LKML[1]
[1] Happened few times, I think RedHat as a whole even got banned from sending changes for a short while
I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment...
It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
Ah, the Ulrich Drepper school of dealing with reported issues. Time for esystemd ;)
systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.
LevelDB was released in 2011, so it existed but was very new.
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
Write-Ahead Log for... logs?
WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread.
I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...
WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.
So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.
(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)
> journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.
You linked copy_file_atomic_at_full(), why? That function is not in the normal rotation path for journald, it's only used in a workaround when clearing FS_NOCOW_FL fails.
Rotation does not rewrite the log file normally, but there is a hole-punching operation though for reclaiming unused space.
> it's only used in a workaround when clearing FS_NOCOW_FL fails.
Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?
When did this become a discussion limited to journald on btrfs?
and that seems like something btrfs should fix at some point
So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.
> When did this become a discussion limited to journald on btrfs?
btrfs is in the HN thread title.
> and that seems like something btrfs should fix at some point
Amazing. The Linux kernel should change to work around journald's inflexibility?
What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.
> btrfs is in the HN thread title.
as is ext4
Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency.
Because Poettering didn’t invent SQLite.
Thank you for your work. ISTM the workload is naturally LSM-shaped.
> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.
Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.
> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).
If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.
Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1.
Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.
Or, you could write a plain text file.
Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.
If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.
In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash.
Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone.
Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log aggregation layer.
You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.
My personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure
The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.
SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work.
An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.)
SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.
Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.
I don’t think there’s any hard work here. Other than the append-only part, all files would be either immutable (the Parquet parts) or maybe mutated by wholesale atomic replacement of the inode (the catalog, although the directory itself, via its contained filenames) could maybe do that. Readers might have to retry sometimes, but readers would neither have boy expect any write privileges.
How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?
Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?
On the disk the live update part would be either a circular buffer (and readers would need to double-check the start/end marks after reading) or just literal append-only streams. In the latter case, reading would be barely more complex than tail -f.
> I'm probably the main person responsible for making journald usable at all.
Thank you for your service!
I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right?
On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process.
Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.
Libtorrent 2.0 switched exclusively to mmaped read/writes for torrent downloads, which resulted in various performance and especially memory consumption issues on ALL platforms.
For some reason Windows handled increased memory consumption the least gracefully.
Many people continued to use v1.2 which use regular files.
V2.1 ended up using pread/pwrite nowz it's fine now.
The issue continued for 3 years more or less.
https://github.com/arvidn/libtorrent/issues/6667
The principled excuse for mmap is when you're reading all over a file at high performance and you want to avoid either excessive syscalls or double caching. Which sounds like what a torrent program does but evidently it doesn't even work well for them.
Cool to see @ValdikSS here as well. The guy never sleeps or he is AI in disguise ;-)
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
One of the first things I do on win10 is disable most of excess logging.
But why, though? I have never seen a performance hit from it that would warrant that.
Dont like SSD wear for no reason. I wont look at those logs anyway on my personal gaming pc so its all useless.
Your issue is greatly exaggerated.
Enterprise users increase the logging and I've never heard of premature SSD failure due to this. The event log is capped in size (adjustable). It's nominally < 100MB.
Your games continually dumping GBs of data into local cache on the other hand...
How do you do that?
Painfully and slowly clicking one item at a time in computer management/event viewer/windows logs/applications and services :|
Im sure this can be automated, but I want to see what Im disabling instead of going bulk all.
Which log daemon do you use?
syslog-ng FTW! Been using it since like 2003. Great router tools in the conf, network support, crazy regex rules for when you're trying to tune your syslog collector system after a few beers. Even has this awesome (footgun?) feature that lets me pipe matching lines to other tools I wrote (also beer influenced).
https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdJourna...
"The systemd journal doesn't force you to not have plain text logs" -- Chris Siebenmann (2024-06-30)
Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?
https://www.freedesktop.org/software/systemd/man/latest/syst...
See StandardOutput= and StandardError=.
> Oh no! Bad Request > Error: access denied: error in challenge meta-refresh: mismatched token
God I hate the modern web. I get that anti-bot measures are necessary, but at what cost?
You get the exact same information in
$ man systemd.exec
Though reading the question again, I should have probably linked to the equivalent of
$ man systemd-system.conf
as well, that's where you can set the default behavior across systemd, not per-service as the first man page is.
Ok, so I should set these to null? I saw elsewhere someone set journald storage to volatile. Which of these approaches is better?
I just set the storage to volatile and the max log size to 10Mbytes and call it a day.
Logs are really only useful at the tail.
Having the previous boot can get you out of a hole if you're dealing with dodgy drivers.
So change it back when you have a dodgy driver to deal with
On Debian, install rsyslogd instead - it's that simple.
years ago, I set Storage=volatile on almost all the journalD configurations I have. This largely solved this kind of problem.
Good for your personal devices, very not good for servers where you need to have any sort of accountability and security trail.
That’s why you ship the logs off host to a central collector.
Nice in theory in practice you always retain them locally as well, just in case the network connection goes down.
If network egress fails and logs are pushed in real time over a connection with no local backing, you face an ugly tradeoff: either drop log data silently (loss of visibility during the very network partition you need to debug) or apply backpressure to services (potentially hanging applications when logging buffers saturate).
Agreed. That's what the log agent's disk buffer is for. That can still be used even if the journal itself is on volatile storage.
hello ValdikSS! nice to see you alive
journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
it has bad scaling properties esp. if you have many journal files
the last time I contributed to journald upstream was to fix a degenerate behavior with many journal files: https://github.com/systemd/systemd/commit/176f73272e6e3116ca...
that makes a dramatic difference for those hitting this case, but it only gets things from nearly unusable to slow-as-usual.
Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd things being horse shit as usual because it was vibecoded even before LLM existed. And there are still people that said that systemd and tools are awesome because they never encountered any of the countless ridicule bugs.
Systemd should just use DuckDB. It's perfect for this job.
"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"
Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.
BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.
Seems like there are a lot of mistaken impressions about DB storage engines out there.
Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.
You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.
What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.
No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.
You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?
For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.
What do you mean? That has described the vast majority of Linux systems I have ever touched, professionally or personally. Even corporate environments with log aggregation tail system logs rather than having them directly shipped elsewhere. The rare exceptions to this are some embedded devices without much durable storage, or tightly regulated environments in which log data is considered radioactive.
I’ve met people that think the log should be remotely stored and not written locally, since it’ll be shipped to splunk or whatever anyway.
Those people change their minds the first time a machine has intermittent network issues, and the logs needed to debug it are lost (or worse, the log buffer fills, then stdout fills, which backpressures the application, creating an outage while simultaneously eating the logs).
By count, most installs will be the large cloud providers
Agreed. And in my experience , most large cloud providers’ Linux systems I’ve worked on (either their VMs as a tenant or their underlying hardware as an employee) log locally and ship additionally.
We did a small amount of system logs, but anything high volume went directly remote
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Sure, a concrete proposal first would be a good idea.
That said, would you look a gift horse in the mouth?
Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.
It doesn't matter how free a turd sandwich is.
There's no proposal that we can evaluate to determine whether it's a turd sandwich or not.
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
You could however choose to maintain a fork. If you had time. Few people do.
Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.
syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour...
See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
> syslog doesn't have nearly the functionality that journald + journalctl does.
A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.
Also, in regards to your original comment:
> This issue report feels like it ought to be accompanied by a fix.
This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!
So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.
[0] AIUI, it came out of Google, but my understanding might be incorrect.
[1] ...regardless of whether or not that effect is intentional...
Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.
(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)
You: [0]
Also you: [1]
The fact that you're personally powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation.
> I'm not entirely sure this is a bug so much as an inefficiency report.
Performance bugs absolutely are bugs... especially when they're in a long-running corporation-backed project that presents itself as the project atop which all Linux systems should be built.
[0] <https://news.ycombinator.com/item?id=49292944>
[1] <https://news.ycombinator.com/item?id=49291746>
Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make.
Per our Guidelines:
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
> Please respond to the strongest plausible interpretation... Assume good faith.
That's what I did. So, right back at you.
> That's what I did
Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.
> Please explain...
I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not.
However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthesized one?
facepalm
Way to double down on the “needlessly picky and combative” angle, dude.
> Way to double down on the “needlessly picky and combative” angle, dude.
Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading.
GLHF and all that.
> you might find enlightenment, in the style of those classic Zen tales
Doctor, heal thyself.
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Indeed. It also means "don't come to me with problems alone." Yes, one can come with a problem, but the exhortation is to come with possible solutions as well and seek guidance on which one is best.
And it's always been cope. Only the worst kind of person would rather not know about a problem if it's not already solved for them.
Yes, a lot of awful practices have always been a thing.
> e.g. hook in a senior engineer that you know is intimate with the system.
unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use
> unless, of course, you don't know said engineer because you don't even work in the same company...
Or they work in a different part of the fairly-large company that you both work for.
I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relevant to the stuff that the company works on.
I would argue the difference is that upstream is well known for being defensive about their ideas and pushing back.
On LKML you might get cursed out, but if your fix is solid fix, it has high chances of getting through. Regardless of how true it would be in reality, the atmosphere created by upstream is that I do not expect the same with journald unless you convince redhat management
What is the real-world use for these features? Who is this built for?
Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
Writing multiple pages for a few hundred actual bytes also thrashes the OS cache...
Not if you use the correct cache hint flags on the write.
I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files.
We have some services at work that log to text files and some to journald.
The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and better than something like `journalctl -u myservice.service --since '2026-08-01' --until '2026-08-02' -g 'query'`. (The tab completion and discoverability is also better, I guess)
What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.
I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020.
$ time journalctl > /tmp/all.log
real 1m11.364s user 0m52.299s sys 0m6.540s
$ time wc -l /tmp/all.log 3659597 /tmp/all.log
real 0m0.152s user 0m0.056s sys 0m0.096s
$ time journalctl | grep sshd | wc -l
12944
real 0m53.973s user 0m49.535s sys 0m5.210s
$ time grep sshd /tmp/all.log | wc -l 12944
real 0m0.429s user 0m0.332s sys 0m0.100s
https://github.com/systemd/systemd/issues/2460#issuecomment-...
I'm no fan of journald, but I have some methodological issues with this test.
The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either.
Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test.
$ sudo sync
$ echo 3 | sudo tee /proc/sys/vm/drop_caches
I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.
Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.
Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
Nothing says Linux like "information is hosted on a site owned by a user-hostile and privacy-mulching corporation."
You forgot to link to the tests that prove it
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
In this case, it's also a very ego-driven project.
Honestly, the few times I went into systemd source (to deal with how they didn't document some critical information without which I couldn't ensure coexistence of other software, software needed for functionality systemd didn't expose), I found it a total mess - combined with very loud and explicit ways the decision of the "stewards" were defended by the team, I would be frankly wary of trying to contribute anything non-trivial.