Don't pay too much attention to the inner-circle feedback. Some people subconsciously use the tone of reception to control you, especially peers/colleagues are prone to this. They see someone having a bright idea, and they genuinely don't like it just because it makes them feel small in comparison. The thing they do next - they try to extinguish the spark by creating an illusion of worthlessness in your mind. By doing so, they are getting rid of a potential competitor on their own way to success. According to psychology, they start feeling normal again by reducing you, this is a natural compensatory mechanism ingrained in human archetypes.
The key thing here is to be able to discern between real and manipulated input information. For this, gut feeling usually works best: it spots that whiny, attention-insisting, importance-seeking, fear-inducing tone of a manipulator, but oftentimes the higher-level nervous system of the brain suppresses that signal (e.g. "How can a well-educated and charismatic person feel a bit off and responsibility-lacking sometimes? It cannot be right, so it must be something with my interpretation of reality. I bet they have the best intentions.")
An even better approach may be not to search for feedback at all, unless it's shared naturally without any strings attached.
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF file
> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
yes you are right, in the article the author thinks he just realized something special "elf is a data base", but no, is not, its just a structured piece of data "a database", sqlite file format is also that, the only diference between Elf and sqlite file is that the latter has a a better "client" to inspect it.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
I wrote a tool[0] for manipulating elf files to support inline assembly for a compiler that doesn’t support it (based on a similar python tool). That required knowing any of the ELF sections that were being changed and how those changes would impact other sections.
sqlite would have made this pretty trivial. Replace the .text fields of a few rows. Insert a few rows for symbols, update a few from the old object.
The best part - there’s tons of library support for sqlite. If it was a new object format, there would be no support and I’d just have to write different parsers and generators.
The schema might the the biggest issue for efficiency. As others have noted, ELF->SELF->ELF might be the best use case for compatibility. That said, a big part of [0] was performance, and I don’t know how sqlite would have done compared to my naive elf parsing and manipulation.
This was a fantastic read thanks for putting it together. I'm always fascinated by binfmt_misc, I remember fooling around with steganography a few years ago and putting executables inside images that could then be executed using a wrapper tool + binfmt_misc.
I think this is awesome though, feels like a lot of things on OS's could just be represented by databases - where does it end?
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1
9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
The TCL community experimented with an idea like this many years ago; they were called "structured documents" or "starpacks". What we found was that (A) it's usually more convenient to keep the data in a separate file, and (B) virus checkers can get suspicious when your app starts modifying itself, leading to unintentional comedy in operations. YMMV.
> The preload table is a list of objects to map last, so their exports win.
made it sound like the 'map last' is a consequence of this new perspective, but it’s really still a loader convention: by definition, the symbol-resolution query needs to give entries in the preload table precedence over ordinary dependencies.
> ELF MECHANISM: .strtab / .dynstr
> The database primitive it reinvents: string interning
> Surprisingly a lot more falls out as well: .dynstr is gone, because name is TEXT and SQLite already interns strings
What's with this claim? SQLite does not intern strings, as far as I can find anywhere, and a quick test shows that duplicate strings are actually duplicated in a database file. You can intern strings in SQLite manually with an intern string table, but it doesn't happen automatically.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
The lack of mmap is a problem, especially given how dynamic link heavy Linux distros are. It’s going to cause a lot of extra RAM usage and the kernel won’t know to dump those pages efficiently and will swap them to disk. The extra RAM usage will probably be recovered - the kernel typically these days has deduplication capabilities - but the lack of efficient swap is a problem.
I've wanted to do something like this for quite a while! Of course I wanted to make my own OS, language and DB from scratch in order to do it, so naturally I've never gotten around to it. It goes to show that there is nothing new under the sun.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
Some of it is convergent evolution at work. 60s/70s/80s specialized formats as much predated modern database designs and data structures as coexisted with them. SQLite is the culmination of decades of database design. Some of that was influenced by those specialized formats of previous decades that SQLite could now in theory replace.
(Which is to say a lot of the specialized formats didn't exist just because of compute and storage resource limitations, but also because training/research/standards work came later. Plus complications like software patents. ZIP files have been around since the 1990s but ZIP only became a common general use container format for "everything" almost exactly 14 years later when patents finally expired.)
This whole article is fantastic, but already at the start, the SQLite virtual tables thing is blowing my mind.
https://www.sqlite.org/vtablist.html
You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing.
This sounds like it could be extremely useful.
Check this out: https://github.com/osquery/osquery
Check out https://hub.steampipe.io/, available as sqlite/postgtesql extensions
(author) I'm enjoying the comments. When I published a short-paper with this idea in academic circles, the feedback wasn't so kind
Don't pay too much attention to the inner-circle feedback. Some people subconsciously use the tone of reception to control you, especially peers/colleagues are prone to this. They see someone having a bright idea, and they genuinely don't like it just because it makes them feel small in comparison. The thing they do next - they try to extinguish the spark by creating an illusion of worthlessness in your mind. By doing so, they are getting rid of a potential competitor on their own way to success. According to psychology, they start feeling normal again by reducing you, this is a natural compensatory mechanism ingrained in human archetypes.
The key thing here is to be able to discern between real and manipulated input information. For this, gut feeling usually works best: it spots that whiny, attention-insisting, importance-seeking, fear-inducing tone of a manipulator, but oftentimes the higher-level nervous system of the brain suppresses that signal (e.g. "How can a well-educated and charismatic person feel a bit off and responsibility-lacking sometimes? It cannot be right, so it must be something with my interpretation of reality. I bet they have the best intentions.")
An even better approach may be not to search for feedback at all, unless it's shared naturally without any strings attached.
More nix content
Curious what that feedback was, specifically?
I won't share any feedback specially but choosing low level primitives as research I found challenging due to the unfamiliarty in peer review circles.
I enjoyed all the papers I read on my graduate classes that bordered on "art" whereas now they seem to be obsessed with performance only.
Yes !
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
nightmare for anything that relies on checksum based security :D
> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
yes you are right, in the article the author thinks he just realized something special "elf is a data base", but no, is not, its just a structured piece of data "a database", sqlite file format is also that, the only diference between Elf and sqlite file is that the latter has a a better "client" to inspect it.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
(And that’s totally ok)
I wrote a tool[0] for manipulating elf files to support inline assembly for a compiler that doesn’t support it (based on a similar python tool). That required knowing any of the ELF sections that were being changed and how those changes would impact other sections.
sqlite would have made this pretty trivial. Replace the .text fields of a few rows. Insert a few rows for symbols, update a few from the old object.
The best part - there’s tons of library support for sqlite. If it was a new object format, there would be no support and I’d just have to write different parsers and generators.
The schema might the the biggest issue for efficiency. As others have noted, ELF->SELF->ELF might be the best use case for compatibility. That said, a big part of [0] was performance, and I don’t know how sqlite would have done compared to my naive elf parsing and manipulation.
0 - https://github.com/ttkb-oss/metrowrap
(author) In the post I say I did write sqlelf which is just that, sqlite + virtual tables prior to this.
This was a fantastic read thanks for putting it together. I'm always fascinated by binfmt_misc, I remember fooling around with steganography a few years ago and putting executables inside images that could then be executed using a wrapper tool + binfmt_misc.
I think this is awesome though, feels like a lot of things on OS's could just be represented by databases - where does it end?
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1 9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
Doesn't redbean do this in a more portable way (not ELF dependent)?
The TCL community experimented with an idea like this many years ago; they were called "structured documents" or "starpacks". What we found was that (A) it's usually more convenient to keep the data in a separate file, and (B) virus checkers can get suspicious when your app starts modifying itself, leading to unintentional comedy in operations. YMMV.
These are the kind of posts i love.
One nit-pick;
> The preload table is a list of objects to map last, so their exports win.
made it sound like the 'map last' is a consequence of this new perspective, but it’s really still a loader convention: by definition, the symbol-resolution query needs to give entries in the preload table precedence over ordinary dependencies.
This was super fun. Dude’s been on an absolute tear recently.
> ELF MECHANISM: .strtab / .dynstr > The database primitive it reinvents: string interning
> Surprisingly a lot more falls out as well: .dynstr is gone, because name is TEXT and SQLite already interns strings
What's with this claim? SQLite does not intern strings, as far as I can find anywhere, and a quick test shows that duplicate strings are actually duplicated in a database file. You can intern strings in SQLite manually with an intern string table, but it doesn't happen automatically.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
The lack of mmap is a problem, especially given how dynamic link heavy Linux distros are. It’s going to cause a lot of extra RAM usage and the kernel won’t know to dump those pages efficiently and will swap them to disk. The extra RAM usage will probably be recovered - the kernel typically these days has deduplication capabilities - but the lack of efficient swap is a problem.
Congratulations, you’ve rediscovered mainframes.
I've wanted to do something like this for quite a while! Of course I wanted to make my own OS, language and DB from scratch in order to do it, so naturally I've never gotten around to it. It goes to show that there is nothing new under the sun.
The word "Your" was dropped from the start of the title for some reason.
How many layers of "SQLite for everything" are we at now?
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
https://sqlite.org/aff_short.html
https://sqlite.org/appfileformat.html
Some of it is convergent evolution at work. 60s/70s/80s specialized formats as much predated modern database designs and data structures as coexisted with them. SQLite is the culmination of decades of database design. Some of that was influenced by those specialized formats of previous decades that SQLite could now in theory replace.
(Which is to say a lot of the specialized formats didn't exist just because of compute and storage resource limitations, but also because training/research/standards work came later. Plus complications like software patents. ZIP files have been around since the 1990s but ZIP only became a common general use container format for "everything" almost exactly 14 years later when patents finally expired.)
ha ha ha
ha ha ha
yes