Thoughts on software packages

2022-10-29, post № 265

opinion, software-design, #package-manager, #dependency-management

Monolithic kernel or not, a base operating system is seldom equipped for all tasks a user expects to perform on it. As such, most OS’ provide facilities to run custom executables on the machine, as well as loosely managing them when running. However, software acquisition is largely thought of as an extra layer on top of the base system, interwoven with it to various degrees: a userland’s package manager’s job.

Unfortunately, fueled by the manic lust for usability by a fabricated usager incompétent, the classical package manager maxim (as acted upon by e. g. Advanced Package Tool (APT)^[1], Pacman^[2], Portage^[3], Dandified YUM (DNF)^[4], Zypper^[5]) seems to fade in popularity, with large GNU/Linux distributions gravitating towards container bundles (see e. g. Flatpak^[6], Snap^[7], AppImage^[8]): whilst their advantages of dependency encapsulation and supposed security benefits are never-endingly chanted by their proponents, in my view they are merely an ad-hoc solution. Instead of crafting solid software with thoughtful third-party inclusions and kempt versions of all parts (which would include replacing abandoned upstreams instead of freezing them together with their reason for abandonement indefinetely), a jumbled mess of any bit that has not evaded the grasp of GitHub is crammed into a bloated archive, coated in a generic runtime and made executable to fool an unsuspecting quick glance into believing one is installing a piece of software worth one’s while.

A cleaner approach which does not use the plethora of unwritten bits found on modern storage devices as a lazy cop-out to understand a package as a miniature virtual computer but nonetheless achieves the desirable property of package atomicity, solves the issue of competing library names or versions and even promises reproducible builds is to improve upon the classical maxim but not discard it (see e. g. Nix^[9], Guix^[10]). However, I feel like these projects get carried away by their founding principles, slowly encroaching on the entire system: understandling themselves as much more than merely a package manager, they become the distribution and even want to manage user-specific configs (see e. g. guix home^[11]). As such, they fall into the trap of complexity and bloatedness, alienating those who came for a Unix experience.

Notably, Guix’ appeal does not stop at boot-level reproducible systems: it is the one entry on the FSF’s list of endorsed GNU/Linux distributions^[12] which I managed to successfully use for a few days. The signficance of the existence of such a distribution should not be underestimated, since — yet again unfortunately — the ideals of free software increasingly fall by the wayside with proprietary firmware, kernel blobs and spyware for the masses being accepted by many. Combined with GNU’s declining relevance^[13], it might be that Stallman’s four freedoms^[14] fail to capture a now relevant dimension of software; in my opinion a pertinent one is to restrict source complexity.
Filling this void, the concept of software minimalism, famously advocated for by suckless.org e.V., prohibits many of the driving forces behind abandoning free software principles: cutting-edge hardware pressure, bloat-cope and dependency trends to name a few.

If now Guix has shown that a package manager can be used as a vehicle to preach a message but — for me more importantly — also comfort believers, I sense there to be an open niche for a package manager representing both software minimalism and software freedom.

Peculiarly, my proposed niche seems to at first glance be diametrically opposed to the core function of a package manager: managing complex package hierarchies. After all, many few-man-show projects choose to go with the in-memory installer route of curl shady.org | sudo sh. Leaving security concerns aside (should one have reason to assume malice, installation via less exploitable means is only a boon when never running the installed), lack of transparency makes uninstalling a daunting task plus further obfuscates the user’s perception of their own system — contrary to the goals of free software.
Furthermore, the separation of program binary and run command (i. e. rc, config file) is blurring evermore in light of suckless-style architectures (see e. g. dmenu^[15], dwm^[16], XMonad^[17], knôtM^[18]) lifting configurations to the compilation stage, thus leading to a deluge of similar forks — a scenario classical package managers are not equipped to satisfactorily handle.

On the topic of architectural changes, the decrease in sheer source volume also makes dropping pre-compilation more attractive: Guix consoles the wary with the promise to only optimise reproducible building by injecting pre-compilations and cryptographically proving identity, whilst Portage bears the burden of hour-long compile times. Yet many package managers rely on binary distribution of clouded origins, which itself undermines the principles of free software: claiming to treat people in a free manner is not just about pointing at a licence file on a project’s homepage and executable poisoning is one of the least arduous vectors a miscreant could dream of.
Suffice it to say, software freedom isn’t worth a cent without the source, which is often unfairly treated like an addendum to the binary and thus forgotten about; never shipped. I think source compilation is paramount and beyond its conceptual merits, minimalistic software does not suffer from excruciating compile times, rendering compilation on delivery viable.

From a deeper architectural view, dependency management and the attitude towards dynamic libraries has also changed over the last decade: go’s toolchain relies heavily on ad fontes source inclusion [C18^[19]] and the benefits of dynamic libraries are nowadays hard to make out [D20^[20]]. My few encounters with such system-wide dependencies of late have only been Google’s brotli wrapper for go^[21] (I am unsure if the hassle to rely on cgo is worth the presumed performance superiority) as well as the never-dying fossils Xlib^[22] and ncurses^[23].
Some research-stage approaches have gone even further, dissecting the concept of a source package into finely hash-indexed language-semantic atoms instead of collections of files (see e. g. Unison^[24], Fragnix^[25]).^[26]
I surmise the disappearance of a pressing need to save disk space, evermore longed for moral and security-related transparency paired with the tremendous potentials e. g. dead code elimination, call-specific fusion and measurement-based custom optimization heuristics open up will help the proliferation and maturing of such heavily source-oriented software design approaches.

My own involvement in the area of package management stems from the realisation that my current dotfile management leaves a lot to be desired: fed up with the myriad of individual files I had to manually manage (a shell rc, an editor rc, color schemes for said editor, and various utilities either written in shell or C — thus also needing compilation), in 2021 I started to verbatim paste or base64 encode everything into my shell rc, effectively devising an ad-hoc archive format.^[27] Whilst undoubtably in line with the Unix tradition [KP84^[28], pp. 97—99: 3.9], it proved to be both fiddly to manage and did not play well with version control. Thus, I began thinking about a unified framework which enables me to portably and recreatably manage my configurations.
This ongoing effort has currently manifested itself in fleetingly^[29], a light-weight package manager with the goal to un-intrusively and atomically add functionality to a running system, journalling what it did to it. As such, fleetingly need not worry about system-integral or deeply dependant packages nor packages of great binary size. It represents my attempt to fill the above described niche, paying attention to proper licencing, proper attribution, software freedom and software minimalism.