SystemD Service Hardening

This discussion on Hacker News primarily revolves around systemd, its design, adoption, and security hardening features, with a notable side tangent on software packaging and naming conventions.

Systemd's Design, Adoption, and Perceived "Win"

A significant theme is the historical debate and eventual widespread adoption of systemd, despite years of criticism. Many users acknowledge the criticism but also point to systemd's practical benefits in solving real problems, especially for distribution maintainers and professional users.

• "The unreasonable effectiveness of writing a security scanner. People will do anything it takes to make the scanner give a perfect score, regardless of whether it makes sense."
• "FooBarWidget: Aside, I'm quite surprise that systemd won out, given all the hate for so many years. You got nearly every forum and HN thread that even tangently mention systemd, flooded with people proclaiming systemd to be the spawn of the devil "forced down their throat". At the same time I noticed that a lot of regular Linux users (like developers that just want to deploy stuff) seemed to like systemd unit files. I guess all those complainers, however vocal, were neither the majority, nor the people who actually maintained distributions."
• "bluGill: Systemd won because for all the deserved hate it solves real problems that nobody else was really trying (or their attempts were worse). There are some really good points to systemd - you just have to takes the bad with it."
• "PhilipRoman: Having the backing of RedHat certainly didn't hurt..."
• "dale_glass: That's because systemd knew who the target users of it were: people making distributions, and professional users with little desire to be woken up at 3 AM to troubleshoot a stuck PID file."
• "eythian: My feeling, without evidence, is that a lot of the hate is vocal people who don't like this thing that does stuff differently to what they're used to."
• "FuriouslyAdrift: A lot of the hate was because the main developers interactions wiht just about everyone was horrible, the code was opaque at best most of the time and constantly changed, scope creep, and the feeling that it was a massive power grab by Red Hat. It can be argued that it didn't solve very many problems and added a huge amount of complexity."
• "bayindirh: systemd is heavily inspired by macOS' launchd."
• "formerly_proven: And Solaris SMF. There basically seem to be ~three generations of unix init: 1. Agglutination of shell scripts 2. "Oh wow, this is getting annoying"-phase: Wrapper for scripts (SRC SMC openrc etc pp) 3. Service supervision daemons (SMF, launchd, systemd)"
• "jcgl: Yeah, I'd probably call systemd something like "an event- and graph-based orchestrator.""
• "candiddevmike: Not really, AIUI pottering just thought launchd's socket activation/inetd like functionality was neat: https://0pointer.de/blog/projects/systemd.html. Upstart is more of a direct ancestor"

Systemd's Security Hardening Features and Adoption

A prominent theme is the discussion around systemd's security hardening capabilities, such as ProtectSystem, TemporaryFileSystem, and credential management, and the slow adoption of these features by package maintainers. Users debated whether these features should be implemented by upstream developers or distribution maintainers, and the practical challenges involved.

• "ibizaman: Nice tip on debugging syscall issues!"
• "johnisgood: By the way, I love that the OP used the Text Fragment (Scroll-to-Text Fragment) feature. I hope it is going to catch on more, quite helpful / useful."
• "jauntywundrkind: Much better article with very real tips about what options to try than yesterday's (weirdly flagged/dead?) post on the topic. Which while I really enjoyed lacked substance; I was in the comments trying to provide a more useful basis with some real examples, but this is an exemplary list of awesome ways systemd can easily quickly readily provide aassive boost to isolation & security. Great write up!"
• "igorramazanov: Automatic systemd service hardening guided by strace profiling https://github.com/desbma/shh"
• "isodude: A nice thing I found is that if you do (which I see they did not in the examples)ProtectSystem=`` you can doTemporaryFileSystem=/:ro`BindReadOnly=/usr/bin/binary /lib /lib64 /usr/lib usr/lib64 <paths you want to read>And essentially just including the binary and the path you want available. ProtectSystem= is currently not compatible with this behavior. EDIT: More info here: https://github.com/systemd/systemd/issues/33688"
• "DyslexicAtheist: these Hardening variables have been discussed some years back[1]. this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files). Maybe they will? But this same approach hasn't worked with apparmor since AppArmor maintainers provide skeleton-templates in many cases that will make the parser stop complaining. ("look I have a profile so apparmor shuts up, but don't take too close a look OK") Then there is firejail, which some argue[2] is snake-oil considering the high level of administrative glue compared to its massive attack-surface (also it's a setuid binary). I didn't mention SElinux since I don't know a single person who had the joy (or pain depending on perspective) of working with it. But again, seems the expectation to implement security with it is shifted to the user."
• "ptman: It would be nice to be possible to do the hardening of services via allowlisting instead. E.g. AllowNothing=true and then start adding what is allowed to make the service function."
• "dralley: > this will not take off I'm afraid, because locking these unitfiles down is offloaded to the end-user (I've yet to see maintainers embrace shipping locked down files). https://fedoraproject.org/wiki/Changes/SystemdSecurityHarden..."
• "DyslexicAtheist: thanks for the link, this is great news."
• "dfc: It says the change was dropped? I guess at least they tried."
• "ciupicri: > Change is dropped because of inactivity. Owner is welcome to resubmit if the work is picked up again."
• "Palomides: imho adding a few lines to the systemd unit file is waaaaaay easier than apparmor and really, these should be written by the developers, not distro maintainers"
• "jcgl: > I think it should be done by the maintainer of the software not the distro Why would you say that? I would agree that the developer likely has better insight into what the software needs. But the security boundary exists at the interface of the application and the system, so I think that both application devs and system devs (i.e. distros) have something to contribute here."
• "jamesponddotco: Another great security feature systemd provides is credential management[1], which allows you to expose credentials to an application in a more secure way than, say, an environment variable or a file in the filesystem."
• "smjburton: Thanks for sharing this. It looks like you can also use "systemd-analyze" with the "--user" flag to inspect systemd user units as well ("systemd-analyze --user security"). I've started using systemd more now that I've transitioned my containers to Podman, and this will be a helpful utility to improve the security of my systemd unit/container services."
• "0xbadcafebee: It would be cool to have a repo with suggested hardening for common services, since there's so many different hardening options."
• "c0balt: From packaging stuff for nixpkgs, a distro that often is without upstream support, it is usually very useful to look at how mainstream distro package services. Those hardening steps also tend to be well tested even if sometimes a bit lax. If you want to find out how, E.g., postgresql can be hardened, consider looking at the Debian, Ubuntu and/or RHEL packages as a starting point."
• "0xbadcafebee: Distros don't usually do security hardening, unless the distro is security-specific. They slap something generic on like AppArmor or SELinux and call it a day. (This article is the proof of that... all the default services are not hardened). Usually this is a good thing, as it prioritizes usability, and lets the user harden as they wish."
• "neuroelectron: Great article. I really appreciate the list of properties and the "check the man pages, good luck" advice. systemd is really a great piece of software I would enjoy deploying on my hosts!"
• "temp0826: Why don't distros flip more of these switches? Are there cons of being more aggressive with these settings? It's really a lot for many people to tinker with."
• "colinsane: > Are there cons of being more aggressive with these settings? well, the con is you might unknowingly break some setups. take NetworkManager: after tightening it down, did you check both IPv4 and IPv6 connectivity?"
• "mkipper: This question is sorta similar to "Why don't distros enable restrictive MAC policies by default". Maintainers could take the time to lock down sshd and limit the damage it can do if exploited, but there are costs associated with that: 1. Upfront development cost 2. Maintenance cost from handling bug reports (lots of edge cases for users) 3. Maintenance cost from keeping this aligned with upstream changes."

Naming Conventions and Minor Nitpicks

A smaller but persistent theme involves discussions about the correct capitalization of "systemd" and other similar software names. This also touches on the user experience of dealing with project branding and the tendency of some users to deliberately misspell names as a form of protest or insult.

• "eliaspro: Quoting https://brand.systemd.io/#:~:text=Yes,%20it%20is%20written%2... "Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. [...]""
• "viraptor: At some point, if you have to write articles about the proper spelling of the name, maybe you should just accept the alternative names as well. Also looking at you Datadog and Cloudflare. (The employees of the second one are especially allergic to CloudFlare for some reason)"
• "pilif: color me surprised to see an article mostly constructive and positive about systemd using the wrong capitalization of the project name. Normally the rule is that people mis-capitalizing the name are usually critical of the project. It's systemd, not SystemD"
• "delusional: Nitpick and title correction: The proper spelling of systemd is systemd, not SystemD. According to their brand page: Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd."
• "bee_rider: Interesting! I’ve mostly seen it written as systemD. I wonder why that seems to be so popular…"
• "roryirvine: Many of the more juvenile systemd haters used to deliberately spell it that way. As an insult, it was rather less successful than the "Micro$oft" / "Slowaris" / "HP-SUX" mockery from the 1990s - but it did manage to sow enough confusion that that it still pops up regularly today, even in contexts that are otherwise neutral or positive about it."
• "brongondwana: We tried very hard to convert FastMail to Fastmail and... it's been about 90% successful but there's definitely a bunch of things out there spelled the old way. We just joke about BIG M occasionally."
• "johnisgood: And I tried telling people "news-teller" (or "news teller") is a better word than "herald", and I got told off, so I dunno."

Software Packaging and Distribution on Linux

The conversation also touched upon the complexities and frustrations of software packaging and distribution on Linux, contrasting it with the perceived ease of development on Windows. This included discussions about binary compatibility, dependency hell, and the challenges of cross-distribution packaging.

• "mschuster91: The actual problem is dependency hell. You might be lucky to be able to transplant a Debian deb into an Ubuntu installation and vice versa, or a SLES rpm to RHEL, but only if the host-side shared libraries that the package depends on are compatible enough on a binary level with what the package expects."
• "dale_glass: IMO, binary compatibility on Linux isn't really solvable. There's just a thousand tiny projects that make up the Linux base that aren't on the same page, and that's not about to change."
• "jcgl: I only started packaging relatively recently. Using OBS definitely made things easier, but it's crazy how much nicer RPM is than dpkg. So much better to have more-or-less everything inside a spec file with macros, versus dpgk's mess of static, purpose-specific files."
• "bee_rider: Anyway, the ecosystem stays heathy because of code contributions. So what’s the point of binary compatibility (from the point of view of the people actually making Linux work: Open Source developers and repo maintainers)?"
• "mschuster91: Want to see Linux on the desktop actually happen? Then allow a hassle free way for commercial software that is not "pray that WINE works good enough" aka use win32 as an ABI layer."

Alternative Init Systems

Brief mentions of alternative init systems like OpenRC and runit surfaced, with some users preferring their simplicity or debugging ease over systemd, though others found them too simplistic or lacking modern features.

• "philliproman: I don't care either way, although I still think OpenRC style scripts are much easier to debug and sometimes have more elegant solutions (templates vs symbolic links)."
• "carlhjerpe: I can't say I know what OpenRC really does, but at it's core systemd is just a glorified script runner as well?"
• "dale_glass: systemd tries to avoid scripts as much as possible. /etc/init.d/whatever were all shell scripts, and they all had to implement all the features themselves."
• "immibis: I use runit for some non-system-level stuff. It's extremely simple, possibly too simple. It doesn't manage load order - if a dependency hasn't loaded yet, you just exit the script and the service manager tries again in 2 seconds. Service scripts are just shell scripts."
• "0xfedcafe: Best systemd hardening is switching to OpenRC or runit"
• "mcpherrinm: Do you have any references for doing similar system hardening under either of those?"
• "yjftsjthsd-h: Why would OpenRC or runit be any less likely to boot?"

Miscellaneous Observations

• The discussion included a minor technical glitch with the website's certificate.
• There was a question about the "low quality" and "low density" of the original post, which was met with a request for elaboration.
• One user expressed that "The overall exposure level for pdns.service: 1.9 OK" and "Overall exposure level for tor.service: 7.1 MEDIUM" indicate how well managed services are.