At least the first one (the getenv thread safety fix) will hopefully make it into glibc 2.41 and it should be quite safe to backport. It turns out that setenv is easier to handle because glibc already never frees environment strings. It's concurrent unsetenv that is rather tricky. Without some snapshot approach, getenv would return null pointers instead of environment variables values that are actually set. I don't want to introduce locking into getenv because getenv without setenv has been async-signal-safe for so long that it would likely break applications.
The environ handling fixes are a bit more controversial because vfork+execve make it complicated to avoid memory leaks, but these further fixes are less important to the stability of the graphics stack.
What about using a linked list for variables added after the start of the process, which can be implemented atomically? Then once it gets "too long", a thread executing setenv could construct a new hashmap and replace the pointer to the old one with the pointer to the new one atomically, all without locking? To prevent two threads from rehashing at the same time, use an atomic flag for whether a thread is rehashing right now. That means if other threads call setenv in the meantime, the extra added variables would be appended to the list to be processed by a later setenv call (if one happens). That list could grow unbounded if enough threads call setenv quickly enough, but I think the simplicity of atomically swapping pointers might be worth it and setenv isn't called very frequently.
POSIX requires that the environment variables can be access as an array, through the environ variable. This array is expected to be used with POSIX interfaces such as posix_spawn and execve. If the array already has to exist, why not use it in getenv?
A purely hash-based implementation is not possible because there is putenv, and some applications expect modifications of environ to be visible via getenv.
As far as I'm concerned this is the only correct way to do it. I believe illumos does this, which is why its env functions are thread safe and have been for decades.
I like this approach, because it lets everything Just Work without anyone outside of the implementation having to think about it at all, and only incurs meaningful overhead if you're doing something really silly, but crucially still won't break — it'll just be slightly slow. That feels like the right trade-off.
That has been around for a long time. I remember it in the early 2000s.
> Decades old footguns - aaargh!
Indeed...
The tricky problem they have is wrapping it into some sort of lock could cause so many issues. Places that dont deadlock, suddenly could. Not a fun problem to solve. In practice it is usually not too bad as you are usually not changing your env vars much. But when you run into it, ugh.
Fortunately, for glibc, the most controversial decision (setenv/unsetenv/clearenv leak) has been made decades ago. It does not look like something that can be changed, so it's actually fixable in the glibc context. But this puts pressure on other libcs to adopt essentially the same approach (even if they don't leak environment strings today), so that's not universally popular.
Solaris/Illumos has a thread-safe setenv()/unsetenv()/putenv(). There is no reason not to have it be thread-safe, though it must leak (Solaris/Illumos retains the references to the deleted envs so as to fool memory debuggers into thinking they are still referenced and so quiet what would be essentially false positives).
Thank you! I deeply appreciate that Steam works so well on Linux these days. I don't take for granted the hard work happening behind the scenes to make that a reality for us.
I really hope they drop a full steam OS release again soon. I’d love to build a gaming tower around it. Might settle for bazzite but I’d like to do steam OS tbh
I honestly don’t want to be tinkering all the time. The steamdeck is pretty much my limit for tinkering with gaming these days. Kids do that to you lol
I imagine it is great but I just don’t want to have to troubleshoot my operating system, games, and hardware. It is probably reasonable to assume that a stable steam OS will be more consistent for me. Maybe I’m underestimating bazzite but in my experience regular Linux users (I have some experience but hardly an expert/daily user) underestimate how frequently one has to tinker with their OS’s.
I use Linux every day and I never have to tinker with it, nor did it need any tinkering to install. An extremely tinker-free experience especially compared to modern Windows.
I feel your anecdotes are at best outdated. Desktop Linux has come along way.
I get people bristle when someone says linux has a little friction, but as someone who does use Elementary and Mint from time to time I just don't get how people can say my view is "outdated" when all one has to do is pick up a console or Mac to see the difference. How many times have we seen folks troubleshooting wifi card drivers on forums? It is not some massive gap, using Linux is not some herculean feat, but surely we can agree that most people would never call it plug and play.
Linux is a great experience these days but you do have to tinker sometimes. You have to mess with drivers and settings and command line. It may be minimal for people comfortable with computers but it's not as friction-less as you're claiming.
I think you're both kind of wrong and right on this. Contemporary linux distributions really don't require tinkering anymore for most cases, yet it's also true that sometimes there is tinkering required. The reason why this is true, yet I don't blame Linux for it, is because it is hardware dependent.
If you buy hardware that is compatible with Linux, then you won't really have to tinker (at least, any more than you would with any other OS, for example, tweaking resolutions, etc). Unfortunately, it's newer hardware that typically requires the tinkering. If you don't want to tinker, I would recommend going with generation n -1 or even n -2. If you go with the latest and greatest, expect to have some tinkering required.
Distro choice does of course matter a great deal. I've been using Fedora as primary OS now for many years and absolutely love it, and it's what I recommend to most people. Ubuntu and derivatives are good of course, though the older kernels do often decrement the generation of hardware. For example, Fedora on n-1 is going to be pretty good. Ubuntu might still lack some support at that age, so should go with n-2 or n-3 to be safe.
The plan is to build a modern PC and the fact that people have to adjust their hardware decisions in such a way (downgrading/using older components) to accommodate linux kind of reiterates my point IMO. If I was installing windows this wouldn’t remotely be a consideration. Though I certainly don’t want windows, it is a notable difference.
As someone that has gamed an equal amount on Windows 10 and Linux, I think you're blowing things a little out of proportion. Windows is a tinker timesink too if you want to uninstall Candy Crush and Xbox Game Bar, disable all telemetry and ads, or even just get a good version of Java installed for Minecraft. Windows can "just work" for some games, but for others it's a nightmare to get running. Another good example is the Fallout games, which have a decades-old bug that crashes the game if you alt-tab away from it. On WINE this bug can be fixed by simply running the game fullscreen in a virtual window. The flexibility is excellent and saves me from trying to inject a DLL file just to get proper borderless fullscreen to work the way it should.
My big takeaway is this; if you are comfortable using the Steam Deck to play games and install software, you will not struggle to get Linux to run games. Pretty much anything that isn't a gaming laptop is going to have some form of support, and even the famously crappy Nvidia drivers were recently updated to support Wayland and other new Linux protocols. Now more than ever before, using Linux to game is probably easier than getting the equivalent experience on Windows.
You compared Linux to Mac a few comments back -- how is that anything but choosing specific hardware to accommodate your OS?
by this standard Mac OS is still a hobby OS because it can't be installed on random hardware.
No, it isn't too much to ask that you make sure the hardware you buy works with the OS you intend to run. If you find Linux fiddly in the modern era it's solely because of this.
You only have to avoid the newer hardware if you don't want to check for compatibility. It's just a rule of thumb to increase your odds of success because most people don't want to investigate every component. If you check for compatibility and it's supported, then you can use the newer hardware. I would have thought that was obvious, but clearly not.
It also matters how far along in the product life cycle it is. If it came out last week, it may not be supported yet. If we're nearing the refresh point then it may be supported.
> When I buy a Mac or a windows machine I don’t have to purposely avoid newer hardware to ensure it works.
But you are also comparing apples and oranges (pun incidental) and shifting the goal posts. If you buy a Mac, then you aren't building a gaming PC, which is what the rule of thumb pertains to. You're buying a complete system that has been integrated and tested. You can do the same thing with a Linux machine from various vendors (Lenovo, Dell, Framework, among others), in which case you don't have to do any investigatory work because (just like with the Mac) it's been done for you by the manufacturer.
Bazzite is atomic and image-based, so it is designed to play your games out-of-the-box without any additional configuration, and instead of package updates you are pulling the new image that's built and tested by Bazzite. From a design perspective it's extremely similar to SteamOS.
Yeah same. I didn't want to install all of those i386 library versions either. But I've found the flatpak steam client to be wonderfully easy and maintenance free, which let's me use my computer for other things, too.
The link you referenced just says that the Debian packages are irrelevant to the current codebase, as used on the Steam Deck. It doesn't say anything about the Steam Deck being the only hardware that will ever run it.
Some of the recent SteamOS release notes have included references to Asus's handheld, which has reinforced the community expectation that it will eventually be available as a distribution you can install on 3rd party hardware. If you go read interviews from Valve employees (Lawrence Yang comes to mind), I believe they've publicly stated that after the OLED shipped, they wanted to start focusing on porting to other devices.
> If you go read interviews from Valve employees (Lawrence Yang comes to mind), I believe they've publicly stated that after the OLED shipped, they wanted to start focusing on porting to other devices.
If so, it is kind of bizarre they haven't reached out to the Bazzite maintainers at all.
In general, it seems like it would save them a tonne of effort if they'd switch from a bespoke Arch-immutable spin to making a spin of Silverblue, something that has been meant to be immutable from the beginning.
They already have a system that works exactly how they want it. They already rebased from Debian to Arch to get it there. They have enough Linux staff on contract to build and maintain that system.
Maybe Bazzite is closer to their goals; maybe it's not. It's certainly not a slam dunk that the best thing they could do is throw away the thing they've been building for years to join a community project on GitHub that's trying to clone that thing.
Upstream Kde is now making an arch based immutable distro too. As steamOS is already using kde and arch, maybe once the kde distro is release, steam will rebase on that instead. Also Valve is now funding Archlinux so they are commited to arch.
But I haven't heard anything one way or the other in a while. But as it stands, they have stated that they plan to do a general release. Unless there is another source where they say they changed their mind?
ChimeraOS is a clone or fork or something of SteamOS. Works great on AMD tiny PC hardware. can't really comment past that. I found the keyboard and mouse setup kinda jarring and just threw windows back on...for now.
As noted in a sibling comment, Valve has released an open-source compositor (`gamescope`), which is what presents Steam as a console-esque UI on the Steam Deck. Using gamescope to present Steam, you can make an arbitrary Linux feel indistinguishable from a Deck.
There are many gaming distributions (e.g. Bazzite, Jovian/NixOS, Nobara, Chimera…) that take this approach.
They're usually just standard desktop distributions (Fedora or NixOS) with gaming packages configured. There is a Russian teenager who's trying to cobble together a SteamOS clone using as many Valve packages as possible. His project is called HoloISO.
No, he's talking about the compositor which has several gaming related features which then runs the Big Picture version of Steam for UI to select and manage games.
They're right that the compositor is gamescope, it's made by Valve and it has game-related features, but the person 3 comment-levels above me did seem to be conflating gamescope with the Big Picture UI. You can absolutely use gamescope and not use Big Picture mode at all, lots of people using Wayland do so by wrapping their games in a call to gamescope.
Same. I have been toying with an idea that I want to turn into a business. I know there are 3rd party attempts to replicate Valve’s stuff, but I would rather use something sanctioned by Valve.
There's a lot more to SteamOS than "it ran `apt install steam -y` for you". Even just having things like Gamescope running out of the box (which is more than just `apt install gamescope`) is a huge amount of headache set aside. Plus the guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
>guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
Flatpak fixes this problem. You don't need gamescope unless for stuff like scaling. The only few times you need to fiddle with anything on desktop is changing proton version or adding a launch variable from protondb.com (just like on steamdeck)
Gamescope does a lot more than scaling. HDR, better framerate limiting, non-rgb gamut handling, isolating "fullscreen" games, shader effect loading, and probably more I'm forgetting. Flatpak let's you run the Steam app itself correctly, nothing more. These "you only need to fiddle when" are exactly what add up and create frustration between users on different systems just wanting to play a game instead of read a manual/guide to find which "few tweaks" they each need to do to get the same experience as SteamOS would give them.
HDR will work soon work everywhere on Wayland by default.
You seem to think steam on Desktop Linux is somehow different from steam on steamdeck Linux. Like I said in my previous post, the only "fiddling" you need to do is copy pasting launch commands and choosing a different Proton version from a drop down _just_like_on_steamdeck.
Bazzite has gamescope running out of the box, and other distributions probably have it too.
It's also an immutable distribution, like SteamOS (except it's based on Fedora Silverblue instead of Arch) so there is no "incompatibility with your exact distro setup": you have the exact same distro setup as every single Bazzite user.
I like console gaming because it just works. I sit down on my couch, I turn on my controller, and I’m back in my game in under 10 seconds. The series S has honestly been a fantastic purchase for me despite the many drawbacks.
PC gaming is very enticing but we all know that’s simply not how it goes down. I would love to build a PC that is literally just discord and Steam. I want to run it in big picture mode for the most part and treat it more or less like a console.
I have really enjoyed my steam deck and it has fit that desired role pretty well. But obviously it is just not that powerful. It’s impressive for what it is, but for me it’s basically a great indie game machine with the occasional AAA option that is tolerable. A well-built machine with that kind of UX (minus the known idiosyncrasies of the deck) would be fantastic.
To answer your question more directly:
Most Linux distros do not offer this either. Bazzite is the closest I’ve seen.
Isn’t best practice to read all environment variables on boot and never use setenv? The only place where setenv would matter is for spawning new processes where you should probably be creating an new environ cloned from the current one and update the new values. Using getenv/setenv as an IPC messaging mechanism seems to be an opportunity for lots of issues aside from it historically not being multithreaded-safe on Linux and having all sorts of potential memory leaks hiding (which is what the post ignores when it says that it’s thread safe on MacOS).
FWIW the decision to leak memory on Mac actually goes back ~26 years to FreeBSD - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 which OSX inherited. I would not be surprised that Windows setenv has BSD roots due to licensing.
26 years ago people knew this API was broken but didn't fix it due to inertia of breaking buggy programs further.
There really shouldn't be a need to change your own process's envvars. For subprocesses just use the proper exec function. For anything else there should be a clear API to call rather than changing a global variable and hoping some code far away from yours rereads it and handles things correctly.
I only partially disagree with the sentiment that it is "impossible to fix". For the current API that is true, but a fairly minor modification API would make it possible. All getenv() has to do is strdup() the return value before sending it back and leaving it on the programmer to free the memory when they are done with it. This does mean that the programmer will need to call getenv() again if they think the value might change, but I think that is a reasonable tradeoff.
This change would make old programs leak memory every time they call getenv() without the subsequent free(), but since the current version also leaks memory that doesn't seem like a dealbreaker. As an added bonus the new version could be made thread safe by wrapping the strdup() in a mutex and doing similar work on the setenv() side.
There exists a ton of working code that does not fiddle with setenv and now would have memory leaks if they don't change their code? Plus I would now need to test if my stdlib requires freeing memory or not because if I try to free on an older libc it is not going to work out well. I don't think the value works out.
From a hygiene perspective - freeing the return of another API is an anti pattern. If you need the caller to release objects there should provide a FooLib_bar_destory(bar) or similar.
Yes, it would cause existing programs to leak memory. Most of the time these leaks would be fairly minor, but some programs could leak a lot if they call getenv() inside of a loop for some reason.
Personally, if the return object is a basic C type, I'm not a fan of creating a wrapper function to call free(). This is one of those code purity things that I don't think buys you anything.
The real question is: is there any case where a program calls `setenv` in one thread and actually wants it to take effect in other already-existing threads?
That said, GLIBC is pretty good at documenting all the dangerous functions, so it is possible to add locking/copying yourself.
Interesting idea. I strongly suspect that there are programs out there that expect that setenv changes the environ array (and they do not treat it as an opaque pointer passed to posix_spawn/execve). With a per-thread setenv, we would need a per-thread environ variable as well. Unfortunately, that's not really compatible with POSIX because environ is not declared in a header. Instead, programmers are expected to write a declaration
extern char **environ;
into their sources, and that declaration is incompatible with environ being a thread-local variable.
Hm, in the end most of the problems do come down to stuff not coming from blessed headers.
Regardless of anything else, how about:
* deprecate direct access to `environ` and add functions to replace it. Have a macro that indicates this and provide a canonical compatibility shim for people to copy if they might use old libcs.
* using linker magic, change the behavior of programs depending on whether they attempt to access `environ` or not, so old-API programs are still thread-unsafe but new ones are thread-safe.
It's amazing how much you can do with the conditionally-linked object files from a static "library". Much of C's cross-TU "UB, no diagnostic required" is inexcusable since we can detect it quite easily with zero overhead (at least, for static linking) using today's linkers by deliberately causing multiple definition errors.
Compatibility with old-ABI programs probably means fixing `environ` is not that simple, but you are libc and libc is in control of dynamic linking ...
Many years ago, glibc did something along those lines for the _res variable (with preprocessor magic instead of linker magic). For the main thread, legacy _res (the actual global data symbol) and new thread-local _res (actually *__res_state()) are the same object, but they diverge for subsequently created threads.
I don't think this would work here because it likely changes semantics too much, and not all binaries that need a thread-safe getenv/setenv combination can be rebuilt, especially since compatibility with both variants from the binaries would likely some changes to each application/library.
All the inconsistencies you suggest sound like a trap, especially when you suggest ABI and API behaviour divergences.
From what I understood of your macro idea this would lead to API changes that would lead to ifdefs for different (g)libcurl versions. Doesn’t feel good especially for software distributors.
Also the changes you mention require changes in the linker scripts distributed by a variety of toolchains and would need to check if the libc target was a blessed one. In effect the deprecation would never move to obsolete and kept around forever.
To clarify the above post libc is in control of dynamic linking through the dl*(dlopen) family of functions.
Yes, I set up environment variables in a plugin that are later read by already started worker threads. It's not a problem for me because the worker threads are all sleeping on a runqueue, but technically I do want to set an env var in one thread and read it in another that is already running.
There are so many better ways to do IPC that this hacky and dangerous getenv/setenv setup is never necessary.
I mean what kind of threading library doesn't have shared memory or message passing?
I'm guessing this mostly happens in situations where the main process can change variables like HTTPS_PROXY and a different thread is running a library that checks those variables before firing up a TCP socket.
Yes it is, if you program application software for Linux (and your other two supported platforms are Windows and macOS, as is the case for OP and their colleague who appear to work for Valve, supporting Steam)
And the reason the colleague dubs it the worst has nothing to do with its specification in POSIX (which doesn't require the function to be thread-safe but also doesn't prevent it from being made thread-safe), but rather its specific implementation in glibc, which is the C library in use on all the Linux distributions that Valve support Steam on and all are equally dubbed as "Linux"
I think you get thread-safety and memory leaks (the article alludes to macOS taking this approach), or it's unsafe but doesn't leak. But that is inherent in the API, as specified by POSIX: setenv must necessarily either invalidate the pointer returned by getenv (which is cannot do safely) or leak it.
Perhaps the leak is "better", in at least there won't be non-OOM crashes, but it still leaves a bad taste in one's mouth.
For a long running program like Steam (that is for some odd reason calling setenv…?) … I'm not sure which is better. Better would be not calling setenv, which it sounds like they've worked on.
I would be happy if getenv() called strdup() on the value and returned that. It would cause loads of minor memory leaks and maybe a handful of serious ones, but they should be reasonably easy to clean up and would avoid returning pointers to mutable private data.
There are a lot of situations where allocating is not acceptable (e.g.: critical applications that don't heap-allocate in order to ensure they never OOM).
Of course, we could have just getenvdup() which does what you just said. Let each application chose which variant is best for them: leaking, or thread-unsafe.
It'd be an API change, sort of? Although POSIX doesn't seem to say anything about ownership of the pointer, beyond thou-shalt-not modify the data it points to. Doesn't seem to specify calling free as valid or invalid.
But honestly, it seems like one might as well do exactly that (strdup & return)? At worst, nothing calls free(), and it is equivalent to the macOS strategy of "just leak the memory" to make it threadsafe. But at best, a program could #ifdef its way into "oh, this semi-sorta-nonstandard-behavior on this particular OS" and call free(), getting both a thread-safe & non-leaking implementation.
The mere existence of Steam is astounding to someone who grew up playing nethack and chess on Linux.
But the Steam client is really strange. Sometimes it works for months, and suddenly a game won't start, or something doesn't work, and I have to do weird stuff to get it working like purging all files or reinstalling. It doesn't make sense, it's like the Steam client rots.
Steam is one of the worst software, it's frankly embarassing for such a huge company. It's a hotchpotch of different design principles and new features only get tacked on with a very different UI + UX. The client is mostly webviews but they are so slow and buggy and they use a very old insecure Chromium version under the hood. The launcher is still a 32 bit program. It's sad that other launchers are even worse though.
It's four versions behind the current one with a lot of CVEs. There seems to be an LTS 126 but it's only for ChromeOS - it's at least behind this one which fixed one critical and two High CVEs: https://chromereleases.googleblog.com/2024/11/long-term-supp...
Even being only one version behind is dangerous for such a huge attack vector.
The interesting thing is that Valve is not a "huge company". They're privately held and only had 336 employees the last time headcount was reported (2021).
In my experience, all of those things are also true of Steam on Windows. In fact, if anything I have to deal with it less frequently since switching exclusively to Linux.
This is really cool insight into both the Steam client and Linux programming. I understand why there may not be detailed release notes every release, but wow "Fixed some miscellaneous common crashes" is an understatement when you know about this work!
> If this can be addressed in glibc, it may involve a tradeoff on features, maybe an opt-in mechanism with a slight departure from the "impossible" POSIX spec. That's something we may pursue in the long term if we can propose something sensible.
I'm really curious why they're using setenv(3) so much. The main usages that I can think of is setting an environment variable before calling something like exec(3). That doesn't seem to be the case here.
The article mentions that they use exevpe for spawning children processes. So what usages of setenv(3) would remain?
It is possible to check for setenv/unsetenv/putenv with nm -D, and a quick sample of my ~/.cargo/bin/* shows far too many programs using those. Yeah they could be single threaded, but who can guarantee they will remain so? Come to think of it listing symbols could detect pthread_create as well.
I'd be interested in a way to do static binary analysis to get from those symbols to a call tree, as well.
I don't see a way to check for **environ usage though, the compiler could turn this one into anything.
> We removed the majority of setenv calls. It was mostly used when spawning processes
Could someone elaborate this for a non-developer? Why would you use `setenv` (which I assume is functionally similar to `export key=value`, but correctly me if I'm wrong) (extensively) for spawning processes?
Environment variables are per-process, but inherited by child processes (not system wide). So when you start other programs you often set up env vars for them, or clear env vars that were used by your parent process to not inadvertantly perturb the child's behaviour.
Unless there is a fork exec family of syscalls are not child processes but the same process itself. Small detail but important because sometimes you really want a separate process altogether.
in that case you do not need setenv to pass values to the underlying process though. And if you are forking without immediately execv in a multithreaded program, setenv is the last of your problems.
edit: what's probably happening is that execve is four or five abstraction layers deeper (possibly in a third party dependency) than where the env variable need to be set without a clean way to pass the values through.
The likely scenarios I was thinking (setenv vs execve) would both be called in the child process after fork. But of course here we get to the fascinating world of fork interactions with threads, and you could even have one thread fork()ing while another thread is doing a setenv().
Regardless of whether it's a hack, my broader question is: is it common in Linux to manipulate environment variables on the fly for a process or program?
I primarily use Windows, both as an end-user and an amateur programmer. From my experience, most programs on Windows don’t do this. If parameters are needed, they’re usually passed as arguments, while environment variables are used for more permanent settings, like %PATH%.
PATH is a good example why steam does this: Steam is a program to launch a wide variety of other programs, which it doesn't fully control. Those programs (games) may do anything including launching further programs (utilities), so steam may set PATH so that the game finds those utilities.
It can also be a way to pass license information or other configuration settings.
Looks like the Steam team moved to control spawning and do execvpe.
I would like to see at least in-process environment modification discouraged. Rust is dealing with the issue by considering getenv unsafe when coming through C, but getting rid of the read side is much harder than the write side.
It is decently common enough in unix. You are correct with my windows stuff it is fairly rare to set env vars to change the launching process. In the unix world though I have seen the pattern a decent number of times. With windows programs usually you see the pattern (not always) if it is ported from a unix system. Windows likes its ini/registry/cli items to do configuration. It doesnt mean the pattern can not be used in windows, I personally just have not seen it as much with native win32 apps. If you fire up something like 'git bash' you can see entire bash functions bound to env variables.
I think it goes back to where windows came from. That environment space in DOS was not exactly huge (256 bytes at one point?). In unix it seems like it was much larger and expressive.
One thing to note about command line arguments on Linux is, any user can typically inspect `/proc/{pid}/cmdline` and get the full command line used to start the process. So if you pass secrets like API keys, passwords, etc, via an argument, they're visible to the rest of the system. However, if you put secrets into the environment of the child process, only the user that owns the child process can inspect `/proc/{pid}/environ`.
Java in general tends to massively dislike environment variables.
Which is a Good Thing, but unfortunately it makes dealing with proxy support so much harder - especially as there are just so damn many HTTP libraries that people use...
execve(2) and friends allow you to specify a new processes's environment variables. How Java may choose to expose that I don't know. Assuming they do, that is all a shell needs. You do not strictly need to call setenv(3) to change a variable that gets evaluated with $, etc. You do need to pass the modified environment to a child process.
1) java shell starts
2) read getenv into a java hashmap
3) evaluate shell input - any sets update the hashmap, any gets read from the hashmap
4) when you get to a point where you need to exec a process you use the hashmap to pass envvars into process builder - which is going to call execvpe for you.
So yeah you definitely could (there'd be other reasons why java wouldn't be a good choice).
TBH reading that folks were calling setenv before an exec to propagate env vars to a child made me sad. I would guess that other use cases were leveraging environ as a poor man's global variable - which is also unfortunate.
I love the steam client on linux these days, especially the compatibility for non-steam games is so great and Ive been using it to play WoW Classic while I have covid
To raise awareness: there's been a bug with the Linux Steam client which has been persistent for a long time.
TL;DR: if you have Steam running for more than a ~day or so, you will run out of window handles so you won't be able to open any new graphical application/window until you restart Steam.
Using Steam Chat appears to make the issue worse (it happens earlier).
I kinda wish HN allowed a visual indicator for subthreads that focus on tangents that don't hold universal appeal: comment grammar, humor, "why did I get downvoted?", etc.
There are lots of little discussions like this that I'd love to have, but which sometimes lead to a lot of downvotes.
I've got several Linux gaming machines (and a couple of business desktops I play FTL on sometimes), and it's weirdly inconsistent - the best and worst machine I have both use Intel Graphics. It almostnever shows up on the A770 - that one stays awake for weeks of daily gaming and I have no problems with it. My laptop with integrated Intel sees it all the time. My laptop with Nvidia graphics sees it often, but not nearly as often.
The one that's so annoying that I've actually developed habits around it is the tendency for the library window to freeze when it becomes unfocused - it's not all the time, but it's often enough that I now habitually close the window anytime I defocus it
Interesting. I've left the Steam client running for weeks at a time and have not seen that issue. I bet it was closed because Valve couldn't replicate the problem.
If you actually look at the issue you can see it was closed because Valve shipped a fix and people confirmed it was fixed. Then later some other people reply that they still have problems but without any useful information.
This has made Steam something to avoid for me. I tried switching to Flatpak version to see if that would help, but unfortunately not. For me it started ~1 year ago.
The performance issues have made me not want to even open Steam anymore so I have nearly completely stopped playing. (not related to this issue but several games have also added kernel anticheats, killing off Linux versions, which also has contributed to this)
Always cool to rediscover people via HN. This post reminded me of the work ttimo did for the Quake 3 engine more than two decades ago. I remember it because I read so many comments written by him (like 15 years ago):
Nvidia in my case, and I agree, the app performance is very poor. The easiest way to see the lag, IMO, is to open up the achievements list for a game and try to scroll around. It takes almost a second for mouse scroll wheel movements to have an effect.
I wish they'd make it more virtualization friendly. I don't want to run untrustworthy proprietary software on my main system. Common sandboxing mechanisms are insufficient since Steam and its games need access to the entire device tree anyway. Nothing short of a real virtual machine would do it for me. Will also make compatibility painless since I can just install the Linux distribution they support.
I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... Only to discover they are exceendingly hostile to users who do this VFIO stuff.
Just yet another reminder not to "buy" games on these platforms, I guess.
> I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance...
So just any standard/decent motherboard bought within the last 3-4 years?
> I wish they'd make it more virtualization friendly.
Which games do you have issues with when virtualizing? I've only been locked out of Halo Infinite and I enable all hyperv flags in libvirt.
> Just yet another reminder not to "buy" games on these platforms, I guess.
If you buy a game on Steam and you're unable to play it they'll refund you straight away, which is why everyone loves Valve and Steam even when the client is crappy.
I couldn't be happier with my setup, which I assume is similar to yours. (2 GPUs)
> So just any standard/decent motherboard bought within the last 3-4 years?
It isn't at all obvious that IOMMU is supported by even current top of the line motherboards. I looked at a lot of products and have yet to see the VT-d and AMD-Vi keywords mentioned in technical specifications. To confirm support I had to read their UEFI firmware manuals and look for instructions on toggling the virtualization support.
At least ECC memory support has started to show up in technical specifications. Looks like IOMMU is not quite there yet.
> Which games do you have issues with when virtualizing?
From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives. I'm currently assuming any game with battleye or easy anti cheat will issue permanent bans on virtualization detection.
> If you buy a game on Steam and you're unable to play it they'll refund you straight away
I guess I'm just lucky, because I've had one MSI and two ASUS boards with this setup without issues, I guess it isn't listed everywhere, my strategy of just buying boards hoping it works seems to be the way!
> From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives
There's a lot of misinformation on the internet, I haven't heard about any cheats used in the wild using a VM and haven't had issues.
>> If you buy a game on Steam and you're unable to play it they'll refund you straight away
> Somehow that doesn't bring me much peace of mind.
If zero risk purchases doesn't bring you peace of mind I recommend dual booting or running two rigs instead of using bleeding edge technology
On Windows, "Environment" is stored in the Win32 Thread Information Block/Thread Environment Block (TIB/TEB), so it's thread-local rather than process-global.
My only issue with Steam at this point is that it’ll just randomly complain it has no connection, no matter which content server I set it to connect to.
If I spam the ‘retry’ button it’ll eventually work, but it’s a massive PITA.
We've got patches under review: https://inbox.sourceware.org/libc-alpha/cover.1722193092.git... (triggered by https://issues.redhat.com/browse/RHEL-42410, a graphics stack stability issue that wasn't as visible in RHEL 9 for some reason)
At least the first one (the getenv thread safety fix) will hopefully make it into glibc 2.41 and it should be quite safe to backport. It turns out that setenv is easier to handle because glibc already never frees environment strings. It's concurrent unsetenv that is rather tricky. Without some snapshot approach, getenv would return null pointers instead of environment variables values that are actually set. I don't want to introduce locking into getenv because getenv without setenv has been async-signal-safe for so long that it would likely break applications.
The environ handling fixes are a bit more controversial because vfork+execve make it complicated to avoid memory leaks, but these further fixes are less important to the stability of the graphics stack.
What about using a linked list for variables added after the start of the process, which can be implemented atomically? Then once it gets "too long", a thread executing setenv could construct a new hashmap and replace the pointer to the old one with the pointer to the new one atomically, all without locking? To prevent two threads from rehashing at the same time, use an atomic flag for whether a thread is rehashing right now. That means if other threads call setenv in the meantime, the extra added variables would be appended to the list to be processed by a later setenv call (if one happens). That list could grow unbounded if enough threads call setenv quickly enough, but I think the simplicity of atomically swapping pointers might be worth it and setenv isn't called very frequently.
POSIX requires that the environment variables can be access as an array, through the environ variable. This array is expected to be used with POSIX interfaces such as posix_spawn and execve. If the array already has to exist, why not use it in getenv?
A purely hash-based implementation is not possible because there is putenv, and some applications expect modifications of environ to be visible via getenv.
As far as I'm concerned this is the only correct way to do it. I believe illumos does this, which is why its env functions are thread safe and have been for decades.
I like this approach, because it lets everything Just Work without anyone outside of the implementation having to think about it at all, and only incurs meaningful overhead if you're doing something really silly, but crucially still won't break — it'll just be slightly slow. That feels like the right trade-off.
Perhaps related: "Setenv Is Not Thread Safe and C Doesn't Want to Fix It" (2023) -- https://www.evanjones.ca/setenv-is-not-thread-safe.html and HN comments regarding how rust was affected: https://news.ycombinator.com/item?id=38342642
Decades old footguns - aaargh!
That has been around for a long time. I remember it in the early 2000s.
> Decades old footguns - aaargh!
Indeed...
The tricky problem they have is wrapping it into some sort of lock could cause so many issues. Places that dont deadlock, suddenly could. Not a fun problem to solve. In practice it is usually not too bad as you are usually not changing your env vars much. But when you run into it, ugh.
Fortunately, for glibc, the most controversial decision (setenv/unsetenv/clearenv leak) has been made decades ago. It does not look like something that can be changed, so it's actually fixable in the glibc context. But this puts pressure on other libcs to adopt essentially the same approach (even if they don't leak environment strings today), so that's not universally popular.
Solaris/Illumos has a thread-safe setenv()/unsetenv()/putenv(). There is no reason not to have it be thread-safe, though it must leak (Solaris/Illumos retains the references to the deleted envs so as to fool memory debuggers into thinking they are still referenced and so quiet what would be essentially false positives).
Thank you! I deeply appreciate that Steam works so well on Linux these days. I don't take for granted the hard work happening behind the scenes to make that a reality for us.
I really hope they drop a full steam OS release again soon. I’d love to build a gaming tower around it. Might settle for bazzite but I’d like to do steam OS tbh
You can run gamescope as your WM and have a steam-deck-like experience on your desktop, ideal for a living room.
https://wiki.nixos.org/wiki/Steam#Gamescope_Compositor_/_%22...
I honestly don’t want to be tinkering all the time. The steamdeck is pretty much my limit for tinkering with gaming these days. Kids do that to you lol
Bazzite is the answer. It’s basically out of the box SteamOS, just somehow even better.
I imagine it is great but I just don’t want to have to troubleshoot my operating system, games, and hardware. It is probably reasonable to assume that a stable steam OS will be more consistent for me. Maybe I’m underestimating bazzite but in my experience regular Linux users (I have some experience but hardly an expert/daily user) underestimate how frequently one has to tinker with their OS’s.
I use Linux every day and I never have to tinker with it, nor did it need any tinkering to install. An extremely tinker-free experience especially compared to modern Windows.
I feel your anecdotes are at best outdated. Desktop Linux has come along way.
I get people bristle when someone says linux has a little friction, but as someone who does use Elementary and Mint from time to time I just don't get how people can say my view is "outdated" when all one has to do is pick up a console or Mac to see the difference. How many times have we seen folks troubleshooting wifi card drivers on forums? It is not some massive gap, using Linux is not some herculean feat, but surely we can agree that most people would never call it plug and play.
Linux is a great experience these days but you do have to tinker sometimes. You have to mess with drivers and settings and command line. It may be minimal for people comfortable with computers but it's not as friction-less as you're claiming.
I think you're both kind of wrong and right on this. Contemporary linux distributions really don't require tinkering anymore for most cases, yet it's also true that sometimes there is tinkering required. The reason why this is true, yet I don't blame Linux for it, is because it is hardware dependent.
If you buy hardware that is compatible with Linux, then you won't really have to tinker (at least, any more than you would with any other OS, for example, tweaking resolutions, etc). Unfortunately, it's newer hardware that typically requires the tinkering. If you don't want to tinker, I would recommend going with generation n -1 or even n -2. If you go with the latest and greatest, expect to have some tinkering required.
Distro choice does of course matter a great deal. I've been using Fedora as primary OS now for many years and absolutely love it, and it's what I recommend to most people. Ubuntu and derivatives are good of course, though the older kernels do often decrement the generation of hardware. For example, Fedora on n-1 is going to be pretty good. Ubuntu might still lack some support at that age, so should go with n-2 or n-3 to be safe.
The plan is to build a modern PC and the fact that people have to adjust their hardware decisions in such a way (downgrading/using older components) to accommodate linux kind of reiterates my point IMO. If I was installing windows this wouldn’t remotely be a consideration. Though I certainly don’t want windows, it is a notable difference.
As someone that has gamed an equal amount on Windows 10 and Linux, I think you're blowing things a little out of proportion. Windows is a tinker timesink too if you want to uninstall Candy Crush and Xbox Game Bar, disable all telemetry and ads, or even just get a good version of Java installed for Minecraft. Windows can "just work" for some games, but for others it's a nightmare to get running. Another good example is the Fallout games, which have a decades-old bug that crashes the game if you alt-tab away from it. On WINE this bug can be fixed by simply running the game fullscreen in a virtual window. The flexibility is excellent and saves me from trying to inject a DLL file just to get proper borderless fullscreen to work the way it should.
My big takeaway is this; if you are comfortable using the Steam Deck to play games and install software, you will not struggle to get Linux to run games. Pretty much anything that isn't a gaming laptop is going to have some form of support, and even the famously crappy Nvidia drivers were recently updated to support Wayland and other new Linux protocols. Now more than ever before, using Linux to game is probably easier than getting the equivalent experience on Windows.
You compared Linux to Mac a few comments back -- how is that anything but choosing specific hardware to accommodate your OS?
by this standard Mac OS is still a hobby OS because it can't be installed on random hardware.
No, it isn't too much to ask that you make sure the hardware you buy works with the OS you intend to run. If you find Linux fiddly in the modern era it's solely because of this.
I addressed this in my previous comment figuring this comment was coming:
> in such a way (downgrading/using older components) to accommodate linux
When I buy a Mac or a windows machine I don’t have to purposely avoid newer hardware to ensure it works.
You only have to avoid the newer hardware if you don't want to check for compatibility. It's just a rule of thumb to increase your odds of success because most people don't want to investigate every component. If you check for compatibility and it's supported, then you can use the newer hardware. I would have thought that was obvious, but clearly not.
It also matters how far along in the product life cycle it is. If it came out last week, it may not be supported yet. If we're nearing the refresh point then it may be supported.
> When I buy a Mac or a windows machine I don’t have to purposely avoid newer hardware to ensure it works.
But you are also comparing apples and oranges (pun incidental) and shifting the goal posts. If you buy a Mac, then you aren't building a gaming PC, which is what the rule of thumb pertains to. You're buying a complete system that has been integrated and tested. You can do the same thing with a Linux machine from various vendors (Lenovo, Dell, Framework, among others), in which case you don't have to do any investigatory work because (just like with the Mac) it's been done for you by the manufacturer.
You brought MacOS into this conversation not me. I’m not sure what the deal is here.
Bazzite is atomic and image-based, so it is designed to play your games out-of-the-box without any additional configuration, and instead of package updates you are pulling the new image that's built and tested by Bazzite. From a design perspective it's extremely similar to SteamOS.
So it works until it doesn't (because your hardware is not their test hardware) then there's no way to debug or fix things.
Also https://github.com/Jovian-Experiments/Jovian-NixOS
GGP literally said he doesn't want to be tinkering all the time. Nix is literally the opposite of no tinkering.
Yeah same. I didn't want to install all of those i386 library versions either. But I've found the flatpak steam client to be wonderfully easy and maintenance free, which let's me use my computer for other things, too.
https://flathub.org/apps/com.valvesoftware.Steam
I thought Valve already said they don't plan to do that. Steam OS 3.0 is only for the Steam Deck isn't it?
From https://repo.steampowered.com/steamos/README.txt
SteamOS version 1 'alchemist' and version 2 'brewmaster' have been discontinued. No further updates are planned.
The SteamOS 'clockwerk' prototype has also been discontinued and will not be released.
The link you referenced just says that the Debian packages are irrelevant to the current codebase, as used on the Steam Deck. It doesn't say anything about the Steam Deck being the only hardware that will ever run it.
Some of the recent SteamOS release notes have included references to Asus's handheld, which has reinforced the community expectation that it will eventually be available as a distribution you can install on 3rd party hardware. If you go read interviews from Valve employees (Lawrence Yang comes to mind), I believe they've publicly stated that after the OLED shipped, they wanted to start focusing on porting to other devices.
> If you go read interviews from Valve employees (Lawrence Yang comes to mind), I believe they've publicly stated that after the OLED shipped, they wanted to start focusing on porting to other devices.
If so, it is kind of bizarre they haven't reached out to the Bazzite maintainers at all.
In general, it seems like it would save them a tonne of effort if they'd switch from a bespoke Arch-immutable spin to making a spin of Silverblue, something that has been meant to be immutable from the beginning.
Is it?
They already have a system that works exactly how they want it. They already rebased from Debian to Arch to get it there. They have enough Linux staff on contract to build and maintain that system.
Maybe Bazzite is closer to their goals; maybe it's not. It's certainly not a slam dunk that the best thing they could do is throw away the thing they've been building for years to join a community project on GitHub that's trying to clone that thing.
Upstream Kde is now making an arch based immutable distro too. As steamOS is already using kde and arch, maybe once the kde distro is release, steam will rebase on that instead. Also Valve is now funding Archlinux so they are commited to arch.
The last I heard, they said they were planning to do it, for example, this article: https://9to5linux.com/valve-says-steamos-3-0-will-be-availab...
But I haven't heard anything one way or the other in a while. But as it stands, they have stated that they plan to do a general release. Unless there is another source where they say they changed their mind?
Last I heard they were still working on a general desktop release but it's slow going, largely due to Nvidia support.
Or not supporting them rather? Do you have a link somewhere they talk about it?
They’ve been pretty quiet about it yeah but last I saw there was some plan to do it some day
For what it’s worth, I have had pretty good luck with Jovian for NixOS. My primary game console is a little gaming PC running it, I like it.
ChimeraOS is a clone or fork or something of SteamOS. Works great on AMD tiny PC hardware. can't really comment past that. I found the keyboard and mouse setup kinda jarring and just threw windows back on...for now.
As noted in a sibling comment, Valve has released an open-source compositor (`gamescope`), which is what presents Steam as a console-esque UI on the Steam Deck. Using gamescope to present Steam, you can make an arbitrary Linux feel indistinguishable from a Deck.
There are many gaming distributions (e.g. Bazzite, Jovian/NixOS, Nobara, Chimera…) that take this approach.
They're usually just standard desktop distributions (Fedora or NixOS) with gaming packages configured. There is a Russian teenager who's trying to cobble together a SteamOS clone using as many Valve packages as possible. His project is called HoloISO.
You're talking about Big Picture, which has been a thing for.. 8 years? Maybe 10?
The compositor is what runs two layers down, under the window manager.
No, he's talking about the compositor which has several gaming related features which then runs the Big Picture version of Steam for UI to select and manage games.
They're right that the compositor is gamescope, it's made by Valve and it has game-related features, but the person 3 comment-levels above me did seem to be conflating gamescope with the Big Picture UI. You can absolutely use gamescope and not use Big Picture mode at all, lots of people using Wayland do so by wrapping their games in a call to gamescope.
Same. I have been toying with an idea that I want to turn into a business. I know there are 3rd party attempts to replicate Valve’s stuff, but I would rather use something sanctioned by Valve.
What do you want from a generic SteamOS that Bazzite doesn't have?
What would that get you over any other distro with the steam software installed?
There's a lot more to SteamOS than "it ran `apt install steam -y` for you". Even just having things like Gamescope running out of the box (which is more than just `apt install gamescope`) is a huge amount of headache set aside. Plus the guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
>guarantee when it doesn't work it's because of a bug rather than something you did or some incompatibility with your exact distro setup.
Flatpak fixes this problem. You don't need gamescope unless for stuff like scaling. The only few times you need to fiddle with anything on desktop is changing proton version or adding a launch variable from protondb.com (just like on steamdeck)
Gamescope does a lot more than scaling. HDR, better framerate limiting, non-rgb gamut handling, isolating "fullscreen" games, shader effect loading, and probably more I'm forgetting. Flatpak let's you run the Steam app itself correctly, nothing more. These "you only need to fiddle when" are exactly what add up and create frustration between users on different systems just wanting to play a game instead of read a manual/guide to find which "few tweaks" they each need to do to get the same experience as SteamOS would give them.
HDR will work soon work everywhere on Wayland by default.
You seem to think steam on Desktop Linux is somehow different from steam on steamdeck Linux. Like I said in my previous post, the only "fiddling" you need to do is copy pasting launch commands and choosing a different Proton version from a drop down _just_like_on_steamdeck.
Bazzite has gamescope running out of the box, and other distributions probably have it too.
It's also an immutable distribution, like SteamOS (except it's based on Fedora Silverblue instead of Arch) so there is no "incompatibility with your exact distro setup": you have the exact same distro setup as every single Bazzite user.
Yeah, Bazzite is probably the best alternative barring official support. It attempts to tackle most of the issues head on.
Yeah you get it lol I just want to play my damn games and not tinker all the damn time!
I like console gaming because it just works. I sit down on my couch, I turn on my controller, and I’m back in my game in under 10 seconds. The series S has honestly been a fantastic purchase for me despite the many drawbacks.
PC gaming is very enticing but we all know that’s simply not how it goes down. I would love to build a PC that is literally just discord and Steam. I want to run it in big picture mode for the most part and treat it more or less like a console.
I have really enjoyed my steam deck and it has fit that desired role pretty well. But obviously it is just not that powerful. It’s impressive for what it is, but for me it’s basically a great indie game machine with the occasional AAA option that is tolerable. A well-built machine with that kind of UX (minus the known idiosyncrasies of the deck) would be fantastic.
To answer your question more directly: Most Linux distros do not offer this either. Bazzite is the closest I’ve seen.
1st party support
Isn’t best practice to read all environment variables on boot and never use setenv? The only place where setenv would matter is for spawning new processes where you should probably be creating an new environ cloned from the current one and update the new values. Using getenv/setenv as an IPC messaging mechanism seems to be an opportunity for lots of issues aside from it historically not being multithreaded-safe on Linux and having all sorts of potential memory leaks hiding (which is what the post ignores when it says that it’s thread safe on MacOS).
No. Because you might end up using a library that calls getenv().
Java does what you say, but it still presents problems for JNI code using libraries that want to getenv().
FWIW the decision to leak memory on Mac actually goes back ~26 years to FreeBSD - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5604 which OSX inherited. I would not be surprised that Windows setenv has BSD roots due to licensing.
26 years ago people knew this API was broken but didn't fix it due to inertia of breaking buggy programs further.
There really shouldn't be a need to change your own process's envvars. For subprocesses just use the proper exec function. For anything else there should be a clear API to call rather than changing a global variable and hoping some code far away from yours rereads it and handles things correctly.
I only partially disagree with the sentiment that it is "impossible to fix". For the current API that is true, but a fairly minor modification API would make it possible. All getenv() has to do is strdup() the return value before sending it back and leaving it on the programmer to free the memory when they are done with it. This does mean that the programmer will need to call getenv() again if they think the value might change, but I think that is a reasonable tradeoff.
This change would make old programs leak memory every time they call getenv() without the subsequent free(), but since the current version also leaks memory that doesn't seem like a dealbreaker. As an added bonus the new version could be made thread safe by wrapping the strdup() in a mutex and doing similar work on the setenv() side.
There exists a ton of working code that does not fiddle with setenv and now would have memory leaks if they don't change their code? Plus I would now need to test if my stdlib requires freeing memory or not because if I try to free on an older libc it is not going to work out well. I don't think the value works out.
From a hygiene perspective - freeing the return of another API is an anti pattern. If you need the caller to release objects there should provide a FooLib_bar_destory(bar) or similar.
Yes, it would cause existing programs to leak memory. Most of the time these leaks would be fairly minor, but some programs could leak a lot if they call getenv() inside of a loop for some reason.
Personally, if the return object is a basic C type, I'm not a fan of creating a wrapper function to call free(). This is one of those code purity things that I don't think buys you anything.
The real question is: is there any case where a program calls `setenv` in one thread and actually wants it to take effect in other already-existing threads?
That said, GLIBC is pretty good at documenting all the dangerous functions, so it is possible to add locking/copying yourself.
Interesting idea. I strongly suspect that there are programs out there that expect that setenv changes the environ array (and they do not treat it as an opaque pointer passed to posix_spawn/execve). With a per-thread setenv, we would need a per-thread environ variable as well. Unfortunately, that's not really compatible with POSIX because environ is not declared in a header. Instead, programmers are expected to write a declaration
into their sources, and that declaration is incompatible with environ being a thread-local variable.Hm, in the end most of the problems do come down to stuff not coming from blessed headers.
Regardless of anything else, how about:
* deprecate direct access to `environ` and add functions to replace it. Have a macro that indicates this and provide a canonical compatibility shim for people to copy if they might use old libcs.
* using linker magic, change the behavior of programs depending on whether they attempt to access `environ` or not, so old-API programs are still thread-unsafe but new ones are thread-safe.
It's amazing how much you can do with the conditionally-linked object files from a static "library". Much of C's cross-TU "UB, no diagnostic required" is inexcusable since we can detect it quite easily with zero overhead (at least, for static linking) using today's linkers by deliberately causing multiple definition errors.
Compatibility with old-ABI programs probably means fixing `environ` is not that simple, but you are libc and libc is in control of dynamic linking ...
Many years ago, glibc did something along those lines for the _res variable (with preprocessor magic instead of linker magic). For the main thread, legacy _res (the actual global data symbol) and new thread-local _res (actually *__res_state()) are the same object, but they diverge for subsequently created threads.
I don't think this would work here because it likely changes semantics too much, and not all binaries that need a thread-safe getenv/setenv combination can be rebuilt, especially since compatibility with both variants from the binaries would likely some changes to each application/library.
All the inconsistencies you suggest sound like a trap, especially when you suggest ABI and API behaviour divergences. From what I understood of your macro idea this would lead to API changes that would lead to ifdefs for different (g)libcurl versions. Doesn’t feel good especially for software distributors.
Also the changes you mention require changes in the linker scripts distributed by a variety of toolchains and would need to check if the libc target was a blessed one. In effect the deprecation would never move to obsolete and kept around forever.
To clarify the above post libc is in control of dynamic linking through the dl*(dlopen) family of functions.
> so it is possible to add locking/copying yourself.
Not if third-party dependent libraries use getenv/setenv. (The article mentions this as a continuing problem with the steam client.)
As a rule you should not assume third-party libraries are at all thread-safe.
Yes, I set up environment variables in a plugin that are later read by already started worker threads. It's not a problem for me because the worker threads are all sleeping on a runqueue, but technically I do want to set an env var in one thread and read it in another that is already running.
Glibc could crib from Illumos, which has a thread-safe putenv()/setenv()/unsetenv()/getenv().
There are so many better ways to do IPC that this hacky and dangerous getenv/setenv setup is never necessary.
I mean what kind of threading library doesn't have shared memory or message passing?
I'm guessing this mostly happens in situations where the main process can change variables like HTTPS_PROXY and a different thread is running a library that checks those variables before firing up a TCP socket.
> One of my colleagues rightly dubbed setenv "the worst Linux API".
Is setenv really a Linux API, since it's neither defined by Linux (it's in POSIX) nor implemented by the Linux kernel (it's entirely in userspace)?
Yes it is, if you program application software for Linux (and your other two supported platforms are Windows and macOS, as is the case for OP and their colleague who appear to work for Valve, supporting Steam)
And the reason the colleague dubs it the worst has nothing to do with its specification in POSIX (which doesn't require the function to be thread-safe but also doesn't prevent it from being made thread-safe), but rather its specific implementation in glibc, which is the C library in use on all the Linux distributions that Valve support Steam on and all are equally dubbed as "Linux"
Technically, this is a GNU libc limitation. A libc _could_ make this thread-safe.
Languages like go just implement this feature internally and don't have this limitation.
The limitation is contemplated in POSIX itself; from the posix documentation:
> The setenv() function need not be thread-safe.
I think you get thread-safety and memory leaks (the article alludes to macOS taking this approach), or it's unsafe but doesn't leak. But that is inherent in the API, as specified by POSIX: setenv must necessarily either invalidate the pointer returned by getenv (which is cannot do safely) or leak it.
Perhaps the leak is "better", in at least there won't be non-OOM crashes, but it still leaves a bad taste in one's mouth.
For a long running program like Steam (that is for some odd reason calling setenv…?) … I'm not sure which is better. Better would be not calling setenv, which it sounds like they've worked on.
I would be happy if getenv() called strdup() on the value and returned that. It would cause loads of minor memory leaks and maybe a handful of serious ones, but they should be reasonably easy to clean up and would avoid returning pointers to mutable private data.
There are a lot of situations where allocating is not acceptable (e.g.: critical applications that don't heap-allocate in order to ensure they never OOM).
Of course, we could have just getenvdup() which does what you just said. Let each application chose which variant is best for them: leaking, or thread-unsafe.
It'd be an API change, sort of? Although POSIX doesn't seem to say anything about ownership of the pointer, beyond thou-shalt-not modify the data it points to. Doesn't seem to specify calling free as valid or invalid.
But honestly, it seems like one might as well do exactly that (strdup & return)? At worst, nothing calls free(), and it is equivalent to the macOS strategy of "just leak the memory" to make it threadsafe. But at best, a program could #ifdef its way into "oh, this semi-sorta-nonstandard-behavior on this particular OS" and call free(), getting both a thread-safe & non-leaking implementation.
The mere existence of Steam is astounding to someone who grew up playing nethack and chess on Linux.
But the Steam client is really strange. Sometimes it works for months, and suddenly a game won't start, or something doesn't work, and I have to do weird stuff to get it working like purging all files or reinstalling. It doesn't make sense, it's like the Steam client rots.
Steam is one of the worst software, it's frankly embarassing for such a huge company. It's a hotchpotch of different design principles and new features only get tacked on with a very different UI + UX. The client is mostly webviews but they are so slow and buggy and they use a very old insecure Chromium version under the hood. The launcher is still a 32 bit program. It's sad that other launchers are even worse though.
Exactly, Steam is very much the worst game launcher ever created, except for all the others.
I checked and its using Chromium 126.0.6478.183 from July, which isn't that old.
It's four versions behind the current one with a lot of CVEs. There seems to be an LTS 126 but it's only for ChromeOS - it's at least behind this one which fixed one critical and two High CVEs: https://chromereleases.googleblog.com/2024/11/long-term-supp...
Even being only one version behind is dangerous for such a huge attack vector.
It seems like a lot of steam client issues come from weird desire to:
1) Keep it 32 bit
2) Have a single store-frontend
At least, it came to a conclusion that it steam itself should be responsible for managing runtimes for games.
The interesting thing is that Valve is not a "huge company". They're privately held and only had 336 employees the last time headcount was reported (2021).
In my experience, all of those things are also true of Steam on Windows. In fact, if anything I have to deal with it less frequently since switching exclusively to Linux.
This is really cool insight into both the Steam client and Linux programming. I understand why there may not be detailed release notes every release, but wow "Fixed some miscellaneous common crashes" is an understatement when you know about this work!
> If this can be addressed in glibc, it may involve a tradeoff on features, maybe an opt-in mechanism with a slight departure from the "impossible" POSIX spec. That's something we may pursue in the long term if we can propose something sensible.
Yes please
I'm really curious why they're using setenv(3) so much. The main usages that I can think of is setting an environment variable before calling something like exec(3). That doesn't seem to be the case here.
The article mentions that they use exevpe for spawning children processes. So what usages of setenv(3) would remain?
It is possible to check for setenv/unsetenv/putenv with nm -D, and a quick sample of my ~/.cargo/bin/* shows far too many programs using those. Yeah they could be single threaded, but who can guarantee they will remain so? Come to think of it listing symbols could detect pthread_create as well.
I'd be interested in a way to do static binary analysis to get from those symbols to a call tree, as well.
I don't see a way to check for **environ usage though, the compiler could turn this one into anything.
> We removed the majority of setenv calls. It was mostly used when spawning processes
Could someone elaborate this for a non-developer? Why would you use `setenv` (which I assume is functionally similar to `export key=value`, but correctly me if I'm wrong) (extensively) for spawning processes?
Environment variables are per-process, but inherited by child processes (not system wide). So when you start other programs you often set up env vars for them, or clear env vars that were used by your parent process to not inadvertantly perturb the child's behaviour.
Yes the proper way to do that is execve, not by changing global variables in your parent process. https://man7.org/linux/man-pages/man2/execve.2.html
Using setenv is mostly always a hack that relies on a bunch of assumptions that could easily change and be hard to debug.
Unless there is a fork exec family of syscalls are not child processes but the same process itself. Small detail but important because sometimes you really want a separate process altogether.
in that case you do not need setenv to pass values to the underlying process though. And if you are forking without immediately execv in a multithreaded program, setenv is the last of your problems.
edit: what's probably happening is that execve is four or five abstraction layers deeper (possibly in a third party dependency) than where the env variable need to be set without a clean way to pass the values through.
The likely scenarios I was thinking (setenv vs execve) would both be called in the child process after fork. But of course here we get to the fascinating world of fork interactions with threads, and you could even have one thread fork()ing while another thread is doing a setenv().
Regardless of whether it's a hack, my broader question is: is it common in Linux to manipulate environment variables on the fly for a process or program?
I primarily use Windows, both as an end-user and an amateur programmer. From my experience, most programs on Windows don’t do this. If parameters are needed, they’re usually passed as arguments, while environment variables are used for more permanent settings, like %PATH%.
PATH is a good example why steam does this: Steam is a program to launch a wide variety of other programs, which it doesn't fully control. Those programs (games) may do anything including launching further programs (utilities), so steam may set PATH so that the game finds those utilities.
It can also be a way to pass license information or other configuration settings.
Looks like the Steam team moved to control spawning and do execvpe.
I would like to see at least in-process environment modification discouraged. Rust is dealing with the issue by considering getenv unsafe when coming through C, but getting rid of the read side is much harder than the write side.
It is decently common enough in unix. You are correct with my windows stuff it is fairly rare to set env vars to change the launching process. In the unix world though I have seen the pattern a decent number of times. With windows programs usually you see the pattern (not always) if it is ported from a unix system. Windows likes its ini/registry/cli items to do configuration. It doesnt mean the pattern can not be used in windows, I personally just have not seen it as much with native win32 apps. If you fire up something like 'git bash' you can see entire bash functions bound to env variables.
I think it goes back to where windows came from. That environment space in DOS was not exactly huge (256 bytes at one point?). In unix it seems like it was much larger and expressive.
One thing to note about command line arguments on Linux is, any user can typically inspect `/proc/{pid}/cmdline` and get the full command line used to start the process. So if you pass secrets like API keys, passwords, etc, via an argument, they're visible to the rest of the system. However, if you put secrets into the environment of the child process, only the user that owns the child process can inspect `/proc/{pid}/environ`.
https://man7.org/linux/man-pages/man5/proc.5.html#DESCRIPTIO...
One can use `hidepid` parameter when mounting procfs to hide cmdlines.
I don't know why this is not implemented today by default in most distros. Probably history reasons.
My guess is it would break stuff like `ps x`, but ideally tools that use the procfs would gracefully degrade in cases like this?
Yes, the fact that the original article exists shows that it's common :)
Maybe among the best decisions Java ever made was hiding setenv. You simply cannot set env vars in Java.
Java in general tends to massively dislike environment variables.
Which is a Good Thing, but unfortunately it makes dealing with proxy support so much harder - especially as there are just so damn many HTTP libraries that people use...
So does that also mean you can't write e.g. a POSIX shell in Java?
execve(2) and friends allow you to specify a new processes's environment variables. How Java may choose to expose that I don't know. Assuming they do, that is all a shell needs. You do not strictly need to call setenv(3) to change a variable that gets evaluated with $, etc. You do need to pass the modified environment to a child process.
1) java shell starts 2) read getenv into a java hashmap 3) evaluate shell input - any sets update the hashmap, any gets read from the hashmap 4) when you get to a point where you need to exec a process you use the hashmap to pass envvars into process builder - which is going to call execvpe for you.
So yeah you definitely could (there'd be other reasons why java wouldn't be a good choice).
TBH reading that folks were calling setenv before an exec to propagate env vars to a child made me sad. I would guess that other use cases were leveraging environ as a poor man's global variable - which is also unfortunate.
You can pass a new environment to a process you spawn, which I think is adequate? You just can't modify the environment of your existing process.
System.getenv exists, so getting is easy... But setting env vars generally involves a convoluted process.
You can modify the JVM's buffer, or spawn a new ProcessBuilder, or a few other things. But it's a nasty place to be.
Or make a little JNI class.
That doesn't work for preserving to child threads, unfortunately. Which is something people expect.
There are tons of reasons you can't write a POSIX shell in Java, starting from the lack of support for non-UTF-8 paths :)
I admire your courage to say something positive about java.
On another note, I believe Zig does the same thing.
I love the steam client on linux these days, especially the compatibility for non-steam games is so great and Ive been using it to play WoW Classic while I have covid
To raise awareness: there's been a bug with the Linux Steam client which has been persistent for a long time.
TL;DR: if you have Steam running for more than a ~day or so, you will run out of window handles so you won't be able to open any new graphical application/window until you restart Steam.
Using Steam Chat appears to make the issue worse (it happens earlier).
This has been documented under https://github.com/ValveSoftware/steam-for-linux/issues/9094 but for some reason that issue has been closed.
I personally just restart Steam every day but if someone else encounters this issue and doesn't know why their windows are not opening, this is why :)
I am using KDE/Wayland but I've observed this under X11 too.
KDE Wayland user here. I have never observed this issue myself, and I leave Steam running all the time (although I never open Steam Chat).
Although, not albeit.
You have a slave sentence there. These are incompatible with albeit.
Thanks, fixed! "Albeit" seemed like it would work, but "although" indeed fits much better.
To put it another way, albeit shouldn't be used with independent clauses.
1. I leave Steam running all the time (although I never open Steam Chat).
2. I leave Steam running all the time (albeit never with Steam Chat).
"I never open Steam Chat" can stand on its own as a sentence, but "never with Steam Chat" does not and thus can be appropriately modified with albeit.
I kinda wish HN allowed a visual indicator for subthreads that focus on tangents that don't hold universal appeal: comment grammar, humor, "why did I get downvoted?", etc.
There are lots of little discussions like this that I'd love to have, but which sometimes lead to a lot of downvotes.
I've got several Linux gaming machines (and a couple of business desktops I play FTL on sometimes), and it's weirdly inconsistent - the best and worst machine I have both use Intel Graphics. It almostnever shows up on the A770 - that one stays awake for weeks of daily gaming and I have no problems with it. My laptop with integrated Intel sees it all the time. My laptop with Nvidia graphics sees it often, but not nearly as often.
The one that's so annoying that I've actually developed habits around it is the tendency for the library window to freeze when it becomes unfocused - it's not all the time, but it's often enough that I now habitually close the window anytime I defocus it
Void Linux, xorg and never has this issue, often going many days without rebooting with Steam constantly open.
Interesting. I've left the Steam client running for weeks at a time and have not seen that issue. I bet it was closed because Valve couldn't replicate the problem.
If you actually look at the issue you can see it was closed because Valve shipped a fix and people confirmed it was fixed. Then later some other people reply that they still have problems but without any useful information.
This has made Steam something to avoid for me. I tried switching to Flatpak version to see if that would help, but unfortunately not. For me it started ~1 year ago.
The performance issues have made me not want to even open Steam anymore so I have nearly completely stopped playing. (not related to this issue but several games have also added kernel anticheats, killing off Linux versions, which also has contributed to this)
Would love an open source Steam client
Always cool to rediscover people via HN. This post reminded me of the work ttimo did for the Quake 3 engine more than two decades ago. I remember it because I read so many comments written by him (like 15 years ago):
https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type...
I can't wait to ditch windows for my tower PC.
The stability is fine for me, but the rendering performance in the steam client when the mouse is in the window is abysmal.
Nvidia?
Nvidia in my case, and I agree, the app performance is very poor. The easiest way to see the lag, IMO, is to open up the achievements list for a game and try to scroll around. It takes almost a second for mouse scroll wheel movements to have an effect.
(Up to date mint, cinnamon)
It's a NUC with 00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
Integrated graphics to be sure, but i'm usually only using it to stream games from PC or laptop and the performance is fine for that in game.
Could just be really old gnome installation and higher poll rate than normal. There used to be issues there..
It's AlmaLinux 9.4. Gnome 40.10.
Yeah, that is old enough that those issues may not have been fixed yet. If you have a high poll rate mouse I'd strongly advise upgrading.
I wish they'd make it more virtualization friendly. I don't want to run untrustworthy proprietary software on my main system. Common sandboxing mechanisms are insufficient since Steam and its games need access to the entire device tree anyway. Nothing short of a real virtual machine would do it for me. Will also make compatibility painless since I can just install the Linux distribution they support.
I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... Only to discover they are exceendingly hostile to users who do this VFIO stuff.
Just yet another reminder not to "buy" games on these platforms, I guess.
> I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance...
So just any standard/decent motherboard bought within the last 3-4 years?
> I wish they'd make it more virtualization friendly.
Which games do you have issues with when virtualizing? I've only been locked out of Halo Infinite and I enable all hyperv flags in libvirt.
> Just yet another reminder not to "buy" games on these platforms, I guess.
If you buy a game on Steam and you're unable to play it they'll refund you straight away, which is why everyone loves Valve and Steam even when the client is crappy.
I couldn't be happier with my setup, which I assume is similar to yours. (2 GPUs)
> So just any standard/decent motherboard bought within the last 3-4 years?
It isn't at all obvious that IOMMU is supported by even current top of the line motherboards. I looked at a lot of products and have yet to see the VT-d and AMD-Vi keywords mentioned in technical specifications. To confirm support I had to read their UEFI firmware manuals and look for instructions on toggling the virtualization support.
https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iom...
At least ECC memory support has started to show up in technical specifications. Looks like IOMMU is not quite there yet.
> Which games do you have issues with when virtualizing?
From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives. I'm currently assuming any game with battleye or easy anti cheat will issue permanent bans on virtualization detection.
> If you buy a game on Steam and you're unable to play it they'll refund you straight away
Somehow that doesn't bring me much peace of mind.
I guess I'm just lucky, because I've had one MSI and two ASUS boards with this setup without issues, I guess it isn't listed everywhere, my strategy of just buying boards hoping it works seems to be the way!
> From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives
There's a lot of misinformation on the internet, I haven't heard about any cheats used in the wild using a VM and haven't had issues.
>> If you buy a game on Steam and you're unable to play it they'll refund you straight away
> Somehow that doesn't bring me much peace of mind.
If zero risk purchases doesn't bring you peace of mind I recommend dual booting or running two rigs instead of using bleeding edge technology
On Windows, "Environment" is stored in the Win32 Thread Information Block/Thread Environment Block (TIB/TEB), so it's thread-local rather than process-global.
My only issue with Steam at this point is that it’ll just randomly complain it has no connection, no matter which content server I set it to connect to.
If I spam the ‘retry’ button it’ll eventually work, but it’s a massive PITA.
All of this stuff goes way over my head. I'm on Pop!_OS and am happy to report "it just works" (tm), though it ignores scaling entirely.
Considering glibc's effort, I have to wonder what the other libc do, and whether they already implement something like this.