[4.1.1] libSPIRV.so: cannot open shared object file: No such

[Rachael]

We don't need to hold that course. I am not one to venerate anyone, even Randi, for being perfect. I know that she knew a lot more about these things than I ever did, though.

If it is better to statically link the internal copy of some of these libraries, then it's what's obviously what we should do. I don't give a flying piss fuck about how bloated GZDoom is on Linux if the "bloat" is what makes it work. I already made a very clear case about that when someone attacked us for the "bloat" on Windows.

Would I prefer a smaller executable size? Are you crazy? Of course I would! I think anyone would. But that does not excuse utilizing solutions that clearly don't work.

And as you can tell, over time I have started crippling CMake's external library searches for certain libraries in order to force it to use the internal versions. That was no accident. It's because it was necessary to do so. Again - it working properly is more important than saving 2 PRECIOUS single bytes of disk space. (And if that seemed extremely sarcastic and mocking - don't worry, it was intended to be)

[dpJudas]

Funny you say that, since dynamic linking was much less prevalent in the 80s and early 90s (let alone 70s). Consoles, which are more memory- and storage-limited than PCs, also stuck with static linking longer than PCs did.

ELF was introduced in 1988 as part of SVR4. They don't spend time making such a large system change unless they feel they have a problem - as computing grew ever more popular the desire to be able to run more software on the same hardware also grew. In the 80's and early 90's memory was still very much at a premium.

Windows 95 was designed to fit into 4 MB of system memory. Think about that for a moment - 4 megabytes. We burn more than that on just one of our upload buffers in GZDoom. So while disk space had indeed gone up (compared to 80's) it doesn't change the fact that at this time in history every single bit you could save counted - and dynamic linking saves both memory and disk space. Look closely at the ELF standard and notice how much care has been put into touching as few pages as theoretically possible so that the maximum amount of memory could be shared. They wouldn't have bothered with all that if it wasn't one of the key points of the entire exercise.

Dynamic linking is necessary for client libraries to update along with the service/server programs, if you want to maintain backwards compatibility as the communication protocol changes.

First of all, this isn't how Linux distros decided to do things. They decided to link *everything* dynamically - from zlib to your favorite xml parsing library. If they truly kept dynamic linking to their platform ABI then I wouldn't have a problem with their policy, but they do not.

Secondly, proper client/server design doesn't require the client library to update as the protocol used to communicate itself can be versioned. X11 is one example of something already designed like this - you can statically link xlib, xrandr and so on and it will work for 20+ years. There are already evidence for this as Unreal Tournament and other Linux games were linked this way. The linux kernel itself is also an example of how you can change things and still support decades old stuff - unlike userland the kernel itself always kept its part of the userspace compatibility bargain.

That said, I agree that actual system libraries does have a case for dynamic linking if they do things right. But once again, that isn't what Linux distros are doing with their /usr/lib folders. Windows went away from this somewhere around 1994 when COM was introduced. It became all to clear what happened when everyone and everything thought they could share random utility libraries and toss it into the System32 folder.

[Marisa the Magician]

Let's just wait for the first hardcore Linuxer to arrive here and explain why this is such a great feature.

Sorry, I'm late. I'm not going to say anything because I know that whatever I say won't change anyone's mind here on the subject, but personally, I think dynamic linking is fine. I have no problems with it.

Oh and I have no complaints about GZDoom's current binary size. I've honestly seen worse in this age where everyone keeps jumping into the bandwagon of the ironically named Electron, which unlike its namesake, is anything but tiny.

you can statically link xlib, xrandr and so on and it will work for 20+ years. There are already evidence for this as Unreal Tournament and other Linux games were linked this way

I don't know where you got that info from but the UT Linux port uses dynamic linking and comes bundled with ancient versions of various libraries like SDL, OpenAL, mikmod and libstdc++. With the exception of those bundled libs, it links to whatever's on the system, including the X11 libs and whatnot.

[dpJudas]

I don't know where you got that info from but the UT Linux port uses dynamic linking and comes bundled with ancient versions of various libraries like SDL, OpenAL, mikmod and libstdc++. With the exception of those bundled libs, it links to whatever's on the system, including the X11 libs and whatnot.

Okay I'll admit I didn't actually run ldd on the Linux version of UT to check. But if it comes bundled with the old versions the linking is effectively static. My info on Desktop Linux is somewhat out of date, but I remember back in the day my Linux distro used static libraries for the X11 extensions.

In any case, do what you like with the Linux version of GZDoom. If it produces compile errors due to altered API or a GLSL compiler that miscompiles I'll consider it not my problem.

[Chris]

Funny you say that, since dynamic linking was much less prevalent in the 80s and early 90s (let alone 70s). Consoles, which are more memory- and storage-limited than PCs, also stuck with static linking longer than PCs did.

ELF was introduced in 1988 as part of SVR4.

I didn't say it wasn't possible, I said it was less prevalent. Dynamic linking was made to solve problems apps were running into in the 80s, but memory and storage concerns weren't the biggest among them. Unix-based systems (Linux wasn't around in the 80s) were designed more for mainframes and server use, which would tend to have more storage and memory than a typical PC. Customization (third-party plugins) and resource sharing (multiple apps/users accessing the same hardware at the same time) were bigger issues to solve on these systems. PCs started utilizing dynamic linking more as they started doing similar things.

Windows 95 was designed to fit into 4 MB of system memory. Think about that for a moment - 4 megabytes. We burn more than that on just one of our upload buffers in GZDoom. So while disk space had indeed gone up (compared to 80's) it doesn't change the fact that at this time in history every single bit you could save counted - and dynamic linking saves both memory and disk space.

Consider games like Doom or some other typical DOS game. When you run its setup program to select your sound and MIDI devices, those are all effectively hardware drivers the game's engine/middleware has code for. Each game that uses CGA, VGA, and/or VESA is specifically coded to use it. DOS games didn't use DLLs to share this between multiple games on the same system, each game contained the code for hardware it supported, and were getting by relatively fine... until you start adding in multi-tasking, a windowing system, and other shared hardware resources.

Look closely at the ELF standard and notice how much care has been put into touching as few pages as theoretically possible so that the maximum amount of memory could be shared. They wouldn't have bothered with all that if it wasn't one of the key points of the entire exercise.

Just because dynamic linking wasn't made to solve problems with memory and storage doesn't mean it can't be memory and storage efficient. Solving problem A, and getting benefits with B as a side effect, isn't an uncommon occurrence.

First of all, this isn't how Linux distros decided to do things. They decided to link *everything* dynamically - from zlib to your favorite xml parsing library. If they truly kept dynamic linking to their platform ABI then I wouldn't have a problem with their policy, but they do not.

It's certainly not *everything*. A fair number of -dev packages here come with static libraries. It may be the policy of various distros to prefer dynamic linking, but it's certainly not a requirement. There's nothing stopping you from static-linking everything from zlib to your favorite xml parsing library, and only dynamic-linking the platform interfaces, if that's what you want for your apps. Some ideologues may complain (with or without valid arguments for a given library), but the actual system will be just fine with you and people can run what you provide.

The linux kernel itself is also an example of how you can change things and still support decades old stuff - unlike userland the kernel itself always kept its part of the userspace compatibility bargain.

The kernel is a pretty different beast than userland. The kernel is the bridge to the hardware, while userland needs to do system management and make sure apps play nice together. But even still, the kernel hasn't gone without breakages... when the kernel switched from its original threading implementation to NPTL, I remember things got rather bumpy. If you go far enough back to apps that use OSS (the audio API), that gets emulated in userland these days. It doesn't get handled in the kernel anymore because it can't share resources very well at that level. OSS4 tried to add sharing, but as we can see, it never really took off on Linux -- you're making the kernel do things it isn't intended to, or you're doing a weird kernel->userland->kernel loop.

But once again, that isn't what Linux distros are doing with their /usr/lib folders. Windows went away from this somewhere around 1994 when COM was introduced. It became all to clear what happened when everyone and everything thought they could share random utility libraries and toss it into the System32 folder.

COM is a fancy dynamic loading interface. Under the hood, it's loading DLLs and calling functions based upon the requested interface to get you a vtable, and automatically freeing DLLs when they're no longer used. COM can also only work with DLLs that are registered with the system, storing information globally in the registry, which causes its fair share of issues (IMO far worse than random dlls tossed into system32; at least if you dump a random foo.dll in system32, an app won't use it if nothing explicitly looks for it, but with COM, foo.dll may be used wherever it is if it got registered for a particular interface).

But if it comes bundled with the old versions the linking is effectively static.

Not really, because I can replace the SDL and OpenAL shared libs it comes with to make it use updated versions, which can use newer modesetting and window management features or directly communicate with PulseAudio with more advanced resampling and mixing capabilities that weren't available back then. If they were actually static-linked, you'd have to do hacks with code-injection to replace internal calls with external ones (assuming none of the library code got inlined, which would make it much more difficult). As an interesting side-note, some years ago SDL implemented a method in which its own calls could be overridden with itself. So you could static link SDL and not worry about supplying an SDL.so or whatever, but the code would look for a user setting that could make it load a dynamic version of itself and forward its calls to there.

[dpJudas]

I'll try not to make a too detailed answer here because I think it is somewhat pointless in the sense that we will never reach an agreement here on whether mainstream Linux distros have a reasonable way of handling libraries. We will just have have to disagree on that.

But I would like to remind you that I'm not just some random that has no idea how COM works or how dynamic libraries work. I'm also not some young kid that don't know anything about the 80's or 90's - I grew up on computers in that age and wrote software that ran in DOS, some on Alpha and SGI machines at university too. As simple examples of this I recently wrote a dinput.dll replacement for UT, or how about the JIT code in GZDoom that generated a valid ELF .eh_frame for Linux and macOS. In DOS I wrote my own tracker software that used the GUS to play sound and I also did graphics in that age - I fully understand that dynamic libraries are good for plugin architectures. I wrote a printer driver in kernel space for both Windows 95 (god damn 16-bit) and Windows NT. So please, stop assuming my viewpoint is some fundamental misunderstanding of what DLLs do or how they work. Or that I don't know the differences between kernel and user space.

[Chris]

I'll try not to make a too detailed answer here because I think it is somewhat pointless in the sense that we will never reach an agreement here on whether mainstream Linux distros have a reasonable way of handling libraries. We will just have have to disagree on that.

FWIW, I'm not arguing whether distros have a good policy on how to handle libraries or not. I'm talking about GZDoom, or anything you may want to you write/distribute for Linux... just because most distros dynamic link everything under the sun for their packages doesn't mean you have to for yours. Most games on Linux do static link most of their libraries, and people are fine with it.

But I would like to remind you that I'm not just some random that has no idea how COM works or how dynamic libraries work.

I didn't say you were, and I apologize if I came off that way. What I'm saying is that the space and memory savings, while nice, wasn't the main motivating factor for dynamic libraries back in the 80s and 90s, so having more space and memory than the 80s and 90s wouldn't be a reason to get rid of them now. What would be a good reason is if a given library has shown itself to have an unstable ABI, or you need custom modifications, or things of that nature where you can't just swap it out for a newer version. There's no point in something being a shared library if it can't be shared or changed independently, I'd certainly agree with that.

[Graf Zahl]

I'll try not to make a too detailed answer here because I think it is somewhat pointless in the sense that we will never reach an agreement here on whether mainstream Linux distros have a reasonable way of handling libraries. We will just have have to disagree on that.

Let me just say this: At my workplace, the maintainer of our server software is fighting a futile battle with the system administrators of the servers at our customers to keep the system in a reliable state. Granted, it's not libraries here but installed services, but the mess of various dependencies outside our own control is making this a very unpleasant job.

But in the end the problem is the same: You depend on components you do not control yourself, which can be buggy, outdated or misconfigured either due to negligence or some admin who tends to tinker around with the configuration too much.

There's a reason why no successful GUI based OS uses a setup where installed applications depend on a random set of dynamically linked libraries. It's not the norm on Windows, it's not the norm on macOS, it's not the norm on Android and it's also not the norm on iOS - in the latter two cases it is outright impossible and with macOS it's unlikely to work well. In all four cases, it is expected that the application ships its own dependencies along with the main executable. Linux is the only system that violates this primary rule of sandboxing an app - do not let it allow to run code that neither belongs to the system nor to the app itself.

I think there's also a reason why Linux does not catch on as a desktop OS - and this dependency hell scenario with software having to depend on non-system components too much - is a big part of it. As long as there is no guarantee that these components are compatible on any imaginable system they are part of the problem, not part of the solution.

[dpJudas]

I agree with you on all that, Graf. Chris does have a point though in the sense that it isn't the Linux kernel and DLL system in itself that prevents bringing your own dependencies. Rather, it is a design choice of the distros to do it this way.

The good news is that lately they have begun to realize this and started new packaging strategies that will eventually solve it (flatpak and snap). We will see if they truly adopt this or not.

[Graf Zahl]

I agree with you on all that, Graf. Chris does have a point though in the sense that it isn't the Linux kernel and DLL system in itself that prevents bringing your own dependencies. Rather, it is a design choice of the distros to do it this way.

I know. but it's actually harder to fight some community attitude than a technical problem.

The good news is that lately they have begun to realize this and started new packaging strategies that will eventually solve it (flatpak and snap). We will see if they truly adopt this or not.

Like I said, the problem isn't technical. Those who are deeply convinced of the 'share everything' idea will never warm up to these things.

[dpJudas]

Like I said, the problem isn't technical. Those who are deeply convinced of the 'share everything' idea will never warm up to these things.

I guess we are about to see how large percentage of the decision making people in the mainstream Linux distros are made of such people.

[vilhelmgray]

Here's the Gentoo team's rationale for preferring external libraries by default: https://wiki.gentoo.org/wiki/Why_not_bu ... pendencies

[dpJudas]

Safe to say I disagree with that analysis. Aside for the security argument, most of those arguments are pretty weak and one-sided. But then again, I consider the entire way of distributing software the way gentoo does pretty brain dead, so it isn't very surprising they have a different stance than me.

[Graf Zahl]

This is a textbook case of people failing to see the bigger picture. I somehow get the feeling they are totally hung up on the concept of a user who self-compiles every bit of code on their system. But that seems to be a large part of Gentoo's users anyway.

Needless to say, with such an attitude Linux will never get some larger market share on the desktop, if it constantly fails to provide a reliable environment.

[Chris]

This is a textbook case of people failing to see the bigger picture. I somehow get the feeling they are totally hung up on the concept of a user who self-compiles every bit of code on their system. But that seems to be a large part of Gentoo's users anyway.

Needless to say, with such an attitude Linux will never get some larger market share on the desktop, if it constantly fails to provide a reliable environment.

I think it's safe to say Gentoo isn't one of the bigger distributions. It may have been a sizable one at one point, due to being an early build-it-yourself/power-user distro that had a decent package manager, but it's apparent that custom building all your packages really didn't give you that much in terms of performance and resource saving, it was easy to break things with bad build settings, and building packages took way more time than just grabbing binaries. You really have to commit to using a distro like that, it's not something a normal user should want or be made to use.