[4.1.1] libSPIRV.so: cannot open shared object file: No such

[vilhelmgray]

I am attempting to run GZDoom version 4.1.1 on a Linux x86_64 target. The following error is reported when the gzdoom executable is run:

Code: Select all

$ gzdoom
gzdoom: error while loading shared libraries: libSPIRV.so: cannot open shared object file: No such file or directory

It does not surprise me that libSPIRV.so is not found, because it is not installed on my system. I noticed the current gzdoom src/CMakeLists.txt file is unconditionally adding the "glslang", "SPIRV", and "OGLCompiler" libraries without first checking if they are installed on the system:

Code: Select all

if (HAVE_VULKAN)
	set( ZDOOM_LIBS ${ZDOOM_LIBS} "glslang" "SPIRV" "OGLCompiler")
endif()

These libraries should be detected first before they are added to the link list.

[Rachael]

The more important thing is - will it even compile without these libraries? As far as I know, while Vulkan support was made optional, it unconditionally compiles which requires the presence of these libraries. Making the libraries optional would require a bit of work invested and I don't know if anyone's willing to do that.

[_mental_]

I noticed the current gzdoom src/CMakeLists.txt file is unconditionally adding the "glslang", "SPIRV", and "OGLCompiler" libraries without first checking if they are installed on the system

These libraries should be detected first before they are added to the link list.

Source code of these libraries is stored in the repository. We can create static libraries and add them unconditionally.

It does not surprise me that libSPIRV.so is not found, because it is not installed on my system.

Most likely, it's quite the opposite and the library was picked from your system. We don't use dynamic libraries as targets for source code in the repository.

The more important thing is - will it even compile without these libraries? As far as I know, while Vulkan support was made optional, it unconditionally compiles which requires the presence of these libraries. Making the libraries optional would require a bit of work invested and I don't know if anyone's willing to do that.

Yes, it works with no dependencies installed. To be sure, you can check Travis configuration and build logs.

[vilhelmgray]

I noticed the current gzdoom src/CMakeLists.txt file is unconditionally adding the "glslang", "SPIRV", and "OGLCompiler" libraries without first checking if they are installed on the system

These libraries should be detected first before they are added to the link list.

Source code of these libraries is stored in the repository. We can create static libraries and add them unconditionally.

It does not surprise me that libSPIRV.so is not found, because it is not installed on my system.

Most likely, it's quite the opposite and the library was picked from your system. We don't use dynamic libraries as targets for source code in the repository.

If the intention of the makefile is to use the version of libraries provided in the source code repository, then something is going wrong since the gzdoom executable built is searching for a dynamic library.

Which part of the makefile sets up the static linking of the version of the libraries in the source code repository? Maybe we can track down where it's failing to statically link the objects for these libraries.

[Rachael]

That is most likely something going wrong with your system. What GZDoom does with most libraries (as long as they haven't been customized with bug fixes and such) is that it will use CMake to search your distro's build environment for the libraries - and if it finds them, it will use them, instead of an internal copy.

If it doesn't find them, then it will use its own internal copy, and most likely use static linking for that. If it finds an external copy it will probably dynamic link to it.

So what's likely the culprit here is that CMake is finding traces of these non-existent libraries on your system somehow. You will have to review the CMake settings by using the cmake-gui utility on your build folder and make sure that the libraries are pointed internally.

Here is an example of how it searches for the libraries - this is where the Vulkan additions are actually done - this line is the one you're most likely going to want to change to disable Vulkan completely - and these files (1, 2, 3) are the ones that CMake pulls in in order to add the Vulkan libraries.

[_mental_]

I guess -lSPIRV linker command line switch is added because of this line.
For some reason, however, the linker found libSPIRV.so in its search paths, and linked GZDoom executable with it.
Also, the path to this library is not in dynamic loader paths. That’s why the executable fails to start.

[Rachael]

That still falls into the HAVE_VULKAN conditional - which if disabled, should prevent that line from being added.

The trick here is to have that variable false, possibly by more thorough checks, if the system cannot fully support it.

[vilhelmgray]

I tracked down the cause of the issue.

I'm running Gentoo Linux, and building GZDoom via a Gentoo ebuild. Gentoo passes several default configurations to CMake, and within these defaults I found the following:

Code: Select all

SET (BUILD_SHARED_LIBS ON CACHE BOOLEAN "")

This was causing CMake to default to dynamic linking of libraries if static linking is not explicitly requested. Setting BUILD_SHARED_LIBS to OFF allows for the static linking behavior.

However, it would still be beneficial for the GZDoom makefile to provide FORCE_INTERNAL_SPIRV, FORCE_INTERNAL_GLSLANG, and FORCE_INTERNAL_OGLCOMPILER options so that users may explicitly choose to use the internal versions of these libraries rather than leaving it up to CMake's ambiguous default behavior.

Regardless, since I found a solution for this issue, you may mark this thread as resolved.

[dpJudas]

Honestly I would prefer it to always force the internal versions. That way GLSL compiler behavior is consistent and well known.

Two months ago I had to explicitly upgrade the GLSL compiler because the version Graf originally had downloaded had a bug in it - I'd really not want bug reports of that sort based on the random nature of Linux distros. If it is up the distro to pick the GLSL compiler an error like that could return, or a new one could appear. One of the advantages Vulkan has is that GLSL compiler behavior is consistent and a dynamic linking throws that right out of the window.

[Graf Zahl]

Heh, heh, one of my favorite Linux topics. Let's just wait for the first hardcore Linuxer to arrive here and explain why this is such a great feature.
In my book, the unpredictability of third party libraries makes the dynamic linking fishy at best and downright dangerous at worst.
For the GLSL compiler this is critical, though, this is a spot where consistency and reliability are far more important than saving a bit of code size. But you can almost bet that someone will submit a PR to reintroduce dynamic linking if static was enforced, because for some people static linking seems to be sacrilege.

In hindsight I wonder if it would have been possible to statically link to LLVM and work around those initialization issues that way.

[_mental_]

It's possible to mark libraries as always static like this. I did that for DUMB only as it won't link as dynamic library anyway.
We can do the same for critical (or even all) dependencies stored in the repository.

[dpJudas]

Thinking a bit further about this - we have to force the linking. The API of the GLSL compiler is not stable - I had to make a few minor code adjustments when I upgraded the compiler.

@Graf: totally agree with you on the sacrilege thing. It seems some people still think it is the 80's or early 90's where memory and disk was an extremely limited resource. Nowadays dynamic linking has more problems than advantages in my opinion.

[vilhelmgray]

Thinking a bit further about this - we have to force the linking. The API of the GLSL compiler is not stable - I had to make a few minor code adjustments when I upgraded the compiler.

@Graf: totally agree with you on the sacrilege thing. It seems some people still think it is the 80's or early 90's where memory and disk was an extremely limited resource. Nowadays dynamic linking has more problems than advantages in my opinion.

If it's critical for these libraries to be statically linked, then it's important to force it to do so in the makefile. In Gentoo, as well as many other distros, dynamic linking is treated as the preference as you've pointedly out -- so without the the makefile forcing static libraries, GZDoom will be configured for dynamic linkage and possibly result in subtle errors for users that will be difficult to track down for developers.

[Chris]

It seems some people still think it is the 80's or early 90's where memory and disk was an extremely limited resource. Nowadays dynamic linking has more problems than advantages in my opinion.

Funny you say that, since dynamic linking was much less prevalent in the 80s and early 90s (let alone 70s). Consoles, which are more memory- and storage-limited than PCs, also stuck with static linking longer than PCs did.

Hard drives and CDs started becoming a thing for PCs in the 80s and 90s, vastly increasing storage capacity beyond floppies, and the expected amount of memory users have is always growing. Far from being a means to save memory and disk space, dynamic linking largely comes into play when you have systems designed for multi-tasking and resource sharing, where it's more about consistency and interoperability. If each app needed to maintain code for handling desktop windows or an audio device or something, you'd have conflicts out the wazoo. Even on modern multi-tasking systems, many operations are service/server-based (e.g. video and audio are handled by a separate process, and apps link to client libraries that talk with those external processes). Dynamic linking is necessary for client libraries to update along with the service/server programs, if you want to maintain backwards compatibility as the communication protocol changes. Dynamic linking is also the reason we can still run closed-source games released 10 to 20+ years ago on modern systems (Win95's gdi32.dll or kernel32.dll certainly wouldn't work on Win10, but apps that linked against them can still run), and its the reason we can make custom replacement components for those games that provide a better experience. For example, there are dsound.dll replacements that emulate the system having a hardware DS3D audio, which not only restores 3D audio for many games, but can also restore environmental audio effects (things that haven't been available for DSound apps since Vista, if you even had a hardware audio device prior to then).

Dynamic linking isn't about working around memory and storage limits, but is for completely different problems that are still (if not more) relevant on modern systems.

[Graf Zahl]

And still, all this pales in comparison to dependency hell. The best way to avoid dependency hell is to avoid unnecessary dependencies, except to the underlying OS itself, of course.
There's a reason why these days very little software installs their DLLs directly into the Windows folder which was common practice in the early days.

But on Linux this still is a major issue because binaries not only link dynamically to the system but they link dynamically to everything by default. All fine and well if it can be guaranteed that all versions of some .so are compatible. But since this is not the case the entire setup isn't stable.
I even had it happen on macOS once with GZDoom - for some reason CMake picked up an external libJPEG which got installed by some other app and all hell broke loose because it was incompatible with GZDoom's code - the only solution was to configure the project to statically link to its own copy provided in the source package.
So why are these even configured to dynamically link? Simply because some Linux users were complaining and bitching that GZDoom was 'too bloated' and asked to change the setup. And Randi complied. Been there, done that, nothing but problems ensued.