vcpkg integration

[Graf Zahl]

I'll just copy my entire post from Discord as a basis for discussion:

The step by step explanation is 3 lines:

'git clone path-to-vcpkg-repo'
run vcpkg bootstrap loader
When starting CMake pass the toolchain file found in 'path-to-toolchain-file'. In the CMake GUI it can be entered when activating 'Specify toolchain file for compiling'

Of course with better to understand phrasing.
Once this is done the rest will be automatic. So the important thing is, what to set up? We currently provide several dependencies as local copies for Windows. And the CMake files for these are old and horrible. We also provide a precompiled version of libvpx, made by the EDuke32 team. This alone is reason enough for me to get rid of it as it is old and unmaintained.

So, now that vcpkg itself is working fine there's a few things to be considered:

clean up the dependencies on these local copies of 3 libraries and libvpx. Like I said, the CMake for this is awful and in an outdated style. It should be modernized, preferably without the local copies. But for this we need to decide on making vcpkg mandatory on Windows. We may have to anyway if more features come along.
modernize the CMake code in general. After updating to CMake 3.27 I get a bootload of warnings for these old subprojects. Unfortunately I do not have the time to familiarize myself with modern CMake to perform this task. Someone else has to help out.
Before going on with video codecs we will have to set up ZMusic so that it a) gets pulled in automatically and b) gets statically compiled into the EXE. This is necessary to have its dependencies available which are also needed by the audio decoders for modern video formats.

Alternatively, of course, we can go the standard route of shipping 100 or so DLLs, like most ports developed on Linux, but I fail to see the advantage of that. Binary size will inevitably be larger because it includes all unused parts of these libraries - doing a static compile would also be easier for building AppImages or Flatpaks on Linux.

The most important thing is that once you know what to do, vcpkg is dead simple to set up. Compiling some libraries may take a while but is normally a one-time affair.

Before merging the PR for this I'd like to get some feedback from the community. Just a note: This is a prerequisite for adding WebP or Webm/Theora support because manually managing these libraries on Windows is a no-go. So if this feature is rejected, there won't be any better graphics or video formats, simple as that.

[Rachael]

vcpkg is a huge important step forward for the project - in order to eventually get to a point where things work reliably the same on all platforms we have to set a standard and vcpkg is the only cross-platform standard that works in all 3 of the major OS's that GZDoom supports. It has the same functionality of yum, apt, homebrew, macports, chocolatey, etc, major difference being that vcpkg tries to pull packages from the original authors' sources and has its own recipe for compiling each one on each OS to whatever architectures that OS supports - pulling directly from the authors also means there's less modifications for each library when it comes downstream to the compiling process, and the recipe is maintained by the same vendor for all platforms - so ideally defects will break the same on all 3 OS's - hopefully - which makes things a lot easier to fix - even when that does not apply, worst case scenario there's a lot less accountability for the breakage rather than depending on 20 different library vendors to fix the same defect.

I know a lot of people hate it strictly because "Microsoft" and the hate toward the software giant is perfectly valid - but eschewing a perfectly good tool solely on the basis of that hatred is the equivalent of shooting oneself in the foot out of spite for someone else. It doesn't hurt them, it only hurts you.

vcpkg isn't perfect by any means but I'd rather work around the problems of a single tool than the whims and arbitrary decisions of various Linux distros that break GZDoom because of how they modify their packages, or the dependency hell that develops because of it.

[Graf Zahl]

Agreed on all points. It will still be a hard sell to the average Linux user who wants to "save disk space" by sharing all the libs. I guess we'll see how it works out, especially with _mental_'s macOS build setup.

[Professor Hastig]

I know a lot of people hate it strictly because "Microsoft" and the hate toward the software giant is perfectly valid - but eschewing a perfectly good tool solely on the basis of that hatred is the equivalent of shooting oneself in the foot out of spite for someone else. It doesn't hurt them, it only hurts you.

The real irony is that despite all the bad stuff, Microsoft has done a lot for Open Source in recent years, far more that some deeply entrenched people want to acknowledge. Vcpkg is the result when a big company with sufficient resources has to find a solution around the mess that has developed in Open Source library development which for some reason is utterly dominated by Linux developers who often lack any awareness of the needs of cross platform development.

I find the situation particularly interesting on macOS. Apple always made it exceedingly difficult to generate distributable packages with their tools when external dependencies were involved, and here vcpkg is a real godsent - finally a solution that takes care of all the behind the back nonsense. And it was made by Apple's biggest competitor...

[Rachael]

Macports and Homebrew always were options on macOS but neither works on Windows like vcpkg does. vcpkg also by design does its business without modifying your base system; in other words if things go fucky you can get rid of vcpkg's folder and start over again. Want to try doing that with Linux or macOS? Really not fun at all. Worst case scenario you have to reinstall the entire OS - and don't get me started on how much of a pain in the ass that is for Mac. (Thankfully, much less so with more recent models)

Yes - Microsoft has been way better about Open Source than they used to be. I hate that they bought Github but I can't deny the sheer volume of stuff they've been open sourcing lately.

[dpJudas]

Meanwhile, in the real world, Microsoft is still only doing their open source contributions whenever it is their own self interest - so let's not overstate how brilliantly good citizens they are!

About vcpkg, my issue with using it is that I don't think it is good enough yet for what it set out to do. Neither is the Visual Studio integration. I have a very small project where I tested it using it to get Qt. Every time Visual Studio updates itself, which is every 2 week, it rebuilds all of Qt! So by using vcpkg I lose 1 hour (thanks to the inefficient build scripts qt uses) where Visual Studio basically appears as hung, because the UI can't properly show what is actually going on.

Also, just for clarity, are you guys planning on using the old or the new way vcpkg is intended to be used? The old method tried it to do it the Linux way with a shared library/package between all projects (with all the problems that produce), while the new method tries to make it configured per project.

If it were up to me, I would not use vcpkg at this time. As annoying as it is to embed third party libraries, I still think it is the one that is the least painful option out there today.

[Rachael]

Never was saying they're great - just better than they used to be.

Back in the Bill Gates/Steve Ballmer days they had massive issues - just some of them were different than the issues they have now. Ironically, the anti-trust lawsuit taught them nothing - Windows 11 is insistent like a child that you really really should absolutely positively use Edge as your browser and if you use something different (like Firefox) you're evil and there's random machinations and traps that they use to try to enforce it - also along with OneDrive. Plus it's full of telemetry and privacy invasion like no previous versions of Windows (except 10) ever had.

So yeah, no one is saying they're perfect in every possible way. Just that they really have improved on the open source front quite a lot - whether it is all about self-interest for them or not - it's still a huge improvement.

[dpJudas]

They are much better yes, but sometimes it feels like there's developing this narrative that Microsoft is pro open source, champions of the open source developer. The truth is they bought Github not for us, but because it was killing their Team Foundation Server garbage project (nowadays called DevOps). They open sourced .net because they were losing the web developer market. Azure is pro open source only because they really really want you to use their server hosting solution. And vcpkg is because web developers have become really happy about their package managers, so they feel they had to do something.

[Graf Zahl]

Ultimately it boils down to one simple thing: Does the software itself have some intrinsic monetary value or not? In all these cases it is the services that are the money makers, not the software itself. Under these circumstances, keeping it proprietary and closed makes no commercial sense.
Which is something any sane CEO would see. In the dark ages their CEO was not sane enough, luckily he's gone now.

Regarding Github, I don't think they bought it because of their Team Foundation solution but because Github was having financial problems and Microsoft as their biggest customer would have suffered some damage, so better buying it out and ensuring it can continue operating. When I compare Github with its competitors it is still by far the best offer around.

Meanwhile, in the real world, Microsoft is still only doing their open source contributions whenever it is their own self interest - so let's not overstate how brilliantly good citizens they are!

About vcpkg, my issue with using it is that I don't think it is good enough yet for what it set out to do. Neither is the Visual Studio integration. I have a very small project where I tested it using it to get Qt. Every time Visual Studio updates itself, which is every 2 week, it rebuilds all of Qt! So by using vcpkg I lose 1 hour (thanks to the inefficient build scripts qt uses) where Visual Studio basically appears as hung, because the UI can't properly show what is actually going on.

The vast majority of these libraries are a horrible clusterfuck of shitty build systems and poor setups. I am not surprised that vcpkg cannot deal with it perfectly.

Also, just for clarity, are you guys planning on using the old or the new way vcpkg is intended to be used? The old method tried it to do it the Linux way with a shared library/package between all projects (with all the problems that produce), while the new method tries to make it configured per project.

The second. The libraries are being generated within GZDoom's build folder. I'm not sure it makes much of a difference here but it is surely preferable to have this contained.

[Professor Hastig]

These are still companies driven by market economy, so of course any decision to open source some software is driven by a cost/benefit analysis. I find it very interesting how things went here.
Once upon a time Microsoft was the king of proprietary software fighting the open competition with every means they got, but now they are taking a far more balanced approach toward open source where they check if it beneficial to their business.
On the other hand, a long time ago, Apple was a company that was friendly to open source and embraced open standards. nowadays their computers are mostly locked down and they are gradually withdrawing from open standards wherever they can, which even reached a point where some software makers whose core business is not where Apple is strong are reducing support for the platform because it requires so much more work.

I think this clearly shows how the business models have changed. Microsoft once was a monopolist who didn't have to play nice to conduct their business and Apple needed to gain potential customers' goodwill.
Nowadays Apple's customers are thoroughly locked into the platform while Microsoft has stiff competition in their core business, i.e. their corporate customers.

About telemetry, I seriously doubt that you can find any commercial software not doing it. The gained data of usage patterns is simply too useful for the developers. It's funny that Microsoft is taking the brunt of the criticism while nobody says a word about Apple (or Adobe or any other manufacturer of costly enterprise software.) They are all doing it, but that 'evil Empire' mindset holds on.

[dpJudas]

Microsoft takes the brunt of the criticisms for one simple reason: they are still a monopoly. I can choose not to use Adobe or Apple (in fact, I use neither now). But I am effectively forced to use Windows, since that is where all the applications and games are. No, Linux is not a viable alternative - that's why they are stuck at 1-2% market share.

Apple and Google are two nice examples of just because Microsoft currently benefits from pushing the pro open source narrative it doesn't mean it will stay that way.

[Professor Hastig]

Their Windows business is a quasi monopoly, but these days it is far from their main route of income.
They currently mainly compete in a space where open source is important, i.e. web server and web services.

I'll be honest here: I find this stance of 'they were evil once, they are less evil now but since they are the Evil Empire they will inevitably turn more evil again' a bit ridiculous.
It's a bit of the inverse of Apple's perception. They once were relatively benevolent, but they more market power they accumulate the more evil they become, yet nobody complains and tries to stop their abusive business practices. One of the worst is ApplePay. Since it is the only means to use an iPhone for payment, businesses are forced into agreements with them because they have no choice, yet nothing happens on that front.

[Rachael]

Oh trust me this place has seen WELL more than its fair share of Apple-bashing! And well deserved, too.

Like when Apple deprecated OpenGL to push their vendor-locked Metal API, or when they decided to deprecate the Intel architecture to push their own proprietary ARM-based "Apple Silicon" architecture. These decisions were not popular with the developers on this forum when either were announced and to this day I have many reservations about Apple and Apple products, especially that product life cycles seem to be so short and recently they even seem to be engineering expensive hardware to fail quicker in order to make more money. Don't even get me started with their Mac Studio line of products - absolutely as close to zero value for the dollar per dollar as you possibly can get. The only economical way to own Apple products nowadays is by second-hand, and with everything seeming to intentionally have short life cycles, even that is iffy.

I would be lying if I said I wasn't excited about getting a new Apple device - I've had a few and was excited with the arrival of each one - but that excitement is so heavily tempered in just knowing just how bad Apple has become lately. Even more disappointing is the fact that, my current computer has cost me less than half as much to build as the most expensive computer that Apple has to offer (it would have to be customized on their website, of course), and even that still has twice the total horsepower on both the raw CPU and GPU capacities and speeds - and much more RAM and SSD storage to boot. Anyone who buys a Mac Studio anything is buying into sham marketing.

[Iluin]

wow the above discussion is getting quite off topic, I just want to address some points about vcpkg:

> bout vcpkg, my issue with using it is that I don't think it is good enough yet for what it set out to do. Neither is the Visual Studio integration.

Don't use the MSBuild integration. MSBuild does not provide a way for library lookup so vcpkg instructs the linker to link literally *.lib. The same applies to additional build flags like include folders etc. Best experience is with CMake since there the necessary usage information can be passed downstream

> I have a very small project where I tested it using it to get Qt. Every time Visual Studio updates itself, which is every 2 week, it rebuilds all of Qt! So by using vcpkg I lose 1 hour (thanks to the inefficient build scripts qt uses) where Visual Studio basically appears as hung, because the UI can't properly show what is actually going on.

Simply learn how vcpkg calculates the ABI hash for your dependencies. You probably want to disable compiler tracking and use exact version matches for e.g. cmake. See https://github.com/microsoft/vcpkg/issu ... 1319244766 (disabling compiler tracking is not recommended). Also only depend on the qtmodules you are actually using instead of the "qt" metaport (If you are not already doing that). Also don't forget to set the baseline version otherwise vcpkg will rebuild if the port dir gets updated.

> The old method tried it to do it the Linux way with a shared library/package between all projects (with all the problems that produce)

No the classic way never forced users to just use one vcpkg folder for all available projects. Only the MSBuild integration is limited to one folder at a time (you can switch if you switch the projects). The main benefit of manifest mode is consistency and baseline pinning if you are not interested in versioning. Classic mode was able to run into issues if people used vcpkg install multiple times and updated the repo between the install commands.

[dpJudas]

I'll be honest here: I find this stance of 'they were evil once, they are less evil now but since they are the Evil Empire they will inevitably turn more evil again' a bit ridiculous.

You are the one using terms like evil. I simply said that narratives saying Microsoft is somehow now pro open source is annoying. I find it to be naive at best. In the same way I also question the logic in giving Microsoft a free card for its dark patterns and uploading data left and right in ways that most certainly violate the GDPR since they didn't give me a choice to opt out. That it also happens post sale in an EULA just makes it even more questionable. Now they doubled down with their Microsoft account shenanigans. If you feel we can't say this just because other companies also do it, OK, that's your point of view I guess. I might even call it a bit ridiculous point of view to be honest.

Don't use the MSBuild integration. MSBuild does not provide a way for library lookup so vcpkg instructs the linker to link literally *.lib. The same applies to additional build flags like include folders etc. Best experience is with CMake since there the necessary usage information can be passed downstream

I never said anything about MSBuild. I was specifically talking about opening the project as a folder. The official CMake integration built in Visual Studio.

Simply learn how vcpkg calculates the ABI hash for your dependencies. You probably want to disable compiler tracking and use exact version matches for e.g. cmake. See https://github.com/microsoft/vcpkg/issu ... 1319244766 (disabling compiler tracking is not recommended). Also only depend on the qtmodules you are actually using instead of the "qt" metaport (If you are not already doing that). Also don't forget to set the baseline version otherwise vcpkg will rebuild if the port dir gets updated.

So you are suggesting I do something that isn't recommended? So the recommendation is to wait an hour every 2 week? That's the kind of thinking that makes me feel it isn't mature enough yet. Now don't get me wrong, I like the general principle of vcpkg, but so far it isn't something I would choose to use in a production environment just yet. I guess we'll just have to see how it works out for the GZDoom team.

No the classic way never forced users to just use one vcpkg folder for all available projects.

That's a matter of perspective. The way they wanted you to do it IMO made it a nightmare to manage. Luckily it seems they themselves also realized it was terrible and changed tactics with the manifest mode.