Is it possible to save the entire Zdoom wiki?

Discuss anything ZDoom-related that doesn't fall into one of the other categories.
hardcore_gamer
Posts: 109
Joined: Sun Jan 14, 2018 12:14 pm

Is it possible to save the entire Zdoom wiki?

Post by hardcore_gamer »

Is there some way to just download the entire zdoom wiki unto my hard drive so that I don't need to access the internet to read it? Ditto for the doom wiki.
JadedLexi
Posts: 134
Joined: Mon Apr 16, 2018 6:24 pm
Preferred Pronouns: She/Her
Location: Ylisse

Re: Is it possible to save the entire Zdoom wiki?

Post by JadedLexi »

User avatar
Kappes Buur
 
 
Posts: 4133
Joined: Thu Jul 17, 2003 12:19 am
Graphics Processor: nVidia (Legacy GZDoom)
Location: British Columbia, Canada

Re: Is it possible to save the entire Zdoom wiki?

Post by Kappes Buur »

I never was successful using HTTrack, but WGET works like a charm.
Instead of the one shown, I use this batch file

Code: Select all

"wget.exe" -rk https://zdoom.org/wiki/Main_Page --no-check-certificate --level=0 --html-extension --include-directories=wiki
Downloads the entire WIKI, about 5600 files, in ca 2 hours 40 minutes.

I just wished I could download only certain files, because about 30% of the files are rather useless.
After trimming those files I was left with about 96 MB of helpful info.
hardcore_gamer
Posts: 109
Joined: Sun Jan 14, 2018 12:14 pm

Re: Is it possible to save the entire Zdoom wiki?

Post by hardcore_gamer »

I know this is a bump but seeing as article 13 just passed I am now interested in downloading the wiki again. As Kappes pointed out I'm not sure how to use HTTrack, so I downloaded WGET instead. But i'm not sure how to use it. Can somebody give me the abridged version on how to use it?
User avatar
Kinsie
Posts: 7402
Joined: Fri Oct 22, 2004 9:22 am
Graphics Processor: nVidia with Vulkan support
Location: MAP33

Re: Is it possible to save the entire Zdoom wiki?

Post by Kinsie »

hardcore_gamer wrote:I know this is a bump but seeing as article 13 just passed I am now interested in downloading the wiki again. As Kappes pointed out I'm not sure how to use HTTrack, so I downloaded WGET instead. But i'm not sure how to use it. Can somebody give me the abridged version on how to use it?
Presumably by doing this:
Kappes Buur wrote:

Code: Select all

"wget.exe" -rk https://zdoom.org/wiki/Main_Page --no-check-certificate --level=0 --html-extension --include-directories=wiki
hardcore_gamer
Posts: 109
Joined: Sun Jan 14, 2018 12:14 pm

Re: Is it possible to save the entire Zdoom wiki?

Post by hardcore_gamer »

Kinsie wrote:
hardcore_gamer wrote:I know this is a bump but seeing as article 13 just passed I am now interested in downloading the wiki again. As Kappes pointed out I'm not sure how to use HTTrack, so I downloaded WGET instead. But i'm not sure how to use it. Can somebody give me the abridged version on how to use it?
Presumably by doing this:
Kappes Buur wrote:

Code: Select all

"wget.exe" -rk https://zdoom.org/wiki/Main_Page --no-check-certificate --level=0 --html-extension --include-directories=wiki
No, I mean how do you use this software? There are no obvious instructions that I can tell. I downloaded it and used the exe but nothing happened.
User avatar
ramon.dexter
Posts: 1547
Joined: Tue Oct 20, 2015 12:50 pm
Graphics Processor: nVidia with Vulkan support
Location: Kozolupy, Bohemia

Re: Is it possible to save the entire Zdoom wiki?

Post by ramon.dexter »

Have you ever heard about command prompt? I bet you have not. So, google what that means and how to use it.

Small advice to you - some programs doesnt have any GUI and are controlled from command prompt only.
User avatar
Kappes Buur
 
 
Posts: 4133
Joined: Thu Jul 17, 2003 12:19 am
Graphics Processor: nVidia (Legacy GZDoom)
Location: British Columbia, Canada

Re: Is it possible to save the entire Zdoom wiki?

Post by Kappes Buur »

hardcore_gamer wrote:No, I mean how do you use this software? There are no obvious instructions that I can tell. I downloaded it and used the exe but nothing happened.
Open a text editor, for example Notepad, and enter the line I gave you. Save as downloadWIKI.bat, or somesuch name, into the WGET/bin folder. Make sure that the file has the .bat extension. Doubleclick on the batch file and the download should start.

Or, as ramon.dexter suggested, open the command prompt and navigate to the WGET/bin folder and copy/paste the line
Spoiler:
Running this command, either from the command prompt or through a batch file, creates a new folder in WGET/bin as zdoom.org and the download starts
Spoiler:
The end result will look like this in your browser, since all files are of the .html type
Spoiler:
The icon for the files will be different for you if you are not using Opera.
You may want to sift through the downloaded files as quite a few do not contain useful information.
kingdomakrillic
Posts: 4
Joined: Thu Sep 13, 2018 2:12 pm

Re: Is it possible to save the entire Zdoom wiki?

Post by kingdomakrillic »

ZDoom's wiki runs on MediaWiki, so WikiTeam's dumpgenerator.py tool will work. Though it'll save the wiki as XML files, which isn't useful if all you want is to read the wiki offline.
User avatar
Enjay
 
 
Posts: 26537
Joined: Tue Jul 15, 2003 4:58 pm
Location: Scotland

Re: Is it possible to save the entire Zdoom wiki?

Post by Enjay »

I'd love to have a decent local copy of the Wiki but when I've tried to download it, the ability to search doesn't work in the local copy, so it's not that useful (to me anyway).
User avatar
Rachael
Posts: 13626
Joined: Tue Jan 13, 2004 1:31 pm
Preferred Pronouns: She/Her

Re: Is it possible to save the entire Zdoom wiki?

Post by Rachael »

Kappes Burr has the right idea with the wget thing, but it doesn't exclude talk pages or any control activities.

I've been considering developing a script to run directly on the web server and upload a complete copy of the wiki to the DRD Team Devbuilds site.
User avatar
m8f
 
 
Posts: 1445
Joined: Fri Dec 29, 2017 4:15 am
Preferred Pronouns: He/Him
Operating System Version (Optional): Manjaro Linux
Location: Siberia (UTC+7)

Re: Is it possible to save the entire Zdoom wiki?

Post by m8f »

I managed to download the ZDoom Wiki using the instructions given on the wiki page with httrack (using HTTraQt GUI), but I had to expand Scan Rules:

Code: Select all

-*.diff.gz -*.rar -*.cab -*/*gitview*/* +*.png +*.gif +*.jpg +*.css +*.js -ad.doubleclick.net/* -*Special:* -*Talk:* -*User:* -*&action=* -*&printable=* -*&oldid=* -*.zip -*.cpp -*.h -*.tar.bz2 -*.c -*.hpp -*.tcc -*.cc -*.deb -*.7z -*.dmg -*.tar.gz -*.exe -*zdoom.org/Changelog/* -zdoom.org/gitdiff/* -*zdoom.org/Changelog* -*zdoom.org/changelog* -github.com/* 
Then with the help of Dashing tool, the offline Dash/Zeal docset was created. I had to do a manual cleanup. The result is far, far, far from perfect, but usable. To use this docset, put the unarchived directory (zdoom.docset) to a directory where other docsets are located (on my machine, it's ~/.local/share/Zeal/Zeal/docsets).

Gez
 
 
Posts: 17842
Joined: Fri Jul 06, 2007 3:22 pm

Re: Is it possible to save the entire Zdoom wiki?

Post by Gez »

Enjay wrote:I'd love to have a decent local copy of the Wiki but when I've tried to download it, the ability to search doesn't work in the local copy, so it's not that useful (to me anyway).
Of course. For that, you'd need to run your own local copy of Wikimedia on a localhost server (something like WAMP or EasyPHP could probably do the trick on Windows) and then use the mediawiki export and import tools.

M8f's alternative solution of creating a Dash or Zeal docset may be much simpler.

Return to “General”