WADbot 2 - automated taking "good" screenshots

[JPL]

UPDATE: first draft of the bot is now posting at wadbot.tumblr.com Thanks for the feedback so far, folks!

Hi folks. Last year I created a twitter bot called WADINFO.TXT that posts the name of a random level from Doomwiki.org. I've always had a more ambitious "phase 2" for the bot, and I've started thinking about it again recently. Basically, Stochastic Planet but for the entirety of the idgames WAD archives. Here are the steps I've broken it down to:

1. Download a random zip from the Doom idgames archives.
2. Use Omgifol to search the files in the archive for map lumps.
3. Pick a random map within the WAD and launch GZDoom with -warp to that map. Other settings include: +notarget, custom log file location, custom screenshot path/filename, desired graphical settings, no HUD, no view weapon.
4. With the map loaded, find a spot to take a "good" screenshot from, take it, log the player's current XYZ location, and quit.
5. Compile the text part of the post: read the log to get the map's name as displayed in-game, try to find the map's author in any included readme. Include a link to the idgames archive download.
6. Submit the image + text post as a draft via the Tumblr API for review.
7. A human (me) reviews draft posts to make sure they're not terrible or broken, and queues them for post using the Tumblr dashboard interface. (Theoretically, if the above steps always produced perfect awesome posts, I'd just automate this too.)
8. Tumblr post at the queued time, with the Twitter auto-post feature linking to that post.

So there are a few different phases for this, and I've tried to tackle the toughest one first: automating (run a script and sit back) the launching of GZDoom and taking a screenshot.

I'm running into a technical snag right now, as I can't seem to run the screenshot command in any automated way. I've tried including it in an autoexec.cfg, and all the other commands run but not the screenshot command. I've tried passing it in as a command line command, eg gzdoom.exe -warp 19 +screenshot, but that doesn't seem to work. I've tried creating an Alias but running that alias in an automated (ie non user instigated) way doesn't seem to work. It's as if any code in ZDoom that runs batches of commands just ignores the screenshot command. Any ideas on how to get around this?

There's also the more open problem of taking "good" screenshots. Initially I tried writing a small ACS library function that warps around to random XYZ coordinates, trying to find a spot that's A) inside the map, B) far enough away from a wall that the screenshot won't just be a wall. I made a bit of progress on this, though it was janky, before realizing that just using deathmatch spawn points (launching the game with deathmatch set to 2) is a good starting point, so that's what I'm doing right now until I can get the other parts of the process working. But I'd be curious to hear if anyone has any thoughts on heuristics for "screenshot that is looking at something interesting", and how to automate this via ACS scripting or something else (I'd be fine hacking the actual source for this purpose if that was easier and/or more effective.)

General thoughts and feedback also quite welcome.

[Tapwave]

I'd subscribe to that, but I wouldn't put this thread in the projects subforum.

[mallo]

I'm have no idea about technical stuff but maybe add delay 1 before screenshot command or something

[TerminusEst13]

I don't think I understand the technical specifications but you had me right at "automated screenshots".

[JPL]

Okay, so I think I've at least proven the concept:

http://wadbot.tumblr.com

Those screenshots are somewhat uninteresting, I suppose most maps aren't classics! :]

Here's what the WADBOT2 python script does so far:
- Follow the "Random File" link on doomworld.con/idgames, looks for a ZIP download link
- Downloads the ZIP, looks for WAD/DEH/PK3 files inside, finds map lumps, picks a random available map
- Opens that map in GZDoom with a special "screenshot mode" config and takes a shot. Also logs player position to the console.
- Nothing on the tumblr/twitter auto-post for now, I manually copied the post text the bot spits out and added it and the screenshot to the post.

A few thoughts / lessons learned so far:
- The "screenshot" and "quit" commands don't run if they're in an autoexec.cfg, even if they're part of an alias. On the one hand, what I'm doing here is about the only practical (to define generously) use I can think of for that; on the other hand it's a weird silent failure case and it caused me some headaches trying to figure out the problem. So maybe it's worth a bug report, if it isn't a desired behavior?
- Deathmatch spawn points are usually merely "okay" as screenshot vantage points. They position the player facing in a cardinal/diagonal direction, so the level lines tend to be orthogonal to the image edges, which looks very static and boring. I think it will be worth the research to try my "warp around randomly looking for a good vantage" approach, though I'm not sure ACS has the power I need for it. One thought is to test a location + POV's subjective "screenshot quality" by checking the total # of sectors visible from that vantage - more usually being better: if you're facing a wall or looking down a boring hallway, you can't see into many sectors, whereas when you're outside or in a courtyard area, you can see more. No idea how to do this, it might involve modifying the GZDoom source - everything about this is a weird ball of hacks anyway, so if it only runs on the machine that hosts the bot, that's okay.
- I need to figure out how to get the URL idgames' "Random File" link resolves to; right now I'm just linking the ZIP download itself because I can't find that, but it's much more preferable to take people to the archive page so they can read the readme, leave a rating, etc.

[IMX]

Really hoping this takes off, friend! Good luck with your project

[JPL]

- Posting via the Tumblr API works now, ridiculously easy thanks to their Python module.
- Links now point to the idgames/ page rather than the direct download.

Of course, there's a problem with the current "random map" selection: idgames returns a random file, but one file may contain a single map while another may contain a 32 map megawad... so the selection process is disproportionately weighted towards single-map WADs, and levels in megaWADs are less likely to come up.
The only way around this I can think of is to actually do a local cache of the entire >12GB idgames archive, build a database of every map in every file, then pick a random entry from that. This is doable! Also I've always liked having a full copy of the archive on a hard drive.

[wildweasel]

Huh, twenty-plus years of /idgames and we've only managed to get to 12 GB of WADs? =P

I really like what you're doing here, JPL. I wonder if WADbot could be used to "index" a local Doom collection for things like Tormentor667's "wad library" idea.

[Gez]

There's also the more open problem of taking "good" screenshots. Initially I tried writing a small ACS library function that warps around to random XYZ coordinates, trying to find a spot that's A) inside the map, B) far enough away from a wall that the screenshot won't just be a wall. I made a bit of progress on this, though it was janky, before realizing that just using deathmatch spawn points (launching the game with deathmatch set to 2) is a good starting point, so that's what I'm doing right now until I can get the other parts of the process working. But I'd be curious to hear if anyone has any thoughts on heuristics for "screenshot that is looking at something interesting", and how to automate this via ACS scripting or something else (I'd be fine hacking the actual source for this purpose if that was easier and/or more effective.)

I think hacking the source code is what would be needed.

Ideas for heuristics of interesting things (that are definitely not possible with just ACS):

1. Check depth. Usually, a close-up shot of a wall is not very interesting. Most of the geometry on the screen should be far away. (Precise values for "most" and "far away" are left as an exercise to the reader.)

2. Check things. If there are a lot of actors visible on the screen, the scene is probably quite interesting; especially if the actors are of varied types. (A room only filled with imps is not as interesting as one with a mix of monsters, plus some torches, pickups, hanging bodies, etc.)

3. Check contrast. If there are varying light levels, the scene is probably more interesting than if everything is flat-lighted to the same value.

4. Check colors. Generate a small thumbnail (like those used for saves) and analyze it for dominant colors, then check if the colors work well together. Not sure there are any good algorithm for that in open source though, and writing it from scratch would require very good grasp of color theory.

As for vantage points: you can warp to player starts, deathmatch starts, teleport destinations, but also important pickups (keys, spheres, non-dropped weapons...) and important monsters (cyberdemons, spiderdemons, any monster type with 300+ hp that is present in less than 10 total on the map...). An idea would be to warp to one of these points, search for another one that has a direct line of sight but is at least some distance away, and once found aim at it. So you'd get pictures like "cyberdemon's view of teleport spot" for example.

[leileilol]

I was trying to refind this thread earlier to post the depth/average+actor/average theory as Gez that's also been in my head but also to brute it from every position and then determine color variance in the screenshot to confirm we have some crazy scene, days ago


so.....gez's idea

[LilWhiteMouse]

I think hacking the source code is what would be needed.

1. Check stuff...

Wouldn't all this be immensely easier if done via a dedicated program looking at the wad data directly? Find a spot/angle/pitch, then pass it on as CVARs written to the INI a custom wad could read to spawn a camera for the screeny? No guess work or sketchy checks, could all be done cleanly and mathematically.

[JPL]

My only thinking using the live game instead of offline analysis is that it's already iterating over the BSP tree... a quick test shows even the "walls drawn" count from stat sight is a useful metric, though far from the best (plus it only prints to screen, not to the log, so extracting that data is tougher w/o modifying source). Offline analysis is tempting though, very systematic... I'll have to try out a few things.

[JPL]

Have poked around in the source code a while, I now have a bit clearer an idea of what I want to try first:
1. Teleport player to center (average of its 3 vertices) of each subsector in the map.
2. At each of these positions, face towards the center (avg position of all subsector vertices) of the sector.
3. Measure "scene complexity" from this PoV: total walls + flats + sprites drawn. I set this number to log to the screen in realtime a la idmypos, and having run around a diversity of levels, the number is generally higher/highest when the scene is more interesting.
4. After having iterated through every subsector in the map, go back to the PoV that had the highest scene complexity and take the screenshot.

I'm still quite lost in the overall structure of the code, and have a fairly poor idea where to start with regard to the first two steps... especially getting at one kind of data (map sectors) from another; I'm not used to the large C/C++ project "maze of includes". If anyone who's more familiar with the code has some good starting points, I'd be grateful. I'm a relatively quick study so I'm not asking anyone to hold my hand :]

[JPL]

Ok, got this pretty much working, and the results are fairly consistently interesting, certainly moreso than using DM spawn points. My code runs after scene render: measuring the scene complexity, comparing it to previous best, then warping to the next point. It runs one subsector per frame (because the renderer has to do its thing for me to have the scene complexity numbers!) which can take quite a while, up past 60 seconds on big maps.

It also looks cool doing its thing:

https://vine.co/v/euVpwbYIJiz
https://vine.co/v/euVljHrnXeA

On maps with VERY low detail it will gravitate towards things like stairwells, as they create more verts than empty rooms, but on such simple (1994-esque) maps a stairwell may be the most interesting feature.

I've queued up some posts on wadbot.tumblr.com for the next few days using this technique, hope yall enjoy!

The code is a pretty brutal hack, if it had any broader utility I'd consider cleaning it up and submitting a pull request to GZDoom, but... yeah.

[Gez]

Ok, got this pretty much working, and the results are fairly consistently interesting, certainly moreso than using DM spawn points. My code runs after scene render: measuring the scene complexity, comparing it to previous best, then warping to the next point. It runs one subsector per frame (because the renderer has to do its thing for me to have the scene complexity numbers!) which can take quite a while, up past 60 seconds on big maps.

How long would it take for ZDCMP2?