AI Upscaling Textures

[Mazioman]

So I was messing around in GIMP last night improving the down-and-out James Norrington texture to my taste (looks a bit less dead inside now) and an idea came to me when I used GIMP's antialiasing and image enhancement tools: Use AI upscaling to improve nearly every texture in the game. If you don't know what that is, Google can show you a lot of results specific to gaming as a medium where AI-upscaling makes a huge difference. I'd recommend looking at r/GameUpscale and Morrowind Enhanced Textures for some very good examples. Anyway, after looking into it, I don't think I have the CUDA-cores I'd need to actually upscale anything for NH/POTC at present.

That being said, while I think it's a good idea for somebody (potentially me...eventually), There are some caveats and concerns that come to my mind: This would probably work best on land, building, and ship textures. I'm not sure if the character textures are at 512x512 (if memory serves) due to engine limitations or in order to ensure good performance. If the reason is the latter, well, most modern graphics cards should be able to handle 1024x1024 without any issues if we could reliably upscale to that degree and keep quality. If it's the former, the only solution would be to see how it fares on Maelstrom maybe? Also, I'm not sure if TGA files are the best to upscale or if there would be issues brought on by compression when converting back to a tga.tx file.

All of that would be moot anyway due to the biggest hurdle that came to my mind with making something like this: Permissions. Given the 'Perpetual Stew' nature of modding Storm Engine games, with so many contributions from so many people over so many years across so many modpacks, it would probably be impossible to ask every original texture artist if it would be fine to re-release their work after running it through an upscaling program. If the permissions and any potential engine limitations vis a vis texture size/conversion/compression can be worked out, I think this could be an excellent idea at least for keeping the game looking modern and crisp going into 2022 and beyond. From a cursory look, I think the amount of time and work it would take is mostly dependent on checking the upscaling for quality and any visual artifacting—I think the actual conversion can be executed in a program or with a script in batch format so you could conceivably knock out an entire folder in one pass if you had the time and HDD space, but I could be completely misinterpreting what I read about how one method uses Python scripting. All in all, I think this idea has enough merit that I should bring it up and if anyone has any experience with upscaling, they could chime in. Thanks everyone.

P.S. Have a slightly edited James Norrington for your time:

 

[Mazioman]

Did a quick first pass using an online upscaler on the first texture for the Fast Merchantman, here are the results:

Pre-Upscale:

Post-Upscale:

Things to note: The difference in image quality isn't extremely noticeable, but this was a high-quality texture to begin with. If you zoom in very closely, you can see that a lot of the visual artifacting and general noise in the texture is gone and there was a seam in the texture just behind the mainmast that's a little less noticeable. I also think the woodgrain is barely improved in the way it looks, which as a hobbyist carpenter, I love. You could probably chalk up the mediocre result to me using an online tool. For the purposes of this test, I used 4x upscaling to see how far it could go and the end result was an 8k resolution file that sits at 10mb. I also didn't consider hard-drive and RAM limitations when I made my original post, so having everything upscaled to that degree wouldn't be as feasible as I'd like. That being said, I think most textures are fine as-is and only a few old/really rough textures would desperately need this treatment. I wonder too, if using a better program would yield a much better result at a lower resolution. Anyway, I'll leave this here for now and if it picks up traction, I'll help however I can or figure out how to do this properly. If this gets done, in any case: abandon all hope all ye HDDs that enter here.

 

[Mazioman]

Update 2: I lied. Did one more test on a Caravela, the texture was 512 at the start and upscaled to 2k due to limitations with TXConverter (Should be noted that the resolution of the earlier test was also 2k, I didn't notice TXConverter automatically resizes images with a 2k cap for some reason, that might bottleneck this to 2k resolution unless the tool can be fixed to allow larger images), however the results were pretty good in my opinion:

Pre-Upscale:

Post-Upscaling:

I think this is a much more noticeable improvement given that the starting textures were low quality. The lines are a lot more crisp and there's a lot less general blurriness to be had, and I will say it's better in person and that these images don't due the changes enough justice. I'd upload the file here as an example for you all to see for yourselves, but until I know more about the permissions thing, I think I'll hold off. I do think this was a stock(ish) model/texture though, and I can comfortably say I'm on to something here, dear mateys...

 

[KillPhil]

wow this looks indeed very good!

I think my RTX2080 would be capable of upscaling everything .. are there plugins for image editing software? This might come in very handy for our Maelstrom project, since those kinda crappy vanilla textures are a lot more noticable with an improved rendering system :/

 

[Mazioman]

wow this looks indeed very good!

I think my RTX2080 would be capable of upscaling everything .. are there plugins for image editing software? This might come in very handy for our Maelstrom project, since those kinda crappy vanilla textures are a lot more noticable with an improved rendering system :/

Thank you, I'd take credit if it wasn't a literal robot that did it for me and I actually play on Maelstrom and know exactly what you mean about the improved rendering fidelity being a double-edged sword where textures are involved.

In regards to your question, I think it would depend on which software is being used. From what I read, if anyone knows Python (failed that course both times I took it, so maybe not me), you can use the source code from a method of upscaling called ESRGAN in a pretty basic Python program to upscale things in batches and a lot of methods for upscaling have API's available, which might be what you're looking for? That said, I think the basic process is: Start with .tga.tx--->convert to .tga--->convert to .png--->upscaling happens--->get high-resolution .png--->convert back to .tga--->convert back to .tga.tx

That's a lot of conversions and mouse clicks, but you get the idea. I asked the upscaling subreddit about if ESRGAN could work with .tga files natively which would cut down on the number of conversions considerably. I'm also still wondering about the TXConverter tool and why there's a limit to texture size that it can convert. Is that because the game would stretch anything beyond a 2k texture? That would put a hard cap on the general resolution of this theoretical texture pack, but ideally you'd have a tiered system of releases (2k, 4k, 8k for the madlads) that would satisfy everyone's needs depending on their hardware. I do think this is the best solution for modernizing the old textures that never caught up to advances in hardware, specifically anything below 2k resolution until the TXConverter thing is figured out. Further, the more I read about upscaling, the more I think it sounds almost like art restoration in a way, and that's really cool.

Here are some info I found:
ESRGAN site for general info.
ESRGAN Wiki.
ESRGAN Windows Installation Guide.
ESRGAN use tutorial and links for GUI programs for usability.

I hope those are helpful for you or anyone else that becomes interested, like I said, I'd love to help as much as I can if this picks up steam. Lastly, I saw that post you made showcasing the weather in the new version of NH for Maelstrom and it's frankly jaw-dropping how you made an almost 20 year-old game look like that with only an updated engine. Amazing, and can't wait to see it in-game myself.

 

[Mazioman]

I am a madman. I am a fool. I have to be at work in four hours. But ladies, gents, and mateys, look what the upscaling does to the islands:

Look how rocky those rocks look! I'm sleep deprived!

 

[Grey Roger]

Note that upscaling x 2, e.g. 512x512 to 1024x1024, makes the file four times larger.

You may be able to improve textures by using TX Convertor to convert to .tga, editing away any artifacts, then converting back using a different setting. A lot of files are converted using DXT1, DTX3 or DXT5 compression, which can introduce artifacts - DXT1 is the most vicious compression and the one most likely to introduce them. If the file does not use the alpha channel to make areas transparent, you can use R5G6B5 instead. For textures which do need the alpha channel, use A1R5G5B5.

 

[KillPhil]

well 4 times larger than a few 100 kb .. I take that for the sake of THAT improved quality!

@Mazioman can you maybe set up a public Google Drive folder where you can share me / us the textures you already upscaled (PM me the link)? I will check upscaling AI myself next weekend

 

[Mirsaneli]

Awesome work and awesome idea mate! I would like to give it a try, but I'm kinda Photoshop guy. Haven't used gimp very much. Do you have a tutorial or something on how it's done?

 

[Mazioman]

Note that upscaling x 2, e.g. 512x512 to 1024x1024, makes the file four times larger.

You may be able to improve textures by using TX Convertor to convert to .tga, editing away any artifacts, then converting back using a different setting. A lot of files are converted using DXT1, DTX3 or DXT5 compression, which can introduce artifacts - DXT1 is the most vicious compression and the one most likely to introduce them. If the file does not use the alpha channel to make areas transparent, you can use R5G6B5 instead. For textures which do need the alpha channel, use A1R5G5B5.

Ahh, and this is unfortunately where it gets beyond me. I'm no texture artist by any stretch, I mainly just piddle around if it's something small I can fix.

well 4 times larger than a few 100 kb .. I take that for the sake of THAT improved quality!

@Mazioman can you maybe set up a public Google Drive folder where you can share me / us the textures you already upscaled (PM me the link)? I will check upscaling AI myself next weekend

If I did this correctly, here's a link to the Drive: The PUP (Drive Link)
I tentatively christen this project: The P.U.P. (Pirate Upscaling Project, but feel free to change that). I know I'm the one that had the idea but this would ideally be a community effort as I don't have the time to put into a project like this due to adjusting to my first 9-5 office job which is...yes...but seriously, beyond simply having the idea and making a nice little proof of concept, I don't deserve any credit, it's the people who have been doing this for over a decade and putting in the work to keep the best Age of Sail simulator out there going that deserve every bit of praise, and that includes all of you guys in the thread currently. Going back to my art restoration analogy, a Rembrandt will always be a Rembrandt no matter if it's restored or original. By extension, a Rembrandt will always be a Rembrandt no matter who does the restoration; be it man or machine. Anyway, I digress...

Awesome work and awesome idea mate! I would like to give it a try, but I'm kinda Photoshop guy. Haven't used gimp very much. Do you have a tutorial or something on how it's done?

Best I can tell, AI-upscaling is usually a basic program written in Python using a library built from other AI's learning how to best enhance images by attempting to enhance millions of images until the AI teaches itself to get it right. You'd use one of these libraries with the source code for the method of your choice (ESRGAN seems to be best for games) and then either write a program to manually convert entire folders, download one of the GUI interfaces that can make it easier, or use an API with an existing program like Photoshop. That said, I did find this from a cursory google search and it operates on the exact same principle, Adobe just calls it super resolution-scaling instead of AI-upscaling: Super Resolution Article (Link). I think the relevant function can be found in Adobe Camera Raw--not sure if it supports batch upscaling though, but this could definitely be a great starting point.

Anyway, I'm glad you're all liking the idea and as a warning: DO NOT GO DOWN THE AI RABBIT-HOLE WHEN RESEARCHING THIS, YOU WILL LOSE SLEEP OVER OUR IMPENDING OVERLORDS .

 

[Mazioman]

I should note that a lot of the folders in the Drive are currently empty and intended to be filled with the corresponding textures. The folder structure for when this is eventually ready for a zippable format has examples in the 'Ships' and 'Islands and Environments' folders as they're currently the only ones I've messed around with. I think the Drive is a good idea because you could conceivably release the upscaled textures in piecemeal format. This would let whoever is interested download only the upscaled (and much larger textures) that they want/deem necessary.