Need Help Keys (?) keep resetting

[Felkvir]

If you have not yet done so, make sure you restore the original files from the "backup" folder to replace the ones I gave you during testing.

Also, I'm curious, what version of Windows are you running and in what language?

If this 'enter' key problem is indeed new to your game, try ditching the "options" file again. It may have gotten corrupted.

Thought I posted a reply already but the forums are being weird so whatever. I can't seem to replicate the issues any longer, idk what is up with that but I'll just keep playing until I notice something again. I'm not entirely convinced its not random at this point... I use windows 10 and norwegian, does that affect anything in the game?

 

[Cerez]

I use windows 10 and norwegian, does that affect anything in the game?

The ISO-8859-1 character encoding (ASCII) is known to have compatibility issues across different computers from different countries (different default languages), with certain language-specific special characters getting garbled. This is the reason Unicode (UTF-8) was invented in the first place. So, while I can't confirm yet, it's possible that your Norwegian-set ASCII is causing problems for the engine to interpret certain special characters in the code (misinterpreting them), and potentially causing the "options" file to get corrupted. This can lead to things like certain keys getting switched around -- but in this case the change should be permanent, not randomly occurring.

What you could try is to set your Windows default language to English (US), restart the computer, and see if the bugs in the game still occur. That would help confirm if this is ASCII/language related.

It is also possible, as Chez has pointed out, that the engine/interpreter itself has a code processing bug:

It's an engine problem where sometimes the null terminator for the strings gets overwritten by a non-null and now the convention of reading strings expecting a null terminator, and not finding one until it overreaches into other non-relevant memory/garbage corrupts it. You can't solve it with anything in the scripts.

I'm not sure what all this really means. @ChezJfrey would need to clarify this. Why is the null terminator for the strings getting overwritten, and what "non-relevant memory/garbage" is the engine overreaching into?

 

[Felkvir]

it's possible that your Norwegian-set ASCII is causing problems for the engine to interpret certain special characters in the code (misinterpreting them), and potentially causing the "options" file to get corrupted. This can lead to things like certain keys getting switched around -- but in this case the change should be permanent, not randomly occurring.

What you could try is to set your Windows default language to English (US), restart the computer, and see if the bugs in the game still occur. That would help confirm if this is ASCII/language related.

Ah I see, I might try that. Do you happen to know also why some battles I get this white texture or something spazzing out on the screen? https://i.gyazo.com/f77cee300141fe9f8ddf6d743871c313.png it gets kind of annoying

 

[ChezJfrey]

The ISO-8859-1 character encoding (ASCII) is known to have compatibility issues across different computers from different countries (different default languages), with certain language-specific special characters getting garbled. This is the reason Unicode (UTF-8) was invented in the first place.

Correct on UTF-8. Unfortunate this inability to decipher exists within the engine, but it is what it is. I dug up the old conversation I had and it had to do with the CT Historical Immersion and the é character. The differences in codepages can also be a problem for characters that reside outside those consistent, first 128 characters. The é in Windows-1252 is code 233 and is one byte. That 233 is not the same character across different codepages, so you don't get the same result when displayed on-screen. And in in UTF-8, 233 is comprised of 4 bytes, but since this source code always does any sort of character string comparisons for equality, assuming each char within a string is 1 byte, then we also run into problems when determining stringa == stringb.

Since so much of the code presumes each char equals one byte, the only tenable solution was to convert any encountered UTF-8 to the engine's default codepage. But during the course of my attempt, I then broke Russian language, because the game's codepage is different, and there is a Russian modding team using it right now. Given the immensely scattered code doing ReadFile calls and no central handler, that is when I did that quick hack and made an option for the Font.ini file that if flagged, will always assume UTF-8 will convert to our base codepage, because there can be no consistency for what country, codepage, or whether any of the files were saved as UTF-8, ISO-?????, Windows-1251, Windows-1252 and trying to handle and convert on the fly, given the thousands of different ReadFile calls, scatter across many different modules where no initial state of what file was actually in process and what codepage it might be was too much. So for now, this flag I implemented was small, quick and minor in that only the Font display method, when passed a string value for output to screen will be converted, just one way, if my flag is true, and for the rest of us, including the Russians, they can omit the flag and it will operate as it always has for those mapped screen font displays.

It is also possible, as Chez has pointed out, that the engine/interpreter itself has a code processing bug:

I'm not sure what all this really means. @ChezJfrey would need to clarify this. Why is the null terminator for the strings getting overwritten, and what "non-relevant memory/garbage" is the engine overreaching into?

Why? Because a bug. This source heavily relies on old char string functions like strcpy, strlen. What often happens, is that a string will pass somewhere, and the recipient method will want a copy, so it will want to keep a copy. So it will get the length, size_t len = strlen(x). Then allocate enough memory, char *y = new char[len]...but we must keep in mind that the memory allocated must contain one extra character for null terminating strings, because that is the standard that something like strlen relies on; it will iterate each character until it reaches null and that's how it knows the string is done and how many chars exist. So that allocation is actually supposed to be new char[len+1]. Then when copying, there is enough space to place a null. In the case of the options, there is a case where that extra null was not provided for. Then, what happens is that string without a null, sits in memory with say 2 bytes x = "ab", with no null. It gets passed along, then somewhere else strlen(x) and when it starts to iterate, it finds 'a' count 1, 'b' count 2, but now no null, so it keeps going to whatever the adjacent memory, byte by byte until it (hopefully) encounters a byte with a zero...null. That is the new count 58, because it read through a bunch of irrelevant, "garbage" memory...it could presumably be very valid memory, being used by other allocations, but "garbage" for the purposes that it has no relevance to our char string pointer x...overreaching what we intended. Later, it asks to copy string x somewhere else, again, but now it copies our string x that was supposed to be 2 chars "ab", as 58 characters, "abíå íàøåë ëó÷øåãî, íî ó íàñ ÃÃ âñåãäà èìååò ýòî ÈÄ, ðàáîòàòü áóäåò..............." you get the idea.

 

[Cerez]

Ah I see, I might try that. Do you happen to know also why some battles I get this white texture or something spazzing out on the screen? https://i.gyazo.com/f77cee300141fe9f8ddf6d743871c313.png it gets kind of annoying

This looks like a graphics card support issue. Are you running the game with an Intel graphics card, by any chance? (The engine is known to have compatibility issues with Intel graphics cards.)

By the chance that you have a dual graphics card, try switching the game to run using the dedicated AMD/NVIDIA graphics card instead of the integrated Intel graphics card (which isn't really designed for video games).

 

[Cerez]

Correct on UTF-8. Unfortunate this inability to decipher exists within the engine, but it is what it is.

Indeed, but it's a really puzzling choice from Akella. UTF-8 was readily available and used/adopted for many years at the time they were developing the very first version of the engine. It just doesn't make sense to go with a multi-language problematic encoding for a Russian game intended for an international release, and using multiple languages.

Since so much of the code presumes each char equals one byte, the only tenable solution was to convert any encountered UTF-8 to the engine's default codepage. But during the course of my attempt, I then broke Russian language, because the game's codepage is different, and there is a Russian modding team using it right now. Given the immensely scattered code doing ReadFile calls and no central handler, that is when I did that quick hack and made an option for the Font.ini file that if flagged, will always assume UTF-8 will convert to our base codepage, because there can be no consistency for what country, codepage, or whether any of the files were saved as UTF-8, ISO-?????, Windows-1251, Windows-1252 and trying to handle and convert on the fly, given the thousands of different ReadFile calls, scatter across many different modules where no initial state of what file was actually in process and what codepage it might be was too much. So for now, this flag I implemented was small, quick and minor in that only the Font display method, when passed a string value for output to screen will be converted, just one way, if my flag is true, and for the rest of us, including the Russians, they can omit the flag and it will operate as it always has for those mapped screen font displays.

Why do we not change our production method (within the NH project at least) to always use the game's default ISO-8859-1 encoding for all files we modify and create?

This means that we would need to go through, convert, and clean up existing files, but we would also eliminate many bugs and inconsistencies along the way.

If every text document was in the ISO-8859-1 encoding, and we didn't deviate from this standard when coding, wouldn't that solve this problem for the most part, or altogether?

I've learnt throughout the years that there is no automated solution that works more reliably than true consistency in neat, manual work. If we didn't allow documents coded in any other encoding into the final build, with a simple review process, that would encourage everyone to work with ISO-8859-1, to find solutions that are compatible with it, and eliminate these problems.

After all, we are working with an older engine here, and we should be catering to its limitations and specifications.

Why? Because a bug. This source heavily relies on old char string functions like strcpy, strlen. What often happens, is that a string will pass somewhere, and the recipient method will want a copy, so it will want to keep a copy. So it will get the length, size_t len = strlen(x). Then allocate enough memory, char *y = new char[len]...but we must keep in mind that the memory allocated must contain one extra character for null terminating strings, because that is the standard that something like strlen relies on; it will iterate each character until it reaches null and that's how it knows the string is done and how many chars exist. So that allocation is actually supposed to be new char[len+1]. Then when copying, there is enough space to place a null. In the case of the options, there is a case where that extra null was not provided for. Then, what happens is that string without a null, sits in memory with say 2 bytes x = "ab", with no null. It gets passed along, then somewhere else strlen(x) and when it starts to iterate, it finds 'a' count 1, 'b' count 2, but now no null, so it keeps going to whatever the adjacent memory, byte by byte until it (hopefully) encounters a byte with a zero...null. That is the new count 58, because it read through a bunch of irrelevant, "garbage" memory...it could presumably be very valid memory, being used by other allocations, but "garbage" for the purposes that it has no relevance to our char string pointer x...overreaching what we intended. Later, it asks to copy string x somewhere else, again, but now it copies our string x that was supposed to be 2 chars "ab", as 58 characters, "abíå íàøåë ëó÷øåãî, íî ó íàñ ÃÃ âñåãäà èìååò ýòî ÈÄ, ðàáîòàòü áóäåò..............." you get the idea.

Ouch, that's nasty, and an easy coding oversight to make. Thanks for explaining it, that makes perfect sense. And there's no way to fixing this without recompiling the engine from the source code. I take it that this is exclusive to POTC, and that it was fixed in later releases of the engine?

Any chance you can copy-paste that particular instance where this bug occurs in the engine code? I'm interested to see what in particular it is copying/processing.

 

[ChezJfrey]

If every text document was in the ISO-8859-1 encoding, and we didn't deviate from this standard when coding, wouldn't that solve this problem for the most part, or altogether?

Yes, that would be a good choice and if remains consistent across files where strings are consumed by the engine to be displayed, used in the controls mapping, compared, then there should not be any problems. I believe this is why the Russian mods work; the files are all saved the same format.

I take it that this is exclusive to POTC, and that it was fixed in later releases of the engine?

No, this source started as 2.8, so it exists across all of them.

Any chance you can copy-paste that particular instance where this bug occurs in the engine code? I'm interested to see what in particular it is copying/processing.

Not likely. There were several long-standing problems I remembered being discussed quite often and I researched them in the code a long time ago and fixed them. The "options" problem was one, so was the battle_interface crash where someone here correctly identified it was the memory delete causing the crash (they must've been well-versed in stepping through asm, because that's how they figured it out...kudos...I can't follow asm that well myself). That battle_interface problem was similar in that memory was allocated to a bare pointer, then sometimes an improper index would get assigned/modified (ptr[index] = y, where index is past the allocation point), then upon delete of that pointer mem, access violation. My point being that I researched and did this years ago and don't really remember exactly where the options problem code sits right now. I'm not bored enough to spend time sifting through in debug mode, while game is reading the options file, essentially to "figure it out" again mode, just to find where and what I changed. I just vividly remember the options problem was missing null terminator in a string allocation.

ETA: Oh, and the save file corruptions! And the infamous GOF mast crashes too...well, there are more, but the famous ones were the first things I tracked down.

 

[Cerez]

For the New Horizons team's coding/development environment, I can heartily recommend Notepad++. It's free and open-source, available to everyone using Windows, and professional enough to allow for precise formatting, as well as flexible and modern enough for to allow for advanced coding tools and practices.

The ISO-8859-1 encoding can be specified here:

And existing text documents can be converted to ISO-8859-1 using the "Convert to ANSI" option in the menu, after selecting the correct "Character set".

For Linux and Mac, Geany provides a very similar development environment, with the same capabilities. (Also free and open-source.)

 

[ChezJfrey]

@Cerez, since you are so mystified whey they did not conform to UTF-8, since it had already been around for quite some time by the time this game was developed, this might make you feel better...

The standard template libraries for strings, array containers, and allocator objects had been around since the mid 90s, yet the source for this had its own, home-rolled allocator object, string class and array container class; they made their own. And the largest problem with crashes and access violations in the game, are directly due to their flawed allocator object. Nobody in their right mind, home-rolls their own, unless they are very adept and must to perform very special operations; nearly everyone is better off using the STL.

I suspect much of this code was developed long ago, and some was adapted and changed, still prior to the STL, when you had to build your own, and so remains a mish-mash of possibly old C code practices, some C++ code (home-rolled containers and such, pre STL days), etc. Some of the char string allocations got "upgraded" to their own crappy class, but most of it was still the old-style new char[], strcpy methods. But they literally had not a single case of using anything from STL, anywhere, yet they would have been far better off, since all they wanted was std::string and std::vector. So, I removed their entire allocator, string, array and hash class and replaced them all with STL. But, in many places, there still exists the strcpy stuff...it's the most prevalent and I have not gotten all that removed.

 

[Pieter Boelen]

Thought I posted a reply already but the forums are being weird so whatever. I can't seem to replicate the issues any longer, idk what is up with that but I'll just keep playing until I notice something again. I'm not entirely convinced its not random at this point... I use windows 10 and norwegian, does that affect anything in the game?

It shouldn't affect anything. In theory.
But computers are weird things. You never do know.

(Or at least... I don't. )

 

[Felkvir]

This looks like a graphics card support issue. Are you running the game with an Intel graphics card, by any chance? (The engine is known to have compatibility issues with Intel graphics cards.)

By the chance that you have a dual graphics card, try switching the game to run using the dedicated AMD/NVIDIA graphics card instead of the integrated Intel graphics card (which isn't really designed for video games).

Nope, I don't use dual graphics. I use a NVIDIA graphics card.

 

[Cerez]

Nope, I don't use dual graphics. I use a NVIDIA graphics card.

What you're seeing in that gray square in certain 2D user interface elements refusing to render. This could be because:

1. They are not from the game, but some overlay graphics from your Windows desktop environment -- of some application running in the background and trying to force its window/UI to float above the game.

2. Your graphics card has compatibility issues with rendering some in-game user interface elements due to no longer supporting certain shaders or overlay rendering methods. If this happened in the stock POTC, it could be a sign that the game is just too old for your computer, as stock POTC has not received an update for more than a decade.

One thing you can try to fix this is to install dgVoodoo 2 for the game. This is a user-made software/patch that restores earlier DirectX API support to Windows and newer graphics cards that have dropped those APIs (rendering 'commands') since, expanding their backwards compatibility.