1. Dismiss Notice
  2. GOG.com logo

    Thanks to YOUR votes, GOG.com now sells:
    - Sea Dogs - Sea Dogs: Caribbean Tales
    - Sea Dogs: City of Abandoned Ships

    Vote now to add Pirates of the Caribbean to the list!

    Dismiss Notice
  3. Under the Crossbones Podcast

    A Pirate Podcast with Interviews
    Music, Comedy and all things Pirate!

    - Episode Guide - About - Subscribe -
    - Twitter - Facebook - iTunes - Android -
    - Youtube - Fill the Coffers -

    Dismiss Notice
  4. New Horizons logo

    Quick links for PotC: New Horizons
    - Download latest version
    - Wiki - FAQ - Report bugs here
    - ModDB profile

  5. GOF logo

    Quick links for AoP2: Gentlemen of Fortune 2
    - Downloads and info
    - Historical Immersion Supermod
    - ModDB Profile

Dismiss Notice
New to the forum?
Please take a moment to read our Welcome Message and Forum Rules.

Solved Heap Corruption, Random Crashes

Discussion in 'Technical Help' started by amplificar, Aug 28, 2012.

  1. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    Heap corruption connected to the usage of DX8Render ReleaseVertexBuffer() eventually leads to the game crashing.

    This current patch skips releasing the vertex buffer, which hurts game performance, but keeps the game running longer and with fewer graphical anomalies.

    Once I learn more about the cause, there will be an improved patch that wont affect game performance.

    Installing The Patch
    A) Backup your PotC MODULES\battle_interface.dll (move it to the desktop or add ".backup" to it's filename)
    B) Download and decompress EU_BI_bypassFDC0.7z into the MODULES folder

    [EDIT] The previous workaround, registry modification to enhance the fault tolerant heap, was removed.
    [EDIT] Improved heap corruption patch, for US/EU and Build 14.
     

    Attached Files:

    Black Iron Oyster likes this.
  2. Thagarr

    Thagarr Pining for the Fjords! Staff Member Administrator Creative Support Storm Modder News Gatherer Hearts of Oak Donator

    Joined:
    Jan 1, 1970
    Messages:
    5,355
    Gender:
    Male
    Occupation:
    CNC Machine Operator
    Location:
    Steeler Country
    This could very well explain many of the apparently random crashes we have experienced over the years, a lot of those have involved game saves and switching between modes. Some of the other strange crashes we have seen involve CTD's on startup that may very well be related to this memory bug, or the loading of the wrong shaders that you mentioned in the other thread. Some others are caused by audio and video codecs, and the rather strange one of keeping your hard drive defragged. :unsure

    I think part of the problem lies with the developers not having an extensive amount of different testing equipment to try the game out on before it's release. The fact that Bethesda and Disney forced them to release it much earlier than originally planned didn't help either. The open beta tests that have become a trend the past few years sure would have helped tremendously with this game, it's unfortunate that wasn't an option 10 years ago.

    The above registry edit is something that not many gamers would be aware of, or even consider, myself included. Personally, I think Microsoft could have done a much better job of implementing the system registry, the way it is now only confuses and confounds the normal computer user. :modding

    In just the past couple of weeks mate, you have helped us narrow down some of the biggest problems that we have had with this game. Thanks once again for your time and effort, it is very much appreciated! :onya
     
    Black Iron Oyster likes this.
  3. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    Ok, I done did this. :read Now why set compatibility to XP SP2? :pirate07: Also, I have 8GB of ram. If a little bit is good a whole lot is better! :woot What is a big number I can use? This game uses almost no ram at all as near as I can tell. In fact I suspect it goes straight to the virtual memory on the hard drive, which might be the reason for the defrag issue.


    For me the CTDs happen during boarding battles and after the battle when I try to sail away on the world map. So I learned to avoid battles. :boom:

    EDIT: Initial testing is encouraging. Selling cargo to a merchant seems to go faster, repairing ships ditto, and no ctd in a boarding battle with me doing all the things that normally cause a ctd. Back to more testing.....:sail
     
  4. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    XP SP2 spoofs DirectX 9.0c, affects memory management, emulates audio and DLL behavior no longer in Windows Vista/7, sets flags for (allowing?) the fault tolerant heap, and more.

    If it helps and the game runs longer on average, then with 8GB of RAM the highest I'd recommend trying is a value of 400 in both fields. That could quadruple the length of time the game averages between crashes. The value is in hexadecimal and megabytes, so FF = 255MB, 400 = 1024MB. The field in FTH that has DelayFree in it's name will benefit from more memory than the other one. Remember to reboot after changing the registry keys.

    If you have programs installed on your computer that crash more often than two or three times per hour (web browser, other games, etc) then modifying FTH could also affect them - so your web browser might start using up a lot more memory. If you notice your computer's performance slowing down then the values for FTH might be set too aggressively.
     
  5. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    The reason I asked about the XP SP2 compatibility is that I tried all of the compatibility settings last year and got a drop in fps with them. I am getting less fps now than before. There might be a conflict as I have DX9c installed already. It does help POTC run better with Win7.

    Does POTC crash that often for you? o_O None of my programs crashes except POTC, and I know when, so avoid that situation except for today in testing and got no CTDs so far. More testing is needed.
     
  6. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    It crashes pretty often for me, but I'm doing repetitive things for debugging. So normally I'm encouraging the game to crash.

    The FTH doesn't require compatibility mode. It'll make a big stability improvement all by itself. I recommended XP SP2 anyway because most people in the last thread said they have DirectX 11, and I've got DirectX 10.

    The way FTH works is unusual. I used the opportunity earlier to point out what could go wrong for anybody using it. The FTH gets activated automatically, so it can affect a whole slew of programs. If a program crashes more than twice an hour the FTH starts protecting it. After seven days, it resets which programs it's affecting until they start crashing again. If you look at the registry inside FTH is a folder called State, and engine.exe will be listed in there if PotC is being protected. A few rare people might have lots of stuff listed in State, if they have bad RAM, a failing hard drive, overheating, or other serious problems. For them, modifying FTH isn't a good idea. Most people will just have PotC in there and FTH never gets used otherwise.

    Thanks for testing :type1
     
  7. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    I'm not a good tester for this. I hope someone with Intel graphics tries this. I have DX11 with DX9c installed also. I am getting a 10-15 fps hit when in XP SP2 compatibility mode, so do not want to do that. Others may get different results.
    I looked in "state" and it is empty. POTC did crash on me last night while I was wandering around town. That is a completely random crash and was the only one.
     
  8. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    I'll start posting real fixes by this weekend. Adjusting FTH was more trouble than future testing will be. Once there are real patches being used, crashing will be less common - finding events leading up to a crash will be the test. Basically, just playing the game normally.

    This afternoon, I tracked down a source of corruption in battle_interface.dll (0xfdc0). I have to stare at the assembly language a while to figure out what it does and how it's supposed to work. It's very patchable, once I determine a good fix for it.

    I'm sure there's more than one source of heap corruption though. I'll just track them down, one after another.
     
    Black Iron Oyster likes this.
  9. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
  10. Armada

    Armada Sea Dog Staff Member Administrator Project Manager 3D Artist Storm Modder

    Joined:
    Jun 11, 2010
    Messages:
    8,078
    Gender:
    Male
    Occupation:
    Game Developer
    Location:
    Somerset, England
    This sounds very promising, mate! It would be fantastic if you could improve the game's stability, and I'm sure many players will appreciate it.
    Unfortunately it just goes to show how poor Akella's coding was... :rolleyes:
     
    Black Iron Oyster likes this.
  11. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    The main problem I'm looking at happens in battle_interface.dll. It's something that has to do with freeing the memory used by a vertex buffer. On my computer I've set up WinDbg to use Microsoft's symbol database. The operating system libraries show their names now, which is very nice.

    The pieces of the puzzle left to identify are the three obscure battle_interface routines shown in the call stack below. Then I'll be able to determine the code in "PROGRAMS\Battle_Interface\BattleInterface.c" that triggers the problem. Ultimately, the root cause might be in the DLL, not how it's used in game... I tested the obvious thing already, commenting out "DeleteClass(&BattleInterface)" in BattleInterface.c - but that didn't prevent the library code from being triggered (nothing else I tried commenting out in BattleInterface.c has prevented triggering the code either).
    Code:
    (The top of the stack is the last thing being done by the program.)
    (Each line down is an earlier routine, which called the next one up.)
    0012f718 6efd9df2 verifier!VerifierStopMessage+0x1f8
    0012f77c 6efda22a verifier!AVrfpDphReportCorruptedBlock+0x1c2
    0012f7d8 6efda742 verifier!AVrfpDphCheckNormalHeapBlock+0x11a
    0012f7f8 6efd90d3 verifier!AVrfpDphNormalHeapFree+0x22
    0012f81c 77805674 verifier!AVrfDebugPageHeapFree+0xe3
    0012f864 777c7aca ntdll!RtlDebugFreeHeap+0x2f
    0012f958 77792d68 ntdll!RtlpFreeHeap+0x5d
    0012f978 759174d9 ntdll!RtlFreeHeap+0x142
    0012f9c0 6d7ba433 KERNELBASE!LocalFree+0x27
    0012f9cc 6d7caf08 d3d8!MemFree+0x13
    0012f9d8 6d80d1ac d3d8!operator delete+0x18
    0012f9f4 6d80b01b d3d8!CBuffer::~CBuffer+0x3c
    0012fa04 6d816e24 d3d8!CVertexBufferMT::`scalar deleting destructor'+0x2b
    0012fa14 6d80b7c8 d3d8!CBaseObject::ReleaseImpl+0x34
    0012fa24 045c694b d3d8!CMipVolume::Release+0x38
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0012fa3c 020be241 DX8RENDER!DMAInterface+0x477b (this is DX8RENDER!ReleaseVertexBuffer)
    0012fa4c 020bfe40 battle_interface+0xe241
    0012fa68 020bfdc8 battle_interface!DMAInterface+0x940
    0012fa70 004186c0 battle_interface!DMAInterface+0x8c3
    0012fd64 004147db ENGINE+0x186c0
    I created a temporary patch, which should work for everyone (attached to this post is "BI_bypassFDC0.7z"). The patch skips the code in "battle_interface.dll" that leads to memory corruption. I've noticed an improvement in game stability, but this patch trades memory corruption for a potential memory leak instead (kinda like what FTH does, except very constant and precise). Once I figure out what the last routines are, I'll come up with a better solution for it.

    battle_interface.dll file offset 0xFDC0 was 56, changed to C3.

    ----------
    I might've found a second bug in the game. I haven't identified it by name yet, but a missing texture gets used by the game sometimes in sailing mode (in vanilla PotC). This causes accelerated SSE2/MMX texture blitting on my computer to crash when it trys using it. It seems to depend on a random sailing event.

    ----------
    I've started working on naming all the routines in DX8RENDER.dll and battle_interface.dll. Doing all the game modules would take months, so I'm holding off on the rest until I need them or I can find time. When I have a mostly complete list of names, I can turn them into debugging symbols. An example of the benefit to doing this is in the call stack shown above, I determined DX8RENDER!DMAInterface+0x477b is in fact a routine named DX8RENDER!ReleaseVertexBuffer. A few of the names like this one were revealed by looking for exception handling text stored in the DX8RENDER.dll. Most of the routines aren't this easy though, they have to be named by figuring out what the code does or what system libraries it calls.

    ----------
    While digging in files with a hex editor, I found the names of the built-in functions for the game's programming language. I wrote them down as a comprehensive reference manual for myself, but I haven't determined most of the parameters and return values. This is probably already known to modders, but I thought I should share it in case it wasn't (or in case some of the functions were missed).

    I was hoping to find a command like Breakpoint() that would drop into the debugger I'm using. That command is meant for a debugger built into the game though, not a separate one like WinDbg (so it does nothing but freeze the game when used). I'm thinking about trying to modify engine.exe so that Breakpoint() triggers a real application interrupt. That'd help a lot.
     

    Attached Files:

  12. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    It looks like you are making great progress here. :onya It looks like I have some reading to do. :read
     
    Black Iron Oyster likes this.
  13. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    I just got done trying out the bypass. It seems to deliver a framerate hit but also seems to reduce the Graphical Anomalies. So it is a "3 steps forward, 2 steps back" kind of thing.

    EDIT: Just got done playing some more and could not get it ti crash during 3 consecutive boarding battles, even when doing things like looting corpses that always caused a crash in the past. Also, the English galleon, the Revenge, has always caused heavy GA when demasted. In this battle there was almost none at all! :woot There was some when approaching port, but it went away with a quick command to raise and lower sails.
    This is a huge improvement! :cheers
     
    Black Iron Oyster likes this.
  14. amplificar

    amplificar Opcode Assemblar Storm Modder

    Joined:
    Aug 20, 2012
    Messages:
    32
    Gender:
    Male
    Occupation:
    Technician
    That's a huge relief. It shows that I've really found the spot heap corruption begins, and it affects the game as much with mods as without.

    Was the drop in framerate gradually getting worse, or did it stay constant? I didn't notice this on my computer because I'm testing at 640x480 32bit windowed. It must be a side effect of simply disabling the release of vertex buffers. When a better fix is made, this problem might disappear.
     
    Black Iron Oyster likes this.
  15. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    The framerate drop is mainly noticed at sea and on land when many npc's are around. It dropped immediately and has stayed constant or gotten a bit better. :treasure:
     
    Black Iron Oyster likes this.
  16. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    Could this thread be pinned to make it easier to find?
     
  17. Armada

    Armada Sea Dog Staff Member Administrator Project Manager 3D Artist Storm Modder

    Joined:
    Jun 11, 2010
    Messages:
    8,078
    Gender:
    Male
    Occupation:
    Game Developer
    Location:
    Somerset, England
    Ayup. Pinned it be. :)
     
  18. JackSwarrow

    JackSwarrow Sailor Apprentice

    Joined:
    Aug 11, 2011
    Messages:
    52
    I used your battle_interface.dll to test it but then i reverted back the the stock one to compare but i noticed the game no longers works anymore. It loads up fine but when i example buy something in the store it just crashes (XINTERFACE.DLL). So i used the modded dll file again but that did not fix it. Any help will be very welcome.
     
  19. Hylie Pistof

    Hylie Pistof Curmudgeon Staff Member QA Tester Storm Modder

    Joined:
    Jan 15, 2010
    Messages:
    13,748
    Gender:
    Male
    Did your game crash there before you made the change?
     
  20. JackSwarrow

    JackSwarrow Sailor Apprentice

    Joined:
    Aug 11, 2011
    Messages:
    52
    No it did not crash. After reverting back the the default dll file it crashed. Its weird that it only crashes at the store, going to sea otherwise works fine
     

Share This Page