What would it take to fix FF8's backgrounds (field files)?

  • Thread starter Thread starter Omzy
  • Start date Start date
Status
Not open for further replies.
Maybe this graphic will help. The blocks cycle through the VRAM in the order of the texture file. At each point, the palettes cycle over the image, changing the colors in VRAM until no more colors are needed, then the cycle continues with the next blocks. The palettes needed for the block in the first position are prioritized, that's the point of keeping the next block in the VRAM next state, because it hasn't finished all of its palettes yet, just the ones in common with the block before it.

Keep in mind, this is a theory based on a very small amount of data so far.

texblocks.png
 
Last edited:
PSX textures are always 256x256 and on a real GPU there would be no reduction in performance if you were to access them on 128 pixel boundaries. From your point of view (I am assuming you're getting this from PIX) you will only see which textures need to be constructed to emulate the GPU commands from the game. There is no "good" reason for the duplication you're seeing other than that its necessary to properly emulate the GPU. This is one of the big issues with replacing textures in FF8, field backgrounds are used as if they were 128x256 but when they're uploaded to a texture the next 128x256 pixels from the VRAM will be uploaded with it because that's just how the GPU works.
 
There is no "good" reason for the duplication you're seeing other than that its necessary to properly emulate the GPU. This is one of the big issues with replacing textures in FF8, field backgrounds are used as if they were 128x256 but when they're uploaded to a texture the next 128x256 pixels from the VRAM will be uploaded with it because that's just how the GPU works.
I understand, so the second block in memory I'm referring to doesn't really do anything, its just there because the commands that load the first block automatically load whatever's next for no good reason. So the palettes cycle on the first block, then the next one loads, etc.
 
If I were to implement modpath/replaceable textures for FF8 the first step would be to rewrite parts of or maybe the entire GPU emulation. Not because it is flawed in any way (although it isn't very efficient) but because then I could start rewriting all the rendering code to use explicit textures instead of just VRAM coordinates.

It would be a massive undertaking. Good luck ::)
 
I'm ready to test my new dictionary of hashes in-game but I need to gain a bit more d3d9 knowledge first. I'm planning on making use of a d3d9 interceptor DLL located here: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html. I'm trying to find out which files I absolutely need and which ones I can change to test texture replacement in-game. Basically I need to capture the SetTexture calls and replace them with any test texture. Any help/advice would be appreciated to help me go faster  :-D

Here's the hash dictionary using the algorithm described above. It adds a numerical suffix to the filenames for each 128x256 block of the texture. There are a total of 6245 blocks in the game.
FF8 Field Hash Dictionary

Edit (note):
http://www.gamedev.net/topic/178004-idirect3dtexture9-and-manipulation-of-single-pixels/

Update (preliminary):
I have verified that my hash function has been successful at identifying my first test case, bghall1, from the VRAM. At first I was a little worried, because the hash value showed up 3 times, but those were bghall1, bghall1a, bghall1b, which are all identical textures. More to come when I have time!

Update2:
I've got the game successfully running the hash function on every texture that is called by D3D. First, I load my hash map from a file of all the known textures (described above). Then, I run the hash function whenever a SetTexture call is made. I then compare that hash value against the hash values in the hash map to find the closest match (the texture with the least different bits of comparison). I've done a test run from bghall1 (first save point) to the front gates of balamb garden and I've recorded these matches in a file to see if the correct textures were found. Here is that file, with a few annotations thrown in regarding where I was at each point in time: http://pastebin.com/9UpAynEd

Explanation of the results--
line structure: filename_blocknum,mismatchedbits,thistexturehashval,closestmatchedtexturehashval

I've only recorded 1/100 texture calls in the results because the file would be too long to navigate if I didn't. Whenever a background is supposed to be set, the hash function correctly identifies the texture and spits out the file name. However, whenever a texture with mostly black or monotone pixels is called, like in intro screens, and some other screens, including some animations that have a lot of black but a few spots here and there like lights, the hash function fails, usually identifying another texture file that has mostly black or monotone pixels. Additionally, whenever a character sprite texture is called, the results show either NO_MATCH or may sometimes identify the wrong texture as a match.

Keep in mind, these are results of a completely untested hash function that has a ton of room for improvement. The number of pixels sampled, number of comparisons between those pixels, randomness of the pixels, and thresholds for matches and mismatches can be varied. Also, secondary hash functions can be used on textures that are mostly black, for example, and eventually special cases can be worked on separately.

Next step: test texture replacement in D3D

Update 3:
Successfully replaced textures in-game with a simple 256x256 fire texture. This slows the system down tremendously if the texture file is loaded from the disk every time a SetTexture is called. This is because SetTexture is called roughly 150 times per frame to render all the various parts of the scene. Since all of the parts of the scene, except the character and object sprites, are loaded from a single field texture file, this 1664x256 (or 1536x256) texture file should be loaded only on the first texture call of the scene and then referenced in memory for each SetTexture call. This should drastically improve framerate and reduce errors in hashing during the other 149 SetTexture calls in the scene to 13/6245 = 0.2% of the original error rate. This all hinges on the ability to detect the correct texture on the first call during the frame.

Update 4: I have now successfully replaced textures in-game with their appropriate textures, using the hash method. It is not perfect, but it reliable chooses the correct texture > 90% of the time so far. I've expanded the pixel selection from 16 to 64 pixels to make it more reliable. It needs some optimization to avoid lag and some other fixes, but I really need to work through the next huge hurdle: allowing resized textures in-game. This, I fear, will be very difficult, since there are many operations performed on the passed-in textures in order to render them to the frame buffer. These operations include drawing all of the geometry of characters/objects, etc, and I'm so far unsure how to resize only the field textures yet keep everything else unaffected. I'll be thinking on that...
 
Last edited:
Just wanted to update you all on my current progress. I've been able to resize a field file to 4x and get it working in-game. All of the animations and lighting effects seem to work. I've been working rather feverishly on this, more than I should readily admit.

My previous hash algorithm caused a lot of lag every time a texture was loaded since it was searching the entire hash. The new hash technique I'm working on should produce much more reliable hashing that is hundreds of times faster (17 iterations to find the texture vs 6256 iterations per texture). This represents the difference between an algorithm that searches linearly through a large data set and one that searches with a binary search algorithm. It should eliminate that lag altogether.

This method should work on all field backgrounds once I get polished, but also has the ability to work for all other texture-like files in the game including character/monster sprites, battle scenes, cards, menus, etc. In order to get it to work for those other texture types, I need to have directories full of those images extracted from the game files. I can see that myst6re's deling works to view most of those files, but extracting the textures from them is something I haven't tried. I'll cross that bridge when I come to it.

Stay tuned for a release within the next couple of weeks! (this will not include a resize pack, that is the next project)
 
Last edited:
Can this include better world map textures as well? The world map is easily the worst looking part of the game.
 
Covarr read my mind, it was a question I had since you started! The world map is muddy when put next the ps1 version.
So that issue being resolved would be amazing!
 
That's great news! Will this new algorithm work on the 3D models' textures, or just the 2D textures in-game?
 
The way this all works is that whenever the game wants to display ANY texture, it calls a CreateTexture command first and then does the processing it needs to draw the texture onto triangles on the screen. Even the field backgrounds are drawn onto strips of triangles that produce a flat rectangle. So far I know that these commands are called when field backgrounds, character sprites (the character models you speak of), object sprites like save points, character portraits, menus with text, and cards are drawn. By extension, I believe they are also called for the world map, battle scenes and spell effects, and everything else, but I haven't proven those because I haven't recorded PIX data on those scenarios yet. So, if a CreateTexture command is called, then this method can replace that texture.

All that I have to know in order to replace a texture is what it looks like in memory. So far, the field backgrounds look exactly like the mim file contents in memory (with various palettes applied). I've already done the legwork and have written algorithms to display every necessary permutation with all necessary palettes for the field files. Character and object sprite textures look just like they do in myst6re's deling program when you open a field file and go to 3d models section. I don't have any experience with those files yet. So the bottom line here is that if any of you guys want non-field textures to work as well, I'll need to get copies of all of those images so that I can run my hash algorithm on them.

Generating these images is not hard--in fact myst6re has generated most of them by reading the ff8 files in her deling program. Its a matter of writing little programs that read the ff8 files and output them to image formats like PNG. In order to see what they look like in memory, you can download the DirectX SDK (June 2010) and use the PIX tool to record all the directx calls. When viewing a PixRUN file, you can click the texture references on all the SetTexture calls and it will display what the textures look like in memory. My guess is that most of them look exactly like they do in deling, because field files are the most convoluted files to display and tex files, etc, are very simple.

If anyone wants to help speed up the progress on those other types of textures, feel free to write python scripts that output images from game files or send me rar's of images. For now, I'm focusing on field files. This process actually becomes more reliable (from 98%->100%) once I have hash values for EVERY texture in the game. As it is now, it is theoretically possible for my hash algorithm to place a field texture over a character sprite, for instance, if they coincidentally have the same hash value. Rare, but possible.
 
Last edited:
If only Myst3re could put a "mass import" feature in Deling to import a directory of field subfiles, rather than having to do them individually by hand. I'm going to need that anyway for my voiceover project.

Pretty please <3

Wrong thread. I'm guessing omzy's method doesn't actually modify the field backgrounds directly but on runtime with a thread interceptor.
 
Last edited:
Nearing the finish line. Here's a proof of concept release that is very preliminary. It includes the textures starting from the first save point in Balamb Garden all the way to the front gate. Please do read the readme before using it. Let me know if you're unable to run it for any reason--I haven't tested it on any other machines so there very well may be a problem or two I didn't consider regarding portability.

Tonberry PRE-RELEASE ALPHA Readme
by Omzy
Born at https://www.ff7catalog.com/threads/10400/

Credits to Matthew Fisher at Stanford University for developing the original d3d9Callback interceptor: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html

*This only works for the Steam FF8 release which uses DirectX
1) Drop the contents of this folder into your FF8 directory that includes your FF8_Launcher.exe
2) Run your launcher
3) Play!

Notes:
*) The textures bundled with this pre-release were improperly resized with photoshop in 30 seconds, do not use fractals (only bilinear), so are meant only as a proof of concept, not as replacers. You will notice there are lines that make the background look like bricks due to the poor quality resize
*) This hasn't been tested on low-end systems, only 1 medium-high end modern system running Windows 7 64-bit, there may be some barely perceptible lag but it isn't that noticeable but if you have a lower-end machine, it may suffer considerably, similarly to running an emulator
*) Anytime a switch/door/trigger occurs in any scene, there is a lag associated with the texture 'pop-in' when it loads. This may be addressed in further releases
*) The hashing algorithm is unfinished and will exclude some ~150 of the ~13000 texture pages used in the game. This will be noticed as parts of the background being high-res and other parts being low-res
*) Only field backgrounds are replaced at this point, but this will be expanded in the future to other texture types
*) Included is a WrapperParameters.txt which has a BaseDirectory that should equal the location of your FF8 directory. The default is "C:\Program Files (x86)\Final Fantasy VIII"
Download : Tonberry Pre-Release Alpha

Edit: Also, is there a project named Tonberry already? I looked but didn't find anything obvious, curious since its such an iconic ff monster.

***Edit2: I had a friend test my first upload and it was missing a few things so I've fixed that and re-uploaded for those of you who downloaded earlier. Don't forget to change your WrapperParameters.txt if you need to!
 
Last edited:
Impressive, as usual Omzy!  :o

Now I just have to buy myself a copy of FF8.  ;)

UPDATE: Just bought it on Steam
 
Last edited:
Any screens yet?
Since it takes a lot of time and effort to write Photoshop scripts to do what FacePalmer did, I don't have any fractalized backgrounds made up yet, if that's what you're wondering. I'm about to release my Tonberry Beta, though, which is the DLL that supports external textures and should pave the way for a beautified game experience. Check the Tools section in a bit.
 
Excellent! Thank you Omzy!!! Looking forward for any in game screens as well!!!
 
Status
Not open for further replies.
Back
Top