[PC] External Texture Support - Tonberry: Enhanced (2.04)

  • Thread starter Thread starter JeMaCheHi
  • Start date Start date
Status
Not open for further replies.
One MAJOR issue regarding hashcodes.

Anyone adding hashes to their own files should check if the hashcode has been used already (which would cause a collision).  The way I do this is compile all of the mods hashes into one "masterhashmap.csv" file that I only use as a test file.  I then search my code against the other codes in the master hash file before making them official. 

This might seem tedious, and it is.  This is just one of the reasons why we need a new hashing algorithm so badly.  I have to go back and do this to a few of my own files, as I had forgotten to do so about halfway through my summons and horizonpack mods.
 
Tripod, SeeD Reborn etc. seem to work now with Tonberry Enhanced 1.6 and your new hashmaps! No more 2/3 backgounds or anything.

I put all of them into the "C:\Final Fantasy VIII\tonberry\hashmap" folder and the old "collisions.csv", "hash1map.csv", "hash2map.csv" and "objmap.csv" files from "Hashmap 1.3", but replaced the "objmap.csv" with the one needed for the Character models from FatedCourage: https://www.ff7catalog.com/posts/207838/ and your GF codes with the upscaled GF textures from rufoos: https://www.ff7catalog.com/posts/207843/

I pasted the GF hashcodes into the "objmap.csv", is that right? Because there are also the character codes.
 
Tripod, SeeD Reborn etc. seem to work now with Tonberry Enhanced 1.6 and your new hashmaps! No more 2/3 backgounds or anything.

I put all of them into the "C:\Final Fantasy VIII\tonberry\hashmap" folder and the old "collisions.csv", "hash1map.csv", "hash2map.csv" and "objmap.csv" files from "Hashmap 1.3", but replaced the "objmap.csv" with the one needed for the Character models from FatedCourage: https://www.ff7catalog.com/posts/207838/ and your GF codes with the upscaled GF textures from rufoos: https://www.ff7catalog.com/posts/207843/

I pasted the GF hashcodes into the "objmap.csv", is that right? Because there are also the character codes.
Click to expand...
I'm glad everything is working for you!

No the GF codes are their own thing - actually... let me release that hashmap right here.  It's for a future mod, but the codes are the same.  New update for summons hashmaps in my main hashmap post!
 
Last edited:
I can't find the new codes for the GF or do you mean the post where you have initially posted them? :D Thanks for your effort, I'm a bit confused how to even find them in any way or so.

BTW, the "Tripod_GR_hm.csv" should be named "Tripod_DE_hm.csv". DE should be the international abbreviation for German. Like "ES" is for Spanish (from Espaniol). DE comes from "Deutsch".

Oh and can it be, that Tripod doesn't effect the in-game menu cards? I mean by pressing Triangle (Y for Xbox) and go to "Cards"?
 
Last edited:
I can't find the new codes for the GF or do you mean the post where you have initially posted them? :D Thanks for your effort, I'm a bit confused how to even find them in any way or so.

BTW, the "Tripod_GR_hm.csv" should be named "Tripod_DE_hm.csv". DE should be the international abbreviation for German. Like "ES" is for Spanish (from Espaniol). DE comes from "Deutsch".

Oh and can it be, that Tripod doesn't effect the in-game menu cards? I mean by pressing Triangle (Y for Xbox) and go to "Cards"?
Click to expand...
Thanks! I fixed it to DE in google drive.  The in-game cards are actually part of SeeD Reborn.  Could you send me the debug information for those menu pages when/if you get a chance?  Also - I got caught up in updating my other mod pages and forgot to post the GF's - they are there now!  Enjoy!
 
One MAJOR issue regarding hashcodes.

Anyone adding hashes to their own files should check if the hashcode has been used already (which would cause a collision).  The way I do this is compile all of the mods hashes into one "masterhashmap.csv" file that I only use as a test file.  I then search my code against the other codes in the master hash file before making them official. 

This might seem tedious, and it is.  This is just one of the reasons why we need a new hashing algorithm so badly.  I have to go back and do this to a few of my own files, as I had forgotten to do so about halfway through my summons and horizonpack mods.
Click to expand...
I'm still working on the hashing algorithm, plus on the performance enhancement. Just need more time, it's being hard to combine this project with my duties.

BTW, glad you liked it, messiahgov!
 
Hello JeMaHeChi,
Could you tell me what do you want to improve in the hashing algorithm?  The main problem is the memory consumption in worldmap exploration. I think the problem is that there are a lot of tiny textures that are loaded from the same big texture. So each time a new portion of the worldmap appears on screen , the big texture is loaded. So the hashing algorithm is run several times when the camera moves. This is not the case with field textures or character textures because they are always on camera.

About the collisions. It is only an assumption but i think sometimes 2 images have the same hashcodes because they are almost the same texture, only slightely blurred ( for depth of field animation). So it may happen that the comparison between pixels give the same results for a blurred texture:

Imagine a white texture with a black circle in the center. Even if i blur the texture the center of the black circle would still be darker than the borders. So if i run the hashing algorithm it would give the same result.
This error won't happen if the compared pixels are close enough.

You necessarily need to run a second algorithm that reads intermediate pixels . That's the reason of hash2map algorithm.
 
Could you send me the debug information for those menu pages when/if you get a chance?
Click to expand...
Thanks Mcindus! If you tell how to do that, it won't be a problem for me. But if we still talking about the "in-menu" cards, it happens with the English version (I think also with any other, haven't tried). I mean they don't have the nice frames around the images. Seems to be the vanilla ones only. But playing a card game is okay, so all frames are new/nice etc. =)

For anyone who has problems, this should be the *.csv files package you need for tonberry 1.6 (if I'm not wrong): http://www34.zippyshare.com/v/YjKm3zpD/file.html
But replace them as soon as Mcindus or FatedCourage update anything.

tonberry folder

  • collisions.csv (Hashmap 1.3 from Tonberry 1.5)
  • hash1map.csv (from FatedCourage's package: https://www.ff7catalog.com/posts/207862/)
  • hash2map.csv (Hashmap 1.3 from Tonberry 1.5)
  • objmap.csv (FatedCourage's "objmap (in combination with hash1map).csv")
hashmap folder (some few a bit renamed)

  • Char_Fujin_hm.csv (from FatedCourage)
  • Char_Irvine_hm.csv (from FatedCourage)
  • Char_Laguna_hm.csv (from FatedCourage)
  • Char_Quistis_hm.csv (from FatedCourage)
  • Char_Rinoa_hm.csv (from FatedCourage)
  • Char_Seifer_hm.csv (from FatedCourage)
  • Char_Selphie_hm.csv (from FatedCourage)
  • Char_Squall_hm.csv (from FatedCourage)
  • Char_Ultimecia_hm.csv (from FatedCourage)
  • Char_Zell_hm.csv (from FatedCourage)
  • HorizonPack_hm.csv (from Mcindus, https://www.ff7catalog.com/posts/210266/)
  • ProjectEden_hm.csv (from Mcindus)
  • ProjectHellfire_hm.csv (from Mcindus, for GF's from rufoos: https://www.ff7catalog.com/posts/207843/)
  • SeeDReborn_DE_hm.csv (from Mcindus)
  • SeeDReborn_EN_hm.csv (from Mcindus)
  • SeeDReborn_ES_hm.csv (from Mcindus)
  • SeeDReborn_FR_hm.csv (from Mcindus)
  • Tripod_DE_hm.csv (from Mcindus)
  • Tripod_EN_hm.csv (from Mcindus)
  • Tripod_ES_hm.csv (from Mcindus)
 
Last edited:
tnx messiahgov, but shouldn't be collision, hash1, hash2 and objmap be inside the hasmap folder? plus why we can't remove hash1map if w have all taht separate hashmaps from single mods?
 
i've moved all csv files of messiahgov in the hashmap directory of tonberry, did i do wrong? because i still have the glitch
 
Could you tell me what do you want to improve in the hashing algorithm?  The main problem is the memory consumption in worldmap exploration. I think the problem is that there are a lot of tiny textures that are loaded from the same big texture. So each time a new portion of the worldmap appears on screen , the big texture is loaded. So the hashing algorithm is run several times when the camera moves. This is not the case with field textures or character textures because they are always on camera.

About the collisions. It is only an assumption but i think sometimes 2 images have the same hashcodes because they are almost the same texture, only slightely blurred ( for depth of field animation). So it may happen that the comparison between pixels give the same results for a blurred texture:

Imagine a white texture with a black circle in the center. Even if i blur the texture the center of the black circle would still be darker than the borders. So if i run the hashing algorithm it would give the same result.
This error won't happen if the compared pixels are close enough.

You necessarily need to run a second algorithm that reads intermediate pixels . That's the reason of hash2map algorithm.
Click to expand...
You hit the nail right on the head with this post. When I built Tonberry, I did not take any time to consider these inefficiencies regarding animations/world map. I also did not spend much time thinking about the best way to select pixels for the hashing algorithms and my lazy solution was to just create the hash2map algorithm by hand-selecting pixels I thought would help distinguish. The second algorithm is only run when the first one fails (has a collision), of course. There must be a better way to do this, or at least a better way to select pixels--I would be rather surprised if what I came up with so quickly was really the best possible solution.

Edit: A potentially 'smarter' solution would be to take every texture in memory dumps (as complete a collection of unique textures as one can amass through an entire complete playthrough speedrun) and run algorithms on them to build a histogram of all of the differences between pixels. This approach could yield the perfect minimal set of pixels to check that would guarantee a collision-free set of codes. The computation and knowledge required to do this would be moderate to advanced, I think, but not beyond the realm of these forums. The eventual technique could also be documented so that it could be extended to FF7 and FF9.
 
Last edited:
Hello JeMaHeChi,
Could you tell me what do you want to improve in the hashing algorithm?  The main problem is the memory consumption in worldmap exploration. I think the problem is that there are a lot of tiny textures that are loaded from the same big texture. So each time a new portion of the worldmap appears on screen , the big texture is loaded. So the hashing algorithm is run several times when the camera moves. This is not the case with field textures or character textures because they are always on camera.

About the collisions. It is only an assumption but i think sometimes 2 images have the same hashcodes because they are almost the same texture, only slightely blurred ( for depth of field animation). So it may happen that the comparison between pixels give the same results for a blurred texture:

Imagine a white texture with a black circle in the center. Even if i blur the texture the center of the black circle would still be darker than the borders. So if i run the hashing algorithm it would give the same result.
This error won't happen if the compared pixels are close enough.

You necessarily need to run a second algorithm that reads intermediate pixels . That's the reason of hash2map algorithm.
 
Click to expand...
The main problem about the hashing algorithm is that it doesn't provide enough "resolution" on the hashes. I mean: it parses some pixels, makes some operations with them, and then it produces the hashcode. The problem is that in some cases, the pixels it analyse are almost the same, but the textures are different ones. That's the case of the collisions, so Omzy made a new Hash_Algorithm which parsed a completely different set of pixels. A better solution would be to make a new one with a higher resolution, parsing more pixels, but this would mean to increase the computation charge on a system which is already inefficient.

On the other hand, we have a huge bottleneck somewhere and I can't identify it. I suspect the texture cache is a big one, and I'm currently working on this, but didn't achieve much on this.

Another solution (for the whole system) would be to take a multithreaded approach, but since FF8 can't use multithread, don't think that is a really good solution...
 
Currently the hash algorithm checks a pixel's color against the next pixel in the series and records a binary value for whether the next color is a higher RGB value or not. This eventual binary string of pixel comparisons is converted to a decimal number and is the final hash code. There is no rule anywhere that says this is how the hash code has to be built.

Some food for thought on the matter (from my gf who is into image processing):
Method: Find 100 or so pixel locations that generate a unique hashcode for ~10,000 images.

Perform transforms on images, then compare transforms to place images into subsets. Repeat until there are maybe no more than 100 images per subset...I don't know, might need more or less, have to find out when you do it. OpenCV is probably the best codebase to use for that.
References:
http://reference.wolfram.com/language/guide/MathematicalMorphology.html
http://www.seas.upenn.edu/~bensapp/opencvdocs/ref/opencvref_cv.htm

Extract from each subset the set of pixels that differentiate each image from another. Then compare those sets of pixels from each other subset and extract the ones that are consistent in  a large percentage of the subsets

boom, you have your essential pixels for the hash code
Click to expand...
So you could separate images based on the overall average color, for example, and make sure you have equal sized subsets. Then you chart the pixels in each subset that are different amongst most images in the set and those become your pixels.

An alternative approach I thought of would be to take the entire database, for each pixel in each image, record the color of the pixel so you can build a frequency chart for each pixel (histogram). You could identify which pixels have the biggest spread (flattest bell curve) and then use those pixels. This would be easy to write an algorithm for and may yield good results. If there were still collisions, a second step could be used.
 
One MAJOR issue regarding hashcodes.

Anyone adding hashes to their own files should check if the hashcode has been used already (which would cause a collision).  The way I do this is compile all of the mods hashes into one "masterhashmap.csv" file that I only use as a test file.  I then search my code against the other codes in the master hash file before making them official. 

This might seem tedious, and it is.  This is just one of the reasons why we need a new hashing algorithm so badly.  I have to go back and do this to a few of my own files, as I had forgotten to do so about halfway through my summons and horizonpack mods.
Click to expand...
I threw together a python script to find hash collisions in the .csv hashmaps. It should be placed inside \FINAL FANTASY VIII\tonberry\ and run from the cmd line.

check_collisions.py

Code: [Select]
Code: 
$> python check_collisions.py -husage: check_collisions.py [-h] [--hash n [n ...]]Find hash collisions in .csv filesoptional arguments:  -h, --help        show this help message and exit  --hash n [n ...]  look for each hash n in the .csv filesIf no hash is given then check_collisions.py will search only for collisionsin the .csv files
check_collisions.py will ignore the collisions.csv and any .csv's in \hashmap\disabled. If no argument is given it looks for existing collisions, otherwise it will look for whatever hashes you provide it (separated by spaces).

Here's what I get when I run it with the current version of Mcindus' mod .csv's (excluding the G.F. Summons hashmap) and FatedCourage's character hashmaps:
Code: [Select]
Code: 
$...\FINAL FANTASY VIII\tonberry>python check_collisions.pyCollisions:18002686233454153068:        ricardinal_13 in hashmap\Rinoa_hm.csv (row 26)        ririsingsun_13 in hashmap\Rinoa_hm.csv (row 29)4298889016580490821:        sel_13 in hashmap\Selphie_hm.csv (row 53)        sel_13 in hashmap\Selphie_hm.csv (row 54)17967494829324066154:        zegauntlet_13 in hashmap\Zell_hm.csv (row 28)        zegauntlet_13 in hashmap\Zell_hm.csv (row 29)17967494829325114858:        zemaverick_13 in hashmap\Zell_hm.csv (row 31)        zemaverick_13 in hashmap\Zell_hm.csv (row 32)16670992488181322982:        wm_texpg_low3_13 in hashmap\HorizonPack_hm.csv (row 206)        wm_texpg_low3_13 in hashmap\HorizonPack_hm.csv (row 208)7381475426756314693:        irlt_13 in hashmap\Irvine_hm.csv (row 34)        sel_13 in hashmap\Selphie_hm.csv (row 55)7770506461894329933:        sel_13 in hashmap\Selphie_hm.csv (row 36)        sel_13 in hashmap\Selphie_hm.csv (row 42)12427237620770590282:        selt_13 in hashmap\Selphie_hm.csv (row 186)        selt_13 in hashmap\Selphie_hm.csv (row 187)9856729730939800021:        wm_texpg6_13 in hashmap\HorizonPack_hm.csv (row 92)        wm_texpg6_13 in hashmap\HorizonPack_hm.csv (row 95)17966657402075467114:        ricardinal_13 in hashmap\Rinoa_hm.csv (row 25)        ririsingsun_13 in hashmap\Rinoa_hm.csv (row 28)16670996886211056878:        wm_texpg_low3_13 in hashmap\HorizonPack_hm.csv (row 222)        wm_texpg_low3_13 in hashmap\HorizonPack_hm.csv (row 224)8526675522373142085:        sel_13 in hashmap\Selphie_hm.csv (row 60)        sel_13 in hashmap\Selphie_hm.csv (row 73)        sel_13 in hashmap\Selphie_hm.csv (row 74)12427237620809890393:        selt_13 in hashmap\Selphie_hm.csv (row 206)        selt_13 in hashmap\Selphie_hm.csv (row 207)
And here's an example of looking for two existing hashes:
Code: [Select]
Code: 
$...\FINAL FANTASY VIII\tonberry>python check_collisions.py --hash 8526675522373142085 124272376207705902828526675522373142085 collides with:        sel_13 in hashmap\Selphie_hm.csv (row 60)        sel_13 in hashmap\Selphie_hm.csv (row 73)        sel_13 in hashmap\Selphie_hm.csv (row 74)12427237620770590282 collides with:        selt_13 in hashmap\Selphie_hm.csv (row 186)        selt_13 in hashmap\Selphie_hm.csv (row 187)
Let me know if you have problems or if this is even helpful.
 
Currently the hash algorithm checks a pixel's color against the next pixel in the series and records a binary value for whether the next color is a higher RGB value or not. This eventual binary string of pixel comparisons is converted to a decimal number and is the final hash code. There is no rule anywhere that says this is how the hash code has to be built.

Some food for thought on the matter (from my gf who is into image processing):
So you could separate images based on the overall average color, for example, and make sure you have equal sized subsets. Then you chart the pixels in each subset that are different amongst most images in the set and those become your pixels.

An alternative approach I thought of would be to take the entire database, for each pixel in each image, record the color of the pixel so you can build a frequency chart for each pixel (histogram). You could identify which pixels have the biggest spread (flattest bell curve) and then use those pixels. This would be easy to write an algorithm for and may yield good results. If there were still collisions, a second step could be used.
Click to expand...
omzy could u pls point me where is located the hashing algorithm in tonberry src files?
 
From: https://www.ff7catalog.com/posts/203979/
I only wrote code in a handful of files. For the D3D9CallbackSC2 I wrote d3d9Callback.cpp/h and GlobalContext.cpp/h. ExtraCode was just unused code. For the D3D9Interceptor, I wrote d3d9Wrapper.cpp/h, Globals.cpp/h. Most all that should be needed to be modified would be in GlobalContext.cpp, I think. There you will find the hash algorithms and the logic for replacing images. It did take me a long time to understand how the directx code works, but if someone simply wants to change the way hashing works they shouldn't need to know that anyways. I think you have a good idea for bundling hash files with their mods--that would make things easier to work with and to update. The biggest thing that is needed, however, is to write new smarter hash algorithms that don't suffer from collisions. This may fix many problems and if more research is done, may decrease problems with animation lag. To compile the code you'll need Microsoft Visual Studio C++ 2010 Express (which is free). I believe you can just open the vcxproj file and then build the DLL. You might have to mess with some configuration, but hopefully it is already configured with the libraries and paths. I remember it being a pain in the ass to do the setup but you might not have to do all that since I sent out the vcxproj file.
Click to expand...
You may need to modify some path variables in build config to match your local directories.
 
Thanks Mcindus! If you tell how to do that, it won't be a problem for me. But if we still talking about the "in-menu" cards, it happens with the English version (I think also with any other, haven't tried). I mean they don't have the nice frames around the images. Seems to be the vanilla ones only. But playing a card game is okay, so all frames are new/nice etc. =)

For anyone who has problems, this should be the *.csv files package you need for tonberry 1.6 (if I'm not wrong): http://www34.zippyshare.com/v/YjKm3zpD/file.html
But replace them as soon as Mcindus or FatedCourage update anything.

tonberry folder

  • collisions.csv (Hashmap 1.3 from Tonberry 1.5)
  • hash1map.csv (from FatedCourage's package: https://www.ff7catalog.com/posts/207862/)
  • hash2map.csv (Hashmap 1.3 from Tonberry 1.5)
  • objmap.csv (FatedCourage's "objmap (in combination with hash1map).csv")
hashmap folder (some few a bit renamed)

  • Char_Fujin_hm.csv (from FatedCourage)
  • Char_Irvine_hm.csv (from FatedCourage)
  • Char_Laguna_hm.csv (from FatedCourage)
  • Char_Quistis_hm.csv (from FatedCourage)
  • Char_Rinoa_hm.csv (from FatedCourage)
  • Char_Seifer_hm.csv (from FatedCourage)
  • Char_Selphie_hm.csv (from FatedCourage)
  • Char_Squall_hm.csv (from FatedCourage)
  • Char_Ultimecia_hm.csv (from FatedCourage)
  • Char_Zell_hm.csv (from FatedCourage)
  • HorizonPack_hm.csv (from Mcindus, https://www.ff7catalog.com/posts/210266/)
  • ProjectEden_hm.csv (from Mcindus)
  • ProjectHellfire_hm.csv (from Mcindus, for GF's from rufoos: https://www.ff7catalog.com/posts/207843/)
  • SeeDReborn_DE_hm.csv (from Mcindus)
  • SeeDReborn_EN_hm.csv (from Mcindus)
  • SeeDReborn_ES_hm.csv (from Mcindus)
  • SeeDReborn_FR_hm.csv (from Mcindus)
  • Tripod_DE_hm.csv (from Mcindus)
  • Tripod_EN_hm.csv (from Mcindus)
  • Tripod_ES_hm.csv (from Mcindus)
Click to expand...
Just a few corrections... you actually don't need the hash1map.csv file if you're using the character hashmaps.  This is probably causing some collisions.  I deleted my hash1map.csv file because all of the hashcodes are already in the hashmap folder if you use the codes we've provided in the .csv's
ProjectEden hashcodes are Omzy's - I just made a separate file for them. ;)
Also - DO NOT USE more than one language pack.. ONLY use the pack that is for your language - otherwise, there will be excess collisions.

I guess I should make a video....

I threw together a python script to find hash collisions in the .csv hashmaps. It should be placed inside \FINAL FANTASY VIII\tonberry\ and run from the cmd line.
Click to expand...
Omg this.  This is sweet.
 
Thanks for the info, was a bit confused with all those files, so just used them all or as given in descriptions, like from FatedCourage's package.
I used the "from ..." to show from who to get the file, if you doesn't want to use the Zippyshare upload.
 
Until the mythical hashing algorithm is found, why not just allow modders to add their own entries to hash2map.csv as well? It would be trivial to separate the csv's in \hashmap\ by suffix, let's say _hm and _hm2, and everyone has access to both hashing algorithms, right? So just include both hashes for every image you modify, build collisions.csv on the fly, and there won't even be problems with future collisions (unless of course both algorithms produce collisions on the same files).

Honestly, I'm not convinced that we will be able to find a perfect hashing algorithm without sacrificing efficiency, there's just too many super-similar textures in menus, loading screens, etc. Certainly we can find the set of pixels with the greatest variance throughout the entire dataset of images, but even then a second round of hashing will probably be necessary for at least a few images. As long as we can keep the quickly-reloaded images (like the worldmap textures, apparently) in their own bucket we shouldn't have too much of a problem. Or am I missing something?
 
Last edited:
Status
Not open for further replies.
Back
Top