FF7 Crisis Core - File Format and Data Investigation

  • Thread starter Thread starter koral
  • Start date Start date
Status
Not open for further replies.
Wow, you really seem to know what you're talking about, X-Dina! it all seems to make quite a lot of sense.
Koral, that's a magnificent job you're doing here, just look at what you've accomplished! A few weeks ago, i would have thought the textures would just be a dream, let alone the actual models! you're so close, so keep going!
 
Wow, you really seem to know what you're talking about, X-Dina! it all seems to make quite a lot of sense.
Koral, that's a magnificent job you're doing here, just look at what you've accomplished! A few weeks ago, i would have thought the textures would just be a dream, let alone the actual models! you're so close, so keep going!
I don't know as much as you think... I'm just making a lot of observations....
 
This thread just randomly caught my eye - nice work, koral. I thought I'd mention that the data you are looking at is bone-relative, meaning the vertex positions need to be cumulatively multiplied by bone matrices to end up with the correct base pose position. There should also be some bone index data and weight values in there with the X-Y-Z's, as per common PS2/PSP vertex blending block arrays. Though that isn't necessarily the case, if there are bone indexes in there they should be pretty easy to spot.

Hope that helps, good luck.
 
if I have knowleged of grafics, but I don't, then I don't have to help, but Koral make a excelent work and I hope to mek progress XD.
 
I really should check here more often  :-P

ignitz, don't worry about it, I think I have almost solved this  :-D
your help with the scripts was useful, so if you want you could try looking into other things, like Monster data maybe?

X-Dina and MrAdults thanks for your observations, but I doubt that those vertices are bone-positions, or in pre-bone-heirarchy-multiplied positions because all the vertices are actually present and parsable.
But I do agree that some kind of skinning-data might also be mixed into those Type-2 vertices.

This was how Tifa looked when only Type-1 vertices are parsed:


And like this when I attemtpted to parse Type-1 vertices out of the Type-2 chunks:


As you can see, all the vertices are there in correct positions, and it's only the polygon indexing which is messed up. The vertices are actually listed as a huge Triangle list within these [!]-files, listed one after the other in approximatly 24-byte chunks.

I have discovered that there seems to be a sort of 0x1C-byte "header" before the beginning of Type-1 or Type-2 vertex groupings, with a short at 0x14 giving a count of how many Type-1 vertices follow. This doesn't seem to work with Type-2 vertex grouping "headers" though, and I am guessing that those are probably multiple "headers" listed one after another until a Type-1 vertex comes along. Or maybe skin data with bone weights.

Or something which isn't imporant at any rate. :lol:
 
Well, I'm not good at reading headers, and as such I won't even attempt it just now...

But I do have an idea about them...

IF the header is structured like this...

Code: [Select]
Code:
File sections       vertex pool size              type one vert amount                            actual verts

then it could be that the type one verts are counted with the type one vert amount, and everything in the vertex pool AFTER the end of the type one's end offset is a type 2.

EX:
Code: [Select]
Code:
Vertex type one amount = offset 0x01AVertex pool starts at 0x010 and ends at 0x01F0x010 This is a Type 10x011 Type 10x012 Type 10x013 Type 10x014 Type 10x015 Type 10x016 Type 10x017 Type 10x018 Type 10x019 Type 10x01A Type 10x01B This is after the specified size... everything after this is Type 2, even though Type 2's size isn't listed0x01C Type 20x01D Type 20x01E Type 20x01F Type 20x020 This is after the end of the vertex pool... Everything from here on is something else...

This is all speculation...
 
Pre-multiplied positions would actually be parsable and appear just as your mog screenshot depicted. In the last model you show, I believe the case might be that it happens to have relatively few bone weights, so most vertices are already in the correct position. However, I still notice some verts in that shot appear out of place, like the one next to the left leg.

It seems likely that type-2 vertices are just vertices with an implicit bone weight reference. The problem is that even though the 1-weight vertices happen to be in triangle list order, that is not the "proper" way to draw them, as there is probably an index key which reference both 1-weight and multi-weight vertices according to face. Again though, totally speculating, as I have no actual data to reference.
 
I guess you guys have a point :oops:
What I should do is take a couple of these files and upload them for you guys to view at your leisure, I am sure I may be missing the obvious.
"1-weight vertices" does sound like a more accurate description of TYPE-1 vertices, but I can't seem to decipher any possible indices anywhere obvious.

Right now, my spare time is going elsewhere.
I have started to Wikify all my findings into Quimms wiki: http://wiki.qhimm.com/FF7:CC
It is still far from complete, but I should manage to get it all filled up in the next few days.

Huge thanks to The SaiNt for giving me access and getting me started!
 :-D
 
Sorry for the double-post, but whilst I was updating the Wiki I noticed something I should have noticed a long long time ago:

click http://wiki.qhimm.com/FF7:CC
then goto the "Miscelenous Findings" section


<TEXT>
 Materia has leveled up!
<END>

<TEXT>
 Nothing more to do, kupo!
<END>
in all ATEL files this script is present, and because the Moogle text is related so strongly with the Debug-Menu data in file 00000.RAW, the first line about the Materia-Levelling-Up can be assumed to also be linked to that.

In other words: Maybe there is a "hidden" Materia to use the Debug-Menu!

All the Materia-Data is present in files 08628 - 08690, so all we have to do now is put two-and-two together!
 :-D
 
Cool, I'd be happy to have a look, if you get the chance to put some binary up somewhere. My Crisis Core disc is currently at a friend's house, and I don't have an early-firmware PSP to assist in ripping the contents myself. :)
 
Here are a couple of the !-Model files that you or anyone else could look at: http://www.mediafire.com/?yiyjj22nm0d

02373.RAW is Moogle-san (representing a simple character sample) and 02190.RAW is Reno (a more complex character).

The actual vertex data begins after a line containing nothing but "wwwwwwww" (or "777777777" hex)
The offset of the Texture-data is located at 0x1C from the beginning of the file, so the size of the entire block can be determined pretty easily.

Best of luck!  :-D
 
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
Code:
FIRE FIRA   FIGABRIZDBRIZRABRIZGATHDGRABIDEANKOKUULTIMA ?EMPTY? ?EMPTY? ?EMPTY?REJNEDREINGRABIDEPOISONQUAKESTOPBARRIERMABARRIERWALL ?EMPTY?LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALLLEAVES_SPIN, SPIN_SPIN, SPIN_KISEKILEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI ?EMPTY? ?EMPTY?HITHIT_01 ?EMPTY?LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARMRECOVERY, SWORD_KISEKIJUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOLRAKURAIFIRE_HIT, SWORD_FIREBRI_HIT, SWORD_BRITHD_HIT, SWORD_THDSHOOT_THROUGH_HITSWORD_KISEKIZENI_COIN, ZENI_HIT, ZENI_PITCHLEAVES_STEAL, STEAL_HANDSP_CHARGELEAVES_SAILESS, SAIRENT2, SAIRENT4HIT01_DEATHASPIL, ASPIL_HIT, ASPIL_SIPPOQUAKE ?EMPTY?HIT_RECOVERYHIT01, SEORD_KISEKIEXPLOSION_HIT, KISEKI_SHINRALEAVES_TEKKEN_PUNCHLEAVES_FLARE, FLARE_FILTER, FLARE_HIT ?EMPTY?THDGALEAVES_GOBURIN_PUNCH ?EMPTY? ?EMPTY? ?EMPTY?DESP
The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:


  • "R"'s are used instead of "L"s, as per Japanese fashion
  • Japanese words like "KISEKI" are mixed in too
  • References to "SHINRA" and "BASTER" and "PRASOL" refer to the 3 swords which Zack has during the game, namely the Shinra standard, Buster Sword and the parasol (used when on vacation at Costa De Sol)
  • The spelling mistakes suggest that these ASCII characters were typed in very quickly without much care, so they probably were nothing more than aides to the developers
  • Most of them seem to be obvious Materia magic, but others (like the "HIT" ones) are standard attacks used by Zack
  • "DESP" is probably DMW attack activation
  • "ASPIL" is probably items being used
  • There are TWO "GRABIDE" files, kindof strange, no?
  • "RAKURAI" and "ANKOKU" seem just as mysterious
  • The one called "LEAVES_WUTAI" is quite interesting too
  • "LEAVES_TEKKEN_PUNCH" :-P

These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D
 
Thanks for putting those files up! I got up and running with them tonight and have textures working and have found pointers to some vertex data (with some extra vertex info, will document that once I have things working a bit more solid). I'm now working on trying to find some kind of index or skeletal data, and will let you know if I have any success.

BTW, you accidentally put up the ShinRa storage box model instead of Mog. :) However, in working with that model, I noticed the texture is 128x128, but it seems I need to treat it as though it has 2 interlace rows (just like the Reno texture which is 256x256) to decode it properly. Is this a specific thing to this one texture in this chunk, or is the "width/128" rule not reliable/correct?
 
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
Code:
FIRE FIRA   FIGABRIZDBRIZRABRIZGATHDGRABIDEANKOKUULTIMA ?EMPTY? ?EMPTY? ?EMPTY?REJNEDREINGRABIDEPOISONQUAKESTOPBARRIERMABARRIERWALL ?EMPTY?LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALLLEAVES_SPIN, SPIN_SPIN, SPIN_KISEKILEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI ?EMPTY? ?EMPTY?HITHIT_01 ?EMPTY?LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARMRECOVERY, SWORD_KISEKIJUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOLRAKURAIFIRE_HIT, SWORD_FIREBRI_HIT, SWORD_BRITHD_HIT, SWORD_THDSHOOT_THROUGH_HITSWORD_KISEKIZENI_COIN, ZENI_HIT, ZENI_PITCHLEAVES_STEAL, STEAL_HANDSP_CHARGELEAVES_SAILESS, SAIRENT2, SAIRENT4HIT01_DEATHASPIL, ASPIL_HIT, ASPIL_SIPPOQUAKE ?EMPTY?HIT_RECOVERYHIT01, SEORD_KISEKIEXPLOSION_HIT, KISEKI_SHINRALEAVES_TEKKEN_PUNCHLEAVES_FLARE, FLARE_FILTER, FLARE_HIT ?EMPTY?THDGALEAVES_GOBURIN_PUNCH ?EMPTY? ?EMPTY? ?EMPTY?DESP
The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:


  • "R"'s are used instead of "L"s, as per Japanese fashion
  • Japanese words like "KISEKI" are mixed in too
  • References to "SHINRA" and "BASTER" and "PRASOL" refer to the 3 swords which Zack has during the game, namely the Shinra standard, Buster Sword and the parasol (used when on vacation at Costa De Sol)
  • The spelling mistakes suggest that these ASCII characters were typed in very quickly without much care, so they probably were nothing more than aides to the developers
  • Most of them seem to be obvious Materia magic, but others (like the "HIT" ones) are standard attacks used by Zack
  • "DESP" is probably DMW attack activation
  • "ASPIL" is probably items being used
  • There are TWO "GRABIDE" files, kindof strange, no?
  • "RAKURAI" and "ANKOKU" seem just as mysterious
  • The one called "LEAVES_WUTAI" is quite interesting too
  • "LEAVES_TEKKEN_PUNCH" :-P

These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D
This is a Debug DATA, it's normal this unique "lenguage" that some the programmer only "understand"   :evil:
The wrong translate leave to this
 
I'm still perplaxed by the lack of face indices - I've tried all kinds of bruteforce searches through the file for every type of index I could think of. The most promising thing I found between both files you put up were from a DWORD pointer at 0x48, it leads to an array of data that looks very much like triangle indices. However, the size and stride does not appear correct, so if it is index data, there is probably another key somewhere else into the file pointing into it. I have not tried getting offsets to the specific "faces" and scanning the file for them yet.

I have to get back to a contract project for a couple days here, so I won't have time to look further until I'm done. In the mean time, good luck! :)
 
Woops, sorry for the missing Mog!  :oops:
I have uploaded it here now, along with the other two as well: http://www.mediafire.com/?yzzmoiandee

But thankyou so much for trying to help!  :-D
The file-data is really quite unusual, and as inefficient as un-indexed triangle-lists are, it seems to be the method they chose to store the data, possibly just directly dumping the contents into RAM for the GPU (although I am not so sure about the internal functioning of PSP's myself).

I haven't fully understood the texture-interlacing, but the rule of dividing it by 128 seems to work majority of the time. Zande had pointed out before that there might be different interlacing modes determined by a value in the header, so it is worth keeping in mind that my interlacing rules are temporary.

The skeletal data probably does follow on after the vertices, in [unknown-chunk-3] and [unknown-chunk-4], because the ShinRa box and the barrel dont contain any data at those points.
If I had to choose, I would say that any face-indexing is in [Unknown-chunk-1], before all the vertices.
But the values are mostly zeros, so maybe they are just initial bone rotation values?

Just do what you have to do, we are not on any strict kind of deadlines here!  :-P
 
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
Code:
FIRE FIRA   FIGABRIZDBRIZRABRIZGATHDGRABIDEANKOKUULTIMA ?EMPTY? ?EMPTY? ?EMPTY?REJNEDREINGRABIDEPOISONQUAKESTOPBARRIERMABARRIERWALL ?EMPTY?LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALLLEAVES_SPIN, SPIN_SPIN, SPIN_KISEKILEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI ?EMPTY? ?EMPTY?HITHIT_01 ?EMPTY?LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARMRECOVERY, SWORD_KISEKIJUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOLRAKURAIFIRE_HIT, SWORD_FIREBRI_HIT, SWORD_BRITHD_HIT, SWORD_THDSHOOT_THROUGH_HITSWORD_KISEKIZENI_COIN, ZENI_HIT, ZENI_PITCHLEAVES_STEAL, STEAL_HANDSP_CHARGELEAVES_SAILESS, SAIRENT2, SAIRENT4HIT01_DEATHASPIL, ASPIL_HIT, ASPIL_SIPPOQUAKE ?EMPTY?HIT_RECOVERYHIT01, SEORD_KISEKIEXPLOSION_HIT, KISEKI_SHINRALEAVES_TEKKEN_PUNCHLEAVES_FLARE, FLARE_FILTER, FLARE_HIT ?EMPTY?THDGALEAVES_GOBURIN_PUNCH ?EMPTY? ?EMPTY? ?EMPTY?DESP
The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:


  • "R"'s are used instead of "L"s, as per Japanese fashion
  • Japanese words like "KISEKI" are mixed in too
  • References to "SHINRA" and "BASTER" and "PRASOL" refer to the 3 swords which Zack has during the game, namely the Shinra standard, Buster Sword and the parasol (used when on vacation at Costa De Sol)
  • The spelling mistakes suggest that these ASCII characters were typed in very quickly without much care, so they probably were nothing more than aides to the developers
  • Most of them seem to be obvious Materia magic, but others (like the "HIT" ones) are standard attacks used by Zack
  • "DESP" is probably DMW attack activation
  • "ASPIL" is probably items being used
  • There are TWO "GRABIDE" files, kindof strange, no?
  • "RAKURAI" and "ANKOKU" seem just as mysterious
  • The one called "LEAVES_WUTAI" is quite interesting too
  • "LEAVES_TEKKEN_PUNCH" :-P

These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D
Rather than treating this as materia with other attacks thrown in, I would suggest that this is all of the battle commands... just some of them happen to be item based like materia and item usage. While this doesn't change anything, it gives a little more insight into the programmers' mindset.

an interesting note about the game that may be useful... or not, is that no matter when you do it, the fight with Minerva ALWAYS uses the Buster Sword. (This may point in some way to how scripted events work in the game at a later point.)

I finally got my custom firmware now, so at any point if you all figure out where the 'DEBUG materia' is if it exists, and you can give me an ISO offset for the memory address materia is in... I can test it out with CWCheat.

I haven't looked into the model data just yet, but I'll post anything that piques my interest once I do.

AH! and about the type 1 and type 2's MrAdults mentioned they might be 1 weight verts. This would mean that the other verts, type 2, are weighted to multiple bones. If that is the case, then we would need to figure out how the weight is displayed...

Or rather, another way to look at it... the type 1 and type 2 verts are exactly the same but the bone data for the type 2's continues after the 1st bone index.

This means, we simply need to figure out how to determine the difference between vertex data and bone relations. I imagine the bone indeces wouldn't extend past 4... and more likely would only hold 2 bones.

Now, there are 2-3 things bone indeces would need...

Vertex in question's XYZ orientation + texture UV + :
1. how many bones are weighted...
2. bone's name (probably just a number)...
3. Bone's actual amount of influence...

here is how I would set it up if I were making the format.

Code: [Select]
Code:
x,y,z,u,v,#bones,bone1,weight1,bone2,weight220 32 12 20-42 02 10 32 11 32This means vertex 32, 50, 18has a UV of 32, 66and has 2 bonesBone 1 is bone number 16, and has a 50% weight.Bone 2 is bone number 17, and has a 50% weight.(I used a 100% scale on the bone weight, because a decimal number would be harder to program. the scale goes from 00(0) to 64(100) )
This is only my thoughts on how it might be worded for efficiency. So, looking at that, the type 2 verts might be as long as 10 offsets if they only have 2 bones, and up to 14 if they had as many as 4 bones weighted.(this is unlikely, but possible...)

Type 1 verts mightn't need the bone amount mini header, and would be implicitly converted.

Still, this is all speculation- Educated speculation, but speculation nonetheless.


In short... We need to decipher the bone data.

EDIT:
Ok, Upon examining I find that the verts are separated by a series of 00 chunks... between 00 00 00 and 00 00 00 00 00 00 long... what are these for? I could speculate that the length of this padded 0 data is used in determining the type of vertex it is. I notice the very first vert begins with a 4 byte long 00 00 00 00.
(In Reno)

So, my guess is that the verts begin with the padded 0's to determine their type, and then use actual coords to determine their location. I could be wrong, but that is my first thought.
 
Last edited:
One other thing that occurs to me is, even though the models seem to have only a single texture map, the models would need to be drawn in sorted/batching order for the purpose of alpha blending. For example, Reno's hair has a very soft-edged alpha, and as I recall playing the game, those surface did use real alpha blending instead of hard-edged alpha tests.

So, there would have to be a way to batch those verts into separate "surfaces", for depth-tested non-alpha (or hard-edged alpha) surfaces (like the whole body), and post-sorted alpha blended surfaces (like the hair). That dictates that there must be some kind of container with references to groups of verts, at least, if not an index list. Maybe if we can find these containers, it will shed more light on the vertex drawing order, or at least provide some clean absolute offsets into the vertex list.

Edit: Although, on second thought, it's possible that it was all hard-edged and I just could not tell because of the small screen and size of characters on the screen. :) My copy of the game is still at my friend's house, but if anyone has it handy, it would be worth looking to see if the character models do indeed use alpha blending.

Second edit: No, they are definitely blended. :) Take a look at Genesis's hair:
http://www.product-reviews.net/wp-c...inal-fantasy-vii-montage-for-the-sony-psp.jpg
 
Last edited:
Status
Not open for further replies.
Back
Top