M
mirex
Guest
Uhhh I would gladly try to work on creating P files that FF7 engine accepts, only if i had some free time. Too many projects are on my hands.
- Status
D
Darkdevil
Guest
Mirex, If you could teach me how to do stuff like that, I will be able to do some stuff as Im doing nothing untill september now...Im out of school so i have ALOT of free time.
M
mirex
Guest
Teaching this kind of hobby would take even more time imho.
Nothing personal.
D
Darkdevil
Guest
Oh, Ok.
M
mirex
Guest
But if you really want to, then i could give you few hints. Are you skilled in programming ? If not then that is the best way to go. C++, Pascal or Visual Basic (which is not much recommended, but works 
) ... choose any and go for it. Try to program something that messes with files, anything.
) ... choose any and go for it. Try to program something that messes with files, anything.
D
Darkdevil
Guest
Well I created a clip-art Veiwer in Vb6 this morning in my 6th Form induction day...Other than that, I have ne means of doing ny programming as i have no applications to do so...
G
Goku7
Guest
By the way, Mirex....what about the code I posted earlier up on the page? Granted, a lot of variables and stuff are just being used as placeholders for the real data types and such that would be used, but would that not be a start for getting direct access to specific models without decompressing the LGPs during run-time of this hypothetical re-written engine?
Looking back through some other threads, it appears there are a lot of talk about offsets to stuff. But what are the offsets relative to? The beginning of the individiual .p files?
Actually, perhaps the iidea behind the earlier code I posted could demystify some data tables that haven't been figured out, they could contain offsets that are only valid when looking at the compressed code of the LGP file, and not the .p files themselves.... resulting in a funny parallel:
LGP file == PSX data that is not stored in the normal CD filesystem
offset tables == coordinates for PSX CD Direct-Sector reads.
Heck, no one's even told me if what I posted up there or here is old ground or not....
F
ficedula
Guest
LGP files aren't themselves compressed in any way. The FF7 engine can read data straight out of them if necessary ... for that matter, so can the Remake engine (not that I've touched that in ages). LGP files just provide a way of packaging up lots of files into one big chunk, so the OS (Windows) doesn't need to bother about creating 20,000 files itself.
The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).
Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.
The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).
Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.
H
halkun
Guest
PC Data != PSX data.
More importantly:
Raw Psy-Q data != FF7-PSX data != FF7-PC data.
The PSX deals with raw Psy-Q data. (TIM, RSD, SEQ, STR, TOD, HRC, etc etc etc) These are munged and stored in special formats for FF7-PSX. (BIN, LZS). The PC version has conversions of the PSY-Q data into PC native data. (SEQ=>MID, STR=>AVI, Binary HRC=>Plain Text HRC)
The only thing that should be platform independant is the core of the kernel. It takes a lot of bit twiddeling to get the data into something that the platfrom can easily handle and that the user can understand.
G
Goku7
Guest
First off -- Long time no see, Fice. 
Oh, so they're NOT compressed....then that should make the re-writting of the engine that much easier, as then it would just need to deal with offsets w/ no real "decompression" per se.
You know....that makes sense.
After I posted the code earlier, it hit me that there just might be a table of contents in the LGP file. On the other hand, I didn't bother to mention it because I had this feeling that all this was indeed old ground.
Either way, thanks for confirming my suspicion, LOL.
I guess that the existance of a table of contents in the LGP files would be how the current FF7 engine could cope with drastically modified LGP files. This would mean that instead of having a table of contents hard-coded into the .exe file (which is what my code example assumed), it would perform the seekg commands using variables previously loaded with offset values from the table of contents.
But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?
At least my train of thought was on the right track. Yay!
It also gave me an idea regarding how to find where the battle animations are stored. Have you (or anyone else for that matter) tried to do a memory dump of the video card RAM during a battle?
My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there. :roll:
Assuming we can get a dump of the the VRAM, we could compare the information in there with the data in the LGP files, and eliminate anything that corresponds to data we already know to not be the battle animations. Any sequences of data that don't appear to be reconizeable to us should have its corresponding sequence in the LGP files noted for later reference as a possibility for the animation sequences.
If there's only one unique set of sequences of unrecognized data, then....well there ya go!! :wink:
I know that, but when I wrote what I wrote, it was supposed to be a metaphor, not a literal statement that the code was the same bit for bit, 'cause its not. Just that the philosophy in how the engine probably looks at how the code is stored is probably the same in both versions.
I.E., those PSX direct sector reads are analogous to the PC version taking an LGP file and directly grabbing the code it needs from it, and then decompressing, using the offsets the same way the PSX uses a list of coordinates to do a direct-sector read from the disk.
I probably should have been a bit more concise when I wrote that, LOL.
Oh, so they're NOT compressed....then that should make the re-writting of the engine that much easier, as then it would just need to deal with offsets w/ no real "decompression" per se.
You know....that makes sense.
After I posted the code earlier, it hit me that there just might be a table of contents in the LGP file. On the other hand, I didn't bother to mention it because I had this feeling that all this was indeed old ground.
Either way, thanks for confirming my suspicion, LOL.I guess that the existance of a table of contents in the LGP files would be how the current FF7 engine could cope with drastically modified LGP files. This would mean that instead of having a table of contents hard-coded into the .exe file (which is what my code example assumed), it would perform the seekg commands using variables previously loaded with offset values from the table of contents.
But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?
At least my train of thought was on the right track. Yay!
It also gave me an idea regarding how to find where the battle animations are stored. Have you (or anyone else for that matter) tried to do a memory dump of the video card RAM during a battle?
My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there. :roll:
Assuming we can get a dump of the the VRAM, we could compare the information in there with the data in the LGP files, and eliminate anything that corresponds to data we already know to not be the battle animations. Any sequences of data that don't appear to be reconizeable to us should have its corresponding sequence in the LGP files noted for later reference as a possibility for the animation sequences.
If there's only one unique set of sequences of unrecognized data, then....well there ya go!! :wink:
I know that, but when I wrote what I wrote, it was supposed to be a metaphor, not a literal statement that the code was the same bit for bit, 'cause its not. Just that the philosophy in how the engine probably looks at how the code is stored is probably the same in both versions.
I.E., those PSX direct sector reads are analogous to the PC version taking an LGP file and directly grabbing the code it needs from it, and then decompressing, using the offsets the same way the PSX uses a list of coordinates to do a direct-sector read from the disk.
I probably should have been a bit more concise when I wrote that, LOL.
H
halkun
Guest
Nope, it's most definatly not in there....
The video ram hold textures, texture caches, frame buffers, and palettte information. You can dump the VRAM from a PSX emulator and see for yourself...
Because the code is PSX native there's also a nother reason why haveing data in VRAM is silly.
The PSX has *no direct access to VRAM*
You can't put data into vram using a memory address as the video memory is not directly attached to the PSX's CPU bus. You have to send data using a GPU packet (Command + arguments) like "Draw triangle", "set viewport", or "load texture" That's why TIM files are used so often. They are pre-formed data that the GPU can accept.
You either put the data into a GPU register, or stream via the GPU's DMA hook. (Meaning you allow the GPU Direct Memory Accees to the PSX's main RAM, but you can't do it the other way around) The GPU has no concept of "Animation" or anything like that. It't can't even do math. That's the GTE's job and that's on another bus all togeather
G
Goku7
Guest
I was refering to the PC version......
F
ficedula
Guest
Unfortunately still isn't going to happen. Modern graphics cards can upload animations and geometry to VRAM, but for that to be *any* use whatsoever the card needs to support hardware T&L at an absolute minimum (and realistically, it needs to have other features too). When FF7 came out, they didn't, so it's not written to upload animations to video cards.
It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!
Back to the older stuff...
Thank you I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.
Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.
That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.
Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.
It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!
Back to the older stuff...
Thank you
I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.
That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.
Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.
D
Darkdevil
Guest
IIRC the most files in an lgp is close to 11,000
Its is either the feil or battle LGP file....im going for the latter.
Its is either the feil or battle LGP file....im going for the latter.
G
Goku7
Guest
Hmm.....I see. However.....
If I'm not mistaken, whatever animations that are about to be drawn on the screen are first transfered to the framebuffer, which should be in the VRAM, so technically the data would in one form or another pass thru the VRAM before going on screen. But, it just occured to me that the data might have already been processed into something other than its original form by that time, so maybe a VRAM dump might not work as I planned.
IIRC, when the case is that no HW TnL is available, then the CPU and system RAM must then caculate the Geometry Transformation instructions before passing it to the framebuffer in the VRAM....so doing a system RAM dump might still be an acceptible lead?
Of course, we'd have to do this on a system with the minimum RAM necessary to get the game to run, as the last thing we need is to go through 1/4 to 1/2 a gig of data looking for the animations.
Basically, I'm saying that since we haven't been able to locate where the animation data is stored, we let the game do it for us, and then we take a picture of what it grabbed from the LGP, and then we go back and cross-reference the data there with what's in the LGP.
There is another remaining point that we ought to remember. Thanks to what halkun dug up regarding what FF7pc was compiled from, we know that FF7pc was written in C++. Therefore, we must think within the limitations of that language when trying to reconstruct how FF7 can access and decode battle operations, etc, and that includes file I/O, and memory storage.
That leaves us with commands like ".read"; ".get"; and ".getline" for loading the battle animation data into memory, if I'm not mistaken. And considering that the data is probably a series of values, it would have to be loaded into some sort of data array, which in turn means the entire data sequence for the animations would be occupying adjacent memory addresses, meaning a RAM dump just might happen to have the battle animation data all conveniently lumped together in one section of memory addresses, instead of being scattered all over the place. This would allow us to very quickly narrow down which sequence it is....I think.
Then again, I do have to admit I've only taken an introductory course in C++, so maybe I'm being too naive or something in regards to how the thing could be coded........
Thank you
Well, congrats on that. If all goes well, I also just might get a full time job in a computer store.....I'm just waiting for them to review my application/resume, and make a decision....
Well, in that case, the beginning of the table of contents isn't hard-coded as an offset either.
But how would they figure out where it ends?
.....oh, duh. There's probably a delimiter character in there. Man, do I feel stupid.
Great. I had the right conclusion (there's a limit to how many files you can put in an LGP file), but got to it by a completely wrong line of reasoning....what are the chances of that?
But still....who in their right mind would need to have 65,500 files for a program?
11,000?! Ok, the programmers need to be taken into the backyard and shot one by one on behalf of the KISS principle.....
D
Darkdevil
Guest
The exact number.
H
halkun
Guest
Keep in mind they are 11,000 really tiny files. some are just a small 32x32 4 bit pictures and such.
C
Cyberman
Guest
That's quite unrealistic
Here is a dosage of reality here. Take a game like NWN you have say 3500 scripts 1200 models etc etc. Now each model consists of about 10 different files.. are you following me yet?
Let me change the way you look at things. If you were to see Windows Source code, think abou tthe number of components and different files that consist of windows. The programers wisely instead of having 11000 different files archived them into a singular file they could extract the data from, however originally they did NEED the 11000 files because they had to be made.
They didn't just WISH the LGP contents into being
A group of artists and scriptors were busily making the contents of that. Hence 11000 is not rediculous.What they did do was KISS they centralized the data and kept information of where things were.
I'm sure FF8 has similar volumes of data in it (and I'm making slow progress on finding the sections of data heh, I have the world textures found already I just need to find the topographic data).
An example is one FF7 Battle mode on the PS1l for cloud.
The one file contains 113 sections
Section 1 consists of 23 bones six of which are joints. That means it has 17 models with vertex pools triangles quads with colored vertices and triangles and quads with textures.
Section 97 is the texture data
Section 98-113 are the weapon models.
Just one single file in reality consists of more than a hundred smaller files originally.
More importantly they HAD to have the 100+ original files and you can't just compile them to one file and throw them out (it will loose information).
There are 300+ enemys the average enemy has 10 bones lets say (no texture) that's 1 HRC and 10 RSD files per enemy, that doesn't include the animation information (there are at least 1 animation but generally there are 5) then a TIM is always include (irregardless if it's used). that's 16 at minimum original files. Well over 4000+ files just for battle models alone.
There is just TOO much information to make it super simple so the programers did the next best thing they archived it
Cyb
G
Goku7
Guest
First, I'm not knocking the fact that they archived it -- I think that's smart, and that more games should nowadays have as much stuff as they can be compressed w/o too much performance loss, instead of unloading 5,000 .bmp files for textures and raw .wav audio for sound.
But that aside, I realize that most of the time a game or powerful program's source code is made up of large numbers of files. However, when I said what I said, it didn't cross my mind that I should have considered those 11,000 files to be in the same vein as the source code, as you essentially pointed out.
However, I was also under the impression that even the final amount of files they ended up with in the LGPs was still needlessly split up past the point of simplicity -- like, for example, the fragments of previous threads on the files and how the engine uses them made it sound like the game had 300 separate files it had to access and load separately, JUST FOR CLOUD'S HEAD. That's what I thought was going on, and that's why I was saying that the programmers didn't seem to be following KISS.
I guess I was mistaken, though.....
But that aside, I realize that most of the time a game or powerful program's source code is made up of large numbers of files. However, when I said what I said, it didn't cross my mind that I should have considered those 11,000 files to be in the same vein as the source code, as you essentially pointed out.
However, I was also under the impression that even the final amount of files they ended up with in the LGPs was still needlessly split up past the point of simplicity -- like, for example, the fragments of previous threads on the files and how the engine uses them made it sound like the game had 300 separate files it had to access and load separately, JUST FOR CLOUD'S HEAD. That's what I thought was going on, and that's why I was saying that the programmers didn't seem to be following KISS.
I guess I was mistaken, though.....
T
trickstar66
Guest
They shouldn't just use compression but at least use an original compression tool. I was playing Return to Castle Wolfenstein recently and got bored so I started to poke about and found the data files. They were compressed in nothing but a .lzs file!! The files themselves weren't too great either. They were simple .tga and .wav files. The only thing that was compressed good were the 3D models. They were in a seperate folder from the compressed .lzs folders.
- Status