[FF4 PC/Steam] - Is there a resource unpacker and repacker available?

Fraggoso · Jun 7, 2017

I also asked in the modding subforum for FFIV:

Is there a tool that can unpack the resource file and repack them back?
I want to mess with my graphics pipeline to get rid of the big pixel/3DS look of the textures.

I couldn't find anything so far. :/

makipl · Jun 9, 2017

Are you sure we are talking about the same game?
Mine Final Fantasy IV for Steam has ALL the files unpacked to bare formats
There's nothing to unpack and repack

Sound/.*AKB files are normal OGG, just play at 0x204
*.NCGR - are normal .PNG files

EFFECT.dat, STAGEMNG_f00.dat and event2d_pack.dat are indeed casual containers, but everything else has .lz extension. I bet it's LZS

EDIT:
Okay, .DAT reversed:

Main:

Code:

Offset | Size | Description
0 | char[4] | MAGIC- "SSAM"
4 | uint | Entries count
8 + (EntryID*40) | 40 bytes | Entry
8 + (EntriesCount*40) | Varies | FILE_data

Entry:

Code:

Offset | Size | Description
0 | uint | File Pointer + (EntryCount*40+8)
4 | uint | File size
8 | char[32] | File name

About LZ:

-Not LZSS algorithm [Tested/ creates rubbish]
-Not LZMA algorithm [Tested/ unknown compression error]
-Not Zlib [Tested/ unknown compression error]

For sure some LZ compression, LZ77?

Okay, I unpacked it succesfully using LZ77 algorithm grabbed from here: https://gist.github.com/Prof9/872e67a08e17081ca00e

Fraggoso · Jun 9, 2017

Sorry I didn't bought the game yet but I thought it had a container file!
Sorry for the hassle I will take a look into the png files of sort.

Lein · Jun 10, 2017

I played this some time back, it would be nice to see a mod that improves upon the chibi models. The chibi look is really off putting as are the animations. Is it possible to export/import models and animations?

Fraggoso · Jun 10, 2017

I looked at the ncgr files and it seems there's only map files stored as normal png.
I also found the tutorial pictures and basically everything that's written with graphics but I can't find any character textures.

Do you know where the character textures are stored?
Are they stored in the dat file you mentioned above?

@Lein without a tool that's not possible at all at this time.

makipl · Jun 10, 2017

Okay. I found some valuable info online. Looks like it's all well-known Nintendo DS formats:

http://www.romhacking.net/documents/[469]nds_formats.htm#Generic

There's also info about compressions.

EDIT: Yeah... but some formats have different MAGIC than it's written in this documentation, also it lacks 3/4 of formats...

EDIT2: I'm quite busy right now, so I'll be able to reverse everything soon, but not now

UPDATE:

Did you know, that this EXE is a gold mine of debug info? Thanks to not deleted asserts and it gives you almost every possible debug info:

it also gives us info how'd they coded the game- using CodeWarrior for Nintendo DS

Oh, this is just great!

Fraggoso · Jun 10, 2017

Thanks maki! Very much appreciated.

Fraggoso · Jun 17, 2017

It's good to see that it's so open.

I really only need the texture files as of now but maybe that can open up the modding to the game as a whole.
The battle UI for instance could really need an overhaul.

makipl · Jun 21, 2017

You see, working as a L2 server support can be sometimes boring, so I coded the unpacker. I didn't have the files, so I can't really test it. Expect LZS decompression+repacking soon:

Code: [Select]

Code:

# Final Fantasy IV - .DAT Container unpacker 0.1# By MaKiPL 21-06-2017# Python 2.7import structimport os.pathmypath = r'FILEPATHHERE' #TYPE FILE PATHf = open(mypath, 'r+b')magic = f.read(4)if magic!= 'SSAM':    print('This is not FFIV DAT container file!')    exitcount = struct.unpack('<I', f.read(4))[0]pointers = [0]sizes = [0]names = ['0']_index = 0while _index < count:    pointers[_index] = struct.unpack('<I', f.read(4))[0]    sizes[_index] = struct.unpack('<I', f.read(4))[0]    names[_index] = f.read(32)    print(pointers[_index])    print(sizes[_index])    print(names[_index])    pointers.append(0)    sizes.append(0)    names.append('0')    _index += 1    relativeFiles = count*40+8OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)if not os.path.exists(OutputDir[:-4] + 'dec'):    os.mkdir(OutputDir[:-4] + 'dec')_index = 0while _index < count:    f.seek(relativeFiles+pointers[_index], 0)    buff = f.read(sizes[_index])    temppath = OutputDir[:-4] + 'dec\\' + names[_index]    if os.path.exists(temppath):        os.remove(temppath)    ff = open(temppath, 'w+b')    ff.write(buff)    ff.close()    _index += 1    f.close()

Fraggoso · Jun 21, 2017

Maki that's wonderful!
How can I test it on my end? xD

Thanks and keep me posted on the repacker!

makipl · Jun 21, 2017

You need to install Python 2.7 (I believe it should work on Python 3 too)
Copy this script, paste into new file and name it as you wish, but add .py extensions (e.g. FF4steamUnpacker.py)
Then edit the file putting the container you want to unpack in line 8

This article should help you set up environment paths:
http://effbot.org/pyfaq/how-do-i-run-a-python-program-under-windows.htm

Then you'll be able to just open cmd and type like:
python.exe NameOfScript, like:

python.exe FF4SteamUnpacker.py

Fraggoso · Jun 21, 2017

Okay I'll give it a try. Thanks again!
Hopefully nothing will blow up on my end. ;P

makipl · Jun 21, 2017

Okay. I came back home and tested it with original files. I made a mistake. Here's corrected code:

Code: [Select]

Code:

# Final Fantasy IV - .DAT Container unpacker 0.2# By MaKiPL 21-06-2017# Python 2.7import structimport os.pathmypath = r'D:\FINAL FANTASY IV\EXTRACTED_DATA\files\STAGEMNG_f00.dat' #TYPE FILE PATHf = open(mypath, 'r+b')magic = f.read(4)if magic!= 'SSAM':    print('This is not FFIV DAT container file!')    exitcount = struct.unpack('<I', f.read(4))[0]pointers = [0]sizes = [0]names = ['0']_index = 0while _index < count:    pointers[_index] = struct.unpack('<I', f.read(4))[0]    sizes[_index] = struct.unpack('<I', f.read(4))[0]    names[_index] = f.read(32).rstrip('\0')    print(pointers[_index])    print(sizes[_index])    print(names[_index])    pointers.append(0)    sizes.append(0)    names.append('0')    _index += 1   relativeFiles = count*40+8OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)if not os.path.exists(OutputDir[:-4] + 'dec'):    os.mkdir(OutputDir[:-4] + 'dec')_index = 0while _index < count:    f.seek(relativeFiles+pointers[_index], 0)    buff = f.read(sizes[_index])    temppath = OutputDir[:-4] + 'dec\\' + names[_index]    if os.path.exists(temppath):        os.remove(temppath)    ff = open(temppath, 'w+b')    ff.write(buff)    ff.close()    _index += 1   f.close()

Also run this code on Python 2.x
I'll copy some files on pendrive to test the code on original files tomorrow for LZS decompression and repacking

Fraggoso · Jun 23, 2017

So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]

Code:

>>> python.exe _ffiv.py  File "<stdin>", line 1    python.exe _ffiv.py                   ^SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?

makipl · Jun 23, 2017

Code:
So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]

Code:

>>> python.exe _ffiv.py File "<stdin>", line 1 python.exe _ffiv.py ^SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?

You should run it more like this:

But worry not. I'm preparing the C version of it, because no any python LZ77 algorithm works for the files, I don't know why

Fraggoso · Jun 23, 2017

Okay I'll try it out and see what the results are.

Fraggoso · Jun 27, 2017

It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

makipl · Jun 30, 2017

It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

I'm just a total moron- it's not LZ77, but GBA LZ77, which in fact is not LZ77 but LZ10. I almost finished the C tool, it has all- auto extract with LZ decompression and repacking too! But... I'm struggling with this for hours and I still get HEAP corruption no matter what.

Fraggoso · Jun 30, 2017

Hi Maki, no problem!
I'll gladly wait for it. =)

makipl · Jul 3, 2017

Okay. I'm done. I finally found the location where I corrupted the memory. Fixed it and everything works fine (yet the software crashes just as it's exits, but that's minor issue, nothing to worry about)

https://github.com/MaKiPL/FFIV_repacker/releases

You can grab it from there ^

There might be a compression issue... After uncompressing GBA LZ file from FFIV and recompressing it again, the file content doesn't match 1:1. The difference are the control bits, they are different, but the whole concept of data structure is the same (therefore that might work)
If anyone have an idea why the software crashes at closing, then feel free to tell

[FF4 PC/Steam] - Is there a resource unpacker and repacker available?

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

Lein

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

Fraggoso

Guest

makipl

Guest

Fraggoso

Guest

makipl

Guest