Ye Olde PSX .DAT files redeux (4SBWY_1.DAT)

  • Thread starter Thread starter Cyberman
  • Start date Start date
Status
Not open for further replies.
C

Cyberman

Guest
I've been looking at these of late (like within the last 2 weeks to be precise)
I modified my program to dump these little images of the structure.

We currently know a few things about these.
1 The first 7 Double Words are pointers to the playstations memory
2 The first section of the 7 sections of this file is the script section.

I'm not sure how to deliniate the sections properly myself.  So as a GUESS I took the pointers and assumed they were a certain amount apart. IE the data is decoded and dumped into memory directly.  I suspect this is not entirely true.

<removed cute cudely images of DATA!>

The sections aren't properly aligned with the pointer information AND all the pointer informat is DWORD aligned. The data is not aligned acording to that it appears. So the only thing I can think of is it's decompressed and a header is parsed to determine the locations the locations the real data gets stuffed into memory.  Since we know what the file is and where it's supposed to be in memory a little debugging I suspect will find the data using a PS1 emulator that should be aligned at these locations.  That's my new plan at least :D

I suspect that section 1 may have some other information in it to find the beginings and ends of these sections. IE Section 1 does not start immediately :lol: so back to the drawing board.

For reference I am using \FIELD\4SBWY_1.DAT file to work on this.

Any thoughts on this anyone?

Cyb
 
Here is my experimental field viewer. It's a bit sloppy, but then again it is not finished.
Code: [Select]
Code: 
// System#include <sys/types.h>#include <sys/stat.h>#include <fcntl.h>// STL#include <string>#include <map>// C#include <cassert>#include <errno.h>#if 1// GLUT#include <GLUT/glut.h>// OpenGL#include <OpenGL/OpenGL.h>#include <OpenGL/gl.h>#include <OpenGL/glu.h>#else// GLUT#include <glut/glut.h>// OpenGL#include <gl/gl.h>#include <gl/glu.h>#endifusing namespace std;typedef unsigned char u8;typedef unsigned short int u16;typedef unsigned long int u32;typedef signed char s8;typedef signed short int s16;typedef signed long int s32;// - utility functions --------------------------------------------------------u16 get_u16le(void const * const buf){ return ((u8*)buf)[0] | (((u8*)buf)[1]<<8);}u32 get_u32le(void const * const buf){ return ((u8*)buf)[0] | (((u8*)buf)[1]<<8) | (((u8*)buf)[2]<<16) | (((u8*)buf)[3]<<24);}#ifndef O_BINARY#define O_BINARY 0#endifint load( const string & name, void ** data ){        size_t size = 0;        *data = NULL;        int fd = open( name.c_str(), O_RDONLY|O_BINARY, 0 );        if (fd >= 0) {                struct stat sb;                if ( fstat( fd, &sb ) >= 0) {                        assert( sb.st_size > 0 );                        void *tmp = malloc( sb.st_size );                        if ( tmp!=NULL ) {                                if (read( fd, tmp, sb.st_size ) == sb.st_size) {                                        *data = tmp;                                        size = sb.st_size;                                } else {                                        free(tmp);                                }                        }                }                 close( fd );        } else {  printf("%i\n", errno); }        return size;}void save(const std::string & name, const void *data, int size){        int fd = open( name.c_str(), O_WRONLY|O_CREAT|O_TRUNC, 0666 );        if (fd >= 0) {  write( fd, data, size );                close( fd );        }}int load_lzs(const string & name, void ** data ){ // read compressed data u8 *buf; int tsize = load( name, (void**)&buf ); if (buf == NULL) {  *data = NULL;  return 0; }  int isize = get_u32le(buf) + 4;  if (isize != tsize) {  free(buf);  *data = NULL;  return 0; }  // decompress; int osize = (isize + 255) & ~255; u8 * obuf = (u8 *)malloc(osize); int iofs = 4, oofs = 0; u8 cmd=0, bit=0; while (iofs < isize) {  if (bit == 0) {   cmd = buf[iofs++];   bit = 8;  }  if (cmd&1) {   obuf[oofs++]=buf[iofs++];   if (oofs==osize) {    osize+=256;    obuf = (u8*)realloc(obuf, osize);   }  } else {   u8 a = buf[iofs++];   u8 b = buf[iofs++];   u16 o = a | ((b&0xF0)<<4);   u8 len = (b&0xF)+3;      int rofs =  oofs - ((oofs - 18 - o) & 0xFFF);   for (int j=0; j<len; j++) {    if (rofs < 0) {     obuf[oofs++]=0;    } else {     obuf[oofs++]=obuf[rofs];    }    if (oofs==osize) {     osize+=256;     obuf = (u8*)realloc(obuf, osize);    }    rofs++;   }  }  cmd>>=1;  bit--; } free( buf );  *data = obuf; return oofs;}// - field class --------------------------------------------------------------class CFieldBackground {private: struct TexInfo {  int page, pal;  bool hc;  bool operator<(const TexInfo & rhs)  {   return memcmp(this, &rhs, sizeof(TexInfo)) < 0;  } }; map<TexInfo, int> texInfoMap;  u8 * mimData; int mimSize; u8 * datData; int datSize; u8 * bsxData; int bsxSize;  struct Texture {  u8 * rgba; // data  int id;  // OpenGL texture ID };  struct Tile {  int u, v, pal; // info from the request, pal=-1 for RGBA  int s, t; // texture position in OpenGL texture  int texture; // Texture };  int getTileI8(int u, int v, int pal); int getTileRGBA(int u, int v);  int getTexture(u8 atr1, u8 atr2, u8 x);public: CFieldBackground(); ~CFieldBackground(); void load(const std::string & name); void draw();};CFieldBackground::CFieldBackground():  mimData(NULL), mimSize(0),  datData(NULL), datSize(0),  bsxData(NULL), bsxSize(0){}CFieldBackground::~CFieldBackground(){}voidCFieldBackground::load(const std::string & name){ // read compressed data string baseName = name.substr(0, name.find('.'));  // MIM string mimName = baseName + ".MIM"; mimData = NULL; mimSize = load_lzs( mimName, (void **)&mimData );  // DAT string datName = baseName + ".DAT"; datData = NULL; datSize = load_lzs( datName, (void **)&datData ); save(datName + ".bin", datData, datSize);  // BSX string bsxName = baseName + ".BSX"; bsxData = NULL; bsxSize = load_lzs( bsxName, (void **)&bsxData ); save(bsxName + ".bin", bsxData, bsxSize);}int CFieldBackground::getTexture(u8 atr1, u8 atr2, u8 x){ TexInfo texInfo; texInfo.page = (atr1 >> 4); // * 128 int px = texInfo.page * 64 + x; texInfo.pal = px < 512 ? px / 256 : ((px - 512) / 128) + 2; texInfo.hc = false; // atr2 & 1; if (texInfoMap.find(texInfo) != texInfoMap.end()) {  return texInfoMap[texInfo]; } printf("%2.2x %2.2x\n", atr1, atr2);  int ofs = get_u32le(mimData); int step = get_u32le(mimData+ofs); int npal = get_u16le(mimData+10); int pal_ofs = 12; int pic_ofs = ofs+12; int xsize = get_u16le(mimData+ofs+8)*2; int ysize = get_u16le(mimData+ofs+10); u8 *texture = new u8[256*256*4];  if (texInfo.hc) {  int page = texInfo.page * 128;    for (int y=0; y<256; y++) {   for (int x=0; x<256; x++) {    u16 col = get_u16le(mimData + (y*xsize+x*2+page) + pic_ofs);         int r = (((col      ) & 31) * 255 + 15) / 31;    int g = (((col >>  5) & 31) * 255 + 15) / 31;    int b = (((col >> 10) & 31) * 255 + 15) / 31;    int a = (col & 0x8000) || (col == 0) ? 0 : 255;        int addr = (y*256+x) * 4;    texture[addr+0] = r;    texture[addr+1] = g;    texture[addr+2] = b;    texture[addr+3] = a;   }  } } else {  int page = texInfo.page * 64;  int pal = texInfo.pal * 512;    for (int y=0; y<256; y++) {   for (int x=0; x<256; x++) {    u8 idx = mimData[(y*xsize+x+page) + pic_ofs];    u16 col = get_u16le(mimData + idx*2 + pal_ofs + pal);         int r = (((col      ) & 31) * 255 + 15) / 31;    int g = (((col >>  5) & 31) * 255 + 15) / 31;    int b = (((col >> 10) & 31) * 255 + 15) / 31;    int a = (col & 0x8000) || (col == 0) ? 0 : 255;        int addr = (y*256+x) * 4;    texture[addr+0] = r;    texture[addr+1] = g;    texture[addr+2] = b;    texture[addr+3] = a;   }  } }  int textureId; glGenTextures(1, &textureId); glBindTexture( GL_TEXTURE_2D, textureId);#if 0 glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST ); glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST );#else glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR ); glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );#endif glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 256, 256, 0, GL_RGBA, GL_UNSIGNED_INT_8_8_8_8, texture);  delete texture;  texInfoMap[texInfo] = textureId; return textureId;}voidCFieldBackground::draw(){ u32 memaddr[7]; for (int i=0; i<7; i++) {  memaddr[i] = get_u32le(datData+i*4); } u32 base = memaddr[0] - 7 * 4; u32 section[7]; for (int i=0; i<7; i++) {  section[i] = memaddr[i] - base; }  u8 * tilemap = datData + section[2] + get_u32le(datData + section[2]); int count = (get_u32le(datData + section[2] + 4) - get_u32le(datData + section[2])) / 8;#if 1 printf("---- %4.4x\n", count); for (int i=0; i<get_u32le(datData + section[2]); i++) {  printf("%2.2x ", (u8*)(datData + section[2])[i]);  } printf("\n----\n"); for (int i=0; i<get_u32le(datData + section[2] + 8)-get_u32le(datData + section[2] + 4); i++) {  printf("%2.2x ", (u8*)(datData + section[2] + get_u32le(datData + section[2] + 4))[i]);  } printf("\n----\n");#endif for (int i=0; i<count; i++) {  s16 tx = (s16)get_u16le(tilemap + 8 * i + 0) + 256;  s16 ty = (s16)get_u16le(tilemap + 8 * i + 2) + 256;  u8 sx = tilemap[8 * i + 4];  u8 sy = tilemap[8 * i + 5];  u8 atr1 = tilemap[8 * i + 6];  u8 atr2 = tilemap[8 * i + 7];  // printf("%2.2x %2.2x|", atr1, atr2);    glBindTexture(GL_TEXTURE_2D, getTexture(atr1, atr2, sx));    glBegin(GL_QUADS);  glTexCoord2i(sx, sy); glVertex2i(tx, ty);  glTexCoord2i(sx+15, sy); glVertex2i(tx+16, ty);  glTexCoord2i(sx+15, sy+15); glVertex2i(tx+16, ty+16);  glTexCoord2i(sx, sy+15); glVertex2i(tx, ty+16);  glEnd(); } //printf("\n");#if 0 tilemap = datData + section[2] + get_u32le(datData + section[2] + 4); count = get_u16le(tilemap); for (int i=0; i<count; i++) {  s16 tx = (s16)get_u16le(tilemap + 14 * i + 0 + 2) + 512;  s16 ty = (s16)get_u16le(tilemap + 14 * i + 2 + 2) + 256;  u8 sx = tilemap[14 * i + 4 + 2];  u8 sy = tilemap[14 * i + 5 + 2];  u8 atr1 = tilemap[14 * i + 6 + 2];  u8 atr2 = tilemap[14 * i + 7 + 2];  // printf("%2.2x %2.2x\n", atr1, atr2);    glBindTexture(GL_TEXTURE_2D, getTexture(atr1, atr2, sx));    glBegin(GL_QUADS);  glTexCoord2i(sx, sy); glVertex2i(tx, ty);  glTexCoord2i(sx+15, sy); glVertex2i(tx+16, ty);  glTexCoord2i(sx+15, sy+15); glVertex2i(tx+16, ty+16);  glTexCoord2i(sx, sy+15); glVertex2i(tx, ty+16);  glEnd(); }#endif}CFieldBackground g_FieldBackground;// - OpenGL stuff --------------------------------------------------------------void ReshapeWindowFunc(int width, int height){ glViewport(0, 0, width, height); glMatrixMode(GL_PROJECTION); glLoadIdentity(); glOrtho(0.0, width, height, 0.0, 0.0, 1.0);}void MainRenderLoop(void){ // render glClearColor(0, 0, 0, 1.00); glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); g_FieldBackground.draw();  glFlush();}void SpecialHandler(int key, int x, int y){#if 0 switch (key) {  case GLUT_KEY_UP:   rot_x -= 4;   if (rot_x < -64) {     rot_x = -64;    }    break;  case GLUT_KEY_DOWN:   rot_x += 4;   if (rot_x > 64) {     rot_x = 64;    }    break;  case GLUT_KEY_LEFT:   rot_y -= 5;   break;  case GLUT_KEY_RIGHT:   rot_y += 5;   break; } glutPostRedisplay();#endif}void KeyboardHandler(unsigned char key, int x, int y){ switch (key) {  case 27:   exit(0);   break; }}int main(int argc, char *argv[]) { if (argc != 2) {  fprintf(stderr, "fieldview <field.dat>\n"); } else {  // get current directory  char cwd[1024];  getcwd(cwd, sizeof(cwd));    // init opengl and glut  glutInit(&argc, (char **) argv);  glutInitWindowSize(640, 480);  glutInitDisplayMode(GLUT_SINGLE | GLUT_RGBA | GLUT_DEPTH);  glutCreateWindow(argv[0]);  glutDisplayFunc(MainRenderLoop);  glutIdleFunc(MainRenderLoop);  glutReshapeFunc(ReshapeWindowFunc);  glutKeyboardFunc(KeyboardHandler);  glutSpecialFunc(SpecialHandler);    // Setup OpenGL  glDisable(GL_LIGHTING);  glDisable(GL_DEPTH_TEST);  glDisable(GL_CULL_FACE);  glEnable(GL_TEXTURE_2D);  glEnable(GL_BLEND);  glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);  glEnable(GL_ALPHA_TEST);  glAlphaFunc(GL_GREATER, 0.0f /*1.0f/255.0f*/);    glDisable(GL_LINE_SMOOTH);    glShadeModel(GL_SMOOTH);    glMatrixMode(GL_MODELVIEW);  glLoadIdentity();    glMatrixMode(GL_TEXTURE);  glLoadIdentity();  glScalef(1.0f/256.0f, 1.0f/256.0f, 1);  glTranslatef(0.5f, 0.5f, 0.0f);    // Setup projection matrices  glViewport(0, 0, 640, 480);    glMatrixMode(GL_PROJECTION);  glLoadIdentity();  glOrtho(0.0, 640, 480, 0.0, 0.0, 1.0);    // if the filename is not absolute try to add the current work directory.  string path;  if (argv[1][0] != 0 && argv[1][0] != '/' && argv[1][1] != 0 && argv[1][1] != ':' && argv[1][2] != 0 && argv[1][2] != '\\') {   path = string(cwd) + "/";  }    // load the background data  g_FieldBackground.load(path + argv[1]);    // main (this will never return)  glutMainLoop(); } return 0;}
 
Hey Micky have you tried OpenGLUT? You can link it with your programs. It is under the MIT license.
 
Hmmm it seems 28 bytes was my problem go figure. Alignment is now somewhat sane, although computing the Width and Height has yet to be completed (since I keep having to reboot). I think I need to reinstall Borland C++ :/

Cyb
 
Thanks Micky your information got me this far.
4SBWY_1
http://www.geocities.com/cyberman_phillips/TimView/4SBWY_1.jpg
ANFRST_1
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1.jpg

Any interesting note there appear to be a few sections to this data likely sprite stuff et al. Micky's data of course is incomplete. Thanks to him I can at least view this much as well as find the dimensions of the Thing.
What I do is iterate through the tiles given to get Xmax and Xmin and Ymax and Ymin.
Then I size a bitmap and create a second one. The second bitmap contains all the tiles which are rendered as needed. When done I just copy a 16x16 block from the rendered tile palette to the drawing and this gets you the image.  White areas have no tiles associated with them in this view.  As you can see no sprite data is shown either.

The interesting thing is the values for the tiles aren't 0,0 as the upper left corner instead 0,0 is somewhere in the screen. My guess is that is the point is the origin of the screen in relative coordinates. Likely it relates to the walkmesh data after perspective transformation in the GTE.
For "fun" I will have to change this to an OpenGL view and add the walk mesh drawn on top of it. I believe the walk mesh data to be identical to that of the PC's so I will use what's available here for that (anyone for wire mesh view overlay?) I suppose I could just draw it overtop instead of using OpenGL (might make it easier since my OpenGL code still isn't working right).

Cyb
 
Hey Micky have you tried OpenGLUT? You can link it with your programs. It is under the MIT license.
Click to expand...
I selected GLUT mostly because it is already available on a lot of systems (Mac + Linux) or can be easily added (windows).
I don't think I need any of the features of OpenGLUT at the moment.
For the first one, try converting pure blue to no alpha.
Click to expand...
I think the green/blue areas are overwritten by higher layers.
I suppose I could just draw it overtop instead of using OpenGL (might make it easier since my OpenGL code still isn't working right).
Click to expand...
That should be quite easy with OpenGL. Just lock the Z-Buffer while drawing the background, then enable it for the mesh.
Did you find the correct algorithm for selecting palettes?
 
I wish I could say "SURE" save my windows computer DIED horribly and I'm backing data up from it as I type this. It means a install of win2K and all those fun security updates then reinstall all that software. This could take a little while I gave up doing a normal install over the stupid thing tell it works. So now I'm working on a wipe clean the slate of ye olde boot partition and install. SIGH.. I've got so much to back up.. thank goodness I have a whole disk to dump stuff into. (it had 30gigs cleared ... had) still busily backing up. Network is toast unless someone knows how to get the network up in 'DOS' setup mode.

So... hopefully I can back up everything and I won't have to start from scratch :D

Cyb
 
I've rebuilt enough of the application I've returned to this point.  (Weee)
Unfortunately it will be a week before the backgrounds are as good as they were LOL. Such is life I guess. In any case there appears to be lots of other data in that section you (Micky) have been using to extra data. Might be sprite data you think?
The information on the display appears to be segmented in groups of 64 horizontal pixels with an associtated palette. This comes out to be 128x256 sections.
The playstation has a display area of 1024 x 512.  The palette for the background appears to be at 0, 480 (leaving roughly 32 palettes possble), obviously the display for the background starts at 0,0.  At 384 (pixels) the actual background tile info starts. This gives a max of 1024-384= 640 pixels. OR 10 128x256 tile sets. with possibly 20 palettes. (hmmm).

I digress. I've made progress obviously (grin). however the offset computation to the tile set seems to not be quite right (wee). I guess it needs work.  I think how it works is something like this you have 640x256 physical pixels Or 1280x256 8 bit pixels in the tile set max. I believe they organized it in 128 8 bit pixel tile sets. However the pitch of the MIM data comes into play here. The palette is choosen depending if the 'page' exceeds the end pitch I think.  That's why if it's > 512 it wraps. However you can have larger tile sets than that so.. I think the palette count relates to the number of 128x128 sections that are used but starts top left to right and increases then (Y wise as you step through the palettes) by the horizontal pitch /64 to the next row. That's my thinking at the moment.

Cyb
 
First Micky you should be using a UINT16 for the attribute information instead of 2 bytes.
Second the page appears to use bits  6-10 or 4-10 this makes for a 5 or 7 bit field they are using for the page # or whatever it might be.  It also appears that using the scheme you have picks out every OTHER section of data. IE it alternates blocks.  Palette finding is not working worth beans with the current scheme. You can only find the first palette so I need to twiddle around more and might resort to some scheme to allow me to tweak the palette used for each section of tiles, so they look like something.
I suspect the alternate block sections between are sprite data or better put non background data. In any case I'm making 'steady' progress in getting the background data viewable, the palettes appear to be the tricky part. :D
 
Ok progress has moved along some
Here is ANFRST_1 again however I've included a tile image.
This pretty much looks identical as before, however I hand tweaked the palettes to look at the tiles.

Notes:
Some tiles are wrong, this means the method used for selecting tiles is .. wrong as well :D Edited <-- this has been fixed and the image reflects that. I used my Tile tracing code to find out where it was grabing the tiles from. Such is life! :D
This likely means I'll have to do some sort of forced tweaking of what tile sets are used with tile collection bits in the attribute word. That might be complicated but if you can rearrange things so they are correct then, you'll learn how it's supposed to work is my theory (it's how I am learning how to select the correct palettes at least).
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1_Image.jpg
Palettes:
Assume that the tiles are actually in 128x128 sections and in this case are arranged in an array of 10x2  (10 on top 10 on bottom).  Assume the top row is 0 -9 and the bottom row is 10-19 (tile set number). The palette ordering is (in tile set:palette order), 0:0. 1:0, 2:1, 3:1, 4:2, 5:3, 6:4, 7:5, 8:6, 9:7, 10:0, 11:0, 12:1, 13:1, 14:2, 15:8, 16:4, 17:9, 18:0, 19:7.  It's interesting to note there are 14 palettes and yet 4 aren't used. The tiles use palettes 0-9 what are 10 - 13 used for? I guess I'll have to check if there are 2 images in file.
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1_Tiles.jpg

Other notes:
The attribute word in the tile mapping structure does a few things however I've yet to know what it's all ment for however it appears bits 10-6 are the only ones used for tiles, palette information might be somewhere else (maybe).

Cyb
 
Sorry about that, Cyberman, but I had to replace your img-tags with url-tags, because I had to scroll horizontally at every single line I wanted to read (although I'm using 1024x768 screen resolution)

 - Alhexx
 
Hmmm well I suppose 1180 and 1280 size bitmaps might be an issue :D

LOL.

Cyb
 
Status
Not open for further replies.
Back
Top