I am trying to reverse engineer an old Wolf3D style Dos game named "Nitemare 3D". I can get a raycasting engine with directional sprites running pretty easily using SDL2, but I have no experience with opening file archives. There is an old game file extractor that sadly does not have any available source code. Does anyone know how I can read these files with c or c++?
-
1You know SE has also reverseengineering.stackexchange.com site... What exactly you want to extract? sprites/textures , maps, sound ? Do you have some screen shots of the game and or more info about it (like screen size etc ... that might narrow down thesearch properties) ...– SpektreAug 11, 2019 at 6:59
-
1btw for locating and determinig encoding of images inside program files I used PV back in the MS-DOS days... it was superior any other tool I saw back then...– SpektreAug 11, 2019 at 8:41
-
I have updated my answer with new findings in the file-format and I also finished the decoder now its matching the game screenshots completely...– SpektreAug 12, 2019 at 11:31
2 Answers
Read and extract SND.DAT
Code is C# but is easy enough to understand and port to another language.
Basically there are 6 bytes entries that tells where is what in the file, some of them are dummy.
I have already identified for you the different types of files in it:
IBK
MID
VOC
Winamp will play MIDI files out of the box.
VLC will play VOC files properly unlike Winamp.
For IBK files check these links:
http://www.shikadi.net/moddingwiki/IBK_Format
https://github.com/Wohlstand/OPL3BankEditor
Code:
using System;
using System.Collections.Generic;
using System.IO;
using JetBrains.Annotations;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace UnitTestProject1
{
[TestClass]
public class UnitTest1
{
[TestMethod]
public void TestMethod1()
{
const string directory = @"C:\Temp\nite3d20";
var path = Path.Combine(directory, "SND.DAT");
using (var stream = File.OpenRead(path))
using (var reader = new BinaryReader(stream))
{
stream.Position = 0;
var samples = new List<Sample>();
while (true)
{
var sample = new Sample(reader);
if (sample.Position == stream.Length)
break;
samples.Add(sample);
}
var index = 0;
foreach (var sample in samples)
{
string extension;
switch (index)
{
case 0:
extension = "IBK";
break;
case int n when n >= 1 && n <= 15:
extension = "MID";
break;
case int n when n >= 34 && n <= 110:
extension = "VOC";
break;
default:
extension = ".BIN";
break;
}
var name = $"{path}.{index++:D3}.{extension}";
if (sample.Length == 0)
continue;
stream.Position = sample.Position;
var bytes = reader.ReadBytes(sample.Length);
File.WriteAllBytes(name, bytes);
}
}
}
}
public struct Sample
{
public readonly short Length;
public readonly int Position;
public Sample([NotNull] BinaryReader reader)
{
if (reader == null)
throw new ArgumentNullException(nameof(reader));
Length = reader.ReadInt16();
Position = reader.ReadInt32();
}
public override string ToString()
{
return $"{nameof(Position)}: {Position}, {nameof(Length)}: {Length}";
}
}
}
You will end up with the following files:
Directory of C:\Temp\nite3d20\New Folder With Items
[.] [..] SND.DAT.000.IBK SND.DAT.001.MID SND.DAT.002.MID SND.DAT.003.MID
SND.DAT.004.MID SND.DAT.005.MID SND.DAT.006.MID SND.DAT.007.MID SND.DAT.008.MID SND.DAT.009.MID
SND.DAT.010.MID SND.DAT.011.MID SND.DAT.012.MID SND.DAT.013.MID SND.DAT.014.MID SND.DAT.015.MID
SND.DAT.034.VOC SND.DAT.035.VOC SND.DAT.036.VOC SND.DAT.037.VOC SND.DAT.038.VOC SND.DAT.039.VOC
SND.DAT.040.VOC SND.DAT.041.VOC SND.DAT.042.VOC SND.DAT.043.VOC SND.DAT.044.VOC SND.DAT.045.VOC
SND.DAT.046.VOC SND.DAT.047.VOC SND.DAT.048.VOC SND.DAT.049.VOC SND.DAT.050.VOC SND.DAT.051.VOC
SND.DAT.052.VOC SND.DAT.053.VOC SND.DAT.054.VOC SND.DAT.055.VOC SND.DAT.056.VOC SND.DAT.057.VOC
SND.DAT.058.VOC SND.DAT.059.VOC SND.DAT.060.VOC SND.DAT.061.VOC SND.DAT.063.VOC SND.DAT.064.VOC
SND.DAT.065.VOC SND.DAT.066.VOC SND.DAT.067.VOC SND.DAT.068.VOC SND.DAT.069.VOC SND.DAT.070.VOC
SND.DAT.071.VOC SND.DAT.073.VOC SND.DAT.074.VOC SND.DAT.078.VOC SND.DAT.079.VOC SND.DAT.080.VOC
SND.DAT.081.VOC SND.DAT.082.VOC SND.DAT.083.VOC SND.DAT.084.VOC SND.DAT.085.VOC SND.DAT.086.VOC
SND.DAT.087.VOC SND.DAT.088.VOC SND.DAT.089.VOC SND.DAT.090.VOC SND.DAT.091.VOC SND.DAT.092.VOC
SND.DAT.093.VOC SND.DAT.094.VOC SND.DAT.095.VOC SND.DAT.096.VOC SND.DAT.097.VOC SND.DAT.098.VOC
SND.DAT.099.VOC SND.DAT.100.VOC SND.DAT.101.VOC SND.DAT.102.VOC SND.DAT.103.VOC SND.DAT.104.VOC
SND.DAT.105.VOC SND.DAT.106.VOC SND.DAT.107.VOC SND.DAT.108.VOC SND.DAT.109.VOC SND.DAT.110.VOC
88 File(s) 560,142 bytes
Note that I increment index before potentially skipping it, you then end up with the intended index for it, i.e. you might find relevant info on other files and they should point at correct index, else adjust.
Better DAT reading algorithm
In fact, DAT files share the same structs describing where is what:
UINT16LE Length
UINT32LE Position
This will read and extract DATs into subdirectories:
What changes from previous code is members in struct and the logic to infer last entry in that FAT.
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using JetBrains.Annotations;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace UnitTestProject1
{
[TestClass]
public class UnitTest1
{
[NotNull]
private static Entry[] GetEntries([NotNull] Stream stream)
{
if (stream == null)
throw new ArgumentNullException(nameof(stream));
using (var reader = new BinaryReader(stream, Encoding.Default, true))
{
var entries = new List<Entry>();
while (true)
{
var entry = new Entry(reader);
if (entry.Position + entry.Length == stream.Length)
break;
entries.Add(entry);
}
return entries.ToArray();
}
}
[NotNull]
private static string GetPath([NotNull] string fileName)
{
if (fileName == null)
throw new ArgumentNullException(nameof(fileName));
const string directory = @"C:\Temp\nite3d20";
var path = Path.Combine(directory, fileName);
return path;
}
[TestMethod]
public void Extract_UIF_DAT()
{
Extract("UIF.DAT");
}
[TestMethod]
public void Extract_SND_DAT()
{
Extract("SND.DAT");
}
private static void Extract([NotNull] string fileName)
{
if (fileName == null)
throw new ArgumentNullException(nameof(fileName));
var path = GetPath(fileName);
using (var stream = File.OpenRead(path))
using (var reader = new BinaryReader(stream))
{
var entries = GetEntries(stream);
var directory = Directory.CreateDirectory($"{path} content");
var index = 0;
foreach (var entry in entries)
{
stream.Position = entry.Position;
var entryPath = Path.Combine(directory.FullName, $"{index++:D3}.BIN");
var entryData = reader.ReadBytes(entry.Length);
File.WriteAllBytes(entryPath, entryData);
}
}
}
}
public struct Entry
{
public readonly ushort Length;
public readonly uint Position;
public Entry([NotNull] BinaryReader reader)
{
if (reader == null)
throw new ArgumentNullException(nameof(reader));
Length = reader.ReadUInt16();
Position = reader.ReadUInt32();
}
public override string ToString()
{
return $"{nameof(Position)}: {Position}, {nameof(Length)}: {Length}";
}
}
}
Unpacking the executable
Use the following to unpack it:
https://github.com/joncampbell123/dosbox-x
Note that the game won't run when you extract it, complaining that the EXE is corrupt.
And I guess that with the free version of IDA decompiler, you will be able to see many interesting strings in it.
UIF.DAT images
Each of the files in it are images as some strings in EXE tells, they probably use some form of RLE encoding.
Looking at their header, I think the following could be valid:
Then the palette is 768 bytes after image data highlighted in yellow.
Good luck for your game!
-
Dude, Awesome! I can create a c++ version as soon as I get home, my only question(sorry, new to reading files), what does "case int n when n >= 1 && n <= 15" do? I know it identifies the file type but what does it actually mean? Aug 14, 2019 at 14:58
-
It is a shortcut syntax to check 'index' value inside the switch block!– aybeAug 14, 2019 at 16:00
-
-
-
The IMG.*
files contains all textures and sprites in uncompressed/unencrypted state so its very easy to render, but there is no VGA palette there (nor in other files you posted). I found this:
and inside the game zip I found GAME.PAL
file containing the palette ... Its 1924 Bytes
long and its look like the palette for IMG.*
sprites is at offset 1156 dec
till end of file.
The IMG.*
file format is as follows:
0000: DWORD 00000000 magic ID?
// directory
0004: DWORD 0000BC00 1st image start
0008: DWORD 2nd image start
000C: DWORD 3th image start
...
// 1st image
BC00: BYTE xs
BC01: BYTE ys
BC02: BYTE unknown[8]
BC0A: BYTE pixels[xs*ys]
// 2nd image
...
However the directory has some hacks in it I did not yet recognize so its safer to scan for image by incrementing start address of image by 10+xs*ys
to move to next image (until end of file reached). I decoded all the images starts and create own linear directory to speed up the rendering and usage instead (see the code).
The pixels are stored 8bpp 256 color pixels in column major order !!! Most likely to speed-up/simplify the ray-casting engine Scanlines. This is my test code in C++/VCL to render it:
//$$---- Form CPP ----
//---------------------------------------------------------------------------
#include <vcl.h>
#pragma hdrstop
#include "Unit1.h"
//---------------------------------------------------------------------------
#pragma package(smart_init)
#pragma resource "*.dfm"
TForm1 *Form1;
Graphics::TBitmap *bmp;
int xs,ys;
//---------------------------------------------------------------------------
class Nightmare3D_img
{
public:
// palette
DWORD pal[256];
// file
BYTE *dat; // file
DWORD *dir;
int siz,num; // filesize, images
// decoded/rendered image
Graphics::TBitmap *bmp;
Nightmare3D_img()
{
dat=NULL; siz=0; num=0; dir=NULL;
bmp=new Graphics::TBitmap;
bmp->HandleType=bmDIB;
bmp->PixelFormat=pf32bit;
}
Nightmare3D_img(Nightmare3D_img& a) { *this=a; }
~Nightmare3D_img() { _free(); }
Nightmare3D_img* operator = (const Nightmare3D_img *a) { *this=*a; return this; }
//Nightmare3D_img* operator = (const Nightmare3D_img &a) { ...copy... return this; }
void _free(){ if (dat) delete[] dat; siz=0; num=0; dir=NULL; }
void load_pal(AnsiString filename) // load palette to memory
{
int hnd,adr,i0,i,j;
BYTE a[1924],*p;
hnd=FileOpen(filename,fmOpenRead); if (hnd<0) return; // open file
FileRead(hnd,a,1924); // load it to memory
FileClose(hnd); // close file
// extract palette
p=(BYTE*)pal;
i0=1156;
for (i=0,j=0;i<3*256;i+=3,j+=4)
{
p[j+0]=a[i0+i+2];
p[j+1]=a[i0+i+1];
p[j+2]=a[i0+i+0];
p[j+3]=0;
}
}
void load(AnsiString filename) // copy file to memory
{
_free();
int hnd,adr,xs,ys;
hnd=FileOpen(filename,fmOpenRead); if (hnd<0) return; // open file
siz=FileSeek(hnd,0,2); // get its size
FileSeek(hnd,0,0); // point to its start
dat=new BYTE[siz];
if (dat) FileRead(hnd,dat,siz); else siz=0; // load it to memory
FileClose(hnd); // close file
// recreate linear directory
dir=(DWORD*)(dat+4);
for (adr=0xBC00,num=0;(adr<siz)&&(num<12030);)
{
dir[num]=adr; num++;
xs=dat[adr+0];
ys=dat[adr+1];
adr+=10+(xs*ys);
}
}
void draw(int ix)
{
if (!siz) return;
DWORD **pyx,c;
int x,y,xs,ys,adr;
// find image start
if ((ix<0)||(ix>=num)) return; // image index out of range
adr=dir[ix];
if ((adr<0)||(adr+2>siz)) return; // end of file check
xs=dat[adr+0];
ys=dat[adr+1];
bmp->SetSize(xs,ys);
pyx=new DWORD*[ys];
for (y=0;y<ys;y++) pyx[y]=(DWORD*)bmp->ScanLine[y];
adr+=10;
x=0; y=0;
for (;adr<siz;)
{
pyx[y][x]=pal[dat[adr]]; adr++;
y++; if (y>=ys){ y=0; x++; if (x>=xs) break; }
}
delete[] pyx;
}
} img;
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
void draw()
{
bmp->Canvas->FillRect(TRect(0,0,xs,ys));
int ix,x=0,y=0,dx,dy,q,w=4;
for (dx=0,dy=0,ix=Form1->sb_img->Position;ix<img.num;ix++)
{
img.draw(ix);
bmp->Canvas->Draw(x,y,img.bmp); x+=w+img.bmp->Width;
q=img.bmp->Width; if (dx<q) dx=q;
q=img.bmp->Height; if (dy<q) dy=q;
if (x>=xs-dx-w){ x=0; y+=w+dy; if (y>=ys-64) break; dx=0; dy=0; }
}
Form1->Caption=AnsiString().sprintf("image: %i-%i/%i ",Form1->sb_img->Position,ix,img.num);
// bmp->SaveToFile("out.bmp");
Form1->Canvas->Draw(0,0,bmp);
}
//---------------------------------------------------------------------------
__fastcall TForm1::TForm1(TComponent* Owner):TForm(Owner)
{
bmp=new Graphics::TBitmap;
bmp->HandleType=bmDIB;
bmp->PixelFormat=pf32bit;
FileListBox1->Directory="..\\Files";
FileListBox1->Mask="IMG.*";
FileListBox1->Update();
img.load_pal("..\\Files\\GAME.PAL");
img.load ("..\\Files\\IMG.1");
}
//---------------------------------------------------------------------------
void __fastcall TForm1::FormPaint(TObject *Sender)
{
draw();
}
//---------------------------------------------------------------------------
void __fastcall TForm1::FormResize(TObject *Sender)
{
xs=sb_img->Left;
ys=ClientHeight;
bmp->Width=xs;
bmp->Height=ys;
draw();
}
//---------------------------------------------------------------------------
void __fastcall TForm1::FormActivate(TObject *Sender)
{
draw();
}
//---------------------------------------------------------------------------
void __fastcall TForm1::FileListBox1Change(TObject *Sender)
{
if (FileListBox1->FileName!="")
{
img.load(FileListBox1->FileName);
draw();
}
}
//---------------------------------------------------------------------------
void __fastcall TForm1::FormDestroy(TObject *Sender)
{
delete bmp;
}
//---------------------------------------------------------------------------
void __fastcall TForm1::sb_imgChange(TObject *Sender)
{
draw();
}
//---------------------------------------------------------------------------
You can ignore the VCL stuff the only important thing is the class ... the img.load
just loads entire IMG.*
file into memory and draw(ix)
render ix
th sprite into img.bmp bitmap.
I have a scroll bar sb_img
that selects which image is rendered first and file list to select which IMG.*
file to load for testing.
Here some previews from IMG.1 file:
If you need help with porting the VCL graphics code see bullets #3,#4 in here:
For the ray-casting see this: