Blades Save File Format

[Celtic Minstrel]

Stareye, you mentioned the idea of an "upgrade until reaching current version" system of converting older format files. I can't think of a way to do this without keeping every single version of the relevant structures around, though. Is there such a way that you've come up with?

[Niemand]

You might have to keep old versions of the structures and surrounding code, but that's what namespaces are for. Your alternative is that everytime you update the format, you have to write a separate function, from scratch, to convert each and every old format to the newest one. Using the approach Stareye suggests, you just add one new function/set of functions, and let the data cascade through the code you've already written until it reaches a state you're ready to use. i'm not sure either way is really much less code, but the cascading approach is nice because it lets you leverage what you've already done (hopefully correctly) to minimize additional work and minimize the introduction of new bugs.

 

Besides all that, the format probably shouldn't change frequently at all. This is the kind of thing for which suggested changes should be considered and accumulated for some time until you have enough of them to justify rolling out a new system.

[*i]

Not especially. That is part of the reason why I was suggesting a keyword/argument way of doing things. Read a line, find the keyword that indicates what the line is, go to the relevant section of code, read the arguments, and place it in the relevant arrays/structures.

 

This makes the format more extensible because it would be mostly independent of record order, array sizes, or contents. If later someone decides to switch things around, the parser would not care about the order. Sure, it's kind of a pain now, but doing this will pay off in the long run.

 

Sure, the file would be written out in a specific order; however, the input would inherently not care about anything except for the first few tokens which give some fundamental information about the version of the file.

 

Also, to save space for large things like the SDF array, you might want to pick some common sense defaults (zero in this case). Only write things to the save file if they differ. Most scenarios only use a small fraction of the SDF array, so there is no need to write out a bunch of zeros.

 

There would be a pain if someone were to modify the fundamental definition of something (say someone for some weird reason decides to make 3D SDFs). The best way around this would be to invent a new keyword for it and just have the new format be a superset of the old one.

[King InuYasha]

Originally Posted By: Niemand

I would suggest that you go ahead with Quicktime. I'm sure Windows has native libraries to load and display PNG images as well, so there doesn't seem to be any reason to do something really low level like building in libpng.

I don't think that it would work very well to try to attack the old code to UI elements stored in a nib file. I'm not even sure if you can get the controls stored that way into an antiquated enough form since Jeff used the Control Manager, rather than the more modern HIView library. I looked at that, and decided that nothing short of a full conversion to nibs and custom controls seemed sensible. I've done a little work in that direction, but it won't be useful for some time to come.

I agree one the campaign flags, my first thought was to also store them by scenario name. I think that this has a couple of advantages: Firstly, the designer doesn't have to worry about choosing a key. Secondly, scenario names are already essentially unique, so one doesn't need to worry too much about collisions. Having the stored data also be a dictionary would also be nice, although designers could probably deal with having to index their data numerically.

I would like to argue that Windows does NOT have a native method of handling PNGs properly.

Also, instead of relying on QuickTime or whatever, why not use libpng? By using libpng, you reduce the overall differences in code by using a cross-platform library. Try to use cross-platform methods over platform-specific ones, that would make it easier in the long run to consolidate the sources down to a single tree.

[Celtic Minstrel]

Hm. I suppose if the format change only involved adding some fields, there would be no need to keep the old structure – conversion could happen in-place...

 

Of course, the format should change as infrequently as possible.

 

 

The only other thing I need to do now is figure out how to use gzip and tar from the code. Or I could use zlib... or is that the same thing?

[Niemand]

You want zlib. (There's not need for a tar step if you only have one file anyway.)

 

@King InuYasha: So you want to add in an entire new code library as a dependency, so that you can write a few common lines of code to load the data, then write a a few platform specific lines of data to hand off the data to the graphics toolkit, instead of just writing a few platform specific lines to have the graphics library load the data (Or in the Mac case, specifically, have a couple of lines to have one native library load the data and a couple of lines to give it to the native graphics library)? That's making thing more complicated without making the code any more portable, unless you want to someday port it to a platform with no native handling for PNGs.

 

EDIT: With regard to the assertion that PNGs can't be loaded with a native Windows libraries:

Google Search for: windows GDI+ Image PNG

The second hit is http://www.experts-exchange.com/Programming/Languages/CPP/Q_23393732.html. Since it's stupid Experts Exchange, setting one's user agent string to match google-bot's is sufficient to trick it into displaying the information which includes an example:

Code:

void Paint(HWND hWnd, HDC hDC){	Graphics	graphics(hDC);	Rect		rect(50, 50, 300, 600);	RECT		cRect;	Rect		CRect;	Image		* pImage = Image::FromFile( L"C:\\download\\Visine4.png"); 	graphics.SetSmoothingMode(SmoothingModeAntiAlias);	graphics.SetTextRenderingHint(TextRenderingHintAntiAlias);	graphics.SetCompositingMode(CompositingModeSourceOver);	graphics.SetCompositingQuality(CompositingQualityGammaCorrected);	graphics.SetInterpolationMode(InterpolationModeHighQualityBicubic);//	mGraphics->SetPageUnit(mMeasurementType); 	GetClientRect(hWnd, &cRect); 	CRect.Width	= cRect.right;	CRect.Height	= cRect.bottom; 	graphics.FillRectangle(&SolidBrush(Color(252, 217, 211)), CRect);	graphics.DrawImage(pImage, rect); 	delete pImage;}

Only one line in there was actually loading the image, the rest is all drawing code for the purposes of the example.

[Celtic Minstrel]

Originally Posted By: Niemand

You want zlib. (There's not need for a tar step if you only have one file anyway.)

Except I will have multiple files. For the saved game, I'll have at least the main party file as well as the graphics file, and I may decide to store each PC in a separate file. And some other data (maps?) may be easier to handle in a separate file.

So yes, I do need a tar step.

[Niemand]

Ack. That makes things harder. Still, not impossible. The first place to look at is the Wikipedia page,which seems to cover the tar format pretty well. What you'll then need to do is this:

-build the data that will make up each 'file' as a block in memory

-to each block of 'file' data attach a tar file header

-concatenate all of these blocks together

-gzip the entire big block, and write it to disk

(If I understand the zlib documentation correctly the way it will actually work can be a bit nicer, as gzwrite() appears to be able to write out blocks of data one by one to the same file, so you could instead have a cycle of composing the data for a 'file', attching its tar header, and handing it off to gzwrite().)

 

At any rate, here's a test program I wrote just now which which creates about the simplest possible tarball. generateHeader() could be made a bit more sophisticated to do things like try to split long file paths, but it should suffice for what I think you need.

Click to reveal..

Code:

#include <iostream>#include <cstdio>#include <cstring>#include <string>#include <stdexcept>#include <ctime>// from http://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5&manpath=FreeBSD+8-currentstruct header_posix_ustar {		   char name[100];		   char mode[8];		   char uid[8];		   char gid[8];		   char size[12];		   char mtime[12];		   char checksum[8];		   char typeflag[1];		   char linkname[100];		   char magic[6];		   char version[2];		   char uname[32];		   char gname[32];		   char devmajor[8];		   char devminor[8];		   char prefix[155];		   char pad[12];	   };header_posix_ustar generateHeader(const std::string& fileName, unsigned long long fileSize,bool directory=false){	if(fileSize>077777777777LL)		throw std::length_error("Specified file size >= 8 GB");	if(fileName.length()>=100)		throw std::length_error("Specified file name longer than 99 characters.");	header_posix_ustar header;		sprintf(header.name,"%s",fileName.c_str());	sprintf(header.mode,"%07o",0600);	//leave uid filled with NULs	//leave gid filled with NULs	sprintf(header.size,"%011o",fileSize);	sprintf(header.mtime,"%011o",time(NULL));	memset(header.checksum,' ',8);	header.typeflag[0]=directory?'5':'0';	//leave linkname filled with NULs	sprintf(header.magic,"ustar  "); //overwrites header with " \0"	//leave uname filled with NULs	//leave gname filled with NULs	//leave devmajor filled with NULs	//leave devminor filled with NULs	//leave prefix filled with NULs	//leave pad filled with NULs		unsigned int sum=0;	unsigned char* ptr=reinterpret_cast<unsigned char*>(&header);	for(unsigned int i=0; i<sizeof(header); i++){		sum+=ptr[i];	}	if(sum>0777777)		throw std::runtime_error("Checksum overflow");	sprintf(header.checksum,"%o",sum);	return(header);}#include <fstream>int main(){	std::string str="Hello World!";		std::ofstream outfile("test.tar");		header_posix_ustar header=generateHeader("test.txt",str.length()+1);		outfile.write((const char*)&header,sizeof(header));	outfile.write((const char*)str.c_str(),str.length()+1);	//write the footer to end the file	for(unsigned int i=0; i<2*sizeof(header_posix_ustar); i++)		outfile.put('\0');	outfile.close();		return(0);}

[Celtic Minstrel]

Whoa, thanks. I shall start writing that save function then.

[Celtic Minstrel]

Okay... the "KEYWORD <arguments>" format is great for the scalar elements of the structure, but it's not quite so good for the sub-structures such as boats, horses, journal notes, saved population data for the last four towns, outdoor wandering encounters, etc.

 

For these, there are two possibilities I have thought of. I could simply put the entire sub-structure (together with any sub-sub-structures!) on a single line – a keyword with many arguments. Or, I could define a keyword whose first argument is another keyword, followed by more arguments. Which do you think is better? Does it even matter? Are there any other possibilities?

 

 

Another place it falls short is on the two-dimensional terrain arrays, and on the bitwise arrays (such as item_taken). For the former I simply wrote out the full array as-is, separating elements on with tabs and lines of elements with newlines. The latter I skipped over – I'm still not sure of the best way to handle it. I could do it just like the terrain array, I suppose.

[Celtic Minstrel]

Well, no-one responded, so I arbitrarily decided.

 

For those interested, here is the current draft of the savefile format.

 

Click to reveal.. (Blades of Exile Save Format version 2 beta)

The file is a gzipped tarball containing up to 12 files. Each file consists of one or more blocks of data; if there's more than one block in a file, they are separated by formfeeds (arbitrary choice).

 

The first block in most of the files stores the majority of the information in the form "KEY <arguments>". Subsequent blocks may have different formats.

 

save/party.txt

Stores the party record and related information

Block 1: Most of the info.

Block 2: The setup save (fields for that last 4 visited towns); formatted as 64 lines of 64 tab-separated numbers.

Blocks 3-5: The recorded strings, for journal entries, special encounters, and conversations (each type gets a block, and the order hasn't yet been defined) - alternatively I could split these into a separate file.

Block 6: Any monsters that can be summoned by an item in a player's inventory that was brought from some other scenario.

 

save/pc*.txt

Stores the information about a single PC; * is a number from 1 to 6.

Block 1: All information is stored in this block.

 

save/town.txt

Only if the party is in town; stores various info about the current town.

Block 1: Information such as which town it is.

Block 2: Fields information.

Blocks 3+: Probably one for terrain info and one for creature info. If deemed necessary, one for lighting info could also be used.

 

save/townmaps.dat

Contains the town maps in binary format with no padding. Hence, its size should always be 200*8*64 = 102400 bytes. Only present if map saving is enabled.

 

save/out.txt

Contains info about the current 4 outdoor sections. At present this is just whether it is explored. I'm not sure if there's any other info that needs storing here. Only present if the party is in a scenario.

 

save/outmaps.txt

Contains the town maps in binary format with no padding. Hence, its size should always be 100*6*48 = 28800 bytes. Only present if map saving is enabled.

 

save/graphics.png

This isn't yet implemented, but it will contain custom graphics required for items, monsters, and even PCs taken from other scenarios. It will be 280 pixels wide by 900 pixels tall for a total of 250 terrain-sized slots.

[Niemand]

Form the save/ prefixes, I take it you decided to put a directory inside the tarball, and the files inside that? I was going to recommend doing so, but forgot to.

 

Is there some mechanism to denote what internal formatting a block will use? (That is, when the loading procedure encounters a block, how will it know how to decode it?) This shouldn't be difficult, just requiring a tiny header on each block.

 

Also, is there a particular purpose to putting multiple blocks in a file? The only one that I can see at a glance is lower overhead, since a block header can be much smaller than the 512 byte file header the tar format requires.

[Celtic Minstrel]

Yes, I decided to put a directory in the tarball. I think it was recommended somewhere that I read.

 

Originally Posted By: Niemand

Is there some mechanism to denote what internal formatting a block will use? (That is, when the loading procedure encounters a block, how will it know how to decode it?) This shouldn't be difficult, just requiring a tiny header on each block.

I was just going to make the blocks have a defined order, so its formatting depends on its position in the file. I could put a small header instead, though.

 

Originally Posted By: Niemand

Also, is there a particular purpose to putting multiple blocks in a file? The only one that I can see at a glance is lower overhead, since a block header can be much smaller than the 512 byte file header the tar format requires.

Mainly just to keep related information together. Lower overhead is a nice bonus, I guess.

 

 

 

Also, I forgot to mention the header in the previous post – it's currently 10 bytes:

1. 0x0B0E
2. 1432 if the party is in town, otherwise 5790
3. 100 if the party is in a scenario, otherwise 200
4. 5567 if maps are saved, otherwise 3422
5. Version number in 0xMMmm format (first new-format version is 2.0, so 0x0200)

 

If there's enough demand, and it seems necessary, I may add two more flags to indicate whether custom graphics are present, and whether journal strings are present.