Porting Blades of Exile Scenarios to BoA

[Ishad Nha]

What we really need here is a fully customizable 3D Windows Scenario Editor.

 

Failing that there are some problems that can be fixed by the use of spreadsheets:

 

Eliminating the mix – up of x,y coordinates in town special encounters, it seems the coordinates are being wrongly rotated, thus (x,y) becomes (y,x), usually x does not equal y, so this is a massive error. (This problem only afflicts Windows scenarios. A Mac scenario becomes a Windows scenario after being saved by the Windows version of a BoE Editor.)

 

Scenario authors massively customizing terrain slots 122:169. The BoA Editor has its porting routine based upon a totally standard scenario, one with no customization of terrain. Slots 122 thru 169 are the wall slots, this really leads to problems with porting.

 

Other minor problems can be reported by the spreadsheet so action can be taken in the BoA scenario after porting. For example, BoE magically locked doors are always assigned the wrong terrain script upon porting to BoA.

 

One approach is to create a special BoE porting scenario where the terrain has been rearranged to avoid problems on porting.

 

This approach should work, I have had success creating spreadsheets that can create BoE scenarios out of Exile 2 and Exile 3. Ditto creating BoA scenarios out of Avernum 1 through Avernum 3.

[Niemand]

Step 1: Somebody get the Windows 3D Editor compiling, then talk to me about changes I've made to the Mac version or changes they want to make.

 

I took a stab at working on the Windows editor a long while ago, but I really didn't have appropriate tools, namely a relatively new PC or IDE. Still not having a suitable PC, it is far more practical for someone else to do this than me. In addition, I hardly have the time these days to do much serious hacking on the code myself.

 

The first problem above should be trivial, with a clear bug report from a user detailing exactly what goes wrong this should take mere minutes to fix. Likewise, the third problem is trivial.

 

The difficulty with the second problem is that the user must be given means to specify BoA terrain definitions to use in place of BoE terrains by number. It seems to me that the best way would be to allow the user to add to the BoE scenario a configuration file much like a BoA data script to supply new terrain definitions. The editor would then using the mapping supplied in this file and insert the given definitions into the newly created data script.

[Ishad Nha]

I have given up on the blasted Windows version, I have made no progress whatsoever, I am still hitting twelve pages of errors on the Dev-C++. We really need a Windows programmer like Ormus here.

[Niemand]

There's no reason to be daunted by the large number of errors; likely if you fix a couple of things many of them will go away. At one point I recall getting over a thousand errors and warnings from the BoE character editor. Given that the code is fundamentally usable, sheer number of errors doesn't mean too much.

[Ishad Nha]

Like most people on these boards I lack the knowledge to fix anything. I can handle Avernumscript, I have also downloaded notes on C programming but I really don't know enough about it.

[nikki.]

I'm going to see if I can compile the Windows code into a working editor this Bank Holiday weekend. I don't know nearly enough C to do much, but I know a little, so hopefully...

[Ishad Nha]

For a brief history of my attempts to compile the Windows Source Code see:

 

Compiling Windows Editor

[Ishad Nha]

In the meantime I have created a spreadsheet that enables correction of the known errors:

 

http://www.freewebs.com/ishadnha/WindowsPortingLargeTown.zip

 

(this address must be manually copied and pasted into your address bar.)

 

How it works:

Port the scenario to BoA

For each town, copy the BoA town entry from the bas file into the BoA Large Towns sheet

Copy the BoE town entry from the exs file into the BoE Large Towns sheet

Make any needed changes

Copy back into the bas file.

 

By its nature it can't handle a bad choice of calls used to translate various BoE special states. Jeff did make a few bad or dubious choices there.

 

 

 

 

 

[Ishad Nha]

Why does the ported terrain frequently look like such a mess?

 

My understanding is that the terrain porting process is tough to design. Currently the translation checks for terrain immediately next to the given square x,y in the relevant BoE town. It only checks to see if the adjoining terrain is wall or non-wall. Thus it can't tell if a north wall or a south wall is needed. Ditto it can't distinguish between the need for a west wall and an east wall.

 

For simple free - standing buildings you could have an algorithm that travels along the wall to find corners.

 

Personally I doubt that the process can ever be fully automated.

[Ishad Nha]

Brilliant idea for porting:

The editor has quite a few glitches in how it ports scenarios, especially in the towns. I can't program so I can't alter the source code to reflect my tastes.

 

Now the BoA town entries are well within the ability of an Excel spreadsheet to handle. Thus the idea is to port it as usual. Then run one town record through a spreadsheet at a time. Said spreadsheet will have access to both the BoA town record from the bas file and the original BoE town record.

[Niemand]

You could do this, but I would discourage it. This not what Excel is intended to do, and Excel will always be a weaker tool for doing this than the simple C framework provided by the editor. The C program has access to the data in its proper form, as fields of a complex structure, while Excel as you are using it only has raw binary data which you must manually split apart into fields of interest.

 

You say that you can't program, but I would argue that you are already doing so, although perhaps in a rudimentary fashion, using Excel's scripting. I think that without much difficulty you can in a few hours learn the C/C++ you need to be able to hack on the porting portion of the BoA Editor; for the most part it uses only fairly rudimentary language features. At a glance, I think you would need to know about arithmetic, variables, the if, for, and switch flow control mechanisms, arrays, and structs.

 

You can of course keep doing things your way using Excel, but I suggest that you give working with the actual editor a try as it has a great deal of potential that you currently miss out on. Looking at this thread and the other, which you mentioned a few posts ago, you were having some difficulty getting the crufty old code to compile at all. If that's still the case I would be happy to try to help further; since discussion related to that may become verbose and technical, please feel free to email me (niemandcw at gmail dot com) about it.

 

PS: Although I had forgotten how much is sucks, I have gotten Dev C++ running under Wine, and it almost compiles the 3D editor except for a single cryptic error. Who knows, I might manage to do some Windows development after all.

[Celtic Minstrel]

What's the cryptic error?

[Ishad Nha]

I am using Dev C++ 4.9.9.2 and I am having no luck getting anything to compile, that goes for the specimen programs too! (I mean the specimen programs that come with the IDE itself.) I get hit with:

 

[Linker error] undefined reference to `__cpu_features_init'

ld returned 1 exit status

C:\Dev-Cpp\Examples\FileEditor\Makefile.win [build Error] [FileEditor.exe] Error 1

 

 

If you can compile the editor do so. We could experiment with new things the way Ormus is rearranging the Blades of Exile editor. Then we could fix the errors and quirks.

[Celtic Minstrel]

This means that there is a function called "cpu_features_init" which is called but not defined. Judging by the name, I'd say it's probably an internal function used by the compiler, which probably means that you need to link with some library that comes with it. You'll need to delve into project settings, I think.

[Niemand]

The cryptic error I'm getting is:

Quote:

make.exe: *** [3D_BoA_Editor_Windows_private.res] Error -1073741819

which is being given by the command (in the generated makefile):

Quote:

windres.exe -i 3D_BoA_Editor_Windows_private.rc --input-format=rc -o 3D_BoA_Editor_Windows_private.res -O coff

The windres documentation lacks any description of the errors it may give, so I've been browsing through its source code to figure out what might cause this. (That's one advantage of using FOSS software, I suppose. I'd have preferred a clear error message, though.)

The same error occurs when I try to compile the windowed examples that came with the IDE, so I fear it may mean that the problem is with trying to run it with Wine. This is too bad, because it almost works.

 

@Ishad Nha: Third google result: Dev-cpp undefined reference to `__cpu_features_init. Have you tried this?

 

With regard to rearranging the editor code to have a single codebase for both platforms: This would be marvelous, and I'm looking into it. The necessary steps are, I think:

-refactor Mac code to isolate all platform dependent code

-refactor Windows code to isolate platform dependent code

-merge code bases to share all platform independent code

Naturally the first two steps may be done in either order or in parallel, but there are both major undertakings, and I have very little spare time these days for any of this (sadly).

[Ishad Nha]

Good, it works out. There was a topic in the Bloodshed forums but it has yet to be answered.

 

As for the Windows Editor, the new Mac seems to have all sorts of good features, we really do need a Windows equivalent.

 

Edit:

So far I have never been able to make the source code compile, that is why I was using Excel.

 

A spreadsheet can show you everything in a town record, neatly laid out. I don't know whether that is needed or not.

[Niemand]

You got the editor to compile and run? Great! I'll see if I can put together a concise description of the changes required for the undo system I implemented, as it's about the simplest and most platform independent.

[Ishad Nha]

"It works out" meant that I was able to compile the programs that were supplied with the IDE, not the Windows editor. I have never succeeded in compiling the editor at all. This may be why:

 

Project1_private.rc:0: fatal error: when writing output to : Invalid argument

compilation terminated.

[Celtic Minstrel]

Is there a file called "Project1_private.rc" in the editor folder? If not, it may be worth trying Google again.

 

I don't think there should be such a file, actually, unless it's something used internally by the compiler.

[Niemand]

That file name sounds wrong; in my experience it should be 3D_BoA_Editor_Windows_private.rc, an intermediate file that the compiler creates, which should refer to Blscened.rc, which in turn defines things like the game's menus and diologs, analogously to the resource file used by the Mac version.

 

Are you using the code as downloaded from the BoAEdRemake project on Sourceforge?

 

EDIT: I would also like to note that hand tweaking diffs is not fun. Why did somebody have to go and change a bunch of key variable names between the two platforms?

[Ishad Nha]

Initially there is a "Blscened.rc" only, found in the Win3DReleaseSource\Source sub-folder. When you attempt to compile the source some files are generated in the main (Win3DReleaseSource) folder. One of these is the "Project1_private.rc", here the name of the project is Project1, so name means "(Project name)_private.rc".

 

Edit:

Yes I am using the Sourceforge project source code.

 

 

[Niemand]

And are you not using the BoA3DEditor.dev included in that copy of the code? If I understand correctly, you are not, so perhaps that is the source of a misconfiguration.

[Ishad Nha]

Bizarre, surely the world will now come to an end, it compiled!

[Ishad Nha]

Two problems that the import BoE scenario function can't easily handle:

getting the 2D walls translated into an acceptable 3D format

handling the fact that in BoE dialog nodes in one town can be called in any other town.

 

Function replaces the non-directional walls of BoE with east and south facing walls in BoA. I don't think the function can ever finally be automated because you need the town/out zone to look right. Then you need to remember to ensure that the party must walk over placed specials. Then it looks good if bookcases stand against walls, when that was the case in BoE. Only thing that I can think of is to ensure that the location of the BoE walls is plainly marked. The floor under those walls will be type 253, give it an appropriate color and these walls really stand out.

Click to reveal..

This requires alteration to the "short old_ter_to_floor[256] = {...}" table and the deletion of the following code:

if (wall_type == 0)

t_d.floor[j] = t_d.floor[i - 1][j];

else t_d.floor[j] = t_d.floor[j - 1];

Dialog is a potential problem because BoA dialog can only be called from the dialog script of the current town. Dialog can be copied from one script to another, you can renumber the nodes. If you run out of nodes you may be able to replace dialog with messages called from a town or creature script. BoA has only 200 nodes. I don't think there is a way around this.

[Niemand]

Quote:

Dialog is a potential problem because BoA dialog can only be called from the dialog script of the current town.

What do you mean? What you've written is not correct; dialogue can only be contained in the dialogue script, but can be (in fact must be) initiated from other scripts. Do you mean that there's a problem accessing dialogue chains across towns? This is certainly true, and the only solution is to duplicate the necessary nodes. Is this a problem because it is more restrictive than what BoE allowed? If so it would seem that the porting tool would need to examine the graph of talk nodes in the source town and copy all nodes to the town which references them. It would be a pain to get all of the details right, but it doesn't seem like it would actually be a fundamentally hard problem.

[Ishad Nha]

What I am saying is that you can only access one dialog script at a time, the script for the current town, you can't access a dialog script from another town. By contrast, in BoE the entire exs file was loaded at once, so you could access dialog from any town.

Currently the town script gives names for all creatures who have one of the ten personalities for the current town. It gives the personality numbers for any creatures who used dialog for other towns. My copy of Scentext.txt lists all the personality names and numbers at the end of the file. So you know what name to insert.

If you have dialog from many towns in the one script you will need to harmonize contradictory node numbers, that is no problem. Then you will need to update Memory Cell 3 values.

[Celtic Minstrel]

Originally Posted By: Ishad Nha

By contrast, in BoE the entire exs file was loaded at once,

This is false. BoE only loads one town and four outdoor sections at a time, plus the general scenario data. When dialogue from another town is required, it loads that dialog on the fly.

I suspect this would likely not be a serious problem, as long as you can copy dialogue into the scripts of the towns that need it. How many more dialogue nodes does BoA have per town compared with BoE?

[Ishad Nha]

The official figure for BoA is 200 nodes per town dialog script. I don't know if there is any way around this or not. Kel and Niemand might be able to work something out.

In BoE there are only 60 per town, but nodes from other towns can be imported.

[Celtic Minstrel]

So, chances are you'll rarely have a problem with exceeding the node limit. Just find a way to duplicate nodes in whichever towns they are needed in.

[Ishad Nha]

The porting process is purely linear, point A to point B, one town at a time is loaded and ported. There is no provision for going back to a town that has already been ported. In BoE this would be quite natural, in BoA it is not because the latter has no dynamic storage, everything is fixed-length.

[Niemand]

To the best of my knowledge this problem is pretty fundamental to BoE as well. If you want to solve the specific problem that interests you, you're going to have to fix it, at least to a degree.

 

I assume that in BoE it's legal for dialogue in a town to refer to dialog nodes in a later town. Then, the only way to handle the problem correctly is to read all BoE dialogue data into memory, transforming it into whatever intermediate representation is convenient, and only then generate all town dialogue scripts, placing into each script all nodes which are necessary, either because they belonged to that town originally, or because they were copied. It would still be possible to hit the 200 node ceiling, at that point there's nothing you could do but abort the generation of that dialogue script and issue an error message.

 

As long as, within a given town, dialogue can only begin at dialogue nodes which actually belong to that town, you can avoid needing to defer generation of any town data besides the dialogue script. If the above assumption holds, when writing a town dialogue script you would preserve the numbers of all nodes original to that town, correcting only the numbers of those which are copied from other towns (aside: it would be very useful for maintainability that when writing out a node, you write out a comment indicating any other nodes in other files which are duplicates), so that any nodes referred to in the town's town or creature data (essentially, where ever it is that BoE stores what node is used when the party talks to a given creature) would never need to be corrected.

 

The most important question to ask, I would argue, is whether you consider this a worthwhile use of your time. It can certainly be done, but is anyone ever going to use it? There's very little interest in the community in porting BoE scenarios (let's not delude ourselves: it's a very small community to begin with), and there are going to be plenty of other details that really don't port well that will not be fixable in any way, or that are absurdly hard to fix (like the totally different collections of spells, or conveyor belts).

[Dantius]

Originally Posted By: Niemand

The most important question to ask, I would argue, is whether you consider this a worthwhile use of your time. It can certainly be done, but is anyone ever going to use it? There's very little interest in the community in porting BoE scenarios (let's not delude ourselves: it's a very small community to begin with)

I don't know about this. I certainly don't consider myself a member of the Blades community (and I don't think anyone else would, either ), but there are lots of scenarios for BoE that I can't play on my shiny 10000 exabye 4096x3072 pixel 512 bit windows that I would absolutely love to play, and porting to BoA might be enough to reawaken interest in BoE, because people might want to go back to and play the original scenarios.

I would say that porting some of the more popular/better scenarios into BoA would be a good use of time. I just don't know what those scenarios could be. I suppose that once the scenario ratings system gets into full swing, you could just take the best rated scenarios from that and port them, but again, what do I know.

[Ishad Nha]

I think the dynamic arrays of BoE are superior to the fixed-length everything of BoA.

My understanding is that a creature can call dialog from any town in BoE, this is done solely by his personality number. Any town can have its dialog called here. One solution is to load dialog for town i, then check all monsters personalities in town i for external dialog, load the relevant towns and add the dialog. Also, make sure that you don't load the same town twice. Another solution is the one that you mentioned.

If you hit the 200 node you will need to trim dialog.

A lot of features can't be easily ported, that is true. They simply force the use of work-arounds. Any scenario can be ported if you want to devote the time to that. Here we are seeing that BoA has a lot of limitations.

Conveyor belts are a hassle, CM was trying to recreate them in BoA, I don't know if he succeeded or not. Mercifully they don't seem to be all that common from the scenarios I have looked at.

[Niemand]

Quote:

I think the dynamic arrays of BoE are superior to the fixed-length everything of BoA.

My point was that I hadn't noticed any code of that style in BoE. If it's there, good, as for cases like this it holds a lot of advantages.

Quote:

My understanding is that a creature can call dialog from any town in BoE, this is done solely by his personality number. Any town can have its dialog called here.

Okay, then you have to do this the hard way, the way I mentioned is a bit too restricted. You could try to do it as you suggest, randomly accessing towns as needed. Personally, I'd prefer not to do this, as you would have to calculate the right places to jump to to read, and be careful about returning to where you were. My preference would be to just put in the necessary dynamic structures, load the whole thing, so as to have it all at my fingertips while writing the output.

An idea occurs to me: The data structures for BoE have been significantly overhauled, and I wonder if we could make use of them as a drop-in replacement for the current reading code and storage data structures in the BoA editor. It might help us avoid some wheel-reinvention.

@Celtic Minstrel: Can you comment on the merits of that train of thought? Are the new BoE data structures mature enough that we would want to use them? Are things modular enough that we could import that code easily (like via an svn external definition)? While I know there's a new generation of data structures, is the reading code actually new and improved, and in particular would it be any better suited to the purpose of loading the whole scenario at once than what we have?

[Celtic Minstrel]

Originally Posted By: Niemand

I assume that in BoE it's legal for dialogue in a town to refer to dialog nodes in a later town.

In fact, this is not the case. A dialogue node in town A can only lead to more nodes in town A, and all nodes defined in town A must belong to one of the personalities of town A.

However, the town data is allowed to reference personalities from other towns.

So your suggestion probably will not work.

Originally Posted By: Ishad Nha

I think the dynamic arrays of BoE are superior to the fixed-length everything of BoA.

Nope. The original code in BoE, at least, is just as "fixed-length everything" as BoA. This is (very slowly) changing, but it's still largely the case if I recall correctly.

Originally Posted By: Ishad Nha

If you hit the 200 node you will need to trim dialog.

And issue an error message. Though, I think maybe it would be better to just generate the nodes anyway and issue a warning (perhaps generate the extra nodes commented-out).

Originally Posted By: Ishad Nha

Conveyor belts are a hassle, CM was trying to recreate them in BoA, I don't know if he succeeded or not. Mercifully they don't seem to be all that common from the scenarios I have looked at.

I have a terrain script which, as I recall, works reasonably well at implementing conveyor belts. It doesn't yet push items, though. It also won't push people with the "hovering feet" status, and may not push people who are hasted. From looking at the code, it seems that simply putting one instance of it somewhere in the town will automatically enable all belts, which may not be what is wanted.

I wouldn't say it's ready to be used.

Originally Posted By: Niemand

An idea occurs to me: The data structures for BoE have been significantly overhauled, and I wonder if we could make use of them as a drop-in replacement for the current reading code and storage data structures in the BoA editor. It might help us avoid some wheel-reinvention.

@Celtic Minstrel: Can you comment on the merits of that train of thought? Are the new BoE data structures mature enough that we would want to use them? Are things modular enough that we could import that code easily (like via an svn external definition)? While I know there's a new generation of data structures, is the reading code actually new and improved, and in particular would it be any better suited to the purpose of loading the whole scenario at once than what we have?

As I recall, the saving and loading code for the new data structures does not yet work. (In fact, I think the latest SVN version has saving disabled altogether.) Also, though the files are vaguely modular, they depend on a lot of other things (such as global variables defined in the core files). So, this is probably not a good idea at this time.

Speaking of which, did you ever find time to figure out why loading legacy save files did not work?

[Ishad Nha]

BoE Scenario Editor seems to be dynamic arrays everywhere, that is why you have the strlens everywhere. The game itself I don't know about.

[Niemand]

Quote:

. . . dynamic arrays everywhere, that is why you have the strlens everywhere

It's not a terribly important point, but no. C strings are arrays of characters whether allocated statically (on the stack) or dynamically (on the heap). They have no knowledge of their own length, so you have no choice but to use strlen() (which does nothing but march along from the pointer it's given, looking for a NUL, and counting how many steps it takes). This is the case even if the string is stored in a statically allocated array of fixed size, since the string can always be shorter than the array it lives in.

[Celtic Minstrel]

Yeah. The presence of strlen only means that C-strings are used, not whether anything is allocated dynamically. Dynamic allocation is the use of malloc.

[Ishad Nha]

Better description, BoA is fixed-length in everything while BoE has variable length strings, for things like outdoor zone names, town names and sign text.

[Niemand]

No, that's still not it. Both use the same system: A fixed length quantity of storage (an array) is allocated statically, in which a variable length (C style) string is placed. Obviously, then, the length of this string must not be allowed to exceed the size of the array, as that would be a buffer overrun, which C unfortunately makes it very easy to blunder into.

[Ishad Nha]

In the BoA Editor, I see "strlen" all over the place in the porting function. Maybe BoE allocates 256 bytes per string, but the strings contained in an exs file are rarely that long.

[Celtic Minstrel]

BoE stores the lengths of all the strings in the global scenario structure if I recall correctly, so it knows how long each one is when it loads them. It may also be using Pascal-style strings. However, whatever it is, it's probably only variable-length in the file; in memory, everything is statically allocated.

[Ishad Nha]

I was talking about the exs file, not what happens when it is loaded. Which is not a problem, given the amount of memory in computers these days.