Resource System

From Armagetron
This wiki page has an associated Blueprint on launchpad.

This is the design document for the resource system. It's currently a work in progress, but it should be followed for future work on the resource system.

Overview of the Resource System

The resource system is our own internal platform-independent, server-independent way to reference dynamic game data. This includes things like textures, sound effects, and game object models. While it would be straightforward to only focus on platform-independent path handling, we require a certain amount of information to be associated with resources, and some resources even contain game logic, such as maps. Resources can also be downloaded and/or installed automatically at the direction of a game server or the user. Therefore, we need a complete resource-handling system capable of much more advanced work while hiding underlying platform details from the rest of the game.

We have tResourceManager. It handles retrieving files from the cache/downloading, and will handle timing out cached resources and deleting them. It transparently handles bundled resources, installed resources, and cached resources, so none of the other components need care about it.

To help tResourceManager do it's work, we have tXmlResource. Currently there's nothing in tXmlResource, it's a placeholder. What it *should* contain, however, is the ability to read tags that are standard across all resources, such as the parent <Resource> tag. It should also understand tags that reference other resources and internal binary data. So the code path to loading files into memory passes through tXmlResource. For an initial implementation, we can just put the actual code into tXmlResource, later we can work out a fancy way to register mimetype handlers to allow components to hook in their own special formats.

tResourceManager, then, will scan all resource directories creating its own internal list of tXmlResources, and it will use that list to do its work. Same for installing, when tResourceManager is called upon to install a resource, it will be given the current location of the resource, load it into a tXmlResource, and then do what it needs to do to install it.

tResourceManager loads tXmlResources, and any other resources, on demand from its caches. When installing, tResourceManager initialized a tXmlResource for the relevant resources (without fully loading them) just enough to determine where they belong.

Components will create their own classes as needed that inherit tXmlResource. Then they'll register that class with tResourceManager. When it's time for a component to request a resource, it will ask tResourceManager for an instance of that resource and tResourceManager will provide it, either by creating a new one, by dynamic_casting the one from its list, or whatever else we dream up.

So the rest of the game, then, will only access resources through tResourceManager. No component will create new instances of a resource, all will always get points from tResourceManager. tResourceManager will always retain ownership of the resource instances, so components will not delete them, ever, though they will inform the tResourceManager when they no longer need it so it can perform cleanup.

Design Goals

The design goals of the system are as follows.

  • Low overhead for implementing new resource types
  • Low overhead for using resources from outside code
  • Scripts should be able to use the same interface to implement and use resources
  • Insure integrity of resources
  • Platform independence for all other code that uses resources (i.e. isolate platform-dependent code to the fewest possible places within the resource system)
  • Separation from the game itself, to be provided as a shared object for other programs to use in a support role

Organization in the Source Tree

Ok, resources need their own directory. Right now, each directory under src represents a layer and they all stack on top of each other. I'd like to move it to where each represents a component, and they may be stacked, maybe not. Lucifer thinks the resource system doesn't belong in tools, it needs access to stuff tools don't have access to. However, to be generic enough to be usable outside the game (as a shared object as in the goals), it needs to submit to the restrictions of the tools category.

Organization of Resources in the Source Tree

Todo: Write this

Organization of Resources in an installation

Todo: write this

Detailed Overview of Resource System

The resource system needs to be something where new resources can be added with a very minimum amount of work. It also needs to be flexible, since there's no point in adding resources if the resource system itself is going to restrict what types of resources can exist in the first place. We will hit a soft upper bound on useful resource types, hopefully before Bacchus gets out the door. But that won't mean we'll be done creating resource types.

Resources should be able to exist in script. So when we add scripting, we need it to have access to resources, and it needs to be able to create new resources.

A resource is a single entity, but may consist of extra components.

We have these xml files that provide at least the meta-data needed for the resource system to manage them. It is the only required element in the resource, and must reference all the data elements.

There may be binary data of arbitrary format, some of which we'll support on our own, some of which we'll expect people that really want them to write scripts to support them. Several examples of binary data are jpg, png, ogg, mpg, wav, flac, tiff. The list goes on.

In order to give the resource system the lowest overhead for people to use it, it needs to have a simple interface to the rest of the game. It should be as simple as: tResource* theResource = tResourceManager::getResource(myresource); and you get a resource initialized with the relevant xml file. Loading of further details, including binary data, must either be performed during this initialization or on demand such that after this line, you can send textures from the resource to openGL for rendering with no intermediate steps required. Streamed files must always be loaded on demand by their nature.

To support such a simple interface to the rest of the game, the resource system needs to be able to load any arbitrary binary file, and it needs to be able to do it in a generic manner. So you can't have, for example, LoadTiff, LoadJpg, LoadOgg, and so forth for methods. You need one method with one return type that is capable of loading any arbitrary binary file. Obviously that return type will have to be a class pointer, this is a problem solved in the cockpit already that can be copied into the resource system.

Normally, when you need one method with one return type in a bunch of classes you use inheritance to get that method there. So for gMap and gCockpitParser and gTheOthers they'll need to subclass tResource in some way or other, and tResource contains the LoadFile method (or whatever it winds up being called). So what does this LoadFile do?

tResource::LoadFile will call tResourceManager::GetFileLoader which will return a function pointer to the loader tResource has asked for, or it will return NULL for error, maybe it'll throw an exception. tResource's subclasses will use this file to load the actual binary data that goes with the resource. So for models this could be textures, could be another xml file that describes the model or part of the model, whatever. For sound effects this could be vorbis-encoded sound.

So tResourceManager keeps a list of loaders internally, indexed by type and mimetype, such as (texture, png) or (sound, mp3). Whatever. The other game modules will register their loaders with tResourceManager using a static nonsense-named object whose constructor just registers the loaders with all the other global variables. When tResourceManager scans its list of loaders, if none is found, it throws an exception. So if it sees, say, (texture, mp3), it can say "I don't have a loader that loads mp3s as textures". It might say "I have an mp3 loader, but it thinks mp3 is music and I can't use it". The loader itself needs to verify that the file is the right kind and the match has been made and so check the integrity of the file. If it fails, it also needs to throw an exception. Each loader returns a pointer to an instance of tResourceComponent.

Game components should already know what they're loading, why, and what they intend to do with it. So as long as they ask for the right stuff, they should be fine. tResource::LoadFile should take two arguments, then. It should take a string that contains a type, such as "texture", "sound", "song", "model". Then it should take an identifier for the file.

The files are identified in a special section at the top of the resource xml file. Under the <Resource> tag should be a <Files> section that contains all referenced files and what they are. tResource understands these tags and is able to load them as described.

So it should be possible to implement a new resource with 3 steps:

  1. Create the xml file and subclass tXmlParser
  2. Create special binary file loaders
  3. Register binary file loaders *and* the new subclass created in step 1

So after all this, we need tResourceManager to provide a single API that allows the rest of the game to retrieve, create, sort, delete, and do other things to resources. (Mind you, tResourceManager might ultimately find a vacation home as a shared object in a python library for other applications to use in resource creation) tResource needs to provide an interface that makes it possible for tResourceManager to do everything it needs to do, which is moving files around, expiring its cache, and so forth. The subclasses of tResource can extend the interface for their specific components, and they should do so. There's no reason gCockpit should be using a gMap to set itself up, after all. So gCockpitParser would have stuff in its interface that gMap doesn't have, and vice versa. But tResourceManager doesn't know about them, nor does it care for its work. (Except that it needs to be able to create instances of the subclass to give when asked, but it can do that and then dynamic_cast them for internal use. The cockpit/arena/whtever will have to dynamic_cast it back to the subclass for its own use)

Components of the Resource System

tResourceManager

This is the top-level interface. All outside code will retrieve resources from tResourceManager. tResourceManager will automatically provide these features:

  • Automatic download and installation of resources - this is an on-demand service provided. When a game server specifies a resource the client should use, tResourceManager will download and install that resource and make it available to the code that requires it on the client side. The game server itself should also automatically download and install resources in the same fashion from the repository specified in its configuration files.
  • Cache management - just like a web browser, tResourceManager should expire resources in the cache and manage the disk space used by the cache.
  • Generate lists of resources - this is an on-demand service as well. Its primary purpose is to provide information needed to build UI elements for the user. Use cases include moviepack selectors, resource browsers, and map rotators. It should be capable of listing not only installed and cached resources, but also resources available in the repository.
  • Loaders for files associated by file type and mimetype.

Features provided that generally require user action of some sort:

  • Resource creation - Used by resource authoring programs, like a map editor.
  • Resource packaging - Used by resource developers to create distributions of their resources
  • More (I seem to have forgotten some of this momentarily)

Loaders

tResourceManager stores a list of loaders. A file loader is something that loads a resource component that is not a resource itself. Actually, that's a little too vague. When a subclass of tResource is registered as a resource type, its file extension will be used and tResourceManager will know that it's also a loader for that type. More on this later.

Other than resources, resource components are generally assumed to be binary files that require special loaders that are fundamentally incompatible with each other. And it does so through a common interface. :) A loader returns a child of class tResourceComponent. The loader's owning module should declare and define that class. The loader itself creates the class instance with tNEW and then returns it. We should probably provide a convenience macro that makes it a dynamic_cast to tResourceComponent. In any case, the loader returns a pointer to tResourceComponent, one way or another. It should take as its only argument a complete path to the file. That means that tResourceManager will already have downloaded the file before it calls the loader, if downloading is necessary.

tResource

tResource is the base class for all xml resources. It is also used as a standalone class by tResourceManager to perform all of the functions needed during the day of managing resources. All resources should inherit this class.

tResource provides a generic way to handle all resources using standard tags that exist in all xml files. It also provides a base class for components to use to implement their own special resources.

An internal registry will exist eventually that components will use to tell the resource system about their own special resources, so that when they query tResourceManager for their own special resources, tResourceManager can provide.

These two aspects combine to make it easy to add new resources, both in new code and potentially in scripting. All that's needed is to make your subclass of tResource and register it with tResourceManager. We also get comprehensive resource management within tResourceManager via tResource, and this is true for even new resource types, and we get this without sacrificing extensibility.

Registering your subclass of tResource is a matter of registering a callback that returns an instance of your subclass. Ideally, this callback would be a static method in your subclass. We should consider making that a requirement. Is it possible to make a pure virtual static method? If so, then that's all we need to do, and in your derived class's constructor, call something like tResourceManager::RegisterResource with it, similar to the mechanism for loaders.

tResourceComponent

tResourceComponent is a base class that will never be used by the Resource System specifically. It is true that the subclasses of tResourceComponent will be instantiated by the loaders registered with tResourceManager, but the loaders themselves exist in the module that needs them.

An example of how this works is best:

rVisual is a class that is capable of rendering models. So it puts textures on them and performs transformations that allow animating the models. To load a texture, it provides a method that is capable of loading PNG files. During global variable initialization, it registers this loader with tResourceManager.

rVisual's PNG loader is marked (texture, png). tResourceManager now knows that any component that wants a texture that is a png can do so using rVisual's PNG loader. If you consider that there may be two implementations of rVisual, one that uses SDL's Load_Image function and one that uses libpng directly, then you might think that there may be a collision between the two. And you are correct. This dilemma is one that isn't solved by the resource system. One solution is to use #defines to make the use of SDL or libpng mutually exclusive. So we should consider adding a third marker for loaders, subsystem, where (texture, png, SDL) would tell it to use SDL's Load_Image.

However that dilemma is solved, the end result is that when a tResource subclass needs to load a texture, it will use the loader provided by rVisual, and it won't even know or care where the loader actually exists. In this example this is important because there's a wall between rVisual and the rest of the game, where rVisual is not allowed to give any data to the rest of the game because it's a rendering object.

So when an rVisual is instantiated, it is told by the rest of the renderer that it needs to load a model resource, and it's given a tResourceDescriptor. It uses its tResourceDescriptor to ask tResourceManager for an instance of the rVisualModel class. tResourceManager calls a callback that rVisual owns that returns a new class and parses the xml file for the resource. Since tResource does its own scan of the file for common tags before rVisualModel does its scan, tResource will be loading the components of the model. tResource will look in the <Files> section at the files it needs to load and load them as directed, using the callbacks for the loaders given to it by tResourceManager when it queries for them. The callbacks themselves are also owned by rVisual. So rVisual knows what's going into the tResourceComponent because it owns the function that creates and populates it with data. It can put anything it wants in there. It may want to just put a tString that contains a path to the file and then later load that file every frame. There may be no good reason for it, the point is that it can.

The fundamental point that's important here is that while it is true that the Resource System initiates loading files for resources, it never knows anything about the files themselves beyond what is required to match it up with a loader, and it never needs to know more. The user of the resource provides all the code needed to load the file and then use it, the Resource System just sits in the middle and manages the file itself.

In that admittedly convoluted example, everything that deals with rVisual doesn't actually exist anywhere in current code, it's just invented for the purpose of giving this example.

Virtual Resources

References From Resources

Virtual Resources allows the player to specify a preferred resource to use in references to a common resource. For example, to specify their favorite wall texture for cycle walls. While at some point, the game data (such as map visual CSS) can specify a specific texture resource to use for cycle walls, it could also specify that rim walls should be the same as the cycle wall... thus, it would set the rim wall texture to the 'virtual' cycle wall texture. The player's preferred cycle wall texture would then be applied to the rim, regardless of what that preference is.

A filepath syntax could be virtual/common-name or local/virtual/common-name

In CSS syntax, this would look like:

Wall {
    texture: resource(virtual/cycle-wall.texture);
}

Virtual resources can be applied to any type of resource, including sounds, models, etc...

For future backward compatibility, references to newly introduced virtual resources should specify a default resource to use in case the user has not selected a preference. With a filepath syntax, this would become virtual/common-name(default-resource-filepath), for example virtual/cycle-wall.texture(AATeam/CycleWall-0.3.texture.xml)

Choosing Resources For Virtuals

The player can choose their default virtual resources either individually or by selecting what is known as a 'moviepack'. A moviepack is a resource which simply associates virtual resources with actual resources. For example,

<Resource author="Anonymous" category="test" name="foo" version="0.1">
  <Moviepack>
    <Prefer virtual="cycle-wall" type="texture">
      <ResourceRef
        author="Anonymous"
        category="test"
        name="foo-cycle-wall"
        type="texture"
        version="0.1"
      />
    </Prefer>
    <Prefer virtual="rim-wall" type="texture">
      <ResourceRef
        author="Anonymous"
        category="test"
        name="foo-rim-wall"
        type="texture"
        version="0.1"
      />
    </Prefer>
  </Moviepack>
</Resource>