Resources and Resource Loaders

Resources are how data from external files enter the program and are therefore an important concept in CaRMeN. In order to configure how such a resource imports data resource definitions in the carmen configuration under the resources: field can be specified. This is also how data can be transformed without requiring the user to modify the original data (i.e. preprocessing). It is therefore one of the key features that allow the automation of the workflow.

Resource Loaders are the hidden engines behind resources. Loaders in resources are thin interface layers that serve to translate input data from one format into another, generally into a format that CaRMeN can interpret. There are several kinds of loaders, and therefore also several kinds of resources, available in CaRMeN. Resources are grouped into two categories. The first is that of the so called single-input resource loaders, which take a single input (often the name of a file, for example a .csv file) and produce an output. The second category is that of the multiple-input resource loaders, which take multiple inputs. These are usually used for more complex data importing or transformation tasks, for example to import a chemical mechanism (in order to validate whether a reaction is balanced, information about the species' atomic composition must be known, and this information is typically stored in a different format in a separate file). However, both resource types always produce a single output. For more information on the resource loaders available in CaRMeN see the chapter Resource Loaders under the reference section of this documentation.

Resources are introduced in CaRMeN's configuration as values of a field, starting with the name of the field followed by a colon :, a blank space, and the resource. Depending on the resource loader and the configuration at hand, the resource can be given either as string or as an object. If given as a string, only the path to the file to be loaded needs to be provided, starting with a point . followed by a slash / and the path to the file. If given as an object, a $resource macro is provided as a key. The $resource key is written on the line below the field name with one additional level of indentation (i.e. two additional blank spaces) with respect to the field name followed by a colon :, a blank space and the name of the resource as a string. A resource always has a name by which it can be referred to.

The following is an example of resources introduced in CaRMeN's configuration in YAML-format:

#...
field_name: ./path/to/file
#...
field_name:
  $resource: resource_name
#...

Resource definitions are referenced to by either the file path or by the resource name. Resources are defined in YAML by writing resources: and hitting enter. Resources are then given as an array, each one starting with one level of indentation (i.e. two blank spaces) followed by a hyphen - and enter. In the current version of CaRMeN one-to-one and transforming resources defer in their definition.

Single-Input Resources

Single-input resources allow four keys in their definition: name, input, loader, and options. The first key is written by convention on the line below the hyphen with one additional level of indentation (i.e. two additional blank spaces) with respect to the hyphen. The name of each key is followed by a colon :, a blank space, and the value of the key. Examples of single-input resources are json, yaml or yml, csv, fwc, ckt, moldata, cki, and ckr.

The following shows a single-input resource definition in YAML-format:

resources:
  -
    name: optional_resource_name
    input: ./path/to/file
    loader: csv
    options:
      delimiter: ;

The name: field is optional and it defines the name of the resource by which it can be referred to. If no name is defined, the input is used as the name.

The input: field is obligatory. For resource loaders that import files, this is the relative path (starting with a ./) to the file. However, loaders exist that take other resources as inputs. In this case case, the input is the name of that resource.

The loader: field is optional. It defines the loader to be used and it is necessary if the format of the data file cannot be inferred from its extension (e.g. csv data files with the extension .data or no extension at all).

The options: field is optional. Some loaders take additional configuration options. The available options depend on the loader at hand. For more information on the options available for resource loaders see the chapter Resource Loaders under the reference section of this documentation.

The resource definition in the above example tells CaRMeN to invoke the loader called csv, where the loader is passed the value of the input and options fields. This particular loader attempts to open the file /path/to/file and interprets its contents as a csv (comma-separated values) file, where a delimiter of ; is used instead of the default ,. This loader then produces a tabular data structure, which can be accessed later by referring to the resource's name optional_resource_name.

Definitions for single-input resources can be omitted entirely under certain circumstances. This is the case when the file extension matches the name of the loader, and when the particular resource can be imported using the loader's default options. The resource can be accessed simply by referring to its path. For example, the following resource definition is unnecessary:

resources:
  -
    name: ./data.csv
    input: ./data.csv
    loader: csv

Here, loader can be inferred from the input field, because the file extension of the ./data.csv file matches the name of the loader (csv). Furthermore, the name field is the same as the input field and hence corresponds to the default value.

Only if CaRMeN cannot infer the correct loader from a file's extension and/or the file does not follow the default format for the file kind at hand, must a resource definition be specified. However, definitions for multiple-input resources must always be specified.

Note: You may occasionally come across resources that are defined using the four keys name, path, format, and/or options. This was how resources were defined in older versions of CaRMeN and was later changed to the format described here for consistency.

Multiple-Input Resources

In contrast to single-input resources, multiple-input resources do not have an input field, because the inputs are embedded within options. The remaining three keys name, loader, and options must all be specified. The following example shows a definition of a multiple-input resource using the chemical-model loader.

resources:
  -
    name: resource_name
    loader: chemical-model
    options:
      species_database:
        - ./thermdata.ckt
      reactions:
        surface: ./surface_reactions.ckr

The name: field defines the name of the resource by which it can be referred to.

The loader: field defines the resource loader to be used.

The options: field is an object with a structure dependent on the loader used. Within the options object, references to the inputs of the loader are made. For more information on the options available for resource loaders see the chapter Resource Loaders under the reference section of this documentation.

In the example, the resource definition tells CaRMeN to use the loader chemical-model. This particular loader produces a complex data structure that can be referred to using the name resource_name. Within the options object, references to other resources are made (./thermdata.ckt and ./surface_reactions.ckr).