MELT Evaluation Client

If you have one or more packaged matching systems (SEALS, Docker Web) or simply a URL pointing to a matching service and want to evaluate those, you can write a Java program to evaluate those.

Alternatively, you can use the MELT Evaluation Client which is discussed in this section. The client is a simple java command line tool that is executed in a terminal and does not require any coding skills or deep understanding of MELT, SEALS, or the Docker Web format. The client requires as input IDs of the available built-in tracks. Alternatively, local tracks/ontologies can alo be given as input.

You can download the latest version of the command line tool on GitHub.

Using the Command Line Tool

The most straight forward way to use the tool is as follows:

java -jar matching-eval-client-latest.jar --systems <system-1-path> <system-2-path> <...> --track <location-URI> <collection-name> <version>

The results will be printed on the command line.

Help/Documentation

java -jar matching-eval-client-latest.jar --help

Types of Systems

The evaluation client will automatically detect the following types of matchers:

  1. Web package (docker file) - automatically detected by path ending with .tar.gz
  2. SEALS package - automatically detected by path ending with .zip
  3. Web package (URL) - in all other cases an URL is assumed

Advanced Features

Java 8

Evaluating SEALS packages requires Java 8. If you do not use Java 8 as your default java runtime, you can specify the Java 8 location using parameter --java8 <path to Java 8>.

Serializing Alignments and Details

If you want to access the system alignments and/or have a more detailed evaluation report, you can specify an output directory via --results. The reports are generated by the EvaluatorCSV and you can find an explanation of the generated file structure in the corresponding section of the user guide.

Evaluating a Local Test Case

If you have a local test case (i.e. two ontologies and a reference alignment) on disk that you want to evaluate, you can use option --local-testcase <onto1-path> <onto2-path> <reference-path> instead of the --track option.

Evaluating a Local Track

If you have multiple local test cases that you want to evaluate simultaneously (micro/macro precision etc.), you can create a local track. Create a folder structure as described in the local track section of the user guide and use the --local-track <folder> <name> <version> option where name and version can be freely set.