Package de.uni_mannheim.informatik.dws.melt.matching_eval.evaluator.significance

Class EvaluatorMcNemarSignificance

java.lang.Object

de.uni_mannheim.informatik.dws.melt.matching_eval.evaluator.Evaluator

de.uni_mannheim.informatik.dws.melt.matching_eval.evaluator.significance.EvaluatorMcNemarSignificance

public class EvaluatorMcNemarSignificance
extends Evaluator

Implementation of a significance test according to information specified in: Mohammadi, Majid; Atashin, Amir Ahooye; Hofman, Wout; Tan, Yaohua. Comparison of Ontology Alignment Systems Across Single Matching Task Via the McNemar's Test. 2018..

Field Summary

Fields

Modifier and Type	Field	Description
protected double	alpha	Alpha for significance tests.
static final double	DEFAULT_TRACK_SIGNIFICANCE_SHARE
static final String	FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC
static final String	FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION
static final String	FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK
static final String	FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK
static final String	FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC
static final String	FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION
static final String	FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK
static final String	FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK
static final String	FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC
static final String	FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION
static final String	FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK
static final String	FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK
static final String	FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC
static final String	FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION
static final String	FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK
static final String	FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK
private static final org.slf4j.Logger	LOGGER	Default Logger
private double	trackSignificanceShare	If we calculate the significance for a complete track, the trackSignificanceShare defines the share of test case alignments that need to be significantly different so that the track counts as significantly different.

Fields inherited from class de.uni_mannheim.informatik.dws.melt.matching_eval.evaluator.Evaluator

results

Constructor Summary

Constructors

Constructor	Description
EvaluatorMcNemarSignificance(ExecutionResultSet results)	Constructor.
EvaluatorMcNemarSignificance(ExecutionResultSet results, double alpha)	Constructor.

Method Summary

Modifier and Type	Method	Description
private Map<McNemarCrossTrackResult,SignificanceCount>	calculateAggregatedTestCaseMap(Map<McNemarIndividualResult,Double> pValues)	Given a Map with p values, aggregate the number of significant/not significant alignments.
Map<McNemarIndividualResult,Double>	calculatePvalues(double alpha, TestType testType)
(package private) static long	fact(int n)	Returns the factorial of n.
double	getTrackSignificanceShare()
(package private) static long	nCr(int n, int r)	From n choose r.
(package private) static BigInteger	nCrBigInt(int N, int R)	From n choose r with large numbers.
private double	pValueConsideringFalsePositives(ExecutionResult executionResult1, ExecutionResult executionResult2)	Given two execution results, it is determined whether the two results are significantly different (p < alpha).
private double	pValueConsideringFalsePositives(ExecutionResult executionResult1, ExecutionResult executionResult2, TestType testType)	GGiven two execution results, it is determined whether the two results are significantly different (p < alpha).
void	setTrackSignificanceShare(double trackSignificanceShare)
private void	writeAggregatedTestcasesResultFile(Map<McNemarIndividualResult,Double> pValues, File fileToWrite)	Write the results file on the granularity of matchers.
private void	writeAggregatedTracksResultFile(Map<McNemarIndividualResult,Double> pValues, File fileToWrite)	Write the results file on the granularity of matchers.
void	writeResultsToDirectory(File baseDirectory)	Two files will be written.
void	writeResultsToDirectory(String baseDirectoryPath)	Two files will be written.
private void	writeTestCaseResultFile(Map<McNemarIndividualResult,Double> pValues, File fileToWrite)	Write the results file on the granularity of test cases.
private void	writeTrackResultFile(Map<McNemarIndividualResult,Double> pValues, File fileToWrite)	Write the results file on the granularity of tracks.

Methods inherited from class de.uni_mannheim.informatik.dws.melt.matching_eval.evaluator.Evaluator

checkAndCreateDirectory, getDefaultResultsDirectory, getDirectoryWithCurrentTime, getResults, getResultsDirectoryTrack, getResultsDirectoryTrackMatcher, getResultsDirectoryTrackTestcaseMatcher, setDefaultResultsDirectory, writeToDirectory, writeToDirectory, writeToDirectory

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

Default Logger

alpha

protected double alpha

Alpha for significance tests.

trackSignificanceShare

private double trackSignificanceShare

If we calculate the significance for a complete track, the trackSignificanceShare defines the share of test case alignments that need to be significantly different so that the track counts as significantly different.

For example: Given a track has 10 test cases and a trackSignificanceShare of 0.5. If 4 alignments are significantly different, the track counts as not significant for the matcher combination. If 5 or more alignments are significantly different, the track counts as significant for the matcher combination.

DEFAULT_TRACK_SIGNIFICANCE_SHARE

public static final double DEFAULT_TRACK_SIGNIFICANCE_SHARE

See Also:

• Constant Field Values

FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC

public static final String FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC

See Also:

• Constant Field Values

FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC

public static final String FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC

public static final String FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC

public static final String FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC

See Also:

• Constant Field Values

FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

public static final String FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

public static final String FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

public static final String FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

public static final String FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION

public static final String FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION

See Also:

• Constant Field Values

FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION

public static final String FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION

public static final String FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION

public static final String FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION

See Also:

• Constant Field Values

FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

public static final String FILE_NAME_TEST_CASE_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

public static final String FILE_NAME_TRACK_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

public static final String FILE_NAME_AGGREGATED_TESTCASES_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

See Also:

• Constant Field Values

FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

public static final String FILE_NAME_AGGREGATED_TRACKS_MC_NEMAR_ASYMPTOTIC_CCORRECTION_EXACT_FALLBACK

See Also:

• Constant Field Values

Constructor Details

EvaluatorMcNemarSignificance

public EvaluatorMcNemarSignificance(ExecutionResultSet results)

Constructor. It runs the test with alpha=0.05

Parameters:

results - The results of the matching process.

EvaluatorMcNemarSignificance

public EvaluatorMcNemarSignificance(ExecutionResultSet results,
 double alpha)

Constructor.

Parameters:

results - The results of the matching process.

alpha - The desired alpha (probability of making a type 1 error.

Method Details

writeResultsToDirectory

public void writeResultsToDirectory(String baseDirectoryPath)

Two files will be written.

Parameters:

baseDirectoryPath - The directory path to which the result shall be written.

writeResultsToDirectory

public void writeResultsToDirectory(File baseDirectory)

Two files will be written.

Specified by:

writeResultsToDirectory in class Evaluator

Parameters:

baseDirectory - The directory to which the result shall be written.

writeAggregatedTestcasesResultFile

private void writeAggregatedTestcasesResultFile(Map<McNemarIndividualResult,Double> pValues,
 File fileToWrite)

Write the results file on the granularity of matchers.

Parameters:

pValues - The p values.

fileToWrite - The file that shall be written.

calculateAggregatedTestCaseMap

private Map<McNemarCrossTrackResult,SignificanceCount> calculateAggregatedTestCaseMap(Map<McNemarIndividualResult,Double> pValues)

Given a Map with p values, aggregate the number of significant/not significant alignments.

Parameters:

pValues - P values.

Returns:

Map with aggregated significance counts.

writeAggregatedTracksResultFile

private void writeAggregatedTracksResultFile(Map<McNemarIndividualResult,Double> pValues,
 File fileToWrite)

Write the results file on the granularity of matchers.

Parameters:

pValues - The p values.

fileToWrite - The file that shall be written.

writeTrackResultFile

private void writeTrackResultFile(Map<McNemarIndividualResult,Double> pValues,
 File fileToWrite)

Write the results file on the granularity of tracks.

Parameters:

pValues - The p values.

fileToWrite - The file that shall be written.

writeTestCaseResultFile

private void writeTestCaseResultFile(Map<McNemarIndividualResult,Double> pValues,
 File fileToWrite)

Write the results file on the granularity of test cases.

Parameters:

pValues - The p values.

fileToWrite - The file that shall be written.

calculatePvalues

public Map<McNemarIndividualResult,Double> calculatePvalues(double alpha,
 TestType testType)

pValueConsideringFalsePositives

private double pValueConsideringFalsePositives(ExecutionResult executionResult1,
 ExecutionResult executionResult2)

Given two execution results, it is determined whether the two results are significantly different (p < alpha). The execution results must be from the same test case.

Parameters:

executionResult1 - Result 1.

executionResult2 - Result 2.

Returns:

p value

pValueConsideringFalsePositives

private double pValueConsideringFalsePositives(ExecutionResult executionResult1,
 ExecutionResult executionResult2,
 TestType testType)

GGiven two execution results, it is determined whether the two results are significantly different (p < alpha). The execution results must be from the same test case.

Parameters:

executionResult1 - Result 1.

executionResult2 - Result 2.

testType - The type of test to be used.

Returns:

p value. NaN if p cannot be calculated.

nCrBigInt

static BigInteger nCrBigInt(int N,
 int R)

From n choose r with large numbers.

Parameters:

N - N of nCr(N,r)

R - R of nCr(n,R)

Returns:

nCr(n, r)

nCr

static long nCr(int n,
 int r)

From n choose r.

Parameters:

n - N of nCr(N,r)

r - R of nCr(n,R)

Returns:

nCr(n, r)

fact

static long fact(int n)

Returns the factorial of n.

Parameters:

n - Value for which the factorial shall be calculated.

Returns:

Factorial of n.

getTrackSignificanceShare

public double getTrackSignificanceShare()

setTrackSignificanceShare

public void setTrackSignificanceShare(double trackSignificanceShare)