Class MissingDataAudit

java.lang.Object

edu.cmu.tetrad.data.missing.MissingDataAudit

public final class MissingDataAudit
extends Object

A descriptive audit of the missingness in a dataset, intended to be computed (and logged) before a search is run on real data, and to give the user the evidence needed to choose a MissingDataPolicy. The audit reports per-variable missing counts and rates, the number of complete rows, the distinct missingness patterns and their frequencies, and the matrix of pairwise complete counts (the number of rows on which each pair of variables is jointly observed), whose minimum is a useful proxy for how much information test-wise deletion actually has available for the worst pair.

For continuous datasets, littlesMcarTest() computes Little's (1988) chi-square test of the MCAR hypothesis, using EM estimates of the mean and covariance under the saturated Gaussian model. A small p-value is evidence against MCAR, in which case deletion-based policies (MissingDataPolicy.LISTWISE, MissingDataPolicy.TESTWISE) are suspect and a MAR-based policy (MissingDataPolicy.EM_COVARIANCE, MissingDataPolicy.MULTIPLE_IMPUTATION) should be preferred. A large p-value is consistent with MCAR but does not prove it.

Missing values are Double.NaN for continuous variables and DiscreteVariable.MISSING_VALUE for discrete variables, per Tetrad convention.

Version:

$Id: $Id

Author:

josephramsey

See Also:

• MissingDataPolicy
• EmCovarianceEstimator

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	MissingDataAudit.LittleResult	The result of Little's MCAR test: the chi-square statistic, its degrees of freedom, the p-value, and pattern accounting.

Field Summary

Fields

Modifier and Type	Field	Description
static final double	HIGH_MISSING_RATE	The per-variable missing rate at or above which advice() flags a variable as having high missingness.
static final double	MCAR_TEST_ALPHA	The alpha level advice() uses to interpret Little's MCAR test.
static final int	SMALL_PAIRWISE_COUNT	The pairwise complete count below which advice() warns that test-wise statistics may be unstable.

Constructor Summary

Constructors

Constructor	Description
MissingDataAudit(DataSet dataSet)	Constructs the audit for the given dataset, computing all descriptive statistics.

Method Summary

Modifier and Type	Method	Description
List<String>	advice()	Rule-based advice for handling the missingness in this dataset, as a list of short paragraphs, in decreasing order of importance.
boolean	anyMissing()	True just in case the dataset contains at least one missing entry.
double	getMeanPairwiseCount()	The mean, over pairs of distinct columns, of the pairwise complete count (see MissingDataSpec.EffectiveSampleSizeMode.MEAN_PAIRWISE).
int	getMinPairwiseCount()	The minimum, over pairs of distinct columns, of the pairwise complete count.
int	getMissingCount(int column)	The number of missing entries for the given column.
double	getMissingRate(int column)	The fraction of entries missing for the given column.
int	getNumCompleteRows()	The number of rows with no missing entries--i.e., the sample size that listwise deletion would retain.
int	getNumPatterns()	The number of distinct missingness patterns among the rows.
double	getOverallMissingRate()	The fraction of all entries that are missing.
int[][]	getPairwiseCompleteCounts()	The matrix of pairwise complete counts: entry (i, j) is the number of rows on which columns i and j are both observed.
Map<String,Integer>	getPatternCounts()	The distinct missingness patterns and their row counts.
static boolean	isMissing(DataSet dataSet, int row, int column)	Returns true just in case the given cell of the dataset is missing, using Tetrad's conventions: DiscreteVariable.MISSING_VALUE for discrete variables and Double.NaN otherwise.
MissingDataAudit.LittleResult	littlesMcarTest()	Computes Little's (1988) chi-square test of the MCAR hypothesis for a continuous dataset, using EM estimates of the mean and covariance under the saturated Gaussian model.
String	report()	A human-readable, multi-line summary of the audit, suitable for logging via TetradLogger before a search is run.
String	toString()	A string representation of the audit.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

HIGH_MISSING_RATE

public static final double HIGH_MISSING_RATE

The per-variable missing rate at or above which advice() flags a variable as having high missingness.

See Also:

• Constant Field Values

MCAR_TEST_ALPHA

public static final double MCAR_TEST_ALPHA

The alpha level advice() uses to interpret Little's MCAR test.

See Also:

• Constant Field Values

SMALL_PAIRWISE_COUNT

public static final int SMALL_PAIRWISE_COUNT

The pairwise complete count below which advice() warns that test-wise statistics may be unstable.

See Also:

• Constant Field Values

Constructor Details

MissingDataAudit

public MissingDataAudit(DataSet dataSet)

Constructs the audit for the given dataset, computing all descriptive statistics. This is a single O(n * p^2) pass over the data (for the pairwise counts).

Parameters:

dataSet - The dataset. May be continuous, discrete, or mixed.

Method Details

isMissing

public static boolean isMissing(DataSet dataSet,
 int row,
 int column)

Returns true just in case the given cell of the dataset is missing, using Tetrad's conventions: DiscreteVariable.MISSING_VALUE for discrete variables and Double.NaN otherwise.

Parameters:

dataSet - The dataset.

row - The row index.

column - The column index.

Returns:

True if the cell is missing.

anyMissing

public boolean anyMissing()

True just in case the dataset contains at least one missing entry.

Returns:

True if any entry is missing.

getMissingCount

public int getMissingCount(int column)

The number of missing entries for the given column.

Parameters:

column - The column index.

Returns:

This count.

getMissingRate

public double getMissingRate(int column)

The fraction of entries missing for the given column.

Parameters:

column - The column index.

Returns:

This rate, in [0, 1].

getOverallMissingRate

public double getOverallMissingRate()

The fraction of all entries that are missing.

Returns:

This rate, in [0, 1].

getNumCompleteRows

public int getNumCompleteRows()

The number of rows with no missing entries--i.e., the sample size that listwise deletion would retain.

Returns:

This count.

getNumPatterns

public int getNumPatterns()

The number of distinct missingness patterns among the rows.

Returns:

This count.

getPatternCounts

public Map<String,Integer> getPatternCounts()

The distinct missingness patterns and their row counts. Each key is a string of '1' (observed) and '0' (missing) characters, one per column, in dataset variable order.

Returns:

An unmodifiable view of this map, in order of first appearance.

getPairwiseCompleteCounts

public int[][] getPairwiseCompleteCounts()

The matrix of pairwise complete counts: entry (i, j) is the number of rows on which columns i and j are both observed. The diagonal gives per-variable observed counts.

Returns:

A defensive copy of this matrix.

getMinPairwiseCount

public int getMinPairwiseCount()

The minimum, over pairs of distinct columns, of the pairwise complete count. This is the sample size available to test-wise deletion for the worst pair, and a candidate conservative effective sample size (see MissingDataSpec.EffectiveSampleSizeMode.MIN_PAIRWISE).

Returns:

This minimum, or the number of rows if there are fewer than two columns.

getMeanPairwiseCount

public double getMeanPairwiseCount()

The mean, over pairs of distinct columns, of the pairwise complete count (see MissingDataSpec.EffectiveSampleSizeMode.MEAN_PAIRWISE).

Returns:

This mean, or the number of rows if there are fewer than two columns.

littlesMcarTest

public MissingDataAudit.LittleResult littlesMcarTest()

Computes Little's (1988) chi-square test of the MCAR hypothesis for a continuous dataset, using EM estimates of the mean and covariance under the saturated Gaussian model. Rows with no observed values are excluded. Patterns whose observed-variable covariance submatrix is singular are skipped (their degrees of freedom are not counted). The result is cached.

Reference: Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198-1202.

Returns:

The test result.

Throws:

IllegalArgumentException - If the dataset is not continuous.

IllegalStateException - If the dataset has no missing values (the test is vacuous).

advice

public List<String> advice()

Rule-based advice for handling the missingness in this dataset, as a list of short paragraphs, in decreasing order of importance. This is shared by the GUI's Data Audit dialog, causal-cmd, and py-tetrad so that all three interfaces give the same recommendations. The rules are deliberately conservative: no test can confirm MCAR or rule out MNAR, so the advice recommends policies rather than asserting assumptions.

Returns:

The advice, one recommendation per entry; never empty.

report

public String report()

A human-readable, multi-line summary of the audit, suitable for logging via TetradLogger before a search is run. Includes Little's MCAR test for continuous datasets with missing values.

Returns:

This summary.

toString

public String toString()

A string representation of the audit.

Overrides:

toString in class Object

Returns:

This string.