Class MissingnessThreshold

java.lang.Object

edu.cmu.tetrad.data.missing.MissingnessThreshold

public final class MissingnessThreshold
extends Object

Works out what dropping every variable whose missingness rate exceeds a threshold would do to a dataset, without dropping anything. Supports a preview-then-act control: the useful threshold is the one where the complete-case count turns, and that is data-specific, so it has to be looked at rather than assumed.

What this remedy is and is not

Dropping a high-missingness column is decided on the missingness indicator alone, never on the values, so under MCAR or MAR it does not bias what remains -- unlike dropping rows, which under MAR does. When a column is MNAR, dropping it also removes the bias it would have contributed to its own family.

The cost is specific to structure learning and is not small. Removing a variable that is a common cause of two retained variables turns a causally sufficient system into an insufficient one: the confounding is manufactured by the remedy, and no search run afterwards can tell. The variables most likely to exceed a missingness threshold -- hardest to measure, most often skipped -- are not a random sample with respect to being upstream of others. MissingnessThreshold.Candidate.maxAbsCorrelation() is reported for exactly this reason: a high-missingness variable strongly associated with several retained ones is the dangerous case.

Thresholds have no theoretical basis. Twenty percent and fifty percent are conventions, not results.

Author:

josephramsey

See Also:

• MissingDataAudit

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	MissingnessThreshold.Candidate	A variable that a threshold would drop, with what is needed to judge whether dropping it is safe.
static final record	MissingnessThreshold.Effect	The consequence of applying one threshold.

Method Summary

Modifier and Type	Method	Description
static double[]	candidateThresholds(MissingDataAudit audit, int p)	The distinct missingness rates present in the data, ascending, as the thresholds worth sweeping: the dropped set only changes as the threshold crosses one of these, so nothing between them is informative.
static List<String>	droppedNames(MissingnessThreshold.Effect effect)	The names a threshold would drop, in the order effectOf(DataSet, MissingDataAudit, double) reports them.
static MissingnessThreshold.Effect	effectOf(DataSet dataSet, MissingDataAudit audit, double threshold)	The effect of a single threshold.
static boolean	isConfoundingRisk(MissingnessThreshold.Candidate candidate)	Whether a dropped variable looks risky to drop: strongly associated with something retained, on enough pairwise cases to believe it.
static List<Node>	resolve(DataSet dataSet, List<String> names)	Resolves the dropped names to the dataset's variables, skipping any that are no longer present.
static List<MissingnessThreshold.Effect>	sweep(DataSet dataSet, MissingDataAudit audit, double... thresholds)	The effect of each of a series of thresholds, for reading off where the complete-case count turns.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

effectOf

public static MissingnessThreshold.Effect effectOf(DataSet dataSet,
 MissingDataAudit audit,
 double threshold)

The effect of a single threshold.

Parameters:

dataSet - The data. Not modified.

audit - An audit of that data, for the per-variable rates.

threshold - Variables with a missingness rate strictly above this are dropped. Must be in [0, 1].

Returns:

The effect.

Throws:

IllegalArgumentException - if the threshold is out of range.

sweep

public static List<MissingnessThreshold.Effect> sweep(DataSet dataSet,
 MissingDataAudit audit,
 double... thresholds)

The effect of each of a series of thresholds, for reading off where the complete-case count turns.

Parameters:

dataSet - The data. Not modified.

audit - An audit of that data.

thresholds - The thresholds to evaluate.

Returns:

One effect per threshold, in the order given.

candidateThresholds

public static double[] candidateThresholds(MissingDataAudit audit,
 int p)

The distinct missingness rates present in the data, ascending, as the thresholds worth sweeping: the dropped set only changes as the threshold crosses one of these, so nothing between them is informative.

Parameters:

audit - An audit of the data.

p - The number of columns.

Returns:

The candidate thresholds.

droppedNames

public static List<String> droppedNames(MissingnessThreshold.Effect effect)

The names a threshold would drop, in the order effectOf(DataSet, MissingDataAudit, double) reports them.

Parameters:

effect - An effect.

Returns:

The names.

isConfoundingRisk

public static boolean isConfoundingRisk(MissingnessThreshold.Candidate candidate)

Whether a dropped variable looks risky to drop: strongly associated with something retained, on enough pairwise cases to believe it.

Parameters:

candidate - A candidate.

Returns:

True when the association is at or above 0.5.

resolve

public static List<Node> resolve(DataSet dataSet,
 List<String> names)

Resolves the dropped names to the dataset's variables, skipping any that are no longer present.

Parameters:

dataSet - The data.

names - The names.

Returns:

The variables.