Class MissingnessInjector

java.lang.Object

edu.cmu.tetrad.data.MissingnessInjector

public final class MissingnessInjector
extends Object

Injects missing values into a dataset under a per-cell logistic propensity, with optional terms that make the mechanism MAR or MNAR as well as MCAR.

For variable j and case i,

logit P(missing_ij) = α_j + λ · u_i + Σ_k β_jk z_ik + Σ_k γ_jk M_ik, u_i ~ N(0, 1)

where u is a latent per-case propensity shared across variables, z_ik is the true (pre-deletion) value of variable k in case i, standardized over the input column, and M_ik is 1 if variable k was deleted in case i. The β terms are value effects and the γ terms are indicator effects; both are supplied through a MissingnessInjector.Mechanism, and both are absent by default. With no mechanism this is exactly the historical injector, and with the same seed it deletes exactly the same cells.

Separately, a MissingnessInjector.Mechanism can name an attrition order. Variables in that order lose values monotonically: each case drops out at some point along the order and is missing on every later variable in it. The dropout hazard at each step has the same logistic form, so it too can carry a row effect and value effects.

Calibration

Each α_j is solved by bisection so that the marginal rate for variable j matches its target rate whatever the other terms are. Without this, adding a row effect or a value effect would also move the realized rates -- Jensen's inequality applied to the logistic link -- and level could not be varied independently of mechanism, which would make any study using this tool hard to interpret. The marginal is evaluated against the realized u values, values and indicators, so the calibration is exact for the sample actually generated rather than in expectation. Under attrition the hazard at each step is calibrated among the cases still present, so that the cumulative rate at that step matches its target; the targets must therefore be nondecreasing along the order.

Mechanism classes

• MCAR: no value effects, directly or through indicator effects. The row effect alone stays MCAR: u is independent of the data, so it clusters the pattern without changing the mechanism class.
• MAR: every value the deletion depends on is certain to be observed. That holds for a driver that is never deleted, and for an attrition hazard driven by variables earlier in the attrition order, which are observed in every case still present when the hazard applies (sequential, or monotone, MAR).
• MNAR: anything else, including self-masking (a variable's deletion depends on its own value). The classification is conservative: a configuration reported as MNAR has not been certified MAR, which is not a proof that it is not MAR in Rubin's pattern-specific sense.

A colluder, in the sense of Nabi, Bhattacharya and Shpitser (2020), is flagged when one variable's deletion depends on both the value and the indicator of the same other variable. Attrition is deliberately not flagged, although its graph has the colluder shape: the dependence of a later indicator on an earlier one is context-specific there, which a missing-data DAG cannot express, and sequential MAR is identified.

The mechanism report also measures how monotone the realized pattern is along an order -- the attrition order if one is given, otherwise column order -- since that decides whether test-wise local scores for one child share their rows.

Author:

josephramsey

See Also:

• DataTransforms.addMissingData(DataSet, double[])

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	MissingnessInjector.Effect	One term in the deletion propensity of a target variable: the effect of a driver variable's standardized value, or of its missingness indicator, on the log odds that the target is deleted.
static final record	MissingnessInjector.Mechanism	The terms of the deletion mechanism beyond the per-variable level and the row effect.
static enum	MissingnessInjector.MechanismClass	The mechanism class of an injection.
static final record	MissingnessInjector.MechanismReport	What the mechanism was, and how monotone the realized pattern is.
static final record	MissingnessInjector.Report	A report of what was actually injected, for checking that the realized pattern is the intended one.
static final record	MissingnessInjector.Result	The injected data together with its reports.
static final record	MissingnessInjector.Spec	The injection specification.

Method Summary

Modifier and Type	Method	Description
static MissingnessInjector.Result	inject(DataSet data, MissingnessInjector.Spec spec)	Injects missing values into a copy of the data, MCAR.
static MissingnessInjector.Result	inject(DataSet data, MissingnessInjector.Spec spec, MissingnessInjector.Mechanism mechanism)	Injects missing values into a copy of the data under the given mechanism.
static double[]	parseRateProfile(String text, List<Node> variables, double defaultRate)	Parses a hand-specified rate profile.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

inject

public static MissingnessInjector.Result inject(DataSet data,
 MissingnessInjector.Spec spec)

Injects missing values into a copy of the data, MCAR. This is inject(DataSet, Spec, Mechanism) with MissingnessInjector.Mechanism.none(), and deletes exactly the cells the injector always has for a given seed.

Parameters:

data - The data. Not modified.

spec - The specification; its rate count must match the column count.

Returns:

The data with values deleted, and a report of the realized pattern.

Throws:

IllegalArgumentException - if the rate count does not match the column count.

inject

public static MissingnessInjector.Result inject(DataSet data,
 MissingnessInjector.Spec spec,
 MissingnessInjector.Mechanism mechanism)

Injects missing values into a copy of the data under the given mechanism.

Value effects read the input's values, which are taken to be the true values; a cell already missing in the input contributes 0, the column mean, to its standardized value. Variables that are neither continuous nor discrete are never deleted, but may drive deletion.

Parameters:

data - The data. Not modified.

spec - The specification; its rate count must match the column count.

mechanism - The mechanism terms.

Returns:

The data with values deleted, a report of the realized pattern, and a report of the mechanism.

Throws:

IllegalArgumentException - if the rate count does not match the column count, a column index is out of range, the indicator effects form a cycle, an indicator effect targets an attrition variable, the attrition order repeats a column or names one that cannot be deleted, or the rates decrease along it.

parseRateProfile

public static double[] parseRateProfile(String text,
 List<Node> variables,
 double defaultRate)

Parses a hand-specified rate profile.

Accepts a comma or whitespace separated list of rates, one per variable in dataset order, or a list of name=rate assignments in any order, with variables not named taking defaultRate. The two forms may not be mixed. An empty or blank text gives the default rate throughout.

Parameters:

text - The profile text.

variables - The dataset variables, in column order.

defaultRate - The rate for variables not named, used only by the name=rate form.

Returns:

The rates, in column order.

Throws:

IllegalArgumentException - on an unparsable number, an out-of-range rate, an unknown variable name, or a positional list of the wrong length. The message says which.