Class MiceLiteImputer

java.lang.Object

edu.cmu.tetrad.data.missing.MiceLiteImputer

All Implemented Interfaces:

MultipleImputer

public final class MiceLiteImputer
extends Object
implements MultipleImputer

A "lite" chained-equations (MICE-style) multiple imputer for continuous, discrete, or mixed data, using predictive mean matching (PMM) as the single imputation engine for both variable types. For each variable with missingness, a linear regression of that variable on all other variables is fit over the rows where it is observed; each missing entry is then filled by copying the observed value of a donor row chosen at random from the k rows whose fitted values are closest to the missing row's fitted value.

Discrete variables with three or more categories are never used as numbers, since their codes carry no order. As a predictor, such a variable enters as one indicator per category (less a reference category). As a target, one regression is fit per category indicator, giving each row a vector of fitted category scores, and donors are the rows nearest in that vector. Both are unchanged by relabeling the categories. A two-category variable is its own indicator and is used as is. Because imputed values are always copied from observed donors, discrete imputations are automatically valid category codes and continuous imputations respect the observed distribution (no Gaussianity assumption). The chain is initialized by marginal hot-deck draws and swept a fixed number of times.

"Lite" caveats, flagged: the conditional models are additive (no interactions), and linear unless powers of the continuous predictors are asked for (see setPredictorDegree(int)); donors for a continuous target are matched on its fitted mean alone unless more is asked for (see setTargetDegree(int)); a discrete target is matched on linear fits to its category indicators, not on a multinomial model; ordered categories are treated like unordered ones, which is valid but ignores the order; and as with MvnImputer this is improper MI (no parameter draws).

The regressions carry a small ridge penalty (as in the R mice package), so that collinear predictors, or more predictors than observed rows, do not stop a fit; predictors that are constant over the rows used are left out. Only if a variable has fewer than two observed rows, has no usable predictor, or the system still cannot be solved does it fall back to marginal hot-deck draws for that update. Such draws are independent of every other variable and so bias toward independence; every fallback, and every fit that was only possible because of the ridge, is recorded (see getLoggedEvents()) and logged, so that none of this happens silently.

Every other variable predicts each incomplete variable unless that would leave too few observed rows per predictor term, in which case only the variables most correlated with it, or with whether it is missing, are used (see setRowsPerPredictor(int)); a minimum correlation can also be required (see setMinCorrelation(double)). Any such selection is recorded with the other events.

Version:

$Id: $Id

Author:

josephramsey

Constructor Summary

Constructors

Constructor	Description
MiceLiteImputer()	Constructs an imputer with the defaults: 5 donors, 5 sweeps.
MiceLiteImputer(int numDonors, int numSweeps)	Constructs an imputer.
MiceLiteImputer(int numDonors, int numSweeps, double ridge)	Constructs an imputer.

Method Summary

Modifier and Type	Method	Description
List<String>	getLoggedEvents()	What went less than cleanly in the last call to impute: fallbacks to marginal hot-deck draws, predictors left out as constant, and fits that were possible only because of the ridge.
List<DataSet>	impute(DataSet dataSet, int m, long seed)	Produces m completed copies of the given dataset.
void	setCoefficientDraws(boolean coefficientDraws)	Sets whether each column update draws its regression coefficients from their approximate posterior before computing the fitted values of the MISSING rows, as R's mice does ("type-1" matching in the taxonomy of Morris et al.
void	setMinCorrelation(double minCorrelation)	Sets the minimum score a variable needs to be used as a predictor of a target (as quickpred does in the R mice package).
void	setPredictorDegree(int predictorDegree)	Sets the highest power of a continuous predictor used in the regressions.
void	setRowsPerPredictor(int rowsPerPredictor)	Sets the fewest observed rows of a target allowed per predictor term (a continuous or two-category variable is one term; a variable with K categories is K - 1).
void	setTargetDegree(int targetDegree)	Sets the number of powers of a continuous variable whose fitted values donors are matched on when that variable is imputed.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MiceLiteImputer

public MiceLiteImputer()

Constructs an imputer with the defaults: 5 donors, 5 sweeps.

MiceLiteImputer

public MiceLiteImputer(int numDonors,
 int numSweeps)

Constructs an imputer.

Parameters:

numDonors - The number of donor candidates for PMM; at least 1.

numSweeps - The number of chained sweeps; at least 1.

MiceLiteImputer

public MiceLiteImputer(int numDonors,
 int numSweeps,
 double ridge)

Constructs an imputer.

Parameters:

numDonors - The number of donor candidates for PMM; at least 1.

numSweeps - The number of chained sweeps; at least 1.

ridge - The ridge penalty for the regressions, as a fraction of each standardized predictor's sum of squares; positive. The default is 1e-5, small enough to leave a well-posed fit unchanged.

Method Details

setMinCorrelation

public void setMinCorrelation(double minCorrelation)

Sets the minimum score a variable needs to be used as a predictor of a target (as quickpred does in the R mice package). The score is the larger of two absolute correlations, computed once from the observed data: the variable with the target, over rows where both are observed, and the variable with the indicator of whether the target is missing. For a discrete variable with three or more categories, the largest over its category indicators is used.

The default, 0, sets no minimum, and every other variable is a predictor unless the row limit applies. A minimum above 0 makes each imputed value independent of the variables left out, given those kept, which weakens weak dependencies in the completed data; for causal search, prefer leaving this at 0 and relying on the row limit, which only acts when a regression would otherwise be poorly determined.

Parameters:

minCorrelation - the minimum, in [0, 1)

setRowsPerPredictor

public void setRowsPerPredictor(int rowsPerPredictor)

Sets the fewest observed rows of a target allowed per predictor term (a continuous or two-category variable is one term; a variable with K categories is K - 1). When using every eligible variable would give fewer, the variables are taken in order of score (see setMinCorrelation(double)) until the limit is reached, and the rest are left out for that target. The default is 3.

Parameters:

rowsPerPredictor - the limit, at least 1; or 0 for no limit, so that every eligible variable is used however few rows there are

setPredictorDegree

public void setPredictorDegree(int predictorDegree)

Sets the highest power of a continuous predictor used in the regressions. With degree d, a continuous predictor x enters as z, z^2, ..., z^d, where z is x standardized by the mean and standard deviation of its observed values. The default, 1, is the linear model. A degree of 2 or 3 lets the fitted value of a variable follow a curved (including non-monotone) dependence on a predictor, such as y = x^2 + e, which a linear fit misses entirely. Discrete predictors are unaffected.

Caveats: the model is still additive, so a pure interaction (y = x * w + e) is not captured; each continuous predictor now costs d terms against the row limit (see setRowsPerPredictor(int)), so fewer variables may be used on small samples; and this only helps where the dependence shows in the mean of the variable being imputed given the predictor, which for y = x^2 + e is true of y given x but not of x given y.

Parameters:

predictorDegree - the degree, from 1 to 5

setTargetDegree

public void setTargetDegree(int targetDegree)

Sets the number of powers of a continuous variable whose fitted values donors are matched on when that variable is imputed. With degree d, a regression is fit for each of z, z^2, ..., z^d, where z is the variable standardized by the mean and standard deviation of its observed values, and donors are the observed rows nearest in the resulting vector of fitted values. Each power is scaled to unit variance, so a power the predictors cannot predict contributes little to the distance. The default, 1, is ordinary predictive mean matching.

A degree of 2 matches on the fitted spread as well as the fitted mean, which matters when the predictors say nothing about the variable's mean but do say how far from it the variable is. The case in point is imputing x where y = x^2 + e and x is symmetric about zero: the mean of x given y is zero whatever y is, so matching on the mean picks donors without regard to y, while the mean of x^2 given y tracks y. Since every variable in a chain is imputed from its effects as well as its causes, this is the counterpart of setPredictorDegree(int) for the other direction. Discrete variables are unaffected.

Caveat: the powers of the variable are themselves fit by regressions that are linear in the predictor terms, so this pays off most with the predictor degree raised as well.

Parameters:

targetDegree - the degree, from 1 to 4

setCoefficientDraws

public void setCoefficientDraws(boolean coefficientDraws)

Sets whether each column update draws its regression coefficients from their approximate posterior before computing the fitted values of the MISSING rows, as R's mice does ("type-1" matching in the taxonomy of Morris et al. 2014): the residual variance is drawn as RSS / chi-squared(nObs - q), the coefficient draw is b + sigma L'^{-1} eta with eta standard normal and L the Cholesky factor of the ridged cross-product matrix, and the intercept gets its own sigma / sqrt(nObs) draw. The observed rows' fitted values keep the point estimates, per type-1 matching. Off by default, which reproduces this class's previous behavior exactly ("type-0" matching): point estimates throughout, so between-imputation variance reflects only donor and chain randomness and Rubin's rules are anti-conservative. Turn this on when pooled test statistics will be computed from the imputations; it mainly matters there, since score averaging never reads the between-imputation variance. An update with nObs - q < 3 skips the draw and is logged.

Parameters:

coefficientDraws - True to draw coefficients (mice-style proper imputation), false for point estimates.

getLoggedEvents

public List<String> getLoggedEvents()

What went less than cleanly in the last call to impute: fallbacks to marginal hot-deck draws, predictors left out as constant, and fits that were possible only because of the ridge. Each entry names the variable being imputed and says in how many of its updates the event occurred. Empty if every regression was well posed.

Returns:

The events, one per line of report.

impute

public List<DataSet> impute(DataSet dataSet,
 int m,
 long seed)

Produces m completed copies of the given dataset.

Specified by:

impute in interface MultipleImputer

Parameters:

dataSet - The dataset; must contain at least one missing value.

m - The number of imputations; at least 1. With 1, the single completed dataset shows nothing of the uncertainty due to the missing values, which is what the differences among several show.

seed - A random seed for reproducibility, or -1 for a random seed.

Returns:

The m completed datasets.