Class MissingDataUtils

java.lang.Object
edu.cmu.tetrad.data.missing.MissingDataUtils
public final class MissingDataUtils extends Object
Small shared utilities for the missing-data policies: listwise deletion, resolution of a possibly-null MissingDataSpec to a policy (with a logged warning where the historical behavior was a silent switch), a cheap missingness summary suitable for logging from constructors, and the effective-sample-size calculation for analyses run on covariance matrices estimated from incomplete data.
Version:
$Id: $Id
Author:
josephramsey

  • Method Details

    • fromParameters

      public static MissingDataSpec fromParameters(Parameters parameters)
      Builds a MissingDataSpec from algcomparison/GUI/py-tetrad parameters (see the MISSING_* constants in Params), or returns null--meaning "legacy default behavior"--if the policy parameter is absent or "default". Policy values (case-insensitive): "fail", "listwise", "testwise", "em" (or "emCovariance"), "pairwise" (or "pairwiseCovariance"; see MissingDataPolicy.PAIRWISE_COVARIANCE), "mi" (or "multipleImputation"). ESS modes: "fullN", "minPairwise", "meanPairwise".
      Parameters:
      parameters - The parameters.
      Returns:
      The spec, or null for the legacy default.
      Throws:
      IllegalArgumentException - If the policy or ESS mode string is unrecognized.

    • resolveDeletionPolicy

      public static DataSet resolveDeletionPolicy(DataSet dataSet, MissingDataSpec spec, String caller)
      Applies a missing-data specification to a dataset for a component that supports only the deletion policies natively: returns the dataset itself when it is complete or the policy is TESTWISE (the component then does its own per-family or per-test row selection), its complete cases under LISTWISE, and throws otherwise. A null spec on missing data is treated as FAIL.
      Parameters:
      dataSet - The dataset.
      spec - The spec, or null (FAIL on missing data).
      caller - The name of the calling component, for messages.
      Returns:
      The dataset to analyze.
      Throws:
      IllegalArgumentException - If the policy is FAIL or EM_COVARIANCE and the dataset has missing values.
      UnsupportedOperationException - If the policy is MULTIPLE_IMPUTATION.

    • listwiseDelete

      public static DataSet listwiseDelete(DataSet dataSet)
      Returns a new dataset consisting of the rows of the given dataset that have no missing entries (listwise deletion). The given dataset is not modified.
      Parameters:
      dataSet - The dataset.
      Returns:
      The complete-case dataset.
      Throws:
      IllegalArgumentException - If no complete rows remain.

    • resolveOrWarn

      public static MissingDataPolicy resolveOrWarn(DataSet dataSet, MissingDataSpec spec, String caller)
      Resolves a possibly-null spec to a policy for a component that historically switched silently to test-wise deletion on missing data. If the spec is non-null, its policy is returned. If the spec is null and the dataset has missing values, MissingDataPolicy.TESTWISE is returned for backward compatibility, but a warning naming the caller, the policy, and its MCAR assumption--together with a brief missingness summary--is logged through TetradLogger, replacing the silent switch. If the dataset is complete, the policy is irrelevant and TESTWISE is returned.
      Parameters:
      dataSet - The dataset.
      spec - The spec, or null if the user did not provide one.
      caller - The name of the calling component, for the log message.
      Returns:
      The resolved policy.

    • briefSummary

      public static String briefSummary(DataSet dataSet)
      A one-line missingness summary cheap enough to compute in a constructor: one O(n * p) pass giving the overall missing rate, the number of complete rows, and the worst variable. (The full MissingDataAudit, which additionally computes pairwise complete counts in O(n * p^2), is opt-in.)
      Parameters:
      dataSet - The dataset.
      Returns:
      This summary.

    • effectiveSampleSize

      public static int effectiveSampleSize(DataSet dataSet, MissingDataSpec spec)
      The effective sample size to use for penalized scores when the analysis runs on a covariance matrix estimated from incomplete data, per the spec's MissingDataSpec.EffectiveSampleSizeMode. For FULL_N this returns -1, the existing convention meaning "use the nominal sample size"; the pairwise modes compute the pairwise complete counts (O(n * p^2)) via MissingDataAudit.
      Parameters:
      dataSet - The dataset the covariance matrix was estimated from.
      spec - The spec.
      Returns:
      The effective sample size, or -1 for the nominal sample size.

    • estimateCovariance

      public static ICovarianceMatrix estimateCovariance(DataSet dataSet, MissingDataSpec spec)
      The covariance matrix of a dataset with missing values under a policy that estimates one: by EM with the spec's ridge, tolerance and iteration limit under EM_COVARIANCE, or by pairwise deletion (PairwiseCovarianceEstimator) under PAIRWISE_COVARIANCE. Every component that accepts an EM covariance gets its matrix here, so each supports both; see MissingDataPolicy.estimatesCovariance().
      Parameters:
      dataSet - The dataset.
      spec - The spec; its policy is expected to be EM_COVARIANCE or PAIRWISE_COVARIANCE.
      Returns:
      The covariance matrix.

    • gate

      public static DataModel gate(DataModel dataModel, Parameters parameters, boolean specAware, String caller)
      The wrapper-level missing-data gate, called at the top of every algcomparison test and score wrapper's getTest/getScore method. This is the point at which the explicit-choice contract for missing data is enforced for all user-facing interfaces (the GUI search box, algcomparison, and py-tetrad, all of which construct tests and scores through these wrappers).

      Semantics: if the data model is null, is not a tabular DataSet (e.g., a covariance matrix, which cannot contain missing values), or is a complete dataset, it is returned unchanged and the policy is irrelevant. Otherwise the dataset contains missing values, and:

      • Policy "default" (or unset): an IllegalArgumentException is thrown asking the user to choose a policy. This replaces the previous behavior in which components silently (later: with a logged warning) switched to test-wise deletion. To restore the previous behavior explicitly, set the policy to "testwise".
      • "fail": an exception is thrown, per that policy's contract.
      • "listwise": the complete-case dataset is returned. This is supported for every test and score, since it is a pure data transformation performed here, upstream of the component.
      • "testwise" or "em": the dataset is passed through unchanged if specAware is true (the wrapper passes a MissingDataSpec to a component that implements these policies natively); otherwise an exception is thrown naming "listwise" as the supported alternative.
      • "mi": an exception is thrown. Multiple imputation runs the whole search over m imputed datasets via ImputationSearch and is not the responsibility of a single test or score; it is not yet wired into the wrapper interfaces. (ImputationSearch hands complete datasets to the algorithm it runs, so this gate passes through in that context.)
      Statistical caveats as elsewhere: test-wise and listwise deletion are unbiased in general only under MCAR; EM is valid under MAR for approximately multivariate normal data.
      Parameters:
      dataModel - The data model handed to the wrapper; may be null or a non-tabular model.
      parameters - The parameters, from which Params.MISSING_DATA_POLICY is read.
      specAware - True if the wrapper passes a MissingDataSpec through to a component with native TESTWISE / EM_COVARIANCE support.
      caller - The user-facing name of the test or score, for error messages.
      Returns:
      The data model, possibly replaced by its complete-case version under the "listwise" policy.
      Throws:
      IllegalArgumentException - As described above.

    • gate

      public static DataModel gate(DataModel dataModel, Parameters parameters, Set<String> nativePolicies, String caller)
      The finer-grained form of gate(DataModel, Parameters, boolean, String), for components that support some native policies but not others (e.g., the Chi-Square test performs test-wise deletion natively but has no EM-covariance route, while Fisher Z supports both). The boolean form delegates here with {"testwise", "em"} or the empty set.
      Parameters:
      dataModel - The data model handed to the wrapper; may be null or a non-tabular model.
      parameters - The parameters, from which Params.MISSING_DATA_POLICY is read.
      nativePolicies - The policies the underlying component implements natively, from among "testwise" and "em". ("fail" and "listwise" are implemented here for every component, and "mi" is a search-level policy.)
      caller - The user-facing name of the test or score, for error messages.
      Returns:
      The data model, possibly replaced by its complete-case version under the "listwise" policy.
      Throws:
      IllegalArgumentException - As for the boolean form.

    • gate

      public static BlockSpec gate(BlockSpec blockSpec, Parameters parameters, String caller)
      The block-spec variant of gate(DataModel, Parameters, boolean, String), for wrappers that construct block tests and scores from a BlockSpec rather than from the data model handed to getTest/getScore. The embedded dataset is checked; under the "listwise" policy a new BlockSpec is returned wrapping the complete-case dataset with the same blocks, block variables, and ranks (blocks index columns, which row deletion does not disturb). This form declares no native policies; block components with native test-wise support (e.g., the Wilks blocks test) use the four-argument form.
      Parameters:
      blockSpec - The block spec.
      parameters - The parameters.
      caller - The user-facing name of the test or score, for error messages.
      Returns:
      The block spec, possibly rebuilt on the complete-case dataset under the "listwise" policy.
      Throws:
      IllegalArgumentException - As for the data-model gate.

    • gate

      public static BlockSpec gate(BlockSpec blockSpec, Parameters parameters, Set<String> nativePolicies, String caller)
      As gate(BlockSpec, Parameters, String), for block components that implement some policies natively; see gate(DataModel, Parameters, java.util.Set, String) for the meaning of the native set.
      Parameters:
      blockSpec - The block spec.
      parameters - The parameters.
      nativePolicies - The policies the block component implements natively, from among "testwise" and "em".
      caller - The user-facing name of the test or score, for error messages.
      Returns:
      The block spec, possibly rebuilt on the complete-case dataset under the "listwise" policy.
      Throws:
      IllegalArgumentException - As for the data-model gate.

    • emCovarianceIfRequested

      public static DataModel emCovarianceIfRequested(DataModel dataModel, Parameters parameters, String caller)
      Implements the "em" missing-data policy at the wrapper level for covariance-consuming scores that have an ICovarianceMatrix constructor but no native MissingDataSpec handling (EBIC, GIC, Poisson prior, Zhang-Shen bound, and the like). Called after gate(DataModel, Parameters, java.util.Set, String) with "em" in the native set: if the data model is a dataset with missing values and the chosen policy is "em", the EM-estimated covariance matrix (valid under MAR for approximately multivariate normal data) is returned in its place, with its sample size adjusted per the spec's effective-sample-size mode if a pairwise mode is chosen; the wrapper's covariance branch then constructs the score from it. In every other case the data model is returned unchanged.
      Parameters:
      dataModel - The data model, already passed through the gate.
      parameters - The parameters.
      caller - The user-facing name of the score, for the log message.
      Returns:
      The data model, or the EM-estimated covariance matrix in its place under the "em" policy.