Class IndTestBasisExpandedGcm

java.lang.Object
edu.cmu.tetrad.search.test.IndTestBasisExpandedGcm
All Implemented Interfaces:
RawMarginalIndependenceTest, IndependenceTest, RowsSettable
public class IndTestBasisExpandedGcm extends Object implements IndependenceTest, RawMarginalIndependenceTest, RowsSettable
Basis-expanded Generalised Covariance Measure (GCM) conditional independence test.

To test X _||_ Y | Z, this test residualizes the basis-expanded blocks of X and Y on the basis-expanded block of Z (Shah & Peters, 2020, generalized to a grid of functionals): each embedded column f_j(X) and g_k(Y) is regressed on [1, basis(Z)] by ridge-stabilized OLS, and the test statistic is the maximum over the (j, k) grid of the studentized residual cross-covariances 

    T_jk = sqrt(n) * mean(e_j * f_k) / sd(e_j * f_k),
where e_j and f_k are the residual columns. Under the null, each T_jk is asymptotically standard normal, and the p-value for the max statistic is computed by a Rademacher multiplier bootstrap over the recentered residual products, which accounts for the dependence among the grid cells; a Sidak bound over the grid is used instead if the number of multiplier samples is set to zero. The bootstrap is seeded deterministically from the variable indices of the independence fact, so repeated calls for the same fact return the same p-value regardless of call order.

Rationale. The GCM construction is Neyman-orthogonal: substituting e = eps + (f - fhat) and expanding the statistic, the only bias term is the product of the two regression errors, so the regressor's complexity never enters the null distribution -- only its convergence rate matters, and only through that product. Because the regressor used here (OLS on the basis expansion of Z) is a linear smoother of modest dimension, no sample splitting is required (Shah & Peters, Sec. 5): the hat-matrix effect on the residual products is second order. Testing the full grid of basis pairs, rather than the single residual covariance of raw X and raw Y, detects dependence expressed in any basis direction (variance dependence, sign dependence, etc.), not only conditional-mean dependence. The max-type statistic retains power under sparse alternatives, where only a few grid cells carry signal.

Contrast with IndTestBasisFunctionBlocks: the Wilks block test assumes a joint Gaussian model over the embedded columns and derives its degrees of freedom from that model; this test makes no distributional assumption beyond moments, is heteroskedasticity-robust through its self-normalization, and its validity is insensitive to misspecification of the Z-regressions up to the product-of-errors term. Its known blind spots are dependencies orthogonal to every tested functional (mitigated, not eliminated, by the basis grid) and interactions strictly within Z (the sieve regressor is additive over Z's basis columns).

With adaptive basis selection enabled, uninformative higher-order basis columns are pruned once, on the full sample, by the BIC-crossing screen of Embedding.pruneUninformativeBasisColumns(DataSet, Map, ICovarianceMatrix), which controls the size of the (j, k) grid as the truncation limit grows and gives the test convergent behavior in the truncation limit.

Limitations: the multiplier bootstrap assumes independent rows; under serial or spatial dependence the residual products would require block or kernel multipliers (cf. the wild-bootstrap Markov check machinery), which is not implemented here. Effective sample size is likewise not supported. Discrete variables enter through their indicator columns via the embedding.

Author:
josephramsey

  • Constructor Details

    • IndTestBasisExpandedGcm

      public IndTestBasisExpandedGcm(DataSet dataSet, int truncationLimit, int basisType, double lambda, boolean adaptiveBasisSelection)
      Constructs the test.
      Parameters:
      dataSet - the dataset to test on; must not be null.
      truncationLimit - the basis truncation limit; must be >= 1.
      basisType - the basis family (see StatUtils.basisFunctionValue); Legendre = 1.
      lambda - ridge added to the Gram matrix of the Z-regressions (>= 0).
      adaptiveBasisSelection - if true, prune uninformative higher-order basis columns once, on the full sample, by the BIC-crossing screen; see Embedding.pruneUninformativeBasisColumns(DataSet, Map, ICovarianceMatrix).

    • IndTestBasisExpandedGcm

      public IndTestBasisExpandedGcm(DataSet dataSet, int truncationLimit, int zTruncationLimit, int basisType, double lambda, boolean adaptiveBasisSelection)
      Constructs the test with a separate truncation limit for the Z-side (regression) embedding.

      The GCM guarantee requires the regression sieve to be rich enough to approximate E[f_j(X) | Z] for every tested grid function f_j. Under nonlinear relations these conditional means are COMPOSITIONS -- if X is roughly a degree-r function of Z, then f_j(X) has conditional mean of degree about j * r in Z -- so the regression basis must be strictly richer than the tested grid, or the product-of-errors bias term is O(1) and the test over-rejects true conditional independence (the same mechanism, one level up, as OLS residualization failing under a nonlinear conditional mean). By default (zTruncationLimit = 0), the Z-side truncation is set to twice the grid truncation, which covers quadratic relations exactly and is a reasonable sieve default; it can be raised for more strongly nonlinear systems.

      Parameters:
      dataSet - the dataset to test on; must not be null.
      truncationLimit - the basis truncation limit for the tested X/Y grid; must be >= 1.
      zTruncationLimit - the truncation limit for the Z-side regression embedding; 0 selects the default 2 * truncationLimit; otherwise must be >= 1 (values below truncationLimit are raised to truncationLimit).
      basisType - the basis family (see StatUtils.basisFunctionValue); Legendre = 1.
      lambda - ridge added to the Gram matrix of the Z-regressions (>= 0).
      adaptiveBasisSelection - if true, prune uninformative higher-order basis columns of the TESTED GRID once, on the full sample, by the BIC-crossing screen; the Z-side regression embedding is never pruned (richer is safer for the nuisance regressions); see Embedding.pruneUninformativeBasisColumns(DataSet, Map, ICovarianceMatrix).

  • Method Details

    • checkIndependence

      public IndependenceResult checkIndependence(Node x, Node y, Set<Node> z)
      Checks the independence of x and y given z using the basis-expanded GCM statistic.
      Specified by:
      checkIndependence in interface IndependenceTest
      Parameters:
      x - the first variable.
      y - the second variable.
      z - the conditioning set.
      Returns:
      the independence result, with the max-statistic p-value.

    • computePValue

      public double computePValue(double[] x, double[] y)
      Computes the marginal p-value between two raw columns by building a two-column dataset and running the marginal version of this test on it.
      Specified by:
      computePValue in interface RawMarginalIndependenceTest
      Parameters:
      x - the first column.
      y - the second column.
      Returns:
      the marginal basis-expanded GCM p-value.

    • computePValue

      public double computePValue(double[] x, double[][] Y)
      Multivariate fallback: tests x against each column of Y and combines with Fisher's method.
      Specified by:
      computePValue in interface RawMarginalIndependenceTest
      Parameters:
      x - the first column.
      Y - the matrix of second columns.
      Returns:
      the Fisher-combined p-value.

    • getVariables

      public List<Node> getVariables()
      Returns the variables of this test (the caller's Node instances).
      Specified by:
      getVariables in interface IndependenceTest
      Returns:
      the variables.

    • getData

      public DataModel getData()
      Returns the original dataset.
      Specified by:
      getData in interface IndependenceTest
      Returns:
      the dataset.

    • isVerbose

      public boolean isVerbose()
      Returns whether verbose output is enabled.
      Specified by:
      isVerbose in interface IndependenceTest
      Returns:
      true if verbose.

    • setVerbose

      public void setVerbose(boolean verbose)
      Sets whether verbose output is enabled.
      Specified by:
      setVerbose in interface IndependenceTest
      Parameters:
      verbose - true to enable.

    • getAlpha

      public double getAlpha()
      Returns the significance level.
      Specified by:
      getAlpha in interface IndependenceTest
      Returns:
      alpha.

    • setAlpha

      public void setAlpha(double alpha)
      Sets the significance level.
      Specified by:
      setAlpha in interface IndependenceTest
      Parameters:
      alpha - the significance level, strictly between 0 and 1.

    • setNumMultiplierSamples

      public void setNumMultiplierSamples(int numMultiplierSamples)
      Sets the number of Rademacher multiplier bootstrap samples used for the max-statistic p-value. Zero selects the (conservative) Sidak bound over the grid instead.
      Parameters:
      numMultiplierSamples - the number of samples; must be >= 0.

    • setControlFunction

      public void setControlFunction(boolean controlFunction)
      Sets whether each side's regression design is augmented with a univariate basis in that side's first-stage fitted mean (a control function), which captures the interaction and composition terms the additive Z-sieve cannot represent. See the field documentation for the rationale and caveats.
      Parameters:
      controlFunction - true to enable the control-function augmentation.

    • getRows

      public List<Integer> getRows()
      Returns the current row subset, or null for all rows.
      Specified by:
      getRows in interface RowsSettable
      Returns:
      the rows.

    • setRows

      public void setRows(List<Integer> rows)
      Sets the row subset to test on (null = all rows). The embedding and any adaptive pruning are fixed at construction, from the full sample, so the block structure is identical across subsets.
      Specified by:
      setRows in interface RowsSettable
      Parameters:
      rows - the row indices, or null for all rows.

    • toString

      public String toString()
      Returns a string representation of this test.
      Specified by:
      toString in interface IndependenceTest
      Overrides:
      toString in class Object
      Returns:
      the name of this test.