Class StratifiedRankTests

java.lang.Object
edu.cmu.tetrad.util.StratifiedRankTests

public final class StratifiedRankTests extends Object
Rank tests for cross-covariance matrices pooled across the strata of a discrete domain (or batch, site, cohort) variable, after Theorem 4 of Dong, Dai, Ng, Spirtes, and Zhang, "Identifying Partially Observed Causal Models from Heterogeneous/Nonstationary Data" (ICML 2026), the Phase 1 test of their LCD-NOD framework.

The null hypothesis rank(ΣX,Y|T) ≤ r is tested by computing, within each stratum t of the domain variable T, the same Bartlett canonical-correlation statistic that RankTests uses on homogeneous data,

    stat_t = -c_t * sum over i >= r of log(1 - rho_{t,i}^2),
    c_t = (n_t - 1) - (p + q + 1)/2,
and summing: Λ = Σt stat_t is referred to a chi-square distribution whose degrees of freedom are (p - r)(q - r) summed over the strata that enter the sum. The strata are disjoint samples, so their statistics are independent and add; this is the point of difference from PooledRankTest, which combines statistics over multiple imputations of one sample (the imputations share rows and must not be treated as independent) and from RankTests.estimateWilksRankConditioned(SimpleMatrix, int[], int[], int[], int, double), which conditions on a set of continuous variables within one homogeneous sample by Schur complement.

Dong et al. prove (their Theorem 1) that when edge coefficients and exogenous noise change freely across and within domains -- their Nonstationary POLCM model, with coefficient processes independent of noise processes given the domain -- the set of achievable within-domain covariance matrices equals the covariance set of the homogeneous linear model with the same graph. So within-stratum rank constraints carry the same t-separation information as in the homogeneous case, while ranks and partial correlations of the strata mixed together generally do not. Practically: concatenating batches and testing the pooled covariance over-rejects badly; this test does not.

Small strata: a stratum enters a given test only if its Bartlett multiplier c_t is positive; strata too small to discriminate are skipped, and their degrees of freedom are not counted. If no stratum is usable the p-value methods return NaN and estimateRank(int[], int[], double) conservatively reports full rank, mirroring the convention of RankTests.estimateWilksRankFast(SimpleMatrix, int[], int[], int, double). The chi-square approximation is asymptotic in each stratum, so many tiny strata make it rough even when the multipliers are positive; merging small domains is advisable.

The per-stratum canonical correlations for a given pair of column sets are cached, so sequential rank estimation costs one CCA per stratum. The class is thread-safe after construction.

Author:
josephramsey
  • Constructor Details

    • StratifiedRankTests

      public StratifiedRankTests(List<org.ejml.simple.SimpleMatrix> covs, int[] ns)
      Constructs the test from per-stratum correlation (or covariance) matrices and sample sizes. All matrices must be square, of the same dimension, and over the same variables in the same column order; this is the caller's responsibility, since only index arithmetic happens here.
      Parameters:
      covs - the per-stratum correlation or covariance matrices.
      ns - the per-stratum sample sizes, parallel to covs.
  • Method Details

    • fromDataSets

      public static StratifiedRankTests fromDataSets(List<DataSet> dataSets)
      Constructs the test from one data set per stratum. The data sets must all have the same variables in the same order; each is reduced to its correlation matrix.
      Parameters:
      dataSets - one continuous data set per stratum.
      Returns:
      the stratified test.
    • split

      public static List<DataSet> split(DataSet dataSet, Node domain)
      Splits a data set into one data set per value of a discrete domain column, dropping that column. The domain variable must be discrete, or a continuous column whose values are all integral (batch codes stored as doubles are accepted).
      Parameters:
      dataSet - the data set, containing the domain column.
      domain - the domain variable; a column of dataSet.
      Returns:
      the per-stratum data sets, over the remaining variables, in ascending order of domain value.
    • resolveDomainColumn

      public static String resolveDomainColumn(DataSet dataSet, String requested)
      Resolves the name of a domain column for a data set: the named column if it exists; otherwise, if the data set has exactly one discrete column, that column, with a note logged; otherwise an exception listing the discrete columns found. Shared by the interface wrappers that take a domain-column parameter.
      Parameters:
      dataSet - the data set.
      requested - the requested column name; may be null or blank.
      Returns:
      the resolved column name.
    • getNumStrata

      public int getNumStrata()
      Returns the number of strata.
      Returns:
      the number of strata.
    • getTotalSampleSize

      public int getTotalSampleSize()
      Returns the total sample size over all strata.
      Returns:
      the total sample size.
    • getDimension

      public int getDimension()
      Returns the dimension (number of variables) of the stratum matrices.
      Returns:
      the dimension.
    • pValueRankLe

      public double pValueRankLe(int[] x, int[] y, int r)
      The p-value for the null hypothesis rank(ΣX,Y|T) ≤ r, pooling the Bartlett statistic over strata. The column sets may overlap, as they do when conditional independence is tested through augmented sets; shared columns produce canonical correlations of one, which occupy the leading positions and do not enter the statistic when r is at least the overlap size.
      Parameters:
      x - the column indices of the first set.
      y - the column indices of the second set.
      r - the rank bound under the null, with 0 ≤ r < min(|x|, |y|).
      Returns:
      the p-value, or NaN if no stratum is large enough to contribute.
    • pValueRankLeConditioned

      public double pValueRankLeConditioned(int[] xRaw, int[] yRaw, int[] z, int r)
      The p-value for the null hypothesis rank(ΣX,Y|Z,T) ≤ r, where within each stratum the cross-covariance of X and Y is first conditioned on the variable set Z by Schur complement, and the Bartlett statistic is then pooled over strata. With Z empty this is pValueRankLe(int[], int[], int). Z is removed from X and Y first.
      Parameters:
      xRaw - the column indices of the first set.
      yRaw - the column indices of the second set.
      z - the column indices of the conditioning set; may be empty.
      r - the rank bound under the null.
      Returns:
      the p-value, or NaN if no stratum is large enough to contribute, or 1.0 if either set is empty after removal of Z.
    • estimateRank

      public int estimateRank(int[] x, int[] y, double alpha)
      Estimates the rank of ΣX,Y|T as the smallest r whose null hypothesis rank ≤ r is not rejected at the given level, mirroring RankTests.estimateWilksRankFast(SimpleMatrix, int[], int[], int, double). If no stratum is large enough to discriminate, min(|x|, |y|) is returned, conservatively reporting full rank.
      Parameters:
      x - the column indices of the first set.
      y - the column indices of the second set.
      alpha - the significance level.
      Returns:
      the estimated rank.
    • estimateRankConditioned

      public int estimateRankConditioned(int[] xRaw, int[] yRaw, int[] z, double alpha)
      Estimates the rank of ΣX,Y|Z,T, conditioning on the variable set Z within each stratum by Schur complement and pooling over strata, mirroring RankTests.estimateWilksRankConditioned(SimpleMatrix, int[], int[], int[], int, double). Z is removed from X and Y first. If either set is empty after removal, 0 is returned; if no stratum is large enough to discriminate, min(|x|, |y|) is returned.
      Parameters:
      xRaw - the column indices of the first set.
      yRaw - the column indices of the second set.
      z - the column indices of the conditioning set; may be empty.
      alpha - the significance level.
      Returns:
      the estimated rank.