Class PerFactEss

java.lang.Object
edu.cmu.tetrad.search.utils.PerFactEss
public final class PerFactEss extends Object

Per-fact effective sample sizes from block structure

Serially dependent data with block (grouped) structure carries less information than n independent rows, so i.i.d.-based independence tests are anti-conservative and Markov checks yield too many dependence judgments for implied independence constraints. Crucially, the right correction is fact-dependent: for a constraint X _||_ Y | Z in which the participating variables are constant within blocks, the data contains only about as many independent observations as there are blocks, while a constraint involving variables that vary within blocks retains most of the nominal sample size.

This class computes, for each independence fact, an effective sample size

    n_eff = n / (1 + (m0 - 1) * icc(X|Z) * icc(Y|Z))

where m0 is the ANOVA average block size for unbalanced designs, m0 = (n - sum_k m_k^2 / n) / (K - 1), and icc(V|Z) is the one-way ANOVA intraclass correlation of V linearly residualized on Z, clamped to [0, 1]. The product-of-ICCs design effect is the standard first-order correction for covariance-type statistics between clustered variables (Kish); it has the right limits: if X and Y are both block-constant (both ICCs 1), n_eff is approximately the number of blocks, and if either varies freely within blocks (ICC near 0), n_eff is approximately n.

Limitations, stated plainly: the residualization is linear, so nonlinear conditioning is corrected only to first order; discrete variables are handled through their numeric category codes, which is crude; and the design-effect rescaling assumes the block dependence changes the information count rather than the shape of the test statistic's null distribution. Where these approximations are in doubt, a block bootstrap of the check is the nonparametric validator.

  • Method Details

    • blockIds

      public static int[] blockIds(DataSet data, List<String> blockColumns)
      Computes block ids (0..K-1, one per row) from the distinct joint values of the given columns.
      Parameters:
      data - The dataset.
      blockColumns - Names of the columns whose distinct joint values define the blocks (e.g., design or grouping variables identified by the data audit).
      Returns:
      An array of length data.getNumRows() assigning each row its block id.

    • effectiveSampleSize

      public static int effectiveSampleSize(DataSet data, int[] blockIds, IndependenceFact fact)
      Computes the per-fact effective sample size for X _||_ Y | Z under the given block structure.
      Parameters:
      data - The dataset (variables matched to the fact's nodes by name).
      blockIds - Row-to-block assignment, as from blockIds(DataSet, List). If null, or if there are fewer than 2 blocks, the nominal sample size is returned.
      fact - The independence fact.
      Returns:
      The effective sample size, clamped to [max(8, |Z| + 5), n].

    • effectiveSampleSize

      public static int effectiveSampleSize(DataSet data, int[] blockIds, Node x, Node y, Set<Node> z)
      Computes the per-fact effective sample size for x _||_ y | z under the given block structure.
      Parameters:
      data - The dataset (variables matched by name).
      blockIds - Row-to-block assignment; null means no block structure.
      x - The first variable.
      y - The second variable.
      z - The conditioning variables.
      Returns:
      The effective sample size, clamped to [max(8, |z| + 5), n].

    • numBlocks

      public static int numBlocks(int[] blockIds)
      Convenience: the number of distinct blocks in a block-id assignment.
      Parameters:
      blockIds - Row-to-block assignment.
      Returns:
      The number of distinct blocks.