Class StratifiedRankTests
The null hypothesis rank(ΣX,Y|T) ≤ r is tested by computing, within each stratum t of the
domain variable T, the same Bartlett canonical-correlation statistic that RankTests uses on homogeneous
data,
stat_t = -c_t * sum over i >= r of log(1 - rho_{t,i}^2),
c_t = (n_t - 1) - (p + q + 1)/2,
and summing: Λ = Σt stat_t is referred to a chi-square distribution whose degrees of freedom
are (p - r)(q - r) summed over the strata that enter the sum. The strata are disjoint samples, so their statistics
are independent and add; this is the point of difference from PooledRankTest,
which combines statistics over multiple imputations of one sample (the imputations share rows and must not be
treated as independent) and from RankTests.estimateWilksRankConditioned(SimpleMatrix, int[], int[], int[], int, double), which conditions on a set of
continuous variables within one homogeneous sample by Schur complement.
Dong et al. prove (their Theorem 1) that when edge coefficients and exogenous noise change freely across and within domains -- their Nonstationary POLCM model, with coefficient processes independent of noise processes given the domain -- the set of achievable within-domain covariance matrices equals the covariance set of the homogeneous linear model with the same graph. So within-stratum rank constraints carry the same t-separation information as in the homogeneous case, while ranks and partial correlations of the strata mixed together generally do not. Practically: concatenating batches and testing the pooled covariance over-rejects badly; this test does not.
Small strata: a stratum enters a given test only if its Bartlett multiplier c_t is positive; strata too small to
discriminate are skipped, and their degrees of freedom are not counted. If no stratum is usable the p-value
methods return NaN and estimateRank(int[], int[], double) conservatively reports full rank, mirroring the convention of
RankTests.estimateWilksRankFast(SimpleMatrix, int[], int[], int, double). The chi-square approximation is asymptotic in each stratum, so many
tiny strata make it rough even when the multipliers are positive; merging small domains is advisable.
The per-stratum canonical correlations for a given pair of column sets are cached, so sequential rank estimation costs one CCA per stratum. The class is thread-safe after construction.
- Author:
- josephramsey
-
Constructor Summary
ConstructorsConstructorDescriptionStratifiedRankTests(List<org.ejml.simple.SimpleMatrix> covs, int[] ns) Constructs the test from per-stratum correlation (or covariance) matrices and sample sizes. -
Method Summary
Modifier and TypeMethodDescriptionintestimateRank(int[] x, int[] y, double alpha) Estimates the rank of ΣX,Y|T as the smallest r whose null hypothesis rank ≤ r is not rejected at the given level, mirroringRankTests.estimateWilksRankFast(SimpleMatrix, int[], int[], int, double).intestimateRankConditioned(int[] xRaw, int[] yRaw, int[] z, double alpha) Estimates the rank of ΣX,Y|Z,T, conditioning on the variable set Z within each stratum by Schur complement and pooling over strata, mirroringRankTests.estimateWilksRankConditioned(SimpleMatrix, int[], int[], int[], int, double).static StratifiedRankTestsfromDataSets(List<DataSet> dataSets) Constructs the test from one data set per stratum.intReturns the dimension (number of variables) of the stratum matrices.intReturns the number of strata.intReturns the total sample size over all strata.doublepValueRankLe(int[] x, int[] y, int r) The p-value for the null hypothesis rank(ΣX,Y|T) ≤ r, pooling the Bartlett statistic over strata.doublepValueRankLeConditioned(int[] xRaw, int[] yRaw, int[] z, int r) The p-value for the null hypothesis rank(ΣX,Y|Z,T) ≤ r, where within each stratum the cross-covariance of X and Y is first conditioned on the variable set Z by Schur complement, and the Bartlett statistic is then pooled over strata.static StringresolveDomainColumn(DataSet dataSet, String requested) Resolves the name of a domain column for a data set: the named column if it exists; otherwise, if the data set has exactly one discrete column, that column, with a note logged; otherwise an exception listing the discrete columns found.Splits a data set into one data set per value of a discrete domain column, dropping that column.
-
Constructor Details
-
StratifiedRankTests
Constructs the test from per-stratum correlation (or covariance) matrices and sample sizes. All matrices must be square, of the same dimension, and over the same variables in the same column order; this is the caller's responsibility, since only index arithmetic happens here.- Parameters:
covs- the per-stratum correlation or covariance matrices.ns- the per-stratum sample sizes, parallel to covs.
-
-
Method Details
-
fromDataSets
Constructs the test from one data set per stratum. The data sets must all have the same variables in the same order; each is reduced to its correlation matrix.- Parameters:
dataSets- one continuous data set per stratum.- Returns:
- the stratified test.
-
split
Splits a data set into one data set per value of a discrete domain column, dropping that column. The domain variable must be discrete, or a continuous column whose values are all integral (batch codes stored as doubles are accepted).- Parameters:
dataSet- the data set, containing the domain column.domain- the domain variable; a column of dataSet.- Returns:
- the per-stratum data sets, over the remaining variables, in ascending order of domain value.
-
resolveDomainColumn
Resolves the name of a domain column for a data set: the named column if it exists; otherwise, if the data set has exactly one discrete column, that column, with a note logged; otherwise an exception listing the discrete columns found. Shared by the interface wrappers that take a domain-column parameter.- Parameters:
dataSet- the data set.requested- the requested column name; may be null or blank.- Returns:
- the resolved column name.
-
getNumStrata
public int getNumStrata()Returns the number of strata.- Returns:
- the number of strata.
-
getTotalSampleSize
public int getTotalSampleSize()Returns the total sample size over all strata.- Returns:
- the total sample size.
-
getDimension
public int getDimension()Returns the dimension (number of variables) of the stratum matrices.- Returns:
- the dimension.
-
pValueRankLe
public double pValueRankLe(int[] x, int[] y, int r) The p-value for the null hypothesis rank(ΣX,Y|T) ≤ r, pooling the Bartlett statistic over strata. The column sets may overlap, as they do when conditional independence is tested through augmented sets; shared columns produce canonical correlations of one, which occupy the leading positions and do not enter the statistic when r is at least the overlap size.- Parameters:
x- the column indices of the first set.y- the column indices of the second set.r- the rank bound under the null, with 0 ≤ r < min(|x|, |y|).- Returns:
- the p-value, or NaN if no stratum is large enough to contribute.
-
pValueRankLeConditioned
public double pValueRankLeConditioned(int[] xRaw, int[] yRaw, int[] z, int r) The p-value for the null hypothesis rank(ΣX,Y|Z,T) ≤ r, where within each stratum the cross-covariance of X and Y is first conditioned on the variable set Z by Schur complement, and the Bartlett statistic is then pooled over strata. With Z empty this ispValueRankLe(int[], int[], int). Z is removed from X and Y first.- Parameters:
xRaw- the column indices of the first set.yRaw- the column indices of the second set.z- the column indices of the conditioning set; may be empty.r- the rank bound under the null.- Returns:
- the p-value, or NaN if no stratum is large enough to contribute, or 1.0 if either set is empty after removal of Z.
-
estimateRank
public int estimateRank(int[] x, int[] y, double alpha) Estimates the rank of ΣX,Y|T as the smallest r whose null hypothesis rank ≤ r is not rejected at the given level, mirroringRankTests.estimateWilksRankFast(SimpleMatrix, int[], int[], int, double). If no stratum is large enough to discriminate, min(|x|, |y|) is returned, conservatively reporting full rank.- Parameters:
x- the column indices of the first set.y- the column indices of the second set.alpha- the significance level.- Returns:
- the estimated rank.
-
estimateRankConditioned
public int estimateRankConditioned(int[] xRaw, int[] yRaw, int[] z, double alpha) Estimates the rank of ΣX,Y|Z,T, conditioning on the variable set Z within each stratum by Schur complement and pooling over strata, mirroringRankTests.estimateWilksRankConditioned(SimpleMatrix, int[], int[], int[], int, double). Z is removed from X and Y first. If either set is empty after removal, 0 is returned; if no stratum is large enough to discriminate, min(|x|, |y|) is returned.- Parameters:
xRaw- the column indices of the first set.yRaw- the column indices of the second set.z- the column indices of the conditioning set; may be empty.alpha- the significance level.- Returns:
- the estimated rank.
-