Class Embedding

java.lang.Object

edu.cmu.tetrad.search.utils.Embedding

public class Embedding
extends Object

The Embedding class provides utilities for transforming datasets into embedded representations through basis expansions and one-hot encoding. This process is commonly used in preprocessing steps for machine learning or statistical analysis, enabling enhanced variable representations.

Author:

josephramsey, bandrews

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	Embedding.EmbeddedData	Represents the embedded data result, holding the original dataset, the transformed embedded dataset, and a mapping between the indices of original variables and their corresponding transformed variables.

Field Summary

Fields

Modifier and Type	Field	Description
static final double	RANK_TRANSFORM	Sentinel for basisScale: rank-transform each continuous column to a uniform grid on [-1, 1] before applying the basis.

Method Summary

Modifier and Type	Method	Description
static @NotNull Embedding.EmbeddedData	getEmbeddedData(DataSet dataSet, int truncationLimit, int basisType, double basisScale)	Generates an embedding of the given dataset based on the specified parameters.
static Map<Integer, List<Integer>>	pruneUninformativeBasisColumns(DataSet dataSet, Map<Integer, List<Integer>> embedding, ICovarianceMatrix embeddedCorr)	Prunes uninformative basis columns from an embedding using a BIC-crossing screen, so that the effective truncation of each continuous variable is chosen by the data rather than by the truncationLimit parameter.
static DataSet	rankTransformToUnitInterval(DataSet dataSet)	Replaces each continuous column by its mid-ranks mapped to a uniform grid on [-1, 1]: the k-th smallest value (0-based) becomes -1 + 2 (k + 0.5) / N.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

RANK_TRANSFORM

public static final double RANK_TRANSFORM

Sentinel for basisScale: rank-transform each continuous column to a uniform grid on [-1, 1] before applying the basis. With the Legendre basis this makes the basis exactly orthogonal under the empirical measure, gives every row the same leverage, and makes the null distribution of the basis-function LRT chi-square on its nominal degrees of freedom. Min-max scaling (basisScale = 1) instead concentrates the higher-order columns on the few most extreme rows; the resulting LRT null has the chi-square mean but a power-law tail (generalized Pareto shape about 0.37 at truncation 3, N = 1000), so a few high-leverage rows can manufacture a spurious nonlinear edge at any penalty. Rank transformation discards marginal shape and models the copula, which is the appropriate target for detecting nonlinear dependence; it is invariant to monotone marginal transforms and costs about 6 percent of the LRT on Gaussian data.

See Also:

• Constant Field Values

Method Details

rankTransformToUnitInterval

public static DataSet rankTransformToUnitInterval(DataSet dataSet)

Replaces each continuous column by its mid-ranks mapped to a uniform grid on [-1, 1]: the k-th smallest value (0-based) becomes -1 + 2 (k + 0.5) / N. Tied values receive the average of the positions they occupy, so a column with m distinct values still has exactly m distinct values afterwards -- a 0/1 column stays two-valued and its higher-order basis columns are then dropped as rank-deficient, which is the correct behaviour. (Breaking ties by row order would instead spread identical values across the grid and inject row-order noise; on integer-valued features with many ties that is not harmless.) Discrete columns are left untouched.

Parameters:

dataSet - The data.

Returns:

A copy with continuous columns rank-transformed.

getEmbeddedData

@NotNull
public static @NotNull Embedding.EmbeddedData getEmbeddedData(DataSet dataSet,
 int truncationLimit,
 int basisType,
 double basisScale)

Generates an embedding of the given dataset based on the specified parameters. This method transforms the dataset using basis functions according to the truncation limit, basis type, and basis scaling parameters. It supports the handling of both discrete and continuous variables with tailored transformations.

Parameters:

dataSet - the input dataset to be transformed. Must not be null.

truncationLimit - the maximum number of basis function orders to include in the transformation. Must be a positive integer.

basisType - the type of basis function to use in the transformation.

basisScale - the scaling mode for the dataset. Must be a positive number, 0 to standardize the dataset, -1 to skip scaling, or RANK_TRANSFORM to apply a rank transformation to the unit interval.

Returns:

the transformed dataset represented as an instance of EmbeddedData.

Throws:

IllegalArgumentException - if the dataset is null, if truncationLimit is not positive, or if basisScale is invalid.

pruneUninformativeBasisColumns

public static Map<Integer, List<Integer>> pruneUninformativeBasisColumns(DataSet dataSet,
 Map<Integer, List<Integer>> embedding,
 ICovarianceMatrix embeddedCorr)

Prunes uninformative basis columns from an embedding using a BIC-crossing screen, so that the effective truncation of each continuous variable is chosen by the data rather than by the truncationLimit parameter.

For each continuous variable's embedded block (columns in low-order-first order, orthonormalized within the block by getEmbeddedData(DataSet, int, int, double)), the first column (spanning the linear term) is always kept. A higher-order column is kept if and only if there exists at least one embedded column of some OTHER variable with which its squared sample correlation crosses the single-regressor BIC threshold, adjusted for the number of partner columns scanned (an RIC/EBIC-style multiplicity guard):

    n * log(1 / (1 - r^2))  >  log(n) + 2 * log(M),

where n is the sample size and M is the number of partner columns. The left side is the deviance gain of a single-regressor Gaussian regression in either direction, so the screen keeps exactly those basis directions that could produce a BIC-positive pairwise association somewhere in the system at unit penalty discount. A column that fails the screen for every partner can only ever contribute penalty, never a BIC-positive likelihood gain, in any pairwise relation; dropping it discounts the uninformative basis function to zero.

The screen is computed once, from the full-sample correlation matrix of the embedded data, independently of any particular graph or test. Because the pruned embedding is a single fixed embedding of the data, downstream scores computed over it remain (penalized) joint likelihoods, and score equivalence across Markov-equivalent DAGs is preserved. As the truncation limit is raised past the data-supported order, newly added columns are either dropped as numerically rank-deficient by getEmbeddedData(DataSet, int, int, double) or screened out here, so the pruned embedding -- and hence the score and test -- converge rather than degrade.

Limitations, by design: (1) the screen is marginal (pairwise, unconditional), so a basis direction that is uncorrelated with every other variable's block marginally but relevant conditionally (a suppression pattern) will be dropped; (2) the screen uses unit penalty discount, which is deliberately more inclusive than any penalty discount ≥ 1 used by a downstream score; (3) the screen uses the full-sample size, not any caller-set effective sample size. Discrete variables' indicator blocks are not nested expansions and are left untouched, as is any continuous block with a single column.

Parameters:

dataSet - the ORIGINAL (pre-embedding) dataset, used to distinguish discrete from continuous variables; must have variable indices aligned with the keys of embedding.

embedding - the embedding map produced by getEmbeddedData(DataSet, int, int, double), mapping each original variable index to its embedded column indices in low-order-first order.

embeddedCorr - the correlation matrix of the embedded dataset (e.g., new CorrelationMatrix(embeddedData)); its sample size is used for the BIC threshold.

Returns:

a new map with the same keys, in which each continuous variable's column list has been pruned to its screened-in columns (first column always retained). The input map is not modified.