Class DataTransforms
DataTransforms class.
- Version:
- $Id: $Id
- Author:
- josephramsey
-
Method Summary
Modifier and TypeMethodDescriptionstatic DataSetaddMissingData(DataSet inData, double[] probs) Adds missing data values to cases in accordance with probabilities specified in a double array which has as many elements as there are columns in the input dataset.alignVariableOrder(List<DataSet> dataSets, List<String> names) Puts the variables of each of the given data sets in the given order of names.static intappendBlockIdColumn(DataSet dataSet, List<String> keyColumns, String columnName) Appends a new block ID column to the given dataset.appendWithinBlockCenteredColumns(DataSet dataSet, String blockColumn, String suffix) Appends within-block centered ("fixed effects" / within-transformed) copies of the data set's continuous columns, in place: for each continuous column that varies within blocks, a new column named name + suffix is appended holding the values minus their within-block observed means.static double[]center(double[] d) Centers the values in the given array by subtracting the mean of the array from each element.static DataSetSubtracts the mean of each column from each datum that column.center.static MatrixcenterData(Matrix data) centerData.static DataSetconcatenate(DataSet... dataSets) concatenate.static DataSetconcatenate(DataSet dataSet1, DataSet dataSet2) concatenate.static Matrixconcatenate(Matrix... dataSets) concatenate.static DataSetconcatenate(List<DataSet> dataSets) concatenate.static DataSetconvertNumericalDiscreteToContinuous.static voidcopyColumn(Node node, DataSet source, DataSet dest) copyColumn.static ICovarianceMatrixcovarianceGaussianCopula(DataSet dataSet) Returns the Gaussian copula ("nonparanormal") correlation matrix of a continuous data set: for each pair of variables the sample Kendall's tau-b is computed and mapped to the latent Gaussian correlation bystatic ICovarianceMatrixcovarianceNonparanormalDrton(DataSet dataSet) Deprecated.This does NOT compute the nonparanormal (Gaussian copula) correlation estimator, despite the name.static DataSetdiscretize(DataSet dataSet, int numCategories, boolean variablesCopied) discretize.static DataSetgetBootstrapSample(DataSet data, int sampleSize) getBootstrapSample.static DataSetgetBootstrapSample(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator) Get dataset sampled with replacement.static MatrixgetBootstrapSample(Matrix data, int sampleSize) getBootstrapSample.getConstantColumns(DataSet dataSet) getConstantColumns.static DataSetgetLinearizedTransformed(DataSet dataSet) Returns a linearized ("un-warped") version of the dataset: each variable is passed through a monotone Yeo-Johnson transform chosen to make its relations to the other variables as linear as possible (post-nonlinear model un-warping).static DataSetgetNonparanormalTransformed(DataSet dataSet) Returns a nonparanormal-transformed version of the dataset.static DataSetgetResamplingDataset(DataSet data, int sampleSize) getResamplingDataset.static DataSetgetResamplingDataset(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator) Get dataset sampled without replacement.static doublekendallsTauB(double[] x, double[] y) Kendall's tau-b for two columns, computed in O(n log n) by merge-sort inversion counting, with pairwise-complete deletion of rows where either value is NaN.static DataSetLog or unlog datastatic DataSetlogData(DataSet dataSet, Map<String, LogTransformSpec> specs) Applies a logarithmic transform to selected variables of the given dataset, each with its own offset, base, and direction.static DataSetremoveConstantColumns(DataSet dataSet) removeConstantColumns.static DataSetremoveRandomColumns(DataSet dataSet, double aDouble) removeRandomColumns.static DataSetreplaceMissingWithRandom(DataSet inData) replaceMissingWithRandom.static DataSetrestrictToMeasured(DataSet fullDataSet) restrictToMeasured.static doublescale(double value, double dataMin, double dataMax, double scaleMin, double scaleMax) Scales a value from one range to another.static DataSetScales the columns of the provided dataset based on the given scale factors.static DataSetScales the continuous variables in the given DataSet to have values in the range [-1, 1].static voidScales the values of a specified node in the given dataset to a specified range [scaleMin, scaleMax].static DataSetshuffleColumns(DataSet dataModel) shuffleColumns.shuffleColumns2(List<DataSet> dataSets) shuffleColumns2.split.static double[]standardizeData(double[] data) standardizeData.static DataSetstandardizeData(DataSet dataSet) standardizeData.static MatrixstandardizeData(Matrix data) standardizeData.static MatrixstandardizeData(Matrix data, List<Node> variables) Standardizes the columns of the given data matrix by centering and scaling.standardizeData(List<DataSet> dataSets) standardizeData.
-
Method Details
-
logData
-
logData
Applies a logarithmic transform to selected variables of the given dataset, each with its own offset, base, and direction.Variables absent from the map are copied through unchanged, as are discrete variables (which are never transformed even if named in the map). Passing a map that assigns the same spec to every continuous variable reproduces
logData(DataSet, double, boolean, int)exactly.No check is made that the offsets keep values in the domain of the logarithm; a variable whose values are not strictly positive under its offset will yield NaN or negative infinity, exactly as the dataset-wide transform does.
LogTransformSpec.safeOffsetFor(double[])is available to callers that want to propose a safe offset to the user first.- Parameters:
dataSet- The dataset to transform.specs- A map from variable name to the transform to apply to that variable.- Returns:
- The transformed dataset.
-
alignVariableOrder
Puts the variables of each of the given data sets in the given order of names. A data set that already has them in that order is returned as it is; one that has exactly those variables in another order is replaced by a reordered copy with the same name; one that does not have exactly those variables is returned as it is, for the caller to refuse. The given list and its data sets are not changed.- Parameters:
dataSets- the data sets.names- the variable names, in the order wanted.- Returns:
- the data sets, in a new list.
-
standardizeData
-
standardizeData
-
center
-
discretize
-
convertNumericalDiscreteToContinuous
public static DataSet convertNumericalDiscreteToContinuous(DataSet dataSet) throws NumberFormatException convertNumericalDiscreteToContinuous.
- Parameters:
dataSet- aDataSetobject- Returns:
- a
DataSetobject - Throws:
NumberFormatException- if any.
-
concatenate
-
concatenate
-
concatenate
-
appendWithinBlockCenteredColumns
public static List<String> appendWithinBlockCenteredColumns(DataSet dataSet, String blockColumn, String suffix) Appends within-block centered ("fixed effects" / within-transformed) copies of the data set's continuous columns, in place: for each continuous column that varies within blocks, a new column named name + suffix is appended holding the values minus their within-block observed means. Centering on the block removes ALL block-level variation, including unobserved block-level confounding - it conditions on the block without introducing a many-level discrete variable - so searches run on the centered columns estimate within-block structure only; between-block relationships are removed by construction. This is the standard treatment for multilevel data where the blocks are nuisance (subjects, towns, batches). It is NOT a remedy for designed sweep data such as Airfoil, where the scientifically meaningful variables are block-constant: centering maps those columns to zero, which is why block-constant columns (within-block variance negligible relative to total variance) are skipped rather than appended.Discrete columns and the block column itself are skipped. Missing values (NaN) are left missing and are excluded from the block means. Rows in blocks with fewer than 2 observed values for a column get NaN in the centered column, since a within-block deviation from a single observation is identically zero by construction and carries no information.
- Parameters:
dataSet- the data set to modify in place.blockColumn- the name of the discrete column identifying blocks.suffix- the suffix for appended column names (must be nonempty).- Returns:
- the names of the appended columns, in order.
- Throws:
IllegalArgumentException- if the block column is missing or not discrete, the suffix is blank, an appended name would collide with an existing column, or no continuous column varies within blocks.
-
appendBlockIdColumn
Appends a new block ID column to the given dataset. The block ID column partitions the dataset's rows into blocks based on changes in the values of the specified key columns. Each block is represented by a unique categorical value.- Parameters:
dataSet- The dataset to which the block ID column will be appended.keyColumns- A list of column names that determine the block boundaries in the dataset.columnName- The name for the new block ID column to be added. Must not already exist in the dataset.- Returns:
- The number of unique blocks created in the dataset.
- Throws:
IllegalArgumentException- If no key columns are provided, the column name is empty, the dataset has no rows, or any of the specified key columns do not exist in the dataset.
-
restrictToMeasured
-
getResamplingDataset
-
getResamplingDataset
public static DataSet getResamplingDataset(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator) Get dataset sampled without replacement.- Parameters:
data- original datasetsampleSize- number of data (row)randomGenerator- random number generator- Returns:
- dataset
-
getBootstrapSample
-
getBootstrapSample
public static DataSet getBootstrapSample(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator) Get dataset sampled with replacement.- Parameters:
data- original datasetsampleSize- number of data (row)randomGenerator- random number generator- Returns:
- dataset
-
split
-
center
-
shuffleColumns
-
shuffleColumns2
-
covarianceGaussianCopula
Returns the Gaussian copula ("nonparanormal") correlation matrix of a continuous data set: for each pair of variables the sample Kendall's tau-b is computed and mapped to the latent Gaussian correlation byrho = sin(pi * tau / 2),
the standard inversion for a Gaussian copula (Liu, Han, Yuan, Lafferty and Wasserman 2012). Diagonal entries are set to exactly 1, so the result is a correlation matrix carrying the source data set's sample size.WHAT THIS BUYS. Under the nonparanormal model each variable is an unknown strictly monotone transformation of a latent Gaussian. Pearson correlation is not invariant to those transformations and is biased toward zero by them; rank correlation is invariant, so this estimator recovers the latent correlation structure that a linear-Gaussian score or likelihood is written against. As a worked case: for a latent pair with rho = 0.7, exponentiating one variable drops the Pearson correlation to about 0.51 while this estimator returns about 0.70.
WHAT IT DOES NOT BUY. The invariance is to monotone MARGINAL transformation only. Genuine nonlinearity in a conditional mean -- an interaction, a non-monotone dependence, X = f(Y, Z) for non-additive f -- is outside the model and this transform does nothing about it. It is a fix for distorted margins, not a nonlinear method.
CAVEATS, all reported as warnings to the log rather than by throwing:
- The elementwise map sin(pi * tau / 2) does not guarantee positive semidefiniteness. The smallest eigenvalue is checked and a warning logged if it is negative. Run a Covariance Audit on the result for the full diagnosis; likelihoods and partial correlations can behave incoherently on a non-PSD matrix.
- Discrete columns are included in the computation but the copula inversion is not correct for them (a polychoric estimator is the right tool there); a warning is logged if any are present.
- Missing values are handled by pairwise-complete deletion, which is itself a route to a non-PSD matrix and makes the single stated sample size approximate; a warning is logged if any are present.
- Parameters:
dataSet- a continuousDataSetobject- Returns:
- the Gaussian copula correlation matrix, as an
ICovarianceMatrix - Throws:
IllegalArgumentException- if the data set has fewer than two rows or no variables
-
kendallsTauB
public static double kendallsTauB(double[] x, double[] y) Kendall's tau-b for two columns, computed in O(n log n) by merge-sort inversion counting, with pairwise-complete deletion of rows where either value is NaN.With
totthe number of usable row pairs,xTiesandyTiesthe pairs tied in x and in y respectively,bothTiesthe pairs tied in both, anddisthe number of discordant pairs obtained as inversions in y after sorting by (x, y):tau_b = (tot - xTies - yTies + bothTies - 2 * dis) / sqrt((tot - xTies) * (tot - yTies)).
- Parameters:
x- the first column.y- the second column, the same length as the first.- Returns:
- tau-b, or NaN if it is undefined (fewer than two complete rows, or a column constant on them).
-
covarianceNonparanormalDrton
Deprecated.This does NOT compute the nonparanormal (Gaussian copula) correlation estimator, despite the name. It stores raw Kendall's tau, omitting the inversion sin(pi * tau / 2) that maps a rank correlation back to the latent Gaussian correlation. Raw tau is severely attenuated relative to the correlation it is standing in for -- a latent rho of 0.9 returns about 0.71, and 0.5 returns about 0.34 -- so anything scored from this matrix systematically under-detects edges. It also uses tau-a rather than tau-b, so ties attenuate it further, and it is O(n^2) per variable pair. UsecovarianceGaussianCopula(DataSet)instead, which corrects all three. Retained only because removing a public method is a breaking change; nothing in Tetrad calls it.Fills a matrix with raw Kendall's tau-a values.- Parameters:
dataSet- aDataSetobject- Returns:
- a
ICovarianceMatrixobject
-
getLinearizedTransformed
Returns a linearized ("un-warped") version of the dataset: each variable is passed through a monotone Yeo-Johnson transform chosen to make its relations to the other variables as linear as possible (post-nonlinear model un-warping). Unlike the nonparanormal transform, this preserves the marginal skewness that skew-based orientation methods rely on, to the extent the post-nonlinear model holds. Columns are standardized. Fitted lambdas and before/after nonlinearity per variable are written to the log; seeLinearizer.- Parameters:
dataSet- a continuousDataSet- Returns:
- the linearized dataset
-
getNonparanormalTransformed
Returns a nonparanormal-transformed version of the dataset. Each continuous column is rank-transformed and mapped through the normal quantile function, then rescaled to the original column mean and standard deviation. Discrete columns are left unchanged. -
removeConstantColumns
-
getConstantColumns
-
removeRandomColumns
-
standardizeData
-
standardizeData
Standardizes the columns of the given data matrix by centering and scaling. For each column representing a continuous variable, the method calculates the mean and standard deviation, subtracts the mean from each value, and divides by the standard deviation. Discrete variables are ignored.- Parameters:
data- The input data matrix to be standardized. Each column corresponds to a variable, and each row represents an observation.variables- A list of nodes representing the variables in the data. The type of each variable (e.g., continuous or discrete) determines whether the variable will be standardized.- Returns:
- A new standardized data matrix where each continuous variable has been mean-centered and normalized by its standard deviation.
-
standardizeData
public static double[] standardizeData(double[] data) standardizeData.
- Parameters:
data- an array of objects- Returns:
- an array of objects
-
center
public static double[] center(double[] d) Centers the values in the given array by subtracting the mean of the array from each element.- Parameters:
d- the array of double values to be centered- Returns:
- a new array where each element is the original value minus the mean of the input array
-
centerData
-
concatenate
-
getBootstrapSample
-
copyColumn
-
addMissingData
Adds missing data values to cases in accordance with probabilities specified in a double array which has as many elements as there are columns in the input dataset. Hence, if the first element of the array of probabilities is alpha, then the first column will contain a -99 (or other missing value code) in a given case with probability alpha. This method will be useful in generating datasets which can be used to test algorithm that handle missing data and/or latent variables. Author: Frank Wimberly- Parameters:
inData- The data to which random missing data is to be added.probs- The probability of adding missing data to each column.- Returns:
- The new data sets with missing data added.
-
replaceMissingWithRandom
-
scale
Scales the continuous variables in the given DataSet to have values in the range [-1, 1].For each continuous column, the method computes the maximum of the absolute values of the minimum and maximum of the column, and divides all values in that column by this maximum value. Discrete columns are not affected.
- Parameters:
dataSet- The DataSet containing variables to be scaled.scaleMin- The minimum value to scale to.scaleMax- The maximum value to scale to.- Returns:
- A new DataSet with scaled continuous variables, while discrete variables remain unchanged.
-
scale
Scales the values of a specified node in the given dataset to a specified range [scaleMin, scaleMax]. This method only processes nodes that are instances of ContinuousVariable.- Parameters:
dataSet- the dataset containing the values to be scaledscaleMin- the minimum value of the target rangescaleMax- the maximum value of the target rangenode- the node corresponding to the column in the dataset to be scaled
-
scale
public static double scale(double value, double dataMin, double dataMax, double scaleMin, double scaleMax) Scales a value from one range to another.- Parameters:
value- The value to scaledataMin- The minimum value of the data rangedataMax- The maximum value of the data rangescaleMin- The minimum value of the scale rangescaleMax- The maximum value of the scale range- Returns:
- The scaled value
- Throws:
IllegalArgumentException- if dataMin is equal to dataMax
-
scale
Scales the columns of the provided dataset based on the given scale factors. Only continuous variables in the dataset are scaled. Discrete variables are ignored. The method returns a new dataset with scaled values, leaving the original dataset unmodified.- Parameters:
dataSet- the input dataset to be scaledscales- an array of scale factors, where each scale corresponds to a column in the dataset- Returns:
- a new dataset with the continuous columns scaled by the given factors
-