Class DataTransforms

java.lang.Object
edu.cmu.tetrad.data.DataTransforms

public class DataTransforms extends Object

DataTransforms class.

Version:
$Id: $Id
Author:
josephramsey
  • Method Details

    • logData

      public static DataSet logData(DataSet dataSet, double a, boolean isUnlog, int base)
      Log or unlog data
      Parameters:
      dataSet - a DataSet object
      a - a double
      isUnlog - a boolean
      base - a int
      Returns:
      a DataSet object
    • logData

      public static DataSet logData(DataSet dataSet, Map<String, LogTransformSpec> specs)
      Applies a logarithmic transform to selected variables of the given dataset, each with its own offset, base, and direction.

      Variables absent from the map are copied through unchanged, as are discrete variables (which are never transformed even if named in the map). Passing a map that assigns the same spec to every continuous variable reproduces logData(DataSet, double, boolean, int) exactly.

      No check is made that the offsets keep values in the domain of the logarithm; a variable whose values are not strictly positive under its offset will yield NaN or negative infinity, exactly as the dataset-wide transform does. LogTransformSpec.safeOffsetFor(double[]) is available to callers that want to propose a safe offset to the user first.

      Parameters:
      dataSet - The dataset to transform.
      specs - A map from variable name to the transform to apply to that variable.
      Returns:
      The transformed dataset.
    • alignVariableOrder

      public static List<DataSet> alignVariableOrder(List<DataSet> dataSets, List<String> names)
      Puts the variables of each of the given data sets in the given order of names. A data set that already has them in that order is returned as it is; one that has exactly those variables in another order is replaced by a reordered copy with the same name; one that does not have exactly those variables is returned as it is, for the caller to refuse. The given list and its data sets are not changed.
      Parameters:
      dataSets - the data sets.
      names - the variable names, in the order wanted.
      Returns:
      the data sets, in a new list.
    • standardizeData

      public static List<DataSet> standardizeData(List<DataSet> dataSets)

      standardizeData.

      Parameters:
      dataSets - a List object
      Returns:
      a List object
    • standardizeData

      public static DataSet standardizeData(DataSet dataSet)

      standardizeData.

      Parameters:
      dataSet - a DataSet object
      Returns:
      a DataSet object
    • center

      public static List<DataSet> center(List<DataSet> dataList)

      center.

      Parameters:
      dataList - a List object
      Returns:
      a List object
    • discretize

      public static DataSet discretize(DataSet dataSet, int numCategories, boolean variablesCopied)

      discretize.

      Parameters:
      dataSet - a DataSet object
      numCategories - a int
      variablesCopied - a boolean
      Returns:
      a DataSet object
    • convertNumericalDiscreteToContinuous

      public static DataSet convertNumericalDiscreteToContinuous(DataSet dataSet) throws NumberFormatException

      convertNumericalDiscreteToContinuous.

      Parameters:
      dataSet - a DataSet object
      Returns:
      a DataSet object
      Throws:
      NumberFormatException - if any.
    • concatenate

      public static DataSet concatenate(DataSet dataSet1, DataSet dataSet2)

      concatenate.

      Parameters:
      dataSet1 - a DataSet object
      dataSet2 - a DataSet object
      Returns:
      a DataSet object
    • concatenate

      public static DataSet concatenate(DataSet... dataSets)

      concatenate.

      Parameters:
      dataSets - a DataSet object
      Returns:
      a DataSet object
    • concatenate

      public static DataSet concatenate(List<DataSet> dataSets)

      concatenate.

      Parameters:
      dataSets - a List object
      Returns:
      a DataSet object
    • appendWithinBlockCenteredColumns

      public static List<String> appendWithinBlockCenteredColumns(DataSet dataSet, String blockColumn, String suffix)
      Appends within-block centered ("fixed effects" / within-transformed) copies of the data set's continuous columns, in place: for each continuous column that varies within blocks, a new column named name + suffix is appended holding the values minus their within-block observed means. Centering on the block removes ALL block-level variation, including unobserved block-level confounding - it conditions on the block without introducing a many-level discrete variable - so searches run on the centered columns estimate within-block structure only; between-block relationships are removed by construction. This is the standard treatment for multilevel data where the blocks are nuisance (subjects, towns, batches). It is NOT a remedy for designed sweep data such as Airfoil, where the scientifically meaningful variables are block-constant: centering maps those columns to zero, which is why block-constant columns (within-block variance negligible relative to total variance) are skipped rather than appended.

      Discrete columns and the block column itself are skipped. Missing values (NaN) are left missing and are excluded from the block means. Rows in blocks with fewer than 2 observed values for a column get NaN in the centered column, since a within-block deviation from a single observation is identically zero by construction and carries no information.

      Parameters:
      dataSet - the data set to modify in place.
      blockColumn - the name of the discrete column identifying blocks.
      suffix - the suffix for appended column names (must be nonempty).
      Returns:
      the names of the appended columns, in order.
      Throws:
      IllegalArgumentException - if the block column is missing or not discrete, the suffix is blank, an appended name would collide with an existing column, or no continuous column varies within blocks.
    • appendBlockIdColumn

      public static int appendBlockIdColumn(DataSet dataSet, List<String> keyColumns, String columnName)
      Appends a new block ID column to the given dataset. The block ID column partitions the dataset's rows into blocks based on changes in the values of the specified key columns. Each block is represented by a unique categorical value.
      Parameters:
      dataSet - The dataset to which the block ID column will be appended.
      keyColumns - A list of column names that determine the block boundaries in the dataset.
      columnName - The name for the new block ID column to be added. Must not already exist in the dataset.
      Returns:
      The number of unique blocks created in the dataset.
      Throws:
      IllegalArgumentException - If no key columns are provided, the column name is empty, the dataset has no rows, or any of the specified key columns do not exist in the dataset.
    • restrictToMeasured

      public static DataSet restrictToMeasured(DataSet fullDataSet)

      restrictToMeasured.

      Parameters:
      fullDataSet - a DataSet object
      Returns:
      a DataSet object
    • getResamplingDataset

      public static DataSet getResamplingDataset(DataSet data, int sampleSize)

      getResamplingDataset.

      Parameters:
      data - a DataSet object
      sampleSize - a int
      Returns:
      a sample without replacement with the given sample size from the given dataset.
    • getResamplingDataset

      public static DataSet getResamplingDataset(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator)
      Get dataset sampled without replacement.
      Parameters:
      data - original dataset
      sampleSize - number of data (row)
      randomGenerator - random number generator
      Returns:
      dataset
    • getBootstrapSample

      public static DataSet getBootstrapSample(DataSet data, int sampleSize)

      getBootstrapSample.

      Parameters:
      data - a DataSet object
      sampleSize - a int
      Returns:
      a sample with replacement with the given sample size from the given dataset.
    • getBootstrapSample

      public static DataSet getBootstrapSample(DataSet data, int sampleSize, org.apache.commons.math3.random.RandomGenerator randomGenerator)
      Get dataset sampled with replacement.
      Parameters:
      data - original dataset
      sampleSize - number of data (row)
      randomGenerator - random number generator
      Returns:
      dataset
    • split

      public static List<DataSet> split(DataSet data, double percentTest)

      split.

      Parameters:
      data - a DataSet object
      percentTest - a double
      Returns:
      a List object
    • center

      public static DataSet center(DataSet data)
      Subtracts the mean of each column from each datum that column.
      Parameters:
      data - a DataSet object
      Returns:
      a DataSet object
    • shuffleColumns

      public static DataSet shuffleColumns(DataSet dataModel)

      shuffleColumns.

      Parameters:
      dataModel - a DataSet object
      Returns:
      a DataSet object
    • shuffleColumns2

      public static List<DataSet> shuffleColumns2(List<DataSet> dataSets)

      shuffleColumns2.

      Parameters:
      dataSets - a List object
      Returns:
      a List object
    • covarianceGaussianCopula

      public static ICovarianceMatrix covarianceGaussianCopula(DataSet dataSet)
      Returns the Gaussian copula ("nonparanormal") correlation matrix of a continuous data set: for each pair of variables the sample Kendall's tau-b is computed and mapped to the latent Gaussian correlation by
          rho = sin(pi * tau / 2),
      the standard inversion for a Gaussian copula (Liu, Han, Yuan, Lafferty and Wasserman 2012). Diagonal entries are set to exactly 1, so the result is a correlation matrix carrying the source data set's sample size.

      WHAT THIS BUYS. Under the nonparanormal model each variable is an unknown strictly monotone transformation of a latent Gaussian. Pearson correlation is not invariant to those transformations and is biased toward zero by them; rank correlation is invariant, so this estimator recovers the latent correlation structure that a linear-Gaussian score or likelihood is written against. As a worked case: for a latent pair with rho = 0.7, exponentiating one variable drops the Pearson correlation to about 0.51 while this estimator returns about 0.70.

      WHAT IT DOES NOT BUY. The invariance is to monotone MARGINAL transformation only. Genuine nonlinearity in a conditional mean -- an interaction, a non-monotone dependence, X = f(Y, Z) for non-additive f -- is outside the model and this transform does nothing about it. It is a fix for distorted margins, not a nonlinear method.

      CAVEATS, all reported as warnings to the log rather than by throwing:

      • The elementwise map sin(pi * tau / 2) does not guarantee positive semidefiniteness. The smallest eigenvalue is checked and a warning logged if it is negative. Run a Covariance Audit on the result for the full diagnosis; likelihoods and partial correlations can behave incoherently on a non-PSD matrix.
      • Discrete columns are included in the computation but the copula inversion is not correct for them (a polychoric estimator is the right tool there); a warning is logged if any are present.
      • Missing values are handled by pairwise-complete deletion, which is itself a route to a non-PSD matrix and makes the single stated sample size approximate; a warning is logged if any are present.
      Kendall's tau-b is computed in O(n log n) per pair by merge-sort inversion counting, not the O(n^2) pair enumeration.
      Parameters:
      dataSet - a continuous DataSet object
      Returns:
      the Gaussian copula correlation matrix, as an ICovarianceMatrix
      Throws:
      IllegalArgumentException - if the data set has fewer than two rows or no variables
    • kendallsTauB

      public static double kendallsTauB(double[] x, double[] y)
      Kendall's tau-b for two columns, computed in O(n log n) by merge-sort inversion counting, with pairwise-complete deletion of rows where either value is NaN.

      With tot the number of usable row pairs, xTies and yTies the pairs tied in x and in y respectively, bothTies the pairs tied in both, and dis the number of discordant pairs obtained as inversions in y after sorting by (x, y):

          tau_b = (tot - xTies - yTies + bothTies - 2 * dis) / sqrt((tot - xTies) * (tot - yTies)).
      Parameters:
      x - the first column.
      y - the second column, the same length as the first.
      Returns:
      tau-b, or NaN if it is undefined (fewer than two complete rows, or a column constant on them).
    • covarianceNonparanormalDrton

      @Deprecated public static ICovarianceMatrix covarianceNonparanormalDrton(DataSet dataSet)
      Deprecated.
      This does NOT compute the nonparanormal (Gaussian copula) correlation estimator, despite the name. It stores raw Kendall's tau, omitting the inversion sin(pi * tau / 2) that maps a rank correlation back to the latent Gaussian correlation. Raw tau is severely attenuated relative to the correlation it is standing in for -- a latent rho of 0.9 returns about 0.71, and 0.5 returns about 0.34 -- so anything scored from this matrix systematically under-detects edges. It also uses tau-a rather than tau-b, so ties attenuate it further, and it is O(n^2) per variable pair. Use covarianceGaussianCopula(DataSet) instead, which corrects all three. Retained only because removing a public method is a breaking change; nothing in Tetrad calls it.
      Fills a matrix with raw Kendall's tau-a values.
      Parameters:
      dataSet - a DataSet object
      Returns:
      a ICovarianceMatrix object
    • getLinearizedTransformed

      public static DataSet getLinearizedTransformed(DataSet dataSet)
      Returns a linearized ("un-warped") version of the dataset: each variable is passed through a monotone Yeo-Johnson transform chosen to make its relations to the other variables as linear as possible (post-nonlinear model un-warping). Unlike the nonparanormal transform, this preserves the marginal skewness that skew-based orientation methods rely on, to the extent the post-nonlinear model holds. Columns are standardized. Fitted lambdas and before/after nonlinearity per variable are written to the log; see Linearizer.
      Parameters:
      dataSet - a continuous DataSet
      Returns:
      the linearized dataset
    • getNonparanormalTransformed

      public static DataSet getNonparanormalTransformed(DataSet dataSet)
      Returns a nonparanormal-transformed version of the dataset. Each continuous column is rank-transformed and mapped through the normal quantile function, then rescaled to the original column mean and standard deviation. Discrete columns are left unchanged.
      Parameters:
      dataSet - a DataSet object
      Returns:
      a DataSet object
    • removeConstantColumns

      public static DataSet removeConstantColumns(DataSet dataSet)

      removeConstantColumns.

      Parameters:
      dataSet - a DataSet object
      Returns:
      a DataSet object
    • getConstantColumns

      public static List<Node> getConstantColumns(DataSet dataSet)

      getConstantColumns.

      Parameters:
      dataSet - a DataSet object
      Returns:
      a List object
    • removeRandomColumns

      public static DataSet removeRandomColumns(DataSet dataSet, double aDouble)

      removeRandomColumns.

      Parameters:
      dataSet - a DataSet object
      aDouble - a double
      Returns:
      a DataSet object
    • standardizeData

      public static Matrix standardizeData(Matrix data)

      standardizeData.

      Parameters:
      data - a Matrix object
      Returns:
      a Matrix object
    • standardizeData

      public static Matrix standardizeData(Matrix data, List<Node> variables)
      Standardizes the columns of the given data matrix by centering and scaling. For each column representing a continuous variable, the method calculates the mean and standard deviation, subtracts the mean from each value, and divides by the standard deviation. Discrete variables are ignored.
      Parameters:
      data - The input data matrix to be standardized. Each column corresponds to a variable, and each row represents an observation.
      variables - A list of nodes representing the variables in the data. The type of each variable (e.g., continuous or discrete) determines whether the variable will be standardized.
      Returns:
      A new standardized data matrix where each continuous variable has been mean-centered and normalized by its standard deviation.
    • standardizeData

      public static double[] standardizeData(double[] data)

      standardizeData.

      Parameters:
      data - an array of objects
      Returns:
      an array of objects
    • center

      public static double[] center(double[] d)
      Centers the values in the given array by subtracting the mean of the array from each element.
      Parameters:
      d - the array of double values to be centered
      Returns:
      a new array where each element is the original value minus the mean of the input array
    • centerData

      public static Matrix centerData(Matrix data)

      centerData.

      Parameters:
      data - a Matrix object
      Returns:
      a Matrix object
    • concatenate

      public static Matrix concatenate(Matrix... dataSets)

      concatenate.

      Parameters:
      dataSets - a Matrix object
      Returns:
      a Matrix object
    • getBootstrapSample

      public static Matrix getBootstrapSample(Matrix data, int sampleSize)

      getBootstrapSample.

      Parameters:
      data - a Matrix object
      sampleSize - a int
      Returns:
      a sample with replacement with the given sample size from the given dataset.
    • copyColumn

      public static void copyColumn(Node node, DataSet source, DataSet dest)

      copyColumn.

      Parameters:
      node - a Node object
      source - a DataSet object
      dest - a DataSet object
    • addMissingData

      public static DataSet addMissingData(DataSet inData, double[] probs)
      Adds missing data values to cases in accordance with probabilities specified in a double array which has as many elements as there are columns in the input dataset. Hence, if the first element of the array of probabilities is alpha, then the first column will contain a -99 (or other missing value code) in a given case with probability alpha. This method will be useful in generating datasets which can be used to test algorithm that handle missing data and/or latent variables. Author: Frank Wimberly
      Parameters:
      inData - The data to which random missing data is to be added.
      probs - The probability of adding missing data to each column.
      Returns:
      The new data sets with missing data added.
    • replaceMissingWithRandom

      public static DataSet replaceMissingWithRandom(DataSet inData)

      replaceMissingWithRandom.

      Parameters:
      inData - a DataSet object
      Returns:
      a DataSet object
    • scale

      public static DataSet scale(DataSet dataSet, double scaleMin, double scaleMax)
      Scales the continuous variables in the given DataSet to have values in the range [-1, 1].

      For each continuous column, the method computes the maximum of the absolute values of the minimum and maximum of the column, and divides all values in that column by this maximum value. Discrete columns are not affected.

      Parameters:
      dataSet - The DataSet containing variables to be scaled.
      scaleMin - The minimum value to scale to.
      scaleMax - The maximum value to scale to.
      Returns:
      A new DataSet with scaled continuous variables, while discrete variables remain unchanged.
    • scale

      public static void scale(DataSet dataSet, double scaleMin, double scaleMax, Node node)
      Scales the values of a specified node in the given dataset to a specified range [scaleMin, scaleMax]. This method only processes nodes that are instances of ContinuousVariable.
      Parameters:
      dataSet - the dataset containing the values to be scaled
      scaleMin - the minimum value of the target range
      scaleMax - the maximum value of the target range
      node - the node corresponding to the column in the dataset to be scaled
    • scale

      public static double scale(double value, double dataMin, double dataMax, double scaleMin, double scaleMax)
      Scales a value from one range to another.
      Parameters:
      value - The value to scale
      dataMin - The minimum value of the data range
      dataMax - The maximum value of the data range
      scaleMin - The minimum value of the scale range
      scaleMax - The maximum value of the scale range
      Returns:
      The scaled value
      Throws:
      IllegalArgumentException - if dataMin is equal to dataMax
    • scale

      public static DataSet scale(DataSet dataSet, double[] scales)
      Scales the columns of the provided dataset based on the given scale factors. Only continuous variables in the dataset are scaled. Discrete variables are ignored. The method returns a new dataset with scaled values, leaving the original dataset unmodified.
      Parameters:
      dataSet - the input dataset to be scaled
      scales - an array of scale factors, where each scale corresponds to a column in the dataset
      Returns:
      a new dataset with the continuous columns scaled by the given factors