Class Linearizer

java.lang.Object
edu.cmu.tetrad.data.Linearizer
public final class Linearizer extends Object
Linearizing ("un-warping") transform: estimates, for each variable, a monotone Yeo-Johnson transform chosen to make that variable's relations to the other variables as LINEAR as possible, and returns the dataset with all transforms applied (columns standardized).

Contrast with the nonparanormal transform: the nonparanormal chooses each variable's monotone map to make its MARGINAL Gaussian, a per-variable decision that never looks at any other variable, and it destroys the marginal skewness that skew-based orientation methods (FASK) rely on. The linearizer chooses each map by looking at the scatterplots -- it minimizes the summed nonlinearity of the variable's pairwise relations -- and alters the marginal only incidentally.

Motivation: under a post-nonlinear model X_i = h_i(linear structural part + error), with h_i a monotone distortion (sensor saturation, for instance), each pairwise scatterplot looks only mildly curved, yet residual-based nonlinearity checks flag most pairs. Estimating the inverse warps and applying them restores the linear world: relationships straighten AND the structural skewness contributed by the errors survives, so both linear adjacency methods and skew-based orientation apply to the transformed data. If the warps are exactly of Yeo-Johnson form this recovers the linear model up to affine maps; general monotone warps are only approximated; and nonlinearity in the LINKS (rather than per-variable distortion) cannot be removed by any per-variable transform -- rerunning a nonlinearity check on the output is the diagnostic for which situation obtains, and the per-variable before/after nonlinearity measures are reported for exactly that purpose.

Method: nonlinearity of an ordered pair (u -> v) is measured as the R-squared gain of a cubic polynomial regression of v on u over the linear regression. Each variable's objective is the sum of the two directed nonlinearities over its neighbor set (variables with absolute Spearman correlation at least 0.15, which is invariant under monotone transforms and so computed once). The Yeo-Johnson parameters are fit by coordinate descent over variables, each step minimizing the variable's objective over a grid of lambda values, with the current transforms of the neighbors in place; two to four sweeps suffice. Lambda = 1 is the identity, so the method can only improve on doing nothing, up to grid resolution.

Author:
josephramsey
See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    Linearizer(DataSet dataSet)
    Constructs a linearizer for the given continuous dataset.

  • Method Summary

    Modifier and Type
    Method
    Description
    double[]
    getLambdas()
    Returns the fitted Yeo-Johnson lambdas, in variable order.
    double[]
    getNonlinearityAfter()
    Returns the post-transform nonlinearity measures, in variable order.
    double[]
    getNonlinearityBefore()
    Returns the pre-transform nonlinearity measures, in variable order.
    DataSet
    linearize()
    Runs the transform and returns the linearized dataset (same variables, columns standardized).

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • Linearizer

      public Linearizer(DataSet dataSet)
      Constructs a linearizer for the given continuous dataset.
      Parameters:
      dataSet - the dataset to linearize

  • Method Details

    • linearize

      public DataSet linearize()
      Runs the transform and returns the linearized dataset (same variables, columns standardized). Fitted lambdas and before/after nonlinearity per variable are available from the getters afterwards, and a summary is written to the log.
      Returns:
      the linearized dataset

    • getLambdas

      public double[] getLambdas()
      Returns the fitted Yeo-Johnson lambdas, in variable order.
      Returns:
      the fitted Yeo-Johnson lambda per variable (1.0 = identity).

    • getNonlinearityBefore

      public double[] getNonlinearityBefore()
      Returns the pre-transform nonlinearity measures, in variable order.
      Returns:
      the mean directed nonlinearity per variable before the transform.

    • getNonlinearityAfter

      public double[] getNonlinearityAfter()
      Returns the post-transform nonlinearity measures, in variable order.
      Returns:
      the mean directed nonlinearity per variable after the transform.