Class DataAudit
MissingDataAudit) - and emits the results as a list of AuditFindings keyed by FindingCode.
This class reports findings only; it makes no recommendations. The interpretation of findings is left to the user and to documentation that dispatches on the finding codes. Raw statistics computed along the way (correlations, Anderson-Darling p-values, R-squared and eta-squared values, distinct-value counts) are available from accessors so that downstream tools can display or reason over the numbers, not just the flags.
Separately from the findings, notes() returns cross-references to other diagnostics whose scope overlaps
a finding that fired, for cases where the finding on its own is ambiguous between explanations that a different
tool can distinguish. Notes are kept out of AuditFinding messages and out of the "findings" array of
toJson() precisely so that the findings-only contract above stays literally true: a note says what else
could be measured, never what the user should do about the data.
Columns found to be exactly constant (see FindingCode.CONSTANT_COLUMN) are flagged and then excluded from
all subsequent checks in the same audit: constant columns contribute nothing to small-cell, correlation,
near-determinism, non-Gaussianity, or serial-dependence diagnostics, and a constant continuous column would
otherwise poison the pairwise-complete correlation matrix (its correlations are undefined), masking findings such
as NEAR_DETERMINISM_LINEAR among the remaining variables. Each CONSTANT_COLUMN finding states this exclusion
in its message. Consequently getContinuousNames() and the correlation-based accessors cover only the
non-constant continuous variables, while whole-dataset summaries (distinct-value counts, the sample-size ratio,
and missingness statistics) continue to describe the dataset as given.
Missing values (NaN for continuous variables, the discrete missing-value marker for discrete variables, as judged
by MissingDataAudit.isMissing(DataSet, int, int)) are excluded pairwise or listwise as appropriate to each
statistic; each check's documentation notes which.
- Version:
- $Id: $Id
- Author:
- josephramsey
- See Also:
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic final classThresholds for the audit's checks, with defaults. -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final doubleFraction of the full R^2 that a greedily chosen predictor subset must reach for the subset to be reported as accounting for the dependence in NEAR_DETERMINISM_LINEAR messages.static final intMaximum number of predictors the greedy explaining-subset search will select.static final StringThe note attached bynotes()when at least oneFindingCode.NON_GAUSSIANfinding fired.static final StringCross-reference emitted when SENTINEL_VALUE fires: the check judges the shape of the observed distribution, and only the data's provenance settles whether the flagged value is a code or a measurement.static final StringThe note attached bynotes()when at least oneFindingCode.SERIAL_DEPENDENCEfinding fired and the check was computed pooled, without a serial grouping variable. -
Constructor Summary
ConstructorsConstructorDescriptionAudits the given dataset with default thresholds.DataAudit(DataSet dataSet, DataAudit.Config config) Audits the given dataset with the given thresholds. -
Method Summary
Modifier and TypeMethodDescriptionexplainingSubset(Matrix corr, int a, double fullR2, double[] achievedR2) Greedy forward selection on a correlation matrix, used to localize a NEAR_DETERMINISM_LINEAR finding: for column a, predictors are added one at a time, each step choosing the column that most increases the R^2 of a on the selected set, stopping when the subset R^2 reachesEXPLAINING_SUBSET_FRACTIONof fullR2 or the set reachesEXPLAINING_SUBSET_MAX_SIZEpredictors.Returns the Anderson-Darling p-values per continuous variable name, for those variables with enough non-missing values to test, unmodifiable.Returns the pairwise-complete Pearson correlation matrix of the continuous variables, in the order given bygetContinuousNames(), or null if there are fewer than two continuous variables.Returns the names of the continuous variables, in column order, excluding constant columns (see the class Javadoc), unmodifiable.Returns eta-squared values for (discrete, continuous) pairs, keyed "discreteName|continuousName", unmodifiable.Returns the findings, in the order the checks run, unmodifiable.getFindings(FindingCode code) Returns the findings with the given code, in order, unmodifiable.Returns the lag-1 autocorrelation in file order per continuous variable name, for those variables with enough observed values to compute it, unmodifiable.Returns the delegated missingness audit, or null if the dataset has no missing values.Returns the number of distinct observed (non-missing) values per column, by variable name, unmodifiable.Returns the multiple R-squared of each continuous variable regressed on the other continuous variables, by name, for those variables where this was computable, unmodifiable.Returns the Ljung-Box p-value per continuous variable name, testing the joint null that the firstserialMaxLagautocorrelations in file order are zero, for those variables with enough observed values to compute it, unmodifiable.booleanhasFinding(FindingCode code) Returns true if any finding has the given code.notes()Returns cross-references to other diagnostics bearing on the findings that fired, in a fixed order, or an empty list if none apply.static intrecodeToMissing(DataSet dataSet, String variable, double value) Sets every cell of the named continuous variable that holds the given value to missing (NaN), in place, and returns the number of cells changed.report()Returns a human-readable multi-section report of the audit.toJson()Returns the findings and summary statistics as a JSON string, suitable for consumption by py-tetrad or other tools.toString()Returns the report for this audit.
-
Field Details
-
NON_GAUSSIAN_NOTE
The note attached bynotes()when at least oneFindingCode.NON_GAUSSIANfinding fired.The Anderson-Darling check is a marginal one, and a non-Gaussian marginal has more than one explanation. The error term may itself be non-Gaussian, with the variable a linear additive function of its parents, which is the case the LiNGAM family exploits. Or the variable may be a nonlinear function of its parents, or a non-additive one, with Gaussian errors throughout: pushing Gaussian parents through a nonlinearity, or through an interaction, produces a non-Gaussian marginal by itself. Marginal non-Gaussianity does not separate these, and the second case violates the functional form that the first case's methods assume, so the cross-reference is to a check of the conditional mean and of additivity across parents rather than to another marginal test.
- See Also:
-
SERIAL_DEPENDENCE_NOTE
The note attached bynotes()when at least oneFindingCode.SERIAL_DEPENDENCEfinding fired and the check was computed pooled, without a serial grouping variable.File-order dependence has two distinct causes with the same signature. The rows may carry genuine serial structure - a time series or spatial sequence - in which case the dependence is a property of the process. Or the file may be sorted by block: rows grouped by subject, configuration, or condition, with within-block homogeneity masquerading as autocorrelation. Variables constant within blocks read as near-1 lag-1 autocorrelation in file order, and smooth within-block sweeps read as strong positive dependence, without any temporal process at all. The pooled check does not separate these; recomputing the check within groups (via the serial grouping variable) does - under the block reading the within-group autocorrelations collapse and block-constant variables are skipped, while under the serial reading the dependence survives grouping. The two readings point at different downstream treatments, so the distinction matters before acting on the finding.
- See Also:
-
SENTINEL_VALUE_NOTE
Cross-reference emitted when SENTINEL_VALUE fires: the check judges the shape of the observed distribution, and only the data's provenance settles whether the flagged value is a code or a measurement.- See Also:
-
EXPLAINING_SUBSET_FRACTION
public static final double EXPLAINING_SUBSET_FRACTIONFraction of the full R^2 that a greedily chosen predictor subset must reach for the subset to be reported as accounting for the dependence in NEAR_DETERMINISM_LINEAR messages.- See Also:
-
EXPLAINING_SUBSET_MAX_SIZE
public static final int EXPLAINING_SUBSET_MAX_SIZEMaximum number of predictors the greedy explaining-subset search will select.- See Also:
-
-
Constructor Details
-
DataAudit
Audits the given dataset with default thresholds.- Parameters:
dataSet- the dataset to audit; may not be null.
-
DataAudit
Audits the given dataset with the given thresholds.- Parameters:
dataSet- the dataset to audit; may not be null.config- the thresholds to use; may not be null.
-
-
Method Details
-
getFindings
Returns the findings, in the order the checks run, unmodifiable.- Returns:
- These findings.
-
getFindings
Returns the findings with the given code, in order, unmodifiable.- Parameters:
code- the code to filter by.- Returns:
- These findings.
-
hasFinding
Returns true if any finding has the given code.- Parameters:
code- the code to look for.- Returns:
- True if present.
-
getObservedDistinctCounts
-
getContinuousNames
-
getContinuousCorrelationMatrix
Returns the pairwise-complete Pearson correlation matrix of the continuous variables, in the order given bygetContinuousNames(), or null if there are fewer than two continuous variables.- Returns:
- This matrix or null.
-
getAdPValues
-
getR2OnOtherContinuous
-
getEtaSquaredValues
-
getLag1Autocorrelations
Returns the lag-1 autocorrelation in file order per continuous variable name, for those variables with enough observed values to compute it, unmodifiable. If a serial grouping variable is configured, autocorrelations are computed within its groups (per-group centering, cross-boundary pairs excluded) and pooled.- Returns:
- This map.
-
getSerialDependencePValues
Returns the Ljung-Box p-value per continuous variable name, testing the joint null that the firstserialMaxLagautocorrelations in file order are zero, for those variables with enough observed values to compute it, unmodifiable. The chi-square reference distribution is approximate when there are missing values or a serial grouping variable.- Returns:
- This map.
-
getMissingDataAudit
Returns the delegated missingness audit, or null if the dataset has no missing values.- Returns:
- This audit or null.
-
recodeToMissing
Sets every cell of the named continuous variable that holds the given value to missing (NaN), in place, and returns the number of cells changed. This is the operation a SENTINEL_VALUE finding describes: it converts a code the file wrote as an ordinary number into a missing entry, so that the missingness audit counts it, the pairwise-complete correlations skip it, and scores and tests apply whatever missing-data policy is in force instead of treating the code as a measurement.The dataset is modified in place, which is destructive and not undoable: callers that need to preserve the original should pass a copy. The comparison is exact equality against the stored double, which is what the audit's own distinct-value tally uses, so the value taken from a finding's "value" entry matches exactly the cells the finding counted. Cells already missing are left alone and are not counted in the return value.
Applying this to a variable the audit did not flag, or with a value the audit did not report, is permitted: the audit's detector is a heuristic and the codebook is authoritative, so a user who knows the code is entitled to recode a column the heuristic missed.
- Parameters:
dataSet- the dataset to modify; may not be null.variable- the name of a continuous variable in the dataset.value- the value to treat as a missing-data code.- Returns:
- the number of cells set to missing.
- Throws:
IllegalArgumentException- if there is no such variable, or if it is discrete.
-
notes
Returns cross-references to other diagnostics bearing on the findings that fired, in a fixed order, or an empty list if none apply. These are not recommendations about the data or the analysis; each note names a further measurement that would resolve an ambiguity the audit cannot resolve on its own, and the audit takes no position on whether the user should make it.- Returns:
- This list of notes.
- See Also:
-
report
Returns a human-readable multi-section report of the audit.- Returns:
- This report.
-
toJson
Returns the findings and summary statistics as a JSON string, suitable for consumption by py-tetrad or other tools. The schema has top-level fields "numRows", "numVariables", "numContinuous", "numDiscrete", "findings", and "notes". "findings" is an array of objects with fields "code", "severity", "variables", "values", and "message"; "notes" is an array of the strings returned bynotes(), kept separate from the findings so that a consumer reading "findings" alone sees findings only.- Returns:
- This JSON string.
-
toString
-
explainingSubset
public static List<Integer> explainingSubset(Matrix corr, int a, double fullR2, double[] achievedR2) Greedy forward selection on a correlation matrix, used to localize a NEAR_DETERMINISM_LINEAR finding: for column a, predictors are added one at a time, each step choosing the column that most increases the R^2 of a on the selected set, stopping when the subset R^2 reachesEXPLAINING_SUBSET_FRACTIONof fullR2 or the set reachesEXPLAINING_SUBSET_MAX_SIZEpredictors. The achieved subset R^2 is written to achievedR2[0], clamped to [0, 1].With near-collinear predictors the selected set is not unique: if a is nearly duplicated by each of two predictors, which one is selected first is an arbitrary consequence of tie-breaking. The result is one small explaining set, not an attribution of cause. On a pairwise-complete matrix that is not positive semidefinite, subset R^2 values describe the assembled matrix, not any single sample. Candidates whose submatrix cannot be inverted are skipped.
- Parameters:
corr- the correlation matrix.a- the column whose linear dependence on the others is being localized.fullR2- the R^2 of column a on all other columns (from the VIF diagonal).achievedR2- a length-1 array receiving the R^2 of a on the returned subset.- Returns:
- the selected column indices in order of selection; empty if no candidate increases R^2.
-