Class ParameterSweep

java.lang.Object
edu.cmu.tetrad.algcomparison.sweep.ParameterSweep
public final class ParameterSweep extends Object
A harness for evaluating a search algorithm over a grid of parameter settings and recording, per setting, the evidence a parameter choice should rest on: the point-estimate graph on the full data, a probability-annotated aggregate of searches on shared resamples (edge probabilities displaying identically to bootstrap results elsewhere in Tetrad), the StARS-style adjacency instability across those resamples, and Markov-check statistics for the point graph if a Markov-check test is configured. Results are returned as a SweepReport, which serializes to markdown and JSON and exposes explicit, overridable selection rules.

The resamples are drawn once per sweep and shared across all settings, so that instability comparisons between settings are paired (the convention of Tetrad's StARS implementation). Resampling is seed-controlled and the resample searches are race-free: each resample writes its graph to its own slot. Note the determinism contract: under a fixed seed the resampled row sets are exactly reproducible, but end-to-end reproducibility of the report additionally requires the wrapped algorithm to be deterministic; algorithms with internal thread pools (e.g. FGES) can break score near-ties differently between runs, producing small run-to-run wobble in instabilities.

Block resampling (added 2026-8-25). Row resampling treats every row as exchangeable, which is wrong when rows are grouped into units that are exchangeable with each other but dependent within - subjects with repeated measurements, time series segments, sites. setResampleGroups(int[]) supplies one group label per row; resamples are then drawn as whole groups (all rows of a drawn group, in their original order; a group drawn twice with replacement contributes its rows twice), with percentResampleSize and withReplacement applied to the number of groups rather than rows. Instabilities and edge probabilities are computed exactly as before, over the resulting resampled datasets. The Markov check and point graph are unaffected. Under a fixed seed the drawn group sets are reproducible.

The harness executes and measures; it does not decide. Choosing among settings - even by the provided rules on SweepReport - is a user decision. Background knowledge should be set on the algorithm wrapper before sweeping, as usual for algcomparison algorithms.

See Zheng, Verma, Gill, Dai, Spirtes, and Zhang (2026), "Causal discovery in the era of agents" (arXiv:2606.23608), for the assistance-versus-evidence principle this harness is designed to support: the sweep makes the evidence behind a parameter recommendation cheap to produce and inspect, wherever that recommendation comes from.

Version:
$Id: $Id
Author:
josephramsey
See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    ParameterSweep(Algorithm algorithm, Parameters baseParameters)
    Constructs a sweep harness for the given algorithm and base parameters.

  • Method Summary

    Modifier and Type
    Method
    Description
    static double
    adjacencyInstability(List<Graph> graphs, List<Node> variables)
    Computes the StARS-style adjacency instability of a set of graphs over the given variables: the mean over unordered variable pairs of 2 * theta * (1 - theta), where theta is the fraction of graphs in which the pair is adjacent.
    static List<DataSet>
    drawBlockResamples(DataSet data, int[] groups, int numResamples, double percent, boolean withReplacement, long seed)
    Draws seed-controlled block resamples: each resample is the concatenation of the rows of percent * G groups drawn from the G distinct groups, with or without replacement, where a group is the set of rows sharing a label in groups.
    static List<DataSet>
    drawResamples(DataSet data, int numResamples, double percent, boolean withReplacement, long seed)
    Draws seed-controlled resamples of the given dataset: row-index samples of size percent * n, with or without replacement.
    int[]
    getResampleGroups()
    Returns the row groups for block resampling, or null for row resampling.
    static List<Graph>
    searchOnResamples(Algorithm algorithm, Parameters parameters, List<DataSet> resamples, List<Node> variables, boolean parallelized)
    Runs the algorithm at the given parameters on each resample and returns the graphs, with nodes replaced by the given variable list so graphs are comparable.
    void
    setConditioningSetType(ConditioningSetType conditioningSetType)
    Sets the conditioning set type for the Markov check.
    void
    setMarkovCheckTest(IndependenceWrapper markovCheckTest)
    Sets the test used to Markov-check each setting's point graph, or null to skip Markov checking.
    void
    setNumResamples(int numResamples)
    Sets the number of resamples drawn per sweep and shared across settings.
    void
    setParallelized(boolean parallelized)
    Sets whether resample searches and the Markov check run in parallel.
    void
    setPercentResampleSize(double percentResampleSize)
    Sets the fraction of the sample size drawn per resample, in (0, 1].
    void
    setResampleGroups(int[] groups)
    Sets the row groups for block resampling, or null (the default) for row resampling.
    void
    setSeed(long seed)
    Sets the random seed for drawing resamples, for reproducibility, or -1 for time-based.
    void
    setVerbose(boolean verbose)
    Sets whether to log progress to the Tetrad log.
    void
    setWithReplacement(boolean withReplacement)
    Sets whether resampling is with replacement.
    SweepReport
    sweep(DataSet data, String parameter, List<?> values)
    Sweeps a single parameter over the given values.
    SweepReport
    sweep(DataSet data, String parameter1, List<?> values1, String parameter2, List<?> values2)
    Sweeps the cross product of two parameters over the given value lists, varying the second parameter fastest.
    SweepReport
    sweepSettings(DataSet data, List<Map<String,Object>> settings)
    Sweeps an explicit list of settings, each a map from parameter name to value.

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ParameterSweep

      public ParameterSweep(Algorithm algorithm, Parameters baseParameters)
      Constructs a sweep harness for the given algorithm and base parameters. The base parameters are copied per setting and never mutated.
      Parameters:
      algorithm - the algorithm to sweep; may not be null. Set knowledge on this wrapper beforehand if desired.
      baseParameters - the base parameters; may not be null.

  • Method Details

    • setMarkovCheckTest

      public void setMarkovCheckTest(IndependenceWrapper markovCheckTest)
      Sets the test used to Markov-check each setting's point graph, or null to skip Markov checking. The test is instantiated per setting from the data and that setting's parameters, so test parameters (e.g. alpha) may themselves be swept.
      Parameters:
      markovCheckTest - the test wrapper, or null.

    • setConditioningSetType

      public void setConditioningSetType(ConditioningSetType conditioningSetType)
      Sets the conditioning set type for the Markov check. Default ORDERED_LOCAL_MARKOV_PROPERTY.
      Parameters:
      conditioningSetType - the type; may not be null.

    • setNumResamples

      public void setNumResamples(int numResamples)
      Sets the number of resamples drawn per sweep and shared across settings. Zero disables resampling (no probability graphs, instability NaN). Default 50.
      Parameters:
      numResamples - the number of resamples; may not be negative.

    • setPercentResampleSize

      public void setPercentResampleSize(double percentResampleSize)
      Sets the fraction of the sample size drawn per resample, in (0, 1]. Default 1.0. For classical StARS-style subsampling, use around 0.5 with setWithReplacement(boolean) false; for bootstrap-style edge probabilities, use 1.0 with replacement.
      Parameters:
      percentResampleSize - the fraction; must be in (0, 1].

    • setWithReplacement

      public void setWithReplacement(boolean withReplacement)
      Sets whether resampling is with replacement. Default true.
      Parameters:
      withReplacement - true for with replacement.

    • setSeed

      public void setSeed(long seed)
      Sets the random seed for drawing resamples, for reproducibility, or -1 for time-based. Default -1.
      Parameters:
      seed - the seed or -1.

    • setResampleGroups

      public void setResampleGroups(int[] groups)
      Sets the row groups for block resampling, or null (the default) for row resampling. The array must have one entry per row of the dataset passed to sweep; any integer labels may be used, and rows with equal labels form a group. Resamples are drawn as whole groups; see the class Javadoc. Validated against the dataset at sweep time.
      Parameters:
      groups - one label per row, or null for row resampling.

    • getResampleGroups

      public int[] getResampleGroups()
      Returns the row groups for block resampling, or null for row resampling.
      Returns:
      a copy of the group labels, or null.

    • setParallelized

      public void setParallelized(boolean parallelized)
      Sets whether resample searches and the Markov check run in parallel. Default true.
      Parameters:
      parallelized - true for parallel.

    • setVerbose

      public void setVerbose(boolean verbose)
      Sets whether to log progress to the Tetrad log. Default false.
      Parameters:
      verbose - true to log.

    • sweep

      public SweepReport sweep(DataSet data, String parameter, List<?> values) throws InterruptedException
      Sweeps a single parameter over the given values.
      Parameters:
      data - the dataset; may not be null.
      parameter - the parameter name (a Params constant); may not be null.
      values - the values to evaluate, in order; may not be null or empty. Elements are typically Double, Integer, or Boolean.
      Returns:
      the report.
      Throws:
      InterruptedException - if a search is interrupted.

    • sweep

      public SweepReport sweep(DataSet data, String parameter1, List<?> values1, String parameter2, List<?> values2) throws InterruptedException
      Sweeps the cross product of two parameters over the given value lists, varying the second parameter fastest.
      Parameters:
      data - the dataset; may not be null.
      parameter1 - the first parameter name; may not be null.
      values1 - the first parameter's values; may not be null or empty.
      parameter2 - the second parameter name; may not be null.
      values2 - the second parameter's values; may not be null or empty.
      Returns:
      the report.
      Throws:
      InterruptedException - if a search is interrupted.

    • sweepSettings

      public SweepReport sweepSettings(DataSet data, List<Map<String,Object>> settings) throws InterruptedException
      Sweeps an explicit list of settings, each a map from parameter name to value. This is the general entry point; the one- and two-parameter overloads delegate here.
      Parameters:
      data - the dataset; may not be null.
      settings - the settings to evaluate, in order; may not be null or empty.
      Returns:
      the report.
      Throws:
      InterruptedException - if a search is interrupted.

    • drawResamples

      public static List<DataSet> drawResamples(DataSet data, int numResamples, double percent, boolean withReplacement, long seed)
      Draws seed-controlled resamples of the given dataset: row-index samples of size percent * n, with or without replacement. Drawing resamples once and reusing them across settings makes between-setting comparisons paired. This method is also used by the StARS and StabilitySelection algorithm wrappers.
      Parameters:
      data - the dataset.
      numResamples - the number of resamples; zero returns an empty list.
      percent - the fraction of the sample size per resample, in (0, 1].
      withReplacement - whether to draw with replacement.
      seed - the random seed, or -1 for time-based.
      Returns:
      the resampled datasets, in draw order.

    • drawBlockResamples

      public static List<DataSet> drawBlockResamples(DataSet data, int[] groups, int numResamples, double percent, boolean withReplacement, long seed)
      Draws seed-controlled block resamples: each resample is the concatenation of the rows of percent * G groups drawn from the G distinct groups, with or without replacement, where a group is the set of rows sharing a label in groups. Rows of a drawn group appear in their original order; with replacement, a group drawn k times contributes its rows k times. Groups are indexed in order of first appearance in groups, so the draws are reproducible under a fixed seed independent of label values. Resample sizes vary with the sizes of the drawn groups.
      Parameters:
      data - the dataset; may not be null.
      groups - one integer label per row; must have data.getNumRows() entries.
      numResamples - the number of resamples; zero returns an empty list.
      percent - the fraction of the groups per resample, in (0, 1].
      withReplacement - whether to draw groups with replacement.
      seed - the random seed, or -1 for time-based.
      Returns:
      the resampled datasets, in draw order.

    • searchOnResamples

      public static List<Graph> searchOnResamples(Algorithm algorithm, Parameters parameters, List<DataSet> resamples, List<Node> variables, boolean parallelized) throws InterruptedException
      Runs the algorithm at the given parameters on each resample and returns the graphs, with nodes replaced by the given variable list so graphs are comparable. Race-free: each resample writes to its own slot. This method is also used by the StARS and StabilitySelection algorithm wrappers.
      Parameters:
      algorithm - the algorithm.
      parameters - the parameters (not mutated).
      resamples - the resampled datasets.
      variables - the variables of the original dataset, for node replacement.
      parallelized - whether to run the resample searches in parallel.
      Returns:
      the graphs, in resample order.
      Throws:
      InterruptedException - if a search is interrupted.

    • adjacencyInstability

      public static double adjacencyInstability(List<Graph> graphs, List<Node> variables)
      Computes the StARS-style adjacency instability of a set of graphs over the given variables: the mean over unordered variable pairs of 2 * theta * (1 - theta), where theta is the fraction of graphs in which the pair is adjacent. In [0, 0.5]; 0 means every pair's adjacency is unanimous across the graphs.
      Parameters:
      graphs - the graphs (nodes must be from the given variable list).
      variables - the variables.
      Returns:
      the instability, or NaN if there are no graphs or fewer than two variables.