weka.experiment

## Class PairedTTester

- java.lang.Object
- - weka.experiment.PairedTTester

- All Implemented Interfaces:
    java.io.Serializable, OptionHandler, RevisionHandler, Tester
  Direct Known Subclasses:
    PairedCorrectedTTester

      public class PairedTTester
      extends java.lang.Object
      implements OptionHandler, Tester, RevisionHandler

  Calculates T-Test statistics on data stored in a set of instances.

  Valid options are:

       -D <index,index2-index4,...>
        Specify list of columns that specify a unique
        dataset.
        First and last are valid indexes. (default none)

       -R <index>
        Set the index of the column containing the run number

       -F <index>
        Set the index of the column containing the fold number

       -G <index1,index2-index4,...>
        Specify list of columns that specify a unique
        'result generator' (eg: classifier name and options).
        First and last are valid indexes. (default none)

       -S <significance level>
        Set the significance level for comparisons (default 0.05)

       -V
        Show standard deviations

       -L
        Produce table comparisons in Latex table format

       -csv
        Produce table comparisons in CSV table format

       -html
        Produce table comparisons in HTML table format

       -significance
        Produce table comparisons with only the significance values

       -gnuplot
        Produce table comparisons output suitable for GNUPlot

  Version:
    $Revision: 11542 $
  Author:
    Len Trigg (trigg@cs.waikato.ac.nz)
  See Also:
    Serialized Form

- - ### Constructor Summary

    Constructors
    Constructor and Description
    `PairedTTester()`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `assign(Tester tester)` retrieves all the settings from the given Tester
    `PairedStats` | `calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn)` Computes a paired t-test comparison for a specified dataset between two resultsets.
    `boolean` | `displayResultset(int index)` Checks whether the resultset with the given index shall be displayed.
    `Range` | `getDatasetKeyColumns()` Get the value of DatasetKeyColumns.
    `int[]` | `getDisplayedResultsets()` Gets the indices of the the datasets that are displayed (if `null` then all are displayed).
    `java.lang.String` | `getDisplayName()` returns the name of the tester
    `int` | `getFoldColumn()` Get the value of FoldColumn.
    `Instances` | `getInstances()` Get the value of Instances.
    `int` | `getNumDatasets()` Gets the number of datasets in the resultsets
    `int` | `getNumResultsets()` Gets the number of resultsets in the data.
    `java.lang.String[]` | `getOptions()` Gets current settings of the PairedTTester.
    `ResultMatrix` | `getResultMatrix()` Gets the instance that produces the output.
    `Range` | `getResultsetKeyColumns()` Get the value of ResultsetKeyColumns.
    `java.lang.String` | `getResultsetName(int index)` Gets a string descriptive of the specified resultset.
    `java.lang.String` | `getRevision()` Returns the revision string.
    `int` | `getRunColumn()` Get the value of RunColumn.
    `boolean` | `getShowStdDevs()` Returns true if standard deviations have been requested.
    `double` | `getSignificanceLevel()` Get the value of SignificanceLevel.
    `int` | `getSortColumn()` Returns the column to sort on, -1 means the default sorting.
    `java.lang.String` | `getSortColumnName()` Returns the name of the column to sort on.
    `java.lang.String` | `getToolTipText()` returns a string that is displayed as tooltip on the "perform test" button in the experimenter
    `java.lang.String` | `header(int comparisonColumn)` Creates a "header" string describing the current resultsets.
    `java.util.Enumeration<Option>` | `listOptions()` Lists options understood by this object.
    `static void` | `main(java.lang.String[] args)` Test the class from the command line.
    `java.lang.String` | `multiResultsetFull(int baseResultset, int comparisonColumn)` Creates a comparison table where a base resultset is compared to the other resultsets.
    `java.lang.String` | `multiResultsetRanking(int comparisonColumn)` returns a ranking of the resultsets
    `java.lang.String` | `multiResultsetSummary(int comparisonColumn)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
    `int[][]` | `multiResultsetWins(int comparisonColumn, int[][] nonSigWin)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
    `java.lang.String` | `resultsetKey()` Creates a key that maps resultset numbers to their descriptions.
    `void` | `setDatasetKeyColumns(Range newDatasetKeyColumns)` Set the value of DatasetKeyColumns.
    `void` | `setDisplayedResultsets(int[] cols)` Sets the indicies of the datasets to display (`null` means all).
    `void` | `setFoldColumn(int newFoldColumn)` Set the value of FoldColumn.
    `void` | `setInstances(Instances newInstances)` Set the value of Instances.
    `void` | `setOptions(java.lang.String[] options)` Parses a given list of options.
    `void` | `setResultMatrix(ResultMatrix matrix)` Sets the matrix to use to produce the output.
    `void` | `setResultsetKeyColumns(Range newResultsetKeyColumns)` Set the value of ResultsetKeyColumns.
    `void` | `setRunColumn(int newRunColumn)` Set the value of RunColumn.
    `void` | `setShowStdDevs(boolean s)` Set whether standard deviations are displayed or not.
    `void` | `setSignificanceLevel(double newSignificanceLevel)` Set the value of SignificanceLevel.
    `void` | `setSortColumn(int newSortColumn)` Set the column to sort on, -1 means the default sorting.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### PairedTTester

          public PairedTTester()

  - ### Method Detail

    - #### setResultMatrix

          public void setResultMatrix(ResultMatrix matrix)

      Sets the matrix to use to produce the output.
      Specified by:
        `setResultMatrix` in interface `Tester`
      Parameters:
        `matrix` - the instance to use to produce the output
      See Also:
        `ResultMatrix`

    - #### getResultMatrix

          public ResultMatrix getResultMatrix()

      Gets the instance that produces the output.
      Specified by:
        `getResultMatrix` in interface `Tester`
      Returns:
        the instance to produce the output

    - #### setShowStdDevs

          public void setShowStdDevs(boolean s)

      Set whether standard deviations are displayed or not.
      Specified by:
        `setShowStdDevs` in interface `Tester`
      Parameters:
        `s` - true if standard deviations are to be displayed

    - #### getShowStdDevs

          public boolean getShowStdDevs()

      Returns true if standard deviations have been requested.
      Specified by:
        `getShowStdDevs` in interface `Tester`
      Returns:
        true if standard deviations are to be displayed.

    - #### getNumDatasets

          public int getNumDatasets()

      Gets the number of datasets in the resultsets
      Specified by:
        `getNumDatasets` in interface `Tester`
      Returns:
        the number of datasets in the resultsets

    - #### getNumResultsets

          public int getNumResultsets()

      Gets the number of resultsets in the data.
      Specified by:
        `getNumResultsets` in interface `Tester`
      Returns:
        the number of resultsets in the data

    - #### getResultsetName

          public java.lang.String getResultsetName(int index)

      Gets a string descriptive of the specified resultset.
      Specified by:
        `getResultsetName` in interface `Tester`
      Parameters:
        `index` - the index of the resultset
      Returns:
        a descriptive string for the resultset

    - #### displayResultset

          public boolean displayResultset(int index)

      Checks whether the resultset with the given index shall be displayed.
      Specified by:
        `displayResultset` in interface `Tester`
      Parameters:
        `index` - the index of the resultset to check whether it shall be displayed
      Returns:
        whether the specified resultset is displayed

    - #### calculateStatistics

          public PairedStats calculateStatistics(Instance datasetSpecifier,
                                                 int resultset1Index,
                                                 int resultset2Index,
                                                 int comparisonColumn)
                                          throws java.lang.Exception

      Computes a paired t-test comparison for a specified dataset between two resultsets.
      Specified by:
        `calculateStatistics` in interface `Tester`
      Parameters:
        `datasetSpecifier` - the dataset specifier
        `resultset1Index` - the index of the first resultset
        `resultset2Index` - the index of the second resultset
        `comparisonColumn` - the column containing values to compare
      Returns:
        the results of the paired comparison
      Throws:
        `java.lang.Exception` - if an error occurs

    - #### resultsetKey

          public java.lang.String resultsetKey()

      Creates a key that maps resultset numbers to their descriptions.
      Specified by:
        `resultsetKey` in interface `Tester`
      Returns:
        a value of type 'String'

    - #### header

          public java.lang.String header(int comparisonColumn)

      Creates a "header" string describing the current resultsets.
      Specified by:
        `header` in interface `Tester`
      Parameters:
        `comparisonColumn` - a value of type 'int'
      Returns:
        a value of type 'String'

    - #### multiResultsetWins

          public int[][] multiResultsetWins(int comparisonColumn,
                                            int[][] nonSigWin)
                                     throws java.lang.Exception

      Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
      Specified by:
        `multiResultsetWins` in interface `Tester`
      Parameters:
        `comparisonColumn` - the index of the comparison column
        `nonSigWin` - for storing the non-significant wins
      Returns:
        a 2d array where element [i][j] is the number of times resultset j performed significantly better than resultset i.
      Throws:
        `java.lang.Exception` - if an error occurs

    - #### multiResultsetSummary

          public java.lang.String multiResultsetSummary(int comparisonColumn)
                                                 throws java.lang.Exception

      Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other. The results are summarized in a table.
      Specified by:
        `multiResultsetSummary` in interface `Tester`
      Parameters:
        `comparisonColumn` - the index of the comparison column
      Returns:
        the results in a string
      Throws:
        `java.lang.Exception` - if an error occurs

    - #### multiResultsetRanking

          public java.lang.String multiResultsetRanking(int comparisonColumn)
                                                 throws java.lang.Exception

      returns a ranking of the resultsets
      Specified by:
        `multiResultsetRanking` in interface `Tester`
      Parameters:
        `comparisonColumn` - the column to compare with
      Returns:
        the ranking
      Throws:
        `java.lang.Exception` - if something goes wrong

    - #### multiResultsetFull

          public java.lang.String multiResultsetFull(int baseResultset,
                                                     int comparisonColumn)
                                              throws java.lang.Exception

      Creates a comparison table where a base resultset is compared to the other resultsets. Results are presented for every dataset.
      Specified by:
        `multiResultsetFull` in interface `Tester`
      Parameters:
        `baseResultset` - the index of the base resultset
        `comparisonColumn` - the index of the column to compare over
      Returns:
        the comparison table string
      Throws:
        `java.lang.Exception` - if an error occurs

    - #### listOptions

          public java.util.Enumeration<Option> listOptions()

      Lists options understood by this object.
      Specified by:
        `listOptions` in interface `OptionHandler`
      Returns:
        an enumeration of Options.

    - #### setOptions

          public void setOptions(java.lang.String[] options)
                          throws java.lang.Exception

      Parses a given list of options.

      Valid options are:

           -D <index,index2-index4,...>
            Specify list of columns that specify a unique
            dataset.
            First and last are valid indexes. (default none)

           -R <index>
            Set the index of the column containing the run number

           -F <index>
            Set the index of the column containing the fold number

           -G <index1,index2-index4,...>
            Specify list of columns that specify a unique
            'result generator' (eg: classifier name and options).
            First and last are valid indexes. (default none)

           -S <significance level>
            Set the significance level for comparisons (default 0.05)

           -V
            Show standard deviations

           -L
            Produce table comparisons in Latex table format

           -csv
            Produce table comparisons in CSV table format

           -html
            Produce table comparisons in HTML table format

           -significance
            Produce table comparisons with only the significance values

           -gnuplot
            Produce table comparisons output suitable for GNUPlot

      Specified by:
        `setOptions` in interface `OptionHandler`
      Parameters:
        `options` - an array containing options to set.
      Throws:
        `java.lang.Exception` - if invalid options are given

    - #### getOptions

          public java.lang.String[] getOptions()

      Gets current settings of the PairedTTester.
      Specified by:
        `getOptions` in interface `OptionHandler`
      Returns:
        an array of strings containing current options.

    - #### getResultsetKeyColumns

          public Range getResultsetKeyColumns()

      Get the value of ResultsetKeyColumns.
      Specified by:
        `getResultsetKeyColumns` in interface `Tester`
      Returns:
        Value of ResultsetKeyColumns.

    - #### setResultsetKeyColumns

          public void setResultsetKeyColumns(Range newResultsetKeyColumns)

      Set the value of ResultsetKeyColumns.
      Specified by:
        `setResultsetKeyColumns` in interface `Tester`
      Parameters:
        `newResultsetKeyColumns` - Value to assign to ResultsetKeyColumns.

    - #### getDisplayedResultsets

          public int[] getDisplayedResultsets()

      Gets the indices of the the datasets that are displayed (if `null` then all are displayed). The base is always displayed.
      Specified by:
        `getDisplayedResultsets` in interface `Tester`
      Returns:
        the indices of the datasets to display

    - #### setDisplayedResultsets

          public void setDisplayedResultsets(int[] cols)

      Sets the indicies of the datasets to display (`null` means all). The base is always displayed.
      Specified by:
        `setDisplayedResultsets` in interface `Tester`
      Parameters:
        `cols` - the indices of the datasets to display

    - #### getSignificanceLevel

          public double getSignificanceLevel()

      Get the value of SignificanceLevel.
      Specified by:
        `getSignificanceLevel` in interface `Tester`
      Returns:
        Value of SignificanceLevel.

    - #### setSignificanceLevel

          public void setSignificanceLevel(double newSignificanceLevel)

      Set the value of SignificanceLevel.
      Specified by:
        `setSignificanceLevel` in interface `Tester`
      Parameters:
        `newSignificanceLevel` - Value to assign to SignificanceLevel.

    - #### getDatasetKeyColumns

          public Range getDatasetKeyColumns()

      Get the value of DatasetKeyColumns.
      Specified by:
        `getDatasetKeyColumns` in interface `Tester`
      Returns:
        Value of DatasetKeyColumns.

    - #### setDatasetKeyColumns

          public void setDatasetKeyColumns(Range newDatasetKeyColumns)

      Set the value of DatasetKeyColumns.
      Specified by:
        `setDatasetKeyColumns` in interface `Tester`
      Parameters:
        `newDatasetKeyColumns` - Value to assign to DatasetKeyColumns.

    - #### getRunColumn

          public int getRunColumn()

      Get the value of RunColumn.
      Specified by:
        `getRunColumn` in interface `Tester`
      Returns:
        Value of RunColumn.

    - #### setRunColumn

          public void setRunColumn(int newRunColumn)

      Set the value of RunColumn.
      Specified by:
        `setRunColumn` in interface `Tester`
      Parameters:
        `newRunColumn` - Value to assign to RunColumn.

    - #### getFoldColumn

          public int getFoldColumn()

      Get the value of FoldColumn.
      Specified by:
        `getFoldColumn` in interface `Tester`
      Returns:
        Value of FoldColumn.

    - #### setFoldColumn

          public void setFoldColumn(int newFoldColumn)

      Set the value of FoldColumn.
      Specified by:
        `setFoldColumn` in interface `Tester`
      Parameters:
        `newFoldColumn` - Value to assign to FoldColumn.

    - #### getSortColumnName

          public java.lang.String getSortColumnName()

      Returns the name of the column to sort on.
      Specified by:
        `getSortColumnName` in interface `Tester`
      Returns:
        the name of the column to sort on.

    - #### getSortColumn

          public int getSortColumn()

      Returns the column to sort on, -1 means the default sorting.
      Specified by:
        `getSortColumn` in interface `Tester`
      Returns:
        the column to sort on.

    - #### setSortColumn

          public void setSortColumn(int newSortColumn)

      Set the column to sort on, -1 means the default sorting.
      Specified by:
        `setSortColumn` in interface `Tester`
      Parameters:
        `newSortColumn` - the new sort column.

    - #### getInstances

          public Instances getInstances()

      Get the value of Instances.
      Specified by:
        `getInstances` in interface `Tester`
      Returns:
        Value of Instances.

    - #### setInstances

          public void setInstances(Instances newInstances)

      Set the value of Instances.
      Specified by:
        `setInstances` in interface `Tester`
      Parameters:
        `newInstances` - Value to assign to Instances.

    - #### assign

          public void assign(Tester tester)

      retrieves all the settings from the given Tester
      Specified by:
        `assign` in interface `Tester`
      Parameters:
        `tester` - the Tester to get the settings from

    - #### getToolTipText

          public java.lang.String getToolTipText()

      returns a string that is displayed as tooltip on the "perform test" button in the experimenter
      Specified by:
        `getToolTipText` in interface `Tester`
      Returns:
        the tool tip

    - #### getDisplayName

          public java.lang.String getDisplayName()

      returns the name of the tester
      Specified by:
        `getDisplayName` in interface `Tester`
      Returns:
        the display name

    - #### getRevision

          public java.lang.String getRevision()

      Returns the revision string.
      Specified by:
        `getRevision` in interface `RevisionHandler`
      Returns:
        the revision

    - #### main

          public static void main(java.lang.String[] args)

      Test the class from the command line.
      Parameters:
        `args` - contains options for the instance ttests