Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## de.jstacs.classifier.assessment
Class ClassifierAssessment

    java.lang.Object
      de.jstacs.classifier.assessment.ClassifierAssessment

Direct Known Subclasses:
  KFoldCrossValidation, RepeatedHoldOutExperiment, RepeatedSubSamplingExperiment, Sampled_RepeatedHoldOutExperiment

    public abstract class ClassifierAssessment

extends Object

Class defining an assessment of classifiers.
It should be used as a super-class for specialized classifier-assessments like k-fold-crossvalidation or subsampling-crossvalidation.
Several standard tasks like classifier- or model-management (testing, training) are implemented. The method `assess` should be used the standard-method to start a classifier-assessment. Subclasses have to implement the method `evaluateClassifier()`. This method mainly has to execute the construction of test- and training-subsets of the given data. These test- and training-subsets may be used by the methods `test()` and `train()` which already are implemented in a standard way.

Author:
  andr|e gohr (a0coder (nospam:@) gmail (nospam:.) com), Jens Keilwagen

Field Summary
`protected AbstractClassifier[]` | `myAbstractClassifier` This array contains the internal used classifiers.
`protected boolean` | `myBuildClassifierByCrossProduct`
`protected Model[][]` | `myModel` This array contains for each class the internal used models.
`protected MeanResultSet[]` | `myTempMeanResultSets`
`protected int` | `skipLastClassifiersDuringClassifierTraining`

Constructor Summary
`` | `ClassifierAssessment(AbstractClassifier... aCs)` Creates a new `ClassifierAssessment` from a set of `AbstractClassifier`s.
`` | `ClassifierAssessment(AbstractClassifier[] aCs, boolean buildClassifiersByCrossProduct, Model[]... aMs)` This constructor allows to assess a collection of given `AbstractClassifier`s and those constructed using the given `AbstractModel`s.
`protected` | `ClassifierAssessment(AbstractClassifier[] aCs, Model[][] aMs, boolean buildClassifiersByCrossProduct, boolean checkAlphabetConsistencyAndLength)` Creates a new `ClassifierAssessment` from an array of `AbstractClassifier`s and a two-dimensional array of `Model`s, which are combined to additional classifiers.
`` | `ClassifierAssessment(boolean buildClassifiersByCrossProduct, Model[]... aMs)` Creates a new `ClassifierAssessment` from a set of `Model`s.

Method Summary
`ListResult` | `assess(MeasureParameters mp, ClassifierAssessmentAssessParameterSet assessPS, ProgressUpdater pU, Sample... s)` Assesses the contained classifiers.
`ListResult` | `assess(MeasureParameters mp, ClassifierAssessmentAssessParameterSet assessPS, ProgressUpdater pU, Sample[][]... s)`
`ListResult` | `assess(MeasureParameters mp, ClassifierAssessmentAssessParameterSet assessPS, Sample... s)` Assesses the contained classifiers.
`protected abstract boolean` | `evaluateClassifier(MeasureParameters mp, ClassifierAssessmentAssessParameterSet assessPS, Sample[] s, ProgressUpdater pU)` This method must be implemented in all subclasses.
`AbstractClassifier[]` | `getClassifier()` Returns a deep copy of all classifiers that have been or will be used in this assessment.
`String` | `getNameOfAssessment()`
`protected void` | `prepareAssessment(Sample... s)` Prepares an assessment.
`protected void` | `test(MeasureParameters mp, boolean exception, Sample... testS)` Uses the given test-Samples to call the `evaluate`-methods of the local `AbstractClassifier`s.
`protected void` | `train(Sample... trainS)` Trains the local classifiers using the given trainingSamples.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### myAbstractClassifier

    protected AbstractClassifier[] myAbstractClassifier

  This array contains the internal used classifiers.

### myModel

    protected Model[][] myModel

  This array contains for each class the internal used models. This is helpful to allow to train some models only once while evaluating them in combination with other models.

### myTempMeanResultSets

    protected MeanResultSet[] myTempMeanResultSets

### skipLastClassifiersDuringClassifierTraining

    protected int skipLastClassifiersDuringClassifierTraining

### myBuildClassifierByCrossProduct

    protected boolean myBuildClassifierByCrossProduct

Constructor Detail

### ClassifierAssessment

    protected ClassifierAssessment(AbstractClassifier[] aCs,
                                   Model[][] aMs,
                                   boolean buildClassifiersByCrossProduct,
                                   boolean checkAlphabetConsistencyAndLength)
                            throws IllegalArgumentException,
                                   WrongAlphabetException,
                                   CloneNotSupportedException,
                                   ClassDimensionException

  Creates a new `ClassifierAssessment` from an array of `AbstractClassifier`s and a two-dimensional array of `Model`s, which are combined to additional classifiers. If `buildClassifiersByCrossProduct` is `true`, the cross product of all `Model`s in `aMs` is built to obtain these classifiers.

  Parameters:
    `aCs` - the pre-defined classifiers
    `aMs` - the `Model`s that are used to build additional classifiers
    `buildClassifiersByCrossProduct` - Determines how classifiers are constructed using the given models. Suppose a k-class problem. In this case, each classifier is supposed to consist of k models, one responsible for each class.
    Let S_i be the set of all models in aMs[i]. Let S be the set S_1 x S_2 x ... x S_k (cross-product).
    true: all possible classifiers consisting of a subset (set of k models) of S are constructed
    false: one classifier consisting of the models aMs[0][i],aMs[1][i],...,aMs[k][i] for a fixed i is constructed . In this case, all second dimensions of aMs have to be equal, say m. In total m classifiers are constructed.
    `checkAlphabetConsistencyAndLength` - indicates if alphabets and lengths shall be checked for consistency
  Throws:
    `IllegalArgumentException`
    `WrongAlphabetException`
    `CloneNotSupportedException`
    `ClassDimensionException`

### ClassifierAssessment

    public ClassifierAssessment(AbstractClassifier... aCs)
                         throws IllegalArgumentException,
                                WrongAlphabetException,
                                CloneNotSupportedException,
                                ClassDimensionException

  Creates a new `ClassifierAssessment` from a set of `AbstractClassifier`s.

  Parameters:
    `aCs` - contains the classifiers to be assessed
    If model-based classifiers are trained, the order of models in classifiers determines, which model will be trained using which sample in method assess().
    For a two class-problem, it is recommended to
    - initiate the classifiers with models in order (foreground-model (positive class), background-model (negative-class))
    - to initiate a assessment-object using models in order (foreground-model (positive class), background-model (negative-class))
    - to give data s in order (s[0] contains foreground-data, s[1] contains background data)
  Throws:
    `IllegalArgumentException`
    `WrongAlphabetException` - if not all given classifiers are defined on the same `AlphabetContainer`
    `ClassDimensionException`
    `CloneNotSupportedException`

### ClassifierAssessment

    public ClassifierAssessment(boolean buildClassifiersByCrossProduct,
                                Model[]... aMs)
                         throws IllegalArgumentException,
                                WrongAlphabetException,
                                CloneNotSupportedException,
                                ClassDimensionException

  Creates a new `ClassifierAssessment` from a set of `Model`s. The argument `buildClassifiersByCrossProduct` determines how these `Model`s are combined to classifiers.

  Parameters:
    `buildClassifiersByCrossProduct` -
    Determines how classifiers are constructed using the given models. Suppose a k-class problem. In this case, each classifier is supposed to consist of k models, one responsible for each class.
    Let S_i be the set of all models in aMs[i]. Let S be the set S_1 x S_2 x ... x S_k (cross-product).
    true: all possible classifiers consisting of a subset (set of k models) of S are constructed
    false: one classifier consisting of the models aMs[0][i],aMs[1][i],...,aMs[k][i] for a fixed i is constructed . In this case, all second dimensions of aMs have to be equal, say m. In total m classifiers are constructed.
    `aMs` -
    Contains the models in the following way (suppose a k-class problem): the first dimension encodes the class (here it is k), the second dimension (aMs[i]) contains the models according to class i.
    If models are trained directly (during assessment), the order of given models during initiation of this assessment-object determines, which sample will be used for training which model. In general the first model will be trained using the first sample in s... .
    For a two class-problem, it is recommended to
    - initiate the classifiers with models in order (foreground-model (positive class), background-model (negative-class))
    - to initiate a assessment-object using models in order (foreground-model (positive class), background-model (negative-class))
    - to give data s in order (s[0] contains foreground-data, s[1] contains background data)
  Throws:
    `WrongAlphabetException` - if not all given models are defines on the same `AlphabetContainer`
    `IllegalArgumentException`
    `CloneNotSupportedException`
    `ClassDimensionException`

### ClassifierAssessment

    public ClassifierAssessment(AbstractClassifier[] aCs,
                                boolean buildClassifiersByCrossProduct,
                                Model[]... aMs)
                         throws IllegalArgumentException,
                                WrongAlphabetException,
                                CloneNotSupportedException,
                                ClassDimensionException

  This constructor allows to assess a collection of given `AbstractClassifier`s and those constructed using the given `AbstractModel`s.

  Parameters:
    `aCs` - contains some `AbstractClassifier` that should be assessed in addition to the `AbstractClassifiers` constructed using the given `AbstractModels`
    `buildClassifiersByCrossProduct` -
    Determines how classifiers are constructed using the given models. Suppose a k-class problem. In this case, each classifier is supposed to consist of k models, one responsible for each class.
    Let S_i be the set of all models in aMs[i]. Let S be the set S_1 x S_2 x ... x S_k (cross-product).
    true: all possible classifiers consisting of a subset (set of k models) of S are constructed
    false: one classifier consisting of the models aMs[0][i],aMs[1][i],...,aMs[k][i] for a fixed i is constructed . In this case, all second dimensions of aMs have to be equal, say m. In total m classifiers are constructed.
    `aMs` -
    Contains the models in the following way (suppose a k-class problem): the first dimension encodes the class (here it is k), the second dimension (aMs[i]) contains the models according to class i.
    If models are trained directly (during assessment), the order of given models during initiation of this assessment-object determines, which sample will be used for training which model. In general the first model will be trained using the first sample in s... .
    For a two class-problem, it is recommended to
    - initiate the classifiers with models in order (foreground-model (positive class), background-model (negative-class))
    - to initiate a assessment-object using models in order (foreground-model (positive class), background-model (negative-class))
    - to give data s in order (s[0] contains foreground-data, s[1] contains background data)
  Throws:
    `WrongAlphabetException` - if not all given models are defines on the same `AlphabetContainer`
    `IllegalArgumentException`
    `CloneNotSupportedException`
    `ClassDimensionException`

Method Detail

### assess

    public ListResult assess(MeasureParameters mp,
                             ClassifierAssessmentAssessParameterSet assessPS,
                             Sample... s)
                      throws IllegalArgumentException,
                             WrongAlphabetException,
                             Exception

  Assesses the contained classifiers.

  Parameters:
    `s` - contains the data to be used for assessment. The order of samples is important.
    If model-based classifiers are trained, the order of models in classifiers determines, which model will be trained using which sample. The first model in classifier will be trained using the first sample in s. If models are trained directly, the order of given models during initiation of this assessment-object determines, which sample will be used for training which model. In general the first model will be trained using the first sample in s... .
    For a two class-problem, it is recommended to
    - initiate the classifiers with models in order (foreground-model (positive class), background-model (negative-class))
    - to initiate a assessment-object using models in order (foreground-model (positive class), background-model (negative-class))
    - to give data s in order (s[0] contains foreground-data, s[1] contains background data)
    `mp` - defines which performance-measure should be used to assess classifiers
    `assessPS` - contains some parameters necessary for assessment (depends on the kind of assessment!)
  Returns:
    ListResult contains the results (mean and standard-errors) of user-specified performance-measures. These performance-measures are user specified via the given `MeasureParameters`.
  Throws:
    `IllegalArgumentException` - if given assessPS is not of right type (see method evaluateClassifier())
    `WrongAlphabetException` - if given Samples s do not use the same `AlphabetContainer` as contained classifiers/models
    `Exception` - forwarded from training/testing of classifiers/models

### assess

    public ListResult assess(MeasureParameters mp,
                             ClassifierAssessmentAssessParameterSet assessPS,
                             ProgressUpdater pU,
                             Sample... s)
                      throws IllegalArgumentException,
                             WrongAlphabetException,
                             Exception

  Assesses the contained classifiers.

  Parameters:
    `s` - contains the data to be used for assessment. The order of samples is important.
    If model-based classifiers are trained, the order of models in classifiers determines, which model will be trained using which sample. The first model in classifier will be trained using the first sample in s. If models are trained directly, the order of given models during initiation of this assessment-object determines, which sample will be used for training which model. In general the first model will be trained using the first sample in s... .
    For a two class-problem, it is recommended to
    - initiate the classifiers with models in order (foreground-model (positive class), background-model (negative-class))
    - to initiate a assessment-object using models in order (foreground-model (positive class), background-model (negative-class))
    - to give data s in order (s[0] contains foreground-data, s[1] contains background data)
    `mp` - defines which performance-measure should be used to assess classifiers
    `assessPS` - contains some parameters necessary for assessment (depends on the kind of assessment!)
    `pU` - this `ProgressUpdater` may be used to cancel this method `assess()` by setting `pU.isCandeled=true`. In that case, `assess` will abort but return results already computed.
    In certain cases aborting a classifier assessment will not be allowed for example in case of `KFoldCrossValidation`. In this case it might be wise to override this method such that it just returns an error-message.
    pU is allowed to be `null` although in this case it may be more convenient to use the second method `code` not requiring a `ProgressUpdater`.
  Returns:
    ListResult contains the results (mean and standard-errors) of user-specified performance-measures. These performance-measures are user specified via the given `MeasureParameters`.
  Throws:
    `IllegalArgumentException` - if given assessPS is not of right type (see method evaluateClassifier())
    `WrongAlphabetException` - if given Samples s do not use the same `AlphabetContainer` as contained classifiers/models
    `Exception` - forwarded from training/testing of classifiers/models

### assess

    public ListResult assess(MeasureParameters mp,
                             ClassifierAssessmentAssessParameterSet assessPS,
                             ProgressUpdater pU,
                             Sample[][]... s)
                      throws IllegalArgumentException,
                             WrongAlphabetException,
                             Exception

  Parameters:
    `s` - Contains the data to be used for assessment.
    The order of samples in s are important. s[iter][train/test][] -> the first dimension codes for which samples (train, test) are used in iteration iter.
    The second dimension codes for training: s[iter][0] or test: s[iter][1]. s[iter][0] contains for each class a training-sample. Analog s[iter][1] contains the test-samples. The order of samples is important. For further details see comment of method `assess(MeasureParameters, AssessParameterSet, Sample)`.
    The user is responsible to take care or not to take care of the given test- and training-dataset to be not overlapping.
    `mp` - defines which performance-measure should be used to assess classifiers
    `assessPS` - Contains some parameters necessary for assessment. Must be of type `ClassifierAssessmentAssessParameterSet`
    `pU` - This `ProgressUpdater` allows to abort this classifier assessment. If `pU.isCalceled()`is true, all results already computed will be returned. It is allowed to give a null-reference.
  Returns:
    ListResult contains the results (mean and standard-errors) of user-specified performance-measures. These performance-measures are user specified via the given `MeasureParameters`.
  Throws:
    `IllegalArgumentException` - if given assessPS is not of right type (see method evaluateClassifier())
    `WrongAlphabetException` - if given Samples s do not use the same `AlphabetContainer` as contained classifiers/models
    `Exception` - forwarded from training/testing of classifiers/models

### getClassifier

    public AbstractClassifier[] getClassifier()
                                       throws CloneNotSupportedException

  Returns a deep copy of all classifiers that have been or will be used in this assessment.

  Returns:
    a deep copy of all used classifiers
  Throws:
    `CloneNotSupportedException` - if it is imppossible to get an deep copy for at least one classifier

### getNameOfAssessment

    public String getNameOfAssessment()

  Returns:
    name of this class

### evaluateClassifier

    protected abstract boolean evaluateClassifier(MeasureParameters mp,
                                                  ClassifierAssessmentAssessParameterSet assessPS,
                                                  Sample[] s,
                                                  ProgressUpdater pU)
                                           throws IllegalArgumentException,
                                                  Exception

  This method must be implemented in all subclasses. It should perform the following tasks:
  1.) create test- and train-datasets 2.) call method train to train classifiers/models using train-data 3.) call method test to cause evaluation (test) of trained classieres

  Parameters:
    `mp` - defines which performance-measures are used to assess classifiers
    `assessPS` - containes assessment-specific parameters (like: number of iterations of a k-fold-crossvalidation)
    `s` - data to be used for assessment (both: test- and train-data)
    `pU` - a `ProgressUpdater` that mainly has to be used to allow the user to cancel a current running alssifier assessment. This `ProgressUpdater` is guaranteed to be not `null`. In certain cases aborting a classifier assessment will not be allowed for example in case of `KFoldCrossValidation`. In this case the given `ProgressUpdater` should be ignored.
    Usage:
    - `pU.setMax()`= number of iterations of the `assessment-loop`
    - iteration=0;
    - `assessment-loop`
      - `pU.setValue()`=iteration+1;
      - `Sample` treatment
      - `train()`;
      - `test()`;
      - iteration++;
    - `repeat unless(ready or not(pU.isCanceled()))`
  Returns:
    true, if no errors occured
  Throws:
    `IllegalArgumentException` - if the given `AssessParameterSet` is of wrong type
    `Exception` - that occured during training or using classifiers/models

### prepareAssessment

    protected void prepareAssessment(Sample... s)
                              throws WrongAlphabetException

  Prepares an assessment. If the given `Sample` may not be used for this assessment, this method throws an exception.
  Further `MeanResultSet`s are initiated for this assessment (one for each contained classifier).

  Parameters:
    `s` - the `Sample` to be checked
  Throws:
    `WrongAlphabetException` - if
    1. s is null or not of required length (number of classes)
    2. `AlphabetContainer` of s are not consistent with `AlphabetContainer` of local models or classifiers

### test

    protected void test(MeasureParameters mp,
                        boolean exception,
                        Sample... testS)
                 throws SimpleParameter.IllegalValueException,
                        MeanResultSet.InconsistentResultNumberException,
                        MeanResultSet.AdditionImpossibleException,
                        Exception

  Uses the given test-Samples to call the `evaluate`-methods of the local `AbstractClassifier`s. The returned `NumericalResults` as well as the numerical characteristics are added to each classifiers `MeanResultSet`.
  It should not be necessary to override this method in subclasses.

  Parameters:
    `mp` - determines which performance-measures are used to assess the classifiers
    `exception` - whether an `Exception` should be thrown if some `MeasureParameters.Measure` could not be evaluated
    `testS` - samples used as test-sets (has to contain one `Sample` for each class)
  Throws:
    `IllegalArgumentException` - if the length of testS is not equal to the dimension of the classification-problem (testS.length!=this.myAbstractClassifier[0].getNumberOfClasses())
    `SimpleParameter.IllegalValueException`
    `MeanResultSet.InconsistentResultNumberException`
    `MeanResultSet.AdditionImpossibleException`
    `Exception`
  See Also:
    `AbstractClassifier.evaluate(MeasureParameters, boolean, Sample...)`

### train

    protected void train(Sample... trainS)
                  throws IllegalArgumentException,
                         Exception

  Trains the local classifiers using the given trainingSamples.
  The classifiers are either directly trained or via training of the local models. The second option always is used, if the `ClassifierAssessment`-object was constructed using `AbstractModel`s.
  It should not be necessary to override this method in subclasses.

  Parameters:
    `trainS` - samples used as train-sets (has to contain one `Sample` for each class)
  Throws:
    `IllegalArgumentException` - if the length of trainS is not equal to the dimension of the classification-problem (trainS.length!=this.myAbstractClassifier[0].getNumberOfClasses())
    `Exception`

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD