T
 the type of the parameter set for the assessmentpublic abstract class ClassifierAssessment<T extends ClassifierAssessmentAssessParameterSet> extends Object
assess( ... )
should be
used as standard method to start a classifier assessment. Subclasses have to
implement the method evaluateClassifier( ... )
. This method
mainly has to execute the construction of test and training subsets of the
given data. These test and training subsets may be used by the methods
test( ... )
and train( ... )
which already are
implemented in a standard way.Modifier and Type  Field and Description 

protected AbstractClassifier[] 
myAbstractClassifier
This array contains the internal used classifiers.

protected TrainableStatisticalModel[][] 
myModel
This array contains for each class the internal used models.

protected MeanResultSet[] 
myTempMeanResultSets
The temporary result set.

protected int 
skipLastClassifiersDuringClassifierTraining
Skip last classifier.

Modifier  Constructor and Description 


ClassifierAssessment(AbstractClassifier... aCs)
Creates a new
ClassifierAssessment from a set of
AbstractClassifier s. 

ClassifierAssessment(AbstractClassifier[] aCs,
boolean buildClassifiersByCrossProduct,
TrainableStatisticalModel[]... aMs)
This constructor allows to assess a collection of given
AbstractClassifier s and, in addition, classifiers that will be
constructed using the given TrainableStatisticalModel s. 
protected 
ClassifierAssessment(AbstractClassifier[] aCs,
TrainableStatisticalModel[][] aMs,
boolean buildClassifiersByCrossProduct,
boolean checkAlphabetConsistencyAndLength)
Creates a new
ClassifierAssessment from an array of
AbstractClassifier s and a twodimensional array of TrainableStatisticalModel
s, which are combined to additional classifiers. 

ClassifierAssessment(boolean buildClassifiersByCrossProduct,
TrainableStatisticalModel[]... aMs)
Creates a new
ClassifierAssessment from a set of TrainableStatisticalModel s. 
Modifier and Type  Method and Description 

ListResult 
assess(NumericalPerformanceMeasureParameterSet mp,
T assessPS,
DataSet... s)
Assesses the contained classifiers.

ListResult 
assess(NumericalPerformanceMeasureParameterSet mp,
T assessPS,
ProgressUpdater pU,
DataSet[] s)
Assesses the contained classifiers.

ListResult 
assess(NumericalPerformanceMeasureParameterSet mp,
T assessPS,
ProgressUpdater pU,
DataSet[][]... s)
Assesses the contained classifiers.

ListResult 
assess(NumericalPerformanceMeasureParameterSet mp,
T assessPS,
ProgressUpdater pU,
DataSet[] s,
double[][] weights)
Assesses the contained classifiers.

protected abstract void 
evaluateClassifier(NumericalPerformanceMeasureParameterSet mp,
T assessPS,
DataSet[] s,
double[][] weights,
ProgressUpdater pU)
This method must be implemented in all subclasses.

abstract T 
getAssessParameterSet()
This method returns an instance of
ClassifierAssessmentAssessParameterSet that can be used in the assess methods. 
AbstractClassifier[] 
getClassifier()
Returns a deep copy of all classifiers that have been or will be used in
this assessment.

String 
getNameOfAssessment()
Returns the name of this class.

protected void 
prepareAssessment(boolean storeAll,
DataSet... s)
Prepares an assessment.

protected void 
test(NumericalPerformanceMeasureParameterSet mp,
boolean exception,
DataSet[] testS,
double[][] weights)
Uses the given test data sets to call the evaluate( ...

protected void 
train(DataSet[] trainS,
double[][] weights)
Trains the local classifiers using the given training data sets.

protected AbstractClassifier[] myAbstractClassifier
protected TrainableStatisticalModel[][] myModel
protected MeanResultSet[] myTempMeanResultSets
protected int skipLastClassifiersDuringClassifierTraining
protected ClassifierAssessment(AbstractClassifier[] aCs, TrainableStatisticalModel[][] aMs, boolean buildClassifiersByCrossProduct, boolean checkAlphabetConsistencyAndLength) throws IllegalArgumentException, WrongAlphabetException, CloneNotSupportedException, ClassDimensionException
ClassifierAssessment
from an array of
AbstractClassifier
s and a twodimensional array of TrainableStatisticalModel
s, which are combined to additional classifiers. If
buildClassifiersByCrossProduct
is true
, the
cross product of all TrainableStatisticalModel
s in aMs
is built to
obtain these classifiers.aCs
 the predefined classifiersaMs
 the TrainableStatisticalModel
s that are used to build additional
classifiersbuildClassifiersByCrossProduct
 Determines how classifiers are constructed using the given
models. Suppose a kclass problem. In this case, each
classifier is supposed to consist of k models, one responsible
for each class. S_i
be the set of all models in
aMs[i]
. Let S
be the set
S_1 x S_2 x ... x S_k
(crossproduct).true
: all possible classifiers consisting of a
subset (set of k models) of S
are constructed false
: one classifier consisting of the models
aMs[0][i]
,aMs[1][i]
,...,
aMs[k][i]
for a fixed i
is
constructed. In this case, all second dimensions of
aMs
have to be equal, say m
. In
total m
classifiers are constructed.checkAlphabetConsistencyAndLength
 indicates if alphabets and lengths shall be checked for
consistencyIllegalArgumentException
 if the classifiers have different lengthsWrongAlphabetException
 if not all given classifiers are defined on the same
AlphabetContainer
CloneNotSupportedException
 if something went wrong while cloningClassDimensionException
 if there is something wrong with the class dimension of the
classifierpublic ClassifierAssessment(AbstractClassifier... aCs) throws IllegalArgumentException, WrongAlphabetException, CloneNotSupportedException, ClassDimensionException
ClassifierAssessment
from a set of
AbstractClassifier
s.aCs
 contains the classifiers to be assessed.assess( ... )
.s
in order (s[0]
contains foreground data, s[1]
contains
background data)
IllegalArgumentException
 if the classifiers have different lengthsWrongAlphabetException
 if not all given classifiers are defined on the same
AlphabetContainer
CloneNotSupportedException
 if something went wrong while cloningClassDimensionException
 if there is something wrong with the class dimension of the
classifierClassifierAssessment(AbstractClassifier[],
TrainableStatisticalModel[][], boolean, boolean)
public ClassifierAssessment(boolean buildClassifiersByCrossProduct, TrainableStatisticalModel[]... aMs) throws IllegalArgumentException, WrongAlphabetException, CloneNotSupportedException, ClassDimensionException
ClassifierAssessment
from a set of TrainableStatisticalModel
s.
The argument buildClassifiersByCrossProduct
determines how
these TrainableStatisticalModel
s are combined to classifiers.buildClassifiersByCrossProduct
 S_i
be the set of all models in
aMs[i]
. Let S
be the set
S_1 x S_2 x ... x S_k
(crossproduct).true
: all possible classifiers consisting of a
subset (set of k models) of S
are constructed false
: one classifier consisting of the models
aMs[0][i]
,aMs[1][i]
,...,
aMs[k][i]
for a fixed i
is
constructed. In this case, all second dimensions of
aMs
have to be equal, say m
. In
total m
classifiers are constructed.aMs
 aMs[i]
) contains the
models according to class i
.s
... . s
in order (s[0]
contains foreground data, s[1]
contains
background data)
IllegalArgumentException
 if the classifiers have different lengthsWrongAlphabetException
 if not all given classifiers are defined on the same
AlphabetContainer
CloneNotSupportedException
 if something went wrong while cloningClassDimensionException
 if there is something wrong with the class dimension of the
classifierClassifierAssessment(AbstractClassifier[],
TrainableStatisticalModel[][], boolean, boolean)
public ClassifierAssessment(AbstractClassifier[] aCs, boolean buildClassifiersByCrossProduct, TrainableStatisticalModel[]... aMs) throws IllegalArgumentException, WrongAlphabetException, CloneNotSupportedException, ClassDimensionException
AbstractClassifier
s and, in addition, classifiers that will be
constructed using the given TrainableStatisticalModel
s.aCs
 contains some AbstractClassifier
s that should be
assessed in addition to the AbstractClassifier
s
constructed using the given TrainableStatisticalModel
sbuildClassifiersByCrossProduct
 Determines how classifiers are constructed using the given
models. Suppose a kclass problem. In this case, each
classifier is supposed to consist of k models, one responsible
for each class. S_i
be the set of all models in
aMs[i]
. Let S
be the set
S_1 x S_2 x ... x S_k
(crossproduct).true
: all possible classifiers consisting of a
subset (set of k models) of S
are constructed false
: one classifier consisting of the models
aMs[0][i]
,aMs[1][i]
,...,
aMs[k][i]
for a fixed i
is
constructed. In this case, all second dimensions of
aMs
have to be equal, say m
. In
total m
classifiers are constructed.aMs
 Contains the models in the following way (suppose a kclass
problem): the first dimension encodes the class (here it is
k), the second dimension (aMs[i]
) contains the
models according to class i
.s
... . s
in order (s[0]
contains foreground data, s[1]
contains
background data)
IllegalArgumentException
 if the classifiers have different lengthsWrongAlphabetException
 if not all given classifiers are defined on the same
AlphabetContainer
CloneNotSupportedException
 if something went wrong while cloningClassDimensionException
 if there is something wrong with the class dimension of the
classifierClassifierAssessment(AbstractClassifier[],
TrainableStatisticalModel[][], boolean, boolean)
public ListResult assess(NumericalPerformanceMeasureParameterSet mp, T assessPS, DataSet... s) throws IllegalArgumentException, WrongAlphabetException, Exception
s
 contains the data to be used for assessment. The order of the
data sets is important. s
. If
the models are trained directly, the order of given models
during initiation of this assessment object determines, which
data set will be used for training which model. In general the
first model will be trained using the first data set in
s
... . s
in order (s[0]
contains foreground data, s[1]
contains
background data)
mp
 defines which performance measure should be used to assess
classifiersassessPS
 contains some parameters necessary for assessment (depends on
the kind of assessment!)ListResult
that contains the results (mean and standard
errors) of user specified performance measures. These performance
measures are user specified via the given
NumericalPerformanceMeasureParameterSet
.IllegalArgumentException
 if the given assessPS
is not of the right type
(see method evaluateClassifier( ... )
)WrongAlphabetException
 if the given samples s
do not use the same
AlphabetContainer
as contained classifiers/modelsException
 forwarded from training/testing of classifiers/modelsassess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, DataSet[])
public ListResult assess(NumericalPerformanceMeasureParameterSet mp, T assessPS, ProgressUpdater pU, DataSet[] s) throws IllegalArgumentException, WrongAlphabetException, Exception
s
 contains the data to be used for assessment. The order of the
data sets is important. s
. If
the models are trained directly, the order of given models
during initiation of this assessment object determines, which
data set will be used for training which model. In general the
first model will be trained using the first data set in
s
... . s
in order (s[0]
contains foreground data, s[1]
contains
background data)
mp
 defines which performance measure should be used to assess
classifiersassessPS
 contains some parameters necessary for assessment (depends on
the kind of assessment!)pU
 this ProgressUpdater
may be used to cancel this method
assess()
by setting
pU.isCancelled()=true
. In that case,
assess()
will abort but return results already
computed.KFoldCrossValidation
.
In this case it might be wise to override this method such
that it just returns an error message. pU
is allowed to be null
although in
this case it may be more convenient to use the second method
code()
not requiring a ProgressUpdater
.ListResult
that contains the results (mean and standard
errors) of user specified performance measures. These performance
measures are user specified via the given
NumericalPerformanceMeasureParameterSet
.IllegalArgumentException
 if the given assessPS
is not of the right type
(see method evaluateClassifier( ... )
)WrongAlphabetException
 if the given data sets s
do not use the same
AlphabetContainer
as contained classifiers/modelsException
 forwarded from training/testing of classifiers/modelsassess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, ProgressUpdater, DataSet[], double[][])
public ListResult assess(NumericalPerformanceMeasureParameterSet mp, T assessPS, ProgressUpdater pU, DataSet[] s, double[][] weights) throws IllegalArgumentException, WrongAlphabetException, Exception
mp
 defines which performance measure should be used to assess
classifiersassessPS
 contains some parameters necessary for assessment (depends on
the kind of assessment!)pU
 this ProgressUpdater
may be used to cancel this method
assess()
by setting
pU.isCancelled()=true
. In that case,
assess()
will abort but return results already
computed.KFoldCrossValidation
.
In this case it might be wise to override this method such
that it just returns an error message. pU
is allowed to be null
although in
this case it may be more convenient to use the second method
code()
not requiring a ProgressUpdater
.s
 contains the data to be used for assessment. The order of the
data sets is important. s
. If
the models are trained directly, the order of given models
during initiation of this assessment object determines, which
data set will be used for training which model. In general the
first model will be trained using the first data set in
s
... . s
in order (s[0]
contains foreground data, s[1]
contains
background data)
weights
 the (nonnegative) weights for the data;
weight for each data set (first dimension) and each sequence (second dimension),
can be null
which is the same as weight 1 for all sequences in all data setsListResult
that contains the results (mean and standard
errors) of user specified performance measures. These performance
measures are user specified via the given
NumericalPerformanceMeasureParameterSet
.IllegalArgumentException
 if the given assessPS
is not of the right type
(see method evaluateClassifier( ... )
)WrongAlphabetException
 if the given data sets s
do not use the same
AlphabetContainer
as contained classifiers/modelsException
 forwarded from training/testing of classifiers/modelspublic ListResult assess(NumericalPerformanceMeasureParameterSet mp, T assessPS, ProgressUpdater pU, DataSet[][]... s) throws IllegalArgumentException, WrongAlphabetException, Exception
assess()
methods this one allows the user to predefine all
train and test data beforehand which should be used for assessment.s
 contains the data to be used for assessment.s
are important.s[iter][train/test][]
> the first dimension
codes for which data sets (train, test) are used in iteration
iter
. s[iter][0]
or test: s[iter][1]
.
s[iter][0]
contains for each class a training
data set. Analog s[iter][1]
contains the test
data sets. The order of the data sets is important. For further
details see comment of method
assess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, DataSet...)
.mp
 defines which performance measure should be used to assess
classifiersassessPS
 contains some parameters necessary for assessment. Must be of
type ClassifierAssessmentAssessParameterSet
pU
 this ProgressUpdater
allows to abort this classifier
assessment. If pU.isCancelled()=true
, all results
already computed will be returned. It is allowed to give a
null reference.ListResult
that contains the results (mean and standard
errors) of user specified performance measures. These performance
measures are user specified via the given
NumericalPerformanceMeasureParameterSet
.IllegalArgumentException
 if the given assessPS
is not of the right type
(see method evaluateClassifier( ... )
)WrongAlphabetException
 if the given data sets s
do not use the same
AlphabetContainer
as contained classifiers/modelsException
 forwarded from training/testing of classifiers/modelspublic AbstractClassifier[] getClassifier() throws CloneNotSupportedException
CloneNotSupportedException
 if it is impossible to get a deep copy for at least one
classifier (if the classifier could not be cloned)public String getNameOfAssessment()
protected abstract void evaluateClassifier(NumericalPerformanceMeasureParameterSet mp, T assessPS, DataSet[] s, double[][] weights, ProgressUpdater pU) throws IllegalArgumentException, Exception
train()
to train classifiers/models using
train data test()
to cause evaluation (test) of trained
classifiersmp
 defines which performance measures are used to assess
classifiersassessPS
 contains assessment specific parameters (like: number of
iterations of a kfoldcrossvalidation)s
 data to be used for assessment (both: test and train data)weights
 the (nonnegative) weights for the data;
weight for each data set (first dimension) and each sequence (second dimension),
can be null
which is the same as weight 1 for all sequences in all data setspU
 a ProgressUpdater
that mainly has to be used to allow
the user to cancel a current running classifier assessment.
This ProgressUpdater
is guaranteed to be not
null
. In certain cases aborting a classifier
assessment will not be allowed for example in case of
KFoldCrossValidation
. In this case the given
ProgressUpdater
should be ignored. pU.setMax()
= number of iterations of the assessment loop
pU.setValue()
=iteration+1;
train()
;
test()
;
pU.isCancelled()
))
IllegalArgumentException
 if the given ClassifierAssessmentAssessParameterSet
is of wrong typeException
 that occurred during training or using classifiers/modelsprotected void prepareAssessment(boolean storeAll, DataSet... s) throws IllegalArgumentException, WrongAlphabetException
DataSet
may not be used for
this assessment, this method throws an Exception
. MeanResultSet
s are initiated for this assessment (one for
each contained classifier).s
 the DataSet
to be checkedstoreAll
 a switch for storing all individual performance measure vales of each iteration
(cf. MeanResultSet.MeanResultSet(boolean, de.jstacs.results.SimpleResult...)
)WrongAlphabetException
 if s
is null
or not of required
length (number of classes)AlphabetContainer
s of s
are not
consistent with AlphabetContainer
of local models or
classifiersIllegalArgumentException
 if the given data sets are not suitableMeanResultSet.MeanResultSet(boolean, de.jstacs.results.SimpleResult...)
protected void test(NumericalPerformanceMeasureParameterSet mp, boolean exception, DataSet[] testS, double[][] weights) throws SimpleParameter.IllegalValueException, MeanResultSet.InconsistentResultNumberException, MeanResultSet.AdditionImpossibleException, Exception
evaluate( ... )
methods of the local AbstractClassifier
s. The returned
NumericalResult
s as well as the numerical
characteristics are added to each classifiers MeanResultSet
. mp
 determines which performance measures are used to assess the
classifiersexception
 whether an Exception
should be thrown if some
AbstractPerformanceMeasure
could not be evaluatedtestS
 data sets used as test sets (has to contain one DataSet
for each class)weights
 the (nonnegative) weights for the data;
weight for each data set (first dimension) and each sequence (second dimension),
can be null
which is the same as weight 1 for all sequences in all data setsSimpleParameter.IllegalValueException
 if a parameter is not validMeanResultSet.InconsistentResultNumberException
 if the number of results between the different result sets
differMeanResultSet.AdditionImpossibleException
 if added result sets do not matchException
 if necessaryIllegalArgumentException
 if the length of testS
is not equal to the
dimension of the classification problem (
testS.length!=this.myAbstractClassifier
[0].getNumberOfClasses()
)AbstractClassifier.evaluate(de.jstacs.classifiers.performanceMeasures.AbstractPerformanceMeasureParameterSet, boolean, DataSet...)
protected void train(DataSet[] trainS, double[][] weights) throws IllegalArgumentException, Exception
ClassifierAssessment
object was constructed using
TrainableStatisticalModel
s. trainS
 data sets used as training sets (has to contain one
DataSet
for each class)weights
 the (nonnegative) weights for the data;
weight for each data set (first dimension) and each sequence (second dimension),
can be null
which is the same as weight 1 for all sequences in all data setsIllegalArgumentException
 if the length of trainS
is not equal to the
dimension of the classification problem (
trainS.length!=this.myAbstractClassifier
[0].getNumberOfClasses()
)Exception
 if necessarypublic abstract T getAssessParameterSet() throws Exception
ClassifierAssessmentAssessParameterSet
that can be used in the assess
methods.ClassifierAssessmentAssessParameterSet
that can be used in the assess
methods.Exception
 if the parameter set could not be created properlyassess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, DataSet...)
,
assess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, ProgressUpdater, DataSet[])
,
assess(NumericalPerformanceMeasureParameterSet, ClassifierAssessmentAssessParameterSet, ProgressUpdater, DataSet[], double[][])