Metrics

Metrics and measures to evaluate the performance of anomaly detection and training.

Calculations and wrappers of various metrics.

class ecgan.evaluation.metrics.classification.ClassificationMetric[source]

Bases: abc.ABC

Classification metric base class.

abstract calculate(y, y_hat)[source]

Calculate the metric based on ground truth labels y and predicted labels y_hat.

Labels can be either integer or boolean arrays but have to be numpy arrays for the flattening.

Return type: float

class ecgan.evaluation.metrics.classification.FScoreMetric(beta=1, average='weighted')[source]

Bases: ecgan.evaluation.metrics.classification.ClassificationMetric

Create F-score objects.

calculate(y, y_hat)[source]

Calculate the f-beta score (beta defaults to 1) and track it if desired.

If the f-score is chosen as a metric, the precision and recall will also be saved but not returned to the user.

Parameters

y (Union[Tensor, ndarray]) -- Ground truth labels.
y_hat (Union[Tensor, ndarray]) -- Predicted labels.

Return type

float

Returns

The resulting F_beta score. Usually weighted to account for imbalanced classes,: returns average score if the classwise F-score was calculated.

class ecgan.evaluation.metrics.classification.MCCMetric[source]

Bases: ecgan.evaluation.metrics.classification.ClassificationMetric

Create a MCC object.

calculate(y, y_hat)[source]

Calculate the mcc score and track if desired.

Parameters

y (Union[Tensor, ndarray]) -- Ground truth labels of shape (num_samples,).
y_hat (Union[Tensor, ndarray]) -- Predicted labels of shape (num_samples,).

Returns

The average mcc value.

Return type

float

class ecgan.evaluation.metrics.classification.AUROCMetric(average='weighted')[source]

Bases: ecgan.evaluation.metrics.classification.ClassificationMetric

Create an AUROC object.

calculate(y, y_hat)[source]

Calculate the AUROC score and track if desired.

If only one class label is present, AUROC is ill-defined. To avoid exceptions, we set the auroc to 0 if only one class is present. Since this can distort results you should make sure that the batches are large enough.

Parameters

y (Union[ndarray, Tensor]) -- Ground truth labels of shape (num_samples,).
y_hat (Union[Tensor, ndarray]) -- Predicted labels of shape (num_samples,).

Return type

float

Returns

AUROC score.

class ecgan.evaluation.metrics.classification.AvgPrecisionMetric[source]

Bases: ecgan.evaluation.metrics.classification.ClassificationMetric

Create a AP object.

calculate(y, y_hat)[source]

Calculate the average precision score.

Parameters

y (Union[Tensor, ndarray]) -- Ground truth labels of shape (num_samples,).
y_hat (Union[Tensor, ndarray]) -- Predicted labels of shape (num_samples,).

Returns

The average precision score.

Return type

float

class ecgan.evaluation.metrics.classification.ClassificationMetricFactory[source]

Bases: object

Meta module for creating classification metric objects.

Implementation of the maximum mean discrepancy to measure distributional differences.

class ecgan.evaluation.metrics.mmd.MaxMeanDiscrepancy(sigma=None)[source]

Bases: object

Maximum Mean Discrepancy handler. Implementation is inspired by https://github.com/SeldonIO/alibi-detect.

infer_sigma(x, y)[source]

Infer heuristic sigma value for gaussian kernel.

Infer sigma used in the kernel by setting it to the median distance between each of the pairwise instances in x and y.

Parameters

x (Tensor) -- Tensor of shape (num_samples x Features).
y (Tensor) -- Tensor of shape (num_samples x Features).

Return type

float

Returns

Determined sigma value.

gaussian_kernel(x, y, sigma=None)[source]

Calculate the Gaussian kernel function between two tensors.

If sigma is not set, the kernel will infer the sigma value via median pairwise distance.

Parameters

x (Tensor) -- Tensor of shape (num_samples x Features).
y (Tensor) -- Tensor of shape (num_samples x Features).
sigma (Optional[float]) -- Sigma for RBF bandwidth. Is usually set automatically by the object but can also be set manually.

Return type

Tensor

Returns

Kernel matrix K(X,Y) with shape [NX, NY]

Implementation of TSTR from Esteban et al. 2017.

class ecgan.evaluation.metrics.tstr.TSTRClassifier[source]

Bases: abc.ABC

Abstract classification baseclass for TSTR.

abstract fit(train_x, train_y)[source]

Fit model on train data and labels.

Return type: None

abstract predict(test_x)[source]

Predict labels for test data.

Return type: ndarray

class ecgan.evaluation.metrics.tstr.TSTRRandomForest(n_estimators)[source]

Bases: ecgan.evaluation.metrics.tstr.TSTRClassifier

ECGAN wrapper for sklearn RandomForestClassifier.

fit(train_x, train_y)[source]: Fit model on train data and labels.

predict(test_x)[source]

Predict labels for test data.

Return type: ndarray

class ecgan.evaluation.metrics.tstr.TSTRSVM(kernel=SklearnSVMKernels.RBF)[source]

Bases: ecgan.evaluation.metrics.tstr.TSTRClassifier

ECGAN wrapper for sklearn SVM classifier.

fit(train_x, train_y)[source]: Fit model on train data and labels.

predict(test_x)[source]

Predict labels for test data.

Return type: ndarray

class ecgan.evaluation.metrics.tstr.TSTRCNN(num_channels, num_classes, seq_len, device)[source]

Bases: ecgan.evaluation.metrics.tstr.TSTRClassifier

Fit a small CNN as a classifier for TSTR.

This means that the fit and predict methods are implemented and accessed as wrappers around the usual training/evaluation steps.

fit(train_x, train_y)[source]

Fit CNN.

Return type: None

predict(test_x)[source]

Infer using trained CNN.

Return type: ndarray

class ecgan.evaluation.metrics.tstr.TSTRClassifierFactory[source]

Bases: object

Meta module for creating a classifier instance for TSTR.

class ecgan.evaluation.metrics.tstr.TSTR(classifier='random_forest', **kwargs)[source]

Bases: object

TSTR: train on synthetic, test on real data.