autoflow.metrics package

Submodules

autoflow.metrics.classification_metrics module

autoflow.metrics.classification_metrics.balanced_accuracy(solution, prediction)

autoflow.metrics.classification_metrics.pac_score(solution, prediction)

Probabilistic Accuracy based on log_loss metric. We assume the solution is in {0, 1} and prediction in [0, 1]. Otherwise, run normalize_array. :param solution: :param prediction: :param ml_task: :return:

autoflow.metrics.classification_metrics.sensitivity(solution, prediction)

autoflow.metrics.classification_metrics.specificity(solution, prediction)

Module contents

class autoflow.metrics.Scorer(name, score_func, optimum, sign, kwargs)

Bases: object

autoflow.metrics.calculate_confusion_matrix(y_true, y_pred) → List[List[int]]

autoflow.metrics.calculate_score(solution, prediction, ml_task: autoflow.utils.ml_task.MLTask, metric, should_calc_all_metric=False)

autoflow.metrics.make_scorer(name, score_func, optimum=1, greater_is_better=True, needs_proba=False, needs_threshold=False, **kwargs)

Make a scorer from a performance metric or loss function.

Factory inspired by scikit-learn which wraps scikit-learn scoring functions to be used in auto-sklearn.

Parameters

• score_func (callable) – Score function (or loss function) with signature score_func(y, y_pred, **kwargs).

• optimum (int or float, default=1) – The best score achievable by the score function, i.e. maximum in case of scorer function and minimum in case of loss function.

• greater_is_better (boolean, default=True) – Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the score_func.

• needs_proba (boolean, default=False) – Whether score_func requires predict_proba to get probability estimates out of a classifier.

• needs_threshold (boolean, default=False) – Whether score_func takes a continuous decision certainty. This only works for binary classification.

• **kwargs (additional arguments) – Additional parameters to be passed to score_func.

Returns

scorer – Callable object that returns a scalar score; greater is better.

Return type

callable