Numerical Stability

NoNansAssertion

CLASS efemarai.assertions.NoNansAssertion(TensorAssertion)

Ensure that no tensor contains any NaN values.

Registered by default.

Examples:

ef.register_assertion(NoNansAssertion())

ef.deregister_assertion(NoNansAssertion)

NoInfsAssertion

CLASS efemarai.assertions.NoInfsAssertion(TensorAssertion)

Ensure that no tensor contains any Inf values.

Registered by default.

Examples:

ef.register_assertion(NoInfsAssertion())

ef.deregister_assertion(NoInfsAssertion)

NoNonZeroGradientsAssertion

CLASS efemarai.assertions.NoNonZeroGradientsAssertion(TensorAssertion)

Ensure that all gradients contain only zeros before the backward pass is executed.

Registered by default.

Examples:

ef.register_assertion(NoNonZeroGradientsAssertion())

ef.deregister_assertion(NoNonZeroGradientsAssertion)

NoExplodingTensors

CLASS efemarai.assertions.NoExplodingTensors(FunctionAssertion)

Ensure that no tensor in the computational graph has an element less / greater than a certain threshold.

def __init__(self, threshold=1e5)

Initialize the assertion.

Parameters

  • threshold (float) – if a tensor has elements with absolute value greater than this threshold then the tensor is considered ‘exploding’.

Examples:

ef.register_assertion(NoExlodingTensors(threshold=1e2))

NoVanishingTensors

CLASS efemarai.assertions.NoVanishingTensors(FunctionAssertion)

Ensure that no tensor in the computational graph becomes almost all zeros.

def __init__(self, threshold=1e-5)

Initialize the assertion.

Parameters

  • threshold (float) – if the mean of the absolute values of all tensor elements is less than this threshold then the tensor is considered ‘vanishing’.

Examples:

ef.register_assertion(NoVanishingTensors(threshold=1e-3))

Activation Functions

NoDeadReLULayers

CLASS efemarai.assertions.NoDeadReLULayers(FunctionAssertion)

Ensure that there are no dead ReLU layers in your model. In order to avoid storing previous ReLU outputs this assertion detects dead ReLU layers based on batch activations.

def __init__(self, threshold=0.8)

Initialize the assertion.

Parameters

  • threshold (float) – if the fraction of inactive ReLU outputs per batch is greater than the threshold, then the layer is considered dead.

Examples:

ef.register_assertion(NoDeadReLUs(threshold=0.5))

NoSaturatedSigmoidUnits

CLASS efemarai.assertions.NoSaturatedSigmoidUnits(FunctionAssertion)

Ensure that there are no saturated sigmoid units in your model.

def __init__(self, threshold=6)

Initialize the assertion.

Parameters

  • threshold (float) – if the absolute value of the input to a sigmoid unit is greater than this threshold then the unit is considered saturated.

NoSaturatedTanhUnits

CLASS efemarai.assertions.NoSaturatedTanhUnits(FunctionAssertion)

Ensure that there are no saturated tanh units in your model.

def __init__(self, threshold=6)

Initialize the assertion.

Parameters

  • threshold (float) – if the absolute value of the input to a tanh unit is greater than this threshold then the unit is considered saturated.

Loss Functions

ValidInputsDiscreteNLLLoss

CLASS efemarai.assertions.ValidInputsDiscreteNLLLoss(TensorAssertion)

When using the NLL loss for discrete problems such as classification the input is probability mass and so its log cannot be positive.

ValidInputsDiscretePoissonNLLLoss

CLASS efemarai.assertions.ValidInputsDiscretePoissonNLLLoss(TensorAssertion)

When using the Poisson NLL loss for discrete problems the input is probability mass and so its log cannot be positive. Takes into account the log_input input argument.

ValidInputsDiscreteKLDivLoss

CLASS efemarai.assertions.ValidInputsDiscreteKLDivLoss(TensorAssertion)

When using the KL divergence loss for discrete problems the input is probability mass and so its log cannot be positive. Takes into account the log_target input argument.