Numerical Stability
NoNansAssertion
CLASS efemarai.assertions.NoNansAssertion(TensorAssertion)
Ensure that no tensor contains any NaN values.
Registered by default.
Examples:
ef.register_assertion(NoNansAssertion()) ef.deregister_assertion(NoNansAssertion)
NoInfsAssertion
CLASS efemarai.assertions.NoInfsAssertion(TensorAssertion)
Ensure that no tensor contains any Inf values.
Registered by default.
Examples:
ef.register_assertion(NoInfsAssertion()) ef.deregister_assertion(NoInfsAssertion)
NoNonZeroGradientsAssertion
CLASS efemarai.assertions.NoNonZeroGradientsAssertion(TensorAssertion)
Ensure that all gradients contain only zeros before the backward pass is executed.
Registered by default.
Examples:
ef.register_assertion(NoNonZeroGradientsAssertion()) ef.deregister_assertion(NoNonZeroGradientsAssertion)
NoExplodingTensors
CLASS efemarai.assertions.NoExplodingTensors(FunctionAssertion)
Ensure that no tensor in the computational graph has an element less / greater than a certain threshold.
def __init__(self, threshold=1e5)
Initialize the assertion.
Parameters
- threshold (
float) – if a tensor has elements with absolute value greater than this threshold then the tensor is considered ‘exploding’.
Examples:
ef.register_assertion(NoExlodingTensors(threshold=1e2))
NoVanishingTensors
CLASS efemarai.assertions.NoVanishingTensors(FunctionAssertion)
Ensure that no tensor in the computational graph becomes almost all zeros.
def __init__(self, threshold=1e-5)
Initialize the assertion.
Parameters
- threshold (
float) – if the mean of the absolute values of all tensor elements is less than this threshold then the tensor is considered ‘vanishing’.
Examples:
ef.register_assertion(NoVanishingTensors(threshold=1e-3))
Activation Functions
NoDeadReLULayers
CLASS efemarai.assertions.NoDeadReLULayers(FunctionAssertion)
Ensure that there are no dead ReLU layers in your model. In order to avoid storing previous ReLU outputs this assertion detects dead ReLU layers based on batch activations.
def __init__(self, threshold=0.8)
Initialize the assertion.
Parameters
- threshold (
float) – if the fraction of inactive ReLU outputs per batch is greater than the threshold, then the layer is considered dead.
Examples:
ef.register_assertion(NoDeadReLUs(threshold=0.5))
NoSaturatedSigmoidUnits
CLASS efemarai.assertions.NoSaturatedSigmoidUnits(FunctionAssertion)
Ensure that there are no saturated sigmoid units in your model.
def __init__(self, threshold=6)
Initialize the assertion.
Parameters
- threshold (
float) – if the absolute value of the input to a sigmoid unit is greater than this threshold then the unit is considered saturated.
NoSaturatedTanhUnits
CLASS efemarai.assertions.NoSaturatedTanhUnits(FunctionAssertion)
Ensure that there are no saturated tanh units in your model.
def __init__(self, threshold=6)
Initialize the assertion.
Parameters
- threshold (
float) – if the absolute value of the input to a tanh unit is greater than this threshold then the unit is considered saturated.
Loss Functions
ValidInputsDiscreteNLLLoss
CLASS efemarai.assertions.ValidInputsDiscreteNLLLoss(TensorAssertion)
When using the NLL loss for discrete problems such as classification the input is probability mass and so its log cannot be positive.
ValidInputsDiscretePoissonNLLLoss
CLASS efemarai.assertions.ValidInputsDiscretePoissonNLLLoss(TensorAssertion)
When using the Poisson NLL loss for discrete problems the input is probability mass and so its log cannot be positive. Takes into account the log_input input argument.
ValidInputsDiscreteKLDivLoss
CLASS efemarai.assertions.ValidInputsDiscreteKLDivLoss(TensorAssertion)
When using the KL divergence loss for discrete problems the input is probability mass and so its log cannot be positive. Takes into account the log_target input argument.