Parameter Updaters

Abstract Base Class

class hebel.parameter_updaters.ParameterUpdater(model)
post_gradient_update(gradients, stream=None)
pre_gradient_update(stream=None)

Simple SGD Update

class hebel.parameter_updaters.SimpleSGDUpdate(model)
post_gradient_update(gradients, batch_size, learning_parameters, stream=None)

Momentum Update

class hebel.parameter_updaters.MomentumUpdate(model)
post_gradient_update(gradients, batch_size, learning_parameters, stream=None)

Nesterov Momentum Update

class hebel.parameter_updaters.NesterovMomentumUpdate(model)
post_gradient_update(gradients, batch_size, learning_parameters, stream=None)

Second step of Nesterov momentum method: take step in direction of new gradient and update velocity

pre_gradient_update()

First step of Nesterov momentum method: take step in direction of accumulated gradient