contrib.opt.VariableClippingOptimizer
tf.contrib.opt.VariableClippingOptimizer
class tf.contrib.opt.VariableClippingOptimizer
Defined in tensorflow/contrib/opt/python/training/variable_clipping_optimizer.py
.
Wrapper optimizer that clips the norm of specified variables after update.
This optimizer delegates all aspects of gradient calculation and application to an underlying optimizer. After applying gradients, this optimizer then clips the variable to have a maximum L2 norm along specified dimensions. NB: this is quite different from clipping the norm of the gradients.
Multiple instances of VariableClippingOptimizer
may be chained to specify different max norms for different subsets of variables.
This is more efficient at serving-time than using normalization during embedding lookup, at the expense of more expensive training and fewer guarantees about the norms.
Methods
__init__
__init__( opt, vars_to_clip_dims, max_norm, use_locking=False, colocate_clip_ops_with_vars=False, name='VariableClipping' )
Construct a new clip-norm optimizer.
Args:
-
opt
: The actual optimizer that will be used to compute and apply the gradients. Must be one of the Optimizer classes. -
vars_to_clip_dims
: A dict with keys as Variables and values as lists of dimensions along which to compute the L2-norm. Seetf.clip_by_norm
for more details. -
max_norm
: The L2-norm to clip to, for all variables specified. -
use_locking
: IfTrue
use locks for clip update operations. -
colocate_clip_ops_with_vars
: IfTrue
, try colocating the clip norm ops with the corresponding variable. -
name
: Optional name prefix for the operations created when applying gradients. Defaults to "VariableClipping".
apply_gradients
apply_gradients( grads_and_vars, global_step=None, name=None )
compute_gradients
compute_gradients( *args, **kwargs )
get_name
get_name()
get_slot
get_slot( *args, **kwargs )
get_slot_names
get_slot_names( *args, **kwargs )
minimize
minimize( loss, global_step=None, var_list=None, gate_gradients=GATE_OP, aggregation_method=None, colocate_gradients_with_ops=False, name=None, grad_loss=None )
Add operations to minimize loss
by updating var_list
.
This method simply combines calls compute_gradients()
and apply_gradients()
. If you want to process the gradient before applying them call compute_gradients()
and apply_gradients()
explicitly instead of using this function.
Args:
-
loss
: ATensor
containing the value to minimize. -
global_step
: OptionalVariable
to increment by one after the variables have been updated. -
var_list
: Optional list or tuple ofVariable
objects to update to minimizeloss
. Defaults to the list of variables collected in the graph under the keyGraphKeys.TRAINABLE_VARIABLES
. -
gate_gradients
: How to gate the computation of gradients. Can beGATE_NONE
,GATE_OP
, orGATE_GRAPH
. -
aggregation_method
: Specifies the method used to combine gradient terms. Valid values are defined in the classAggregationMethod
. -
colocate_gradients_with_ops
: If True, try colocating gradients with the corresponding op. -
name
: Optional name for the returned operation. -
grad_loss
: Optional. ATensor
holding the gradient computed forloss
.
Returns:
An Operation that updates the variables in var_list
. If global_step
was not None
, that operation also increments global_step
.
Raises:
-
ValueError
: If some of the variables are notVariable
objects.
Class Members
GATE_GRAPH
GATE_NONE
GATE_OP
© 2017 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/api_docs/python/tf/contrib/opt/VariableClippingOptimizer