I sometimes stumble upon features in TensorFlow which look potentially powerful or interesting, which I did not know about before. This wiki page is supposed to be a collection of such features. (Once some of them becomes more central to RETURNN, we might want to have a separate wiki page for it. E.g. Distributed TensorFlow. To find them, it is interesting to just browse through tf.raw_ops.

Also note that it is actually quite simple to write native simple ops to return custom information (e.g. like the OS pid).

Operations / functions

• tf.math.divide_no_nan/tf.div_no_nan. Computes x/y and return 0 if y==0 (no nan).
• tf.math.xdivy. Computes x/y and return 0 if x==0.
• tf.math.reciprocal_no_nan. Computes 1/x and return 0 if x==0 (no nan).
• tf.math.reduce_euclidean_norm. Also reduce_variance, reduce_std, cumulative_logsumexp.
• tf.switch_case and tf.case
• tf.strings.to_hash_bucket
• tf.strings.ngrams
• tf.compat.v1.get_session_handle. Keep tensor alive in the session, and access it in future calls via get_session_tensor, or use it for feed_dict.
• tf.searchsorted, also lower_bound or LowerBound
• tf.timestamp. Provides the time since epoch in seconds.
• tf.raw_ops.NonDeterministicInts / non_deterministic_ints. Uses OS-provided source of non-determinism (e.g. an RNG) to return random int.

Resources

• DenseHashTable and other (hash) tables, LookupTable, MutableHashTable, ...
• MapStagingArea
• TensorList. List of tensors.
• Stack. Stack of tensors.
• TensorArray (probably more well known). Array of tensors.
• BoostedTrees
• ConditionalAccumulator. Accumulate / aggregate gradients.
• TemporaryVariable. Variable which only lives within a single step.
• Barrier / Mutex
• Optional. Wraps a value (tensor) that may/may not be present at runtime. Wrapped in a tf.variant tensor.
• RaggedTensor. Tensor with one or more ragged dimensions, which are dimensions whose slices may have different lengths.