tutorials/Tutorial-4-Adding-Support-for-More-Operations.md

+
+### Overview
+The initial release of the TFMin library supports a basic set of TensorFlow operations supporting a range
+of common network models, it is inevitable that at some point support for additional operations will need to be added
+during development of larger projects. This tutorial will guide you through the process of creating a new `op_kernel`
+object and adding it to the library.
+
+### OpKernel Architecture
+TFMin includes a set of op_kernel objects which are used to generate the C++ equivalents of given tensorflow
+operations. Each of these objects has a `match()` method which takes a TensorFlow operation object and returns *True*
+if it is able to generate it. If it is then the objects `gen_code()` method is invoked, returning a 
+string containing the generated code.
+
+All op_kernel objects are specialisations of the `BaseOpKernel` base class, this provides a common interface, helper
+methods, and a mechanism for python to list all defined op_kernels. These classes are located in a module in the
+`op_kernels` sub-directly of the tf_min python source.
+
+The `op_kernels` module has been setup so that any additional python
+files are automatically loaded. This means that to create a new op_kernel you can simply create a python
+file in this directory, import the base class and define the new op_kernel. Loading and detection of
+op_kernels is handled automatically by the library.
+
+### OpKernel Base Class
+
+#### Methods to override
+The `BaseOpKernel` object contains four methods that should be overridden in your own operations:
+
+```python
+@staticmethod
+def matches(tf_op):
+    return tf_op.type == "YourOpType"
+```
+This method is used to identify which operations this kernel is compatible with, in most cases it will
+simply check the operation type but it can also perform more complex checks if it is only compatible
+with certain forms of the operation.
+
+```python
+@staticmethod
+def description():
+    return "Short description, will be printed when a developer lists all supported operations."
+```
+This one's pretty self explanatory, can be used to describe the status and origin of this kernel.
+
+```python
+@staticmethod
+def status():
+    return "Production"  # "Testing" or "Development"
+```
+Kernels report their own development status, so that when a flow-graph is being converted a warning can be
+shown if any development or testing kernels are being used.
+
+```python
+@classmethod
+def gen_code(cls, tf_op, inputs):
+    
+    # Python code to generate C++ equivalent of TensorFlow operation        
+    return "// Some C++ code"
+```
+The `gen_code()` method does the heavy lifting for the op_kernel. It is passed the TensorFlow Operation object
+`tf_op` and a list of input tensors `inputs`, it then returns a string of the generated C++ code. The TensorFlow 
+operation also contains a list of input tensors, but this will not always be the same as the list of inputs being used
+by TFMin. During flow-graph analysis some operations may be skipped, identity operations, control flow operations
+with inputs resolve that to constants, etc. It is necessary to use the given input list or the code will fail to compile
+because it tries to access tensors which have not been created.
+
+**Note:** Unlike the others this method needs to be defined with `@classmethod`
+
+#### Helper Methods
+
+##### output_assignment Method 
+```python
+@staticmethod
+def output_assignment(tf_op, idx=0):
+```
+This method returns a string of the statements that create a tensor and the left hand side of an assignment to it,
+corresponding to the specified output of the operation. 
+The statements needed to do this vary depending on the settings used and should be abstracted from the code generated
+for the right hand side of the assignment. The kernel object can assume that the left hand side will be an `Eigen::Tensor` or
+`Eigen::TensorMap` object of the type and size required. Shown below is an example of the statements produced by this method:
+```cpp
+Eigen::TensorMap<Eigen::Tensor<float, 2, Eigen::RowMajor>> Reshape_0((float*)(memoryBlock + 4000),  1, 1000 );  
+Reshape_0 =
+```
+This code is producing an `Eigen::TensorMap` of a pre-defined region of memory to avoid dynamic memory allocation, and
+producing the left hand side and `=` of an assignment to it. The left hand side can then be generated by the specific
+operation kernel.
+
+##### print_operation_details Method
+```python
+@staticmethod
+def print_operation_details(tf_op):
+```
+This method is useful when trying to find how TensorFlow has defined the given operation, which isn't always
+as clear as it could be from the documentation. It prints the shape and type of input and output tensors is shown
+along with any attributes of the operation, the output for a typical 2D convolution is shown below:
+```
+Details of operation type [Conv2D] -------------------
+Attributes:
+   "T"		type(<class 'tensorflow.python.framework.dtypes.DType'>)		value(<dtype: 'float32'>)
+   "data_format"		type(<class 'bytes'>)		value(b'NHWC')
+   "padding"		type(<class 'bytes'>)		value(b'SAME')
+   "strides"		type(<class 'list'>)		value([1, 1, 1, 1])
+   "use_cudnn_on_gpu"		type(<class 'bool'>)		value(True)
+2 input tensors:
+   [ 0] "concat_7:0" float rank(3) [ 13  13 512] : source op ("concat_7" - ConcatV2)
+   [ 1] "Const_25:0" float rank(4) [   1    1  512 1000] : source op ("Const_25" - Const)
+1 output tensors:
+   [ 0] "Conv2D_25:0" float rank(3) [  13   13 1000]
+--------------------------------------------------
+
+```
+
+### Creating an OpKernel Object for the Reciprocal Operation
+
+We will use a simple unary element-wise operation, the reciprocal, to demonstrate how to create a new
+op_kernel in its own python file. Make of a copy of the SqueezeNet example in a new directly called
+`squeeze_net_inv` and add a reciprocal operation in-between the final reshape and softmax operations. 
+This can be found at the end of the `net_preloaded` function, this code should look like the snippet below:
+```python
+    x = tf.reshape(x, (1, 1000))
+
+    x = tf.reciprocal(x)  # Added reciprocal operation
+
+    x = tf.nn.softmax(x)
+    net['classifier_actv'] = x
+
+print("Network instance created: %fs" % (time.time() - cr_time))
+
+return net
+```
+
+Now when we run the script to build and export this model it will fail with the error below, in fact this
+is probably the error message that brought you to this tutorial in the first place:
+
+```
+Analysed flow-graph
+Error : This model uses the following 1 types of operation that are not currently supported by TFMin:
+Reciprocal
+```
+
+#### Source Code of the New OpKernel
+```python
+import tensorflow as tf
+import numpy as np
+import tf_min.cpp_code_gen as code_gen
+import tf_min.tf_utils as tf_utils
+import tf_min.op_kernels.base_op as base_op
+
+"""
+    Layout operation kernels of the TFMin library
+    
+    Operations supported
+    -------------------------------
+    Reciprocal       (dev)
+"""
+
+
+class ReciprocalOpKernel(base_op.BaseOpKernel):
+    """Code generator kernel for the Reciprocal elementwise tensorflow operation."""
+
+    @staticmethod
+    def matches(tf_op):
+        return tf_op.type == "Reciprocal"
+
+    @staticmethod
+    def description():
+        return "Reciprocal operation, currently in development."
+
+    @staticmethod
+    def status():
+        return "Development"
+
+    @classmethod
+    def gen_code(cls, tf_op, inputs):
+
+        input0_identifier = code_gen.c_safe_identifier(inputs[0].name)
+
+        code = "%s %s.inverse();" % \
+               (base_op.BaseOpKernel.output_assignment(tf_op, idx=0),
+                input0_identifier)
+
+        return code
+```
+
+First we have defined our new kernel object as a descendant of the BaseOpKernel object. We have used the convention of `<TensorFlow Op Type>OpKernel` to name kernel objects, although any name will work in
+practice we encourage developers to stick to the same convention.
+
+Next we have overridden the `matches` method so this object will only be used to generate code for Reciprocal
+operations. Then the `description` and `status` methods are overridden to provide appropriate information about
+our new kernel.
+
+Finally the `gen_code` method is overridden, which uses a range of helper functions to generate the C++ code itself.
+This operation produces a single tensor, so a single assignment operation is generated by the following line:
+```python
+code = "%s %s.inverse();" % \
+       (base_op.BaseOpKernel.output_assignment(tf_op, idx=0),
+        input0_identifier)
+```
+
+#### Adding the New OpKernel
+
+Create a new python file called `demo_ops.py` within the `tf_min/op_kernels` directory and copy the code shown above
+into it to add it into the TFMin library. You will now be able to re-run the SqueezeNetInv python exporter script to
+attempt to export the model again. This time around it should be successful, although a warning will have been shown
+telling you that non-production OpKernels are being used in the export of this graph.
+```
+Analysed flow-graph
+Warning : This model will use development or testing OpKernels:
+(Development) Reciprocal
+```
+
+### Summary
+This tutorial should have given you an overview of the process to add support for new operations to the TFMin library.
+A simple example was used here for clarity, there are more complex OpKernels within the existing code base that can be 
+explored to get a deeper understanding. 
+
+In keeping with the open-source ethos we encourage any developers to share any new OpKernels they develop under the same
+GPL licence that this library has been distributed under, and we are happy to provide support to developers who are
+sharing their work.
+
+---
+[Previous Tutorial](/Tutorials/Tutorial-3-Validating-the-Accuracy-of-a-Model) - Validating the Accuracy of a Model
+
+[Next Tutorial](/Tutorials/Tutorial-5-Building-Code-for-the-LEON-3) - Building Code for the LEON 3