英特尔优化的Tensorflow不支持oneDNN

Question

案例1

框架: Tensorflow 2.5.0，Intel-Tensorflow 2.5.0

环境: Google

我有一个由LPOT量化的量化模型，可以在不使用LPOT的情况下进行推理，因此我编写了以下推理代码：

with tf.compat.v1.Session() as sess:
    tf.compat.v1.saved_model.loader.load(sess, ['serve'], model)
    output = sess.graph.get_tensor_by_name(output_tensor_name)
    predictions = sess.run(output, {input_tensor_name: x})
    mse = tf.reduce_mean(tf.keras.losses.mean_squared_error(y, predictions))
    print(mse.eval())

运行行predictions = sess.run(output, {input_tensor_name: x})时

---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1374     try:
-> 1375       return fn(*args)
   1376     except errors.OpError as e:

7 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1359       return self._call_tf_sessionrun(options, feed_dict, fetch_list,
-> 1360                                       target_list, run_metadata)
   1361 

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
   1452                                             fetch_list, target_list,
-> 1453                                             run_metadata)
   1454 

InternalError: Missing 0-th output from {{node model/layer_1/Conv2D_eightbit_requantize}}

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-6-2bddd853d111> in <module>()
      2     tf.compat.v1.saved_model.loader.load(sess, ['serve'], model)
      3     output = sess.graph.get_tensor_by_name(output_tensor_name)
----> 4     predictions = sess.run(output, {input_tensor_name: x[:64]}) # 64, 257, 60, 1
      5     mse = tf.reduce_mean(tf.keras.losses.mean_squared_error(y[:64], predictions))
      6     print(mse.eval())

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    966     try:
    967       result = self._run(None, fetches, feed_dict, options_ptr,
--> 968                          run_metadata_ptr)
    969       if run_metadata:
    970         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1189     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1190       results = self._do_run(handle, final_targets, final_fetches,
-> 1191                              feed_dict_tensor, options, run_metadata)
   1192     else:
   1193       results = []

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1367     if handle is None:
   1368       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1369                            run_metadata)
   1370     else:
   1371       return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1392                     '\nsession_config.graph_options.rewrite_options.'
   1393                     'disable_meta_optimizer = True')
-> 1394       raise type(e)(node_def, op, message)
   1395 
   1396   def _extend_graph(self):

InternalError: Missing 0-th output from node model/layer_1/Conv2D_eightbit_requantize (defined at <ipython-input-6-2bddd853d111>:2) 

无论是否安装了Intel-Tensorflow==2.5.0 ，都会发生此错误，在显式设置os.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'时也不会解决。

另一方面，当我在VS代码中使用Python 3.6.8 64-bit base: Conda运行相同的代码时，它将返回与Case 2相同的错误消息。

案例2

框架: Tensorflow 2.4.0，Intel-Tensorflow 2.4.0

环境: Google

本例运行良好，并输出了预测的MSE丢失，但当我卸载Intel-Tensorflow 2.4.0并仅使用官方Tensorflow运行时，在Case 1 (predictions = sess.run(output, {input_tensor_name: x}))中运行同一行时：

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1374     try:
-> 1375       return fn(*args)
   1376     except errors.OpError as e:

7 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1357       # Ensure any changes to the graph are reflected in the runtime.
-> 1358       self._extend_graph()
   1359       return self._call_tf_sessionrun(options, feed_dict, fetch_list,

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _extend_graph(self)
   1397     with self._graph._session_run_lock():  # pylint: disable=protected-access
-> 1398       tf_session.ExtendSession(self._session)
   1399 

InvalidArgumentError: No OpKernel was registered to support Op 'QuantizedMatMulWithBiasAndDequantize' used by {{node model/dense/Tensordot/MatMul_eightbit_requantize}} with these attrs: [input_quant_mode="MIN_FIRST", T1=DT_QUINT8, Toutput=DT_FLOAT, T2=DT_QINT8, Tbias=DT_QINT32, transpose_a=false, transpose_b=false]
Registered devices: [CPU]
Registered kernels:
  <no registered kernels>

     [[model/dense/Tensordot/MatMul_eightbit_requantize]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-6-2bddd853d111> in <module>()
      2     tf.compat.v1.saved_model.loader.load(sess, ['serve'], model)
      3     output = sess.graph.get_tensor_by_name(output_tensor_name)
----> 4     predictions = sess.run(output, {input_tensor_name: x[:64]}) # 64, 257, 60, 1
      5     mse = tf.reduce_mean(tf.keras.losses.mean_squared_error(y[:64], predictions))
      6     print(mse.eval())

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    966     try:
    967       result = self._run(None, fetches, feed_dict, options_ptr,
--> 968                          run_metadata_ptr)
    969       if run_metadata:
    970         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1189     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1190       results = self._do_run(handle, final_targets, final_fetches,
-> 1191                              feed_dict_tensor, options, run_metadata)
   1192     else:
   1193       results = []

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1367     if handle is None:
   1368       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1369                            run_metadata)
   1370     else:
   1371       return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1392                     '\nsession_config.graph_options.rewrite_options.'
   1393                     'disable_meta_optimizer = True')
-> 1394       raise type(e)(node_def, op, message)
   1395 
   1396   def _extend_graph(self):

InvalidArgumentError: No OpKernel was registered to support Op 'QuantizedMatMulWithBiasAndDequantize' used by node model/dense/Tensordot/MatMul_eightbit_requantize (defined at <ipython-input-6-2bddd853d111>:2)  with these attrs: [input_quant_mode="MIN_FIRST", T1=DT_QUINT8, Toutput=DT_FLOAT, T2=DT_QINT8, Tbias=DT_QINT32, transpose_a=false, transpose_b=false]
Registered devices: [CPU]
Registered kernels:
  <no registered kernels>

     [[model/dense/Tensordot/MatMul_eightbit_requantize]]

即使显式地设置了os.environ['TF_ENABLE_ONEDNN_OPTS'] = '1'，错误仍然存在。

结论

我认为这两种情况都是由相同类型的错误引起的，即没有注册OpKernel来支持Op。

据了解，在安装了正式的Tensorflow v2.5和环境变量TF_ENABLE_ONEDNN_OPTS=1 set (参考文献)之后，量化模型应该在支持oneDNN的情况下运行。但无论是2.4版还是2.5版，情况似乎都不是这样。

我的问题是，如何在不需要安装Tensorflow 2.5的情况下获得oneDNN支持的官方oneDNN环境？或者为什么Intel-Tensorflow 2.5不能工作？谢谢。

Answer 1

LPOT是在Intel AI Analytics工具包中发布的，它与英特尔优化的TensorFlow一起工作。LPOT可以在任何Intel CPU上运行，以量化AI模型。英特尔优化的TensorFlow 2.5.0要求在运行TF_ENABLE_MKL_NATIVE_FORMAT=0量化或部署量化模型之前设置环境变量TF_ENABLE_MKL_NATIVE_FORMAT=0。

有关更多信息，请参考这。

请您检查一下，您是否在2.4中量化了Tensorflow的模型，并在Tensorflow 2.5上运行了推断？对于不运行在Tensorflow 2.5中和在Tensorflow 2.4中运行的模型，一个合理的解释是，支持Tensorflow 2.5的操作员可能不支持在Tensorflow 2.4中创建的模型。