devbox
代码吟游诗人2
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

使用 trt 的int8 量化和推断 onnx 模型_trt量化

作者：代码吟游诗人2 | 2024-02-03 18:55:21

赞

踩

trt量化

2022-04-06 更新：

理清几个概念：

1、onnx模型本身要有动态维度，否则只能转静态维度的trt engine。

2、只要一个profile就够了，设个最小最大维度，最优就是最常用的维度。在推断的时候要绑定一下。

3、builder 和 config 里有很多相同的设置，如果用了 config，就不需要设置 builder中的相同参数了。

  1.  
  2. def onnx_2_trt(onnx_filename, engine_filename, mode='fp32', max_batch_size=1, min_wh=(160,160), max_wh=(320,320), int8_calib=None):
  3.     ''' convert onnx to tensorrt engine, use mode of ['fp32', 'fp16', 'int8']
  4.     :return: trt engine
  5.     '''
  6.  
  7.     assert mode in ['fp32', 'fp16', 'int8'], "mode should be in ['fp32', 'fp16', 'int8']"
  8.  
  9.     G_LOGGER = trt.Logger(trt.Logger.WARNING)
  10.     # TRT7中的onnx解析器的network，需要指定EXPLICIT_BATCH
  11.     EXPLICIT_BATCH = 1 << (int)(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
  12.     with trt.Builder(G_LOGGER) as builder, builder.create_network(EXPLICIT_BATCH) as network, \
  13.             trt.OnnxParser(network, G_LOGGER) as parser:
  14.  
  15.         print('Loading ONNX file from path %s...'%(onnx_filename))
  16.         with open(onnx_filename, 'rb') as model:
  17.             print('Beginning ONNX file parsing')
  18.             if not parser.parse(model.read()):
  19.                 for e in range(parser.num_errors):
  20.                     print(parser.get_error(e))
  21.                 raise TypeError('Parser parse failed.')
  22.  
  23.         print('Completed parsing of ONNX file')
  24.         
  25.         # wujp 2022-03-29 如果使用了config，builder就不用设置了。
  26.         builder.max_batch_size = max_batch_size # max_batch_size 在config中没有
  27.         #builder.max_workspace_size = 1 << 30
  28.             
  29.         if mode == 'int8':
  30.             assert (builder.platform_has_fast_int8 == True), 'not support int8'
  31.             #builder.int8_mode = True
  32.             #builder.int8_calibrator = calib
  33.         elif mode == 'fp16':
  34.             assert (builder.platform_has_fast_fp16 == True), 'not support fp16'
  35.             #builder.fp16_mode = True
  36.  
  37.         profile = builder.create_optimization_profile()
  38.         inputs = [network.get_input(i) for i in range(network.num_inputs)]
  39.         for inp in inputs:
  40.             fbs, shape = inp.shape[0], inp.shape[1:]
  41.             if (shape[1] == -1 and shape[2] == -1): # height 和 width 都是动态的。
  42.                 profile.set_shape(inp.name, min=(1, shape[0], *min_wh), opt=(8, shape[0], *min_wh), max=(max_batch_size, shape[0], *max_wh))
  43.             else:
  44.                 profile.set_shape(inp.name, min=(1, *shape), opt=(8, *shape), max=(max_batch_size, *shape))
  45.         
  46.         config = builder.create_builder_config()
  47.         config.max_workspace_size = 1 << 30
  48.         if mode == 'int8':
  49.             config.set_flag(trt.BuilderFlag.INT8)
  50.             config.int8_calibrator = int8_calib
  51.         elif mode == 'fp16':
  52.             config.set_flag(trt.BuilderFlag.FP16)
  53.         config.add_optimization_profile(profile)
  54.         config.set_calibration_profile(profile) # 不加会有警告 [TensorRT] WARNING: Calibration Profile is not defined. Runing calibration with Profile 0
  55.         print('Building an engine from file %s; this may take a while...'%(onnx_filename))
  56.         #engine = builder.build_cuda_engine(network, config)
  57.         engine = builder.build_engine(network, config)
  58.         print('Created engine success! ')
  59.  
  60.         # 保存计划文件
  61.         print('Saving TRT engine file to path %s...'%(engine_filename))
  62.         with open(engine_filename, 'wb') as f:
  63.             f.write(engine.serialize())
  64.         print('Engine file has already saved to %s!'%(engine_filename))
  65.         return engine

--------------------------------------------------------------------

2022-03-27

以下代码单batch没问题，多batch不行。

目录

生成 trt 模型

1、使用代码

2、onnx模型和图片

3、修改代码

4、结果

推断 trt 模型

生成 trt 模型

1、使用代码

https://github.com/rmccorm4/tensorrt-utils.git

2、onnx模型和图片

模型：动态batch输入（假设为 mob_w160_h160.onnx，输入是 [batchsize, 3, 160, 160]）。

图片：一堆图片（假设有1024张），不需要其他描述文件。

 在 tensorrt-utils/int8/calibration/ 下创建子文件夹（假设叫 my），模型和图片放入其中。

3、修改代码

来到 tensorrt-utils/int8/calibration/ 目录。

修改 onnx_to_tensorrt.py 中 main，主要是修改路径和校准图片数。

  1. def main():
  2.     my_dir = os.path.dirname(os.path.realpath(__file__)) + '/my/'
  3.     
  4.     parser = argparse.ArgumentParser(description="Creates a TensorRT engine from the provided ONNX file.\n")
  5.     parser.add_argument("--onnx", help="The ONNX model file to convert to TensorRT",
  6.                             default=my_dir+'mob_w160_h160.onnx')
  7.     parser.add_argument("-o", "--output", type=str, help="The path at which to write the engine",
  8.                             default=my_dir+'mob_w160_h160.trt')
  9.     parser.add_argument("-b", "--max-batch-size", type=int, default=32, help="The max batch size for the TensorRT engine input")
  10.     parser.add_argument("-v", "--verbosity", action="count", help="Verbosity for logging. (None) for ERROR, (-v) for INFO/WARNING/ERROR, (-vv) for VERBOSE.")
  11.     # 如果explicit-batch 是 False，会有错误 
  12.     # In node -1 (importModel): INVALID_VALUE: Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() &&
  13.     # "This version of the ONNX parser only supports TensorRT INetworkDefinitions with an explicit batch dimension. Please ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."
  14.     parser.add_argument("--explicit-batch", action='store_true', help="Set trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH.",
  15.                             default=True)
  16.     # 如果explicit-precision 是 True, 会有错误
  17.     # "python: ../builder/cudnnBuilderWeightConverters.cpp:162: std::vector<float> nvinfer1::cudnn::makeConvDeconvInt8Weights(nvinfer1::ConvolutionParameters&, 
  18.     # const nvinfer1::rt::EngineTensor&, const nvinfer1::rt::EngineTensor&, float, bool, bool): Assertion `sI.count() == 1' failed."    
  19.     parser.add_argument("--explicit-precision", action='store_true', help="Set trt.NetworkDefinitionCreationFlag.EXPLICIT_PRECISION.",
  20.                             default=False)
  21.     parser.add_argument("--gpu-fallback", action='store_true', help="Set trt.BuilderFlag.GPU_FALLBACK.")
  22.     parser.add_argument("--refittable", action='store_true', help="Set trt.BuilderFlag.REFIT.")
  23.     parser.add_argument("--debug", action='store_true', help="Set trt.BuilderFlag.DEBUG.")
  24.     parser.add_argument("--strict-types", action='store_true', help="Set trt.BuilderFlag.STRICT_TYPES.")
  25.     parser.add_argument("--fp16", action="store_true", help="Attempt to use FP16 kernels when possible.")
  26.     parser.add_argument("--int8", action="store_true", help="Attempt to use INT8 kernels when possible. This should generally be used in addition to the --fp16 flag. \
  27.                                                              ONLY SUPPORTS RESNET-LIKE MODELS SUCH AS RESNET50/VGG16/INCEPTION/etc.",
  28.                             default = True)
  29.     parser.add_argument("--calibration-cache", help="(INT8 ONLY) The path to read/write from calibration cache.", 
  30.                             default=my_dir+"calibration.cache")
  31.     parser.add_argument("--calibration-data", help="(INT8 ONLY) The directory containing {*.jpg, *.jpeg, *.png} files to use for calibration. (ex: Imagenet Validation Set)",
  32.                             default=my_dir+'images/')
  33.     parser.add_argument("--calibration-batch-size", help="(INT8 ONLY) The batch size to use during calibration.", type=int, default=32)
  34.     parser.add_argument("--max-calibration-size", help="(INT8 ONLY) The max number of data to calibrate on from --calibration-data.", type=int,
  35.                             default=1024)
  36.     parser.add_argument("-p", "--preprocess_func", type=str, default=None, help="(INT8 ONLY) Function defined in 'processing.py' to use for pre-processing calibration data.")
  37.     parser.add_argument("-s", "--simple", action="store_true", help="Use SimpleCalibrator with random data instead of ImagenetCalibrator for INT8 calibration.")
  38.     args, _ = parser.parse_known_args()
  39.  
  40.     # 下面的代码不用改

修改 processing.py 中 preprocess_imagenet，修改宽高为160，使用自己的均值和标准差。

  1. #def preprocess_imagenet(image, channels=3, height=224, width=224):
  2. def preprocess_imagenet(image, channels=3, height=160, width=160):
  3.     """Pre-processing for Imagenet-based Image Classification Models:
  4.         resnet50, vgg16, mobilenet, etc. (Doesn't seem to work for Inception)
  6.     Parameters
  7.     ----------
  8.     image: PIL.Image
  9.         The image resulting from PIL.Image.open(filename) to preprocess
  10.     channels: int
  11.         The number of channels the image has (Usually 1 or 3)
  12.     height: int
  13.         The desired height of the image (usually 224 for Imagenet data)
  14.     width: int
  15.         The desired width of the image  (usually 224 for Imagenet data)
  17.     Returns
  18.     -------
  19.     img_data: numpy array
  20.         The preprocessed image data in the form of a numpy array
  22.     """
  23.     # Get the image in CHW format
  24.     resized_image = image.resize((width, height), Image.ANTIALIAS)
  25.     img_data = np.asarray(resized_image).astype(np.float32)
  26.  
  27.     if len(img_data.shape) == 2:
  28.         # For images without a channel dimension, we stack
  29.         img_data = np.stack([img_data] * 3)
  30.         logger.debug("Received grayscale image. Reshaped to {:}".format(img_data.shape))
  31.     else:
  32.         img_data = img_data.transpose([2, 0, 1])
  33.  
  34.     #mean_vec = np.array([0.485, 0.456, 0.406])
  35.     #stddev_vec = np.array([0.229, 0.224, 0.225])
  36.     my_mean = np.array([104, 117, 123])
  37.     assert img_data.shape[0] == channels
  38.  
  39.     for i in range(img_data.shape[0]):
  40.         # Scale each pixel to [0, 1] and normalize per channel.
  41.         #img_data[i, :, :] = (img_data[i, :, :] / 255 - mean_vec[i]) / stddev_vec[i]
  42.         img_data[i, :, :] = img_data[i, :, :] - my_mean[i]
  43.         
  44.     return img_data

4、结果

在 my 中生成的 mob_w160_h160.trt 可以做推断，calibration.cache 没啥用。

为啥 trt比onnx大了很多，是由于动态batch，onnx_to_tensorrt.py 中使用 [1, 8, 16, 32, 64] 五种规格的batch来生成优化 profile。

推断 trt 模型

来到 tensorrt-utils/inference/ 目录。

修改 infer.py 中的 main，主要是修改路径和模型输入。

  1. def main():
  2.     import os
  3.     my_dir = os.path.dirname(os.path.realpath(__file__)) + '/../int8/calibration/my/'
  4.     
  5.     parser = argparse.ArgumentParser()
  6.     #parser.add_argument("-e", "--engine", required=True, type=str,
  7.     #                    help="Path to TensorRT engine file.")
  8.     parser.add_argument("-e", "--engine", type=str, help="Path to TensorRT engine file.",
  9.                             default = my_dir+'mob_w160_h160.trt')
  10.     parser.add_argument("-s", "--seed", type=int, default=42,
  11.                         help="Random seed for reproducibility.")
  12.     args = parser.parse_args()
  13.     
  14.     ...
  15.     
  16.     # Generate random inputs based on profile shapes
  17.     #host_inputs = get_random_inputs(engine, context, input_binding_idxs, seed=args.seed)
  18.     
  19.     # 把 host_inputs 换成需要的输入就可以了（以单batch为例，从test.jpg中读取数据）。
  20.     # host_inputs 是一个list，有几个输入就有几个元素，元素type是np.array,shape是[N,3,160,160]。
  21.     batch_img = []
  22.     img = cv2.imread('test.jpg')
  23.     img = np.float32(img)
  24.     img = cv2.resize(img, (160,160))
  25.     img -= my_mean
  26.     img = img.transpose(2, 0, 1)
  27.     batch_img.append(img)
  28.     host_inputs = [np.array(batch_img)]
  29.     
  30.     ...

注意在 create_execution_context 和 execute_v2 之间不要有非trt 的 CUDA 操作，（比如 初始化一个 EP为CUDA 的onnxruntime），否则执行时会有莫名其妙的错误，比如：

使用同步的execute_v2时，会有错误：（需要设置 context.debug_sync = True，才能看到错误）

[TensorRT] ERROR: safeContext.cpp (184) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception

使用异步的 execute_async 时，会有错误：（异步推断的完整代码可参考https://github.com/RizhaoCai/PyTorch_ONNX_TensorRT）

[TensorRT] ERROR: ../rtSafe/cuda/reformat.cu (740) - Cuda Error in NCHWToNCQHW4: 400 (invalid resource handle)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/article/detail/57688
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号