mmpycocotools包安装的问题：源码安装出现：“gcc: error : ../common/maksApi.c: 没有那个文件或目录“

问题背景

在配一篇detection论文时，安装README文档进行配置，由于论文代码采用的mmdetection(2.X.X)版本中使用了mmpycocotools这个模块（也就是pycocotools外面再套了一层），我环境当中安装了mmdet 3.0.1运行起来会出现如下的错误：

(oln) jiang@jiang:~/model/object_localization_network/object_localization_network-main$ bash tools/dist_train.sh configs/oln_box/oln_box.py 1
config file is : configs/oln_box/oln_box.py
the number of gpus is : 1
the port is : 29500
Traceback (most recent call last):
  File "tools/train.py", line 15, in <module>
    from mmdet.apis import set_random_seed, train_detector
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/__init__.py", line 1, in <module>
    from .inference import (async_inference_detector, inference_detector,
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/apis/inference.py", line 11, in <module>
    from mmdet.datasets import replace_ImageToTensor
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/__init__.py", line 2, in <module>
    from .coco import CocoDataset
  File "/home/jiang/model/object_localization_network/object_localization_network-main/mmdet/datasets/coco.py", line 21, in <module>
    assert pycocotools.__version__ >= '12.0.2'
AttributeError: module 'pycocotools' has no attribute '__version__'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

报错的源码如下（assert引起的版本校验错误）：

try:
	import pycocotools
	if not hasattr(pycocotools, '__sphinx_mock__'):	# for doc generation
		assert pycocotools.__version__>='12.0.2'
except AssertionError:
	raise AssertionError('Incompatible version of pycocotools is installed.'
									'Run pip uninstall pycocotools first. Then run pip'
									'install mmpycocotools to install open-mmlab forked'
									'pycocotools.')
1
2
3
4
5
6
7
8
9

解决方案：

起初mmdet拥有自己的一个mmpycocotools工具包，功能和pycocotools类似。
最新版的mmdet框架已经遗弃了mmpycocotools这个包，也是采用通用的pycocotools。（emmm，话不多说）
现有的方案有两个：

• 采用最新版的mmdet来运行模型，无需安装mmpycocotools包
• 按照论文的mmdet版本来进行，安装mmpycocotools包

下面来一一说明两种方案的结果。

方案1：不安装mmpycocotools包

将代码中出现pycocotools版本验证的源码进行注释，就不会出现这种问题，但是由于mmpycocotools是在pycocotools上层（外面套了一层），所以mmpycocotools有的方法pycocotools不一定有，就会存在差异。
这个时候再继续按照这个方案进行的话，就需要对代码中的大部分与coco有关的部分进行“大手术”。所以这个方法被我pass掉了。

方案2：安装mmpycocotools包

采用pip进行在线安装，命令如下：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# pip install mmpycocotools > /home/jiang/桌面/a.txt
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-37
      creating build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-37/pycocotools
      running build_ext
      Compiling pycocotools/_mask.pyx because it changed.
      [1/1] Cythonizing pycocotools/_mask.pyx
      /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-a5br_mr2/mmpycocotools_0a6b2c42ae12414093ca041fe7591fa0/pycocotools/_mask.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      building 'pycocotools._mask' extension
      sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
      creating build/common
      creating build/temp.linux-x86_64-cpython-37
      creating build/temp.linux-x86_64-cpython-37/common
      creating build/temp.linux-x86_64-cpython-37/pycocotools
      build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
      gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
      gcc: error: ../common/maskApi.c: 没有那个文件或目录
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mmpycocotools
ERROR: Could not build wheels for mmpycocotools, which is required to install pyproject.toml-based projects
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

就会出现一个 " gcc: error: …/common/maskApi.c: 没有那个文件或目录"

---

---

下载mmpycocotools源码进行源码编译安装：

执行python setup.py build出现如下错误：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
gcc: error: ../common/maskApi.c: 没有那个文件或目录
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# 
1
2
3
4
5
6
7
8
9
10
11

这个错误跟pip在线安装一样

后来我就尝试着在mmpycocotools根目录上一层复制一份common文件夹，本以为能够编译成功，但是接着就出现了如下的错误：

(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# python setup.py build
running build
running build_py
running build_ext
building 'pycocotools._mask' extension
sources:['../common/maskApi.c', 'common/maskApi.c', 'pycocotools/_mask.c']
build:{'build/temp.linux-x86_64-cpython-37/../common/maskApi.o': ('../common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/common/maskApi.o': ('common/maskApi.c', '.c'), 'build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o': ('pycocotools/_mask.c', '.c')}
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
../common/maskApi.c: In function ‘rleDecode’:
../common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |    ^~~
../common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |                                                   ^
../common/maskApi.c: In function ‘rleFrPoly’:
../common/maskApi.c:171:2: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
  171 |  for (j = 0; j < k; j++) x[j] = (int)(scale * xy[j * 2 + 0] + .5); x[k] = x[0];
      |  ^~~

gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c common/maskApi.c -o build/temp.linux-x86_64-cpython-37/common/maskApi.o
common/maskApi.c: In function ‘rleDecode’:
common/maskApi.c:48:4: warning: this ‘for’ clause does not guard... [-Wmisleading-indentation]
   48 |    for (k = 0; k < R[i].cnts[j]; k++) *(M++) = v; v = !v;
      |    ^~~
common/maskApi.c:48:51: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘for’
  235 |   if (m > 2) x += (long)cnts[m - 2]; cnts[m++] = (uint)x;
      |                                      ^~~~
gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c pycocotools/_mask.c -o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o
In file included from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from pycocotools/_mask.c:1111:
/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
gcc -pthread -shared -B /home/jiang/anaconda3/envs/oln/compiler_compat -L/home/jiang/anaconda3/envs/oln/lib -Wl,-rpath=/home/jiang/anaconda3/envs/oln/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-37/../common/maskApi.o build/temp.linux-x86_64-cpython-37/common/maskApi.o build/temp.linux-x86_64-cpython-37/pycocotools/_mask.o -o build/lib.linux-x86_64-cpython-37/pycocotools/_mask.cpython-37m-x86_64-linux-gnu.so
/home/jiang/anaconda3/envs/oln/compiler_compat/ld: build/temp.linux-x86_64-cpython-37/common/maskApi.o: in function `rleArea':
/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/common/maskApi.c:76: multiple definition of `rleArea'; build/temp.linux-x86_64-cpython-build/temp.linux-x86_64-cpython-37/../common/maskApi.o:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3/../common/maskApi.c:225: first defined here
collect2: error: ld returned 1 exit status
error: command '/usr/bin/gcc' failed with exit code 1
(oln) root@jiang:/home/jiang/model/object_localization_network/mmpycocotools-12.0.3# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

大概意思就是c语言一个头文件编译了两次到同一个.o文件当中，在链接的时候会出现错误。

分析：
gcc命令看不懂的可以参考 这儿

gcc -pthread -B /home/jiang/anaconda3/envs/oln/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jiang/anaconda3/envs/oln/lib/python3.7/site-packages/numpy/core/include -Icommon -I/home/jiang/anaconda3/envs/oln/include/python3.7m -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-37/../common/maskApi.o
1

gcc编译命令里面，编译的源代码（c文件）相对路径是 ‘-c …/common/maskApi.c’，代表gcc在读取命令的时候出现绝对路径映射错误。

---

mmpycocotools中setup.py源码如下：

import numpy as np
from setuptools import Extension, setup

# To compile and install locally run "python setup.py build_ext --inplace"
# To install library to Python site-packages run
# "python setup.py build_ext install"
# Note that the original compile flags below are GCC flags unsupported by
# the Visual C++ 2015 build tools.
# They can safely be removed.

ext_modules = [
    Extension(
        'pycocotools._mask',
        sources=['common/maskApi.c', 'pycocotools/_mask.pyx'],
        include_dirs=[np.get_include(), 'common'],
        # extra_compile_args=['-Wno-cpp', '-Wno-unused-function', '-std=c99'],
        extra_compile_args=[],
    )
]

setup(name='mmpycocotools',
      packages=['pycocotools'],
      package_dir={'pycocotools': 'pycocotools'},
      install_requires=[
          'setuptools>=18.0', 'cython>=0.27.3', 'matplotlib>=2.1.0'
      ],
      version='12.0.3',
      ext_modules=ext_modules)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

试过使用pdb工具进行调试，但是不太直观，很难发现问题所在

然后我就使用pycharm对这个模块建立项目，进行debug：
发现如下：

主要的问题出在：sources=[‘common/maskApi.c’, ‘pycocotools/_mask.pyx’],的处理上，记录如下：

1. 问题出现地方：
   

2. 原来的sources为：
   

3. 经过步骤1图中【122】【123】行函数的处理之后，得到的新值如下：
   

4. 中间添加了一个"…/common/maskApi.c"源码到sources当中
   
   最后换了另外的一个conda虚拟环境进行编译就没有这个错误

在重新编译mmpycocotools源码时发现，两者调用的build_extension(self, ext)这个函数不同，编译失败找不到"…/common/maskApi.c"是由于其调用的是Cython里面的build_extension()函数，【Cython是对原生的disutils工具包二次封装了的】，而另外一个能编译通过的环境中没有安装Cython，直接调用disutils的build_extension()函数，调用位置如下：

问题总结

这个问题主要虚拟环境中的Cython包所导致的；
当删除Cython包时就可以正常编译
Tips：
所使用的setuptools版本是61.2.0；
mmpycocotools版本为：12.0.3（好像mmpycocotools已经停止维护了）废了老大劲，唉！！！