docker build生成损坏的pythin库安装

Question

在很多情况下，我的docker build运行都会导致python库安装损坏。有时，这会导致docker构建本身在库安装步骤中失败。但更严重的是，它可能会静悄悄地失败，人们只能在运行应用程序时看到错误。被破坏的python库在不同的版本中是不同的；重新安装相同版本的被破坏的库可以修复它。使用--no-cache似乎更好，但即使使用此构建选项，仍然会发生这种情况。

示例如下：

$ docker build --no-cache --force-rm --rm -f Dockerfile . -t mypy                           

$ docker run --rm -it --entrypoint /bin/bash mypy                   
root:/# python
Python 3.6.6 (default, Jul 31 2018, 22:36:30)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm_api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/statsmodels/__init__.py", line 8, in <module>
    from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File "/usr/local/lib/python3.6/site-packages/statsmodels/tools/__init__.py", line 1, in <module>
    from .tools import add_constant, categorical
  File "/usr/local/lib/python3.6/site-packages/statsmodels/tools/tools.py", line 11, in <module>
    from statsmodels.datasets import webuse
  File "/usr/local/lib/python3.6/site-packages/statsmodels/datasets/__init__.py", line 5, in <module>
    from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File "/usr/local/lib/python3.6/site-packages/statsmodels/datasets/scotland/__init__.py", line 1
    6,�R��A�L
     ^
SyntaxError: invalid syntax

root:/# pip list | grep statsmodels
statsmodels            0.8.0

root:/# pip uninstall statsmodels

root:/# pip install statsmodels==0.8.0
Collecting statsmodels==0.8.0
  Using cached https://files.pythonhosted.org/packages/0d/e9/70d80b48c8c52a8de3ec7cd50e2aa2b1f3cf3f95e42b15fdcb59bd7189f3/statsmodels-0.8.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: pandas in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (0.19.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (1.1.0)
Requirement already satisfied: patsy in /usr/local/lib/python3.6/site-packages (from statsmodels==0.8.0) (0.5.0)
Requirement already satisfied: numpy>=1.7.0 in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (1.12.1)
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (2.7.3)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/site-packages (from pandas->statsmodels==0.8.0) (2018.5)
Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from patsy->statsmodels==0.8.0) (1.11.0)
Installing collected packages: statsmodels
Successfully installed statsmodels-0.8.0
root:/# python
Python 3.6.6 (default, Jul 31 2018, 22:36:30)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels.api as sm_api
>>>
root:/#

Dockerfile非常简单，如下所示：

FROM python:3.6-slim
ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN test -n "$aws_access_key_id" && test -n "$aws_secret_access_key"

RUN mkdir -p ${app_dir} /root/.aws

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

ADD requirements.txt ${app_dir}/requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran procps less nano wget

RUN pip install -r ${app_dir}/requirements.txt

这些无声的错误是相当危险的，因为我们不会使用特定于应用程序的单元测试对docker图像执行广泛的验证。我们希望看到可复制的映像构建。如何防止这些错误？

Answer 1

我没有答案，而是一个建议。

如果依赖项只是构建依赖项而不是运行时依赖项，则使用多阶段构建。

这样，您可以在构建的第一步生成轮子，并在第二步安装轮子。如果创建的轮子被损坏，你有可能无法安装它们，但我不确定。

基于您的Dockerfile的示例：

FROM python:3.6-slim as builder

ADD requirements.txt requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran gcc gfortran g++  

RUN mkdir /wheels && \
    pip install -U pip wheel setuptools && pip wheel -r requirements.txt --wheel-dir=/wheels

FROM python3.6-slim as prod

ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 

WORKDIR $app_dir

如果您不介意较长的构建过程，您可以在prod和builder之间添加一个步骤，该步骤将安装并导入所有模块，以便在导入失败时进行测试：

FROM python:3.6-slim as builder

ADD requirements.txt requirements.txt

RUN apt-get update -qqy && apt-get install -qqy --no-install-recommends \
  build-essential libopenblas-dev gfortran gcc gfortran g++  

RUN mkdir /wheels && \
    pip install -U pip wheel setuptools && pip wheel -r requirements.txt --wheel-dir=/wheels

FROM python3.6-slim as test

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 
# Will fail if one of requirements cannot be imported
RUN python -c 'f = open("requirements.txt") ; modules = [line.strip() for line in f.readlines() if line.strip()] ; f.close() ; list(map(__import__, modules))'

FROM python3.6-slim as prod

ARG app_dir=/root/myapp
ARG aws_access_key_id
ARG aws_secret_access_key

RUN echo "[default]" > /root/.aws/credentials && \
  echo "aws_access_key_id = $aws_access_key_id" >> /root/.aws/credentials && \
  echo "aws_secret_access_key = $aws_secret_access_key" >> /root/.aws/credentials

COPY --from=builder /wheels /tmp/wheels
ADD requirements.txt requirements.txt

RUN pip install --no-cache-dir --no-index --find-links=/tmp/wheels/. -r requirements.txt 

WORKDIR $app_dir

这是一个相当丑陋的提议，但也许它可以帮助你……