构建conda环境时版本冲突的原因

Question

在我的项目的CI管道中，我们构建了一个包含conda环境的码头映像。以下是Dockerfile：

FROM debian:buster-slim

RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
        install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
        gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
        --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
        > /etc/apt/sources.list.d/conda.list

WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
        MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
        CONDA_VERSION='4.10.3' && \
        CONDA_DIR=/opt/conda && \
        curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
        /bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
        rm ${MINICONDA_SCRIPT} && \
        $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
        $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
        $CONDA_DIR/bin/conda config --system --remove channels defaults && \
        $CONDA_DIR/bin/conda config --system --add channels main && \
        $CONDA_DIR/bin/conda config --system --add channels conda-forge  && \
        $CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
        $CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
        $CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
        $CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
        $CONDA_DIR/bin/conda create -n py3 python=3.8

RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
     bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
                avro-python3 \
                backports.zoneinfo \
                behave \
                black \
                conda-tree \
                dirty-equals \
                flake8 \
                google-api-python-client \
                google-auth-oauthlib \
                google-cloud-bigquery \
                google-cloud-bigquery-storage \
                google-cloud-bigtable \
                google-cloud-datacatalog \
                google-cloud-firestore \
                google-cloud-logging \
                google-cloud-monitoring \
                google-cloud-pubsub \
                google-cloud-secret-manager \
                google-cloud-storage \
                google-cloud-workflows \
                invoke \
                ipdb \
                jinja2 \
                jsonpath-ng \
                oauth2client \
                pandas \
                pipdeptree \
                pydantic \
                pylint \
                pytest \
                pytest-xdist \
                python-dateutil \
                responses \
                rich \
                rope \
                semantic_version \
                sh \
                sqlfluff \
                tenacity \
                tqdm \
                zenpy \
    && pip install \
                functions-framework \
                google-cloud-bigquery==2.31.0 \
                google-cloud-scheduler"

我想更新conda环境，使其使用python 3.10而不是python 3.8，因此我更改了适当的行：

$CONDA_DIR/bin/conda create -n py3 python=3.10

然后尝试构建码头映像。conda映像的构建失败了，原因如下：

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
  - backports.zoneinfo -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
  - google-cloud-bigquery -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
  - google-cloud-bigquery-storage -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
  - oauth2client -> python[version='2.7.*|3.5.*|3.6.*|3.4.*']
  - python-dateutil -> python[version='>=3.8,<3.9.0a0|>=3.9,<3.10.0a0']
  - tenacity -> python[version='3.4.*|3.8.*|3.7.*']
Your python: python=3.10
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

其中最重要的一行似乎是：

当python出现在右边时，这意味着左侧的东西对您受限的python版本来说是不可用的。

我一直在钻研，试图发现这个问题，并特别关注这一行：

谷歌-云-bigquery -> pythonversion=‘2.7.x.3.3.**>=2.7，<2.8.0a0|>=3.6，<3.7.0a0|>=3.8，<3.9.0a0|>=3.7，<3.0a0’

这个错误表明google对于python3.10是不可用的，但是我在https://anaconda.org/conda-forge/google-cloud-bigquery上检查过，它声明这个包应该可以用于python3.10。

描述google bigquery安装google bigquery以及对熊猫和py箭头集成的额外要求。

支持PythonVersionPython3.6

与Python2.7和3.5兼容的这个库的最后一个版本是=1.28.0。

从表面上看，在我看来，我试图安装的包是用于python3.10的，所以我不明白为什么会出现这种冲突。很明显，我遗漏了一些重要的信息，但我对conda还不太了解。任何建议都很感激。

Answer 1

我通过从conda安装中删除backports.zoneinfo ( python3.9和更高版本中不需要它，因为它被放入了标准库中)和google来解决这个问题。google bigquery无论如何都是由pip安装的(正如我在这个线程上的最初文章中所看到的)，因为conda正在安装我们想要的版本(我不知道为什么)。

我还删除了tqdm，因为我们不再使用它了。生成的Dockerfile如下所示：

FROM debian:buster-slim

RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
        install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
        gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
        --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
        > /etc/apt/sources.list.d/conda.list

WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
        MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
        CONDA_VERSION='4.10.3' && \
        CONDA_DIR=/opt/conda && \
        curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
        /bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
        rm ${MINICONDA_SCRIPT} && \
        $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
        $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
        $CONDA_DIR/bin/conda config --system --remove channels defaults && \
        $CONDA_DIR/bin/conda config --system --add channels main && \
        $CONDA_DIR/bin/conda config --system --add channels conda-forge  && \
        $CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
        $CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
        $CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
        $CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
        $CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
        $CONDA_DIR/bin/conda create -n py3 python=3.10

RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
     bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
                avro-python3 \
                behave \
                black \
                conda-tree \
                dirty-equals \
                flake8 \
                google-api-python-client \
                google-auth-oauthlib \
                google-cloud-bigtable \
                google-cloud-datacatalog \
                google-cloud-firestore \
                google-cloud-logging \
                google-cloud-monitoring \
                google-cloud-pubsub \
                google-cloud-secret-manager \
                google-cloud-storage \
                google-cloud-workflows \
                invoke \
                ipdb \
                jinja2 \
                jsonpath-ng \
                oauth2client \
                pandas \
                pipdeptree \
                pydantic \
                pylint \
                pytest \
                pytest-xdist \
                python-dateutil \
                responses \
                rich \
                rope \
                semantic_version \
                sh \
                sqlfluff \
                tenacity \
                zenpy \
    && pip install \
                functions-framework \
                google-cloud-bigquery \
                google-cloud-scheduler \
                object-config-models"