Github操作CI:使PyCall了解Julia变量

Question

我想要设置一个Github动作ci.yml文件，它安装Python和Julia，然后使用PyCall构建和测试一个包。在这种情况下，PyCall应该使用先前安装的Python版本，而不是它默认的特定于朱莉娅的Python版本，它附带的是miniconda。

到目前为止，我剥离的ci.yml如下所示：

name: CI
on:
  - push
  - pull_request

test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        # python-path: [RUNNER_TOOL_CACHE/Python/*]
        os:
          - macOS-latest
         
        arch:
          - x64
steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

- name: Install dependencies 
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

# Julia Tasks
- uses: julia-actions/setup-julia@v1
- name: Setup Julia
  with:
    version: ${{ matrix.version }}
    arch: ${{ matrix.arch }}

- name: Set ENV variables for  
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

- name: Build Julia pkg
        run: |
          echo $ENV["PYTHON"]
          echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
          echo $ENV["PYCALL_JL_RUNTIME"]
      - uses: julia-actions/julia-buildpkg@v1
              
      - uses: julia-actions/julia-runtest@v1
      - name: Test Julia

它可以正常运行，直到julia-actions/julia-runtest@v1 (到目前为止原因很清楚)，因为PyCall使用它自己的miniconda环境和Python安装，而我试图用PyCall加载到pkg.jl中的Python包安装在我运行CI的系统Python中。

我不想使用内部PyCall -Julia特定的- Python版本，因为我希望能够通过矩阵调用测试不同的-Julia版本。

我能够通过Github操作传递环境变量，如下所示：

- name: Set ENV Variables for  
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

其中产出：

Run echo $ENV["PYTHON"]
  echo $ENV["PYTHON"]
  echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
  echo $ENV["PYCALL_JL_RUNTIME"]
  shell: /bin/bash -e {0}
  env:
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYTHON] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME_PYTHONHOME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
[PYTHON]
[PYCALL_JL_RUNTIME_PYTHONHOME]
[PYCALL_JL_RUNTIME]

这是由PyCall文档建议的

但是，PyCall仍然使用：

┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

[ Info: PyCall is using /Users/runner/.julia/conda/3/bin/python (Python 3.8.5) at /Users/runner/.julia/conda/3/bin/python, libpython = /Users/runner/.julia/conda/3/lib/libpython3.8.dylib
[ Info: /Users/runner/.julia/packages/PyCall/BcTLp/deps/deps.jl has not changed
[ Info: /Users/runner/.julia/prefs/PyCall has not changed

我可以在julia-actions/setup-julia@v1中传递一个startup.jl文件吗

我如何让PyCall知道这些ENV变量？

编辑:将env添加到julia-actions/julia-buildpkg@v1并重新构建PyCall

它给了我一个权限错误：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME_PYTHONHOME: ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME: ${{ env.pythonLocation }}
      - run: |
          julia 
          using Pkg
          Pkg.build("PyCall")

     Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/b6dff5fa725eff4f775f472acd86756d6e31fb02/build.log`
┌ Info: No system-wide Python was found; got the following error:
/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* │ Base.IOError("could not spawn setenv(`/Users/runner/hostedtoolcache/Python/3.7.9/x64 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`,[\"PATH=/Users/runner/hostedtoolcache/julia/nightly/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64:/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/ruby/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/usr/local/go/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Users/runner/Library/Android/sdk/ndk-bundle:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.5.1/x64\", \"GITHUB_RUN_NUMBER=90\", \"PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG\", \"NUNIT3_PATH=/Library/Developer/nunit/3.6.0\", \"NVM_CD_FLAGS=\", \"GITHUB_ACTOR=mkschulze\", \"ANDROID_NDK_18R_PATH=/Users/runner/Library/Android/sdk/ndk/18.1.5063045\", \"ENV[PYCALL_JL_RUNTIME_PYTHONHOME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"USER=runner\", \"PYCALL_JL_RUNTIME_PYTHONHOME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"CONDA=/usr/local/miniconda\", \"VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg\", \"GITHUB_SHA=a08b0b9021e3138eb34ff298f2ccc0498713e627\", \"JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_API_URL=https://api.github.com\", \"GITHUB_ACTIONS=true\", \"ANDROID_SDK_ROOT=/Users/runner/Library/Android/sdk\", \"GITHUB_ENV=/Users/runner/work/_temp/_runner_file_commands/set_env_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"SHELL=/bin/bash\", \"GITHUB_ACTION_PATH=/Users/runner/work/_actions/julia-actions/julia-buildpkg/v1\", \"RUNNER_PERFLOG=/usr/local/opt/runner/perflog\", \"JAVA_HOME_12_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-12.jdk/Contents/Home\", \"EDGEWEBDRIVER=/usr/local/share/edge_driver\", \"TMPDIR=/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/\", \"GITHUB_RUN_ID=390315562\", \"ImageOS=macos1015\", \"NVM_DIR=/Users/runner/.nvm\", \"GITHUB_HEAD_REF=\", \"GITHUB_RETENTION_DAYS=90\", \"GITHUB_SERVER_URL=https://github.com\", \"GITHUB_JOB=test\", \"RUNNER_TOOL_CACHE=/Users/runner/hostedtoolcache\", \"RUNNER_TRACKING_ID=github_bab8cac0-d4b7-441b-9529-36f2faf19750\", \"HOMEBREW_NO_AUTO_UPDATE=1\", \"CHROMEWEBDRIVER=/usr/local/Caskroom/chromedriver/86.0.4240.22\", \"GITHUB_WORKFLOW=CI\", \"GITHUB_ACTION_REPOSITORY=julia-actions/julia-buildpkg\", \"PYCALL_JL_RUNTIME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"pythonLocation=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"HOME=/Users/runner\", \"GITHUB_ACTION=julia-actionsjulia-buildpkg\", \"ENV[PYTHON] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"PYTHONIOENCODING=UTF-8\", \"JAVA_HOME_8_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_EVENT_PATH=/Users/runner/work/_temp/_github_workflow/event.json\", \"XPC_FLAGS=0x0\", \"PYTHON=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"XCODE_10_DEVELOPER_DIR=/Applications/Xcode_10.3.app/Contents/Developer\", \"XPC_SERVICE_NAME=0\", \"XCODE_12_DEVELOPER_DIR=/Applications/Xcode_12.2.app/Contents/Developer\", \"GITHUB_REF=refs/heads/py-cli\", \"XCODE_11_DEVELOPER_DIR=/Applications/Xcode_11.7.app/Contents/Developer\", \"__CF_USER_TEXT_ENCODING=0x1F5:0:0\", \"LC_ALL=en_US.UTF-8\", \"GITHUB_REPOSITORY=Humans-of-Julia/GraknClient.jl\", \"LC_CTYPE=en_US.UTF-8\", \"ImageVersion=20201115.1\", \"LANG=en_US.UTF-8\", \"GITHUB_GRAPHQL_URL=https://api.github.com/graphql\", \"ENV[PYCALL_JL_RUNTIME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"SHLVL=2\", \"LOGNAME=runner\", \"RUNNER_WORKSPACE=/Users/runner/work/GraknClient.jl\", \"SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yxv10u7SRe/Listeners\", \"DOTNET_MULTILEVEL_LOOKUP=0\", \"HOMEBREW_CASK_OPTS=--no-quarantine\", \"GITHUB_BASE_REF=\", \"_=/Users/runner/hostedtoolcache/julia/nightly/x64/bin/julia\", \"GITHUB_ACTION_REF=v1\", \"DOTNET_ROOT=/Users/runner/.dotnet\", \"PIPX_HOME=/usr/local/opt/pipx\", \"***", \"JAVA_HOME_13_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home\", \"GITHUB_WORKSPACE=/Users/runner/work/GraknClient.jl/GraknClient.jl\", \"NUNIT_BASE_PATH=/Library/Developer/nunit\", \"ANDROID_HOME=/Users/runner/Library/Android/sdk\", \"CI=true\", \"POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-macos1015\", \"GECKOWEBDRIVER=/usr/local/opt/geckodriver/bin\", \"RCT_NO_LAUNCH_PACKAGER=1\", \"GITHUB_PATH=/Users/runner/work/_temp/_runner_file_commands/add_path_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"RUNNER_OS=macOS\", \"JAVA_HOME_7_X64=/Library/Java/JavaVirtualMachines/zulu-7.jdk/Contents/Home\", \"JULIA_LOAD_PATH=@:/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/jl_XGe9fC\", \"PIPX_BIN_DIR=/usr/local/opt/pipx_bin\", \"BOOTSTRAP_HASKELL_NONINTERACTIVE=1\", \"JAVA_HOME_14_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-14.jdk/Contents/Home\", \"JAVA_HOME_11_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home\", \"RUNNER_TEMP=/Users/runner/work/_temp\", \"GITHUB_REPOSITORY_OWNER=Humans-of-Julia\", \"GITHUB_EVENT_NAME=push\", \"OPENBLAS_MAIN_FREE=1\", \"ANDROID_NDK_HOME=/Users/runner/Library/Android/sdk/ndk-bundle\", \"AGENT_TOOLSDIRECTORY=/Users/runner/hostedtoolcache\"])
*/
: permission denied (EACCES)", -13)
└ using the Python distribution in the Conda package
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Answer 1

好了，开始运行了。将这个PYTHON设置为确切的Python二进制，并且至少在macOS和Ubuntu x64上可以工作。

macOS / linux解决方案：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"

Windows解决方案：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/python.exe"

所有人的解决办法：

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : python

下面是一个完整的工作ci.yml，它传递macOS和Ubuntu x64：

name: CI
on:
  - push
  - pull_request

jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        os:
          - ubuntu-latest
          - macOS-latest
        arch:
          - x64
    steps:
      - uses: actions/checkout@v2
      - name: Set up  ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}

      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

      - name: Set ENV Variables for  
        run: echo ENV["PYTHON"] = "${{ env.pythonLocation }}/bin/python" >> $GITHUB_ENV

      - name: Install dependencies 
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # Julia Tasks
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
        
      - uses: actions/cache@v2
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-

      
      - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"
            
      - uses: julia-actions/julia-runtest@v1
      - uses: julia-actions/julia-processcoverage@v1
      - uses: codecov/codecov-action@v1
        with:
          file: lcov.info

Answer 2

每次更改ENV["PYTHON"]时，都需要运行Pkg.build("PyCall")。这是朱莉娅真正读取Python配置的唯一时刻，否则它根本不在乎。

如果您想测试一个Python配置矩阵，我仍然建议您为其中的每个配置设置一个迷你Python (尽管它不需要由Julia安装--它可以是Julia外部的迷你Python)。当使用非Conda运行Pkg.build("PyCall")时，可能会出现各种“问题”，最终您可能会对这些安装进行良好的隧道化操作。