将2条Azure管道转换为一条管道

Question

我有2条天蓝色管道供Devops/MLops使用

1. 第一个训练和注册模型的管道
2. 将模型部署到azure容器实例的第二个管道

我想将第二个管道转换成第一个管道中的一个阶段，但是我有一些变量依赖于第一个管道。

这是第一条管道：

    resources:
  containers:
  - container: mlops
    image: mcr.microsoft.com/mlops/python:latest

pr: none
trigger:
  branches:
    include:
    - master
  paths:
    include:
    - diabetes_regression/
    - ml_service/pipelines/diabetes_regression_build_train_pipeline.py
    - ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r.py
    - ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r_on_dbricks.py

variables:
- template: diabetes_regression-variables-template.yml
- group: devopsforai-aml-vg

pool:
  vmImage: ubuntu-latest

stages:
- stage: 'Model_CI'
  displayName: 'Model CI'
  jobs:
  - job: "Model_CI_Pipeline"
    displayName: "Model CI Pipeline"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - template: code-quality-template.yml
    - task: AzureCLI@1
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          # Invoke the Python building and publishing a training pipeline
          python -m ml_service.pipelines.diabetes_regression_build_train_pipeline
      displayName: 'Publish Azure Machine Learning Pipeline'

- stage: 'Trigger_AML_Pipeline'
  displayName: 'Train and evaluate model'
  condition: succeeded()
  variables:
    BUILD_URI: '$(SYSTEM.COLLECTIONURI)$(SYSTEM.TEAMPROJECT)/_build/results?buildId=$(BUILD.BUILDID)'
  jobs:
  - job: "Get_Pipeline_ID"
    condition: and(succeeded(), eq(coalesce(variables['auto-trigger-training'], 'true'), 'true'))
    displayName: "Get Pipeline ID for execution"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - task: AzureCLI@1
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          python -m ml_service.pipelines.run_train_pipeline --output_pipeline_id_file "pipeline_id.txt" --skip_train_execution
          # Set AMLPIPELINEID variable for next AML Pipeline task in next job
          AMLPIPELINEID="$(cat pipeline_id.txt)"
          echo "##vso[task.setvariable variable=AMLPIPELINEID;isOutput=true]$AMLPIPELINEID"
      name: 'getpipelineid'
      displayName: 'Get Pipeline ID'
  - job: "Run_ML_Pipeline"
    dependsOn: "Get_Pipeline_ID"
    displayName: "Trigger ML Training Pipeline"
    timeoutInMinutes: 0
    pool: server
    variables:
      AMLPIPELINE_ID: $[ dependencies.Get_Pipeline_ID.outputs['getpipelineid.AMLPIPELINEID'] ]
    steps:
    - task: ms-air-aiagility.vss-services-azureml.azureml-restApi-task.MLPublishedPipelineRestAPITask@0
      displayName: 'Invoke ML pipeline'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        PipelineId: '$(AMLPIPELINE_ID)'
        ExperimentName: '$(EXPERIMENT_NAME)'
        PipelineParameters: '"ParameterAssignments": {"model_name": "$(MODEL_NAME)"}, "tags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}, "StepTags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}'
  - job: "Training_Run_Report"
    dependsOn: "Run_ML_Pipeline"
    condition: always()
    displayName: "Publish artifact if new model was registered"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - template: diabetes_regression-publish-model-artifact-template.yml

我的第二个管道(部署到Aci)：

# Continuous Integration (CI) pipeline that orchestrates the deployment of the diabetes_regression model.

# Runtime parameters to select artifacts
parameters:
- name : artifactBuildId
  displayName: Model Train CI Build ID. Default is 'latest'.
  type: string
  default: latest

pr: none

# Trigger this pipeline on model-train pipeline completion
trigger: none
resources:
  containers:
  - container: mlops
    image: mcr.microsoft.com/mlops/python:latest
  pipelines:
  - pipeline: model-train-ci
    source: Model-Train-Register-CI # Name of the triggering pipeline
    trigger:
      branches:
        include:
        - master

variables:
- template: diabetes_regression-variables-template.yml
- group: devopsforai-aml-vg

stages:
- stage: 'Deploy_ACI'
  displayName: 'Deploy to ACI'
  condition: variables['ACI_DEPLOYMENT_NAME']
  jobs:
  - job: "Deploy_ACI"
    displayName: "Deploy to ACI"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - download: none
    - template: diabetes_regression-get-model-id-artifact-template.yml
      parameters:
        projectId: '$(resources.pipeline.model-train-ci.projectID)'
        pipelineId: '$(resources.pipeline.model-train-ci.pipelineID)'
        artifactBuildId: ${{ parameters.artifactBuildId }}
    - task: AzureCLI@1
      displayName: 'Install AzureML CLI'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: 'az extension add --source https://azurecliext.blob.core.windows.net/release/azure_cli_ml-1.27.0-py3-none-any.whl --yes'
    - task: AzureCLI@1
      displayName: "Deploy to ACI (CLI)"
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)/$(SOURCES_DIR_TRAIN)/scoring
        inlineScript: |
          set -e # fail on error
          
          az ml model deploy --name $(ACI_DEPLOYMENT_NAME) --model '$(MODEL_NAME):$(get_model.MODEL_VERSION)' \
          --ic inference_config.yml \
          --dc deployment_config_aci.yml \
          -g $(RESOURCE_GROUP) --workspace-name $(WORKSPACE_NAME) \
          --overwrite -v
    - task: AzureCLI@1
      displayName: 'Smoke test'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          python -m ml_service.util.smoke_test_scoring_service --type ACI --service "$(ACI_DEPLOYMENT_NAME)"

我不知道该如何获得的管道是：

  parameters:
    projectId: '$(resources.pipeline.model-train-ci.projectID)'
    pipelineId: '$(resources.pipeline.model-train-ci.pipelineID)'

因为他们从模型列车-ci管道中获得了项目和管道标识。

我寻找如何从当前正在运行的管道中获取i，但没有得到任何信息。

我试图从作业$AMLPIPELINEID中使用变量Get_pipline_ID，但它是空的

下面是我尝试过的内容以及当前文件的样子：

    # Continuous Integration (CI) pipeline that orchestrates the training, evaluation, and registration of the diabetes_regression model.
parameters:
- name : artifactBuildId
  displayName: Model Train CI Build ID. Default is 'latest'.
  type: string
  default: latest

resources:
  containers:
  - container: mlops
    image: mcr.microsoft.com/mlops/python:latest

pr: none
trigger:
  branches:
    include:
    - master
  paths:
    include:
    - diabetes_regression/
    - ml_service/pipelines/diabetes_regression_build_train_pipeline.py
    - ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r.py
    - ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r_on_dbricks.py

variables:
- template: diabetes_regression-variables-template.yml
- group: devopsforai-aml-vg

pool:
  vmImage: ubuntu-latest

stages:
- stage: 'Model_CI'
  displayName: 'Model CI'
  jobs:
  - job: "Model_CI_Pipeline"
    displayName: "Model CI Pipeline"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - template: code-quality-template.yml
    - task: AzureCLI@1
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          # Invoke the Python building and publishing a training pipeline
          python -m ml_service.pipelines.diabetes_regression_build_train_pipeline
      displayName: 'Publish Azure Machine Learning Pipeline'

- stage: 'Trigger_AML_Pipeline'
  displayName: 'Train and evaluate model'
  condition: succeeded()
  variables:
    BUILD_URI: '$(SYSTEM.COLLECTIONURI)$(SYSTEM.TEAMPROJECT)/_build/results?buildId=$(BUILD.BUILDID)'
  jobs:
  - job: "Get_Pipeline_ID"
    condition: and(succeeded(), eq(coalesce(variables['auto-trigger-training'], 'true'), 'true'))
    displayName: "Get Pipeline ID for execution"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - task: AzureCLI@1
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          python -m ml_service.pipelines.run_train_pipeline --output_pipeline_id_file "pipeline_id.txt" --skip_train_execution
          # Set AMLPIPELINEID variable for next AML Pipeline task in next job
          AMLPIPELINEID="$(cat pipeline_id.txt)"
          echo "##vso[task.setvariable variable=AMLPIPELINEID;isOutput=true]$AMLPIPELINEID"
      name: 'getpipelineid'
      displayName: 'Get Pipeline ID'
  - job: "Run_ML_Pipeline"
    dependsOn: "Get_Pipeline_ID"
    displayName: "Trigger ML Training Pipeline"
    timeoutInMinutes: 0
    pool: server
    variables:
      AMLPIPELINE_ID: $[ dependencies.Get_Pipeline_ID.outputs['getpipelineid.AMLPIPELINEID'] ]
    steps:
    - task: ms-air-aiagility.vss-services-azureml.azureml-restApi-task.MLPublishedPipelineRestAPITask@0
      displayName: 'Invoke ML pipeline'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        PipelineId: '$(AMLPIPELINE_ID)'
        ExperimentName: '$(EXPERIMENT_NAME)'
        PipelineParameters: '"ParameterAssignments": {"model_name": "$(MODEL_NAME)"}, "tags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}, "StepTags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}'
  - job: "Training_Run_Report"
    dependsOn: "Run_ML_Pipeline"
    condition: always()
    displayName: "Publish artifact if new model was registered"
    container: mlops
    timeoutInMinutes: 0
    steps:
    - template: diabetes_regression-publish-model-artifact-template.yml
- stage: 'Deploy_ACI'
  displayName: 'Deploy to ACI'
  condition: variables['ACI_DEPLOYMENT_NAME']
  jobs:
  - job: "Deploy_ACI"
    displayName: "Deploy to ACI"
    container: mlops
    timeoutInMinutes: 0
    variables:
      AMLPIPELINE_ID: $[ dependencies.Get_Pipeline_ID.outputs['getpipelineid.AMLPIPELINEID'] ]
    steps:
    - download: none
    - template: diabetes_regression-get-model-id-artifact-template.yml
      parameters:
        projectId: '$(AMLPIPELINE_ID)'
        pipelineId: '$(AMLPIPELINE_ID)'
        artifactBuildId: ${{ parameters.artifactBuildId }}
    - task: AzureCLI@1
      displayName: 'Install AzureML CLI'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)
        inlineScript: 'az extension add --source https://azurecliext.blob.core.windows.net/release/azure_cli_ml-1.27.0-py3-none-any.whl --yes'
    - task: AzureCLI@1
      displayName: "Deploy to ACI (CLI)"
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        workingDirectory: $(Build.SourcesDirectory)/$(SOURCES_DIR_TRAIN)/scoring
        inlineScript: |
          set -e # fail on error
          
          az ml model deploy --name $(ACI_DEPLOYMENT_NAME) --model '$(MODEL_NAME):$(get_model.MODEL_VERSION)' \
          --ic inference_config.yml \
          --dc deployment_config_aci.yml \
          -g $(RESOURCE_GROUP) --workspace-name $(WORKSPACE_NAME) \
          --overwrite -v
    - task: AzureCLI@1
      displayName: 'Smoke test'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        inlineScript: |
          set -e # fail on error
          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
          python -m ml_service.util.smoke_test_scoring_service --type ACI --service "$(ACI_DEPLOYMENT_NAME)"

在第二管道资源中，有：

resources:
  pipelines:
  - pipeline: model-train-ci
    source: Model-Train-Register-CI 

我现在不能定义它，因为我有一个管道，它不能自我引用

而不是：

'$(resources.pipeline.model-train-ci.pipelineID)'

我需要将模型-train(这是第一个管道名)更改为当前合并的管道。

应该是这样的，但我在网上什么都没找到：

 parameters:
projectId: '$(<current-projectID>)'
pipelineId: '$(<current-pipelineID)'

我真的被困在这里了，如果有人能指引我走向正确的方向，我就不熟悉了。

Answer 1

我想将第二个管道转换成第一个管道中的一个阶段，但是我有一些变量依赖于第一个管道。

就个人而言，你目前所做的是朝着正确的方向：

更改模型-train(这是第一个管道名称)到当前合并的管道

管道资源变量resources.pipeline.<Alias>.projectID以预定义变量的形式提供给所有作业。

因此，在您合并之前，它应该等同于Build.TriggeredBy.ProjectID。

但是，当我们合并了这些管道后，就没有资源管道，只有当前的管道。要获得当前管道的projectID，可以尝试使用System.TeamProjectId。

您可以查看文档系统变量(DevOps服务)以获得更多细节。