Deploy Azure Bicep (ARM Templates) to Multiple Environments using Azure DevOps Pipelines

Previously I’ve posted on developing Azure Bicep code using Visual Studio Code and on how to use an Azure DevOps Pipeline to deploy bicep code to Azure. In this post we’re going to go one step further and look at deploying resources defined using Azure Bicep to multiple environments. Our goal is to fully automate this process, so we’re going to leverage Azure DevOps Pipelines to define a build and release process.

Let’s summarise the goals:

• Azure resources to be defined using Azure Bicep
• Build and Release process to be defined in yaml in Azure DevOps Pipelines
• Build process to be triggered whenever the bicep code changes
• Build process should output the compiled ARM template as an artifact
• Release process needs to support multiple environments
• Each environment should use a different resource group
• Release process should gate the release to particular environments based on a mix of branch and approval gates

You might think these goals are too hard to achieve using a single build and release process but the good news is that most of the heavy lifting is already done by Azure DevOps.

Here’s a quick summary of what we needs to setup:

1. Azure Bicep file – this will define the resource(s) that we’re going to deploy to each environment
2. Azure DevOps Environments – this will define the different environments we’re going to deploy to. At this stage this is limited to defining the approvals and checks that will be done before code/resources are deployed to an environment
3. Azure DevOps Variable Groups – this will define variables that are used across all build and deployment steps, as well as variables that are specific to individual environments
4. Azure DevOps Pipeline – this will define the actual build and release process for the bicep code.

Defining Resources Using Azure Bicep

Let’s start by defining a very simple Azure Bicep (services.bicep) that defines a storage account.

/* 
******************  
    Literals
****************** 
*/

// Resource type prefixes
var storageAccountPrefix = 'st'

/* 
******************  
    Parameters
****************** 
*/

// ** General
param applicationName string 
param location string 
param env string 

/* 
******************  
    Resources
****************** 
*/

var appNameEnvLocationSuffix  = '${applicationName}${env}'

// Storage Account

var storageAccountName  = '${storageAccountPrefix}${appNameEnvLocationSuffix}' 

resource attachmentStorage 'Microsoft.Storage/storageAccounts@2019-06-01' = {
    name: storageAccountName
    location: location
    sku: {
        name: 'Standard_LRS'
        tier: 'Standard'
    }
    kind: 'StorageV2'
    properties: {
        accessTier: 'Hot'
        minimumTlsVersion: 'TLS1_2'
        supportsHttpsTrafficOnly: true
        allowBlobPublicAccess: true
        networkAcls: {
            bypass: 'AzureServices'
            defaultAction: 'Allow'
            ipRules: []
        }
    }
}

A couple of things to note about this bicep file

• It defines a constant, storageAccountPrefix, that is used to define the full name of the generated resource. This prefix comes from the list of recommended resource-type prefixes that the Microsoft documentation lists
• There are three parameters defined: applicationName, location and env. Location is used to define the region that the resource will be created (alternatively you could use resourceGroup().location to use the same location as the resource group where this resource is being created). The applicationName and env parameters are combined with the storageAccountPrefix to define a unique name for the resource being created.
• All three parameters are required, meaning that they will need to be supplied when deploying the generated ARM template. To simplify the process you may want to define default values for applicationName and location. It’s important that you supply the env value during deployment to ensure the resources in each resource group are unique across your subscription.

Setting Up Multiple Environments

Next we’ll define the different environments in Azure DevOps. We’ll keep things relatively simple and define three environments:

1. Development – from the develop branch
2. Testing – from the release branch
3. Production – from the release branch but requiring approval

To create these environments, click on the Environments node under Pipelines from the navigation tree on the left side of the Azure DevOps portal for the project. Click the New environment button and enter a Name and Description for the each environment.

Branch Control

For each environment we need to limit deployments so that only code from the appropriate branch can be deployed to the environment. To setup a branch control check for an environment you first need to open the environment by selecting it from the list of Environments. From the dropdown menu in the top right corner, select Approvals and checks. Next, click the Add check (+) button. Select Branch control and then click the Next button.

In the Branch control dialog we need to supply the name of the branch that we want to limit deployments from. For example for the Development environment we would restrict the Allowed branches to the develop branch (specified here as refs/heads/develop).

Note here that we’ve also included refs/tags/* in the list of Allowed branches. This is required so that we can use the bicep template from the Pipeline Templates repository. I’d love to know if there’s a way to restrict this check to only a specific repository, since adding refs/tags to the Allowed branches will mean that any tagged branch in my repository will also be approved. Let me know in the comments if you know of a workaround for this.

The Testing and Production environments are both going to be restricted to the Release branch. However, we’re also going to enforce a check on the branch to Verify branch protection.

What this means is that the Release branch will be checked to ensure branch policies are in place. For example the Release branch requires at least one reviewer and a linked work item.

Approvals

The only difference between the Testing and Production environments is that deployment to the Production environment requires a manual approval. This makes sense in most cases considering you may need to co-ordinate this with a marketing announcement or a notification to existing customers.

Setting up an approval gate starts similar to adding branch control. Open the environment and go to Approvals and checks. Click the Add check button and select Approvals. In the Approvals dialog, enter the list of users that can approve the deployment, adjust any of the other properties and then click Create.

We typically set a very short approval timeout period to avoid the scenario where multiple deployments get queued up behind each other. If we create another deployment before the first has been approve, we prefer to only have the latter deployment pushed to the Production environment. Of course, this is something your team should discuss and agree on what strategy you want to employ.

Environment Variable Groups

Since we’re going to be deploying the same set of resources to each of the environments, we’re going to need a way to specify build and release variables, some that are common across all stages of the pipeline, and some that are specific to each environment. To make it easy to manage the variables used in the pipeline we’re going to use variable groups which can be defined within the Library tab within Azure DevOps Pipelines.

Common Build Variables

We’ll create a variable group called Common Build Variables and we’ll add two properties, ResourceGroupLocation and AzureSubscriptionConnectionName.

As you can probably deduce, the ResourceGroupLocation will be the region where all the resources will be created. For simplicity this will assumed to be the same across all environments. The AzureSubscriptionConnectionName we’ll come back to but needless to say, it’s the same connection for all stages in the pipeline.

Environment Specific Variables

For each environment we’re going to define a variable group that is named Common.[enviornment]. These variable groups will contain variables that are specific to each environment. In this case, we’re going to define the EnvironmentName, the EnvironmentCode and ResourceGroupName

We’ll show these variable in action shortly but it’s important to remember that any variable that needs to vary, based on which Enviornment it’s being deployed to, should be defined in the appropriate variable group.

Build and Release Process

The build and release process is going to be defined as a yaml pipeline in Azure DevOps.

Service Connections

Before we can jump in to write some yaml, we need to setup a couple of service connections.

• Pipeline-Templates – this is a github service connection so that the build process can download the appropriate pipeline template to assist with the compilation of the bicep file.
• Azure-Subscription – this is a link to the Azure subscription where the resource groups will be created and subsequently the resources created.

I’m not going to step through process of creating these connections, since you can simply follow the prompts provided in the Azure portal. However, it’s important to take note of the name of the service connections.

Build and Deploy Process

Here’s the full build and release pipeline, which we’ll step through in more detail below.

trigger:
  branches:
    include:
    - '*'  # must quote since "*" is a YAML reserved character; we want a string
  paths:
    include:
    - azure/services.bicep
    - pipelines/azure-services.yml
  
resources:
  repositories:
    - repository: pipelinetemplates
      type: github
      name: builttoroam/pipeline_templates
      ref: refs/tags/v0.7.0
      endpoint: Pipeline-Templates
  
pool:
  vmImage: 'windows-latest'
  
variables:
  - group: 'Common Build Variables'
  - name: application_name
    value: inspect
  - name: bicep_filepath
    value: 'azure/services.bicep'
  - name: arm_template_filepath
    value: '$(Pipeline.Workspace)/BicepArtifacts/services.json'
  
stages:
- stage: Compile_Bicep
  pool:
    vmImage: 'windows-latest'

  jobs:
  - job: Bicep
    steps:
      - template: azure/steps/bicep/bicep.yml@pipelinetemplates
        parameters:
          name: Bicep
          bicep_file_path: '$(System.DefaultWorkingDirectory)/$(bicep_filepath)'
          arm_path_variable: ArmFilePath

      - task: CopyFiles@2
        displayName: 'Copying bicep file to artifacts folder'
        inputs:
          contents: '$(Bicep.ArmFilePath)'
          targetFolder: '$(build.artifactStagingDirectory)'
          flattenFolders: true
          overWrite: true

      - task: PublishBuildArtifacts@1
        displayName: 'Publish artifacts'
        inputs:
          pathtoPublish: '$(build.artifactStagingDirectory)' 
          artifactName: 'BicepArtifacts' 
          publishLocation: Container


- template:  templates/deploy-arm.yml
  parameters:
    stage_name: 'Deploy_Development'
    depends_on: 'Compile_Bicep'
    deploy_environment: 'Development'

- template:  templates/deploy-arm.yml
  parameters:
    stage_name: 'Deploy_Testing'
    depends_on: 'Deploy_Development'
    deploy_environment: 'Testing'
  
- template:  templates/deploy-arm.yml
  parameters:
    stage_name: 'Deploy_Production'
    depends_on: 'Deploy_Testing'
    deploy_environment: 'Production'

Trigger – we’ve setup this process to kick off whenever code is committed to the develop environment.

Resources – this process leverages the templates from PipelineTemplates, which requires the definition of a resource pointing to the appropriate tagged release in the pipeline templates github repository.

Stages – there’s one build stage, followed by three release (aka deploy) stages. The steps for the build stage leverage the bicep template from pipeline templates, in order to generate the ARM template. The ARM template is then executed in each of the environments and deployed to the appropriate resource group.

The three deploy stages all used the same template, that we’ll see in a minute, coupled with an environment specific parameter value. For example the first stage passes in a parameter ‘Development’.

Deploy Stage Template

As I mentioned, the three deployment stages are identical, except for some parameter values that are defined for each environment. In this case the template referenced for each stage includes creating the resource group and then planting it.

parameters:
- name: stage_name
  type: string
  default: 'Deploy_ARM_Resources'

- name: depends_on
  type: string
  default: ''

  # deploy_environment - Environment code
- name: deploy_environment
  type: string

stages:
- stage: ${{ parameters.stage_name }}
  dependsOn: ${{ parameters.depends_on }}
  variables:
  - group: 'Common.${{ parameters.deploy_environment }}'
  
  pool:
    vmImage: 'windows-latest'

  jobs:
  - deployment: 'Deploy${{ parameters.stage_name }}'
    displayName: 'Deploy ARM Resources to ${{ parameters.deploy_environment }}' 
    environment: ${{ parameters.deploy_environment }}
    strategy:
      runOnce:
        deploy:
          steps:
          - task: PowerShell@2
            name: ${{ parameters.stage_name }}
            inputs:
              targetType: 'inline'
              workingDirectory: $(Pipeline.Workspace)
              script: |
                  $envParam = '${{ parameters.deploy_environment }}'
                  Write-Host "Deployment deploy environment parameter: $envParam"

                  $envName = '$(EnvironmentName)'
                  Write-Host "Deployment environment name variable: $envName"

          - task: AzureCLI@2
            displayName: 'Create resource group - $(ResourceGroupName)'
            inputs:
              azureSubscription: $(AzureSubscriptionConnectionName)
              scriptType: ps
              scriptLocation: inlineScript
              inlineScript: |
                Write-Host "Creating RG: $(ResourceGroupName)"
                az group create -n $(ResourceGroupName) -l $(ResourceGroupLocation)
                Write-Host "Created RG: $(ResourceGroupName)"

          - task: AzureResourceGroupDeployment@2
            displayName: 'Deploying ARM template to $(ResourceGroupName)'
            inputs:
              azureSubscription: $(AzureSubscriptionConnectionName)
              action: 'Create Or Update Resource Group' 
              resourceGroupName: $(ResourceGroupName)
              location: $(ResourceGroupLocation) 
              templateLocation: 'Linked artifact'
              csmFile: '$(arm_template_filepath)' # Required when  TemplateLocation == Linked Artifact        
              overrideParameters: '-location $(ResourceGroupLocation) -env $(EnvironmentCode) -applicationName $(application_name)'

Throughout this pipeline, there are various variables referenced. The important thing to note is that the variables need to exist for each environment, or are environment independent.

The Common Build Variables group was imported in the build and release process yaml file. The ResourceGroupLocation is the only variable from this group that’s used within this template.

The variable group for each environment is imported within the deploy stage template. The environment name, which is passed in as the deploy_environment parameter, is concatenated with “Common.” in order to import the correct variable group. The imported variables can be referenced the same way as other locally defined variables.

The deployment pipeline template has three steps: The first simply outputs variables so that it’s clear what environment is being built. Then there’s a task for creating the resource group, and then lastly a task for deploying the ARM template.

Running the Pipeline End to End

In the post we’ve defined three different environment and configured Azure DevOps to have different variable groups for each of the environments. Here you can see an execution of the pipeline with the build stage, followed by three deployment stages.

In this case note the Testing deployment failed because the resources were being deployed from the wrong branch (develop instead of release). Unfortunately because this one stage failed, the entire pipeline was marked as failed.