United Kingdom: +44 (0)208 088 8978

Working with Azure Resource Manager in F#

We're hiring Software Developers

Click here to find out more

By

Posted: December 3, 2017

Categorised: Technology, Uncategorized

Introduction

In case you've not used it before, Azure Resource Manager (ARM) is nowadays the standard way to specify and deploy resources into Azure. Resources (such as Virtual Machines, Storage Accounts, App Services etc.) are deployed into resource groups - logical containers for multiple resources.

Azure also has support for scripting these deployments, in the form of ARM Templates - JSON files that contain the declarative make-up of a resource group. We can observe and report on these deployments through the Azure portal etc., with full tracing of what happened on each deployment.

ARM through CI

It's a common task to try to incorporate your ARM templates into your deployment process - so when deploying your code, you can also deploy your infrastructure at the same time. In fact, some build services (including Microsoft's own VSTS) have built-in tasks for doing exactly this.

However, .NET developers may be familiar with the FAKE build system, which provides a number of useful helper libraries alongside a simple DSL to chain together build steps. Until recently, it wasn't especially easy to deploy an ARM template through .NET, which meant your build process needed to be a two-step process:

  1. Deploy your ARM template through either Powershell, Azure CLI or e.g the VSTS task.
  2. Run your FAKE script.

Mixing CI tooling and FAKE

There are some concerns I have around this approach of mixing multiple tools / languages for a build process:

  1. This immediately means that you increase complexity i.e. two languages / tools for build to learn.
  2. In addition, it means that you may restrict your ability to easily run an entire build locally - very useful for development and debugging purposes.
  3. It's difficult to manage a proper build chain across multiple tools or frameworks etc. - it's better to try to keep as much as possible in one system.
  4. When using the supplied ARM task in VSTS, it does not provide you with key outputs from the template deployment, which you may need to feed into your FAKE script. For example, imagine your ARM template creates a storage account; you might like to access the storage account key afterwards in order to "prime" the storage account with default data.

It would be much nicer to perform the deployment of our infrastructure directly in our FAKE script along with the other required tasks! But before we go much further, let's first look at an example ARM template which we'll later on try to deploy through FAKE.

Sample ARM template

Here's a sample ARM template, sample-rg.json which simply creates a Storage Account into an existing Azure Resource Group:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountName": { "type": "string" }
  },
  "variables": {},
  "resources": [
    {
      "name": "[(parameters('storageAccountName')]",
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2016-01-01",
      "sku": { "name": "Standard_LRS" },
      "kind": "Storage",
      "location": "West Europe",
      "tags": {},
      "properties": {}
    }
  ],
  "outputs": {
    "storageAccountKey": {
      "type": "string",
      "value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
    }
  }
}

If you've never used ARM templates before, there are several points of interest:

  1. Input parameters: You can supply input arguments to the template. Here, we're supplying the name we want the storage account to be called (storageAccountName). We can then reference it later on as [parameters('storageAccountName')].
  2. Resources: We specify that we want to create a resource of kind Storage, using the Standard_LRS SKU in the West Europe data centre. Note that ARM is smart enough that if the resource already exists, it'll apply any changes (where possible) to the existing resource; if it doesn't exist, it'll create it.
  3. Output parameters: After we create the storage account, we want to return back out to the caller the key of the newly-created storage account. In ARM templates, you can actually specify dependencies and use properties across resources e.g. create a storage account and then set an app setting in a web app with the storage account key etc. But here, we want to pass the storage account key back to the caller - in our case, a FAKE script - so that we can use it downstream in the FAKE build process.

Authenticating with Azure

In order to authenticate with Azure, you'll need to create an Azure application account in your Azure AD, and then grant it access to either the entire subscription, or the existing resource group as needed. The Azure application has a Client ID, Client Secret and a Tenant Id. You'll need to get all three of these from the portal.

These credentials will be used later on in the process to allow our .NET code to securely communicate with Azure.

The ARM Helper

Once you've got your credentials, you can create a new F# script, adding a Paket dependency to the ARM Helper:

github CompositionalIT/fshelpers src/FsHelpers/ArmHelper/ArmHelper.fs

As the ARM Helper is a simple file, you can easily work with it without worrying about NuGet packages etc. - it's just a standalone .fs file that'll compile into your project or FAKE scripts etc.

The ARM Helper is actually just a simple wrapper around the relatively new .NET Resource Manager Fluent library (part of the collection of .NET Fluent library packages). This library is itself simply a wrapper around the native REST API for accessing the Resource Manager that Azure exposes.

The wrapper provides a few simple functions. Here's an example of it in use:

open System
open Cit.Helpers.Arm

// Get a handle to the Azure resource manager.
let authContext =
    let azureCredentials =
        { ClientId = Guid "0ab9dbf4-3423-49e4-b40b-8e2118053527"
          ClientSecret = "CLIENT SECRET GOES HERE"
          TenantId = Guid "bb7f7453-15af-4ab0-9d45-cdb4a56293bc"}
    let subscriptionId = Guid "536309b7-121d-4a8a-81ed-9a5a25240086"

    authenticate azureCredentials subscriptionId

let outputs =
    // Kick off a deployment. Block until it's complete, returning any outputs from the deployment.
    deploySimple "test-build" "sample-rg" (File.ReadAllText "sample-rg.json") [ "storageAccountName", "armhelper" ] authContext

Initially, we create our credentials and select a subscription, which we then call authenticate on. Note in this simple example, we've hard-coded the client secrets into the script. In a continuous deployment model, you'd pass these arguments into the script from your build system e.g VSTS, AppVeyor etc.

The next section uses a simple wrapper around the Fluent SDK to create and execute deployment into Azure - in this case, the contents of the sample-rg.json file we saw, earlier into the resource group sample-rg. Notice how we supply any required input parameters using a simple list of string * string pairs. The API will convert these into a format understood by ARM (note: ARM templates also allow optional parameters with defaults embedded in the template). Once the ARM template has been deployed, the API retrieves any outputs from the deployment and returns them as a Map<string, string>:

let storageAccountKey = outputs.TryFind "storageAccountKey"

Since there's no reliance on FAKE or any other library, you can call these function however you want. However, it's very easy to put into a FAKE script, as shown in this gist. 

let mutable azure = None

Target "AzureAuthentication" <| fun _ ->
    tracefn "Authenticating with Azure..."
    let azureCredentials =
        { ClientId = getBuildParam "ClientId" |> Guid
          ClientSecret = getBuildParam "ClientSecret"
          TenantId = getBuildParam "TenantId" |> Guid }
    let subscriptionId = getBuildParam "SubscriptionId" |> Guid
    azure <- Some(authenticate azureCredentials subscriptionId)

Target "CreateResourceGroup" <| fun _ ->
    let resourceGroup = getBuildParam "ResourceGroup"
    let armTemplate = getBuildParam "ArmTemplate"
    tracefn "Deploying template '%s' to resource group '%s'..." armTemplate resourceGroup

    let outputs =
        let deployment =       
            { DeploymentName = getBuildParamOrDefault "BUILD_BUILDNUMBER" (sprintf "LOCAL-%s" (getMachineEnvironment()).MachineName)
              ResourceGroup = Existing resourceGroup
              ArmTemplate = File.ReadAllText armTemplate
              Parameters = [ "storageAccountName", getBuildParam "StorageAccountName" ]
              DeploymentMode = Incremental }

        deployment
        |> deployWithProgress azure.Value
        |> Seq.choose(function
        | DeploymentInProgress (state, operations) -> tracefn "State is %s, completed %d operations." state operations; None
        | DeploymentError (statusCode, message) -> traceError <| sprintf "DEPLOYMENT ERROR: %s - '%s'" statusCode message; None
        | DeploymentCompleted outputs -> Some outputs)
        |> Seq.head

    let storageAccountKey = outputs.TryFind "storageAccountKey"
    tracefn "Done! Storage account key is %A" storageAccountKey

// Build order
"AzureAuthentication"
    ==> "CreateResourceGroup"

// Start build
RunTargetOrDefault "CreateResourceGroup"

Notice in this example we've created a full configuration for deployment (rather than the deploySimple function used earlier) and applied it to deployWithProgress, which provides a sequence of messages with status updates whilst deployment occurs; the final message in the sequence contains the outputs from ARM. What's also nice is that with VSTS, variables implicitly show as environment variables and therefore appear as standard build parameters for FAKE to consume. And as a final proof that this does indeed work, here's a version of that FAKE script running in VSTS (using the optional FAKE Runner extension!).

Future ideas

The ARM Helper is at a very early stage at this point, and there are several features we'd like to add to it.

Improved VSTS integration

Currently there's no good story around taking advantage of VSTS's excellent Azure authentication integration, so we have to explicitly create an Azure Application and supply the Client ID / Secret / Tenant ID. It would be preferable if we could avoid this.

Secondly, this apporach doesn't utilise VSTS support for "secure" variables, whose values, once entered, do not show in logs or any screens. This would be preferable for the client secrets.

Async support?

As one of the most common uses for this will be running within FAKE, the helper is deliberately synchronous. You can of course wrap the API calls into your own async block, but we will probably add a native async version as well.

Remove dependency on Fluent API

We're considered going straight to the REST API in order to remove the dependency on the REST API. This would enable putting a helper such as this directly into FAKE.

ARM Type Provider

We've been experimenting with creating an ARM type provider. This may allow features such as:

  • Strongly typed input and outputs for a given ARM template.
  • Ability to create ARM templates based on strongly-typed sourced ARM schema.
  • Usage of ready-made ARM templates sourced from the Azure ARM gallery on GitHub.

Conclusion

There's definitely room for improvement with the ARM Helper. Nonetheless, this post illustrates how its possible to do away with hybrid Azure CLI / Powershell / FAKE scripts and perform everything directly inside FAKE - providing you with greater control and reuse of your existing skills.

Compositional IT

© 2024 Compositional IT