Accessing Azure Key Vault Secrets in JMeter Tests Running from Azure Pipelines

When running JMeter load tests, you often need to access sensitive information like API keys, passwords, or connection strings. Azure Key Vault is the perfect solution for storing these secrets, but connecting JMeter to Key Vault requires proper authentication, especially when you are intending to run it from self-hosted agents.

I recently had build and run load tests, and my initial intent was to leverage Azure Load Testing service. Azure Load Testing service provides a convenient way to retrieve secrets from Key Vault using a managed identity through the __GetSecret JMeter function. However, there were a couple of reasons why we ended up running JMeter from Azure Pipelines:

• We wanted to have a bit more control over the test environment, including the ability to use a custom JDK version
• Our test plan is saving the results using SimpleDataWriter result collector under specific path and configuration, but Azure Load Testing was not uploading the results from that collector to the storage account
• We wanted to be able to run the tests using self-hosted agents

At the same time, we already had the test plan developed and tested, and we didn’t want to have to make changes to it. It would allow us to use the same test plan for both Azure Load Testing service and Azure Pipelines without any changes to the test plan in the future.

You can read more about Azure Load Testing Service here.

Prerequisites

Below are the prerequisites for integrating Azure Key Vault with JMeter tests running in Azure Pipelines:

Required Azure Resources

• Azure DevOps project
• Azure Key Vault instance
• Azure service connection with Key Vault read permissions

Azure Self-hosted Agent Environment

• Self-hosted Azure Pipelines agent with internet access
• JMeter installation
• Azure CLI installation
• jmeter-plugins-azure-load-testing-stub in JMeter’s lib/ext folder

Test Plan Requirements

• JMeter test plan using the __GetSecret function

How it works

Let’s assume that we have a JMeter test plan that is using the __GetSecret JMeter function to fetch the secrets from the Key Vault. Name of the secret is api-key and in our test plan we are using it like this:

${__GetSecret(api-key)}

if we would be running this from Azure Load Testing service, the secret value would be fetched automatically using the managed identity. However, since we are running it from Azure Pipelines and self-hosted agents, we need to handle the secret fetching manually.

Properties file

To enable JMeter to fetch secrets from Azure Key Vault when running from Azure Pipelines, we need to create a properties file that will configure the authentication and secret retrieval settings. We’ll use the Azure CLI authentication method since it’s already available in our pipeline environment.

Create a file named azure-key-vault.properties with the following content:

# Configure the secret store type and authentication method
azure_load_testing_stub.get_secret.api-key.store_type=keyvault
azure_load_testing_stub.get_secret.api-key.auth_type=azure_cli

# Key Vault configuration
azure_load_testing_stub.get_secret.api-key.vault_uri=https://your-keyvault.vault.azure.net
azure_load_testing_stub.get_secret.api-key.secret_name=api-key

The properties file configuration follows these key points:

1. We specify keyvault as the store type and azure_cli as the authentication method
2. Authentication will use the Azure CLI credentials that are already authenticated in the pipeline
3. The Key Vault URI points to your specific vault instance
4. The secret_name matches the name of your secret in Key Vault

Make sure to replace api-key in the property names with the actual secret name you’re using in your JMeter test plan’s __GetSecret function.

Azure Pipeline Configuration

To run the JMeter tests with Azure Key Vault integration, we need to set up our pipeline to handle the Azure CLI authentication and JMeter execution. Here’s an example pipeline configuration:

trigger:
  - main

pool:
  name: 'your-agent-pool'  # Use your self-hosted agent pool name

steps:
- task: AzureCLI@2
  inputs:
    azureSubscription: 'your-azure-service-connection'  # Use your Azure service connection name
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      # Run JMeter test with properties file
      & jmeter -n `
        -t your-test-plan.jmx `
        -q azure-key-vault.properties `
        -l results.jtl `
        -j jmeter.log      

Let’s break down the key components of this pipeline:

1. We use the AzureCLI@2 task with scriptType: 'pscore' for PowerShell Core support
2. The script runs within the authenticated Azure CLI context, ensuring the JMeter process has access to the Azure credentials
3. The -q azure-key-vault.properties parameter loads our Key Vault configuration
4. Test results are saved to results.jtl and logs to jmeter.log
5. The PowerShell backtick (`) is used for line continuation instead of bash’s backslash

Make sure your Azure service connection has the necessary permissions to read secrets from your Key Vault. You can configure this in the Azure Portal by adding an access policy for the service principal used by your service connection.

It’s crucial to understand that the JMeter test execution must happen within the AzureCLI@2 task context. If you try to run JMeter in a regular powershell or script task, it will not have access to the Azure CLI credentials and will fail.

Using managed identities

Instead of using service principal authentication, you can also use managed identities to authenticate to Azure Key Vault. This approach requires that the the agent has access to the managed identity assigned and that the managed identity has the necessary permissions to access the Key Vault.

Then, in the properties file, you would need to set the auth_type to managed_identity and (optionally, according to the documentation) provide the managed identity client ID.

This setup would allow run the JMeter tests without the need for AzureCLI@2 task in the pipeline as the wrapper for providing the Azure CLI credentials. JMeter would be able to fetch the secrets using the managed identity automatically.

Troubleshooting

When running JMeter tests with Azure Key Vault integration, you might encounter some common issues. The best way to diagnose these problems is by examining the jmeter.log file. You can publish this file as a pipeline artifact using the following task:

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: 'jmeter.log'
    artifact: 'jmeter-logs'

It will allow you to download the log file from the pipeline run and inspect it for any errors or warnings. I helped me a lot when I was setting up the integration for the first time.

Common Issues and Solutions

1. Misconfigured Key Vault URI and/or secret name

2. Access Permission Issues If you see authentication or authorization errors in the logs, verify that the Azure CLI credentials have access to the Key Vault and can read the secret(s) you are trying to access.

Conclusion

Integrating Azure Key Vault with JMeter tests running in Azure Pipelines provides a secure way to manage sensitive data in your load testing scenarios. By using Azure CLI authentication, we’ve simplified the setup process while maintaining security best practices. It allows us to keep our test plans consistent between Azure Load Testing service and Azure Pipelines without any changes to the test plan.

Happy testing!

Cover photo by DALL-E 3