Azure Key Vault: Best Practices for Secret Management

Today’s modern applications pay great attention to security, which includes safer access secrets such as API keys, connection strings, certifications, and passwords. Microsoft Azure proposes the Key Vault service to solve this matter.

If you’re wondering why you should use the Key Vault service, you should consider that it comes with features that meet today’s business needs.

• Securely store secrets and certificates
• High-availability and backups
• Integrate with Azure services
• Network security
• Access model

How does Key Vault work?

Azure Key Vault is designed to store secrets and certificates that your applications will need to access private data and protected services.

For example, let’s say we’re building an application that uses a database, instead of hard-coding the connection string, you can store it in a key vault and let your app retrieve it when required.

The application in this demonstrative example will be hosted within a Web App and will use a service principal to get access to the Key Vault, as follows:

If you don’t know what is a service principle, …

Once the connection string secret is retrieved, our Web App can establish secure access to its database.

Deep dive into Key Vault features

As mentioned earlier, Key Vault comes with many interesting features that meet with business’ modern needs

Securely store secrets and certificates

Key Vault is capable of storing and managing different types of data, such as:

• Secrets: you can store and control access to secrets such as tokens, API keys, passwords, or connection strings.
• Certificates: you can manage and deploy public and private TLS/SSL certificates for use with Azure services such as App Service, or Front Door.
• Keys: you can create and manage encryption keys to control your data hosted in Azure services such as storage accounts, or disks.

High-availability and backups

The Key Vault service is designed to maintain availability in disaster scenarios. You don’t have to manually intervene, as it will automatically fail over requests to a paired Azure region.

It protects against accidental, or malicious deletion of your secrets. Thanks to the soft-delete and purge protection features, you can restore not just your secrets but also your Key Vault service even after you delete it.

Soft-delete will keep your deleted secrets in its rubbish for 90 days. During this period, you will be able to restore your secrets, and you can adjust this period.

Purge protection will prevent you from completely deleting your key vault before the soft-delete period completes. You will have to wait.

Although Key Vault is a highly available service, you can still download a copy of your secrets, keys, and certificates. Restore it as simple as downloading a backup.

Integration with Azure services

When you deploy services to Azure, in some cases these services will require access to secrets. Microsoft thought of building tight integration to make it easier to combine these technologies.

For example, when we build applications on top of virtual machines, azure functions, or web apps, they will be able to access secrets in Key Vault natively through a service principle or a managed identity.

Other kinds of Azure services like Front Door can use Key Vault to store SSL certificates, or Bastion can access VM password secrets within a Key Vault.

Network security

By default, authorized applications and users can access Key Vault through the internet. In addition to this, it’s possible to limit this access to select virtual networks and IP addresses. You can also disable public access entirely.

When you have public access disabled, the only option to access Key Vault is by using Private Endpoints.

It’s like having a virtual internal interface within a private virtual network in Azure that allows only traffic going through it to access the Key Vault service.

Access model

If you’re wondering how your app will retrieve secrets, you should know there are two ways to access the Key Vault service.

Using Azure role-based access control

When you invite users to access Azure resources, you grant them RBAC roles. It’s an authorization system that provides access management at a specific scope level.

Using vault access policy

On the other hand, this kind of policy determines what operations can be performed on secrets, keys, and certificates for a given security principal, namely a user or an application.

Secret management best practices

Solid software design requires always following best practices to eliminate technical debt, security vulnerabilities, and bugs. That is valid for every technology you can use, including Microsoft Azure services such as the Key Vault.

Here’s a list of best practices that will save your life

Grant access for a limited scope

If you never heard about the “Zero-Trust” model in the universe of IT systems security, it’s an approach to the idea of “never trust, always verify”

Assigning the least privileged access to users and applications is a foundational element of the zero-trust model.

If you have an application that will access a database password. Why would you give it access to manage secrets, or why give it access to read other secrets?

Least privilege is not only limited to what actions can be performed, it’s possible to define a time range where your application or user can perform these actions.

Periodically rotate secrets

Periodic secret rotation is replacing secrets based on a calendar, for example, replacing your email account’s password every 3 months.

This is one of the security best practices, it helps to avoid the risk of unauthorized access and potential breaches, here’s why:

• Anomalies detection: replacing secrets on a basis will make older versions unusable, which means any hacker actively using them will no longer have access to resources, and this will be detectable when you monitor failed access logs.
• Compliance requirements: regulatory standards and compliance frameworks, such as PCI DSS, HIPAA, and others, require rotating secrets. Periodic rotation helps ensure having secrets that meet with the latest secure methods.
• Limiting insider threats impact: employees hold sensitive information, and when they leave they can abuse it, periodic rotation ensures that they no longer have access to the new secrets.
• Reducing the impact of compromises: if a secret is leaked or stolen, regular rotation of the secret will limit attackers from using the compromised secret because the damage will end as soon as the secret is changed.

One of the ways to implement a secret rotation workflow is by using other Azure automation services such as Event Grid and Azure Functions.

In this example, when you create a secret and set an expiration date for it, you can set the Event Grid service to automatically trigger a Function once a secret is nearly expiring.

Your Azure Function will generate a new secret, update the resource related to this secret, a database for example, and update the secret in Key Vault to the newly generated value.

Secret rotation frequency can vary based on your needs, and on your enterprise policy, what’s important is that you set a regular schedule and automate the process.

Monitor access to your data and systems

You should never consider that your system is 100% secure, as vulnerabilities can exist, and human mistakes can occur.

It’s critical to actively monitor how, when, from where and by who your data is being accessed. As mentioned earlier, the periodic rotation of secrets can reduce some threats, but this won’t eliminate it.

One more significant step to consider in your everyday routine is to actively monitor both your data and systems. Microsoft Azure offers ways to monitor what is happening to your services and data.

For example, using a service like Log Analytics, you’ll be able to trace unauthorized access attempts.

Technical Demonstration

As the given example earlier, we’re creating an App Service web application and an Azure SQL database. The goal is to let the web application securely retrieve the connection string to the database from within a Key Vault.

In this demonstration, we will build a Node.js application, and we will also use the Azure CLI tool to deploy the needed services to Microsoft Azure.

Demonstration Project setup

To get started, create a new directory and run the following commands to initiate a Node.js project and to install the needed dependencies.

mkdir kv-demo
cd kv-demo
npm init
npm install express mssql @azure/identity @azure/keyvault-secrets

Writing the code sample

Create an index.js file and add the following code, it will create a web server and will establish a connection to the Azure SQL database that we will create later. This code will also retrieve the connection string from the Key Vault service.

const KeyVault = require("@azure/keyvault-secrets");
const Identity = require("@azure/identity");
const SQLServer = require('mssql');
const Express = require('express');

async function main() {
  const app = Express();
  const keyVaultName = process.env.KEY_VAULT_NAME;
  const keyVaultUrl = "https://" + keyVaultName + ".vault.azure.net";
  const keyVaultCredentials = new Identity.DefaultAzureCredential();
  const keyVaultClient = new KeyVault.SecretClient(keyVaultUrl, keyVaultCredentials);
  const connectionStringSecret = await keyVaultClient.getSecret("db-connection-string");

  SQLServer.connect(connectionStringSecret.value).then(pool => {
    app.get('/', async (_, res) => {
      try {
        await pool.request().query('SELECT 1;');
        res.send({ status: 'healthy' });
      } catch {
        res.status(500).send({ status: 'unhealthy' });
      }
    });

    app.listen(process.env.PORT);
  });
}

main();

The code above will be deployed to an App Service and depends on Key Vault and Azure SQL service.

This simple web application will create connect to Azure Key Vault to retrieve the database connection string.

After establishing a secure connection to the database, an API server will listen for incoming requests and will send the status of the database connection.

You can think about it as a health check endpoint for a SQL server database.

Login to Azure using the CLI tool

If this is your first time using the Azure CLI tool, follow this guide to install it, and run the following command to login.

az login --device-code

You should expect a link and a random code in the output. Open that link, copy that code and paste it into the link you’ve just opened.

You will see your Terminal magically connected to your Microsoft Azure account.

Deploy Azure services

To start with deploying the needed services to Azure. Run the following commands, some services require a unique name, so make sure to adjust these commands based on your needs.

az group create --name KVDemo --location northeurope
az appservice plan create --resource-group KVDemo --name asp-demo --is-linux --sku B1
az webapp create --resource-group KVDemo --name azurehacks-web-demo --plan asp-demo  --runtime "NODE|18-lts"
az sql server create --resource-group KVDemo --name azurehacks-db-server-demo --admin-user azurehacks --admin-password "bcf!980c8&83843*a72f10e2"
az sql db create --resource-group KVDemo --server azurehacks-db-server-demo --name demonstration-db --edition Free
az keyvault create --resource-group KVDemo --name azurehacks-kv-demo --enable-rbac-authorization

The SQL server services do not allow any incoming requests by default, so it’s important to configure its firewall component to allow requests from the web app we’re going to configure later. To achieve this, use the following command.

az sql server firewall-rule create --resource-group KVDemo --server azurehacks-db-server-demo --name AllowAzureServices --start-ip-address 0.0.0.0 --end-ip-address 0.0.0.0

The web app will need to access the Key Vault service to read the connection string secret as well, we will use a managed identity to deal with this.

Configure access to the Key Vault

az webapp identity assign --resource-group KVDemo --name azurehacks-web-demo
az role assignment create --role "Key Vault Secrets User" --assignee $(az webapp identity show --resource-group KVDemo --name azurehacks-web-demo --query principalId --output tsv) --scope $(az keyvault show --name azurehacks-kv-demo --query id --output tsv)
az role assignment create --role "Key Vault Administrator" --assignee anas@azurehacks.io --scope $(az keyvault show --name azurehacks-kv-demo --query id --output tsv)

The commands above will do the following actions:

• Enable a managed identity for the web app.
• Assign the Key Vault Secrets User role to the managed identity.
• Assign the Key Vault Administrator role to your account.
  Make sure to replace anas@azurehacks.io by your Azure email.

The next step is to retrieve the connection string to the database and save it as a secret within the Key Vault, this command will print your precious secret.

az sql db show-connection-string --server azurehacks-db-server-demo --name demonstration-db --client ado.net

The printed connection string contains <username> and <password>. Make sure to update them to the credentials you used earlier when you run the az sql server create command.

Now it’s time to set this connection string secret in your key vault, run the following command, you should replace YOUR_CONNECTION_STRING with the one generated previously.

az keyvault secret set --name db-connection-string --vault-name azurehacks-kv-demo --value YOUR_CONNECTION_STRING

Deploy the web app

Before deploying the code to the web app, we have to set up the deployment credentials, use this command and make sure to replace the username by a unique one.

az webapp deployment user set --user-name "azurehacks" --password "5IuSNaJEUoCRELCvBcYUAQ=="

We also have to make sure that desired environment variables are configured within the web app, update the value of KEY_VAULT_NAME variable when you run this command.

az webapp config appsettings set --resource-group KVDemo --name azurehacks-web-demo --settings PORT=80
az webapp config appsettings set --resource-group KVDemo --name azurehacks-web-demo --settings KEY_VAULT_NAME=azurehacks-kv-demo

To deploy the code to the web app, we’ll use a git repository hosted by the web app, run these commands to proceed with the deployment operation.

git init
git add .
git commit -m "Initial commit"
git checkout -b master
git remote add azure $(az webapp deployment source config-local-git --resource-group KVDemo --name azurehacks-web-demo --query url --output tsv)
git push azure master

The git push command will ask you for a password, it’s the one you’ve used when you set up the deployment credentials, this step may take some time, so be patient.

To validate your work, open the web app in your browser, you should expect a JSON response with a “healthy” content, you can use the following command to test.

az webapp browse --resource-group KVDemo --name azurehacks-web-demo

Conclusion

Throughout this tutorial, we explored the essential features and best practices of the Key Vault service. But keep in mind, there are more things that can be done using this service.

Unfortunately, a single pager can’t tell everything. So if you have questions, or ideas to share, then don’t hesitate to reach us through the contact page.

You can find in this GitHub repository the code and script used in this technical demonstration.