Creating a containerized build agent for Azure DevOps and Azure DevOps Server

In this article, we’ll look at how to create a containerized build agent for Azure DevOps and Azure DevOps Server. This question came from a customer who wanted to retire their VM-based build agent in favor of something that required less manual patching and maintenance. The build agent needed to be injected into the VNet so it could communicate with the customer’s Azure DevOps Server (it works perfectly with the Azure DevOps service) and deploy to App Service in the VNet. Our build agent needed the ability to build both our customers’ Dotnet and JavaScript projects and then deploy them to Azure. The customer was also using artifact feeds from Azure DevOps for NuGet and npm packages, so the build agent needed access to these feeds.

Try #1: Windows Containers

Because our customers were more familiar with Windows, we decided to use Windows-based container images, specifically Windows 2022 LTSC. I added the Dotnet 8 SDK, PowerShell 7, Az PowerShell module, Node/npm, and AzCopy to the dockerfile of this base image. The first observation is that the size of the Windows 2022 container image started at 3.24 GB and increased up to 8.5 GB when adding various packages.

The next step was to upload this image to Azure Container Registry. As previously mentioned, this took quite some time because the images were so large.

*Note: If you don’t have Docker installed, you can use the “az acr build” task to build a container image from a dockerfile and push the image to Azure Container Registry. This will be explained in a later step.

I chose to host it on Azure App Service because it supports VNet integration and is a very simple hosting platform for my containers. Added the following four environment variables:

• AZP_URL – URL of the Azure DevOps Server and collection of projects (or organization of the Azure DevOps service). yes: https://devops.contoso.com/myProjectCollection
• AZP_POOL – Agent pool name where the build agent will reside.
• AZP_TOKEN – A PAT token that authorizes the build agent to interact with Azure DevOps. This value has full access to the DevOps organization or collection and should be treated as a secret (we recommend storing it in Azure KeyVault).
• AZP_AGENT_NAME – A friendly name that identifies this build agent.

I restarted App Service so that my container could pick up the environment variables and check with Azure DevOps Server to see that my build agent was registered with my agent pool. We created a sample dotnet application and a sample Node application to test the build pipeline. Both applications were built successfully using the new containerized build agent.

success!!! Or so I thought…

I handed the build agent over to the customer, who tried building (larger, more complex) projects using the containerized build agent. The dotnet project was restored and built without issue, but the corresponding Node application exited with the following error during the “npm install” step:Fatal error: CALL_AND_RETRY_LAST allocation failure – JavaScript out of heap memory.” I tried several ways to fix this issue.

• Many articles recommended adjusting Node’s max-old-space-size parameter (i.e. the amount of memory to allocate to old objects on the heap before garbage collection).
• There is also a default memory limit for Windows containers running on Azure App Service associated with an App Service plan SKU. you can Please update this limit WEBSITE_MEMORY_LIMIT_MB Sets the app settings to the limits of the App Service plan.
• Finally, if all else fails, scale your App Service plan to its maximum (11).

Although these steps seemed to lessen the impact of the issue, pipeline execution still occasionally failed with a “JavaScript heap out of memory” exception. Additionally, running the highest SKU available will cost your customers more than they actually want to spend. Back to the drawing board.

Attempt #2: Linux Containers

My next thought was to do this on Linux. The Ubuntu 22.04 image is only 77.86MB in size, a fraction of the size of Windows, and even when installing PowerShell Core, Dotnet 8 SDK, Azure CLI, and Node JS, the overall package size is still only 2GB. , which is also about 25% of the size of a Windows container.

After creating and building the dockerfile and pushing the container image to Azure Container Registry, I attempted to run it in Azure App Service, but found that the container kept failing after startup with an error. This error indicates that the service is not responding to health probes. This observation made some sense. This is because containers do not have a front end, they are agents that listen for signals from Azure DevOps. Luckily, Azure has many container hosting options, so we decided to switch to using Azure Container Instances instead.

Networking and container instances

However, one thing I immediately noticed was that while the test container running against the Azure DevOps service was working fine, the container injected into the network was causing a DNS lookup error while trying to resolve the name of the Azure DevOps server. Typically, Azure services inserted into a VNet inherit the DNS settings of the VNet itself. I checked my DNS settings and found that my VNet has custom DNS servers specified, what is happening in my container???

If you want your container instances to use custom DNS, you must specify those custom DNS servers when the container instances are created. Unfortunately, the portal is somewhat limited in what you can specify during creation, so I wrote a little bicep script to build the container instance. In addition to setting up a custom DNS, you could also create a user-assigned managed identity and assign it to your container instances for secure access to Container Registry.

*As a side note, if you are restricting access to your container registry, you should use user-assigned and system-assigned managed identities here. The reason is a bit of a “chicken/egg” problem. Specifying a user-assigned identity allows you to create and assign access to container instances before they are created. If you use a system-assigned ID, the container instance will attempt to pull the image as part of the deployment process and will fail before the container instance and associated system-assigned ID are created.

Once the container instance was deployed and the build agent code was running, we were able to run the build pipeline successfully. We started out very small, with a single CPU and 1.5GB of RAM, and we were getting the occasional “JavaScript heap out of memory” exception, but increasing the RAM completely resolved this issue.

Microsoft Defender for Containers and Automatic Container Registry Updates

One of the nice things about using build agents as containers is that you can configure Microsoft Defender to scan your Container Registry for vulnerabilities using Defender for Containers. You can also use Microsoft Defender for Servers to detect VM-based build agents, but running them in a containerized manner gives you the opportunity to actually “self-heal” your container images by periodically re-running the dockerfile and getting updated versions of the underlying OS. Provided. And various packages (assuming you are not importing a specific version of the software). This can be accomplished using a few simple az cli commands in your pipeline.

az acr build . -r registryname -t imagename:latest --platform linux --file dockerfile
#Trim the number of manifests to 1 to cleanup Defender for Container results
az acr run --registry registryname --cmd 'acr purge --filter "imagename:.*" --keep 1 --untagged --ago 1d' /dev/null

finish

I’ve placed the scripts and dockerfiles used in this blog in a GitHub repository. here. This includes a dockerfile to build the Linux agent, a bash script to install the Azure DevOps agent code, a (failed) Windows version of the container, custom DNS, and container instance bicep code to deploy the managed container instance. Identity. We hope you find this information helpful and please let us know if you encounter any issues.