Microsoft Azure Batch: cloud batch processing

In every work we have those batch processes that require longer than other business tasks: billing, rendering of images, reporting (payroll, calculations, forms) and a long etcetera. They are that kind of tasks at night, carrying time to process or who need a highperformance computing. For this type of scenario is Azure Batch, which allows you to programmatically define a set of resources on Azure which is responsible for performing this kind of work ondemand or scheduled, without the need to configure an HPC cluster, virtual machines, and so on.

Before seeing an example, there are some basics you should know of this service:

  • Pool: it’s the container that will manage the nodes and the jobs.
  • Nodes: they are virtual machines that will process the tasks.
  • Job: It‘s work that will be associated with the tasks to perform.
  • Tasks: can be from a command line with an echo to launch an executable that performs the most complex process. The task doesn’t have to be developed in. NET. In fact, in this example I will show you how to launch a jar task.

To test the service, the first thing you need is to create an Azure Batch account. To do this, go to https://portal.azure.com > New > Compute > Batch Service.

Create new Azure Batch Service
Create new Azure Batch Service

For this example, I’ve created a project with the Azure Batch .NET library, which you will be able to generate a job that process different tasks:

  1. Echo call.
  2. Retrieve the packages installed via Chocolatey on the node.
  3. Know the value of the java_home environment variable.
  4. Launch an executable in Java that performs a series of actions.

As you can see, most of them are too simple, but it is important that you know how to configure this assortment of tasks so that the nodes are able to process them.

The first thing I do is to create an asynchronous method, CreateTasksAsync, which is my entry point to all the service configuration:

        static void Main(string[] args)
        {
            try
            {
                CreateTasksAsync().Wait();
            }
            catch (AggregateException aggregateException)
            {

                foreach (var exception in aggregateException.InnerExceptions)
                {
                    Console.WriteLine(exception.ToString());
                    Console.WriteLine();
                }
            }

            Console.WriteLine("Press return to exit...");
            Console.ReadLine();
        }

As the ideal is that the process is asynchronous, the tasks of configuration, creation and waiting has been divided into the code. CreateTaskAsync is responsible for the configuration:

        private static async Task CreateTasksAsync()
        {

            //1. Connect with you Azure Batch account
            BatchSharedKeyCredentials credentials = new BatchSharedKeyCredentials(
                                                        Settings.Default.BatchURL,
                                                        Settings.Default.AccountName,
                                                        Settings.Default.AccountKey);


            using (var batchClient = await BatchClient.OpenAsync(credentials))
            {
                //2. Add a retry policy
                batchClient.CustomBehaviors.Add(RetryPolicyProvider.LinearRetryProvider(TimeSpan.FromSeconds(10), 3));

                //3. Create a job id
                var jobId = string.Format("MyTasks-{0}", DateTime.Now.ToString("yyyyMMdd-HHmmss"));

                try
                {
                    //4. Submit the job
                    await SubmitJobAsync(batchClient, jobId);

                    //5. Wait for the job to complete
                    await WaitForJobAndPrintOutputAsync(batchClient, jobId);
                }
                finally
                {
                    Console.WriteLine("Press enter to delete the job");
                    Console.ReadLine();

                    if (!string.IsNullOrEmpty(jobId))
                    {
                        Console.WriteLine("Deleting job: {0}", jobId);
                        batchClient.JobOperations.DeleteJob(jobId);
                    }
                }
            }
        }
  1. The first thing is to retrieve values from the account of Azure Batch that you just created. In this example, they are stored in Settings file and you can find them on the portal:

    Batch info - Azure portal
    Batch info – Azure portal
  2. Add a retry policy every 10 seconds with three retries.
  3. Generates an id for the job that will be set up.
  4. Pass the client and the new id to the next step, through the SubmitJobAsync method.
  5. In order to wait for the completion of the process, use a method called WaitForJobAndPrintOutputAsync.

Regarding the last two steps, let’s see what the SubmitJobAsync is:

        private static async Task SubmitJobAsync(BatchClient batchClient, string jobId)
        {
            Console.WriteLine("Creating the job {0}", jobId);

            //1. Create a job
            CloudJob job = batchClient.JobOperations.CreateJob();
            job.Id = jobId;

            //2. Define a start tasks for the nodes

            //Start task: Installing Chocolatey - A Machine Package Manager
            var startTask = new StartTask()
            {
                ResourceFiles = new List<ResourceFile>() {
                    new ResourceFile("https://returngisbatch.blob.core.windows.net/resources/install-choco.bat", "install-choco.bat")
                },
                CommandLine = "install-choco.bat",
                WaitForSuccess = true, //Specifies if other tasks can be scheduled on a VM which has not run the start task
                RunElevated = true
            };

            //3. For this job, ask the Azure Batch service to automatically create a pool of VMs when the job is submitted
            job.PoolInformation = new PoolInformation
            {
                AutoPoolSpecification = new AutoPoolSpecification
                {
                    AutoPoolIdPrefix = "returngis",
                    PoolSpecification = new PoolSpecification
                    {
                        TargetDedicated = 3,
                        OSFamily = "4",
                        VirtualMachineSize = "small",
                        StartTask = startTask
                    },

                    KeepAlive = false,
                    PoolLifetimeOption = PoolLifetimeOption.Job
                }
            };

            //4. Commit job to create it in the service
            await job.CommitAsync();

            //5. Define my tasks
            var commandLine = "cmd /c echo Hello world from the Batch Hello world sample!";
            Console.WriteLine("Task #1 command line: {0}", commandLine);
            await batchClient.JobOperations.AddTaskAsync(jobId, new CloudTask("taskHelloWorld", commandLine));

            var commandChoco = "cmd /c choco list --local-only";
            Console.WriteLine("Task # 2 command line: {0}", commandChoco);
            await batchClient.JobOperations.AddTaskAsync(jobId, new CloudTask("taskchoco", commandChoco));

            var echoJavaHome = "cmd /c echo %java_home%";
            Console.WriteLine("Task # 3 command line: {0}", echoJavaHome);
            await batchClient.JobOperations.AddTaskAsync(jobId, new CloudTask("taskecho", echoJavaHome));

            var jarTask = new CloudTask("jartask", "cmd /c java -jar AzureStorage-0.0.1.jar");
            Console.WriteLine("Task # 4 command line: {0}", "java -jar AzureStorage-0.0.1.jar");
            jarTask.ResourceFiles = new List<ResourceFile>();
            jarTask.ResourceFiles.Add(new ResourceFile("https://returngisbatch.blob.core.windows.net/resources/AzureStorage-0.0.1.jar", "AzureStorage-0.0.1.jar"));
            await batchClient.JobOperations.AddTaskAsync(jobId, jarTask);

        }

This method is the one with the entire burden of creating both the pool and each of the tasks:

  1. Create the job with the id you chose during the configuration part.
  2. Define a start task for the nodes. What does it mean? When I did a review of the tasks we were going to launch, you could see that two of them are making use of Chocolatey and Java, which are not available in the nodes that run tasks by default. Azure Batch allows you to define start tasks to prepare the nodes before the execution of the tasks, in order to set up and install the software needed to run them. For this scenario, I’ve created a .bat with information necessary to Chocolatey and also install Java JDK.
    @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
    choco install jdk8 --force --yes --acceptlicense --verbose
    exit /b 0

    Both of them, install-choco.bat and the jar are stored in an Azure Storage account, with the access in Blob mode.

    Azure Batch - Resource Files - Microsoft Azure Storage Explorer - Public Read Access Blob
    Azure Batch – Resource Files – Microsoft Azure Storage Explorer – Public Read Access Blob
  3. There are different ways to create a pool. In this case you are doing it in such a way that when the job is queued in the service will generate the pool and will be deleted after processing it. You can set that the number of machines (TargetDedicated = 3), the OS family (1 = Windows Server 2008, 2 = Windows Server2008 R2, 3 = Windows Server 2012 and 4 = Windows Server 2012 R2), the size of the machines and the start task that you defined previously.
  4. Commit the job into the service with the configuration that you have just set.
  5. Add each of the tasks that I promised in the first point. The last, which launches an executable Java, differs from the rest, since it is necessary to use a ResourceFile as part of the task and invoke it within the paragraph command line.

Finally is the waiting phase, the WaitForJobAndPrintOutputAsync method:

        private static async Task WaitForJobAndPrintOutputAsync(BatchClient batchClient, string jobId)
        {
            Console.WriteLine("Waiting for all tasks to complete on job: {0} ...", jobId);

            //1. Use a task state monitor to monitor the status of your tasks
            var taskStateMonitor = batchClient.Utilities.CreateTaskStateMonitor();

            List<CloudTask> myTasks = await batchClient.JobOperations.ListTasks(jobId).ToListAsync();

            //2. Wait for all tasks to reach the completed state.
            bool timedOut = await taskStateMonitor.WhenAllAsync(myTasks, TaskState.Completed, TimeSpan.FromMinutes(15));

            if (timedOut)
            {
                throw new TimeoutException("Timed out waiting for tasks.");
            }

            //3. Dump task output
            foreach (var task in myTasks)
            {
                Console.WriteLine("Task {0}", task.Id);

                //4. Read the standard out of the task
                NodeFile standardOutFile = await task.GetNodeFileAsync(Constants.StandardOutFileName);
                var standardOutText = await standardOutFile.ReadAsStringAsync();                
                Console.WriteLine("Standard out: ");
                Console.WriteLine(standardOutText);

                Console.WriteLine();
            }
        }
  1. Creates an object of type TaskStateMonitor that will help you wait for all tasks in the list that pass you as a parameter.
  2. This is the point of the application where will require more patience, since the monitor will be waiting until all tasks are Completed within 15 minutes. Depending on the type of task would have to increase the timeout of waiting time.
  3. If all goes well, you can see the output on the screen, the file stdout.txt for the task, for each processed tasks. There is another file per task, stderr.txt, that contains the errors that have occurred during the execution.

While you are waiting for, the only thing that you can see through the console is the following:

Azure Batch - Waiting for the tasks - Console
Azure Batch – Waiting for the tasks – Console

Or through the Azure portal:

Azure Batch - Info on Azure portal
Azure Batch – Info on Azure portal

There is a tool that I recommend you: Azure Batch Explorer. With it you will be able to see what is happening in the service, in which state the pool is, tasks, nodes, etc.

Batch Explorer - Jobs - Tasks
Batch Explorer – Jobs – Tasks

Also it has a very interesting option: the Heatmap, which allows you to view the status of the nodes: if they are waitingrunning something, starting or if some error occurred.

Azure Batch - Heatmap
Azure Batch – Heatmap

It also allows you to easily manage the creation of a user and the RDP for remote access to any of the nodes.

Batch Explorer - Add User and Connect
Batch Explorer – Add User and Connect

Once the job is completed you can see the result in the console and finish the job.

Azure Batch - All tasks done
Azure Batch – All tasks done
In this link you can see the service limits and quotas. For now It’s only available processing tasks on Windows machinesalthough you can see videos where the use of Linux is in progress.

Here is the example project in my GitHub account.

Happy batching!