I’ve spent a lot of time talking about Azure Media Services, attempting to show step by step how to get started in the media world through the Microsoft solution. It is true that many times we are dealing with new projects but what if I have my media content and I need to add to my Azure account? Perform this process through SDK Azure Media Services itself can create bottlenecks, as it should be creating asset by asset and, in turn, add and upload each file (some of them may oversized) . For this scenario, there is a process called Ingest Assets in Bulk which I’m going to write today, while I enjoy my vacation.
When we perform an assets ingestion massively one thing we should do is to decouple the creation of the assets from uploading files. Through the SDK we can prepare a structure where we create the assets and define the files that contain each before making the uploading. We’ll have a container associated with this process and It will be watched by Azure Media Services, to expect all the files defined in the structure and locate them in the appropriate asset.
using System;
using System.Configuration;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using Microsoft.WindowsAzure.MediaServices.Client;
namespace IngestingAssetsBulk
{
class Program
{
static void Main()
{
//0. Install-Package windowsazure.mediaservices
//1. Variables
var AccountName = ConfigurationManager.AppSettings["AccountName"];
var AccountKey = ConfigurationManager.AppSettings["AccountKey"];
var storageKey = ConfigurationManager.AppSettings["StorageKey"];
var folder = ConfigurationManager.AppSettings["Folder"];
const string ingestManifestName = "MyIngest";
var tempFolder = string.Format(@"{0}temp", folder);
//2. Get AMS context
var context = new CloudMediaContext(AccountName, AccountKey);
//3. Create the Ingest Manifest: Represents a set of assets to be created through bulk ingesting along their associated asset files.
var manifest = context.IngestManifests.Create(ingestManifestName);
//4. Create temporary folder
Directory.CreateDirectory(tempFolder);
//5. Loop directories and create the assets
var directories = Directory.GetDirectories(folder);
foreach (var directory in directories)
{
var files = Directory.GetFiles(directory);
if (!files.Any()) continue;
var directoryName = Path.GetFileName(directory);
if (directoryName == "temp") continue;
Console.WriteLine("Create asset from {0} directory", directoryName);
manifest.IngestManifestAssets.Create(context.Assets.Create(directoryName, AssetCreationOptions.None), files);
foreach (var file in files)
{
File.Copy(file, String.Format(@"{0}{1}", tempFolder, Path.GetFileName(file)));
}
}
//6. AzCopy task
var process = new Process
{
StartInfo =
{
FileName = @"C:Program Files (x86)Microsoft SDKsAzureAzCopyAzCopy.exe",
Arguments = string.Format(@"/Source:{0} /Dest:{1} /DestKey:{2} /Y /V /NC:5", tempFolder, manifest.BlobStorageUriForUpload, storageKey)
}
};
process.Start();
//7. Monitor Bulk Manifest
var bContinue = true;
var consoleWriteLock = new Object();
while (bContinue)
{
//=== We need a new context here because IIngestManifestStatistics is considered an expensive ===//
//=== property and not updated realtime for a context. ===//
context = new CloudMediaContext(AccountName, AccountKey);
var myManifest = context.IngestManifests.Where(m => m.Id == manifest.Id).FirstOrDefault();
if (myManifest != null)
{
lock (consoleWriteLock)
{
Console.WriteLine("nWaiting on all file uploads.");
Console.WriteLine("PendingFilesCount : {0}", myManifest.Statistics.PendingFilesCount);
Console.WriteLine("FinishedFilesCount : {0}", myManifest.Statistics.FinishedFilesCount);
Console.WriteLine("{0}% complete.n", myManifest.Statistics.FinishedFilesCount / (float)(myManifest.Statistics.FinishedFilesCount + myManifest.Statistics.PendingFilesCount) * 100);
if (myManifest.Statistics.PendingFilesCount == 0)
{
Console.WriteLine("Completedn");
bContinue = false;
}
}
if (myManifest.Statistics.PendingFilesCount > 0)
{
Thread.Sleep(60000);
}
}
}
Console.WriteLine("Ingest bulk completed!");
//8. Delete temporary folder
Directory.Delete(tempFolder, true);
//9. Delete manifest
manifest.Delete();
Console.ReadLine();
}
}
}- Install the package called windowsazure.mediaservices from Nuget.
- We will need the Azure Media Services account, the Azure Storage key, the name of the folder where the content are, a name for the manifest and a temporary folder name.
- Get the Azure Media Services context.
- An ingest manifest is created through context.IngestManifests.Create. This action will be responsible for creating a container where we upload all the files corresponding to the assets that we’ll define later.
- In this example we need to create a temporary directory where we will copy the files to upload. The reason is that AzCopy copies the content of the folder including directories in it, but can not flatten the copy to a single folder, it replicates the structure in the container. The ingest manifest requires that all files are in the root.
- Using the path associated with Folder variable, recover all the directories contained in it and through the foreach create the assets using the name of the directories and pass an array with the name of files that will be part of them. On the other hand, copy each of the files in the temporary folder created in the previous step .
- Now we have established the structure of assets and files will upload. In this example, we perform call to the tool AzCopy to make the ingest. It could be a completely independent operation, but I wanted to have everything together in order to show the whole process. In addition to AzCopy, we also have antoher upload clients such as Aspera, which is available through the Azure marketplace.
- Now basically what we do is wait … Thanks to this piece of code can see the status of the process: how many items remain to upload, how many have been processed and also a call to Thread.Sleep with a minute between checks to allow time to perform the uploading.
- Once the process is completed, the temporary folder is deleted.
- Finally, we also eliminate the ingest manifest, leaving only the processed assets in the Azure Media Services account.
Hope this helps,
Happy ingesting!