The rate of new Azure services designed to address Big Data type problems continues to accelerate. This is due in large part to the continued maturity of Azure as a stable and reliable Public Cloud offering. Indeed, the key to profitability in a business data science effort has much more to do with the ways in which today’s cloud services deliver big data capabilities cost effectively.
Several pieces of the Microsoft Big Data solution are delivered in Azure, allowing us to truly build Big Data solutions at “Cloud-Scale”. In the context of the 3-V’s of Big Data (Volume, Variety and Velocity), “Cloud-Scale” means massive cost-effective storage, schema-less data and “ingestion” of millions of rows of data per second.
1. Azure Blob Storage eliminates the need for your Data Science team to provision a Petabyte or more of redundant storage for your Data Lake.
2. Azure Service Bus and Event Hubs deliver telemetry ingestion from websites, apps, and devices. Intake millions of events per second from a wide variety of sources.
3. Azure Stream Analytics handles the transformation of data “In Motion”. In traditional BI, we aggregate and summarize data “At Rest”. The velocity of Big Data requires technology which aggregates and summarizes data in motion.
4. Azure HD Insight is Microsoft’s Apache Hadoop distribution. Developed by Hortonworks, its 100% compatible with Hadoop toolsets such as Pig, Hive, SQOOL, etc. The Map and Reduce components can be deployed as Microsoft.NET assemblies, written in Microsoft C#
I’ll add another V, because “Cloud-Scale” does not translate well in the land of humans. In order for insights to be actionable (and profitable), our Big Data solution must simplify the information Visually for mere mortals.
5. Power BI is a cloud service that lets you share, collaborate and access your Excel reports anywhere on any device.
The Big Data picture is coming into focus and it does not require a legion of consultants, hardware, or Hadoop experts to achieve. Enterprises which have invested in Microsoft Business Intelligence can move into the Internet of Things era with a mid-size data science project, and grow to “Cloud-Scale”.