Latest Microsoft Big Data Offerings Start to Enable a Comprehensive Big Data Cloud Platform

The release of several new big data services within Azure shows the evolution of Azure from a virtual machine IAAS world to a PAAS big data enablement service.


The following is a summary of what each of these services provide and a link to more information.

Data Input/Output Services

  • Event Hubs: Internet of Things style ingestion of millions of events coming from hundreds of thousands of concurrent clients.
  • Web Sites: Use ASP.NET, PHP, Java technologies to deliver a public facing web site.  Built in support for staging, backup/recovery, high availability and auto-scaling.
  • Notification Hubs: Broadcast push notifications to millions of mobile devices.
  • API Management: Publish APIs to be consumed by internal teams, partners and developers at scale.  Includes provisioning, usage plans, throttling and alerts.

Data Processing Services

  • HD Insight: Hadoop running in the cloud as a service.  Latest version of Hadoop is supported and HD Insight adds additional integration with Excel as a BI platform.
  • Stream Analytics: Part of HD Insight, Stream Analytics provides real time processing of incoming analytics data (e.g. web logs, events, sensor readings, etc.)
  • Machine Learning: Leverage enterprise grade analytics algorithms using a design tool.  Combine algorithms for predictive analytics together in machine learning workflows.
  • Search: General purpose search engine for indexing text based documents.
  • Data Factory: Cloud based ETL engine for data processing across multiple data services including SQL Server, Azure SQL, Azure Blob, Azure Table.  Use multiple languages including C#, Hive and Pig for data processing tasks.
  • Batch: Massively parallel processing for batch jobs.  Harness large pools of CPUs on demand.

Data Storage Services  

  • Table Storage: simple name/value storage but high performance and incredibly cheap. 
  • Document DB: NoSQL document database offering.  Store data as self-describing JSON structures.
  • Blob Storage: storage for unstructured documents, files, etc.  Blob Storage is also used for storing of virtual machine volumes, HD insight volumes, etc. underneath these other services.
  • Azure SQL: traditional SQL Server running as a service. 

The number of services available is growing – some of these services have just been announced while others have been production ready for a while now. Each of these services is PAAS based, e.g. you are buying an automatically scaling service that is entirely managed by Microsoft on a per use basis.  Imagine the number of virtual machines you would need to manage, patch and ensure high availability to replicate these services that you can now scale up in minutes using the Azure portal!

Read More

Microsoft Azure Event Hubs Enable Internet of Things

Microsoft has launched an extension to the existing Microsoft Azure Service Bus that provides a massively scalable data capture solution for capturing data from large volumes of clients such as sensors, devices, etc. 


In business terms, this means that it is designed for commercial scenarios where you have millions of devices, sensors or clients sending readings to a central logging service.  For example, oil pipeline companies are outfitting their pipes with sensors on every inch of their pipeline that generate safety readings as oil flows through the pipeline.  In the next few years, cars will be connected to the Internet and will be sending reliability, service and safety data to provide consumers with warnings, service notifications, etc.  Within media, each set top box will need to send events for every channel we watch, every time we flip the channel, etc. to further maximize the efficiency of advertising.  All of these scenarios require a very robust and high performance collector for all these events being generated.

Event Hubs is a PAAS service that allows for ingestion of events being generated by hundreds of thousands of concurrently connected clients.  You can purchase needed throughput and dynamically scale up your needs as the number of devices increases. 

Read More

If You Can’t Beat em, Join em – Microsoft Office and Dropbox to Work Together

With Dropbox and Microsoft slugging it out in the market over cloud storage, it’s interesting that they have just announced a partnership and integration between the two services.  Like Microsoft’s partnership with, Microsoft seems to have acknowledged that having a connected eco-system of cloud services an important aspect to cloud adoption, even if it means allowing for non-Microsoft partners integrating with Microsoft services.

Dropbox has become home to more than 35 BILLION office files.  Microsoft would love to convert Dropbox users to OneDrive but this isn’t going to happen immediately.  In the meantime, they have announced a partnership that integrates Office and DropBox together in the following ways:

  • Ability to access Dropbox from within Office apps to get to files
  • Edit Office files directly from Dropbox
  • Share new or edited files using Dropbox sharing functionality

Office and Dropbox 1

The functionality will first be included in the next updates to the Office apps for iOS and Android, coming in the next few weeks. The Web integrations between the Dropbox website and Office Online will be available in the first half of 2015. Dropbox will also make its application available on the Windows Phone and Windows tablet platforms in the coming months.

There is no mention of Office running on Windows laptops – it’s not clear whether this will be supported on the traditional desktop versions of Office.

Read More