Microsoft Azure Data Lake Coming Soon

In the big data world, the latest buzz concept is the “Data Lake”.

In the industry, the concept of a data lake is relatively new. It’s as an enterprise wide repository of every type of data collected in a single place prior to any formal definition of requirements or schema. This allows every type of data to be kept without discrimination regardless of its size, structure, or how fast it is ingested. Organizations can then use Hadoop or advanced analytics to find patterns of the data. Data lakes can also serve as a repository for lower cost data preparation prior to moving curated data into a data warehouse.

Essentially, the concept is a massive (e.g. Petabytes) repository of raw files coming in from various sources such as web analytics, Internet of things, etc.

Azure Data Lake

Microsoft announced a new Azure Data Lake service that builds on top of their existing Hadoop investments to provide cloud based scalability at massive sizes.

The service is in private preview and interested developers can sign up here.

Read More

Top 10 2015 Predictions for Microsoft Azure

Like Office 365, Microsoft Azure has undergone a lot of change in the past year.  Throughout 2014, it seemed like there was a new service being released in either preview or production release every few weeks.   Looking forward to 2015, here is list of predictions for Azure in 2015.

10. Continued Focus on Security and Encryption

One of Microsoft’s key differentiators as an enterprise cloud provider is its focus on security.  Microsoft already has many of the key certifications, public commitments to enterprise grade security, etc. and in 2015 expect the investments to continue. 

9. Microsoft Shrinks the Market Share Gap with Amazon

Amazon is still the dominant cloud supplier and will continue to be in 2015.  However, in 2014, Microsoft’s growth rate outpaced Amazon and I think we’ll see the same in 2015 as Microsoft closes the gap in market share. 

cis q114

Also watch for data center expansion in 2015 as a way to take away market share from Amazon – Microsoft has launched now in China and Australia for example and this geographic expansion will continue as we move into 2015.

8. New Versions of Visual Studio, ASP.NET, .NET Framework

While not technically an Azure feature, the new releases of Visual Studio, ASP.NET and the .NET framework will further integrate, support and provide new ways of building solutions for Microsoft Azure.  The new Azure SDK provides better diagnostics, improved deployment tools for Azure, support for Blob storage, improved HD Insight support and the new version of Visual Studio supports Azure connected services, enterprise SSO, code analysis for Azure, and publishing to Azure integration.

.net_2015.

7. Services in Preview Launch as Production Services

There were many new services launched in 2014 for Microsoft Azure in preview including:

  • Improvements to Azure SQL
  • Better storage and shared file systems
  • A number of big data services (Batch, Machine Learning, Data Factory, Storm, Stream Analytics, etc)
  • Live media streaming
  • NoSQL document based database
  • Azure Search
  • Site recovery through replication

In particular the big data services are a major strategic investment for Microsoft and a key differentiator in the fight with other cloud service providers for market share.  Expect these to be promoted to enterprise class production ready services and a LOT of marketing and promotion around them in 2015.

6. Microsoft Struggles with the Cannibalization of SQL Server

Microsoft’s number one product is SQL Server with a massive $6 billion in revenue for Microsoft.  The SQL Server business grew by 11% in 2014, but Azure growth was more than 100%. 

Microsoft has a fundamental problem with SQL Server as it pivots towards the cloud:

  • The cost of SQL Server running in the cloud through IAAS is quite expensive, especially compared to NOSQL alternatives.  I can run a basic Windows VM for as little as $14 / month, but installing SQL Standard drives that price up to $315 / month or $1,1777 / month for SQL Enterprise.  The cost of a real enterprise class SQL cluster running through Azure on IAAS would cost thousands of dollars per month when you account for high availability and clustering requirements.
  • Microsoft has Azure SQL, which is a PAAS based offering effectively competing with traditional SQL Server running either on premise or in the cloud.  Azure SQL is significantly cheaper than running a full SQL Server license especially for smaller databases. 
  • Microsoft has multiple NOSQL alternatives including Hadoop, Table Storage, and Document DB.  Each of these services can replace a traditional SQL database in certain scenarios.

In the same way that Office 365 has cannibalized on premise implementations of SharePoint, Exchange and Office, Azure will start to cannibalize all those SQL Server databases running on premise.  It will happen slowly because of the nature of migrating any kind of database but over the next 3-5 years expect the shift to become more visible. 

Microsoft’s struggle in 2015 will be how to position its traditional SQL Server business (in particular all of those customers being sold on upgrades to SQL 2014) vs. customers who might start to look at moving to NOSQL alternatives as they move to the cloud.

5. Microsoft Continues to Pivot on Open Source, Linux and Partnerships

Microsoft will continue its pivot on embracing Linux, open source, and other non-Microsoft partnerships such as SalesForce, IBM, Oracle, Dropbox, etc.  Microsoft has been busy in 2014 open sourcing a number of their core platforms including ASP.NET and big chunks of the .NET framework.  They have also moved to a cross-platform model for the .NET framework.  Microsoft has also announced partnerships with SalesForce and Dropbox to support integration between Office 365 and Azure.

Expect this to continue and expand in 2015 as Microsoft moves from a proprietary software company to an open cloud services company.  Microsoft has recognized that their future is hosting EVERYTHING, not just Microsoft designed and engineered products and this will continue to expand in 2015.

4. Price Wars

Microsoft and all the other major cloud providers are in a massive price war which is driving prices down.  If you had purchased basic cloud storage in 2012, you would have paid $0.14 per GB.  In 2013, that price was $0.07 per GB.  At the end of 2014, it’s as low as $0.03 per GB.   Similarly, an A3 VM would have cost $0.48 per hour in 2012 and is now running at $0.32 per hour.

Expect the price drops to continue in 2015 as Microsoft competes with Amazon and others for market share and the economies of hardware, storage, etc. continue to improve over time. 

3. Hadoop and Big Data Go Mainstream

Big data has been a buzzword in the industry for the past several years, but it has been more hype in many cases than practice.  Hadoop became the key platform for big data in 2014 with Microsoft embracing it as its core platform.  Hadoop providers have received massive investments and their revenue is expected to grow in 2015 by 60% year over year.

However, there have also been significant barriers to adoption and CIO’s have been slow to commit to big data platforms.  2014 was a year of many proof of concepts, investigations and hype demos but also lots of concerns, trepidations and adoption challenges for big data technologies.

In the same way that cloud saw massive growth in 2014 as it went from hype to mainstream, I see 2015 as being a pivotal year in the big data story as CIO’s start to move from hype, research and proof of concept stage to mainstream use of these technologies.

In addition, as Microsoft’s new big data services come online such as Batch, Storm, Stream, Machine Learning, etc. this will start to reduce the complexity of engineering big data services and move the market forward to embrace the big data promise with less of the need for data scientists and PHD computer scientists to figure it out. 

2. The Resurgence of PAAS

Microsoft originally championed the cloud as PAAS and quickly had to backtrack into the IAAS business as Amazon took the market.  However, running virtual machines in the cloud in the same way as on premise is not economical in the long run because you still end up owning the costs for maintenance, patching, upgrades, etc.  Microsoft has gone head to head with Amazon and other IAAS vendors and can now compete well in basic VM hosting – however, this is ultimately a race to the bottom as the prices continue to drop. 

The real differentiator for Microsoft in the long run is PAAS – it has the development tools, the APIs, the stacks and the developer community to making running your own virtual machines seem as antiquated as running them on premise.  Some key changes to Azure that have happened in 2014 will make PAAS an increasingly compelling option in 2015:

  • Azure Web Sites as a low cost, high scale option for running public facing web sites is a very attractive option over running your own servers.  Scalability of the Azure Web Site offering is already really good.
  • Azure SQL continues to drop in price while increasing in features and performance.  The new version of Azure SQL (currently in preview) brings almost complete compatibility as well as improved performance to the existing service. 
  • Microsoft’s new big data offerings are all PAAS services and they provide quasi control over how many VMs, instances, and performance is provided to scale as needed while not requiring any management of the underlying infrastructure.  For example, the new Azure Batch provides access to pools of VMs on demand with no need to maintain them – the service manages them as a generic pool including provisioning and deprovisioning.

For most customers, PAAS will provide a more economical, easier to maintain and scalable service than building your own virtual infrastructure using IAAS.  As additional services and finer grain control over PAAS services in terms of dedicated performance units, scalable tiers, etc. the case for IAAS based services is being undermined by easier to use PAAS services.

1. Performance Challenges and Opportunities

One of the key challenges and opportunities we have seen with Azure is performance and scalability.  For example, running SQL Server on IAAS has been a challenge because of poor I/O performance.  We have also seen really good scalability from Azure Web Sites, especially with the Shared Tier. 

The bottom line is that a cloud based VM isn’t the equivalent of a VM running on premise for lots of different reasons.  The I/O performance tends to be poorer, the network latency is harder to control, and if Microsoft is selling performance on Azure SQL as a set of “Dedicated Performance Units” which don’t map well to traditional servers.  Before launching any cloud service, performance testing is a must to ensure that your particular scenario will scale as expected and perform economically.  In some cases, the cost of the scalability is dirt cheap (for example, scaling up an Azure Web Site or load on an Azure SQL database) where in other cases it can be quite costly (for example, scaling up VM’s running SQL Server in IAAS). 

image

Microsoft has been introducing new service tiers that start to address some of these challenges including:

Expect more of these types of improvements as we move into 2015 as customers to start to leverage Azure for performance demanding workloads. 

Read More