The PASS Summit has kicked off again with a tremendous keynote from Ranga. He’s been in the role at Microsoft for a little over a year now, and has really come into his own, as can be seen by the presentation this morning. The changes to the data picture haven’t changed hugely over the past year, although the "Internet of Things" space is increasing quickly.
With that, the speed of growth in data volume has kicked in harder than ever. Being able to collect, process, and analyse the kinds of volume that we’re now facing means that scaling is major feature being discussed. In recent years, this meant looking at Big Data and the ways that this can hook into our existing solutions, and technologies like Hekaton have allowed us to scale up to handle huge numbers of transactions in a scale-up scenario.
This year, though, we see scale-out having a refreshed focus. We’re hearing talk of ‘sharding’ more, and the idea of being able to use multiple databases (including cloud-based ones) to achieve scale on demand – an elasticity that suits business more than ever.
Most of our customers at LobsterPot see changes in the amount of business that’s going on across the year, with some having certain key days requiring orders of magnitude more traffic than on ‘normal’ days. They already scale out their websites, but data is another matter. Databases typically scale UP, not OUT.
My work in the Analytics Platform System / Parallel Data Warehouse space makes me acutely aware of the challenges around scaling out data. When you need to perform joins between tables which have their data in different databases, on different servers, there are problems that need addressing. A lot of it happens behind the scenes through complex data movement techniques, so that it looks like a normal query. This is stuff that is hard to do through clever data
What we’re seeing this morning are some of the ways that Microsoft is providing scale out technology in SQL Server and SQL Database. Considering they now have over a million SQL Database databases in Azure, thinking about how to leverage this technology to enhance on-prem SQL Server databases to provide a new level of hybrid is very interesting.
One of these technologies is Stretched Tables, which we saw this morning. This is about being able to take a table in SQL Server and stretch it into SQL Databases in Azure. This means that the table will be sharded across on-prem and cloud – hot data being stored locally, and more-rarely used data being stored in the cloud. For queries that need to access data that’s in the cloud, data can can be extracted from the cloud tables, pushing predicates down to pull back part of the data, transparently (as far as the user is concerned).
This is not like using linked servers and views, handling inserts with triggers. This is achieving hybrid behind the scenes, giving users a logical layer they can query to access their information whether it’s local or in the cloud.
Until now, I’ve always felt that ‘hybrid’ has been about using some components locally and other components in the cloud. But what we’re seeing now are ways that ‘hybrid’ can mean that we have the core of our database – the tables themselves – are handled in a hybrid way.
Exciting times ahead…
