Steve is hosting this month’s T-SQL Tuesday, and asks about what we’ve learned from conference sessions, things which impacted us and how we work. It’s an interesting topic for two reasons – firstly, I enjoy giving conference presentations, and secondly, they’re really not my preferred way of learning.

Don’t get me wrong – conferences are a great way to learn things for a lot of people. I just find that my brain isn’t wired for learning during sessions. If I’m at a conference, I’m probably learning more from conversations, and from hearing what people are saying outside the sessions they’re giving. But this is part of why I enjoy giving presentations. Let me explain, by way of describing some sessions where I definitely did learn something.

I’m not sure which things I learned in conference sessions per se, rather than at conferences in general. I remember an incredibly insightful session at an MVP Summit in 2023 where a few things about Fabric really clicked in ways they hadn’t before. It was so insightful that I grabbed a few others who hadn’t been there (Alex Whittles comes to mind) to join the rest of Bogdan Crivat’s session. But I can’t say it impacted the way I work – just that it set a stronger foundation than what I’d had before. And also, it wasn’t a “traditional” session, with someone talking in front of hundreds of people. There were a dozen or so of us in a fairly small room, and Bogdan was writing on a whiteboard.

Whereas, a session from TechEd Australia 2005 impacted me a lot.

It was my second TechEd event (the first was 1999), but the first after I learned the significance of the technical community. And I’d become a regular presenter myself, just not large events yet. I couldn’t tell you about many of the sessions I attended (I learned a lot more in the expo hall that year), but one session certainly had an impact. It was about security, and the speaker was Microsoft’s Jesper Johansson. LinkedIn tells me he now works for Walmart. https://www.linkedin.com/in/jesperjo

Of course there were things about security that I learned that day, although I couldn’t tell you much about that. The impact that he had on me was more than that. He held the audience’s attention like few technical presenters I’d seen before. I had done user group presentations and spoken in church plenty of times. I understood the need to connect with an audience, but Jesper validated the significance of that for technical sessions. I could remember university lecturers who connected with the room, but mostly that only applied if you were already interested in the subject matter. Jesper was attracting an audience who didn’t even care that much about security. He was leaving a mark. People were learning. Really learning.

Jesper’s session inspired me to aim for a style that would engage people. Technical sessions that would be interesting to people who didn’t necessarily have a background in the material. I wanted to be able to grab the attention of the people in the room (or at least most of them), and hold onto it. To teach their hearts and not just their minds, hopefully so they would leave changed. I try to do the same with my consulting, with the people I encounter in life in general. Of course I’m not always successful at that, but I do ask myself what people need from me whether I’m teaching a large room, a small room, or an individual. Sometimes it’s just a one-on-one conversation, but I’m there for the audience. I’m not trying to perform to be entertaining or interesting to them, I’m genuinely interested in the people who are there and I want to create that connection.

I know that my first TechEd Australia presentation didn’t achieve that. But I hope some of my presentations since then have. I try not to be bound by a slide deck or script (although I’ll have an idea about how the talk will go, and may have even used slides during the preparation), and instead reach into the room and be led by how the people are responding (definitely harder for online presentations, I know). I learn my material well, so that I have the freedom to explain it in ways that work that day, whether it’s through demos, whiteboards, flip-charts, or something else. It means I’m not a typical presenter, and while some audience members don’t like my presentation style, I hope that enough of them do. I’ve had a lot of good feedback from a lot of sessions, and even seen some other presenters adopt similar styles with great success.

I’m keen to see which sessions other people mention in response to Steve’s blog party this month, because I feel like there’s always something to learn about presenting. When I hear people saying that someone blew their mind, I want to know what it was that caused that. I would encourage every conference presenter to learn from the posts that respond to Steve’s question, and to inspire to blow people’s mind, to change their lives, and leave them different. I mean, why wouldn’t you?

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Presentations with impact! appeared first on LobsterPot Solutions.

I’m sure you know I’m a business owner. I’ve had my company since 2008. Sure, there have been ups and downs over that time, but in general it’s been good. We take on clients who want our services, but also try to maintain a level of flexibility with our time so that we can be responsive to whatever happens. Even if I’m spending most days at a particular client, I try to make sure I can respond to other things. Because just like how an employee might get sick, another client might have an urgent issue and I like to be able to respond. I like to be able to pause what I’m doing if necessary. Maybe this is to avoid risk, by not feeling like I have all my eggs in one basket. But maybe it increases the risk, because I’m less inclined to take on long-term full-time contracts.

Before I had the business, things were different. I craved stability, and I’m sure my career suffered for it.

I finished my university time in Melbourne at the very end of 1997. I’d had jobs as a research assistant in each of the three faculties I got degrees from, and it all wound up on New Year’s Eve at the end of 1997. I handed in my last pieces of work and after the long weekend I started working as a consultant, in a job that I had been offered six months earlier. I turned down the offer of a PhD for the stability of a wage.

In 2000, we moved to the UK with our two sons, and that felt risky. When we arrived I did not have a job to step into, but within a week I was deciding which of two I would take. Both were permanent positions – I was still risk-averse. I didn’t want to go contracting. I didn’t back myself enough. Financially, I could’ve made better choices, but I had a young family, and money has never been a goal. In 2002 we moved to Adelaide (where my wife grew up), and I still wanted permanent roles.

It took more than ten years of working in permanent positions (’98 to ’08) for me to take the career risk of starting my own company.

Someone called me an entrepreneur recently. I disagreed with them. I’ve never sought funding from anywhere. I’ve never tried to build up a company to sell it. I’ve always pushed against hiring a salesperson and never had anyone in the company with a sales-related title.

What I have done is lean into the community. I’ve done work for charities for little or no revenue – I even remember charging a client less than the hourly rate I was paying a subcontractor to do the work. And I’ve volunteered my time (and that of my employees) to work among the data community, running events, speaking at conferences, helping community members with what they need, even sometimes people who would consider us competition. And I’ve tried to listen to what God is saying. That’s been hard a lot of the time, but I try.

Is that a career risk? Sure – community work is definitely a risk. It would be better for me to focus on the bottom line and limit how much we do that isn’t tied directly to revenue. Community work is not how people are supposed to run a business. I’m in a group for business leaders and I know that I’m not the same as most of the others there.

This stretches to other parts of my life as well. In the comedy community, I’m hoping to start another open mic room again (I ran one in May/June last year before the venue closed). And I’m not running it to make money – I’m doing it to provide more options for local comedians. People need stage time to improve, and I can help with that. That might help me improve as a comedian too, but that’s not it. There is a need for more open mic space in Adelaide, and I think I can help. Is that a risk? Nah – it’s not taking anything away from my livelihood. But people have called me “commercially naïve” because of it.

Like I wrote further up – I’m not an entrepreneur. I’m good at helping other people solve their stuff, which makes me a good consultant. But running my own company still feels like a career risk to me. At the end of the day though, I like that my company reflects who I am, and I think the bigger risk would be for me to be someone I’m not.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Career risks? appeared first on LobsterPot Solutions.

Now, as much as I’d like to wax lyrical about how English ages and changes over time, and how people nowadays would’ve said “don’t @ me”, I’m going to fit into Pat’s brief slightly better and talk about code ageing – T-SQL code even!

I could write about how new features in T-SQL can help, such as the NULL-aware “IS [NOT] DISTINCT FROM”. If you have code that uses the non-sargable ISNULL function on a column, then that might be something to consider when you next review that code. But I’m going to talk about T-SQL that can age badly because of more sneaky reasons.

There’s a truth that I think many coders don’t really appreciate, and that’s that a T-SQL query is rarely good by itself. Sure, there are queries that have strong potential to be good, but performance depends on a few other things too. And this can mean that a query that was good when it was written might have become bad over time, even though it’s the same query it always was.

Bad plans in the cache

One obvious example is when a bad plan has crept into the cache, and now every time that query runs, the bad plan gets chosen, and performance is now rubbish.

Plans show us how the Query Optimizer has decided to run a query. And I’m saying ‘decided’ to personify it a little. Of course it’s just maths, following a precise algorithm with heuristics to try to make the best choices, but at the of the day, as much as we know that there’s no arbitrary decision-making going on, it can really feel like there’s some tiny person in there, choosing to punish us for every mistake.

One mistake that gives us a bad plan can be simply calling the query with a parameter that is rarely used but causes the query to run a very different way. That plan could get cached, and suddenly every subsequent call for that query (with more typical parameters) uses a less-than-ideal plan.

Imagine the query:

SELECT TOP (100) t.Col1
FROM SomeTable t
WHERE t.Col2 IS NOT DISTINCT FROM @p1
ORDER BY t.Col3 DESC;

There are two obvious ways that this query could run. One would be to apply a TopN Sort on the results of a Seek on Col2, the other would be to perform an ordered (backward) Scan of an index on Col3 with a residual Predicate on Col2. Which one is better? No idea. It depends on how selective the filter is. It also depends on whether particular indexes are available, although right now I’m going to assume there are covering indexes that could produce both plans, one on “(Col3) include (Col1, Col2)”, and one on “(Col2, Col3) include (Col1)”. And it depends on whether the SQL engine knows about the selectivity of different values (ie, the statistics). All this is where the strength of a query is not just about how the query is written, but about indexes, statistics, the cardinality estimate model, and more.

If the table has 100M rows and 10% of them have the value we’re looking for, then the perhaps the best option is the ordered Scan. On average we might need to look through 1000 rows to find the top 100, if those values are mostly around the largest values of Col3.

But what if only 0.001% of the rows have the value we’re looking for. A total of 1000 rows in the whole table. Now we’d be better off using a Seek to find the 1000 rows. But that Sort might need a large memory grant.

And what if there are 10M of them, but they’re all around the smallest values of Col3… oh, the joys of query tuning.

Maybe the value is usually selective, but every so often it’s not – maybe when it runs at 4am it uses a value which isn’t selective, so I’m willing to take the hit on performance, but during the day I want it to use the Seek + Sort plan. And usually this is fine, and there’s a cached plan that does the Seek + Sort that gets reused every time. Terrific.

Then one night, there’s an index rebuild, and the next time the query runs it’s the overnight non-selective parameter. The compiler comes up with a plan which is good for that value, and it’s the ordered Scan plan, which gets cached. Then when users arrive in the morning, that’s the plan that gets used every time, and it’s slow. Agonisingly, it gets often (but not always!) fixed by restarting the SQL service, and let’s stop imagining how much pain organisations can get into when they start down these rabbit holes.

Could we force the Seek? Sure. Should we force the Seek? That’s a tougher question. Maybe we need the overnight query to run better too. Maybe we need two queries. If the parameter value NULL is the only one which is non-selective, then making two queries is easy. We could test the value for NULL and then run a different query. That’d be easy to do. Or maybe it could be fixed by using the OPTION (RECOMPILE) hint, but that’s not exactly free either.

Inside knowledge is great, but perhaps we didn’t have that knowledge when we wrote the query, and the code has aged badly because we didn’t anticipate the future.

There are a bunch of other changes that can cause a perfectly adequate query to age poorly. Some quick examples:

More data than expected

It doesn’t have to be a ‘bad plan’ scenario for a query to be less good than when it was written, because conditions can simply change. The system was so popular than the amount of data grew and now some process takes a lot longer than before. The scan was fine, but now the table is a lot larger, and the values that we thought would be selective aren’t (or vice versa).

Or maybe because the volume of data has increased, the number of physical reads is higher. Perhaps adding RAM is a simple solution, but maybe queries that were fast enough with full scans need to be rewritten.

Better (!) estimates

Maybe the version has been upgraded, and the cardinality estimates are improved. The Optimizer used to think that 20 rows would be the results of a filter, but it was always wildly wrong, with 20,000 rows being more typical. Now with a later version of SQL, the estimate is correct, but because of this, there’s a join that’s using a Hash Match instead of a Nested Loop. Way fewer logical reads, but now it needs a larger memory grant to run the query, and system performance is hurting.

Concurrency

Speaking of memory, maybe the concurrency requirements have increased. Maybe there are 100x as many as expected users trying to use the system at once, and those memory grants are causing a bottleneck. Maybe queries need to be tuned to discourage large memory grants.

Index changes

Maybe partitions have been introduced, and the indexes haven’t been reviewed.

Or maybe a well-meaning DBA or external consultant (ahem!) has been brought in and a new index has been added to reduce the reads. Except that now some other query has started to use that index and although the Query Optimizer has good reason for choosing that index, it’s slower than it was before, or that query we’d tuned to reduce the memory has returned to its old patterns.

So… what should we do?

Naturally what we need is a crystal ball, and write our queries (and our indexes, partitions, statistics, hints, whatever) according to how we expect when things have changed. Typically if you write with scale in mind, you won’t be impacted too much by scale.

But also, understand that there will be some things that you can’t expect so easily. You can’t necessarily expect to anticipate that a particular index will be created, and I’m definitely not suggesting you force the use of a particular index from day 1 of your query, because the change might be about the shape of the data – something that was marvellously selective no longer is for example.

The general feel of T-SQL code might not age as obviously as, say, .Net application code, with techniques that have been superseded over the years. Or a UI that that has aged like a trendy haircut. A lot has changed in the database world, but a T-SQL query now looks roughly the same as it did in the days of SQL Server 2005. But you should totally expect that a query you wrote in a small dev environment might behave quite different in production as the size and shape of the data increases.

So don’t beat yourself up about it – just schedule a code review, or at least a good tuning session. There’s bound to be some things you can fix up, and maybe those fixes won’t age nearly so badly, because you have a better idea about how things will look in the future.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Ageing code appeared first on LobsterPot Solutions.

The name of this post “Really, Louis?” is nothing to do with how Louis thought for a number of years that Arnie Rowland was me. But rather about how people (especially “conscientious types” (that’s only ‘code’ if you want it to be) like so many of us in the data world) frequently take ourselves back through our mistakes, beating ourselves up about it, thinking about how we could’ve avoided it, what we could’ve done better (before, during and after), as if time machines could exist.

I mean, sure, there’s a school of thought that because we learn through our mistakes, we become stronger. But you know, I’d rather learn through near misses. “Oh, that could’ve been bad, let me code up something so that doesn’t happen” is a better experience than having to learn when a disaster has actually happened. And hands up if you know that having a good ‘disaster recovery’ plan only saves you from certain problems, not all problems… thank you, I see that hand. And that one. Yup. All the hands.

There’s an article on Psychology Today called “Coming Out of the Shadows of Our Worst Mistakes“, by Rodney Luster, PhD (he has a blog called “More Than A Feeling” which has a lot of great posts and is worth looking at if you have mental health struggles). One of his points is that even a single mistake can heighten anxiety and depression symptoms. And still, we all make mistakes. All of us. Data professionals included.

Our mistakes shouldn’t define us. We can quote Sting when he sings that “history will teach us nothing”, but we hope that each mistake makes us likely to fall into the same patterns that led us into trouble before. I heard something recently that says our histories might describe us, but they don’t define us. And I want to assure you, reader (and myself), that your history is more about successes than your failures. You wouldn’t be where you are today if you weren’t good at what you’d do. (You would’ve been promoted into management instead.)

As Dr Luster writes, our mistakes can cast shadows over us. Over our mental wellbeing. Over our general mood. We can feel guilt and shame about mistakes, and it can take proper effort to feel redeemed. As a Christian, I know that this is a major aspect of Christianity. The Bible says that we’ve been set free from our guilt and shame, and God loves (and likes) us despite our failings. And yet, we relive our mistakes. Especially the ones involving those areas in which we know we’re meant to do better. Where we’re supposedly ‘experts’.

I’m not going to dwell on my mistakes today. Well, I’m not going to dwell on my mistakes any more today. I’ve already spent time doing that, even though I knew it wasn’t a good idea. I thought at some point in my life I would work out how to stop reliving mistakes, but that’s not a skill I’ve learned yet.

I say to my clients, “Don’t worry – we’ve all done it”, and “You’re not a proper data professional until you’ve stuffed something up”. For some reason, I still remember a question in the Microsoft Certified Master that dealt with corruption, and (without going into any detail, because NDA, and because I really don’t remember the detail) failing to grab a copy of the database files first. That’s not a mistake I’d ever made with a corrupt database in the real world (and there had been plenty I’d handled – I think I had a reputation for fixing corruption, back when it was more common than nowadays), but in the exam – yeah, made the mistake. Maybe it was because it was an exam. Maybe it was because I was sitting by myself (not alongside a customer). That day in the exam, I found myself wanting those files (because of another mistake!), and didn’t have them.

The consequence of my mistake in that exam was basically nothing. I passed the exam and earned my MCM status. Apparently I scraped through with the lowest passing score I could get. But I still passed. That corrupt database question didn’t matter at all. And yet, every so often, it appears in my thoughts. I wonder whether I would make that mistake again, perhaps in an environment where it would matter more. So you can imagine how much I relive the mistakes that actually count for something…

The mistake I’m going to relive for you/Louis is the mistake of reliving my mistakes, as I’ve been describing. I know it sounds meta – but I learned recently that ‘meta’ is the Portuguese word for ‘goal’ (not the football goal, but the thing you’re trying to achieve), so maybe it’s okay that it’s meta. It’s a useful goal to stop reliving your mistakes, even if it is meta.

My advice to you is to learn forgiveness. Forgive everyone else their mistakes. It’s easy to forgive people their mistakes because, by definition, they’re unintentional. So do that, and get used to doing that. Then work out how to forgive yourself for your own.

…and teach me how to do it… 😉

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Mistakes? Really, Louis? ;) appeared first on LobsterPot Solutions.

I’m in a business leaders group called Vistage. In Australia it used be to called TEC (I’m quite happy about the rebranding because “TEC” sounds like “tech”. Plus each group has a chair, and “TEC Chair” sounds a lot like “deckchair” to me). Anyway, when there’s a discussion going on, my Vistage chair sometimes says, “If you knew the answer, what would it be?”, which comes from Sir John Whitmore’s book “Coaching for Performance”.

It’s an annoyingly good question because it shifts the perspective a little. It’s not just asking what the answer is (to which “that’s the problem – I don’t know!” is a common response), it’s asking the person to imagine a situation where they already know. It subtly creates courage in the responder, because they’re asked to imagine that they know and that what they know is correct. I say it’s “annoyingly good” because it’s essentially using mind games to get us to move us past not knowing.

Mike’s asking us to write a note to ourselves from ten years in the future is framed similarly. He wants us to imagine a world in which we’ve got through the next ten years, and we now have the benefit of hindsight. He wants us to look back at 2015 with hindsight, to point out that a thing we were going through at the time would work out okay (or maybe the opposite), and then give ourselves similar advice about the thing we’re in now. Essentially, he’s asking, “If you knew you were going to be successful, what would you do/feel differently?” He wants us to prod ourselves into action about something in the world.

For me, I’ve just turned 51. I remember turning 31 and finding it’s still one of the more jarring age-changes I’ve had. Something about changing from “thirty” to “thirty-something” felt like I had entered a different phase of life, and I’m trying not to feel similar this year now that I’m “fifty-something”. Looking back at being 41, and imagining what it might be like to be 61, it’s interesting.

Ten years ago the world felt like it was changing quickly. And now it feels like it’s changing even quicker. In another ten years, I suspect the rate of change will not have slowed at all, although there is a significant chance that a breaking point might have been reached along the way. Ten years ago we had no idea of the events that were coming. Australia had seen five changes of Prime Minister in the previous ten years, but despite only having two more changes since then, the impact of fires, lockdowns, and other global events have made the world seem vastly different.

My advice to 2015 me is to ride with the big waves and to not sweat the small ones. I don’t think I did let the small waves knock me around, but I know there were times that were harder than I’d like. I’m still here though! Data is still data, whether the tools change, or the trend is towards or away from relational, or whether we’re working on-site or remote. Basic needs of society are still the same, and businesses still need their data to be good, insightful, and well-looked after. New ways of doing things might change the day-to-day activity, and people might be dealing with jarring stresses from the world more than ever, but the core needs of society and data-focused businesses are mostly unchanged.

I suspect I’ll feel similar in 2035 looking back at now. I have no idea what will happen in those ten years. AI is bringing change, bringing new tools (as I wrote in August), but I don’t think it will change what people or businesses need – just how we deliver them. People still need people to be there for them, and businesses still need to trust their data and have insight into it. This year I passed 20 years of running the user group in Adelaide, and in another 10 years I might have just gone past 30 – which will be just about half my life by then. The pandemic caused a lot of change to community, and even within the local user group things feel different now to then. But people do still need community, and I will try to help make that happen, whether that’s in the technical community, the comedy community, or the church community.

So the advice I’m going to give myself from the future is to make the right decisions. Keep doing the things that I should be doing, and find the right balance between pushing myself and going easy on myself. Tonight is a case in point – my back is sore, but it often is so I shouldn’t let that stop me going out to Monday night open-mic comedy. I haven’t been in a long while, but I need to get that back into my weekly schedule.

If current me knows things are going to work out, what will I have been doing to help that? Yeah – doing those things that I know are what I should be doing. I’m pretty sure hindsight will thank me for making those good decisions.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Notes to self… appeared first on LobsterPot Solutions.

Therefore Steve Jones asks us to write about what is making us excited about this new release. Perhaps ‘excitement’ is the wrong word, but certainly Steve wants to know which features will make our lives easier. He lists a few ideas but for me the biggest thing is the probably the new suite of functions about fuzzy string comparison.

One of the things that I feel has been significant in the last 20 years has been tidying data. In some ways, I feel like Data Quality is less of a thing nowadays, especially with the advent of Big Data – one of the fundamental things about structured data is that the quality can be poor sometimes. AI prompts seem to make us bad at typing because it’s just so forgiving. But if you’re trying to know whether someone’s name is Steven or Stephen, then even though AI might not care, Steve probably does. And even though two names might sound the same, there is no bounds to the ‘creativity’ of parents to name their daughters Khloe, which probably gets written down by call centre employees as Chloe or Clooey or something else entirely. Cafes even get confused with my name, let alone anyone with a name outside the top ten from the last hundred years (fun fact: apparently “Robert” was the number one name in the US from 1924-1939, and then top 10 until 1972. I was born two years later, so clearly my parents weren’t following trends – but I was named after a Scottish King, and was probably lucky that there was already a Bruce in my mum’s family). So even though data is becoming more and more UNstructured, we need to care more and more that our Big data is also Good data. High quality. Spelling things the right way.

I’m very sad that Master Data Services is leaving the product with SQL Server 2025. Master Data is a great way of handling lists of things, such as known verified data from trusted sources. But even though there are alternatives (such as keeping an instance of SQL Server 2022 around, or just using an external product like Profisee, which has some great tools for identifying data matches), I’m not thrilled about that aspect of SQL Server 2025.

However, SQL Server 2025 does bring some great options for doing fuzzy string matches, making custom Data Quality options even richer. I’ve spoken about this at some user groups recently (including tomorrow, remotely for TriPASS, and in a few weeks in Melbourne and Sydney for Difinity), and in that session I go much deeper into how I see data matching going. I’ll also write more about these methods in future posts, but it’ll take a few posts, covering quite a few sub-topics.

In the meantime though, have look at the Microsoft documentation on Fuzzy String Matching. You’ll see there are four new functions, covering similarity and distance across two algorithms (Edit Distance and Jaro-Winkler).

We’ve had Fuzzy Lookup and Fuzzy Grouping in SSIS and Power Query (well, Power Query Online) for over twenty years, but they’ve felt like closed boxes. Sure you can put all your data through a Fuzzy Grouping transform, but there weren’t many options for being able to fine-tune things. Now I can find a bunch of candidates for a match, and then apply my own set of logic using these new functions.

Keep your eye out for lengthier, more technical posts in a month or so (after the conferences are done), and for now, take the time to explore where these new functions can fit into your current data matching code. You might not be rolling out SQL Server 2025 for a few months, but you can start using this in Azure SQL already!

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post SQL Server 2025 excitement – fuzzy is coming! appeared first on LobsterPot Solutions.

As Steve writes, there are a bunch of ways to parse strings in SQL. And (which Steve doesn’t write) 2025 is going to change a lot of people’s approaches, with its native regular expression functions. I’ve looked at these, but so far my reaction is more like “Oh finally!”, rather than “Ooh, think of the potential!” – I think these new functions are most useful for when you’ve got a regular expression that you’re using in non-SQL code, and now it’s much easier to handle it within SQL.

I should also point out that the days of new features appearing only-when-a-new-SQL-Server-version-appears feel like they’re well-and-truly over, since Azure SQL gets stuff first. So when I write “2025”, I really mean the year, rather than the version of SQL Server. The feature-set, rather than the product.

But let’s talk about non-regex methods for parsing strings and the patterns that I use. I find that the biggest issue with most string parsing is complexity. Even something as simple as finding the value between the 2nd and 3rd hyphens can be done in different ways with different levels of complexity, and even if it works, maintaining that code can become really hard.

For example, finding the position of the first hyphen might be as simple as using the CHARINDEX function. Finding the second might involve two CHARINDEX functions, and calling SUBSTRING with parameters that have increasingly nested CHARINDEX calls… well, you can see how the complexity quickly builds

Another way to find that ‘between two hyphens’ thing would be to do a STRING_SPLIT call with the enable_ordinal parameter set, and then filter it to the third entry. All well and good, but depending on what else is needed, STRING_SPLIT’s creation of additional rows can make life harder, and if you’re fighting complexity, expanding the number of rows in a dataset and then combining them again can be hard.

As much as we data people like ‘set-based’ solutions, rather than thinking how to transform individual values, string parsing seems to fall into the bracket of okay to deal with this way.

So if we start with a string value in a variable, there are a lot of things that can be done without freaking out the person who will need to read your code later (or “future you” even). It probably doesn’t matter too much if you split the string up in various ways, concatenate parts back together in different orders, STRING_AGG some stuff, and so on. It’s certainly much easier than trying to parse all the values in a table at once in these more complex ways.

I like to separate logic for this kind of thing away into a function, where the logic can be tested and proven and anyone using it can have confidence that it works. And if there’s a bug or an enhancement, it can be fixed in one place.

But not a scalar function, of course. A table-valued function. One that can take a string as a parameter and produce a row with all the different things that could be wanted. And I can guarantee that it will produce a single row.

Now, this table-valued function should be inline, rather than multi-statement. I hope you’re asking why. It’s because I want the Query Optimizer to simplify it all out. I can call the function with APPLY in my FROM clause and then refer to whichever columns I like, and expect the QO not to bother with working out the ones I don’t want. So if I have a function that carves up an address string into various components, does some lookups to get reference values, maybe coming up with a JSON version and an XML version, and so on… if I don’t reference the JSON column in my outer query, my function won’t bother to produce it. A multi-statement will do all the work every time because it’s procedural. Inline will leverage the Query Optimizer to figure out which things it doesn’t need.

APPLY won’t just get used to call the function, but I’ll use it extensively inside the function too. By using APPLY, I can do a bunch of working calculations, making the building blocks of my work, and then combine them in the different ways, while keeping the whole thing as a single query for my inline function. Remember that an inline TVF is a stored correlated subquery. With APPLY, I can layer my logic, making it easier to maintain. If I need to expand to multiple rows, I can fold it back into one in a single APPLY subquery, unpivotting or string_split and then pivotting back into a single row. If I need a new set of return columns added, some more APPLY clauses can keep that new logic separate, so that I can confidently know that I still have only one row of data coming out, and this new logic isn’t going to confuse any of my existing code.

Here’s an example, using one of the ideas that Steve mentioned – pulling dates out of a string, where there are separators that are either a semicolon or full-stop. Notice that my TVF is a series of APPLY statements. Notice also that I’m using OUTER APPLY and TRY_CONVERT so that if I get NULLs, I’m still returning the row. And all the way along, I have my working calculation columns available in case I need them again.

If I needed to make sure I only returned a single row per starter row, I could do that by just combining more into a single APPLY and then aggregating the results. I’ve also added a column to return the string length, which I’ll use in a moment.

As for what happens when I ignore one of the values… well like I said earlier, the work gets cut back. Compare what happens if I’m not interested in the JSON block, only the string length.

Naturally, this can work against a table of values just fine – I’m sure I don’t need to show you what’s going on there.

Don’t misunderstand me – I love the new regex stuff that’s come in this year. But my general pattern for string parsing is unlikely to change – the regex will just be another tool in my arsenal. I’ll still create useful inline TVFs that suit my customers’ individuals need, letting them pick and choose the columns they want.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post String parsing appeared first on LobsterPot Solutions.

On Thursday, I was sat in a “Cloud & AI Innovation Summit” put on by the Adelaide Microsoft sub. I could’ve been visiting a customer and working on their data quality, or helping to sort out the details of an application, or writing documentation. But instead I was in sessions listening to what’s being said about tech, letting my head wander around the space, having conversations with Adelaide leaders.

I mention this because on that day I was reflecting on the fact that I learn differently to many others. In a session, I’ll probably be tapping on my phone, taking notes about things, jotting down ideas, looking things up, and generally letting my head roam around the concepts. People with pens often draw shapes or doodle. Questions appear in my head, which I either want to ask or research myself – if I don’t I end up feeling distracted by them. I’ve known for a long time that classroom environments and conference sessions are not really my preferred learning style, but if they can be conducive to letting my mind want through idea, then that’s good.

But Todd asked about learning (well, mastering!) new skills, rather than getting familiar with concepts. I’m talking about the latter because I feel it forms a foundation for skills to be built on. The concepts are the bedrock for the skills to attach to. I want to know the physics about why a ball will spin a certain way before I even put my shorts on. (Bad example – those days are long gone.)

I know that a lot of people want to experience the physical aspects of the world before they learn why things behaved that way, and that it’s very important to learn muscle memory to have a skill, but there’s a lot to be same for explaining concepts and inner workings, so that the why can be grasped. For me, at least, it’s that becomes a bit of a prerequisite for most new skills.

If I’m going to become truly proficient with a skill, it has to include knowing how the thing works. As a kid I had muscle memory that let me move around a piano keyboard, and I’d done some amount of theory as well, but it was only later that chord progressions gave me more freedom. When I picked up guitar-shaped instruments, the knowledge I had let me get comfortable much quicker.

I’m curious to see what other people say about how they master skills. There are a lot of people who like learning new skills in the data community.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post Learning new skills appeared first on LobsterPot Solutions.

There are definitely plenty of people who use AI to help with emails, or as a replacement search engine, and anyone who’s writing code in a tool like VS Code will be getting code suggestions like never before, even if they’re not writing prompts for Copilot to write the code for them. But most of these people wouldn’t really say their careers have been changed – just that they’re becoming better at their job because of the tools that are now available.

I could ask an AI tool to write this post for me – but in some ways I feel like that’s avoiding the writing task, rather than just getting help with it.

But I think this has always been how it is with technology. There’s a process that is done by a person, and over time, the person gets better tools to help improve quality or speed or scale. Pretty soon, the job can be done by someone who doesn’t have the same skills as the original person, and later it can be done without a person being involved at all. My car’s ignition mechanism is a button I press. The car I had beforehand had a key that needed to be turned. Some electric cars just “turn on” when your phone is in the car. It’s all very different to my uncle’s 1902 Peugeot. It used to be a process that a person needed to be very much involved in, but now you barely even need to know that a car needs to be turned on.

When I learned to code, it was a Commodore 64. People told me I should learn to program using Assembly, but I saw benefit to the speed with which I could get programs created by leveraging the functionality provided within BASIC. I didn’t keep writing BASIC code, and different languages came with different frameworks to get different things done. The purists writing in low-level languages got left behind by the people leveraging other frameworks. Using those frameworks meant they could produce more functionality in less time.

As far as coding today, AI tools are like having another framework available. The packages available in most languages, including Python, PowerShell, JavaScript, .Net, and so many more, aren’t about to go away, but now AI provides the tools to use those packages more quickly. We’ve gone from grabbing a package and figuring out the nuances of how to use it to having AI recommend a package and write a bunch of code for us. The ‘no-code’ options provided by UI Wizards are being replaced by systems where we can talk to AI and get a system that works the way we want. It’s not exactly ‘no-code’, but it’s pretty close to ‘code-you-never-need-to-see’. For me, I’m hoping it’ll be like ‘produce tests to demonstrate the correctness of this’, especially given AI’s tendency to hallucinate.

I was doing AI at university in the early 90s. My lecturers were trying to persuade me to accept an offer of a PhD at Edinburgh University, but I went into consulting instead. AI back then was very different, but I know we’ve seen AI change our jobs for many years – we just might not have been noticing. LLMs have brought AI into the forefront of our thinking, brought it to regular people, so now it’s more obvious that AI is having an impact.

For everyone in IT – heck, everyone in every industry! – staying current is essential. We all need to be developing our skills and using the latest tools, so that we don’t get overtaken and made redundant by new technology that’s coming up. Our jobs aren’t necessarily going to be replaced, but the tools we were using yesterday have been replaced, and AI is just another tool that we should be loading into our toolsets.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post AI’s impact on careers appeared first on LobsterPot Solutions.

I’ll come back to the ‘what I’m doing’ question. First let’s talk about the age-demographic of people in the SQL community.

Yeah – there is definitely a lack of young people. But I think this has always been the case. Even when I got into SQL properly, this was after I’d spent about seven years as a consultant / programmer. Databases had always been part of what I did, but I didn’t consider myself a specialist until later. Even the first presentations I did in the technical community were about coding, not about SQL.

My story wasn’t unique, either. Other people I met in the SQL world had taken a while to realise the significance of data, even though their journeys were sometimes different. Some came to databases after spending time as a sysadmin, some came from business applications like Excel, and some had been programmers like I had. Few seemed to get into data right away, and at 30 I was still seen as fairly young for the data community. When I was first awarded MVP status (when I was 31) there were just four other SQL MVPs in Australia – all older than me.

These days, data is understood much earlier. Data science and analytics have become legitimate university courses in their own right! Back in my day, data was a single uni subject. SQL varieties don’t quite get the same appreciation as other platforms, but data itself is more popular than ever.

The community though… well, it’s like Fight Club. People don’t always talk about community like they could. They might attend meetings to learn, but the benefit of time spent with other professionals can take a while to click – especially in a post-pandemic world where people are used to doing their learning online.

Community seems to be lacking in a lot of parts of society – certainly here in Australia where there were significant periods of lockdown (much worse in Melbourne than Adelaide where I am). People still need people, but losing loved ones to contagious diseases has meant people just don’t choose to be around other people so much. Online groups are good, but people can watch in isolation.

Getting to the other side of the room – moving from being in an audience to presenting – that’s even harder for people today. If they’re not even in the room much, even having the conversation about presenting is tricky. What am I doing to change this? Well, I still mentor various people, and some of that is about presenting, but more of it is about getting established as experts – helping them learn what they want to learn, and encouraging them to take steps to go where they want to.

In my mentoring, I encourage people those who don’t know about data to learn about data. I encourage people who know about data to attend community events. And I encourage people who are attending to start thinking about what they can present about and then to take the step to do it. Because presenting on a topic is a great way to make sure you really know your stuff.

Presenting can be part of someone’s journey, but it can also take time to warm up to the idea. Whenever you step onto a stage you become very visible, and that can be daunting. But I also think that if someone is feeling comfortable in their own skin as an expert, knowing that they are legitimately good at what they do (often takes quite a bit of mentoring and encouragement), then they will often start to realise that they belong on the other side of the room.

I hope that my user group will be able to have a local first-time presenter later this year – and ideally, it’ll be someone under 30. I have some ideas about who that might be, but I know the timing has to be right for them, so I’m not setting it as an actual goal.

I’m definitely interested to see what other people are doing to encourage younger presenters – but I think when you look at the wider data community, not just SQL, things aren’t as grey as they might seem.

@robfarley.com@bluesky (previously @rob_farley@twitter)

The post The younger generation in the SQL community appeared first on LobsterPot Solutions.