As we approach the end of another remarkable year at LobsterPot Solutions, it’s time to reflect on our journey, celebrate our achievements, thank our customers, team members and the community, and look forward to the exciting opportunities that lie ahead.

One of the highlights of this year has been our work on several groundbreaking projects. Our team has been at the forefront of developing cutting-edge solutions that have significantly enhanced our clients’ digital experiences. This year, we have completed a high-value data warehouse implantation, continued to provide ongoing technical expertise to various projects and even delivered expert advice to a legal team about best practice database system implementation ahead of a significant court case.

Our success is built on the strong relationships we have with our clients. This year, we’ve continued to deepen these connections by delivering exceptional service and tailored solutions. We’ve worked closely with our clients to understand their unique needs and challenges, ensuring that we provide the most effective and impactful solutions. Our clients have included engineering and manufacturing firms, vehicle financial services, hire equipment specialists, waste management services, high schools and education providers, state government units, and even other database consultancies, although Rob’s usual seasonal work as Father Christmas has been called off this year because he’s still on crutches after fracturing his tibia a few months ago.

To everyone on our team, thank you for representing us so well. It’s wonderful to have such dedicated individuals helping us deliver outstanding results to our customers. We truly appreciate your commitment, flexibility, and exceptional skills. Your efforts make us proud every day. Thank you!

Our commitment to the community has been a cornerstone of our values at LobsterPot Solutions. This year, between the multiple user groups we coordinate, we have conducted at least 14 meetings, bringing the Adelaide and Melbourne data platform communities together for learning and networking. Rob and Kelly have also shared their expertise at more than eight different Australian and international conferences. Oh, and Kelly was finally recognised as an Microsoft MVP, recognising her continued impact in the technical community. Well done Kelly!

As we look forward to the coming year, we are excited about the opportunities that await us. The recent announcements about SQL Server 2025 and Fabric SQL DB, along with the development of the Azure SQL and Fabric platforms give us confidence that we’ve got busy times ahead, helping people do what they do better with this technology. Thank you to our clients, partners, and team members for making this year a success. We look forward to continuing this journey together and reaching even greater heights in the year ahead.

Merry Christmas, and we’ll see you in 2025!

The post Happy end of 2024 appeared first on LobsterPot Solutions.

In this series of blog posts, I want to explain different ways to extrapolate business requirements to write great reports. In this post, we’ll look at steps to connect to source data to create a new Power BI report. Firstly, let me explain what I mean by “Great Reports”. These are reports that are user-friendly, scalable and reusable. Most importantly, reports that the business keeps using, well after you finish developing them.

Great reports MUST have a great data model (or semantic model for those who work in Power BI/Fabric) behind them, so most of my advice here is actually about the data model design, not the reports themselves. I work predominantly with Power BI, Fabric, and SQL Server Analysis Services, so my advice is tailored to those technologies. If you want to catch up on the previous posts, go to:

• Useful tips for designing a really great report model from Agile user stories
• Design a really terrific Star Schema by interpreting an existing report

Here are the connection details. Start developing!

Who has been told by a manager or business person to just connect to the source data and start creating a new report? Here is my tip:

DON’T DO IT

All Power BI and Fabric reports must have a semantic model, which Microsoft describes as “a logical description of an analytical domain, with metrics, business-friendly terminology, and representation, to enable deeper analysis.” – Source

If you try to build a report from a transactional source, you will end up with a data model replicating the tables structured from the source system, which is (hopefully) a third-normal form relational database. Or worse, a flat, single table because the API presented all the data in one endpoint. This is not ideal for a Power BI or Fabric Semantic model because it relies on many-to-many and bi-directional relationships. Many very smart people have written about why these are bad here and here, so I’m not going to repeat their teachings. Just know that a poorly designed data model can reduce the report performance, introduce logic errors and require complicated and erroneous DAX measures. You want your Semantic model to be dimensional, that is, a star schema.

What to do instead

Here are my recommended steps to connect to source data to create a new Power BI report:

Step 1: Talk to the business to identify the business problem or reporting opportunity

My recommendation is to follow the Kimball methodology. For more information, check out the official pages.

In this methodology, the Kimball Group says that you need to identify the business process, grain, dimensions, and facts to design a dimensional/semantic model. And “for a dimensional modeller, the business process is an event or activity which generates or collects metrics” Quote from Design Tip #69 Identifying Business Processes.

So, go back to your manager or business person and ask them any of the following questions:

• What business process do you want to report on?
• What business activity or event are you interested in?
• What about that activity or event is of interest?
• What behaviour in the business are you looking to understand more, track or change?

OK, that last question might be a bit deep for some managers, but it might also start a good discussion about what they really want the reports for. From there, you can treat their requirements as user stories and follow my advice in the Useful tips for designing a really great report model from Agile user stories.

If they still just want a report from a source system…

If the manager or business person still insists that they need a report from a source system, there are a few things you could educate them on:

• Business systems are large and complex. Do they want a report on all features and elements of the business system? If so, they’ll unfortunately be waiting a really long time for you to finish. Can they narrow down the scope to get you started? Or can they set a highest priority business element to start with?
• Off-the-shelf software is designed to meet the requirements of many different organisations/businesses/companies, not just yours. Therefore, they allow different configurations and implementation of business rules. Is there someone in the organisation that you can talk to about the config of the system and the business rules so that you don’t misinterpret the data?
• Regardless of how you receive your user requirements, there are still a number of questions about the security, timeliness and usability of your report that need to be asked. I will cover these questions in the next blog post, but I don’t want to forget to mention them here.

Maybe your manager has said that he needs a new report to analyse the patients who visit a healthcare practitioner for medical advice and leave with a medication prescription.

Step 2: Do some research to help yourself out

Operational systems are typically implemented for a specific business process or processes. For example, iChris is a HR software solution, Salesforce is a CRM, Xero is accounting software used for invoice management and payroll, and Microsoft Dynamics is an ERP and CRM suite. You could research the source system to find what business processes it supports. The trap here is to still focus on too many processes at once. You will need to prioritise one to start with. I like to remind my clients that “Priority 1” and “High Priority” are singular. It isn’t “Priority ones” or “High priorities”; there is no “s”; there can be only one.

Next, you need to start to think about the grain, dimensions, and facts. Microsoft has published some articles on Learn about the Common Data Model (https://learn.microsoft.com/en-us/common-data-model/), which would be very helpful to look at next. Business processes like healthcare, human resources, campaigns, budgeting, sales, etc., are described in detail, along with attributes, semantic metadata, and relationships.

These published resources can save you a heap of time and help highlight anything in the dimensional/semantic model that you may have missed because you don’t have someone to ask about the business rules or because you’re a report developer, not a subject matter expert in healthcare, sales, finance, marketing, etc.

You don’t need to use Dynamics 365 to leverage the resources available for the subject areas. The Learn pages and GitHub repo are free and available to anyone.

Step 3: Document the business process, grain, dimensions, and facts

Research has been done, but you must still formally identify the business process, grain, dimensions, and facts to design a dimensional/semantic model. It’s worth noting that the business process often becomes the Fact or Facts in your model. Remember, your manager needs a new report to analyse the patients who visit a healthcare practitioner for medical advice and leave with a medication prescription. This feels complicated because the person requesting the report, the Kimball books and the CDM mentioned above all use slightly different language to say the same things, so I break it down like follows:

Business Process

Dimensions

Facts

Step 4: Draw the model on paper

As I’ve said in previous posts, I will ALWAYS draw the dimensional/semantic model on a whiteboard or a blank piece of paper (physical or online), as it’s the quickest and easiest to change and evolve.

Then, use the drawing to see if you can answer some of the questions your manager or business person originally asked. For example;

• “I want to know how many patients who visit a healthcare practitioner for medical advice leave with a medication prescription” – the model above allows you to slice the Medications by Patient, so this requirement would be met.
• “I want to know, on average, how often a healthcare practitioner prescribes medication to patients” – the model includes physician, number of encounters, and number of encounters with a prescribed mediation, so this requirement can be met.
• “I want to know the average time it takes patients to fulfil their prescription” – the model doesn’t appear to include any prescription fulfilment information, so this requirement would not be met.

Step 5: Load the data into your model

Now, you are finally ready to start developing in Power BI by loading data from the source system into your carefully designed dimensional/semantic model.

Please always apply this well-known theory:

That means that, if you can, transform the data at the source system layer before you load it into Power Query or Pipelines. This may mean:

• Create T-SQL views in a SQL Server database
• Create reports in the application (Salesforce and Xero are good at this) and then connect Power Query to the report
• Build or use a custom API from the application development tools

If you can’t control the source system enough to transform your data before ingestion, consider implementing a Medallion architecture in Fabric. Tasks Flows are a great new feature to help here.

Finally, after you have designed a dimensional/semantic model that reflects a business process with facts and dimensions at the appropriate grain and you have loaded the data, you can start developing the report visuals.

I hope these recommended steps to connect to source data to create a new Power BI report will get you started when you’ve been told by a manager or business person to create a report from raw, system data.

Happy designing!

The post Recommended steps for connecting to source data and creating a new Power BI report appeared first on LobsterPot Solutions.

Where to start

I have worked as a business intelligence developer for several years, and I’m always asked: “How do you convert user requirements to a functioning data model?”

I follow the Kimball methodology. For more information, check out the official pages.

But, here are some specific tips on what works for me.

Firstly, understand that requirements evolve. Very few people know precisely what they want before they start, mainly because they don’t know what’s possible. Once I start developing a new report, especially in a new tool, they will ask for something new or different to the original requirements. This is why Agile principles work well in business intelligence development. The trick is to get as close as possible without doing more work than required. Do not get attached to the first data model; know and understand that it will evolve over time.

Before we get into more details, I follow a few ground rules when I interpret requirements; these include:

Have a process

Many stakeholders, especially those paying for the solution, don’t appreciate ‘loose’ requirements and poorly defined deliverables. To address their concerns, I show them a pre-defined process to gather requirements and build report parts or facts and dimensions. Each project may use a slightly different process, but create a method that works best for you. You can start with the one I describe below.

Set expectations

Once I know what process I will use, I explain that each iteration of the development will take a specific number of hours or days and what should be expected from each repetition. It’s essential to be clear on the process and what can be achieved each time you repeat it.

Always create a star schema

I don’t want to see data models that replicate the tables from the source data or ‘Flat’ data models, where everything is in one or two tables. These models perform poorly, are unscalable (try adding a new piece of data), and are challenging to understand and find things in. They often contain repeated data, only work when there is a single data source and are difficult to write DAX expressions on (I could rant for hours on why you need a star schema, but that’s reserved for another post). Regardless of size, I will always design and build a star schema for all reports. In case you are not familiar with star schemas, here are the basics:

Facts in the middle

• Also known as measure groups (in SSAS Multidimensional) because they measure things
• They align to a business process, or some event that happens in your business, e.g. a Sale is completed, a Customer places an Order
• Normally referred to in the plural, e.g. Sales, Orders

Dimensions on the points

• Describe the facts
• What you’d see in filters and groupings
• Normally referred to in the singular, e.g. Product, Date, People

I understand that Power BI is an ELT (“Extract, Load, and Transform”) tool, as opposed to an ETL (“Extract, Transform, and Load”) tool, and individual queries need to be written for each source table. These queries become the tables in your model, but I still follow Fact and Dimension logic to split or combine source tables to ensure I create a star schema. Now that you have some ground rules, let’s review the reporting requirements. Remember this image about understanding requirements? It totally applies here. Your ability to understand the customers’ requirements is vital to delivering a brilliant, usable, well-performing, adaptable and scalable report.

Congratulations, you have been asked to replicate an existing report.

More often than not, customers (or clients/business users/stakeholders, call them what you want; it’s the person requesting the report) will ask you to rebuild an existing report, either from a new source system or because there is a problem with the existing report. Alternatively, you may get a report mock-up drawn on paper (or a napkin – yes, that has happened to me). Examining the sample or mick-up provided is essential as a first step in your process.

Star Schema from an existing report

No matter what, you need to create a Star Schema by interpreting this existing report and mock-up. Start with version 1 of the star schema by looking for the actions, acts or things that need to be counted (facts) and the descriptions, filters, and groupings (dimensions).

Just looking at these two sample reports, I can see that they need the following in the Star Schema for our existing report and mock-up:

Dimensions

• Division and Territory Name – could be combined into a Sales Region dimension.
• Employees – including Employee Last Name
• Product – including Product Category
• Lines (I’m not sure what this means)
• MHs (I’m not sure what this means)
• CAGR – 12 month, quarterly and Order Year – so ensure to include a Date dimension with quarters and all dates listed (to use Power BI’s inbuilt time intelligence features)

Facts

• Income
• Budget
• Actual (is this the same as Income?)
• Actual Variance to Budget $
• Actual Variance to Budget %
• Total (I don’t know what these charts are showing as the total amount)

As you can see, this process has identified some dimensions and facts to start the star schema design, but it has also highlighted some serious questions about what data is needed in the report. It’s vital at this stage to find the people who will actually be using the report. Too often, I find that analysts or BI developers think that they understand the needs of the business, but they know the requirements from a technical perspective only. Go and ask these questions of the businesspeople and customers who created the mock-up:

• What do you mean by Lines in the mock-up?
• What do you mean by MHs in the mock-up?
• Is Actual the same as Income? If they are different, how so?
• I don’t know what these charts show as the total amount. Can you please be more specific?

Once you have some answers, you can start to design the star schema. Just a note: I will ALWAYS begin on a whiteboard or a blank piece of paper (physical or online), as it’s the quickest and easiest to change and evolve.

It’s always a good idea to combine different requirements-gathering techniques to understand better what the customer wants from the reports. Refer to this post for more ideas (Useful tips for designing a really great report model from Agile user stories).

Once you are clear on the requirements, start to look at source systems and understand how the data is structured. This will undoubtedly form part of your process and improve the star schema. Remember to keep changing it as you find out more information.

I hope this gives you some understanding and ideas for designing a user-friendly, scalable and re-usable Star Schema by interpreting an existing report or report mock-ups.

Happy designing!

The post Design a really terrific Star Schema by interpreting an existing report appeared first on LobsterPot Solutions.

I start by asking questions of the person requesting the report.

• Who will look at the report?
• Why will they look at the report?
• What are they trying to determine?
• What actions do they need to take from the report?
• What questions does the report need to answer?
• How often does the data need to be refreshed?

If you want business people to use the report, talk to them about their requirements

It’s vital to design a great data model from Agile user stories to find the people who will actually be using the report. Too often, I find that analysts or BI developers think that they understand the needs of the business, but they know the requirements from a technical perspective only.

If I’m working in an Agile project, I may be lucky enough to get a concise and detailed user story or something similar to articulate the requirements. It may look like this:

Look for the verbs and nouns

As the model designer (or BI developer), my job is to find the verbs and nouns in the story. Remember back to grade 3; verbs are words that show actions, motions, doing, or states of being; nouns are words or phrases that are the name of something or describe something.

Sometimes, a word might technically not be a verb or a noun, but another way to think about this exercise is to find the actions or acts (facts) and the descriptions (dimensions).

I will highlight the verbs, actions or acts (facts) and nouns or descriptions (dimensions) in your user story:

Conceptualise the star schema

From here, I will conceptualise the star schema. Just a note: I will ALWAYS begin on a whiteboard or a blank piece of paper (physical or online), as it’s the quickest and easiest to change and evolve.

Start with the Facts

I start with the verbs, actions or acts (facts). Have I highlighted one or many in the user story? If there are many, do they all relate to the same business process? A good fact table models an actual process that the business does, for example, hiring people and terminating employees. An employee movements fact table could include new hires, bad hires and terminations and have the flexibility to include promotions or team changes later.

I will draw the employee movements fact table in the middle of the page.

How will we describe the Facts?

Now, I look for the nouns, attributes or descriptions (dimensions) I highlighted. I’ll ask the same questions as for the facts: Have I highlighted one or many in the user story? If there are many, do they all relate to the same business process? “12 months” is an attribute of a calendar, so I will need a time, date or calendar dimension (call it what you want). I’ll then add a dimension for gender and a dimension for ethnicity.

Age is a little more complicated because an employee’s age changes yearly. Instead of adding an Age dimension, I may add a date of birth to calculate age at any time, or age buckets (21 or younger, 22-30, 31-40, etc.) that will change less often. In this scenario, I might even prefer to rename the attribute to “age at the time of employee movement,” then I know it won’t change.

Full time, part time, hourly, and salaried can all relate to an employment type, so I’ll draw an employment type dimension in the star schema.

I also highlighted “bad” in relation to bad hires. First, I need to go to the business people and confirm what defines a “bad hire”. Then I need to ensure that I have all the data I need to determine if a hire is “bad”. This attribute could be a calculated column based on the business rule and other attributes or a new attribute derived from the source system. Finally, if I choose to implement an employee movements fact table, I will need an employee movements type to specify if the movement is a new hire, termination, promotion or team change. This may be an attribute in the employee movements fact table or a new dimension.

Record attributes and measures

Now that we have facts and dimensions listed start to record attributes and measures. You may be able to get these from the questions you asked and the user story, but if not, ASK.

Now design a great data model from Agile user stories – rework as needed

You may also realise that some dimensions are so small they don’t need an entire dimension to define. For example, gender has only 1 description field plus a primary key, and the primary key could be the same as the description. This applies to movement type as well. I can include the whole description on the fact table and remove the gender and movement type dimensions, which makes them “degenerate dimensions”. I can then remove the gender and movement type dimensions.

Determine where the data will come from

Which systems and objects/tables within the system contain the fields? Most likely, the business user is a user of the system, and they can step you through how they would find the data manually. Use this to understand the complexity of the data sources. Are there 10 different sources, or 1? Is all the data on a single screen, or must you navigate multiple places to find everything?

Ask more questions

Importantly, at this stage, ask questions about how the Facts and Dimensions connect to each other. If Budgets are in a different system to Actuals, is there a common identifier/dimension you can use? Do you need additional objects/tables to get ID columns that mean something?

If you don’t have the budget or time to develop the entire model, design and build a basic star schema. Be sure to include a fact table and multiple dimensions that directly connect to that fact table. You won’t get approval to build more if you can’t prove the value of the first part; if there is nothing for the user to see in the report, they won’t be willing to pay for more of that ‘nothing’.

The best way to get started designing a great report model from Agile user stories is to get the user story and begin drawing the star schema on a whiteboard or a blank piece of paper. Then keep changing it as you find out more information. I hope this helps you to understand the business requirements and translate them into a scalable data model. In my next post, I’ll talk about designing a report model from an existing report.

Happy designing!

Kelly Broekstra | twitter | LinkedIn

The post Useful tips for designing a really great report model from Agile user stories appeared first on LobsterPot Solutions.

We’ve helped a community health care provider deliver services to COVID positive people, a government commission to deliver a data governance and regulatory intelligence system that is both robust and secure, and continued our work assisting a government department in the management and analysis of critical debt-related data.

Although most of our prominent projects have focused on data warehouses, business intelligence, and self-service reporting solutions, several clients have again called us to improve poor performing database systems. Our focus on getting quick but significant wins in performance tuning is world-renowned and heavily sought-after. 

Power BI has been the big driver of new projects this year. Many clients approach us for report development, and then realise that we can assist with data management, governance, warehousing and system implementations. Several of these projects have morphed into more extensive engagements that have included training, participation in Centre of Excellence discussions and the tuning of source systems.

A newer endeavour for LobsterPot Solutions has been data warehouse health checks, which have now been requested by multiple clients. Our comprehensive review of your data warehouse environment includes documentation of your existing elements as well as recommendations for performance improvements, upgrade and migration strategies, and best-practice improvements.

Rob’s even found time to step away from the data platform consulting and spend time entertaining kids and parents alike with his Santa antics. But unfortunately, he had to perform some of his duties from home after a COVID close contact.

We feel that 2021 has had a bit of everything. Our clients have said the following about LobsterPot Solutions this year:

“LobsterPot Solutions has assisted the Commission in establishing a centralised information platform using Azure Data Lake Storage, Azure Data Factory, Azure SQL DB, Azure Purview and Power BI. The ongoing tailored support and expertise provided by LobsterPot Solutions has helped the team to grow their knowledge and expertise and deliver solutions that have exceeded expectations.”

“The LobsterPot Solutions consultant was willing to work as a member of the team and be part of the team to solve the problem. They had the technical expertise and also took the time to understand how we work. They would challenge us and identify possible gaps and flaws but provide potential solutions.”

“I just wanted to give a shout out to the LobsterPot Solutions consultant for the support they have provided to the team over the last two weeks. They helped them to develop a dashboard of indicators (some with challenging logic) within a very short time. We couldn’t have done it without them, and they did it under challenging circumstances, so we really appreciate it.”

Everyone at LobsterPot Solutions is looking forward to the challenges that the new year will bring.

Merry Christmas to all of you.

The post The year that was 2021 for LobsterPot Solutions appeared first on LobsterPot Solutions.