Archive for Software

Create a Pareto Chart in Power BI

“Baseball is 90% mental, and the other half is physical.” – Yogi Berra

You just have to love Yogi Berra quotes like this. We all pretty much know what he’s talking about, even if his math is not spot on. It’s a restatement of the Pareto Principle, the 80/20 rule! It applies to just about anything in life or business. If I had to write a definition of it for technical documentation, it would look something like this:

“A situation where eighty percent of events attributed to a group are caused by twenty percent of the members of the group.”

Re-stated as examples:

  • Eighty percent of your human resource issues are caused by twenty percent of your employees.
  • Eighty percent of your maintenance issues are caused by twenty percent of your equipment.
  • Eighty percent of your sales are attributed to twenty percent of your products.
  • Eighty percent of the wealth is controlled by twenty percent of the population.


(And I’m certainly not in that last twenty percent. If I was, I wouldn’t have to write articles like this one!)

Now that we’ve got an understanding of the principle, let’s look at how it can be visualized. Excel has a very simple wizard for creating a Pareto Chart that can be found on the Insert menu:

But we want one of these in Power BI. And Power BI doesn’t have one (yet, maybe later). We’ll need to ‘roll our own’.  Let’s discuss the various parts of the chart itself, so we know what we’re shooting for.

  • The categories, or series, at the bottom (One, Two, Three, etc.) represent the different members of the ‘group’ we are trying to analyze. They may be employees, machines on the manufacturing line, or products in our catalog.
  • The blue bar above each ‘member’ is its respective measurement (count of HR issues, money spent on maintenance, annual sales, etc.). The scale of this measurement is on the left side, in our case going from 0 to 120.
  • The final element is the curved line and the right-side scale measuring from zero to one hundred percent. This represents, at each category member, the percentage of the cumulative total of all members to the left of the member in question, inclusive. To put it another way, as we add each category’s number to the running total of those on its left, the line represents that running total divided by the entire total for all members of the category.

The arrow points to the spot on the line where it crosses 80%, in our case, after about the first four members, as can be seen by following the green dashed line from right to left, then down. The first four members would be the ‘twenty percent’ of the Pareto Principle, and their cumulative measure would be the eighty.

Note: Math wizards may point out that four members divided by a total member count of fifteen is closer to thirty percent than twenty, but remember that this is a rule of thumb, and we all know that some thumbs are bigger or smaller than others.

To plot some data in a Pareto Chart, we’ll need a couple of pieces of information from it:

  • Each member’s respective total
  • The grand total
  • The running total at each member, sorted from largest to smallest
  • The percentage that running total represents compared to the grand total

Now that we understand what we’re shooting for, let’s get started.

If your data includes a running sum of the measurement for each member, sorted by the respective member’s measurement, then you’re golden and can skip to the section titled Add the Grand Total and Running Percent. Your data may include a Ranking column so you may be able to skip the respective steps in each of the following two sections. For the rest of you, keep reading. We’ll look at two approaches to getting the intermediate bits of data: Power Query (M), and DAX.

Create the Rank and Running Sum in Power Query

Let’s start with some simple data in Excel, in fact the same data used to generate the Excel Pareto chart we used to explain the concepts:

We’ll load this data (it’s in an Excel table called “Table1”) and edit it in the Power BI Query Editor. First, we need to sort the data by the [Measure] column, sorted descending. Click the down-arrow next to the Measure column title and select Sort Descending.

Next, on the Add Column menu, select Index Column. Keep the defaults of Starting Index of 1 and Increment of 1.

I renamed my column to [Power Query Rank] to differentiate it from ranking step we’ll introduce in the model later via DAX.

Next, we’ll add the running total as a Custom Column with a formula as shown below:

Hint: If you can’t read the formula from the screen shot, it is:

= Table.Range ( #”Renamed Columns”, 0, [Power Query Rank] )

Attribution should go to Sam Vanga and SQL Server Central for this bit of M code:

The Power Query function Table.Range can be explained like this: Given a table of data, in our case the last of our query steps, a.k.a. #”Renamed Columns”, start at the 0 row (top), and go down the number of rows represented by the value in column [Power Query Rank]. The result is a table associated with each row in the query. The first row of the query has a table with one row of data in it. The second row has a table with two rows, and so forth. This table is represented by the word “Table” on each row of the column we just added.

From here, click on the ‘expand’ arrow in the column header and select the Aggregate radio button, check off the “Sum of Measure” column, and un-check “Use original column name as prefix”:

I renamed the resulting column [Power Query Running Total] (not shown).

Click Close and Apply on the Home menu.

Create the Rank and Running Sum in DAX

As with all things Microsoft, there is more than one way to accomplish a goal. In our case, the goal is to get the running total, and just like before, we’ll need the ranking first. For this exercise, we’ll be using DAX instead of Power Query, but should get the same results.

Create a column with the formula as follows:

DAX Rank = RANKX (All ( Table1 ), [Measure] )

Next, create the DAX Running Total measure as:

DAX Running Total =


SUM ( Table1[Measure] ),



Table1[DAX Rank] <= MAX ( Table1[DAX Rank] )



This DAX formula does pretty much the same thing as the Power Query Range.Table function above, the only difference is that it includes the aggregate within, eliminating the need for an extra column.

Note: Know the difference between Columns and Measures in DAX. Mistaking the two will cause error, frustration, and hair loss.

Plotting all these columns and measures on a simple table visual shows that Power Query and DAX come up with the same answers for Rank and Running Total, a good sanity check. Also, the ranks are easy to verify as to accuracy, and with a little mental math, running totals are as well. I had to re-format some of the numbers to make them show without decimals.

Add the Grand Total and Running Percentage

There’s two more pieces we need: [Grand Total] which is self-explanatory, and [Running Percent], which is the ‘percentage of the [Running Total] compared to the [Grand Total]’. These can only be done in DAX. Add a measure as follows:

Grand Total = CALCULATE ( SUM ( Table1[Measure] ) , ALL ( Table1 ) )

This calculates the Grand Total and makes it available at every slice (row of each Member).

Now add the last item, a column with the expression:

Running Percent = [Power Query Running Total] / [Grand Total]


Running Percent = DIVIDE ( [Power Query Running Total] , [Grand Total] )

Note: The column [DAX Running Total] would work just as well as its Power Query equivalent since we know it has the same number.

Format this last one as a percent.

Create the Chart

Now the fun part. For this we’ll need either a “Line and Stacked Column Chart” or a “Line and Clustered Column Chart”. This is the easiest part of the whole exercise:

  • The Shared Axis is the [Member] column (“One”, “Two”, “Three”, etc.)
  • The Column values is the [Measure] column
  • The Line values in the [Running Percent] column

Like I said, simple if you have all of the data pieces in front of you.

Need help getting the right data pieces? Not sure what charts you can generate from the data pieces you have? There’s probably a way to get to where you want to be. Reach out to our team of data scientists at BlumShapiro Consulting to learn more about how data can help guide your organization into the future.

Microsoft Announces Power BI Premium: Removes Functionality on Free Version

Many of our clients come to us looking for solutions to help them achieve “Business Intelligence for Everyone” in their organization while avoiding the pitfalls of reporting in Excel. Our response is simple: Microsoft Power BI is an easy-to-use, non-technical business intelligence tool which is far more robust than Microsoft Excel for reporting. End users who rely upon Excel for reporting often view Power BI as a logical step up. With Power BI, users can automate mundane data transformation steps, connect to a broad range of data sources and securely collaborate with colleagues  —all within an environment that looks and feels just like Excel. Our clients have reported that Power BI’s free edition includes enough functionality to get started on any reporting initiative, automate data extraction and transformation activities and share the results with a team of executives, analysts, managers and colleagues. However, as Power BI data and report volumes grow, organizations may choose to step up to Power BI Pro, which upgrades users from 1GB to 10GB of data and enables complex analytics sharing capabilities, even outside the organization.

Finding a Solution for Larger Organizations

The current Power BI service does present some challenges to larger, more sophisticated organizations. Some of the issues include:

  •   Sharing and collaboration features would often become complex and difficult to manage
  • Compute resources are shared, not dedicated, and there is no ability to provision additional compute resources
  • Structured reporting capabilities are not well suited for interactive reports and “single pane of glass” dashboards delivered in Power Bi

These issues begged for a simpler, more manageable model for large organizations.

Introducing Power BI Premium

In early May 2017, Microsoft announced its intention to introduce a new licensing level for Power BI, Power BI Premium. Power BI Premium is designed to address the shortcomings of Power BI Pro. Here are three things to know about Power BI Premium:

  1. Power BI Premium Edition will support Power BI Apps. Power BI Apps replace Content Packs and Power BI Embedded. Organizations that currently share Power BI content externally with Power BI Embedded should plan to migrate to Power BI Premium Edition.
  1. Power BI Premium Edition offers dedicated capacity for organizations that need more control. Instead of paying strictly per user, Power BI Premium is licensed on a combined capacity and usage model. This enables organizations who struggle with the per user data limits enforced on Free and Pro Edition users (1 GB and 10GB maximums, respectively) to load data models that are much larger. As with other Azure services, organizations can scale up and scale down capacity as their needs change.
  1. Power BI Premium Edition includes a license for Power BI Report Server—a full featured on-premises solution supporting both Power BI (interactive) reports and Reporting Services (paginated, structured) reports.

Important Note for Power BI Free Edition Users

Power BI Free Edition became quite attractive because many users within the same organization could share content without paying any fee. Unfortunately, Power BI Free Edition functionality will be changing soon. Users on the Free Edition will no longer be able to share dashboards with colleagues, other than by printing them out, or showing others their “personal dashboard” in a browser. As of June 1, users enjoying dashboard sharing will no longer be able to do so under the Free Edition.

June 1st is right around the corner, and some organizations have built fully functional company dashboards using Free Edition licenses. These organizations now face the prospect of having to either upgrade to Power BI Pro Edition ($10/user/month) or lose vital collaboration features. This is why Microsoft is offering a 1-year trial of Power BI Pro licenses to users who have previously signed up for Power BI Free Edition. This allows organizations to carefully consider which users need Power BI Pro for data model, report and dashboard creation and collaboration and which do not. Some organizations will stay on the Free Edition, and simply share their BI content via PowerPoint. Others will look at Power BI Pro or Premium licensing and continue to see value.

Next Steps

Microsoft has stated that general availability of Power BI Premium is on the horizon, but no specific release date has been communicated. If your organization has many users creating reports and dashboards with the Free Edition, here are some things you can do to get ready for the change.

  1. Take advantage of the 1-Year Power BI Pro trial – encourage users to respond to any email communication from Microsoft and take advantage of the grace period
  1. Download the Power BI Report Server and take it for a spin
  1. Review the Power BI Premium Calculator to understand what your costs would look like under the Power BI Premium model

For more information on how to achieve high performance analytics and reporting with Power BI, contact Brian Berry and our Data Analytics team at, or by phone at 860.570.6368.

Berry_Brian-240About Brian: Brian Berry leads the Microsoft Business Intelligence and Data Analytics practice at BlumShapiro. He has over 15 years of experience with information technology (IT), software design and consulting. Brian specializes in identifying business intelligence (BI) and data management solutions for upper mid-market manufacturing, distribution and retail firms in New England. He focuses on technologies which drive value in analytics: data integration, self-service BI, cloud computing and predictive analytics

Using Real Time Data Analytics and Visualization Tools to Drive Your Business Forward

Business leaders need timely information about the operations and profitability of the businesses they manage to help make informed decisions. But when information delivery is delayed, decision makers lose precious time to adjust and respond to changing market conditions, customer preferences, supplier issues or all three. When thinking about any business analytics solution, a critical question to ask is: how frequently can we (or should we) update the underlying data? Often, the first answer from the business stakeholders is “as frequently as possible.” The concept of “real time analytics,” with data being provided up-to-the minute, is usually quite attractive. But there may be some confusion about what this really means.

While the term real time analytics does refer to data which is frequently changing, it is not the same as simply refreshing data frequently. Traditional analytics packages which take advantage of data marts, data warehouses and data cubes are often collectively referred to as a Decision Support System (DSS). A DSS helps business analysts, management and ownership understand historical trends in their business, perform root cause analysis and enable strategic decisions. Whereas a DSS system aggregates and analyzes sales, costs and other transactions, a real time analytics system ingests and processes events. One can imagine a $25 million business recording 10,000 transactions a day. One can imagine that same business recording events on their website: login, searches, shopping cart adds, shopping card deletes, product image zoom events. If the business is 100% online, how many events would that be? The answer may astonish you.

Why Real Time Analytics?

DSS solutions answer questions such as “What was our net income last month?”, “What was our net income compared to the same month last year?” or “Which customers were most profitable last month?” Real time analytics answers questions such as “Is the customer experience positive right now?” or “How can we optimize this transaction right now?” In the retail industry, listening to social media channels to hear what customers are saying about their experience in your stores, can drive service level adjustments or pricing promotions. When that analysis is real-time, store managers can adjust that day for optimized profitability. Some examples:

  1. Social media sentiment analysis – addressing customer satisfaction concerns
  2. Eliminating business disruption costs with equipment maintenance analytics
  3. Promotion and marketing optimization with web and mobile analytics
  4. Product recommendations throughout the shopping experience, online or “brick and mortar”
  5. Improved health care services with real time patient health metrics from wearable technology

In today’s world, customers expect world class service. Implicit in that expectation is the assumption that companies with whom they do business “know them”, anticipate their needs and respond to them. That’s easy to say, but harder to execute. Companies who must meet that expectation need technology leaders to be aware of three concepts critical to making real time analytics a real thing.

The first is Internet of Things or IoT. The velocity and volume of data generated by mobile devices, social media, factory floor sensors, etc. is the basis for real time analytics. “Internet of Things” refers to devices or sensors which are connected to the internet, providing data about usage or simply their physical environment (where the device is powered on). Like social media and mobile devices, IoT sensors can generate enormous volumes of data very, very quickly – this is the “big data” phenomenon.

The second is Cloud Computing. The massive scale of IoT and big data can only be achieved with cloud scale data storage and cloud scale data processing. Unless your company’s name is Google, Amazon or Microsoft, you probably cannot keep up. So, to achieve real-time analytics, you must embrace cloud computing.

The third is Intelligent Systems. IBM’s “Watson” computer achieved a significant milestone by out-performing humans on Jeopardy. Since then, companies have been integrating artificial intelligence (AI) into large scale systems. AI in this sense is simply a mathematical model which calculates the probability that data represents something a human would recognize: a supplier disruption, a dissatisfied customer about to cancel their order, an equipment breakdown. Using real time data, machine learning models can recognize events which are about to occur. From there, they can automate a response, or raise an alert to the humans involved in the process. Intelligent systems help humans make nimble adjustments to improve the bottom line.

What technologies will my company need to make this happen?

From a technology perspective, a clear understanding of cloud computing is essential. When evaluating a cloud platform, CIO’s should look for breadth of capability and support for multiple frameworks. As a Microsoft Partner, BlumShapiro Consulting works with Microsoft Azure and its Cortana Intelligence platform. This gives our clients cloud scale, low cost and a wide variety of real time and big data processing options.

CIO Article 1

This diagram describes the Azure resources which comprise Cortana Intelligence. The most relevant resources for real time analytics are:

  1. Event Hubs ingest high velocity streaming data being sent by Event Providers (i.e. Sensors and Devices)
  2. Data Lake Store provide low cost cloud storage which no practical limits
  3. Stream Analytics perform in-flight processing of streaming data
  4. Machine Learning, or AzureML, supports the design, evaluation and integration of predictive models into the real-time pipeline
  5. Cognitive Services are out-of-the-box Artificial Intelligence services, addressing a broad range of common machine intelligence scenarios
  6. Power BI supports streaming datasets made visible in a dashboard context

Four Steps to Get Started with Real Time Analytics

Start with the Eye Candy – If you do not have a dashboard tool which supports real-time data streaming, consider solutions such as Power BI. Even if you are not ready to implement an IoT solution, Power BI makes any social media or customer marketing campaigns much more feasible. Power BI can be used to connect databases, data marts, data warehouses and data cubes, and is valuable as a dashboard and visualization tool for existing DSS systems. Without visualization, it will be very difficult to provide human insights and actions for any kind of data, slow or fast.

Get to the Cloud – Cloud storage costs and cloud processing scale are the only mechanisms by which real time analytics is economically feasible (for most companies). Learn how investing in technologies like Cloud Computing can really help move your business forward.

Embrace Machine Intelligence – To make intelligent systems a reality, you will need to understand machine learning technologies, if only at a high level. Historically, this has meant developing a team of data scientists, many of whom have PhD’s in Mathematics or Statistics, and open source tools like R or Python. Today, machine learning is much more accessible then it has ever been. AzureML helps to fast track both the evaluation and operationalization of predictive models.

Find the Real-Time Opportunity – As the technology leader in the organization, CIO’s will need to work closely with other business leaders to understand where real-time information can increase revenue, decrease costs or both. This may require imagination. Start with the question – what would we like to know faster? If we knew our customer was going to do this sooner, how would we respond? If we knew our equipment was going to fail sooner, how would we respond? If we knew there was an opportunity to sell more, how would we respond?

Berry_Brian-240About Brian: Brian Berry leads the Microsoft Business Intelligence and Data Analytics practice at BlumShapiro. He has over 15 years of experience with information technology (IT), software design and consulting. Brian specializes in identifying business intelligence (BI) and data management solutions for upper mid-market manufacturing, distribution and retail firms in New England. He focuses on technologies which drive value in analytics: data integration, self-service BI, cloud computing and predictive analytics

Power BI Demo CTA


Row Level Security in Power BI – Part 2

In Part 1 of this series, we discussed Row Level Security in Power BI, that it is different from RLS in SQL Server 2016, and then went on to demonstrate two simple scenarios where RLS can be used to filter data in a model based on Role assignment utilizing some DAX filter expressions. We introduced the USERNAME() DAX function and demonstrated its usefulness. In this second article, we’ll be diving a little deeper into RLS.

Row Level Security in Power BI Scenario 3:

Those first two scenarios from Part 1 were not that bad to implement. One line of a DAX expression and you’ve got a simple filter covered. But life seldom ever hands us a scenario that is so cut and dry. For this scenario, we’re going to add some requirements that might get handed down by the business users such as:

  • Ability to grant access to a group of countries, like continents, or regions.
  • Ability to have any one country be in multiple geo-political groups.
  • Ability to grant and revoke access to an entire group of people at once rather than needing to address each person individually
  • Ability to grant access on any one country to any one individual

For this we’re going to again go back to the database and create a few tables to help us. We already have the Country table so no need to do anything there. But we’ll add four more:

  • Users
  • Groups
  • User Group Membership
  • Country Group Membership

The T-SQL code for this can be found in the attachment, and a simple database drawing is shown here:

Row Security 3-1-1

We’ll also add some rows to the respective tables via basic INSERT statements:

For those of you who, like me, are full-blooded SQL nerds, I have included the CREATE TABLE, INSERT INTO, and CREATE VIEW scripts that can be executed in your database.  But we don’t need to import all four additional tables into your model, we simply need the distinct list of Users and the Countries to which each has access. This is represented in the [dbo].[SecurityQuery] view (Script #05). If you look at the view definition, note the DISTINCT key word in the SELECT clause, the optional fields to show the User’s Full Name and the Country Name, the absence of any fields from the [Group] or[ GroupUser] tables, and the WHERE clause at the bottom that filters for only Active records.

First, we’ll need to remove the Continent table from the model, then import the SecurityQuery dataset. Since this is not an exercise in how to import data, I’ll leave it up to you to get that done. And while you’re adding it, you might as well add the [SecurityReference] view as well and we’ll cover its usage at the end of the article.

Once you have the [SecurityQuery] in your model, it needs to be joined to the Country table, on Country code:

Row 3-1-2


Hint: If you have sufficient rows in the Security Query dataset, when you set up the relationship between [Country Code] in the Country table and the [Country Code] in the Security Query table, Power BI will recognize that this is a one-to-many relationship with the Country table being at the “one” side.

As before, we will use the Manage Roles button on the Modelling tab to configure the DAX Filter for the Roles. We’ll start with what we learned from before:  a filter on the [UserName] field and making use of the USERNAME() DAX function as before:

Row 3-1-3

But if we were to browse the model now for any one member in our set, we’ll see that he or she is limited in the rows returned by the Security Query, but NOT limited by rows returned in the Country table. Why not? It worked before. It is because the table on which the USERNAME() filter is applied is not at the top of the hierarchy, it’s at the bottom. To make things more complicated, there are two branches from the top of that hierarchy, one branch to sales, through the country table, and one branch directly to the security table. If only there was a way to filter the countries in that table based on those listed in the Security Query table. We need something more.

Enter the CONTAINS() DAX function. The description from MDSN is as follows:


CONTAINS(<table>, <columnName>, <value>[, <columnName>, <value>]…)

Return Value

A value of TRUE if each specified value can be found in the corresponding columnName, or are contained, in those columns; otherwise, the function returns FALSE.

If you’re like me, you’ll read that and think, “say WHAT?” Let’s implement it first, then we’ll sort out and explain how the parameters are used. Back under the Manage Roles, add a Filter to the Country table as the following (hint: it doesn’t matter which column you select as you will be replacing it with the entire text below):








Note: The line breaks are not required in this, but are added for clarity of reading.

Now here’s the layman’s description of how the DAX statement above works with the five parameters, in order of their appearance:

Go to the ‘SecurityQuery’ table (1st parameter), and in the column ‘SecurityColumn’[UserName] (2nd parameter), look for any rows that match the value returned by the function USERNAME() (3rd parameter).  Then for those rows, take the values in the column ‘SecurityQuery’[CountryCode] (4th), and see if those values exist in the column ‘Country’[CountryCode] (5th). If that [CountryCode] value is found return TRUE for that row, and allow it to be viewed in this context.

There are a couple of things to watch for when implementing this approach. First, you need to have a relationship defined between the Country table and the SecurityQuery table. Second, the <value> parameter (3rd parameter) of the CONTAINS() function can be just that, a single value, not a list or table. But the USERNAME() function fits this bill nicely. Third, you still need the [UserName] = USERNAME() security filter on the Security Query dataset.

If you followed the INSERT scripts included, you will recall that there were only two people involved: Fred and Bob. And you may recall that we granted Fred access to a Group called “Pacific Rim” (which included countries such as Japan, Hong Kong, and the Philippines) but that [GroupMembership] row was NOT flagged with [IsActive] = 0. Fred’s mapped data looks like this:

row 3-1-4


Now here’s where the [SecurityRefernce] dataset comes in. After adding it as part of the model, you need to make sure that A) it has NO relationships to any other tables and that B) it also has a [UserName] = USERNAME() DAX Filter expression applied.

I added a report page to my Power BI model and a simple Matrix visualization based on this table and configured it as follows:

  • Rows = [Country Name]
  • Columns = [GroupName]
  • Values = [IsActive], with a SUM aggregation
  • For the [IsActive], I set up Conditional Formatting with the following properties

row 3-1-5

The resulting visualization is a nice way to see exactly HOW a person has gotten access to any one particular country (via membership in which Group), and what countries are in any Group they have are also a member of, even if that membership is not active.

Row 3-1-6


It is true that there is a lot introduced here that may not specifically be “Row Level Security” stuff, but rather T-SQL overhead. After all, all you really need for this 3rd scenario to work is the distinct list of Users and Countries that are allowed access. But given the 173 countries in the world and say, two dozen people to which access control is required, that’s potentially up to 4,000 rows of data controlling who has access to what. Breaking it into Users and Groups and Memberships is a way to manage the mess.

About Todd: Todd Chittenden started his programming and reporting career with industrial maintenance applications in the late 1990’s. When SQL Server 2005 was introduced, he quickly became certified in Microsoft’s latest RDBMS technology and has added certifications over the years. He currently holds an MCSE in Business Intelligence . He has applied his knowledge of relational databases, data warehouses, business intelligence and analytics to a variety of projects for BlumShapiro since 2011. 

Power BI Demo CTA