Archive for Digital Transformation

Technology Talks Podcast: Introduction to Blockchain

Listen to our podcast, Technology Talks, hosted by Hector Luciano, Consulting Manager at BlumShapiro Consulting. Each month, Hector will talk with about the latest news and trends in technology with different leaders in the field.

In this episode, Hector provides an introduction to blockchain technology, the connections to bitcoin and how this technology will have an impact on business going forward. Listen to previous episodes of Technology Talks on our SoundCloud page.

Create a Pareto Chart in Power BI

“Baseball is 90% mental, and the other half is physical.” – Yogi Berra

You just have to love Yogi Berra quotes like this. We all pretty much know what he’s talking about, even if his math is not spot on. It’s a restatement of the Pareto Principle, the 80/20 rule! It applies to just about anything in life or business. If I had to write a definition of it for technical documentation, it would look something like this:

“A situation where eighty percent of events attributed to a group are caused by twenty percent of the members of the group.”

Re-stated as examples:

  • Eighty percent of your human resource issues are caused by twenty percent of your employees.
  • Eighty percent of your maintenance issues are caused by twenty percent of your equipment.
  • Eighty percent of your sales are attributed to twenty percent of your products.
  • Eighty percent of the wealth is controlled by twenty percent of the population.

 

(And I’m certainly not in that last twenty percent. If I was, I wouldn’t have to write articles like this one!)

Now that we’ve got an understanding of the principle, let’s look at how it can be visualized. Excel has a very simple wizard for creating a Pareto Chart that can be found on the Insert menu:

But we want one of these in Power BI. And Power BI doesn’t have one (yet, maybe later). We’ll need to ‘roll our own’.  Let’s discuss the various parts of the chart itself, so we know what we’re shooting for.

  • The categories, or series, at the bottom (One, Two, Three, etc.) represent the different members of the ‘group’ we are trying to analyze. They may be employees, machines on the manufacturing line, or products in our catalog.
  • The blue bar above each ‘member’ is its respective measurement (count of HR issues, money spent on maintenance, annual sales, etc.). The scale of this measurement is on the left side, in our case going from 0 to 120.
  • The final element is the curved line and the right-side scale measuring from zero to one hundred percent. This represents, at each category member, the percentage of the cumulative total of all members to the left of the member in question, inclusive. To put it another way, as we add each category’s number to the running total of those on its left, the line represents that running total divided by the entire total for all members of the category.

The arrow points to the spot on the line where it crosses 80%, in our case, after about the first four members, as can be seen by following the green dashed line from right to left, then down. The first four members would be the ‘twenty percent’ of the Pareto Principle, and their cumulative measure would be the eighty.

Note: Math wizards may point out that four members divided by a total member count of fifteen is closer to thirty percent than twenty, but remember that this is a rule of thumb, and we all know that some thumbs are bigger or smaller than others.

To plot some data in a Pareto Chart, we’ll need a couple of pieces of information from it:

  • Each member’s respective total
  • The grand total
  • The running total at each member, sorted from largest to smallest
  • The percentage that running total represents compared to the grand total

Now that we understand what we’re shooting for, let’s get started.

If your data includes a running sum of the measurement for each member, sorted by the respective member’s measurement, then you’re golden and can skip to the section titled Add the Grand Total and Running Percent. Your data may include a Ranking column so you may be able to skip the respective steps in each of the following two sections. For the rest of you, keep reading. We’ll look at two approaches to getting the intermediate bits of data: Power Query (M), and DAX.

Create the Rank and Running Sum in Power Query

Let’s start with some simple data in Excel, in fact the same data used to generate the Excel Pareto chart we used to explain the concepts:

We’ll load this data (it’s in an Excel table called “Table1”) and edit it in the Power BI Query Editor. First, we need to sort the data by the [Measure] column, sorted descending. Click the down-arrow next to the Measure column title and select Sort Descending.

Next, on the Add Column menu, select Index Column. Keep the defaults of Starting Index of 1 and Increment of 1.

I renamed my column to [Power Query Rank] to differentiate it from ranking step we’ll introduce in the model later via DAX.

Next, we’ll add the running total as a Custom Column with a formula as shown below:

Hint: If you can’t read the formula from the screen shot, it is:

= Table.Range ( #”Renamed Columns”, 0, [Power Query Rank] )

Attribution should go to Sam Vanga and SQL Server Central for this bit of M code: http://www.sqlservercentral.com/blogs/samvangassql/

The Power Query function Table.Range can be explained like this: Given a table of data, in our case the last of our query steps, a.k.a. #”Renamed Columns”, start at the 0 row (top), and go down the number of rows represented by the value in column [Power Query Rank]. The result is a table associated with each row in the query. The first row of the query has a table with one row of data in it. The second row has a table with two rows, and so forth. This table is represented by the word “Table” on each row of the column we just added.

From here, click on the ‘expand’ arrow in the column header and select the Aggregate radio button, check off the “Sum of Measure” column, and un-check “Use original column name as prefix”:

I renamed the resulting column [Power Query Running Total] (not shown).

Click Close and Apply on the Home menu.

Create the Rank and Running Sum in DAX

As with all things Microsoft, there is more than one way to accomplish a goal. In our case, the goal is to get the running total, and just like before, we’ll need the ranking first. For this exercise, we’ll be using DAX instead of Power Query, but should get the same results.

Create a column with the formula as follows:

DAX Rank = RANKX (All ( Table1 ), [Measure] )

Next, create the DAX Running Total measure as:

DAX Running Total =

CALCULATE (

SUM ( Table1[Measure] ),

FILTER (

ALLSELECTED ( Table1 ),

Table1[DAX Rank] <= MAX ( Table1[DAX Rank] )

)

)

This DAX formula does pretty much the same thing as the Power Query Range.Table function above, the only difference is that it includes the aggregate within, eliminating the need for an extra column.

Note: Know the difference between Columns and Measures in DAX. Mistaking the two will cause error, frustration, and hair loss.

Plotting all these columns and measures on a simple table visual shows that Power Query and DAX come up with the same answers for Rank and Running Total, a good sanity check. Also, the ranks are easy to verify as to accuracy, and with a little mental math, running totals are as well. I had to re-format some of the numbers to make them show without decimals.

Add the Grand Total and Running Percentage

There’s two more pieces we need: [Grand Total] which is self-explanatory, and [Running Percent], which is the ‘percentage of the [Running Total] compared to the [Grand Total]’. These can only be done in DAX. Add a measure as follows:

Grand Total = CALCULATE ( SUM ( Table1[Measure] ) , ALL ( Table1 ) )

This calculates the Grand Total and makes it available at every slice (row of each Member).

Now add the last item, a column with the expression:

Running Percent = [Power Query Running Total] / [Grand Total]

Or:

Running Percent = DIVIDE ( [Power Query Running Total] , [Grand Total] )

Note: The column [DAX Running Total] would work just as well as its Power Query equivalent since we know it has the same number.

Format this last one as a percent.

Create the Chart

Now the fun part. For this we’ll need either a “Line and Stacked Column Chart” or a “Line and Clustered Column Chart”. This is the easiest part of the whole exercise:

  • The Shared Axis is the [Member] column (“One”, “Two”, “Three”, etc.)
  • The Column values is the [Measure] column
  • The Line values in the [Running Percent] column

Like I said, simple if you have all of the data pieces in front of you.

Need help getting the right data pieces? Not sure what charts you can generate from the data pieces you have? There’s probably a way to get to where you want to be. Reach out to our team of data scientists at BlumShapiro Consulting to learn more about how data can help guide your organization into the future.

Technology Talks Episode 5: Digital Transformation’s Impact on the Manufacturing Industry

Listen to our new podcast, Technology Talks, hosted by Hector Luciano, Consulting Manager at BlumShapiro Consulting. Each month, Hector will talk about the latest news and trends in technology with different leaders in the field.

In this episode, Hector speaks with Janet Prisloe, Partner at BlumShapiro about the impact of the 4th Industrial Revolution and Digital Transformation on the manufacturing industry.

Listen to our previous episodes on our SoundCloud page >>

 

Send Custom Emails Using Azure Functions as Scheduled Tasks in SharePoint Online

Recently, a client of ours was looking to have a daily email sent to each user that has an overdue task, or a task that is set to expire. The email had a specific format. Each task needed direct links and had to be grouped by the date that it was due. Since it was an on-premises Project Server and SharePoint 2013 farm, it was not too difficult to build and embed the solution into the farm. I took care of this by building a SharePoint timer job, which leveraged a SharePoint search to retrieve all tasks where the due date was before the next business day. Once deployed, and activated, this timer job was automatically scheduled to run every morning, and the SharePoint admins could trigger it manually.  Everything worked great.

Another client of ours was looking for a solution exactly like this, except they were strictly SharePoint Online / Project Online. They had no on-premises farm, there were no real servers to even speak of. One option would have been to create a PowerShell script or .NET executable to run the code, and have that process run as a Scheduled Task on some server. However, there were no servers. Even if they did, what was the point of being in the cloud, if you are still stuck with a foot (or process) on the ground?

So, I turned to Microsoft Azure, and that’s where Azure Functions came into play.

Azure Functions

Azure Functions are programs or code snippets that run in the cloud. They can run on schedules or can be triggered by different types of events (HTTP request, item added to Azure Blob Storage, Excel file saved to OneDrive, etc.). Think of this as a Windows Scheduled Task that can be triggered by modern events and activities.

The programs or code snippets can be created and edited within the Azure Portal, making it easy to get up and running with an Azure Function. The languages supported by Azure Functions are more than adequate: PowerShell, C#, JavaScript, F#, Python, PHP, Bash, and Batch.

Note that I could have also used Azure WebJobs to accomplish this, but I felt that Azure Functions had many positives. Azure Functions are easy for the client to maintain, it has automatic scaling, they only pay for how long the code executes, it supports WebHooks and can be easily triggered via an HTTP request.

Send Custom Emails from SharePoint Online

For this solution, I created the Azure Function in Visual Studio pre-compiled with the SharePoint Online client-side object model (CSOM) DLLs. The solution was straightforward, as it would use CSOM to query SharePoint Online’s search service for all overdue tasks and tasks due within the next business day. It would then do some logic to build the email content based on the assigned users, and then send out emails using SendGrid. SendGrid is built into Microsoft Azure, so configuring it was a breeze, and you get 25,000 free emails a month!

Once deployed, I configured the Azure Function to run on schedule (like before), and it can even be triggered by an HTTP request, so putting an HTTP request in a SharePoint site workflow or Microsoft Flow means that any site user would be able to trigger this function as needed.

Long gone are the days where there are integration servers laying around in the data center waiting to get more processes to help them consume more of their over-allocated resources. Most servers, virtual machines, really, are now dedicated to a specific application, and shouldn’t share their resources with one-off processes.

Azure Functions is a great server-less architecture solution to these problems. Whether you need to send emails, calculate metrics, or analyze big data, Azure Functions can be a solution for you. Learn more about how BlumShapiro can help your organization with issues like this.

About Brent:

Brent

Brent Harvey has over 10 years of software development experience with a specific focus on SharePoint, Project Server, and C #and web development. Brent is an Architect at BlumShapiro Consulting. Brent is a Microsoft Certified Solutions Expert in SharePoint 2013, Solutions Associate in Windows Server 2012, Specialist in Developing Azure Solutions and Professional Developer in SharePoint 2010.