Archive for Machine Learning

Using Real Time Data Analytics and Visualization Tools to Drive Your Business Forward

Business leaders need timely information about the operations and profitability of the businesses they manage to help make informed decisions. But when information delivery is delayed, decision makers lose precious time to adjust and respond to changing market conditions, customer preferences, supplier issues or all three. When thinking about any business analytics solution, a critical question to ask is: how frequently can we (or should we) update the underlying data? Often, the first answer from the business stakeholders is “as frequently as possible.” The concept of “real time analytics,” with data being provided up-to-the minute, is usually quite attractive. But there may be some confusion about what this really means.

While the term real time analytics does refer to data which is frequently changing, it is not the same as simply refreshing data frequently. Traditional analytics packages which take advantage of data marts, data warehouses and data cubes are often collectively referred to as a Decision Support System (DSS). A DSS helps business analysts, management and ownership understand historical trends in their business, perform root cause analysis and enable strategic decisions. Whereas a DSS system aggregates and analyzes sales, costs and other transactions, a real time analytics system ingests and processes events. One can imagine a $25 million business recording 10,000 transactions a day. One can imagine that same business recording events on their website: login, searches, shopping cart adds, shopping card deletes, product image zoom events. If the business is 100% online, how many events would that be? The answer may astonish you.

Why Real Time Analytics?

DSS solutions answer questions such as “What was our net income last month?”, “What was our net income compared to the same month last year?” or “Which customers were most profitable last month?” Real time analytics answers questions such as “Is the customer experience positive right now?” or “How can we optimize this transaction right now?” In the retail industry, listening to social media channels to hear what customers are saying about their experience in your stores, can drive service level adjustments or pricing promotions. When that analysis is real-time, store managers can adjust that day for optimized profitability. Some examples:

  1. Social media sentiment analysis – addressing customer satisfaction concerns
  2. Eliminating business disruption costs with equipment maintenance analytics
  3. Promotion and marketing optimization with web and mobile analytics
  4. Product recommendations throughout the shopping experience, online or “brick and mortar”
  5. Improved health care services with real time patient health metrics from wearable technology

In today’s world, customers expect world class service. Implicit in that expectation is the assumption that companies with whom they do business “know them”, anticipate their needs and respond to them. That’s easy to say, but harder to execute. Companies who must meet that expectation need technology leaders to be aware of three concepts critical to making real time analytics a real thing.

The first is Internet of Things or IoT. The velocity and volume of data generated by mobile devices, social media, factory floor sensors, etc. is the basis for real time analytics. “Internet of Things” refers to devices or sensors which are connected to the internet, providing data about usage or simply their physical environment (where the device is powered on). Like social media and mobile devices, IoT sensors can generate enormous volumes of data very, very quickly – this is the “big data” phenomenon.

The second is Cloud Computing. The massive scale of IoT and big data can only be achieved with cloud scale data storage and cloud scale data processing. Unless your company’s name is Google, Amazon or Microsoft, you probably cannot keep up. So, to achieve real-time analytics, you must embrace cloud computing.

The third is Intelligent Systems. IBM’s “Watson” computer achieved a significant milestone by out-performing humans on Jeopardy. Since then, companies have been integrating artificial intelligence (AI) into large scale systems. AI in this sense is simply a mathematical model which calculates the probability that data represents something a human would recognize: a supplier disruption, a dissatisfied customer about to cancel their order, an equipment breakdown. Using real time data, machine learning models can recognize events which are about to occur. From there, they can automate a response, or raise an alert to the humans involved in the process. Intelligent systems help humans make nimble adjustments to improve the bottom line.

What technologies will my company need to make this happen?

From a technology perspective, a clear understanding of cloud computing is essential. When evaluating a cloud platform, CIO’s should look for breadth of capability and support for multiple frameworks. As a Microsoft Partner, BlumShapiro Consulting works with Microsoft Azure and its Cortana Intelligence platform. This gives our clients cloud scale, low cost and a wide variety of real time and big data processing options.

CIO Article 1

This diagram describes the Azure resources which comprise Cortana Intelligence. The most relevant resources for real time analytics are:

  1. Event Hubs ingest high velocity streaming data being sent by Event Providers (i.e. Sensors and Devices)
  2. Data Lake Store provide low cost cloud storage which no practical limits
  3. Stream Analytics perform in-flight processing of streaming data
  4. Machine Learning, or AzureML, supports the design, evaluation and integration of predictive models into the real-time pipeline
  5. Cognitive Services are out-of-the-box Artificial Intelligence services, addressing a broad range of common machine intelligence scenarios
  6. Power BI supports streaming datasets made visible in a dashboard context

Four Steps to Get Started with Real Time Analytics

Start with the Eye Candy – If you do not have a dashboard tool which supports real-time data streaming, consider solutions such as Power BI (link to our web page). Even if you are not ready to implement an IoT solution, Power BI makes any social media or customer marketing campaigns much more feasible. Power BI can be used to connect databases, data marts, data warehouses and data cubes, and is valuable as a dashboard and visualization tool for existing DSS systems. Without visualization, it will be very difficult to provide human insights and actions for any kind of data, slow or fast.

Get to the Cloud – Cloud storage costs and cloud processing scale are the only mechanisms by which real time analytics is economically feasible (for most companies). Learn how investing in technologies like Cloud Computing can really help move your business forward.

Embrace Machine Intelligence – To make intelligent systems a reality, you will need to understand machine learning technologies, if only at a high level. Historically, this has meant developing a team of data scientists, many of whom have PhD’s in Mathematics or Statistics, and open source tools like R or Python. Today, machine learning is much more accessible then it has ever been. AzureML helps to fast track both the evaluation and operationalization of predictive models.

Find the Real-Time Opportunity – As the technology leader in the organization, CIO’s will need to work closely with other business leaders to understand where real-time information can increase revenue, decrease costs or both. This may require imagination. Start with the question – what would we like to know faster? If we knew our customer was going to do this sooner, how would we respond? If we knew our equipment was going to fail sooner, how would we respond? If we knew there was an opportunity to sell more, how would we respond?

Berry_Brian-240About Brian: Brian Berry leads the Microsoft Business Intelligence and Data Analytics practice at BlumShapiro. He has over 15 years of experience with information technology (IT), software design and consulting. Brian specializes in identifying business intelligence (BI) and data management solutions for upper mid-market manufacturing, distribution and retail firms in New England. He focuses on technologies which drive value in analytics: data integration, self-service BI, cloud computing and predictive analytics

Power BI Demo CTA

 

Face API and Power BI

At last week’s Build2015 developer conference, Microsoft demonstrated many great new tools. One demo which got quite a bit of attention was the How Old Am I? app (http://how-old.net) The demo allows users to upload pictures and let the service “guess” the age and gender of the individuals in the photo. Within a few hours, the demo went viral, with over 210,000 images uploaded to the site from all over the world. The result was a dashboard of requests from all over the globe.

Power BI

This solution shows off the use of a number of powerful technologies.

Face APIProject Oxford is a set of Artificial Intelligence API’s and REST services which developers can use today to build Intelligent Systems. In addition to Facial Recognition, the Project Oxford AI services include Speech Recognition, Vision (or Image Recognition and OCR), and Language Understanding Intelligent Services – leveraging the technology capabilities of Bing and Cortana.

Azure Event Hubs –  a highly scalable publish-subscribe ingestor that can intake millions of events per second, the Event Hubs API is used to stream the JSON document from the web page when the user uploads a picture.

Azure Stream Analytics – a fully managed low latency high throughput stream processing solution. Azure Stream Analytics lets you write your stream processing logic in a very simple SQL -like language.   This allows the solution to take measurements every 10 seconds of how many requests, from which countries, of which gender and age.  These measurements become Facts for your analysis.

Power BI – choose PowerBI as the output of our stream analytics job (click here to learn how). Then the team went to http://www.powerbi.com, and selected the dataset and table created by Azure Stream Analytics. There is no additional coding needed to create real time dashboards.

The only down side to this is that my worst fears have been confirmed – I look older than I actually am by over 10 years! 🙁
How old do I look?!?!

Fun with Machine Learning

All my blog articles over the years have been technical in nature. I decided to break out of that mold today. I almost titled this article “It’s not a train robbery, it’s a science experiment” (Doc Brown, in Back to the Future III). I hope you enjoy reading it as much as I did writing it.

The title is not meant to imply that machine learning isn’t inherently fun (I personally happen to think it’s a cool use of aggregated technologies). Rather it’s to say that we’re going to have some fun with machine learning in a way you wouldn’t have otherwise considered. But in order to do so, the reader must understand at least the fundamental concepts of machine learning. Don’t worry, we’re not going to be diving into data mining algorithms or the R language or python code or anything remotely technical. Instead, a real life analogy is best, and we’ll dumb this one right down to the level of a two-year-old toddler! Kids between the ages of about one and six are GREAT at ‘machine learning,’ but NOT in the LEARNING side of machine learning. No, they’re on the TEACHING side of machine learning, the ‘writing of the algorithms’, the ‘Python and R code’, that the ‘machines’ (their parents) use to learn. Let’s take a look at how this works.

Ever try to get a two-year-old to eat something he or she just does NOT want to eat? Like broccoli or cauliflower? Even adults are split about evenly on the likes and dislikes of vegetables. Two-year-olds, on the other hand, tend to swing to the dislike side on just about all varieties. So what happens? The child absolutely will not eat said vegetables. Babies and toddlers being spoon-fed from a jar tend to take a different and sometimes visually humorous approach: they let you spoon it into their mouth, but it quickly comes back out like toothpaste accompanied by a grimace. Having wasted an entire jar of baby food on the bib, the father (as a new father I had to take my turn feeding the kids!) turns to his wife and says, “Honey, he doesn’t like the green beans, but he loves the applesauce.” “OK,” comes the reply, “I won’t buy the beans again.”

What just happened here? Believe it or not, that was “machine learning” on a micro scale. The ‘machine’, the parents, just ‘learned’ something. Two data points, in fact. Green beans are icky, while applesauce gets a ‘thumbs up.’ Now if all the toddlers in the town were to teach those various bits of knowledge to their respective parents, you have just built yourself a ‘reference dataset.’ Suppose now a bunch of those mothers interact at the weekly “Mommy and Me”. Just now joining their group is a new mother whose daughter is ready to switch from the bottle to semi-solid food. The discussion is likely to descend around what each child likes and dislikes in that area. The new mother listens intently and comes away with knowledge of what her daughter is MOST LIKELY to prefer, but WITHOUT actually having to experience a bib full of pureed sweet potatoes! This is machine learning in action. The machine has applied an algorithm to a reference dataset to predict a probable outcome.

Now, no child dislikes ALL foods, even three and four-year-olds, as much as some parents tend to perceive. (My six-year-old son wouldn’t eat a peanut butter and honey sandwich unless it was cut diagonally! Go figure!) If you think your child dislikes ALL foods, it’s more likely he or she only dislikes all the foods YOU like. Since you’re not likely to buy stuff you personally wouldn’t eat, the child has no chance to find what he or she actually enjoys. The parents will then broaden their variety to find something acceptable.

Let’s take a look at another real world scenario, this time closer to the topic at hand.

Many on-line retailers use machine learning and data mining to present to the consumer things they are MOST LIKELY to purchase based on any number of information points and reference datasets. These include your past purchases, your demographics, and the things other consumers have purchase together. The algorithms employed can be ‘market basket analyses’, ‘clustering’, or others (and I promise that’s as technical as we’ll get in this article). We’ve all seen it in action at Amazon and Netflix. “Based on your viewing history…” or “People who bought X also bought…” Even grocery stores learned that beer was often purchased in conjunction with diapers. Seems that young mothers often sent their husbands to the store in times of diaper needs, hence the beer.

I decided to try an experiment this morning, and this is where the fun comes in. I wanted to take a finicky two-year-old’s stance on my internet steaming audio. Pandora, Rhapsody, iHeartRadio and the like often apply machine learning type of logic to decide the next song to queue to your personal listening stream based on your likes and dislikes. What would happen if I started a new ‘radio station’, then flagged every song it presented to me as ‘thumbs down?’ Would it just keep letting me spit out the offerings until it found something I actually liked? What if I didn’t like ANYTHING? Would it cut me off and kick me out for being impossible to please? I decided I just had to find out.

I started by naming my new station “Billy Joel.” (Hey, if the experiment were to fail, I figured why not fail with something decent!) Within 5 seconds starting the first song, I had hit the ‘Thumbs down’ button. OK, no, problem, it moved on to the next. Six more songs were dispatched in similar fashion. “Hey, this is fun” I thought. On the next song, however, it allowed me to dislike it, but I was forced to listen to the entire song while a banner displayed the message about not being fed that particular vegetable variety again. Five more disliked songs all brought up the same message while still playing the song to completion. Oh, well, at least I had some good music to listen to. After a dozen similar results, and realizing I wasn’t getting anywhere trying to fool the machine, I threw it a curve and hit the ‘Thumbs up’ on the next few tracks. (I think I smelled smoke coming from my router.) The next six tracks were all skipped by flagging as disliked in similar fashion to the first batch. I settled into a back-and-forth of liking and disliking bunches of songs in groups. In the end, I had to like at least a couple of songs it presented to me before I could dislike AND SKIP a bunch of other tracks.

After two hours the machine won as I had to produce some useful work at the office. There was a practical limit to how much it could ‘learn’ from this picky two-year-old music consumer. Likewise, parents all think they win in the end, too, or do they? They will tell you they eventually ‘got their child to like’ certain foods when in fact they simply settled on a repertoire of foods that their child wouldn’t reject, kind of like…wait for it…machine learning.