Real time analyses for the Internet of things (IoT)
Based on current forecasts, by the year 2020, 24-30 billion electronic devices will be connected to the Internet. Deployed into this IoT scenario, Stream Analytics allows you to analyse queries in real-time for data streams generated by smartphones, consumer electronics, sensors, websites, social media, infrastructure systems and connected cars.
With Stream Analytics, decisions are no longer based on human interpretations and evaluations, but on automated, real-time calculations. Furthermore, considerably more information and thus more useable knowledge are taken into account, because data from all available internal and external sources (so-called data silos) are brought together and combined (correlation-based Big Data analyses).
Where is Stream Analytics used?
- Implementation of real-time analyses for Internet of Things (IoT) solutions
- Implementation of mission-critical forecast results with a high degree of speed and reliability (stream of millions of events per second)
- Generation of real-time dashboards and alerts using data generated by devices and applications
- Using machine learning spot pattern recognition undetectable by humans, (e.g. for real-time fraud detection)
- Personalised stock trading analyses and alarms as offered by financial institutions
- Warnings in CRM programs if customers do not respond within a given time window
In order to be successful in highly competitive markets, companies need a flexible, reliable and cost-effective way to perform such event-driven real-time streaming data analyses themselves.
Real-Time versus Streaming Analytics
It is important to distinguish between "real-time analytics" and "Streaming Analytics".
Normally, "real-time business analytics" refers to a system that provides answers within a certain narrowly defined time frame. Exchange systems are a classic example of these types of systems. Results are delivered within a guaranteed response time or as a result of a fixed deadline.
Streaming Analytics, however, is different. Stream processing refers to the on-going calculation of data that is continuously flowing through a system. Unlike "real-time analysis", there are no specific deadlines or tolerances used in processing the stream. The output of a stream processing system does not have a fixed time frame.
Analytics for the IoT era
When working with IoT analytics, real-time Streaming Analytics plays a central role in the overall process. IoT analytics requires actions that record events in real-time, correlate different data streams and compare streams with historical values and models. It detects anomalies, transforms incoming data, triggers an alarm when a specific error or state occurs in the stream and displays this real-time data in a dashboard.
Scenario 1: Real-time fraud detection in telecommunications
Telecommunications companies have to deal with a large volume of incoming call data. A typical profile of requirements for a fraud detection analytics system might look like this:
- Reduce the amount of incoming data to a manageable quantity and provide customer insights based on their usage time and geographical region
- Recognise SIM card fraud in real-time (e.g. simultaneous calls from the same person, but in different geographical regions) so that the customer can be informed quickly or the service can be switched off immediately
The technology for such real-time anomaly detection as used in this telecommunications example would be equally suitable for detecting fraud such as you might find in credit card or identity theft scenarios.
Scenario 2: Analysing process data from IoT devices
Market-ready Industry 4.0 applications such as Predictive Maintenance have strongly placed the manufacturing industry's focus on production IT: machines and production equipment that independently exchange information, control each other, and provide independent maintenance make factories smarter. Take, for example, an industrial automation company that has completely automated its manufacturing process. Its machines have sensors that record data streams in real time. In this scenario, a production manager needs real-time views of the sensor data so he can search for patterns and react accordingly. This is where Pentaho comes in, because it can be used to find interesting patterns in the incoming data stream.
Companies that can react to the incoming flood of data in real time dramatically improve their efficiency and distinguish themselves on the market. Pentaho Stream Analytics gives you this power, because it can process millions of events per second.
By combining descriptive and predictive analyses with prescriptive analyses (and starting from a predefined goal), the necessary measures to achieve these goals can be automated using Streaming Analysis.
The results of these exact, automated real-time decisions are independently integrated into the intelligent front ends and operational processes by the system. Machine learning based on, for example, Apache Spark, allows continuous optimisation of processes in all conceivable areas – completely without any human intervention, if desired.
The combination of streaming analytics in real-time with predictive analytics based on historical analyses allows companies to make decisions and take actions that dramatically increase the business value of IoT use cases.
The high volume of real-time data provides great potential for new insights into business processes that can be used to increase revenue, reduce costs and risks and increase operational efficiency.