CEP/ESP Technology, IoT and Cloud

When I got invite for PUG Reboot event this weekend, I registered myself quickly-mainly because I was interested in 2 of the 4 sessions of track#1. Track#2 was focused Visual Studio and related topics. Track#1 has sessions involving Azure. The two sessions I was interested in learning more about DevOps kind of features using PowerShell workflow introduced in Azure recently. And the other was about certain new features around supporting IoT and complex event processing(CEP)/event stream processing(ESP) applications. I had particular interest in understanding more on the later topic because of my stint on CEP/ESP sometime back using open source technologies.

The first session was on Azure as IaaS platform which was essentially an intro to many features of a typical IaaS platform and its capabilities. The second one was on Azure automation, PowerShell and desired state configuration. It talked about PowerShell Workflow feature, to automate many DevOps tasks for your application hosted on Azure. The similar features have been available on Puppet and similar technologies for some time now. The advantage of this feature in Azure over Puppet kind of tools is that, it not only allows automation configuration management related use cases but also monitoring, log processing etc.

The third session was of my most interest. The topic was on Azure’s support for CEP/ESP kind of applications(which are primarily arising due to IoT possibilities). During my consulting assignments few years back, I was involved in a project where we were looking to develop CEP/ESP platform for water utilities domain where in IoT kind of applications such as intelligent sludge management, waste and waste water treatment plant management, intelligent and predictive operations for water utilities companies.The key thing in such applications is huge number of events which is generated by various sources which needs to be analyzed, processed and used for patterns detection, actionable analytics. So the velocity, variety and volume of events is under consideration in such cases. The IoT is involved here in a sense, as events are typically generated by telemetry devices. For example, in water utilities domain, sensors, flow meters, GPS devices on fleet of trucks moving sludge, sensors and other telemetry devices in process equipment in treatment plants etc. The infrastructure required for processing large volume of events was least of the concerns during our initial prototyping and design. We were more focused on event gathering, pre-processing, processing using tools such as Esper/Drools, machine learning algorithms for analytics and visualization framework. What cloud service providers such as Azure have done is they have solved problem of infrastructure and ecosystem required for ultra-fast processing of events by way of providing services such as Event Hub and Stream Analytics services. Of course, storage of events data for archival and historical trending is anyways part of any cloud infrastructure. All in all, this make much more attractive from time to market perspective on IoT applications such as what I was talking about water utilities domain. This is shown in below diagram from Azure documentation.


Now IoT application developers can focus more on processing, analytics logic part of the puzzle. The possibilities are immense using modelling, simulation, optimization and combining that with big data analytics techniques for business benefits.

A side note: It is interesting to note how technologies involve and also morph under a new terminologies and names. In the past we used to have data acquisition applications, SCADA to handle, and derive actionable insights from data generated by various sensors on a manufacturing plant or any other industrial setting.  This is now called at Industrial IoT(IIoT), of course, now some data is coming from Internet enabled devices(not necessarily from one manufacturing plant).

Update on Azure: Got to read just now, that Microsoft is acquiring a data science company(which produced commercial versions of software used for statistical programming called R) to further its machine learning features on Azure.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s