IOT Data needs to be SMART, not Big

 

 Two of the most over hyped concepts out there right now are IOT and Big Data. Every time I hear people using the term Big Data I shudder. They have no idea what they mean and no idea what they are going to do with it. All they know is that Gartner and others tell them it will save the world and create universal peace and prosperity. Yeah, right… And fast food isn’t fattening?

Now mix IOT in with this. I have seen and read so much rubbish that the veritable explosion of devices ^ is going to create so much data that we will be swamped. I don’t know what this means: will Western civilisation collapse or if we yawn long enough, will it just go away? Or will we need gum boots to push our way through?

But here is the problem and yes, IOT is the cause. A whole bunch of companies are racing off to build devices that are going to project data into something somewhere that someone somehow is going to visualise and goggle at endlessly. Apparently. So yes, I can set 1000 sensors on 50 farms to start sending data off to some cloud based storage somewhere. Repeated thousands of times over everywhere in the world, these little sensors are going to start pumping data at some massive rate (seconds? minutes? hours?) into cloud storage. How incredibly stupid and how incredibly wasteful.

We need SMART data, not big data and anybody who tells me they are going to create tonnes of data from their remote monitoring solutions is confirming their complete lack of credibility in this space. For data to be useful it needs to be relevant and accessible. Terabytes of unstructured drivel is still drivel. So in order to save the world from collapse we have worked on a SMART data solution:

Sparsely MAnaged Relevant Transactions

First, we need to be SPARSE.

Let me give you an example. You are monitoring an airlock and you need to know when a door is opened and how often. You might even want to know how long it is open for bio-security rating purposes. Your sample frequency is per second because the open door will trigger alerts. So if the door is shut, I don’t need to be told that over and over. As long as a heartbeat exists, I only want to know when the state changes – ie the door opens or closes. Another example: a cool room temperature which we monitor once a minute. Under normal conditions, the room will chill to within 1.5C and 4C. if the room temp doesn’t vary more than (say) 0.5C then I don’t want to know about it. Don’t send the data! It can easily be extrapolated at the storage end without fuss or complexity. Save the bandwidth and storage cost.

Secondly, the process needs to be MAnaged

Data has to be controlled at the source. It is too late once it gets into the hubs and storage pools. This is my problem with most LoRA solutions. They just pump data regardless because that is all they can do. There needs to be some intelligence at the device in order restrict outgoing packets to be sparse data and not big data. My contention is that the sending gateway must have application code and an operating system in order to manage what ends up in our IOT solution space. Managed devices are just as important to bandwidth and storage control as sparse data protocols are.

Thirdly, data needs to be Relevant

We must not send data just because we can. That is just vain: the thought that somebody one day might want it. Phooey. We need a reason for the data we capture if for no other reason than there is a cost to storage and a cost for processing. Let’s go back to our cool room example. We know that the thermostat should set the room to 1.5 – 4C but it is misbehaving. The repair man decides to change the precision to 0.1C and get readings every second instead of every minute. He wants to watch how fast the compressor pulls down the room and how fast it warms up. You see, we don’t really need this data for the last six months, just now when a problem has been detected. Once the problem is fixed I revert back to minute readings on a lower precision. We could always collect this data but that just ignores the cost and performance hits we suffer as a consequence. We need to collect relevant data, not everything just because we can. This is the bower bird syndrome on steroids. One more comment: this process of changing the capture parameters could be automated, if you think about it, assuming the device is managed. The repair man will take a profile of the data so that he can review comparative performance down the track. It is idle wastefulness to collect that data continually. It also means you don’t understand your system’s basic metrics.

Finally, we must think of our data as being Transaction based

So what do I mean by that? Not only should data be sparse, managed and relevant but it should be part of some logical business process. It should belong to a business transaction, not just recorded ‘because’. For example, if I am monitoring the pressure differential across a filter system, the differential value will have levels to create alerts which are aimed at advising me when to replace the filters. Transactions accumulate into trend sets and usually have events that are generated from level settings. Yes, it might be interesting to watch the pressure differential but it isn’t useful if it cannot result in a practical transaction of some kind. The important outcome here is how often I have to replace the filters, not the pressure differentials leading up to the event. Ok, that data can be predictive in itself but after a few months, I have data for proactive replacement rather than waiting for failure. Note here how the transaction actually modifies with experience? It is all about being smart and not just reactive. This is how we give our customers true value and it doesn’t derive from big data at all. Just smart data.

In summary, we need to collect sparsely managed data that is relevant and transaction oriented. Obviously we need to know why we want it and what we are going to do with it.

This is where most IOT solutions go wrong. They go out and collect data then look for a problem to solve with it. Data without relevance and without a transaction outcome. Ladies and gentlemen, turn this around. Go out there and find problems that you can build an IOT solution to solve.

Geoff Schaller
@ArcoflexIOT

Note: ^Gartner again, 50 billion by 2020, although the number changes more often than the font

ArcoFlex Sensei – Creating Virtual Sensors

 

 

ArcoFlex Sensei introduces a brand new concept to the IOT market: virtual sensors. Not possible on LoRA or single device gateway solutions, Sensei can combine one or more real sensors into virtual sensors to represent real data or conditions that are dependent on multiple real world inputs.

 

What a virtual sensor solves is important. Taking the example of a bio-security airlock, there are two doors with door switches and the airlock is in breach if both doors are opened simultaneously. Without virtual sensors you are constrained to writing code in the visualisation layer or stream analytics layer which constantly monitors the outputs of two sensors, the two door switches. Not only do you have to customise the code each time you need this implementation, the processing workload is now on the server side and scaling is a problem. If 1000 nurseries need an airlock, the server has to continually monitor this process for a 1000 sites. It is a massive performance hit and an unwanted complexity.

Enter ArcoFlex Sensei and a virtual sensor. The Raspberry PI code will create a sensor from the combined input of both door switches and report breach data like any other sensor. This is a simple Logical AND where both doors open creates a breach and even the length of time in breach can be measured in the visualisation. All processing is done on the remote PI so now scaling is not a problem. The dashboard merely displays data as if it were another sensor.

The key to sensor and device performance is in making the 1000’s of PI’s do the processing rather than the visualisation layer or the server. The resulting data stream becomes raw data just like any other data stream. This also allows for universal coding in the visualisation layer because the output is treated just like any other sensor.

So far, we have come across several very useful implementations:

·       Hothouse Airlock – At the Toolangi Seed Potato farm, bio-isolation is required in order to obtain certification to sell into Queensland. Whilst the doors can be rigged with alarms locally, Sensei records breaches (or lack of) and provides documentation for compliance purposes.

·       Filter Differential – in our standard irrigation monitoring solution there is a pressure sensor either side of the micro-filter. A virtual sensor is created to monitor the difference in pressure across the filter. Once it gets too great, a new filter is ordered and a replacement scheduled. Again, proactive alerts are generated to warn of pressure deterioration.

·       Irrigation line Leaks – when the line is pumping, there is a certain back pressure that is normal. If the outward leg pressure drops too much when the pump is running, there is a leak that must be attended to.

·        Pump Run Confirm – the float switch for refill can be actuated but if the UV sterilizer breaks, the pump is disconnected. This means that the float switch alone isn’t enough of an indicator that the pump is actually running. This virtual sensor might check three inputs to determine if it is actually running.

·        Fungal Growth Profile – there is a relationship between humidity and temperature that creates conditions ideal for fungal growth in a hothouse. The formula for this relationship can be coded into the PI so that raw data can be fed to the customer as a percentage of ideal.

·       Heartbeat – based around a simple timer, a single byte can be sent through to the dashboard confirming that comms and data channels are all in order. This is critical where you have data which of its nature, changes only very slowly or not at all for long periods. The heartbeat allows you to very economically detect the overall health of your sensor array.

·       Trending Patterns – perhaps you are monitoring air pressure or temperature and you have a need for a dashboard gauge which simply shows rising/steady/falling or drier/wetter and so on. Quite often, this fact is a metric of itself or can be used to combine into alerts and notifications. Defining these at the PI level solves all those scaling issues mentioned and ensures the output is recorded as data points of themselves.

With ArcoFlex Sensei, you can build virtual sensors to represent quite complex relationships between other sensors. These virtual sensors generate real time data in their own right but keep all the processing down at the sensor gateway. This makes for fully scalable business logic solutions to generate compound data without processing overheads.

But there is one other use for virtual sensors: output! Being able to control digital or analogue output is a vital part of remote monitoring and control. This isn’t just internal feedback, important as that can be, this is configurable control. Implemented with simple relays, a digital output could be instructed to turn on or off a motor, a fan, a compressor, a pump or in fact anything that can take a switched power supply. With a little bit of extra wiring we can also make an analogue output (something like 0V to 5V) to control a potentiometer. An example of this might be a thermostat or a stepping motor.

The virtual sensor gives us a structured way to provide configurable metrics to a digital or analogue input. Expect this to be the growth zone for IoT going forward. Well, once they’ve solved all the other issue first.

Geoff Schaller@ArcoflexIOT

ArcoFlex Sensei – Solving the IOT Sensor Problem

 

The IOT space – the bit that talks about remote monitoring and control (and most people ignore the control bit) - is full of companies scrambling to provide sensors and devices, vainly hoping someone will know what to do with them. The problem for the solution developer is that none of them are compatible and all of them are different. So how is the average business to navigate this sea of differences? This is where ArcoFlex Sensei comes to the rescue.

Add to that the fact that everyone will tell you WHAT you should be doing, very few, if anyone, actually tells you how. Whilst it is easy enough to code visualisations and dashboards once you have data, how on earth do you get the data out of a sensor and into an event hub in order to extract it down to a visualisation. Let’s look at the problem:

·         you have sensors and now need to wire it to “something” to extract the data

·         Your sensor gateway needs to be able to address the sensor and extract the actual data

·         There is scaling and limits and DAC conversion to be concerned about

·         Digital inputs need some way to get their value into the gateway

·         Code is needed somewhere to find these values, connect with the Hub and send data

Each of these sensors needs to connect to the central gateway computer via an IO card. Unless you’re an electrical engineer or hobbyist with some experience, this is going to be a massive problem so let’s go over the sensor types you will encounter:

One-Wire.          One-Wire devices have an address and you need to determine (or read from the packet someone binned 6 weeks ago). Once you have the address your gateway code will simply read the value in the form of a string. No conversion necessary but you will get 86 decimals places if you aren’t careful. One-Wire is good because the sensors are very cheap and quite accurate. Code is need to loop around all possible addresses each time you want a value.

Analogue.           To confuse matters, there are 4ch and 8ch analogue devices and they scale differently. 4ch sensors are vastly more accurate but you can only have 4 per IO card. They can be either current oriented (4ma) or voltage oriented (5V). they may or may not have a zero setting as an offset. This will further reduce resolution. 8ch analogue sensors can only provide 0-255 as a maximum resolution and if there is an offset, this is often 25% smaller. You need to know all of these characteristics so that you can correctly scale the resulting value.

Digital.                 To operate a digital sensor, you are required to supply a voltage (or 0V) to a set of pins that can be read as on or off. This means providing power to the circuit being switched or sensed.  The same goes if you are supplying a signal back. You will now be operating a relay to switch something on or off that will carry current to the device being controlled.

So the IO cards used to connect sensors need to have compatible input channels and if there are multiple cards, dip switches set to addresses to the cards. The code in your gateway device needs to accommodate all this and try as we might, the smaller Arduino Yun boards just couldn’t carry the programming. This was especially true once we came to the conclusion we needed to manage and compress the collected data. Our conclusion was to move on to a Raspberry PI3 B processor board and Windows 10 IOT Core as an operating system. Linux would have worked too but we’re a Microsoft shop and the attraction of using one language for the entire code base was just too attractive.

So ArcoFlex Sensei was built to solve all these sensor related problems. The basic model includes the following:

·         Raspberry PI3B with 1GB RAM and 8GB SSD card

·         W10 IOT Core and an application to manage the IO cards, all comms and management

·         One one-wire IO card (256+ sensors possible)

·         One 4 channel analogue IO card

·         One Digital IO card with up to 8 inputs

·         A power supply

·         A 4G gateway router and/or RJ45 LAN socket

·         Instructions (Yes!) on how to wire in your sensors

With just a few bits of wire, some pliers and a soldering iron, anyone can wire in any sensor they choose. $5 temperature sensors are a good start but we have taken all the complexity out of knowing how to get data out of a sensor and onto the web. If you obtain one of our starter kits we can also provide the back-end code to visual your sensor data through to a dashboard and even generate alerts.

With ArcoFlex Sensei, you can build a proof of concept device in just a few minutes.

Geoff Schaller
@ArcoflexIOT