💾 Archived View for ew.srht.site › en › 2021 › 20210111-environmental-data-1.gmi captured on 2023-09-08 at 16:26:00. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

2021-01-11

Progress on Project HausBus

tags: data keepitsimple

Projects

My desk at home is hopelessly cluttered with /projects/ in various stages, ranging from contemplating to just about being finished --- its only one more thing ... you all know this, I guess.

So one of my long term projects is "collecting environmental data on premise".

It started out in maybe 2003 and it went a long way and several iterations of technology. State of affairs last summer was this:

A handfull of AVR microcontrollers and sensors on printed circuit boards, distributed throughout the house, implement the sensor electronics..
Each board is connected to a cable providing power (12V) and a data connection via RS485 bus.
Each controller is programmed with assembly and Forth[a].
The collection of sensors includes temperature and humidity (inside and outside the house), atmospheric pressure, use of electricity and water, the water level in the underground cistern[b] and a number of readings related to heating and solar thermal energy.
A small /always on/ system[c] is running Linux and serveral services.
Among Them A Perl Script, Which periodically addresses the microcontrollers one by one and collects their current sensor readings. The script thus is aptly called "collector.pl", you see?
The /protocol/ on the bus consists of plain ascii text strings.
The collector would dissect the data strings and feed them into a sqlite database.
Another perl script, named "viewer.pl" would query said database, wrangle the resulting data a little and draw a diagram using perl/pgplot5! Repeat after me: pee-gee-plot-five. Very well!
Most important feature: the whole thing ist /not/ /smart/! No! Not even a tiny bit!

This odd collection has served me well for more than 10 years! The WAF (wifes acceptance factor) was actually quite high, and we gained a bit of insight from this collection.

From the description some of you (assuming there are readers at all) will already have slight doubts whether this author is in a sane state of mind. perl? Wuttt? PGPLOT??? Yes well, pgplot itself used to be a Fortran library, if memory serves me well. It has been replaced with giza, which is not a drop in replacement for how I used it.

However, the pgplot based viewer remained working only, because all the neccessary pieces had survived the upgrade of the /viewing/ system to Debian 9 (stretch). It was clear, that on a newly installed system I would fail to piece everything together. Plus there was a number of shortcomings which were asking for more sophisticated tools.

So last summer I set out to make this whole thing more modern[d]. I placed another controller, with new firmware on my desk, together with a Pi running a new collector plus mqtt, influxdb, grafana and the MagicMirror3 software. Plus display, cables, power supply and whatever is needed for a happy controller's life. And the whole thing failed on me. Miserably so! The whole thing turned out to be unstable. It would run for maybe an hour, or for a whole week, and then it would just stall. Each component still blinking, but the communication would fail. And powercycle would /not/ make the system run again! Very strange indeed.

I spent hours and hours on this. No dice.

Then some day I put everything away and declared unconditional surrender.

But a tinkerer's brain will not give up that easily, right?

So over year end season I decided to keep as much as possible of the existing solution and try /the simplest thing, that could possibly work/ to copy the collected data into a newfangled influxdb. Keep it simple!

This /keeping it simple/ required three iterations nonetheless. But I did get out of this with small changes:

The collector script will call "mosquitto_pub" externally for every good data point collected --- everything else remains in place.
The collector script is now started from systemd replacing the old init.d script, journalctl replaces the screen hack used so far.

New in the odd collection are these:

Mosquitto will forward the data telegrams to any interested subscriber.
Telegraf will subscribe and feed the telegrams into influxdb.
Grafana will query the database and make any webbrowser act as a display to visualize the data --- this acts as an additional viewer. Look Ma, no perl!

None of these new toys are self-explanatory to my sort of brain, but heck, I can show new diagrams without resorting to the perl viewer.

Mission accomplished!

And if I look at this, it is already so much easier to use:

Any system in the network can look at this now! I can repurpose the viewer system, or at the very least upgrade its operating system!
Annotations are one click away --- they didn't really exist before.
Zooming into the data or out is a "mouse-down - move - mouse-up" sequence away --- this involved fiddly command lines before!
Switching a given dataset on or off is a click or ctrl-click away!

Of course there are minor niggles:

I have not found out yet how to calculate the dew point from different temperature and humidity time series.
The cross hair is actually lacking the /cross/ feature.
Unfortunately grafana currently cannot display more than 2 different y-axes on a planel. I would like to have tree, all of them standing free on the right hand side.

tl;dr: You have to start somewhere. And keep it simple!

Bartender? a Talisker 57North, please, with water at the side. Thank you so much!

Cheers,

~ew

[a] amforth home page

[b] 2020-11-04 Leaking Cistern --- Fixed?

[c] PC Engines APU2 single board computer

[d] for some weird definition of modern, perhaps.

Home