I am curious about what everyone does for managing your devices outside the realm of general operation ie. sending MQTT messages.
I have a growing number of ESP devices now and I have started to write a firmware that is unified between them to make firmware and general parameter updates easier. They have a section of the memory that is specific to the application of the device, but general stuff like Wifi, MQTT, OTA, and a range of functions like deepsleep routines are unified in the firmware.
I have baked in a Webserver to each device so that pretty much all settings and parameters can be updated through a web ui (similar to Tasmota, but better-looking ).
This is great for updating stay the wifi credentials on 5-10 devices, but as I get more devices (and probably add more options to the unified firmware this will be a pain for updating anything.
Currently in progress
To solve this problem I am using my central server to host a central UI, this host has a REST API I have built in and each ESP has a simple API for GET/POST requests from the host, and also POST from the device for things like Watchdog, Error/Warning logs, Device status, General device info like free memory, uptime etc.
I also wanted the ability for my devices like temp/humid sensors to update the openhab persistence DB with values that are collected when we have wifi scheduled off, then it can do a bulk dump to the host for processing rather than writing directly to the DB.
I am also baking in the ability for bulk device setting updates, so essentially do a multi-select and update device settings, but also the ability to write the unified firmware to all selected devices.
Have I gone crazy?
I have yet to get the partial firmware update finished, and all the GET request built in. The basic function is working and very nicely at that, but I am interested in what others are doing in this regard.
Obviously, it’s a bit more of a pain the more different chips one has, as the libraries will vary across them but currently have found the ESP8266 covers most of my requirements and if I need more GPIO I just use an extender chip.
I would say that you have exceeded the level of management that our small group here uses for their own setups! When I start thinking of building my own software to run my automation devices en masse, i get the urge to curl up into the fetal position in the corner. Sheesk. I think it’s incredible that you have the ability to TRULY make your setup your own. Serious kudos! Offering no input on how to make it better, just bowing at the feet of a master.
It is a mission, but I am the kind of person who isn’t just happy when it’s working, I either have to break it or nitpick some small detail till my eyes pop out my head.
I am kind of enjoying C++ now, I think it seemed impossible at the start of the year but once I got stuck in its not so bad. Building the host web server wasn’t too hard as its in PHP, I finished that last night, though I still need to finish the encryption as its pretty rough for a local network.
I would say I am the opposite of a master @Guru_Of_Nothing , I was explaining to a friend some of the devices I have been building at a dinner party (he’s way smarter than me too) his wife says you must be cleaver to do all this stuff… I said no I am just really stubborn and don’t mind failing lots and lots of times till it sinks in… and that’s why I don’t play around with AC circuits
Just thought I would give an update on this, for anyone stumbling across it.
So after much pondering and a working system but unrefined, I have decided on a more sustainable/less device specific approach. I am limited to writing and maintaining the software accross all ESP devices and also run the challange of new chips like the ESP32, then I could have to modify the code for them. Pretty silly for just one home.
While I pretty much use MQTT for everything in terms of automation, why am I using HTTP messages for telemetry and commands??? good question. If i use MQTT I can leverage other software on the devices like Tasmota, that excepts MQTT commands and sends telemetry. I can also create a library for my custom devices that use deepsleep and send retained messages with settings changes to them, is well as report telemetry, no web ui really needed for these actual devices, all over MQTT.
So I am now modifying my server to handle as a MQTT client that can publish and subscribe to each device. It will operate much the same as it currently does, creating an object for each devices based on the topic (much how OpenHab Things work).
Originally I thought I could just use OpenHab for this but its a real pain adding a ton of items for all the telemetry and settings. There is also the restriction on the type of content and the way it is displayed, much easier just creating a php server that handles this, it can be easily customized based on the sub topics.
Once I have it all up and working I will post all the code up on Github and link here, I will also probably write a post on my new blog aussiesmarthome.com.au that covers finance and tech.
Sounds like a winner! It never ceases to amaze me at how insanely versatile and modular MQTT makes things in our automation world. I am starting to see the benefits of EVERYTHING connecting to MQTT in a meaningful way so that things can be manipulated and monitored. I will definitely be checking out your blog as well.
MQTT is definitely a powerful tool, I only wanted to bypass it originally as it is an extra service to depend on with both device and server but after rethinking it I see it as “if MQTT is not working, nothing is” lol
Down the track I will put MQTT on its own server, currently my server is running File Server, MQTT, OpenHab, MySQL, Laragon stack, Unifi Controller, UPS Power Panel and a few other things, its a fair bit to deal with and uses about 9GB of ram static and about 15GB under heavy load.
Maybe in the future I will get a nice 16 core rig and run all the services as VMs, but will wait till we add all the lighting to the home management.
I personally use PlatformIO for everything. Library management, device management, firmware updates, device profiles, serial debugging.
It has a small learning curve, but once you get rolling it simplifies everything. Now, I wouldn’t use any of the code from this project (everything is way better now), but this will give you an idea:
You could in theory script it to build the firmware and remote update everything. I haven’t yet, because I need to re-write all my stuff to be less, ugly.
I’m actually going to move on from a lot of my custom stuff to Mozilla’s WebThings library and ditch my OpenHAB2 install for Mozilla Webthings. It’s very impressive and probably the first OpenSource IoT platform that I’d consider installing for someone else. I hacked up a device to open our gate at work and have a webthings gateway running inside a docker container, in an ECS cluster on AWS
wow @techman83 thats really interest, man there are so many different optiongs out there.
I am kind of happy I am not completely doing the house till next year, gives me some time to explore the options with the devices I have before hardwiring.
Can Mozilla’s WebThings run locally? just looking through the docs now and it seems very capable.
Webthings absolutely can run locally, in fact they have a bunch of different ways to install it, including a Raspberry Pi image and Docker (I just read they have an OpenWRT build in the works too!!). I chose to run it in AWS because I didn’t want the burden of managing hardware in the office and as low stakes way to flesh out our DevOps tools.
They recently released a new version. Looks like the work install is already running it
).
) & this makes me happy.
