The Micro32/Fabgl Board Died Like a Little Bitch
Trying to find something funny here. First stress testing, the $15 board experienced multiple failures and the VGA was the first to go every time
I knew I said I would not discuss anything technical of much complexity over here on Vault-Co Communications. I am trying to keep it sorted so it is over on the “Vault-Co” website where it belongs. I think because many people signed up as paid because of the marvels and wonders I was promising both for the main program I have been working on 20 years or so and possibly this little $15 board. I thought I should report on my progress here. It turned up some critical flaws in the cheap hardware.
I kept imagining becoming a legend for writing this system not just because it was useful … I kept dreaming about people saying after the first round produced “CD-OS” on the little Micro32 board … this guy is some kind of ultimate badass. He got all this working on this $15 dollar board. Basically the guy squeezed an Apache web server equivalent into a memory space about the size of the Apache logo bitmap. So I won’t lie, I was increasingly excited about this prospect and have been doing 8 hours a day or more just getting it to an alpha version by porting snippets of my existing code over where they were minimalist and tightly functional. I think I must have done at least 200+ hours working on it, learning the IDE, compiling the code, learning how peripherals connect and how to drive a serial device from it over UART. (This was going to be the tiny Geiger Counter project + temp/humidity sensor) I would call it “version 0.1” when I proved it could serve web pages and run the Modbus.
I got a very impressive prototype working for the FabGL board on Monday. It had a little configuration screen followed by a monitor screen as it began to serve web pages. I learned quite a bit about FreeRTOS and launching a process in it’s own memory space to run on the other chip on the ESP-32. (This was going to be the Modbus real time but I never actually got that running before doing some tests) Just for the freaky factor, I threw my tilemaps onto the SD card that is the main repository for user data on the chip and actually got the map server working, inside one of my all-purpose any-browser screen desktop/mobile/tablet windows. Here is a screenshot off my Android tablet which was in the other room.
Worked on the first test! The pages loaded surprisingly fast after the main SPA (Single Page Application) code was pulled down. (After that the browser timestamps and checks the digest hash, caches all the code and will never load it again until the hash value changes) Yes, that’s the map server delivering the correct tiles for my capital city in Virginia, plus the editing controls for graphics and markers worked with persistence!!!!
I got on another browser that sits five feet away on another table that runs on a thin client, the cheapest and most generic setup possible for the network stations. I almost wet my pants when my ported Javascript/JQuery code loaded up from that browser perfectly as well. Pretty exciting stuff to think you can tell people that they can have a semi-complete Vault management system with Modbus over TCP/IP on a $15 board.
I thought, my head is spinning with excitement so it is time to get methodical about this. This is a good time before progressing any further to do some stress testing. See if something will kick this little board over, make it crash, socket problems … you never know, most of these mechanisms are unknown to me despite seeming similar to what I have written on the desktop. You can code in Windows for quite some time before you discover it’s a good idea to put a one second delay before some socket calls because Windows has such a crappy API for sockets it can crash if the next command comes too soon. Got better after XP but not much better.
So I actually edited the Javascript code in the browser to keep making AJAX requests on a timer, again and again every 10 seconds. Then I left that running. After 6 hours, the web pages were still popping up perfectly. That’s when I noticed the VGA display was flickering like crazy and missing scanlines.
Before I panicked, I rebooted everything and fired up the testing again. This time the VGA display started to mess up in less than 20 minutes, indicative of a recurrent failure that has something to do with static, heat or some other environmental influence possibly like EMF. That’s tragic and in most hobbyist boards you would just work around it by turning it off every once in a while. After all, you wouldn’t expect too much of a $15 board despite the impressive things it can do.
I tried other VGA screens and checked the cables again, ran the tests from the start. Three minutes later the screen was messed up all over again. In all fairness to this little board, you could tell the ESP32 chip was a tough bugger running without problems. I decided that maybe I had just gotten a board with a bad VGA chip. I bought two for this very reason. I hooked up the other one and let it run for four hours with my fingers crossed. Yep, the VGA began to flicker again.
Perhaps it has something to do with humidity in the air and static buildup - it has been raining cats and dogs here. It doesn’t matter because any board so sensitive to humidity the display fails cannot be trusted to run the VAULT-SYS/CD-OS system. I would not tell people to buy one to use for this purpose because it would be a terrible idea to put any faith into it, especially entering inventory, personnel, configuring monitoring and sensors, etc.
I knew it sounded too good to be true … and it was. I continue to keep hope alive that someday a board will appear on the market that is the equivalent of a mini-desktop with mouse and keyboard plus VGA display, built like a tank, runs VAULT-SYS and costs $15. With advances in semiconductors it could still happen. I have to say I think you could build it on top of the ESP32 as well. In the meantime, the only boards I have ever tested I thought were fit as military electronics was the STM32 and related products. I will probably work some on a board like this after August because I have a ROM burner for these devices - problem is they often cost at least $50 or more. They are however very hardened systems and have similar built-in capacity for display and input peripherals. I was actually thinking it an ideal ARM device to put into my Pip-Boy replica to use as a portable showcase! I can’t work on it until later in the year, however.
I have spent at least two months faffing with this Micro32 board and it certainly was not wasted … I learned a lot about the ESP32 ecosystem, compiler, implementation and operating environment. Maybe I will put it to use somewhere in the future in a smaller device with fewer responsibilities than this was going to assume.
I now have about two months to get out VAULT-SYS like I promised, so I am returning to that immediately with full commitment. My complete development environment is set up apart from my desktop on the other side of my study and ready to produce that code for Windows/Linux/Mac/FreeRTOS etc. That’s cross platform code that I always planned for the hardware to be an exercise for the user … they could run it on a laptop or wherever they found a cheap host for it. I think some of my experience hooking up Modbus is absolutely applicable to the big brother version of what was intended to run on the little board - because I learned how to do discovery of Modbus devices as a separate and distinct process running outside of the web server, which can be very useful to aim for a plug’n’play setup where almost all the logic is user customized once the devices are found.
I am genuinely sorry this happened, my village idiot enthusiasm got the best of me and I should have been working on VAULT-SYS this entire time as promised on the subscriber page. All I can say that is optimistic is that the current state of the VAULT-SYS code is around 80% and I have two months to make that 100%. Also, the big brother version of my system that is VAULT-SYS is about a million times more powerful than the little board was ever going to be, albeit with larger memory and operating system requirements.
I realized after yanking it out that SQLite is really the organizational core of the software and it was sorely missed when I used a tiny SQL mimic library on the Micro32 board because SQLite wouldn’t fit. Half the features went missing instantly after leaving it out. SQLite is so much more than a database that it is an insult to call it just that. It is the way the real time monitoring works, with shared memory tables in place of the alternate process I talked about above running Modbus on the Micro32. Once you move to this format for messaging, you can have a hundred protocols running from all over your shelter or retreat connected to anything. You will see the power of this idea when I bring out the first version of VAULT-SYS.
I’m not LARPING here and I’m not Terry Davis (He was a pretty cool guy but crazy as a three dollar bill). There will be something released as scheduled in August 2023. Whether or not it will be perfect I don’t know at this time. I give you my word that I will keep that deadline. I first need to solve the software problem to prove my ideas are viable.
Then, likely with the help of others, we will solve the hardware problem on what is the ideal board and setup to use for it. I am in favor of conformity on the recommended solution and I hope at that time I can point out the board I have found that is the best use case. I am sorry to say it is not going to be the Micro32/FabGL board because it has proven it will never be tough enough to be reliable. The single most important quality that my system needs to have will be that reliability.
Also, after 25 years, I swear to you … threads suck. Don’t use them. They are never going to be right for a survival manager system because they cause problems of indeterminacy that can never truly be debugged. Just to be a contrarian and because this list that follows was the #1 reason I designed VAULT-SYS architecture to run as intercommunicating processes for reasons of stability.
Why threads suck:
Threaded programs are impossible to test. One successful run means nothing.
Threaded programs tendency to fail varies with load. This frequently leads to misplaced optimism at a project's start, when load is low.
Threaded programs are difficult to read, because you need to know in which thread each piece of code could execute. This is difficult to determine from inspection.
Threaded programs are difficult to maintain for the same reason.
Threaded programs are difficult to debug. Being unable to repeat failed execution runs hurts.
Threaded code is frequently slower then single-threaded code because of the extra overhead.
Threaded programs don't scale. Even if you buy hardware with more processors, the speed up will be sub-linear while the increase in cost is super-linear. This is a space where distributed computation wins: there is a reason Google buys many commodity boxes rather then big multi-processor boxes.
Threaded programs will work fine for years, and then start failing without having been changed.
I mean for the first version of this software to run for 50 years without failure once it is started up and when something does fail, I hope it’s the hardware even as the software continues to chug away.
Regards, Tex
On 21 June I paid $40 to support and read this SS. Still unable to.