Initial project commit

2026-08-08 20:58:06 -06:00
commit ea34c399d7
18 changed files with 1039 additions and 0 deletions
@@ -0,0 +1,506 @@
+# High level overview
+
+This application is less like an Arduino sketch and more like an **embeded application**. It should include the patterns of seperation of concerns, services, interfaces, event-driven communication, configuration management and dependency boundries. Here are the design requirements:
+
+
+This project is the **Hub** in a Hub and spoke architechure. The Hub is responsible for creating an access point that all of the spokes (Nodes) can connect to. The Hub will use a round-robbin scheduling to collect sensor data from the spokes. The spokes are collecting audio data and storing it in a buffer that is read by the Hub. This means that we have to be on time and can not fall behind schedule. This means that we should either use or lean heavily toward a real time system. 
+
+The Hub is also responsible for storing the audio data it is collecting onto an SD card. This **must** be done using 4 bit SDIO to be able to achieve the write speeds required to save all of the audio data. We estimate to support up to 10 Nodes each with 4 microphones. The raw audio should be saved to the SD card for post processing.
+
+The Hub also will act as the interface with the users. It will contain a web dashboard that users can use to view the status of the Hub and the Nodes. 
+
+
+## Define Hub responsibilities
+
+
+My Hub is going to have several independent responsibilities:
+
+1. **Network Management**
+
+   * Creates WiFi network
+   * Handles connected devices
+   * Manages IP addresses
+   * Handles OTA
+   * Handles Node authentication
+
+2. **Device Management**
+
+   * Knows what sensor nodes exist
+   * Tracks online/offline state
+   * Stores metadata:
+
+     * device ID
+     * last seen time
+     * firmware version
+     * sensor capabilities
+     * signal strength
+
+3. **Sensor Data Collection**
+
+   * Polls other ESP32 nodes
+   * Parses responses
+   * Validates data
+   * Converts into a common format
+
+4. **Data Storage**
+
+   * SD card management
+   * Logging
+   * File rotation
+   * Handling SD card failures
+
+5. **Web Interface**
+
+   * Dashboard
+   * API endpoints
+   * Configuration pages
+   * OTA interface
+
+6. **System Monitoring**
+
+   * Free heap
+   * Uptime
+   * CPU usage
+   * Connected clients
+   * Errors
+
+7. **Configuration**
+
+   * WiFi settings
+   * Sensor definitions
+   * Logging intervals
+   * Device names
+
+
+# Proposed High-Level Structure
+
+I would lean toward something like this:
+
+```
+src/
+│
+├── main.cpp
+│
+├── core/
+│   ├── system_manager.cpp
+│   ├── event_bus.cpp
+│   └── config.cpp
+│
+├── network/
+│   ├── wifi_manager.cpp
+│   ├── ota_manager.cpp
+│   └── api_server.cpp
+│
+├── sensors/
+│   ├── sensor_manager.cpp
+│   ├── sensor_node.cpp
+│   └── sensor_protocol.cpp
+│
+├── storage/
+│   ├── sd_manager.cpp
+│   └── data_logger.cpp
+│
+├── web/
+│   ├── web_server.cpp
+│   ├── dashboard.cpp
+│   └── web_assets/
+│
+└── models/
+    ├── sensor_data.h
+    ├── device_info.h
+    └── system_status.h
+```
+
+The goal is that `main.cpp` becomes boring.
+
+Ideally:
+
+```text
+initialize system
+start services
+run loop
+```
+
+
+# How Modules Communicate
+
+
+```
+             +----------------+
+             |  Web Interface |
+             +-------+--------+
+                     |
+                     |
+             +-------v--------+
+             |  System State  |
+             +-------+--------+
+                     |
+       +-------------+-------------+
+       |             |             |
+       v             v             v
+
+ Sensor Manager  SD Logger   Network Manager
+```
+
+The web interface does not know how sensors work.
+
+It asks:
+
+> "Give me current system state."
+
+Not:
+
+> "Go ask the sensor manager, which talks to WiFi, which parses packets..."
+
+---
+
+# Introduce a Data Model Layer
+
+Instead of passing random variables around:
+
+```cpp
+temperature
+humidity
+battery
+deviceName
+```
+
+Create common structures.
+
+Conceptually:
+
+```
+SensorReading
+
+{
+    deviceId,
+    timestamp,
+    temperature,
+    humidity,
+    batteryVoltage
+}
+```
+
+Then everything speaks this language:
+
+```
+Sensor Node
+      |
+      |
+      v
+SensorReading
+      |
+      +------> SD Logger
+      |
+      +------> Dashboard
+      |
+      +------> API
+```
+
+Now if I change from ESP-NOW to WiFi HTTP to MQTT later, the rest of the system does not care.
+
+---
+
+# Consider a Service-Oriented Architecture
+
+I would like a service oriented architechure like this:
+
+```
+WifiService
+SensorService
+StorageService
+WebService
+OTAService
+```
+
+Each service has:
+
+```
+begin()
+loop()
+status()
+```
+
+Conceptually:
+
+```
+setup()
+{
+    wifi.begin();
+    storage.begin();
+    sensors.begin();
+    web.begin();
+}
+
+
+loop()
+{
+    wifi.update();
+    sensors.update();
+    storage.update();
+    web.update();
+}
+```
+
+This is very ESP32-friendly.
+
+---
+
+# Use Interfaces Where Things Might Change
+
+For example, how do sensors communicate?
+
+Today:
+
+```
+Hub ---- HTTP ----> Sensor ESP32
+```
+
+Tomorrow:
+
+```
+Hub ---- ESP-NOW ----> Sensor ESP32
+```
+
+Later:
+
+```
+Hub ---- MQTT ----> Sensor ESP32
+```
+
+If we build our sensor system around:
+
+```
+SensorTransport
+```
+
+we can swap the communication method without rewriting our sensor manager.
+
+Example concept:
+
+```
+             SensorManager
+
+                    |
+                    |
+
+          SensorTransport Interface
+
+          /          |          \
+
+       HTTP       ESP-NOW      MQTT
+```
+
+This is a very powerful pattern.
+
+---
+
+# Event-Driven vs Polling
+
+My system naturally has events:
+
+Examples:
+
+* Sensor came online
+* Sensor stopped responding
+* New reading received
+* SD card full
+* OTA started
+
+Instead of:
+
+```cpp
+if(sensorOffline)
+{
+   updateWebPage();
+}
+```
+
+you could have:
+
+```
+SensorManager
+      |
+      |
+      emits:
+      SensorOfflineEvent
+
+              |
+              |
+       +------+------+
+       |             |
+ Web Interface   Logger
+```
+
+This prevents modules from knowing about each other.
+
+For an ESP32 project, I would not over-engineer this with a full framework, but a simple event queue could be very clean.
+
+---
+
+# Configuration Management
+
+Avoid hardcoding:
+
+```cpp
+const char* wifiName = "MyNetwork";
+```
+
+Instead:
+
+```
+/config.json
+```
+
+Example:
+
+```
+{
+  "Hub_name": "greenhouse-Hub",
+  "logging_interval": 60,
+  "sensors": [
+      {
+        "id": "sensor01",
+        "location": "north bed"
+      }
+  ]
+}
+```
+
+Then your web dashboard could eventually modify configuration.
+
+---
+
+# Think About Failure Modes Early
+
+Embedded systems fail differently than servers.
+
+Plan for:
+
+## SD card removed
+
+If the SD card is removed, it is acceptable to crash/halt (of course with a
+nice error message) We do not have enough RAM to store the data so it is not
+feasable to buffer until the SD card comes back.
+
+## Sensor disappears
+
+If one of the Nodes dissappear, we need to alert the user. This should be done
+with a warning light on the device however I do not think it is a critical
+failure. As we are recording audio, we can just insert silence in that section
+if the Node drops off.
+
+## Power loss
+
+With power loss I want the stored audio capture to not be corrupted. That is
+about the only requirement. If the system dies mid recording I want to be able
+to still use whatever was recorded before it died.
+
+## Corrupt configuration
+
+I want to have the device configurable through USB. So I can give it a default
+configuration however then end users can use a yaml file to reconfigure the
+device. This should then reboot the device and start it up again.
+
+# Logging Architecture
+
+I would avoid:
+
+```
+sensor_manager.cpp
+
+writeFile()
+```
+
+Instead:
+
+```
+SensorManager
+
+produces:
+
+SensorReading
+
+        |
+        v
+
+DataLogger
+
+        |
+        v
+
+SDManager
+```
+
+Then the logger can do it's own thing and the sensor system doesn't care.
+
+---
+
+# Mental Model
+
+Think of the Hub as a small operating system:
+
+```
+                 HUB
+
+        +----------------+
+        | Web Dashboard  |
+        +----------------+
+
+        +----------------+
+        | System State   |
+        +----------------+
+
+ +----------+ +----------+ +----------+
+ | Sensors  | | Storage  | | Network  |
+ +----------+ +----------+ +----------+
+
+        +----------------+
+        | Hardware Layer |
+        +----------------+
+```
+
+---
+
+# Node Hub communication
+
+This will be done using UDP packets. The Hub will send out a packet to ask for
+data and the Node will reply with a packet containnig the binary audio data.
+
+---
+
+# Node count
+
+I estimate having 10 nodes sampling 4 microphones each. Doing the math this
+ends up with about 6 MB/sec which is doable if I use 4 bit SDIO and wifi is
+totally capable of handling those speeds.
+
+
+---
+
+## Data frequency
+
+The nodes do not have much RAM so I want to collect frequently. I am thinking
+that when the Nodes connect to the Hub they are assigned a "time slot" so that
+they can dump their data during that time.
+
+---
+
+## Power
+
+The Hub is batery powered however it will be a large power bank. The nodes
+however are not and will be kept light weight.
+
+---
+
+
+My initial recommendation would be:
+
+* **Service-oriented architecture**
+* **Strong data models**
+* **Central system state**
+* **Event-based communication where useful**
+* **Hardware abstraction boundaries**
+* **Separate firmware projects with shared libraries**
+
+The next thing I would design is the **communication protocol between the Hub and sensor nodes**, because that decision will influence almost every other piece.