# Audio Logging Design This document defines how the Hub writes the audio it collects from the Nodes to the SD card. It is the contract that the future Node-collection task (producer) and the existing `StorageTask` (consumer) are written against. The deliverable for this design is: * `src/storage/data_logger.h` - the public interface (skeleton). * `src/storage/data_logger.cpp` - stub bodies + fully implemented WAV metadata helpers. The chunk-pool/queue/file logic is marked `TODO` and is the remaining work. * The constants in `src/storage/storage_config.h` under the "Audio logging" section. --- ## 1. Data model | Parameter | Value | |-----------------------|------------| | Sample rate | 48 kHz | | Bit depth | 16 bit (signed little-endian PCM) | | Microphones per node | 4 | | Max nodes | 10 | | **Total channels** | **40** | One microphone == one channel == one "track". A single file holds all of them as an interleaved multichannel WAV. ## 2. Throughput budget ``` raw stream rate = 40 ch x 48 kHz x 2 B = 3,840,000 B/s (3.66 MiB/s) ``` * SDIO 4-bit @ 40 MHz sustains roughly 8-12 MB/s -> the card write has ample headroom at less than half of its budget. * WiFi (AP) carries the same 3.84 MB/s inbound as UDP payload, which the design already budgets for. * The 128 KB chunk pool represents **~34 ms** of audio. The write must be able to absorb bursts longer than that only by dropping (Section 8). ## 3. File format: single multichannel WAV The stream is a **RIFF/WAVE file with 40 interleaved channels**. The 44-byte header is the only metadata; no per-chunk headers are written to the file. Header layout (all little-endian): | Offset | Size | Value | |--------|------|--------------------------------------------| | 0 | 4 | `"RIFF"` | | 4 | 4 | chunk size = file size - 8 (patched) | | 8 | 4 | `"WAVE"` | | 12 | 4 | `"fmt "` | | 16 | 4 | 16 (PCM fmt chunk size) | | 20 | 2 | 1 (PCM) | | 22 | 2 | 40 (channels) | | 24 | 4 | 48000 (sample rate) | | 28 | 4 | 3,840,000 (byte rate = rate x blockAlign) | | 32 | 2 | 80 (block align = channels x 2) | | 34 | 2 | 16 (bits per sample) | | 36 | 4 | `"data"` | | 40 | 4 | data size = file size - 44 (patched) | `buildWavHeader()` and `finalizeWavHeader()` in `data_logger.h/cpp` produce these bytes and the two 32-bit values patched at offsets 4 and 40. ### Header finalization The sizes at offsets 4 and 40 are unknown while the file is being written. They are patched when the file is closed (rotation or session end): ``` file.seek(4); file.write(riffSize, 4); file.seek(40); file.write(dataSize, 4); ``` ### Crash tolerance If power is lost mid-file, the header still contains the placeholder sizes. PCM decoders that read to EOF (Audacity, ffmpeg, Python `wave`) play the valid audio regardless; the last partial sector may be zero-padded. This satisfies the design.md requirement that a mid-recording power loss still yields usable audio. ### Opening the file on a PC * **Audacity** / **ffmpeg** / **Python `wave`**: handle 40 channels. * VLC and stock Windows players generally will not render 40 channels even though the file is structurally valid. Convert with ffmpeg first if needed. ## 4. Architecture: producer / consumer with a chunk pool ``` Node collection task (core 1) StorageTask (core 0) assembles one round of 40 channels | loop: interleaved into frame order v +----------------------+ +----------------------+ | c = acquireChunk(5ms)| <-- freeQ ---- | pool: 8 x 16 KB | | fill c->data[...] | | (MALLOC_CAP_DMA) | | submitChunk(c) | ---- filledQ -->| nextChunk(1000ms) | +----------------------+ | writeChunk(c) | | releaseChunk(c) | | rotateIfNeeded() | | updateStats() | +----------------------+ DataLogger owns both queues and the pool. Producers and the consumer never touch the file or the card directly - only DataLogger does. ``` * **Pool**: 8 chunks x 16 KB = 128 KB, allocated once in `begin()` with `heap_caps_malloc(MALLOC_CAP_DMA)`. * **Queues**: `freeQ` holds pointers to empty chunks, `filledQ` holds pointers to chunks waiting to be written. Both are FreeRTOS queues of `AudioChunk*`. * **Cores**: the producer runs on core 1 (network), the consumer on core 0 (the existing pinned `StorageTask`). SD card work never touches core 1. ## 5. The interface Types (from `data_logger.h`): ```cpp struct AudioChunk { uint8_t* data; // DMA-capable buffer owned by the pool uint32_t capacity; // buffer size in bytes (STORAGE_LOG_CHUNK_SIZE) uint32_t length; // valid bytes written by the producer uint32_t sequence; // monotonic order, for diagnostics only }; class DataLogger { public: bool begin(fs::FS& files); // allocate pool, create log dir void end(); // finalize current file, free pool bool openSession(); // create rec__.wav bool closeSession(); // finalize + close current file bool rotateIfNeeded(); // called by the consumer // Producer (core 1) AudioChunk* acquireChunk(TickType_t timeout); void submitChunk(AudioChunk* chunk); // Consumer (core 0) AudioChunk* nextChunk(TickType_t timeout); bool writeChunk(AudioChunk* chunk); void releaseChunk(AudioChunk* chunk); // Stats uint32_t bytesWritten() const; uint32_t chunksWritten() const; uint32_t droppedChunks() const; uint32_t writeSpeedBps() const; bool overflowing() const; }; ``` ### Producer contract (Node collection task, not yet written) ```cpp AudioChunk* c = logger.acquireChunk(pdMS_TO_TICKS(5)); if (c == nullptr) { /* collection is behind; drop this round */ return; } // Fill c->data[0..c->length) with interleaved PCM, frame order: // frame0: ch0..ch39, frame1: ch0..ch39, ... // 40 samples of 2 bytes per frame = 80 bytes per frame. c->length = interleavedBytes; c->sequence = nextSeq++; logger.submitChunk(c); ``` * Chunks **must be submitted in strict stream order** (single producer, FIFO queue -> order is preserved automatically). * A chunk is not required to end on a frame boundary, but `c->length` should stay a multiple of 80 bytes so the file is always frame-aligned. * `acquireChunk` can return a chunk even when the pool is momentarily empty; see the drop-oldest rule in Section 8. ### Consumer contract (StorageTask on core 0, future) ```cpp while (true) { AudioChunk* c = logger.nextChunk(pdMS_TO_TICKS(1000)); if (c != nullptr) { logger.writeChunk(c); // one file.write(c->data, c->length) logger.releaseChunk(c); // returns the chunk to freeQ } logger.rotateIfNeeded(); logger.updateStats(storageState); // bps + drops -> dashboard } ``` * The 1 s timeout lets the loop pulse for stats/capacity updates even when no audio is flowing (this replaces the current 1 s `vTaskDelayUntil` heartbeat). * `writeChunk` is the only place the card is touched. It appends the chunk's PCM directly to the WAV data section in one `file.write`. ### Ownership rules * The pool owns the buffers; producers and the consumer borrow them. * A borrowed chunk is in exactly one place at a time: a producer (between acquire and submit), a queue, or the consumer (between next and release). * `releaseChunk` returns it to `freeQ`. `writeChunk` never frees. ## 6. Upstream interleaving contract The WAV must be written frame-interleaved, but the Nodes deliver one UDP packet per Node (4 contiguous mono channels). Frames cannot be interleaved until all Nodes in a round have been collected. That re-ordering is the job of the collection task, **not** the logger: 1. Collect all 10 Node dumps for round `n`. 2. A missing/offline Node contributes silence: zero-fill its 4 channels. 3. Re-order into frame order: `frame j = node0[mic0..3], node1[mic0..3], ...` 4. Feed the resulting bytes into chunks and submit them in order. Consequences the collection task must honor: * **Round size**: with the 128 KB pool, a round must stay <= ~96 KB (~25 ms of audio) so the pool can hold more than one round. See the sizing math in Section 7. * **Silence for offline Nodes** comes from this zero-fill; the logger never invents data. ## 7. Chunk pool and RAM budget ``` chunk size = STORAGE_LOG_CHUNK_SIZE = 16 KiB pool depth = STORAGE_LOG_POOL_SIZE = 8 pool total = 128 KiB (MALLOC_CAP_DMA) buffering = 128 KiB / 3.84 MB/s ~= 34 ms round size = 48 kHz x 2 B x 40 ch x round_s 25 ms round -> 96 KB (~6 chunks) [fits the pool with slack] 50 ms round -> 192 KB [does NOT fit - must shrink] ``` The collection round duration is therefore bounded by the pool unless the pool grows. Keep rounds at <= 25 ms, or raise `STORAGE_LOG_POOL_SIZE`/chunk size and re-run the math. The round's interleave buffer lives in the **collection task's own RAM** (up to 96 KB), not in the logger pool; it is freed after the round is submitted. ## 8. Overflow: drop-oldest + warning If the SD card cannot keep up, the pool drains and the producer has no chunk. Policy (chosen): **drop the oldest buffered chunk until caught up** - never block, never halt. Mechanism inside `acquireChunk`: on timeout, the logger pops one chunk off the back of `filledQ` (the oldest unwritten data), returns it to `freeQ`, hands it to the producer, increments `droppedChunks`, and raises the overflow warning. The stream keeps flowing with the newest data at the cost of a gap. The warning is visible two ways: * **Serial**: a rate-limited `[Logger] X chunks dropped, Y MB behind` line. * **LED on GPIO `STORAGE_WARN_LED_GPIO`** (default 4, active-high): on while `overflowing()`, off once the queue drains below a low-water mark again. ## 9. Rotation, naming, flush * **Rotate by size**: when `bytesThisFile >= STORAGE_LOG_ROTATE_BYTES` (default 1 GiB, ~4.6 min at 3.84 MB/s) the consumer calls `rotateIfNeeded()`: finalize + close the current file, open the next. * **Files**: created under `STORAGE_LOG_DIR` (`/sdcard/audio`), named `rec__.wav`. `n` increments per rotation within a boot; the uptime prefix keeps names unique across boots. If a name already exists, skip forward until it does not (never overwrite). * **Flush**: `file.flush()` (f_sync) every `STORAGE_LOG_FLUSH_BYTES` (default 16 MiB, ~4.3 s) so an unclean power-off loses at most that window and never corrupts earlier data. * **Card full**: rotation cannot create a file -> report a fatal error, light the warning LED solid, and halt (matches design.md's SD-failure stance). ## 10. DMA requirements The SDIO path (production) uses the SDMMC controller's internal IDMA engine: block data moves to the card without CPU cycles. Two rules make this work: * Buffers must be in DMA-capable memory: allocate the pool with `heap_caps_malloc(MALLOC_CAP_DMA)` (guarantees internal DRAM + alignment). * Never hand the card a buffer that lives in PSRAM or a stack array. The SPI path (`STORAGE_IFACE_SPI`, current bring-up) does **not** use DMA - the Arduino SPI driver busy-waits the FIFO. It is bring-up only and cannot sustain the 3.84 MB/s target; production logging must run on SDIO. ## 11. Metering -> dashboard `writeChunk` accumulates `bytesWritten`/`chunksWritten`; `writeSpeedBps` is derived from a sliding 1 s window of real writes. The consumer publishes `writeSpeedBps` and `droppedChunks` into `StorageState` each loop, replacing the boot-time `measureWriteSpeed()` benchmark (whose interval is already 0). Add a `droppedChunks` field to `StorageSnapshot` and surface it on the dashboard when the logging loop lands. ## 12. Future integration points (StorageTask) When the consumer loop is implemented, `StorageTask::run()`: 1. After mount: `logger.begin(storage.fs())`, `logger.openSession()`. 2. Replace the 1 s heartbeat loop with the consumer loop in Section 5. 3. Stop calling `measureWriteSpeed()` (the logger provides real bps). ## 13. SPI -> SDIO migration checklist Already documented in `src/storage/storage_config.h`. Restated for logging: 1. `STORAGE_IFACE` SPI -> SDMMC. 2. Wire SD to `STORAGE_SDMMC_*` pins (freely re-routable on classic ESP32). 3. Keep 4-bit mode (`STORAGE_SDMMC_MODE_1BIT == false`) - 1-bit halves the throughput margin. 4. `MALLOC_CAP_DMA` buffers work unchanged; nothing else moves. ## 14. Open TODOs (to finish the feature) - [ ] `data_logger.cpp`: chunk pool + queue creation in `begin()`. - [ ] `data_logger.cpp`: `acquireChunk` drop-oldest path + overflow warning (Serial + LED on `STORAGE_WARN_LED_GPIO`). - [ ] `data_logger.cpp`: `writeChunk`/`rotateIfNeeded`/`closeSession` file handling + header finalization + flush cadence. - [ ] `data_logger.cpp`: `writeSpeedBps` sliding window. - [ ] `StorageTask`: consumer loop (Section 5). - [ ] `StorageState`/dashboard: `droppedChunks` field. - [ ] Node collection task: round assembly + interleave + zero-fill (Section 6). ## Appendix: constants | Constant | Default | Meaning | |-----------------------------------|--------------|---------------------------------| | `STORAGE_AUDIO_SAMPLE_RATE_HZ` | 48000 | WAV sample rate | | `STORAGE_AUDIO_CHANNELS` | 40 | = nodes x mics, WAV channels | | `STORAGE_AUDIO_BITS` | 16 | WAV bit depth | | `STORAGE_LOG_CHUNK_SIZE` | 16 * 1024 | pool chunk size (bytes) | | `STORAGE_LOG_POOL_SIZE` | 8 | pool chunk count | | `STORAGE_LOG_DIR` | "/sdcard/audio" | recording directory | | `STORAGE_LOG_ROTATE_BYTES` | 1 GiB | rotate when a file reaches this | | `STORAGE_LOG_FLUSH_BYTES` | 16 MiB | f_sync cadence | | `STORAGE_WARN_LED_GPIO` | 4 | overflow warning LED | | `STORAGE_WARN_LED_ACTIVE_HIGH` | true | LED polarity |