audio_project_hub/docs/audio_logging.md

Audio Logging Design

This document defines how the Hub writes the audio it collects from the Nodes to the SD card. It is the contract that the future Node-collection task (producer) and the existing StorageTask (consumer) are written against.

The deliverable for this design is:

• src/storage/data_logger.h - the public interface (skeleton).
• src/storage/data_logger.cpp - stub bodies + fully implemented WAV metadata helpers. The chunk-pool/queue/file logic is marked TODO and is the remaining work.
• The constants in src/storage/storage_config.h under the "Audio logging" section.

---

1. Data model

Parameter	Value
Sample rate	48 kHz
Bit depth	16 bit (signed little-endian PCM)
Microphones per node	4
Max nodes	10
Total channels	40

One microphone == one channel == one "track". A single file holds all of them as an interleaved multichannel WAV.

2. Throughput budget

raw stream rate = 40 ch x 48 kHz x 2 B = 3,840,000 B/s  (3.66 MiB/s)

• SDIO 4-bit @ 40 MHz sustains roughly 8-12 MB/s -> the card write has ample headroom at less than half of its budget.
• WiFi (AP) carries the same 3.84 MB/s inbound as UDP payload, which the design already budgets for.
• The 128 KB chunk pool represents ~34 ms of audio. The write must be able to absorb bursts longer than that only by dropping (Section 8).

3. File format: single multichannel WAV

The stream is a RIFF/WAVE file with 40 interleaved channels. The 44-byte header is the only metadata; no per-chunk headers are written to the file.

Header layout (all little-endian):

Offset	Size	Value
0	4	"RIFF"
4	4	chunk size = file size - 8 (patched)
8	4	"WAVE"
12	4	"fmt "
16	4	16 (PCM fmt chunk size)
20	2	1 (PCM)
22	2	40 (channels)
24	4	48000 (sample rate)
28	4	3,840,000 (byte rate = rate x blockAlign)
32	2	80 (block align = channels x 2)
34	2	16 (bits per sample)
36	4	"data"
40	4	data size = file size - 44 (patched)

buildWavHeader() and finalizeWavHeader() in data_logger.h/cpp produce these bytes and the two 32-bit values patched at offsets 4 and 40.

Header finalization

The sizes at offsets 4 and 40 are unknown while the file is being written. They are patched when the file is closed (rotation or session end):

file.seek(4);  file.write(riffSize, 4);
file.seek(40); file.write(dataSize, 4);

Crash tolerance

If power is lost mid-file, the header still contains the placeholder sizes. PCM decoders that read to EOF (Audacity, ffmpeg, Python wave) play the valid audio regardless; the last partial sector may be zero-padded. This satisfies the design.md requirement that a mid-recording power loss still yields usable audio.

Opening the file on a PC

• Audacity / ffmpeg / Python wave: handle 40 channels.
• VLC and stock Windows players generally will not render 40 channels even though the file is structurally valid. Convert with ffmpeg first if needed.

4. Architecture: producer / consumer with a chunk pool

 Node collection task (core 1)                 StorageTask (core 0)
   assembles one round of 40 channels              |  loop:
   interleaved into frame order                    v
   +----------------------+                 +----------------------+
   | c = acquireChunk(5ms)|  <-- freeQ ---- |   pool: 8 x 16 KB    |
   | fill c->data[...]    |                 |   (MALLOC_CAP_DMA)   |
   | submitChunk(c)       | ---- filledQ -->|   nextChunk(1000ms)  |
   +----------------------+                 |   writeChunk(c)      |
                                            |   releaseChunk(c)    |
                                            |   rotateIfNeeded()   |
                                            |   updateStats()      |
                                            +----------------------+

 DataLogger owns both queues and the pool. Producers and the consumer never
 touch the file or the card directly - only DataLogger does.

• Pool: 8 chunks x 16 KB = 128 KB, allocated once in begin() with heap_caps_malloc(MALLOC_CAP_DMA).
• Queues: freeQ holds pointers to empty chunks, filledQ holds pointers to chunks waiting to be written. Both are FreeRTOS queues of AudioChunk*.
• Cores: the producer runs on core 1 (network), the consumer on core 0 (the existing pinned StorageTask). SD card work never touches core 1.

5. The interface

Types (from data_logger.h):

struct AudioChunk
{
    uint8_t* data;      // DMA-capable buffer owned by the pool
    uint32_t capacity;  // buffer size in bytes (STORAGE_LOG_CHUNK_SIZE)
    uint32_t length;    // valid bytes written by the producer
    uint32_t sequence;  // monotonic order, for diagnostics only
};

class DataLogger
{
public:
    bool begin(fs::FS& files);   // allocate pool, create log dir
    void end();                  // finalize current file, free pool

    bool openSession();          // create rec_<uptime>_<n>.wav
    bool closeSession();         // finalize + close current file
    bool rotateIfNeeded();       // called by the consumer

    // Producer (core 1)
    AudioChunk* acquireChunk(TickType_t timeout);
    void        submitChunk(AudioChunk* chunk);

    // Consumer (core 0)
    AudioChunk* nextChunk(TickType_t timeout);
    bool        writeChunk(AudioChunk* chunk);
    void        releaseChunk(AudioChunk* chunk);

    // Stats
    uint32_t bytesWritten() const;
    uint32_t chunksWritten() const;
    uint32_t droppedChunks() const;
    uint32_t writeSpeedBps() const;
    bool     overflowing() const;
};

Producer contract (Node collection task, not yet written)

AudioChunk* c = logger.acquireChunk(pdMS_TO_TICKS(5));
if (c == nullptr) { /* collection is behind; drop this round */ return; }

// Fill c->data[0..c->length) with interleaved PCM, frame order:
//   frame0: ch0..ch39, frame1: ch0..ch39, ...
// 40 samples of 2 bytes per frame = 80 bytes per frame.
c->length = interleavedBytes;
c->sequence = nextSeq++;
logger.submitChunk(c);

• Chunks must be submitted in strict stream order (single producer, FIFO queue -> order is preserved automatically).
• A chunk is not required to end on a frame boundary, but c->length should stay a multiple of 80 bytes so the file is always frame-aligned.
• acquireChunk can return a chunk even when the pool is momentarily empty; see the drop-oldest rule in Section 8.

Consumer contract (StorageTask on core 0, future)

while (true)
{
    AudioChunk* c = logger.nextChunk(pdMS_TO_TICKS(1000));
    if (c != nullptr)
    {
        logger.writeChunk(c);   // one file.write(c->data, c->length)
        logger.releaseChunk(c); // returns the chunk to freeQ
    }
    logger.rotateIfNeeded();
    logger.updateStats(storageState);  // bps + drops -> dashboard
}

• The 1 s timeout lets the loop pulse for stats/capacity updates even when no audio is flowing (this replaces the current 1 s vTaskDelayUntil heartbeat).
• writeChunk is the only place the card is touched. It appends the chunk's PCM directly to the WAV data section in one file.write.

Ownership rules

• The pool owns the buffers; producers and the consumer borrow them.
• A borrowed chunk is in exactly one place at a time: a producer (between acquire and submit), a queue, or the consumer (between next and release).
• releaseChunk returns it to freeQ. writeChunk never frees.

6. Upstream interleaving contract

The WAV must be written frame-interleaved, but the Nodes deliver one UDP packet per Node (4 contiguous mono channels). Frames cannot be interleaved until all Nodes in a round have been collected. That re-ordering is the job of the collection task, not the logger:

1. Collect all 10 Node dumps for round n.
2. A missing/offline Node contributes silence: zero-fill its 4 channels.
3. Re-order into frame order: frame j = node0[mic0..3], node1[mic0..3], ...
4. Feed the resulting bytes into chunks and submit them in order.

Consequences the collection task must honor:

• Round size: with the 128 KB pool, a round must stay <= ~96 KB (~25 ms of audio) so the pool can hold more than one round. See the sizing math in Section 7.
• Silence for offline Nodes comes from this zero-fill; the logger never invents data.

7. Chunk pool and RAM budget

chunk size  = STORAGE_LOG_CHUNK_SIZE  = 16 KiB
pool depth  = STORAGE_LOG_POOL_SIZE   = 8
pool total  = 128 KiB  (MALLOC_CAP_DMA)

buffering   = 128 KiB / 3.84 MB/s  ~= 34 ms
round size  = 48 kHz x 2 B x 40 ch x round_s
  25 ms round -> 96 KB  (~6 chunks)   [fits the pool with slack]
  50 ms round -> 192 KB                [does NOT fit - must shrink]

The collection round duration is therefore bounded by the pool unless the pool grows. Keep rounds at <= 25 ms, or raise STORAGE_LOG_POOL_SIZE/chunk size and re-run the math.

The round's interleave buffer lives in the collection task's own RAM (up to 96 KB), not in the logger pool; it is freed after the round is submitted.

8. Overflow: drop-oldest + warning

If the SD card cannot keep up, the pool drains and the producer has no chunk. Policy (chosen): drop the oldest buffered chunk until caught up - never block, never halt.

Mechanism inside acquireChunk: on timeout, the logger pops one chunk off the back of filledQ (the oldest unwritten data), returns it to freeQ, hands it to the producer, increments droppedChunks, and raises the overflow warning. The stream keeps flowing with the newest data at the cost of a gap.

The warning is visible two ways:

• Serial: a rate-limited [Logger] X chunks dropped, Y MB behind line.
• LED on GPIO STORAGE_WARN_LED_GPIO (default 4, active-high): on while overflowing(), off once the queue drains below a low-water mark again.

9. Rotation, naming, flush

• Rotate by size: when bytesThisFile >= STORAGE_LOG_ROTATE_BYTES (default 1 GiB, ~4.6 min at 3.84 MB/s) the consumer calls rotateIfNeeded(): finalize + close the current file, open the next.
• Files: created under STORAGE_LOG_DIR (/sdcard/audio), named rec_<uptimeSeconds>_<n>.wav. n increments per rotation within a boot; the uptime prefix keeps names unique across boots. If a name already exists, skip forward until it does not (never overwrite).
• Flush: file.flush() (f_sync) every STORAGE_LOG_FLUSH_BYTES (default 16 MiB, ~4.3 s) so an unclean power-off loses at most that window and never corrupts earlier data.
• Card full: rotation cannot create a file -> report a fatal error, light the warning LED solid, and halt (matches design.md's SD-failure stance).

10. DMA requirements

The SDIO path (production) uses the SDMMC controller's internal IDMA engine: block data moves to the card without CPU cycles. Two rules make this work:

• Buffers must be in DMA-capable memory: allocate the pool with heap_caps_malloc(MALLOC_CAP_DMA) (guarantees internal DRAM + alignment).
• Never hand the card a buffer that lives in PSRAM or a stack array.

The SPI path (STORAGE_IFACE_SPI, current bring-up) does not use DMA - the Arduino SPI driver busy-waits the FIFO. It is bring-up only and cannot sustain the 3.84 MB/s target; production logging must run on SDIO.

11. Metering -> dashboard

writeChunk accumulates bytesWritten/chunksWritten; writeSpeedBps is derived from a sliding 1 s window of real writes. The consumer publishes writeSpeedBps and droppedChunks into StorageState each loop, replacing the boot-time measureWriteSpeed() benchmark (whose interval is already 0). Add a droppedChunks field to StorageSnapshot and surface it on the dashboard when the logging loop lands.

12. Future integration points (StorageTask)

When the consumer loop is implemented, StorageTask::run():

1. After mount: logger.begin(storage.fs()), logger.openSession().
2. Replace the 1 s heartbeat loop with the consumer loop in Section 5.
3. Stop calling measureWriteSpeed() (the logger provides real bps).

13. SPI -> SDIO migration checklist

Already documented in src/storage/storage_config.h. Restated for logging:

1. STORAGE_IFACE SPI -> SDMMC.
2. Wire SD to STORAGE_SDMMC_* pins (freely re-routable on classic ESP32).
3. Keep 4-bit mode (STORAGE_SDMMC_MODE_1BIT == false) - 1-bit halves the throughput margin.
4. MALLOC_CAP_DMA buffers work unchanged; nothing else moves.

14. Open TODOs (to finish the feature)

• data_logger.cpp: chunk pool + queue creation in begin().
• data_logger.cpp: acquireChunk drop-oldest path + overflow warning (Serial + LED on STORAGE_WARN_LED_GPIO).
• data_logger.cpp: writeChunk/rotateIfNeeded/closeSession file handling + header finalization + flush cadence.
• data_logger.cpp: writeSpeedBps sliding window.
• StorageTask: consumer loop (Section 5).
• StorageState/dashboard: droppedChunks field.
• Node collection task: round assembly + interleave + zero-fill (Section 6).

Appendix: constants

Constant	Default	Meaning
STORAGE_AUDIO_SAMPLE_RATE_HZ	48000	WAV sample rate
STORAGE_AUDIO_CHANNELS	40	= nodes x mics, WAV channels
STORAGE_AUDIO_BITS	16	WAV bit depth
STORAGE_LOG_CHUNK_SIZE	16 * 1024	pool chunk size (bytes)
STORAGE_LOG_POOL_SIZE	8	pool chunk count
STORAGE_LOG_DIR	"/sdcard/audio"	recording directory
STORAGE_LOG_ROTATE_BYTES	1 GiB	rotate when a file reaches this
STORAGE_LOG_FLUSH_BYTES	16 MiB	f_sync cadence
STORAGE_WARN_LED_GPIO	4	overflow warning LED
STORAGE_WARN_LED_ACTIVE_HIGH	true	LED polarity