auto_el/analyze/design.md

# Design

## Purpose

This project watches a running instance of the game **Eternal Lands** (an X11
window) and continuously analyzes screenshots of it. In its current form it
detects three kinds of state and records them to a JSONL log:

1. **Harvesting** — whether the player is currently harvesting (mining,
   cutting, etc.), detected from the color of a single pixel near the bottom
   of the screen.
2. **HUD values** — the numeric readouts next to the five status bars at the
   bottom of the screen (mana, food, health, inventory capacity, action
   points), extracted with OCR.
3. **Crafting windows** — the player can drag the Manufacturing and Inventory
   windows anywhere on screen, so they are located relative to their own title
   bars (not fixed coordinates). The colored status line inside the
   Manufacturing window is OCR'd and mapped to a category (`working`,
   `success`, `failed`, `failed_lost`, `hungry`, `overloaded`, `stopped`,
   `none`).

The near-term goal is observation only: the log is produced so a future
**observer** component can read it and act — e.g., automatically resume
harvesting when the player stops, or react when health/food get low. Keeping
the log format stable and machine-parseable is therefore a first-class
requirement.

## Capabilities

- **Window capture** (`capture.py`) — finds a window by title (substring,
  case-insensitive) on X11 and grabs its pixels even when unfocused or
  occluded, via `XGetImage` with a root-window fallback.
- **Harvesting detection** (`analyze.is_harvesting`) — samples the pixel at
  `(1710, 1068)`. Light tan (~`#f2c285`, luma ≈ 201) means harvesting; dark
  brown (~`#664c33`, luma ≈ 81) means idle. Decision is `luma >= 140`.
- **Bar value extraction** (`analyze.read_bar_values`) — crops the small
  number that precedes each bar, binarizes it, upscales 3x, and runs tesseract
  with a digits-only whitelist (`--psm 7`).
- **Window verification** (`analyze.verify_window`) — checks a `150x42` crop
  around a candidate point for the title substring `anufact` (Manufacturing)
  or `nven` (Inventory). This is the per-frame cache check; it passes on all
  53 reference frames.
- **Window recovery** (`analyze.find_windows`) — full-frame OCR scan
  (`--psm 11`, run at two scales) that collects candidate title words and
  verifies each. Used only when a cached position stops verifying, i.e. after
  the window moves.
- **Craft message OCR** (`analyze.read_manufacturing_message`) — crops a
  window-relative region, isolates green or red text pixels, tight-bboxes
  them, and OCRs the isolated color channel.
- **Message classification** (`analyze.classify_message`) — maps an OCR'd
  message plus its color to a stable category string.
- **Window cache** (`main.WindowTracker`) — keeps last-known window positions,
  re-verifies them each frame, and triggers a full-frame rescan only when a
  cache miss happens.
- **JSONL logging** (`main.py`) — appends one JSON object per frame.
- **Replay mode** (`main.py --replay DIR`) — processes existing PNGs instead
  of a live window, so analysis and OCR can be verified offline.
- **Test suite** (`tests/`) — folder-driven pytest tests; dropping a new
  screenshot into a folder automatically covers it.

## Architecture

```
capture.py        X11 window capture -> PIL Image (RGBA)
analyze.py        Pure functions: summarize, is_harvesting, read_bar_values,
                  verify_window, find_windows, read_manufacturing_message,
                  classify_message, load_image, difference. No side effects.
main.py           CLI entry point. Live capture loop or replay mode; owns the
                  WindowTracker cache, builds the JSON record, appends to log.
tests/            Folder-driven pytest tests + test images.
```

- `capture.py` is the only module that talks to X11.
- `analyze.py` is pure image analysis — every function takes a PIL image (or
  path) and returns data. This makes it directly testable.
- `main.py` wires them together and owns I/O (capturing, logging, printing).

## HUD calibration

All screen coordinates are hardcoded for a **1920x1080 captured window** and
live at the top of `analyze.py`. They were measured from real screenshots and
must move only if the game window resolution/layout changes.

| Constant | Value | Meaning |
|---|---|---|
| `HARVEST_PIXEL` | `(1710, 1068)` | pixel that flips color while harvesting |
| `HARVEST_LUMA_THRESHOLD` | `140` | above = harvesting, below = idle |
| `NUMBER_REGIONS["mana"]` | `(0, 1033, 62, 1048)` | crop before the blue bar |
| `NUMBER_REGIONS["food"]` | `(195, 1033, 226, 1048)` | crop before the yellow bar |
| `NUMBER_REGIONS["health"]` | `(355, 1033, 389, 1048)` | crop before the red bar |
| `NUMBER_REGIONS["inventory"]` | `(520, 1033, 552, 1048)` | crop before the dark bar |
| `NUMBER_REGIONS["action_points"]` | `(610, 1033, 714, 1048)` | crop before the purple bar |

Regions are `(x0, y0, x1, y1)`. `y0` starts at 1033 because row 1032 is the
panel's top border, which would confuse OCR.

Observed values on the reference screenshots: mana `32`, food `39`, health
`40`, action points `140`; inventory capacity varies (`110` → `108` → `106`)
as items are collected.

## Window calibration

The Manufacturing and Inventory windows are semi-transparent panels the player
can drag anywhere, so their positions are **relative to the window title bar**,
never fixed screen coordinates. All offsets below are measured from real
captures and live at the top of `analyze.py`.

| Constant | Value | Meaning |
|---|---|---|
| `TITLE_VERIFY_CROP` | `(14, 12, 150, 42)` | `(left, top, width, height)` crop around a candidate title point |
| `TITLE_VERIFY_THRESHOLD` | `115` | binarization threshold for the title crop |
| `TITLE_SUBSTRING` | `anufact` / `nven` | OCR substring that proves the title bar |
| `MESSAGE_SEARCH` | `(175, 105, 600, 70)` | craft message search region relative to the title |
| `MESSAGE_COLOR_GREEN` | `(170, 120, 30)` | green-text pixel filter `(min g, min g-r, min g-b)` |
| `MESSAGE_COLOR_RED` | `(170, 100, 60)` | red-text pixel filter `(min r, min r-g, min r-b)` |
| `FIND_OCR_SCALES` | `(1.0, 0.75)` | full-frame scan scales; some frames only recover at one |

- The craft message sits at `(title_x - 161, title_y + 121)` — measured at 5
  different Manufacturing window positions, all the same offset. `MESSAGE_SEARCH`
  is a wider region around it that absorbs title-detection jitter.
- The title bar is ~11px tall. The verify crop is deliberately generous
  (`±14/±12` px) because the verify point is the title-bar center, which OCR
  occasionally reports a few pixels from the glyphs.
- Full-frame recovery is approximate: OCR word boxes land within ~15px of the
  true title. Positions are re-anchored by `verify_window`, so exactness only
  matters for the message offset, and `MESSAGE_SEARCH` absorbs the jitter.
- Reference window positions across the capture sessions: Manufacturing at
  `(488, 480)`, `(376, 262)`, `(1330, 194)`, `(1005, 565)`, `(281, 676)`,
  and `(1104, 327)`/`(1104, 337)`; Inventory always `(1629, 686)`.

## Log format

One JSON object per line, appended to `analysis.jsonl` (configurable via
`--log`):

```json
{"time": "20:31:02", "frame": 4, "is_harvesting": true, "harvest_pixel": "#f2c285", "harvest_luma": 201, "bars": {"mana": 32, "food": 39, "health": 40, "inventory": 106, "action_points": 140}, "manufacturing_window": [1104, 327], "manufacturing_message": {"text": "you successfully created 1 potion of minor healing", "color": "green", "category": "success"}, "inventory_window": [1629, 686]}
```

- `frame` is an integer in live mode, the file name in replay mode.
- `bars` values are integers, or `null` when OCR could not read them (e.g.
  tesseract not installed). Consumers must tolerate `null`.
- `manufacturing_window` / `inventory_window` are `[x, y]` title-bar
  coordinates of the (currently verified) window, or `null` when unverified.
- `manufacturing_message` is `{"text", "color", "category"}` or `null`.
  `text`/`color` are `null` when no message is showing (category `none`);
  `manufacturing_message` itself is `null` when the Manufacturing window
  position is unverified. Colors: `green` (success/working), `red`
  (failed/failed_lost/hungry/overloaded/stopped).
- Message categories: `working`, `success`, `stopped`, `failed`,
  `failed_lost`, `hungry`, `overloaded`, `none`, `unknown`.
- This shape is the contract for the future observer — it should not change
  without updating `tests/` and the observer accordingly.

## Testing

`tests/images/` holds the screenshots used as fixtures, grouped by expected
behavior:

```
tests/images/harvesting/      is_harvesting must be True
tests/images/not_harvesting/  is_harvesting must be False
tests/images/corrupt/         must be skipped gracefully (None)
tests/images/manufacturing/   craft message at the pinned title position must
  <category>/*.png              classify to the folder's category
tests/images/inventory/       verify_window must pass at the pinned position
```

Tests are **folder-driven**: new edge cases are added by dropping a screenshot
into the appropriate folder; no test code changes needed unless a new expected
*value* must be pinned. `tests/test_bars.py` pins per-frame expectations (e.g.
inventory = 110/108/106 depending on frame). `tests/test_windows.py` pins the
Manufacturing title position per fixture frame (`MFG_POSITION`), the Inventory
position, and a representative subset of `find_windows` recovery cases.

## Dependencies and setup

System:

- `tesseract-ocr` (OCR engine; the Python binding is `pytesseract`)
- X11 (for live capture)

Python (installed in the project virtualenv `.venv/`, see `requirements.txt`):

- `python-xlib`, `pillow`, `pytesseract`, `pytest`

```bash
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest tests/
.venv/bin/python main.py                 # live capture
.venv/bin/python main.py --replay tests/images/harvesting
```

## Known limitations

- Coordinates are resolution-specific (1920x1080). Resizing the game window
  requires recalibration of `HARVEST_PIXEL` / `NUMBER_REGIONS`.
- OCR depends on the system `tesseract` binary; if missing, bar values log as
  `null` (harvesting detection is unaffected). Window/message OCR degrades the
  same way (`verify_window` returns False, messages read as `null`).
- The panel appears to be semi-transparent; some scene pixels show through.
  So far this has not affected detection, but a very different background
  could.
- Full-frame window recovery (`find_windows`) is not 100% reliable — it found
  the Manufacturing window in all 53 reference frames but missed the Inventory
  window in one harvesting frame. `WindowTracker` compensates by retrying on
  every frame while a window stays stale, so a missed recovery only delays the
  update by a frame or two. The cheap per-frame `verify_window` cache check is
  the workhorse (53/53) and recovery only runs after a window moves.
- The steady-state loop adds roughly 0.5s/frame for the two window verifies
  plus message OCR; a full-frame recovery scan costs ~5-9s but only happens
  after a window moves.