Working harvesting detection, working manufacturing detection.
This commit is contained in:
@@ -0,0 +1,213 @@
|
||||
# Design
|
||||
|
||||
## Purpose
|
||||
|
||||
This project watches a running instance of the game **Eternal Lands** (an X11
|
||||
window) and continuously analyzes screenshots of it. In its current form it
|
||||
detects three kinds of state and records them to a JSONL log:
|
||||
|
||||
1. **Harvesting** — whether the player is currently harvesting (mining,
|
||||
cutting, etc.), detected from the color of a single pixel near the bottom
|
||||
of the screen.
|
||||
2. **HUD values** — the numeric readouts next to the five status bars at the
|
||||
bottom of the screen (mana, food, health, inventory capacity, action
|
||||
points), extracted with OCR.
|
||||
3. **Crafting windows** — the player can drag the Manufacturing and Inventory
|
||||
windows anywhere on screen, so they are located relative to their own title
|
||||
bars (not fixed coordinates). The colored status line inside the
|
||||
Manufacturing window is OCR'd and mapped to a category (`working`,
|
||||
`success`, `failed`, `failed_lost`, `hungry`, `overloaded`, `stopped`,
|
||||
`none`).
|
||||
|
||||
The near-term goal is observation only: the log is produced so a future
|
||||
**observer** component can read it and act — e.g., automatically resume
|
||||
harvesting when the player stops, or react when health/food get low. Keeping
|
||||
the log format stable and machine-parseable is therefore a first-class
|
||||
requirement.
|
||||
|
||||
## Capabilities
|
||||
|
||||
- **Window capture** (`capture.py`) — finds a window by title (substring,
|
||||
case-insensitive) on X11 and grabs its pixels even when unfocused or
|
||||
occluded, via `XGetImage` with a root-window fallback.
|
||||
- **Harvesting detection** (`analyze.is_harvesting`) — samples the pixel at
|
||||
`(1710, 1068)`. Light tan (~`#f2c285`, luma ≈ 201) means harvesting; dark
|
||||
brown (~`#664c33`, luma ≈ 81) means idle. Decision is `luma >= 140`.
|
||||
- **Bar value extraction** (`analyze.read_bar_values`) — crops the small
|
||||
number that precedes each bar, binarizes it, upscales 3x, and runs tesseract
|
||||
with a digits-only whitelist (`--psm 7`).
|
||||
- **Window verification** (`analyze.verify_window`) — checks a `150x42` crop
|
||||
around a candidate point for the title substring `anufact` (Manufacturing)
|
||||
or `nven` (Inventory). This is the per-frame cache check; it passes on all
|
||||
53 reference frames.
|
||||
- **Window recovery** (`analyze.find_windows`) — full-frame OCR scan
|
||||
(`--psm 11`, run at two scales) that collects candidate title words and
|
||||
verifies each. Used only when a cached position stops verifying, i.e. after
|
||||
the window moves.
|
||||
- **Craft message OCR** (`analyze.read_manufacturing_message`) — crops a
|
||||
window-relative region, isolates green or red text pixels, tight-bboxes
|
||||
them, and OCRs the isolated color channel.
|
||||
- **Message classification** (`analyze.classify_message`) — maps an OCR'd
|
||||
message plus its color to a stable category string.
|
||||
- **Window cache** (`main.WindowTracker`) — keeps last-known window positions,
|
||||
re-verifies them each frame, and triggers a full-frame rescan only when a
|
||||
cache miss happens.
|
||||
- **JSONL logging** (`main.py`) — appends one JSON object per frame.
|
||||
- **Replay mode** (`main.py --replay DIR`) — processes existing PNGs instead
|
||||
of a live window, so analysis and OCR can be verified offline.
|
||||
- **Test suite** (`tests/`) — folder-driven pytest tests; dropping a new
|
||||
screenshot into a folder automatically covers it.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
capture.py X11 window capture -> PIL Image (RGBA)
|
||||
analyze.py Pure functions: summarize, is_harvesting, read_bar_values,
|
||||
verify_window, find_windows, read_manufacturing_message,
|
||||
classify_message, load_image, difference. No side effects.
|
||||
main.py CLI entry point. Live capture loop or replay mode; owns the
|
||||
WindowTracker cache, builds the JSON record, appends to log.
|
||||
tests/ Folder-driven pytest tests + test images.
|
||||
```
|
||||
|
||||
- `capture.py` is the only module that talks to X11.
|
||||
- `analyze.py` is pure image analysis — every function takes a PIL image (or
|
||||
path) and returns data. This makes it directly testable.
|
||||
- `main.py` wires them together and owns I/O (capturing, logging, printing).
|
||||
|
||||
## HUD calibration
|
||||
|
||||
All screen coordinates are hardcoded for a **1920x1080 captured window** and
|
||||
live at the top of `analyze.py`. They were measured from real screenshots and
|
||||
must move only if the game window resolution/layout changes.
|
||||
|
||||
| Constant | Value | Meaning |
|
||||
|---|---|---|
|
||||
| `HARVEST_PIXEL` | `(1710, 1068)` | pixel that flips color while harvesting |
|
||||
| `HARVEST_LUMA_THRESHOLD` | `140` | above = harvesting, below = idle |
|
||||
| `NUMBER_REGIONS["mana"]` | `(0, 1033, 62, 1048)` | crop before the blue bar |
|
||||
| `NUMBER_REGIONS["food"]` | `(195, 1033, 226, 1048)` | crop before the yellow bar |
|
||||
| `NUMBER_REGIONS["health"]` | `(355, 1033, 389, 1048)` | crop before the red bar |
|
||||
| `NUMBER_REGIONS["inventory"]` | `(520, 1033, 552, 1048)` | crop before the dark bar |
|
||||
| `NUMBER_REGIONS["action_points"]` | `(610, 1033, 714, 1048)` | crop before the purple bar |
|
||||
|
||||
Regions are `(x0, y0, x1, y1)`. `y0` starts at 1033 because row 1032 is the
|
||||
panel's top border, which would confuse OCR.
|
||||
|
||||
Observed values on the reference screenshots: mana `32`, food `39`, health
|
||||
`40`, action points `140`; inventory capacity varies (`110` → `108` → `106`)
|
||||
as items are collected.
|
||||
|
||||
## Window calibration
|
||||
|
||||
The Manufacturing and Inventory windows are semi-transparent panels the player
|
||||
can drag anywhere, so their positions are **relative to the window title bar**,
|
||||
never fixed screen coordinates. All offsets below are measured from real
|
||||
captures and live at the top of `analyze.py`.
|
||||
|
||||
| Constant | Value | Meaning |
|
||||
|---|---|---|
|
||||
| `TITLE_VERIFY_CROP` | `(14, 12, 150, 42)` | `(left, top, width, height)` crop around a candidate title point |
|
||||
| `TITLE_VERIFY_THRESHOLD` | `115` | binarization threshold for the title crop |
|
||||
| `TITLE_SUBSTRING` | `anufact` / `nven` | OCR substring that proves the title bar |
|
||||
| `MESSAGE_SEARCH` | `(175, 105, 600, 70)` | craft message search region relative to the title |
|
||||
| `MESSAGE_COLOR_GREEN` | `(170, 120, 30)` | green-text pixel filter `(min g, min g-r, min g-b)` |
|
||||
| `MESSAGE_COLOR_RED` | `(170, 100, 60)` | red-text pixel filter `(min r, min r-g, min r-b)` |
|
||||
| `FIND_OCR_SCALES` | `(1.0, 0.75)` | full-frame scan scales; some frames only recover at one |
|
||||
|
||||
- The craft message sits at `(title_x - 161, title_y + 121)` — measured at 5
|
||||
different Manufacturing window positions, all the same offset. `MESSAGE_SEARCH`
|
||||
is a wider region around it that absorbs title-detection jitter.
|
||||
- The title bar is ~11px tall. The verify crop is deliberately generous
|
||||
(`±14/±12` px) because the verify point is the title-bar center, which OCR
|
||||
occasionally reports a few pixels from the glyphs.
|
||||
- Full-frame recovery is approximate: OCR word boxes land within ~15px of the
|
||||
true title. Positions are re-anchored by `verify_window`, so exactness only
|
||||
matters for the message offset, and `MESSAGE_SEARCH` absorbs the jitter.
|
||||
- Reference window positions across the capture sessions: Manufacturing at
|
||||
`(488, 480)`, `(376, 262)`, `(1330, 194)`, `(1005, 565)`, `(281, 676)`,
|
||||
and `(1104, 327)`/`(1104, 337)`; Inventory always `(1629, 686)`.
|
||||
|
||||
## Log format
|
||||
|
||||
One JSON object per line, appended to `analysis.jsonl` (configurable via
|
||||
`--log`):
|
||||
|
||||
```json
|
||||
{"time": "20:31:02", "frame": 4, "is_harvesting": true, "harvest_pixel": "#f2c285", "harvest_luma": 201, "bars": {"mana": 32, "food": 39, "health": 40, "inventory": 106, "action_points": 140}, "manufacturing_window": [1104, 327], "manufacturing_message": {"text": "you successfully created 1 potion of minor healing", "color": "green", "category": "success"}, "inventory_window": [1629, 686]}
|
||||
```
|
||||
|
||||
- `frame` is an integer in live mode, the file name in replay mode.
|
||||
- `bars` values are integers, or `null` when OCR could not read them (e.g.
|
||||
tesseract not installed). Consumers must tolerate `null`.
|
||||
- `manufacturing_window` / `inventory_window` are `[x, y]` title-bar
|
||||
coordinates of the (currently verified) window, or `null` when unverified.
|
||||
- `manufacturing_message` is `{"text", "color", "category"}` or `null`.
|
||||
`text`/`color` are `null` when no message is showing (category `none`);
|
||||
`manufacturing_message` itself is `null` when the Manufacturing window
|
||||
position is unverified. Colors: `green` (success/working), `red`
|
||||
(failed/failed_lost/hungry/overloaded/stopped).
|
||||
- Message categories: `working`, `success`, `stopped`, `failed`,
|
||||
`failed_lost`, `hungry`, `overloaded`, `none`, `unknown`.
|
||||
- This shape is the contract for the future observer — it should not change
|
||||
without updating `tests/` and the observer accordingly.
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/images/` holds the screenshots used as fixtures, grouped by expected
|
||||
behavior:
|
||||
|
||||
```
|
||||
tests/images/harvesting/ is_harvesting must be True
|
||||
tests/images/not_harvesting/ is_harvesting must be False
|
||||
tests/images/corrupt/ must be skipped gracefully (None)
|
||||
tests/images/manufacturing/ craft message at the pinned title position must
|
||||
<category>/*.png classify to the folder's category
|
||||
tests/images/inventory/ verify_window must pass at the pinned position
|
||||
```
|
||||
|
||||
Tests are **folder-driven**: new edge cases are added by dropping a screenshot
|
||||
into the appropriate folder; no test code changes needed unless a new expected
|
||||
*value* must be pinned. `tests/test_bars.py` pins per-frame expectations (e.g.
|
||||
inventory = 110/108/106 depending on frame). `tests/test_windows.py` pins the
|
||||
Manufacturing title position per fixture frame (`MFG_POSITION`), the Inventory
|
||||
position, and a representative subset of `find_windows` recovery cases.
|
||||
|
||||
## Dependencies and setup
|
||||
|
||||
System:
|
||||
|
||||
- `tesseract-ocr` (OCR engine; the Python binding is `pytesseract`)
|
||||
- X11 (for live capture)
|
||||
|
||||
Python (installed in the project virtualenv `.venv/`, see `requirements.txt`):
|
||||
|
||||
- `python-xlib`, `pillow`, `pytesseract`, `pytest`
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt
|
||||
.venv/bin/python -m pytest tests/
|
||||
.venv/bin/python main.py # live capture
|
||||
.venv/bin/python main.py --replay tests/images/harvesting
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- Coordinates are resolution-specific (1920x1080). Resizing the game window
|
||||
requires recalibration of `HARVEST_PIXEL` / `NUMBER_REGIONS`.
|
||||
- OCR depends on the system `tesseract` binary; if missing, bar values log as
|
||||
`null` (harvesting detection is unaffected). Window/message OCR degrades the
|
||||
same way (`verify_window` returns False, messages read as `null`).
|
||||
- The panel appears to be semi-transparent; some scene pixels show through.
|
||||
So far this has not affected detection, but a very different background
|
||||
could.
|
||||
- Full-frame window recovery (`find_windows`) is not 100% reliable — it found
|
||||
the Manufacturing window in all 53 reference frames but missed the Inventory
|
||||
window in one harvesting frame. `WindowTracker` compensates by retrying on
|
||||
every frame while a window stays stale, so a missed recovery only delays the
|
||||
update by a frame or two. The cheap per-frame `verify_window` cache check is
|
||||
the workhorse (53/53) and recovery only runs after a window moves.
|
||||
- The steady-state loop adds roughly 0.5s/frame for the two window verifies
|
||||
plus message OCR; a full-frame recovery scan costs ~5-9s but only happens
|
||||
after a window moves.
|
||||
Reference in New Issue
Block a user