Design

Purpose

This project watches a running instance of the game Eternal Lands (an X11 window) and continuously analyzes screenshots of it. In its current form it detects three kinds of state and records them to a JSONL log:

Harvesting — whether the player is currently harvesting (mining, cutting, etc.), detected from the color of a single pixel near the bottom of the screen.
HUD values — the numeric readouts next to the five status bars at the bottom of the screen (mana, food, health, inventory capacity, action points), extracted with OCR.
Crafting windows — the player can drag the Manufacturing and Inventory windows anywhere on screen, so they are located relative to their own title bars (not fixed coordinates). The colored status line inside the Manufacturing window is OCR'd and mapped to a category (working, success, failed, failed_lost, hungry, overloaded, stopped, none).

The near-term goal is observation only: the log is produced so a future observer component can read it and act — e.g., automatically resume harvesting when the player stops, or react when health/food get low. Keeping the log format stable and machine-parseable is therefore a first-class requirement.

Capabilities

Window capture (capture.py) — finds a window by title (substring, case-insensitive) on X11 and grabs its pixels even when unfocused or occluded, via XGetImage with a root-window fallback.
Harvesting detection (analyze.is_harvesting) — samples the pixel at (1710, 1068). Light tan (~~#f2c285, luma ≈ 201) means harvesting; dark brown (~~#664c33, luma ≈ 81) means idle. Decision is luma >= 140.
Bar value extraction (analyze.read_bar_values) — crops the small number that precedes each bar, binarizes it, upscales 3x, and runs tesseract with a digits-only whitelist (--psm 7).
Window verification (analyze.verify_window) — checks a 150x42 crop around a candidate point for the title substring anufact (Manufacturing) or nven (Inventory). This is the per-frame cache check; it passes on all 53 reference frames.
Window recovery (analyze.find_windows) — full-frame OCR scan (--psm 11, run at two scales) that collects candidate title words and verifies each. Used only when a cached position stops verifying, i.e. after the window moves.
Craft message OCR (analyze.read_manufacturing_message) — crops a window-relative region, isolates green or red text pixels, tight-bboxes them, and OCRs the isolated color channel.
Message classification (analyze.classify_message) — maps an OCR'd message plus its color to a stable category string.
Window cache (main.WindowTracker) — keeps last-known window positions, re-verifies them each frame, and triggers a full-frame rescan only when a cache miss happens.
JSONL logging (main.py) — appends one JSON object per frame.
Replay mode (main.py --replay DIR) — processes existing PNGs instead of a live window, so analysis and OCR can be verified offline.
Test suite (tests/) — folder-driven pytest tests; dropping a new screenshot into a folder automatically covers it.

Architecture

capture.py        X11 window capture -> PIL Image (RGBA)
analyze.py        Pure functions: summarize, is_harvesting, read_bar_values,
                  verify_window, find_windows, read_manufacturing_message,
                  classify_message, load_image, difference. No side effects.
main.py           CLI entry point. Live capture loop or replay mode; owns the
                  WindowTracker cache, builds the JSON record, appends to log.
tests/            Folder-driven pytest tests + test images.

capture.py is the only module that talks to X11.
analyze.py is pure image analysis — every function takes a PIL image (or path) and returns data. This makes it directly testable.
main.py wires them together and owns I/O (capturing, logging, printing).

HUD calibration

All screen coordinates are hardcoded for a 1920x1080 captured window and live at the top of analyze.py. They were measured from real screenshots and must move only if the game window resolution/layout changes.

Constant	Value	Meaning
`HARVEST_PIXEL`	`(1710, 1068)`	pixel that flips color while harvesting
`HARVEST_LUMA_THRESHOLD`	`140`	above = harvesting, below = idle
`NUMBER_REGIONS["mana"]`	`(0, 1033, 62, 1048)`	crop before the blue bar
`NUMBER_REGIONS["food"]`	`(195, 1033, 226, 1048)`	crop before the yellow bar
`NUMBER_REGIONS["health"]`	`(355, 1033, 389, 1048)`	crop before the red bar
`NUMBER_REGIONS["inventory"]`	`(520, 1033, 552, 1048)`	crop before the dark bar
`NUMBER_REGIONS["action_points"]`	`(610, 1033, 714, 1048)`	crop before the purple bar

Regions are (x0, y0, x1, y1). y0 starts at 1033 because row 1032 is the panel's top border, which would confuse OCR.

Observed values on the reference screenshots: mana 32, food 39, health 40, action points 140; inventory capacity varies (110 → 108 → 106) as items are collected.

Window calibration

The Manufacturing and Inventory windows are semi-transparent panels the player can drag anywhere, so their positions are relative to the window title bar, never fixed screen coordinates. All offsets below are measured from real captures and live at the top of analyze.py.

Constant	Value	Meaning
`TITLE_VERIFY_CROP`	`(14, 12, 150, 42)`	`(left, top, width, height)` crop around a candidate title point
`TITLE_VERIFY_THRESHOLD`	`115`	binarization threshold for the title crop
`TITLE_SUBSTRING`	`anufact` / `nven`	OCR substring that proves the title bar
`MESSAGE_SEARCH`	`(175, 105, 600, 70)`	craft message search region relative to the title
`MESSAGE_COLOR_GREEN`	`(170, 120, 30)`	green-text pixel filter `(min g, min g-r, min g-b)`
`MESSAGE_COLOR_RED`	`(170, 100, 60)`	red-text pixel filter `(min r, min r-g, min r-b)`
`FIND_OCR_SCALES`	`(1.0, 0.75)`	full-frame scan scales; some frames only recover at one

The craft message sits at (title_x - 161, title_y + 121) — measured at 5 different Manufacturing window positions, all the same offset. MESSAGE_SEARCH is a wider region around it that absorbs title-detection jitter.
The title bar is ~11px tall. The verify crop is deliberately generous (±14/±12 px) because the verify point is the title-bar center, which OCR occasionally reports a few pixels from the glyphs.
Full-frame recovery is approximate: OCR word boxes land within ~15px of the true title. Positions are re-anchored by verify_window, so exactness only matters for the message offset, and MESSAGE_SEARCH absorbs the jitter.
Reference window positions across the capture sessions: Manufacturing at (488, 480), (376, 262), (1330, 194), (1005, 565), (281, 676), and (1104, 327)/(1104, 337); Inventory always (1629, 686).

Log format

One JSON object per line, appended to analysis.jsonl (configurable via --log):

{"time": "20:31:02", "frame": 4, "is_harvesting": true, "harvest_pixel": "#f2c285", "harvest_luma": 201, "bars": {"mana": 32, "food": 39, "health": 40, "inventory": 106, "action_points": 140}, "manufacturing_window": [1104, 327], "manufacturing_message": {"text": "you successfully created 1 potion of minor healing", "color": "green", "category": "success"}, "inventory_window": [1629, 686]}

frame is an integer in live mode, the file name in replay mode.
bars values are integers, or null when OCR could not read them (e.g. tesseract not installed). Consumers must tolerate null.
manufacturing_window / inventory_window are [x, y] title-bar coordinates of the (currently verified) window, or null when unverified.
manufacturing_message is {"text", "color", "category"} or null. text/color are null when no message is showing (category none); manufacturing_message itself is null when the Manufacturing window position is unverified. Colors: green (success/working), red (failed/failed_lost/hungry/overloaded/stopped).
Message categories: working, success, stopped, failed, failed_lost, hungry, overloaded, none, unknown.
This shape is the contract for the future observer — it should not change without updating tests/ and the observer accordingly.

Testing

tests/images/ holds the screenshots used as fixtures, grouped by expected behavior:

tests/images/harvesting/      is_harvesting must be True
tests/images/not_harvesting/  is_harvesting must be False
tests/images/corrupt/         must be skipped gracefully (None)
tests/images/manufacturing/   craft message at the pinned title position must
  <category>/*.png              classify to the folder's category
tests/images/inventory/       verify_window must pass at the pinned position

Tests are folder-driven: new edge cases are added by dropping a screenshot into the appropriate folder; no test code changes needed unless a new expected value must be pinned. tests/test_bars.py pins per-frame expectations (e.g. inventory = 110/108/106 depending on frame). tests/test_windows.py pins the Manufacturing title position per fixture frame (MFG_POSITION), the Inventory position, and a representative subset of find_windows recovery cases.

Dependencies and setup

System:

tesseract-ocr (OCR engine; the Python binding is pytesseract)
X11 (for live capture)

Python (installed in the project virtualenv .venv/, see requirements.txt):

python-xlib, pillow, pytesseract, pytest

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest tests/
.venv/bin/python main.py                 # live capture
.venv/bin/python main.py --replay tests/images/harvesting

Known limitations

Coordinates are resolution-specific (1920x1080). Resizing the game window requires recalibration of HARVEST_PIXEL / NUMBER_REGIONS.
OCR depends on the system tesseract binary; if missing, bar values log as null (harvesting detection is unaffected). Window/message OCR degrades the same way (verify_window returns False, messages read as null).
The panel appears to be semi-transparent; some scene pixels show through. So far this has not affected detection, but a very different background could.
Full-frame window recovery (find_windows) is not 100% reliable — it found the Manufacturing window in all 53 reference frames but missed the Inventory window in one harvesting frame. WindowTracker compensates by retrying on every frame while a window stays stale, so a missed recovery only delays the update by a frame or two. The cheap per-frame verify_window cache check is the workhorse (53/53) and recovery only runs after a window moves.
The steady-state loop adds roughly 0.5s/frame for the two window verifies plus message OCR; a full-frame recovery scan costs ~5-9s but only happens after a window moves.

11 KiB Raw Blame History