auto_el/analyze/AGENTS.md

AGENTS.md

Guidance for AI agents working in this repo. Read before editing. The goal of these rules is to keep the code clean, consistent, and easy to extend without breaking the JSONL log contract or the test suite.

Project in one line

Capture the Eternal Lands X11 window, detect harvesting + the 5 HUD bar numbers, track the draggable Manufacturing/Inventory windows, OCR the craft status message, and append one JSON object per frame to a JSONL log for a future "observer" to consume.

Commands

Run everything through the project virtualenv — the system Python is externally managed (PEP 668) and lacks the deps:

source .venv/bin/activate          # then plain python/pytest work
# or, without activating:
.venv/bin/python -m pytest tests/ -v
.venv/bin/python main.py
.venv/bin/python main.py --replay tests/images/harvesting

• Always run the full test suite before finishing: .venv/bin/python -m pytest tests/
• If you add a Python dependency, add it to requirements.txt and install it into .venv.

Architecture and responsibilities

• capture.py — the ONLY module that touches X11. Returns PIL RGBA images.
• analyze.py — pure functions only (no I/O, no side effects). Every function takes a PIL image (or path) and returns data. Includes window detection (verify_window, find_windows), craft-message OCR (read_manufacturing_message), and classify_message.
• main.py — CLI, capture loop, replay mode, JSON record construction, and logging. Owns all file/stdout I/O. Also owns the window-position cache (WindowTracker): verify the cached position each frame, rescan the full frame only when a window moves.
• tests/ — folder-driven pytest tests plus fixture images.

Keep this split. New analysis logic goes in analyze.py. New ways to run the loop go in main.py. Never put capture logic in analyze.py or analysis in capture.py.

Code style

• No comments unless they explain WHY (non-obvious decisions, measured constants). Docstrings on public functions and modules are expected. Match the existing tone: short, declarative docstrings; a brief comment only where the reasoning is not obvious (e.g. "y0 starts at 1033 because row 1032 is the panel border, which confuses OCR").
• Standard library + the deps in requirements.txt only. Do not introduce new heavy dependencies without a strong reason and a requirements.txt update.
• Return plain dicts/lists of primitives (JSON-serializable) from analyze.py functions — never objects or PIL types — so results drop straight into the log.
• Function names are snake_case and descriptive (is_harvesting, read_bar_values, load_image).
• Module-level constants are UPPER_SNAKE_CASE; private helpers/constants are _leading_underscore.
• No type hints required, but if you add them keep them consistent across a file.

Screen-coordinate calibration

All pixel coordinates are measured values for a 1920x1080 window and live as constants at the top of analyze.py (HARVEST_PIXEL, HARVEST_LUMA_THRESHOLD, NUMBER_REGIONS).

Window coordinates are relative to the window title bar, never fixed screen positions — the player can drag the Manufacturing/Inventory windows. Calibrated constants: MESSAGE_SEARCH (the craft message sits at (title_x - 161, title_y + 121), verified at 5 different window positions) and the TITLE_VERIFY_CROP/FIND_TITLE_OFFSETS tuning.

• When you change a coordinate or threshold, you MUST re-run the tests. If you have a screenshot proving the new value, drop it in the right tests/images/ folder first so the tests verify your change.
• Never "adjust" a coordinate in a test to make it pass. Verify against the actual images.
• If the game window resolution changes, the calibration must be redone from real screenshots — do not guess coordinates.

Tests

• Tests are folder-driven:
  • tests/images/harvesting/*.png → is_harvesting must be True
  • tests/images/not_harvesting/*.png → must be False
  • tests/images/corrupt/*.png → load_image must return None, no crash
  • tests/images/manufacturing/<category>/*.png → the craft message read at the pinned title position must classify to <category>
  • tests/images/inventory/*.png → verify_window at the pinned position
• Adding an edge case = dropping a PNG into the right folder. Do that instead of writing one-off test code, unless a new expected value must be pinned (then update tests/test_bars.py, e.g. INVENTORY_EXPECTED, or tests/test_windows.py, e.g. MFG_POSITION).
• Keep fixture images small in number and real (actual captured frames), not synthetic.
• Pinned OCR values in test_bars.py are ground truth from real frames. If a frame's expected value must change, verify the number by reading the image — never just update the test to match a new wrong OCR result.

The JSONL log contract

main.py appends one JSON object per line to analysis.jsonl (default). Shape:

{"time": "…", "frame": 4, "is_harvesting": true, "harvest_pixel": "#f2c285", "harvest_luma": 201, "bars": {"mana": 32, "food": 39, "health": 40, "inventory": 106, "action_points": 140}, "manufacturing_window": [1104, 327], "manufacturing_message": {"text": "you successfully created 1 potion of minor healing", "color": "green", "category": "success"}, "inventory_window": [1629, 686]}

• The future observer parses this log. Do not change field names, types, or add/remove fields casually. If you must extend it, add new keys (never repurpose existing ones) and update design.md.
• bars.* may be null when OCR fails — consumers must tolerate that; keep it that way.
• manufacturing_window / inventory_window are [x, y] title-bar coordinates, or null when the window is not currently verified. manufacturing_message is {text, color, category} (text/color are null when no message shows), or null when the window position is unverified. Message categories: working, success, stopped, failed, failed_lost, hungry, overloaded, none, unknown. Consumers must tolerate all null cases.
• Keep the log append-only and line-per-record (JSONL). It is tail-friendly.

Error handling

• Analysis functions must never crash the capture loop. analyze.py catches per-item failures (e.g. OCR returns None on error) and returns None rather than raising.
• Optional dependency pattern: pytesseract is imported inside try/except ImportError and the module degrades gracefully (pytesseract = None). Follow this pattern for any new optional dep.
• Corrupt images → load_image returns None; callers skip them.

Docs

• design.md — the source of truth for capabilities, calibration, architecture, and the log format. Keep it in sync when you change behavior.
• AGENTS.md — these rules. Update it if a convention changes.