InterferenceETL/aidocs/project_context.md

InterferenceETL project context

2026-07-31: Initial hourly interference pipeline

• This repository is developed as the InterferenceETL submodule under Metrix and is intended to run later in Metrix Script Management on an hourly schedule.
• main.py reads the configured Metrix SFTP storage through Metrix API Token authentication or a local directory for tests. It selects the newest hour containing all seven known interference source types and falls back from a newer incomplete hour.
• Each selected ZIP must contain exactly one XLSX. The script strictly validates the known Sheet0 header, writes one full UTF-8-BOM CSV per source type, and writes one merged CSV containing hour_start, hour_end, source_type, cgi, cell_name, interference_dbm, and source_path.
• LTE/SDR CGI currently uses 460-00-{node}-{cell}; NR uses masterOperatorId unchanged. This is an explicit assumption pending user confirmation.
• Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
• manifest.json records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
• The Metrix script container needs openpyxl==3.1.5, PyMySQL==1.1.2, bridge networking, python main.py, reachable API/MySQL addresses, and credentials injected through project environment settings. Secrets must not be committed.
• Read-only validation against the current Metrix storage selected window 2026073110001100, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were 7 / 804 / 45 / 86 / 700 / 164 / 25 in EXPECTED_TYPES order; sampled CGI, cell name, and interference values matched the source workbooks.
• The current Linux/amd64 runtime image is interference-etl-runtime:1.1. Its offline archive is generated locally at dist/interference-etl-runtime-1.1.tar (46,944,256 bytes, SHA-256 BF92BF776D52865CA484D0B7D1B10336C998A07E8310D6F01576C74A65A1FF6A) and remains excluded from Git.
• Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.

2026-07-31: Latest-hour MySQL retention

• Scheduled runs write to MySQL by default. File-only development runs must explicitly pass --no-database; connection values come from INTERFERENCE_DB_* environment variables.
• The default table is interference_hourly_summary. Its only time column is metric_time DATETIME, populated from the source KPI 开始时间, so users can identify the hour represented by every row.
• One transaction deletes the selected hour for idempotent refresh, inserts its complete batch, then deletes every other database hour. Any failure rolls back the data changes, and an older selected hour cannot replace a newer hour already stored.
• Database retention never deletes or modifies Metrix Storage/SFTP source ZIP or XLSX files. Generated CSV retention remains a separate pending decision.