InterferenceETL/aidocs/project_context.md

InterferenceETL project context

2026-07-31: Initial hourly interference pipeline

• This repository is developed as the InterferenceETL submodule under Metrix and is intended to run later in Metrix Script Management on an hourly schedule.
• main.py reads the configured Metrix SFTP storage through Metrix API Token authentication or a local directory for tests. It selects the newest hour containing all seven known interference source types and falls back from a newer incomplete hour.
• Each selected ZIP must contain exactly one XLSX. The script strictly validates the known Sheet0 header, writes one full UTF-8-BOM CSV per source type, and writes one merged CSV containing only hour_start, hour_end, cgi, cell_name, and interference_dbm.
• NR CGI uses {gNBplmn}-{gNBId}-{cellId}. All 4G sources use 460-00-{node}-{cell}, mapped to the actual node and cell column names in each workbook schema.
• Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
• manifest.json records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
• The Metrix script container needs openpyxl==3.1.5, bridge networking, and python main.py. Runtime-specific source and output settings may be injected through project environment settings.
• Read-only validation against the current Metrix storage selected window 2026073110001100, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were 7 / 804 / 45 / 86 / 700 / 164 / 25 in EXPECTED_TYPES order; sampled CGI, cell name, and interference values matched the source workbooks.
• interference-etl-runtime:1.1 is now a local build helper only. Script projects use the standard python:3.13.11-slim image and upload dist/InterferenceETL-offline-1.1.zip, which vendors openpyxl and et_xmlfile at the workspace root.
• Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.

2026-07-31: Latest-hour MySQL retention

• Scheduled runs write to MySQL by default. File-only development runs must explicitly pass --no-database; the MySQL host, port, account, password, database, and table are constants at the top of main.py.
• The fixed script-owned database is interference_etl and the fixed table is interference_hourly_summary. Its only time column is metric_time DATETIME, populated from the source KPI 开始时间, so users can identify the hour represented by every row.
• One transaction deletes the selected hour for idempotent refresh, inserts its complete batch, then deletes every other database hour. Any failure rolls back the data changes, and an older selected hour cannot replace a newer hour already stored.
• Database retention never deletes or modifies Metrix Storage/SFTP source ZIP or XLSX files. Generated CSV retention remains a separate pending decision.

2026-07-31: Self-contained Script Management package

• scripts/build_offline_package.py copies the three runtime distributions from the verified dependency image, adds the application files and __main__.py, and creates a ZIP that Metrix can extract without preserving executable bits or symlinks. The archive can also run directly as python InterferenceETL-offline-1.1.zip when a Script Management server stores the upload without extracting it.
• The builder runs an import check inside python:3.13.11-slim; the package therefore needs neither a custom server image nor online package installation.
• The builder excludes bytecode caches, and the ignored dist/ directory is the only location for this generated package.

2026-07-31: Fixed CGI and database configuration

• Replaced the NR masterOperatorId passthrough with gNBplmn-gNBId-cellId. The remaining five 4G sources use the fixed 460-00 prefix plus their schema-specific base-station and cell ID columns.
• Removed CGI PLMN and MySQL connection command-line/environment options. Database configuration now lives as a small constant block at the top of main.py; the verified bridge-network address is 172.17.0.1:3306 because ShareMySQL DNS is unavailable from the default script network.
• Read-only online validation selected complete window 2026073115001600 and checked all 2,275 generated CGI values against the seven converted source CSV files. Per-source row counts were 6 / 1,198 / 48 / 91 / 731 / 179 / 22 in EXPECTED_TYPES order.
• Before the dedicated-database correction, a connection-only check returned the platform database metrix; that check did not create a table or modify data. Production use now targets only interference_etl.

2026-08-03: Dedicated database initialization

• InterferenceETL no longer writes into the Metrix platform database. Its fixed database is interference_etl, while the table remains interference_hourly_summary.
• The default MySQL path first connects without selecting a database, runs CREATE DATABASE IF NOT EXISTS interference_etl with utf8mb4, then reconnects to that database and creates the table if needed. The configured account therefore needs database creation permission on first run.
• Offline dependencies are stored below the workspace vendor/ directory instead of separate package directories at the project root. main.py prepends this directory to sys.path, so the run command remains python main.py.
• Summary CSV rows and database rows do not expose source_type or source_path. The database primary key is (metric_time, cgi); the current online hour was checked for CGI uniqueness before migrating from the former source-aware key.

2026-08-03: CellData coordinate enrichment

• Each run selects the latest filename-dated XLSX from five fixed CellData directories for 5G, 700M, reverse-activated 5G, TDD LTE, and FDD LTE. The filename date must end in YYYYMMDD.xlsx; source files remain read-only.
• CellData workbooks use sheet 小区信息表 and required columns eNB/gNB, CI, 经度, and 纬度. CellData CGI is always 460-00-{eNB/gNB}-{CI} and conflicting coordinates for the same CGI stop the run instead of silently overwriting data.
• The summary CSV now ends with longitude,latitude. The script-owned interference_hourly_summary table has nullable DECIMAL(10,6) columns with the same names; existing tables are migrated automatically and unmatched CGI values are stored as NULL.
• Read-only validation of the five 20260728 workbooks produced 66,250 unique CGI coordinates with no conflicts. Against the sampled latest-hour summary, 1,529 of 1,539 rows matched (99.35%); the remaining 10 rows correctly stay empty.

2026-08-03: Metrix API database output

• Storage reads and database writes now share one API Token authenticated MetrixApiClient. Direct PyMySQL access, MySQL host/account/password constants, and the PyMySQL offline dependency were removed.
• ApiSummaryStore creates the dedicated interference_etl database and interference_hourly_summary table through the Database API. It checks the latest stored hour, then uses /run-script with single_session=true and an explicit transaction to refresh one complete hour and delete all others.
• API address, Token, and ShareMySQL conn_id are constants loaded from ignored runtime_config.py; the tracked runtime_config.example.py documents the required names. The current API address is http://188.5.127.115:18271.
• Deployment remains SSH based. The Metrix Script Management API is not used for uploads, execution, or log inspection.
• Eleven containerized unit tests cover the pipeline, CellData enrichment, API database transaction, newer-hour protection, decimal validation, and API failure propagation.
• SSH deployment and a real Metrix runner execution succeeded with run eead5c315fb04836a6327be201f561e6. Window 2026080310001100 produced 1,991 rows, matched 1,977 coordinates, left 14 unmatched, and deleted 1,789 rows from the prior hour.
• Final API verification found exactly one stored hour (2026-08-03 10:00:00) and exactly six columns: metric_time, cgi, cell_name, interference_dbm, longitude, and latitude. The obsolete online source backup and environment-level API URL/Token entries were removed after the successful run.