InterferenceETL/aidocs/project_context.md

InterferenceETL project context

2026-07-31: Initial hourly interference pipeline

• This repository is developed as the InterferenceETL submodule under Metrix and is intended to run later in Metrix Script Management on an hourly schedule.
• main.py reads the configured Metrix SFTP storage through Metrix API Token authentication or a local directory for tests. It selects the newest hour containing all seven known interference source types and falls back from a newer incomplete hour.
• Each selected ZIP must contain exactly one XLSX. The script strictly validates the known Sheet0 header, writes one full UTF-8-BOM CSV per source type, and writes one merged CSV containing only hour_start, hour_end, cgi, cell_name, and interference_dbm.
• NR CGI uses {gNBplmn}-{gNBId}-{cellId}. All 4G sources use 460-00-{node}-{cell}, mapped to the actual node and cell column names in each workbook schema.
• Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
• manifest.json records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
• The Metrix script container needs openpyxl==3.1.5, bridge networking, and python main.py. Runtime-specific source and output settings may be injected through project environment settings.
• Read-only validation against the current Metrix storage selected window 2026073110001100, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were 7 / 804 / 45 / 86 / 700 / 164 / 25 in EXPECTED_TYPES order; sampled CGI, cell name, and interference values matched the source workbooks.
• interference-etl-runtime:1.1 is now a local build helper only. Script projects use the standard python:3.13.11-slim image and upload dist/InterferenceETL-offline-1.1.zip, which vendors openpyxl and et_xmlfile at the workspace root.
• Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.

2026-07-31: Latest-hour MySQL retention

• Scheduled runs write to MySQL by default. File-only development runs must explicitly pass --no-database; the MySQL host, port, account, password, database, and table are constants at the top of main.py.
• The fixed script-owned database is interference_etl and the fixed table is interference_hourly_summary. Its only time column is metric_time DATETIME, populated from the source KPI 开始时间, so users can identify the hour represented by every row.
• One transaction deletes the selected hour for idempotent refresh, inserts its complete batch, then deletes every other database hour. Any failure rolls back the data changes, and an older selected hour cannot replace a newer hour already stored.
• Database retention never deletes or modifies Metrix Storage/SFTP source ZIP or XLSX files. Generated CSV retention remains a separate pending decision.

2026-07-31: Self-contained Script Management package

• scripts/build_offline_package.py copies the three runtime distributions from the verified dependency image, adds the application files and __main__.py, and creates a ZIP that Metrix can extract without preserving executable bits or symlinks. The archive can also run directly as python InterferenceETL-offline-1.1.zip when a Script Management server stores the upload without extracting it.
• The builder runs an import check inside python:3.13.11-slim; the package therefore needs neither a custom server image nor online package installation.
• The builder excludes bytecode caches, and the ignored dist/ directory is the only location for this generated package.

2026-07-31: Fixed CGI and database configuration

• Replaced the NR masterOperatorId passthrough with gNBplmn-gNBId-cellId. The remaining five 4G sources use the fixed 460-00 prefix plus their schema-specific base-station and cell ID columns.
• Removed CGI PLMN and MySQL connection command-line/environment options. Database configuration now lives as a small constant block at the top of main.py; the verified bridge-network address is 172.17.0.1:3306 because ShareMySQL DNS is unavailable from the default script network.
• Read-only online validation selected complete window 2026073115001600 and checked all 2,275 generated CGI values against the seven converted source CSV files. Per-source row counts were 6 / 1,198 / 48 / 91 / 731 / 179 / 22 in EXPECTED_TYPES order.
• Before the dedicated-database correction, a connection-only check returned the platform database metrix; that check did not create a table or modify data. Production use now targets only interference_etl.

2026-08-03: Dedicated database initialization

• InterferenceETL no longer writes into the Metrix platform database. Its fixed database is interference_etl, while the table remains interference_hourly_summary.
• The default MySQL path first connects without selecting a database, runs CREATE DATABASE IF NOT EXISTS interference_etl with utf8mb4, then reconnects to that database and creates the table if needed. The configured account therefore needs database creation permission on first run.
• Offline dependencies are stored below the workspace vendor/ directory instead of separate package directories at the project root. main.py prepends this directory to sys.path, so the run command remains python main.py.
• Summary CSV rows and database rows do not expose source_type or source_path. The database primary key is (metric_time, cgi); the current online hour was checked for CGI uniqueness before migrating from the former source-aware key.

2026-08-03: CellData coordinate enrichment

• Each run selects the latest filename-dated XLSX from five fixed CellData directories for 5G, 700M, reverse-activated 5G, TDD LTE, and FDD LTE. The filename date must end in YYYYMMDD.xlsx; source files remain read-only.
• CellData workbooks use sheet 小区信息表 and required columns eNB/gNB, CI, 经度, and 纬度. CellData CGI is always 460-00-{eNB/gNB}-{CI} and conflicting coordinates for the same CGI stop the run instead of silently overwriting data.
• The summary CSV now ends with longitude,latitude. The script-owned interference_hourly_summary table has nullable DECIMAL(10,6) columns with the same names; existing tables are migrated automatically and unmatched CGI values are stored as NULL.
• Read-only validation of the five 20260728 workbooks produced 66,250 unique CGI coordinates with no conflicts. Against the sampled latest-hour summary, 1,529 of 1,539 rows matched (99.35%); the remaining 10 rows correctly stay empty.

2026-08-03: Metrix API database output

• Storage reads and database writes now share one API Token authenticated MetrixApiClient. Direct PyMySQL access, MySQL host/account/password constants, and the PyMySQL offline dependency were removed.
• ApiSummaryStore creates the dedicated interference_etl database and interference_hourly_summary table through the Database API. It checks the latest stored hour, then uses /run-script with single_session=true and an explicit transaction to refresh one complete hour and delete all others.
• API address, Token, and ShareMySQL conn_id are constants loaded from ignored runtime_config.py; the tracked runtime_config.example.py documents the required names. The current API address is http://188.5.127.115:18271.
• Deployment remains SSH based. The Metrix Script Management API is not used for uploads, execution, or log inspection.
• Eleven containerized unit tests cover the pipeline, CellData enrichment, API database transaction, newer-hour protection, decimal validation, and API failure propagation.
• SSH deployment and a real Metrix runner execution succeeded with run eead5c315fb04836a6327be201f561e6. Window 2026080310001100 produced 1,991 rows, matched 1,977 coordinates, left 14 unmatched, and deleted 1,789 rows from the prior hour.
• Final API verification found exactly one stored hour (2026-08-03 10:00:00) and exactly six columns: metric_time, cgi, cell_name, interference_dbm, longitude, and latitude. The obsolete online source backup and environment-level API URL/Token entries were removed after the successful run.

2026-08-03: CellData azimuth enrichment

• All five latest CellData workbooks expose direction angle through column 方向角. CellData metadata now maps each 460-00-{eNB/gNB}-{CI} to longitude, latitude, and azimuth.
• Summary CSV and interference_hourly_summary add azimuth. The database type is DECIMAL(6,2) NOT NULL DEFAULT 0; existing tables add the column automatically. Empty CellData direction angles and unmatched CGI values both become 0, while unmatched coordinates remain empty/NULL.
• Real CellData validation loaded 66,524 CGI metadata rows: 54,129 non-zero azimuth values and 12,395 zero/default values. Full window 2026080313001400 produced 1,863 summary rows, including 1,677 non-zero azimuth values and 186 zero values.
• SSH deployment run e5462179fa8c4f9f9c2dfb2e4541e048 succeeded. Database API verification found exactly one hour (2026-08-03 13:00:00), 1,863 rows, and columns metric_time,cgi,cell_name,interference_dbm,longitude,latitude,azimuth.
• Twelve containerized unit tests pass, including empty azimuth defaulting and CellData metadata conflict detection.

2026-08-05: Network type and high-interference filtering

• Summary CSV and interference_hourly_summary add network_type: the two 5G... source types map to 2.6G, the two 700M... source types map to 700M, and SDR FDD/TDD plus reverse-activated RD map to 4G.
• Only high-interference rows enter the summary and database. 2.6G keeps values greater than or equal to -107 dBm; 700M and 4G keep values greater than or equal to -110 dBm. Values strictly below those thresholds are discarded; equal values remain.
• The seven converted source CSV files remain unfiltered source conversions. manifest.json and stdout record threshold_filtered_rows for the summary filter.
• Real read-only validation of window 2026080514001500 reduced 1,868 source rows to 1,109 summary rows: 2.6G=137, 700M=108, 4G=864, with 759 lower-interference rows removed. Thirteen containerized unit tests pass.
• SSH deployment run 98a44e88b3684bd1b9ad7edcb866fb17 succeeded for the same window. The database contains exactly one hour (2026-08-05 14:00:00) and 1,109 rows; grouped API verification found zero threshold violations and confirmed minimums 2.6G=-106.970, 700M=-110.000, and 4G=-109.996.

2026-08-06: Nearby high-interference cell count

• Summary CSV and interference_hourly_summary add nearby_count, the number of other retained high-interference cells within 1 km during the same selected hour. Counts include all network types, exclude the row itself, include the exact 1 km boundary, and default to 0 when coordinates are unavailable.
• Candidate lookup uses a dependency-free 1 km Earth-centered three-dimensional grid index. Only the current and 26 adjacent buckets are checked, then Haversine distance confirms the exact radius; this avoids a full all-pairs scan while preserving distance accuracy.
• Unit tests pass in both the project virtual environment and interference-etl-runtime:1.1 image. A read-only real run for window 2026080613001400 produced 1,116 rows, including 1,110 with coordinates, 925 with non-zero nearby counts, and a maximum count of 39. All 1,116 indexed results matched a separate brute-force comparison, which found 3,973 qualifying pairs.
• SSH deployment and Metrix runner execution 8f2fd51d7de14c93b4b0eb9bee8fafa0 succeeded for the same window. Database API verification found only 2026-08-06 13:00:00, with 1,116 rows, 925 non-zero counts, a maximum of 39, and 3,973 nearby pairs. All six rows without coordinates have count 0.

2026-08-06: Adaptive source waiting and history export

• Source recognition now fixes only the seven known prefixes and final _YYYYMMDDHHMMHHMM.zip; middle text may change with provider granularity. The first 12 digits are rounded down to natural 15-minute boundaries, while the final four end-time digits do not participate in grouping.
• Each run follows only the group containing the globally newest source start time. It does not fall back or backfill older groups. Missing sources or no matching files produce status=waiting and a successful exit; storage API failures remain task failures. Multiple files from one source in a group select the latest source start time.
• All workbook row start times must round into the target group. Result CSV and database rows use one normalized metric_time and the database result columns only; hour_start and hour_end were removed from summary CSV output.
• Database time is checked before source ZIP and CellData downloads. A newer or equal database time skips source processing. Successful database output is archived through the Storage API under 干扰历史数据/YYYY-MM-DD/干扰数据处理结果_YYYYMMDDHHMMSS.csv; an equal database time with a missing/empty history file is exported from the database and repaired.
• Nineteen tests pass on Windows and in the offline runtime image. Live read-only selection saw incomplete newest group 2026-08-06 15:00:00, correctly waited for missing 5G下FDD干扰监控, and did not produce output. Explicit read-only validation of complete group 2026-08-06 14:00:00 produced 1,092 rows with one normalized metric_time and the expected nine-column result schema.
• SSH deployment run 7a7c0bba3a0c41f5a5a9e83ad25ff2fd processed the completed 2026-08-06 15:00:00 group successfully. It retained 1,082 high-interference rows, removed 853 lower-interference rows, matched coordinates for 1,074 rows, left 8 unmatched, and replaced 1,116 rows from the previous database time.
• Database verification found only 2026-08-06 15:00:00 and 1,082 rows. Of those, 887 have non-zero nearby_count, the maximum is 40, and all 8 rows without coordinates remain valid. The history CSV was uploaded with the expected nine columns and 1,082 rows to 干扰历史数据/2026-08-06/干扰数据处理结果_20260806150000.csv.
• A second run e1c81ada63234b3c85a377e1b46a1370 returned success with status=skipped, confirming that an existing database time plus a non-empty history file avoids duplicate work.
• MetrixApiClient uses a proxy-free opener because the API is an intranet service and the Windows system proxy previously converted direct requests into 502 responses. The verified Metrix API remains http://188.5.127.115:18271; external port 9082 currently serves CapacityReport and must not be used by this script.

2026-08-07: Previous-period comparison fields

• Summary CSV, database table, and history CSV add nullable previous-period fields: prev_interference_dbm and prev_nearby_count.
• Before downloading source ZIP/XLSX files, the runner still checks the current database time. When a newer source group is being processed, it reads the currently retained database rows before replacement and maps them by CGI. The current row receives the previous row's interference_dbm and nearby_count; unmatched CGI values remain empty in CSV and NULL in the database.
• On the first run, when the database has no previous group, both fields remain empty/NULL. Existing-table migration adds both columns automatically. Twenty containerized tests pass, including first-run empty values, CGI matching, and nullable column migration.
• Offline package SHA-256 is 74EEA84058C5F14E7108B840561C070DC47752D67FEC68262F0CC6C78EE8EC0D. SSH deployment run 984ffc6f98024a2d81b63d4359aabc71 processed 2026-08-07 07:00:00 with 820 rows and deleted 1,137 rows from the prior database time.
• Production verification matched 616 current CGI values to the previous period and left both previous fields NULL for the remaining 204 CGI values. prev_nearby_count matched exactly; prev_interference_dbm uses the table's existing DECIMAL(10,3) precision. Repeat run 6b38030f6d9e4ab1a0ee6a4ba0dc5d6e returned status=skipped.

2026-08-07: GBK Storage history output

• Only final history CSV files uploaded to Metrix Storage use GBK without a UTF-8 BOM, including database-based history repairs. The seven local source conversions and local merged summary remain UTF-8 with BOM; manifest.json, API JSON, and database character sets remain unchanged.
• Production data contains non-breaking spaces (U+00A0), which Python's GBK codec cannot encode. History-output normalization converts them to ordinary spaces while keeping strict encoding for every other unsupported character.
• Encoding behavior is separated through LOCAL_CSV_ENCODING and HISTORY_CSV_ENCODING; tests verify the local UTF-8 BOM and GBK history parsing independently.
• Twenty-one containerized tests and offline-package execution pass. The deployed package SHA-256 is 11B0010E8FDFEC32058BC8B282E5A5608FC92C48572611A3F9F4632962C18D98.
• The current 2026-08-07 14:00:00 database result was exported directly through the history-only path and replaced in Storage as a verified GBK CSV: 1,082 rows, 150,251 bytes, no UTF-8 BOM, and the expected eleven columns. The temporary UTF-8 backup was deleted after validation.
• The newer 2026-08-07 15:00:00 source group currently contains duplicate CGI rows, for example 460-00-122737-22. Its database transaction rolled back on the existing (metric_time, cgi) primary key, so the retained 14:00 database and history result were not replaced. Duplicate-row selection requires a separate business rule.

2026-08-11: Per-network-type nearby counts

• Summary CSV, database table, and history CSV add nearby_26g, nearby_700m, nearby_tdd, and nearby_fdd: the same 1 km high-interference neighbors split by the neighbor's network_type. Each defaults to 0 and the four values always sum to nearby_count.
• Previous-period fields prev_nearby_26g, prev_nearby_700m, prev_nearby_tdd, and prev_nearby_fdd follow the prev_nearby_count rules: matched by CGI from the previously retained database time, empty/NULL when unmatched or on the first run. On the first run after this upgrade the previous rows only have NULL in the new columns, so the previous split fields stay empty for that one period and self-heal afterwards.
• Existing tables migrate automatically: current count columns are INT NOT NULL DEFAULT 0, previous count columns are INT NULL. The history export SELECT is now generated from SUMMARY_HEADER, so result columns stay in one place. Twenty-two tests pass locally and in the runtime image.
• Deployed package SHA-256 is 638EEC791FACA55937FCF335402F40776B0749375216D27FBB9F8F1864007EBE. A first run (9ccdc7cba5bc48fd8c57a698de3dde84) returned status=skipped and migrated the schema; all eight new columns were confirmed through the Database API.
• Because the 16:00 sources were late, the user chose to truncate interference_hourly_summary and reprocess the 15:00 group instead of waiting. Run 23d870b5d1434a548520c7fc2413e2ba inserted 1,062 rows (869 threshold-filtered, 0 duplicate CGI, 1,059 coordinates matched).
• Production verification: every row satisfies nearby_26g + nearby_700m + nearby_tdd + nearby_fdd = nearby_count (totals 847/216/1,194/3,229 = 5,486); cross-type pair sums are symmetric; network_type contains only 2.6G=145, 700M=68, TDD=206, FDD=643. The history CSV was rewritten in GBK with the nineteen-column header and 1,062 rows.
• The reprocessed batch has empty previous-period fields by design because the 14:00 data no longer existed; from the next group onward the previous fields, including the new splits, populate normally.

2026-08-11: Network type relabeled to 2.6G/700M/TDD/FDD

• network_type keeps its column name but now labels radio type per source prefix: 5G干扰监控 is 2.6G (the user treats 2.6G as the unambiguous 5G label because 700M is also 5G), 700M干扰监控 is 700M, SDR_TDD干扰监控 and 反开RD干扰监控 are TDD, and the three FDD-schema sources (SDR_FDD干扰监控, 5G下FDD干扰监控, 700M下FDD干扰监控) are FDD. The old 2.6G/700M/4G grouping is gone.
• Real source filenames only carry these keywords in the fixed prefix; the middle text (for example LWP_每小时_过滤110) has no usable network keyword.
• High-interference thresholds are unchanged and now keyed per source type through MIN_INTERFERENCE_BY_SOURCE: the two 5G-directory sources keep -107 and the other five keep -110, so relabeling 5G下FDD干扰监控 to FDD does not change which rows are kept.
• No database migration is needed: the column type stays VARCHAR(16) and old-label rows disappear on the next successful replacement run. Twenty-two tests pass locally.

2026-08-07: Duplicate CGI selection

• Retained high-interference rows are deduplicated by CGI before nearby counting and database insertion. When duplicates exist, the row with the numerically largest interference_dbm is kept; equal values keep the first encountered row. Converted source CSVs remain complete.
• The deduplication count is recorded as duplicate_cgi_rows in manifest.json and stdout. This prevents the existing (metric_time, cgi) database primary key from failing and prevents duplicate source rows from inflating nearby_count.
• Deployment package from commit bb0e3d0 has SHA-256 ED5B5C9A543C9E3283642D6A76D6693A200DA938B28449C09CEE2FBCED534121. Successful run 068761c633634c4fb1a66aff921c62ed processed 2026-08-07 15:00:00: 895 rows filtered by threshold, 582 duplicate CGI rows removed, 1,197 rows inserted, and 1,082 old rows deleted.
• Final Storage history verification found 1,197 rows, eleven expected columns, GBK encoding without a UTF-8 BOM, and one metric time 2026-08-07 15:00:00. A subsequent manual run fc38be72da6d496ca5e84e529f79ab53 returned status=skipped.