Metrix/InterferenceETL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 改用Metrix API写入干扰数据

Browse Source
This commit is contained in:
nixevol
2026-08-03 12:05:20 +08:00
parent 470763a818
commit 9776391ad1
8 changed files with 294 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files
aidocs/project_context.md
+12 -2
View File
@@ -8,9 +8,9 @@
- NR CGI uses `{gNBplmn}-{gNBId}-{cellId}`. All 4G sources use `460-00-{node}-{cell}`, mapped to the actual node and cell column names in each workbook schema.
- Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
- `manifest.json` records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
- The Metrix script container needs `openpyxl==3.1.5`, `PyMySQL==1.1.2`, bridge networking, and `python main.py`. Only Metrix Storage API settings are injected through project environment settings.
- The Metrix script container needs `openpyxl==3.1.5`, bridge networking, and `python main.py`. Runtime-specific source and output settings may be injected through project environment settings.
- Read-only validation against the current Metrix storage selected window `2026073110001100`, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were `7 / 804 / 45 / 86 / 700 / 164 / 25` in `EXPECTED_TYPES` order; sampled CGI, cell name, and interference values matched the source workbooks.
- `interference-etl-runtime:1.1` is now a local build helper only. Script projects use the standard `python:3.13.11-slim` image and upload `dist/InterferenceETL-offline-1.1.zip`, which vendors `openpyxl`, `et_xmlfile`, and `PyMySQL` at the workspace root.
- `interference-etl-runtime:1.1` is now a local build helper only. Script projects use the standard `python:3.13.11-slim` image and upload `dist/InterferenceETL-offline-1.1.zip`, which vendors `openpyxl` and `et_xmlfile` at the workspace root.
- Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.
## 2026-07-31: Latest-hour MySQL retention
@@ -46,3 +46,13 @@
- CellData workbooks use sheet `小区信息表` and required columns `eNB/gNB`, `CI`, `经度`, and `纬度`. CellData CGI is always `460-00-{eNB/gNB}-{CI}` and conflicting coordinates for the same CGI stop the run instead of silently overwriting data.
- The summary CSV now ends with `longitude,latitude`. The script-owned `interference_hourly_summary` table has nullable `DECIMAL(10,6)` columns with the same names; existing tables are migrated automatically and unmatched CGI values are stored as `NULL`.
- Read-only validation of the five `20260728` workbooks produced 66,250 unique CGI coordinates with no conflicts. Against the sampled latest-hour summary, 1,529 of 1,539 rows matched (99.35%); the remaining 10 rows correctly stay empty.
## 2026-08-03: Metrix API database output
- Storage reads and database writes now share one API Token authenticated `MetrixApiClient`. Direct PyMySQL access, MySQL host/account/password constants, and the `PyMySQL` offline dependency were removed.
- `ApiSummaryStore` creates the dedicated `interference_etl` database and `interference_hourly_summary` table through the Database API. It checks the latest stored hour, then uses `/run-script` with `single_session=true` and an explicit transaction to refresh one complete hour and delete all others.
- API address, Token, and ShareMySQL `conn_id` are constants loaded from ignored `runtime_config.py`; the tracked `runtime_config.example.py` documents the required names. The current API address is `http://188.5.127.115:18271`.
- Deployment remains SSH based. The Metrix Script Management API is not used for uploads, execution, or log inspection.
- Eleven containerized unit tests cover the pipeline, CellData enrichment, API database transaction, newer-hour protection, decimal validation, and API failure propagation.
- SSH deployment and a real Metrix runner execution succeeded with run `eead5c315fb04836a6327be201f561e6`. Window `2026080310001100` produced 1,991 rows, matched 1,977 coordinates, left 14 unmatched, and deleted 1,789 rows from the prior hour.
- Final API verification found exactly one stored hour (`2026-08-03 10:00:00`) and exactly six columns: `metric_time`, `cgi`, `cell_name`, `interference_dbm`, `longitude`, and `latitude`. The obsolete online source backup and environment-level API URL/Token entries were removed after the successful run.