feat: 自适应等待干扰数据并归档结果

This commit is contained in:
2026-08-06 17:02:20 +08:00
parent 1dad8b3465
commit 0c85dd93c2
4 changed files with 521 additions and 167 deletions
+24 -10
View File
@@ -1,36 +1,50 @@
# InterferenceETL
InterferenceETL processes the newest complete hour of interference KPI files from Metrix storage. It reads seven ZIP/XLSX source types, converts each `Sheet0` to CSV, and creates one merged interference summary.
InterferenceETL follows the newest interference KPI time group in Metrix storage. It reads seven ZIP/XLSX source types, converts each `Sheet0` to CSV, and creates one merged high-interference summary.
Source file names are identified by one of the seven fixed prefixes plus a final `_YYYYMMDDHHMMHHMM.zip`; text between the prefix and time is unrestricted. Only the first `YYYYMMDDHHMM` start time is used for grouping. Start times are rounded down to natural 15-minute boundaries, so `12:00` through `12:14` belong to `12:00`. This also works unchanged when all providers switch together to hourly or daily delivery.
Every run follows only the group containing the newest source start time. It never falls back or backfills an older group. If any source type is missing, the script prints `status=waiting` with the missing types and exits successfully. When one source has multiple files in the group, its latest start time wins.
## Output
Each run writes one window directory:
```text
output/2026073110001100/
output/20260731100000/
├── converted/
│ ├── 5G下FDD干扰监控_2026073110001100.csv
│ ├── 5G干扰监控_2026073110001100.csv
│ └── ... seven source CSV files
├── interference_summary_2026073110001100.csv
├── interference_summary_20260731100000.csv
└── manifest.json
```
For every run, the script also reads the latest filename-dated XLSX from each configured CellData directory. It builds CellData CGI as `460-00-{eNB/gNB}-{CI}` and adds coordinates plus direction angle to matching interference rows. Unmatched rows keep empty coordinates and use `0` for azimuth.
For every processed group, the script also reads the latest filename-dated XLSX from each configured CellData directory. It builds CellData CGI as `460-00-{eNB/gNB}-{CI}` and adds coordinates plus direction angle to matching interference rows. Unmatched rows keep empty coordinates and use `0` for azimuth.
The summary columns are:
```text
hour_start,hour_end,network_type,cgi,cell_name,interference_dbm,longitude,latitude,azimuth,nearby_count
metric_time,network_type,cgi,cell_name,interference_dbm,longitude,latitude,azimuth,nearby_count
```
All selected workbook rows must belong to the selected natural 15-minute group. Every summary row receives the same normalized `metric_time`; a row outside the group fails the run before database or history output.
`network_type` is derived from the seven source types: both `5G...` sources are `2.6G`, both `700M...` sources are `700M`, and SDR/反开 sources are `4G`. The summary and database keep only high-interference rows: `2.6G >= -107 dBm` and `700M/4G >= -110 dBm`. Converted source CSV files remain full source conversions.
`nearby_count` is the number of other high-interference cells from the same selected hour within 1 km, across all network types. The cell itself is excluded, the 1 km boundary is included, and rows without coordinates use `0`. Candidate cells are found through a 1 km spatial grid index and confirmed with exact Haversine distance.
`nearby_count` is the number of other high-interference cells from the same selected time group within 1 km, across all network types. The cell itself is excluded, the 1 km boundary is included, and rows without coordinates use `0`. Candidate cells are found through a 1 km spatial grid index and confirmed with exact Haversine distance.
The script never modifies or deletes source storage files. By default it writes the complete selected hour through the Metrix Database API and keeps only that hour in the target table.
The script never modifies or deletes source storage files. Before downloading source ZIP files or CellData, it compares the normalized target time with the database. A target already present, or older than the database, is not processed again.
The database table contains one time column, `metric_time DATETIME`, which is the source KPI start time, `network_type VARCHAR(16)`, nullable `longitude` and `latitude` columns, `azimuth DECIMAL(6,2) NOT NULL DEFAULT 0`, and `nearby_count INT NOT NULL DEFAULT 0`. A same-hour rerun replaces that whole batch. After a successful insert, rows for every other hour are deleted in the same transaction. The script refuses to replace a newer database hour with an older source hour.
The database table contains normalized `metric_time DATETIME`, `network_type VARCHAR(16)`, nullable `longitude` and `latitude`, `azimuth DECIMAL(6,2) NOT NULL DEFAULT 0`, and `nearby_count INT NOT NULL DEFAULT 0`. A successful transaction replaces the target batch and deletes every other database time, so the table retains only the latest processed group.
After the database succeeds, the same nine result columns are uploaded as UTF-8-BOM CSV to:
```text
/网优日常优化数据文档/(勿删)干扰定时小时指标/干扰历史数据/YYYY-MM-DD/干扰数据处理结果_YYYYMMDDHHMMSS.csv
```
The filename time comes from normalized `metric_time`, never the server clock. If the database already contains the target time but its history CSV is missing or empty, the script exports that time from the database and repairs the history file without reprocessing source data.
CGI is generated with fixed rules:
@@ -49,7 +63,7 @@ python -m unittest discover -s tests -v
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output --no-database
```
Use `--window 2026073110001100` to require one exact source window. Without it, the script selects the newest window containing all seven source types and falls back if the newest observed window is incomplete.
Use `--window 2026073110001100` to select the natural 15-minute group containing that source start time. Without it, the newest observed source start time determines the group. Incomplete groups wait and never fall back.
## Metrix Script Management
@@ -93,7 +107,7 @@ Both storage reads and database writes use the Metrix API at `http://188.5.127.1
```text
--source-dir PATH Read a local directory tree instead of Metrix API
--output-dir PATH Output root, default output or INTERFERENCE_OUTPUT_DIR
--window WINDOW Require an exact 16-digit source window
--window WINDOW Select the natural 15-minute group for a 16-digit source window
--lookback-days N Number of newest date directories scanned, default 3
--storage-id ID Metrix storage connection ID
--root PATH Source directory in Metrix storage