98 lines
4.9 KiB
Markdown
98 lines
4.9 KiB
Markdown
# InterferenceETL
|
|
|
|
InterferenceETL processes the newest complete hour of interference KPI files from Metrix storage. It reads seven ZIP/XLSX source types, converts each `Sheet0` to CSV, and creates one merged interference summary.
|
|
|
|
## Output
|
|
|
|
Each run writes one window directory:
|
|
|
|
```text
|
|
output/2026073110001100/
|
|
├── converted/
|
|
│ ├── 5G下FDD干扰监控_2026073110001100.csv
|
|
│ ├── 5G干扰监控_2026073110001100.csv
|
|
│ └── ... seven source CSV files
|
|
├── interference_summary_2026073110001100.csv
|
|
└── manifest.json
|
|
```
|
|
|
|
For every run, the script also reads the latest filename-dated XLSX from each configured CellData directory. It builds CellData CGI as `460-00-{eNB/gNB}-{CI}` and adds coordinates to matching interference rows. Unmatched rows keep empty coordinates.
|
|
|
|
The summary columns are:
|
|
|
|
```text
|
|
hour_start,hour_end,cgi,cell_name,interference_dbm,longitude,latitude
|
|
```
|
|
|
|
The script never modifies or deletes source storage files. By default it writes the complete selected hour through the Metrix Database API and keeps only that hour in the target table.
|
|
|
|
The database table contains one time column, `metric_time DATETIME`, which is the source KPI start time, plus nullable `longitude` and `latitude` columns. A same-hour rerun replaces that whole batch. After a successful insert, rows for every other hour are deleted in the same transaction. The script refuses to replace a newer database hour with an older source hour.
|
|
|
|
CGI is generated with fixed rules:
|
|
|
|
- NR (`5G干扰监控`, `700M干扰监控`): `{gNBplmn}-{gNBId}-{cellId}`.
|
|
- 4G (the other five source types): `460-00-{eNodeBID}-{小区ID}`, using each source schema's actual equivalent column names.
|
|
|
|
The Metrix API address, API Token, and database connection ID are constants in `runtime_config.py`; they are not project environment variables. Copy `runtime_config.example.py` to `runtime_config.py` and fill in the actual Token and the `conn_id` of `ShareMySQL`. The fixed database is `interference_etl`, the fixed table is `interference_hourly_summary`, and the script creates both automatically when they do not exist.
|
|
|
|
The five CellData directories are fixed in `CELL_DATA_DIRECTORIES`. In each directory, only XLSX files ending with a valid `YYYYMMDD.xlsx` date are considered, and the latest date is selected.
|
|
|
|
## Local development
|
|
|
|
```powershell
|
|
python -m pip install -r requirements.txt
|
|
python -m unittest discover -s tests -v
|
|
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output --no-database
|
|
```
|
|
|
|
Use `--window 2026073110001100` to require one exact source window. Without it, the script selects the newest window containing all seven source types and falls back if the newest observed window is incomplete.
|
|
|
|
## Metrix Script Management
|
|
|
|
The target server is offline. Build the dependency image locally, then generate a self-contained Script Management upload ZIP:
|
|
|
|
```powershell
|
|
docker build -t interference-etl-runtime:1.1 .
|
|
python scripts\build_offline_package.py
|
|
```
|
|
|
|
The generated `dist/InterferenceETL-offline-1.1.zip` contains `main.py`, `runtime_config.py`, a ZIP execution entry point, and a `vendor/` directory containing `openpyxl` and `et_xmlfile`. The builder verifies imports and ZIP execution using the standard `python:3.13.11-slim` image, so the uploaded workspace does not need an online `pip install` or a custom runtime image. The build fails when `runtime_config.py` is missing.
|
|
|
|
Create the project, then upload and extract the ZIP in its Script Management workspace. The normal file tree should contain `main.py` at `/workspace/main.py`. Use these project settings:
|
|
|
|
```text
|
|
Name: InterferenceETL
|
|
Language: python
|
|
Base image: python:3.13.11-slim
|
|
Network: bridge
|
|
Run command: python main.py
|
|
Timeout: 1800 seconds
|
|
```
|
|
|
|
If the server keeps the ZIP as one file instead of extracting it, use `python InterferenceETL-offline-1.1.zip` as the run command. The ZIP contains `__main__.py` for this mode.
|
|
|
|
Configure only run-specific source and output settings in the project environment:
|
|
|
|
```json
|
|
{
|
|
"METRIX_STORAGE_ID": "stg_4d9a910d72",
|
|
"INTERFERENCE_SOURCE_ROOT": "/网优日常优化数据文档/(勿删)干扰定时小时指标",
|
|
"INTERFERENCE_OUTPUT_DIR": "/workspace/output",
|
|
"INTERFERENCE_LOOKBACK_DAYS": "3"
|
|
}
|
|
```
|
|
|
|
Both storage reads and database writes use the Metrix API at `http://188.5.127.115:18271`. SSH remains the deployment and operational channel for uploading files, starting runs, and reading logs.
|
|
|
|
## CLI options
|
|
|
|
```text
|
|
--source-dir PATH Read a local directory tree instead of Metrix API
|
|
--output-dir PATH Output root, default output or INTERFERENCE_OUTPUT_DIR
|
|
--window WINDOW Require an exact 16-digit source window
|
|
--lookback-days N Number of newest date directories scanned, default 3
|
|
--storage-id ID Metrix storage connection ID
|
|
--root PATH Source directory in Metrix storage
|
|
--no-database Generate CSV files without writing the database
|
|
```
|