feat: 实现干扰指标最新小时入库

This commit is contained in:
2026-07-31 16:02:53 +08:00
parent 1c0f0eff62
commit dbe2024b6e
8 changed files with 344 additions and 18 deletions
+3
View File
@@ -0,0 +1,3 @@
*
!Dockerfile
!requirements.txt
+2 -1
View File
@@ -1,6 +1,7 @@
FROM python:3.13.11-slim
RUN pip install --no-cache-dir openpyxl==3.1.5
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
WORKDIR /workspace
+23 -7
View File
@@ -22,14 +22,16 @@ The summary columns are:
hour_start,hour_end,source_type,cgi,cell_name,interference_dbm,source_path
```
The script never modifies source storage and does not write a database yet. Pending decisions are recorded in `docs/assumptions.md`.
The script never modifies or deletes source storage files. By default it writes the complete selected hour to MySQL and keeps only that hour in the target table.
The database table contains one time column, `metric_time DATETIME`, which is the source KPI start time. A same-hour rerun replaces that whole batch. After a successful insert, rows for every other hour are deleted in the same transaction. The script refuses to replace a newer database hour with an older source hour.
## Local development
```powershell
python -m pip install -r requirements.txt
python -m unittest discover -s tests -v
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output --no-database
```
Use `--window 2026073110001100` to require one exact source window. Without it, the script selects the newest window containing all seven source types and falls back if the newest observed window is incomplete.
@@ -39,19 +41,20 @@ Use `--window 2026073110001100` to require one exact source window. Without it,
The target server is offline, so build the runtime image on a connected machine and import it through Metrix Container Management:
```powershell
docker build -t interference-etl-runtime:1.0 .
docker build -t interference-etl-runtime:1.1 .
New-Item -ItemType Directory -Force dist | Out-Null
docker save -o dist\interference-etl-runtime-1.0.tar interference-etl-runtime:1.0
docker save -o dist\interference-etl-runtime-1.1.tar interference-etl-runtime:1.1
```
The local `dist/` directory is intentionally excluded from Git. Transfer the generated TAR as a deployment artifact rather than source code.
The current `1.1` TAR is 46,944,256 bytes with SHA-256 `BF92BF776D52865CA484D0B7D1B10336C998A07E8310D6F01576C74A65A1FF6A`.
Create the Metrix script project with:
```text
Name: InterferenceETL
Language: python
Base image: interference-etl-runtime:1.0
Base image: interference-etl-runtime:1.1
Network: bridge
Run command: python main.py
Timeout: 1800 seconds
@@ -67,11 +70,17 @@ Configure project environment variables in Metrix instead of committing secrets:
"INTERFERENCE_SOURCE_ROOT": "/网优日常优化数据文档/(勿删)干扰定时小时指标",
"INTERFERENCE_OUTPUT_DIR": "/workspace/output",
"INTERFERENCE_LOOKBACK_DAYS": "3",
"INTERFERENCE_PLMN": "460-00"
"INTERFERENCE_PLMN": "460-00",
"INTERFERENCE_DB_HOST": "<reachable MySQL host>",
"INTERFERENCE_DB_PORT": "3306",
"INTERFERENCE_DB_USER": "<MySQL user>",
"INTERFERENCE_DB_PASSWORD": "<MySQL password>",
"INTERFERENCE_DB_NAME": "<database name>",
"INTERFERENCE_DB_TABLE": "interference_hourly_summary"
}
```
`172.17.0.1` is the current Linux Docker default-bridge gateway used to reach the Metrix host port. Verify it before creating the online schedule because Docker bridge configuration can differ by server.
`172.17.0.1` is the current Linux Docker default-bridge gateway used to reach the Metrix host port. Verify both API and MySQL addresses before creating the online schedule because Docker network configuration can differ by server. A container name such as `ShareMySQL` resolves only when the script container joins the same Docker network.
## CLI options
@@ -85,4 +94,11 @@ Configure project environment variables in Metrix instead of committing secrets:
--storage-id ID Metrix storage connection ID
--root PATH Source directory in Metrix storage
--plmn PLMN LTE/SDR CGI PLMN prefix, default 460-00
--no-database Generate CSV files without writing MySQL
--db-host HOST MySQL host or INTERFERENCE_DB_HOST
--db-port PORT MySQL port, default 3306
--db-user USER MySQL user
--db-password VALUE MySQL password; prefer the environment variable
--db-name NAME MySQL database name
--db-table NAME Table name, default interference_hourly_summary
```
+11 -4
View File
@@ -7,8 +7,15 @@
- Each selected ZIP must contain exactly one XLSX. The script strictly validates the known `Sheet0` header, writes one full UTF-8-BOM CSV per source type, and writes one merged CSV containing `hour_start`, `hour_end`, `source_type`, `cgi`, `cell_name`, `interference_dbm`, and `source_path`.
- LTE/SDR CGI currently uses `460-00-{node}-{cell}`; NR uses `masterOperatorId` unchanged. This is an explicit assumption pending user confirmation.
- Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
- Database persistence is intentionally not implemented because the target schema and credentials are pending. `manifest.json` records input paths, sizes, SHA-256 hashes, row counts, warnings, and generated files for later ingestion auditing.
- The Metrix script container needs `openpyxl==3.1.5`, bridge networking, `python main.py`, a reachable `METRIX_API_BASE_URL`, and `METRIX_API_TOKEN` injected through project environment settings. Secrets must not be committed.
- `manifest.json` records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
- The Metrix script container needs `openpyxl==3.1.5`, `PyMySQL==1.1.2`, bridge networking, `python main.py`, reachable API/MySQL addresses, and credentials injected through project environment settings. Secrets must not be committed.
- Read-only validation against the current Metrix storage selected window `2026073110001100`, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were `7 / 804 / 45 / 86 / 700 / 164 / 25` in `EXPECTED_TYPES` order; sampled CGI, cell name, and interference values matched the source workbooks.
- The Linux/amd64 runtime image is `interference-etl-runtime:1.0`. Its offline archive is generated locally at `dist/interference-etl-runtime-1.0.tar` (46,828,032 bytes, SHA-256 `B65B94941BB304CA955224B885582E36BE05E390CBD76FC4C069494352E77746`) and remains excluded from Git.
- Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, and cross-midnight window parsing.
- The current Linux/amd64 runtime image is `interference-etl-runtime:1.1`. Its offline archive is generated locally at `dist/interference-etl-runtime-1.1.tar` (46,944,256 bytes, SHA-256 `BF92BF776D52865CA484D0B7D1B10336C998A07E8310D6F01576C74A65A1FF6A`) and remains excluded from Git.
- Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.
## 2026-07-31: Latest-hour MySQL retention
- Scheduled runs write to MySQL by default. File-only development runs must explicitly pass `--no-database`; connection values come from `INTERFERENCE_DB_*` environment variables.
- The default table is `interference_hourly_summary`. Its only time column is `metric_time DATETIME`, populated from the source KPI `开始时间`, so users can identify the hour represented by every row.
- One transaction deletes the selected hour for idempotent refresh, inserts its complete batch, then deletes every other database hour. Any failure rolls back the data changes, and an older selected hour cannot replace a newer hour already stored.
- Database retention never deletes or modifies Metrix Storage/SFTP source ZIP or XLSX files. Generated CSV retention remains a separate pending decision.
+4 -2
View File
@@ -10,10 +10,12 @@
- NR CGI uses the source `masterOperatorId` unchanged. Confirm whether the final database should retain this source format or normalize it.
- Output CSV files use UTF-8 with BOM so they open correctly in Excel.
- Source files are read-only. The script writes only below its configured output directory.
- MySQL is the scheduled-run output. The table uses one `metric_time DATETIME` column containing the source KPI start time.
- A successful run transactionally refreshes the selected hour and deletes rows for all other hours. A selected hour older than the newest database hour is rejected to prevent data rollback.
## Pending user decisions
- Target database connection, database name, table name, and credentials.
- Target database host, database name, user, and credentials. The default table name is `interference_hourly_summary` and can be overridden.
- Whether the seven converted full CSV files must be retained after successful database import.
- Whether the `指标(计数器)` metadata sheet must also be stored.
- Final CGI normalization rules, especially for NR `masterOperatorId`.
@@ -23,4 +25,4 @@
## Mock boundary
Automated tests generate seven in-memory XLSX/ZIP sources plus a newer incomplete hour. The mock verifies complete-hour fallback, schema validation, CSV conversion, CGI extraction, summary merging, and cross-midnight windows. No mock credentials or fake database writes are present in production code.
Automated tests generate seven in-memory XLSX/ZIP sources plus a newer incomplete hour. The mock verifies complete-hour fallback, schema validation, CSV conversion, CGI extraction, summary merging, cross-midnight windows, transactional latest-hour replacement, and database rollback protection. No mock credentials or fake database writes are present in production code.
+179 -3
View File
@@ -14,7 +14,7 @@ import re
import shutil
import sys
import time
from typing import Protocol
from typing import Callable, Protocol
from urllib.error import HTTPError, URLError
from urllib.parse import quote, urlencode
from urllib.request import Request, urlopen
@@ -138,6 +138,9 @@ SUMMARY_HEADER = (
"source_path",
)
DEFAULT_DB_TABLE = "interference_hourly_summary"
DB_IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9_]+$")
class ProcessingError(RuntimeError):
pass
@@ -157,6 +160,134 @@ class Source(Protocol):
def download(self, path: str) -> bytes: ...
class SummaryStore(Protocol):
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]: ...
class MySQLSummaryStore:
def __init__(
self,
host: str,
port: int,
user: str,
password: str,
database: str,
table: str = DEFAULT_DB_TABLE,
connect_factory: Callable[[], object] | None = None,
) -> None:
if not DB_IDENTIFIER_RE.fullmatch(table):
raise ProcessingError(f"Invalid database table name: {table!r}")
self.host = host
self.port = port
self.user = user
self.password = password
self.database = database
self.table = table
self.connect_factory = connect_factory
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]:
if not rows:
raise ProcessingError("Refusing to replace database data with an empty batch")
metric_times = {datetime.strptime(row["hour_start"], "%Y-%m-%d %H:%M:%S") for row in rows}
if len(metric_times) != 1:
raise ProcessingError("Database batch must contain exactly one metric hour")
metric_time = next(iter(metric_times))
table = f"`{self.table}`"
connection = self._connect()
cursor = None
try:
cursor = connection.cursor()
cursor.execute(
f"""
CREATE TABLE IF NOT EXISTS {table} (
metric_time DATETIME NOT NULL COMMENT '指标开始时间',
source_type VARCHAR(64) NOT NULL,
cgi VARCHAR(128) NOT NULL,
cell_name VARCHAR(255) NOT NULL,
interference_dbm DECIMAL(10,3) NOT NULL,
source_path VARCHAR(1024) NOT NULL,
PRIMARY KEY (metric_time, source_type, cgi)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cursor.execute(f"SELECT MAX(metric_time) FROM {table}")
latest_row = cursor.fetchone()
latest_time = latest_row[0] if latest_row else None
if isinstance(latest_time, str):
latest_time = datetime.fromisoformat(latest_time)
if latest_time is not None and latest_time > metric_time:
raise ProcessingError(
f"Database already contains newer metric time {latest_time:%Y-%m-%d %H:%M:%S}; "
f"refusing to replace it with {metric_time:%Y-%m-%d %H:%M:%S}"
)
cursor.execute(f"DELETE FROM {table} WHERE metric_time = %s", (metric_time,))
refreshed_rows = cursor.rowcount
cursor.executemany(
f"""
INSERT INTO {table}
(metric_time, source_type, cgi, cell_name, interference_dbm, source_path)
VALUES (%s, %s, %s, %s, %s, %s)
""",
[
(
metric_time,
row["source_type"],
row["cgi"],
row["cell_name"],
row["interference_dbm"],
row["source_path"],
)
for row in rows
],
)
cursor.execute(f"DELETE FROM {table} WHERE metric_time <> %s", (metric_time,))
old_rows_deleted = cursor.rowcount
connection.commit()
except ProcessingError:
connection.rollback()
raise
except Exception as exc:
connection.rollback()
raise ProcessingError(f"MySQL write failed: {exc}") from exc
finally:
if cursor is not None:
cursor.close()
connection.close()
return {
"enabled": True,
"table": self.table,
"metric_time": metric_time.strftime("%Y-%m-%d %H:%M:%S"),
"inserted_rows": len(rows),
"refreshed_rows": refreshed_rows,
"old_rows_deleted": old_rows_deleted,
}
def _connect(self) -> object:
if self.connect_factory is not None:
return self.connect_factory()
try:
import pymysql
except ImportError as exc:
raise ProcessingError("PyMySQL is required for database output") from exc
try:
return pymysql.connect(
host=self.host,
port=self.port,
user=self.user,
password=self.password,
database=self.database,
charset="utf8mb4",
autocommit=False,
connect_timeout=15,
read_timeout=60,
write_timeout=60,
)
except Exception as exc:
raise ProcessingError(f"MySQL connection failed: {exc}") from exc
class ApiSource:
def __init__(self, base_url: str, token: str, storage_id: str, root: str) -> None:
if not token:
@@ -302,7 +433,14 @@ def parse_workbook(raw_zip: bytes, source_type: str) -> tuple[tuple[str, ...], l
workbook.close()
def process(source: Source, output_root: Path, lookback_days: int, requested_window: str = "", plmn: str = "460-00") -> Path:
def process(
source: Source,
output_root: Path,
lookback_days: int,
requested_window: str = "",
plmn: str = "460-00",
store: SummaryStore | None = None,
) -> Path:
candidates = source.candidates(lookback_days)
window, selected, warnings_out = select_window(candidates, requested_window)
temp_dir = output_root.resolve() / f".{window}.tmp-{os.getpid()}"
@@ -315,6 +453,7 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
summary_rows: list[dict[str, str]] = []
manifest_files: list[dict[str, object]] = []
database_result: dict[str, object] = {"enabled": False}
try:
for source_type in EXPECTED_TYPES:
candidate = selected[source_type]
@@ -340,6 +479,8 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
summary_rows.sort(key=lambda item: (EXPECTED_TYPES.index(item["source_type"]), item["cgi"], item["cell_name"]))
summary_name = f"interference_summary_{window}.csv"
write_dict_csv(temp_dir / summary_name, SUMMARY_HEADER, summary_rows)
if store is not None:
database_result = store.replace_latest(summary_rows)
manifest = {
"generated_at": datetime.now(timezone.utc).isoformat(),
"window": window,
@@ -351,6 +492,7 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
"warnings": warnings_out,
"files": manifest_files,
"summary_csv": summary_name,
"database": database_result,
}
(temp_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
output_root.resolve().mkdir(parents=True, exist_ok=True)
@@ -365,6 +507,10 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
print(f"selected_window={window}")
print(f"source_files={len(manifest_files)}")
print(f"summary_rows={len(summary_rows)}")
if database_result["enabled"]:
print(f"database_metric_time={database_result['metric_time']}")
print(f"database_inserted_rows={database_result['inserted_rows']}")
print(f"database_old_rows_deleted={database_result['old_rows_deleted']}")
for message in warnings_out:
print(f"warning={message}")
print(f"output={final_dir}")
@@ -471,15 +617,45 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--storage-id", default=os.getenv("METRIX_STORAGE_ID", DEFAULT_STORAGE_ID))
parser.add_argument("--root", default=os.getenv("INTERFERENCE_SOURCE_ROOT", DEFAULT_ROOT))
parser.add_argument("--plmn", default=os.getenv("INTERFERENCE_PLMN", "460-00"))
parser.add_argument("--no-database", action="store_true", help="Generate files without writing MySQL")
parser.add_argument("--db-host", default=os.getenv("INTERFERENCE_DB_HOST", ""))
parser.add_argument("--db-port", type=int, default=int(os.getenv("INTERFERENCE_DB_PORT", "3306")))
parser.add_argument("--db-user", default=os.getenv("INTERFERENCE_DB_USER", ""))
parser.add_argument("--db-password", default=os.getenv("INTERFERENCE_DB_PASSWORD", ""))
parser.add_argument("--db-name", default=os.getenv("INTERFERENCE_DB_NAME", ""))
parser.add_argument("--db-table", default=os.getenv("INTERFERENCE_DB_TABLE", DEFAULT_DB_TABLE))
return parser
def mysql_store_from_args(args: argparse.Namespace) -> MySQLSummaryStore:
required_values = {
"INTERFERENCE_DB_HOST": args.db_host,
"INTERFERENCE_DB_USER": args.db_user,
"INTERFERENCE_DB_PASSWORD": args.db_password,
"INTERFERENCE_DB_NAME": args.db_name,
}
missing = [name for name, value in required_values.items() if not value]
if missing:
raise ProcessingError(f"Missing database settings: {', '.join(missing)}; use --no-database for file-only runs")
if not 1 <= args.db_port <= 65535:
raise ProcessingError("db-port must be between 1 and 65535")
return MySQLSummaryStore(
host=args.db_host,
port=args.db_port,
user=args.db_user,
password=args.db_password,
database=args.db_name,
table=args.db_table,
)
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
if args.lookback_days < 1:
raise ProcessingError("lookback-days must be at least 1")
source: Source = LocalSource(args.source_dir) if args.source_dir else ApiSource(args.api_base, args.api_token, args.storage_id, args.root)
process(source, args.output_dir, args.lookback_days, args.window, args.plmn)
store = None if args.no_database else mysql_store_from_args(args)
process(source, args.output_dir, args.lookback_days, args.window, args.plmn, store)
return 0
+1
View File
@@ -1 +1,2 @@
openpyxl==3.1.5
PyMySQL==1.1.2
+121 -1
View File
@@ -26,8 +26,9 @@ class PipelineTest(unittest.TestCase):
for source_type in main.EXPECTED_TYPES:
create_archive(root, source_type, COMPLETE_WINDOW)
create_archive(root, main.EXPECTED_TYPES[0], INCOMPLETE_WINDOW)
store = RecordingStore()
result = main.process(main.LocalSource(root), output, lookback_days=3)
result = main.process(main.LocalSource(root), output, lookback_days=3, store=store)
self.assertEqual(result.name, COMPLETE_WINDOW)
converted = sorted((result / "converted").glob("*.csv"))
@@ -45,6 +46,8 @@ class PipelineTest(unittest.TestCase):
self.assertEqual(manifest["summary_rows"], 7)
self.assertEqual(len(manifest["warnings"]), 1)
self.assertIn(INCOMPLETE_WINDOW, manifest["warnings"][0])
self.assertEqual(manifest["database"]["metric_time"], "2026-07-31 10:00:00")
self.assertEqual(len(store.rows), 7)
def test_requested_incomplete_window_is_rejected(self) -> None:
with tempfile.TemporaryDirectory() as temp:
@@ -69,6 +72,123 @@ class PipelineTest(unittest.TestCase):
("2026-07-30 23:00:00", "2026-07-31 00:00:00"),
)
def test_mysql_store_replaces_same_hour_and_deletes_other_hours(self) -> None:
connection = FakeConnection(datetime(2026, 7, 31, 10), same_hour_rows=7, old_rows=20)
store = main.MySQLSummaryStore(
host="",
port=3306,
user="",
password="",
database="",
connect_factory=lambda: connection,
)
result = store.replace_latest([database_row()])
self.assertEqual(result["metric_time"], "2026-07-31 10:00:00")
self.assertEqual(result["inserted_rows"], 1)
self.assertEqual(result["refreshed_rows"], 7)
self.assertEqual(result["old_rows_deleted"], 20)
self.assertEqual(connection.cursor_instance.inserted[0][0], datetime(2026, 7, 31, 10))
self.assertTrue(connection.committed)
self.assertFalse(connection.rolled_back)
self.assertTrue(connection.closed)
def test_mysql_store_refuses_to_replace_a_newer_hour(self) -> None:
connection = FakeConnection(datetime(2026, 7, 31, 11))
store = main.MySQLSummaryStore(
host="",
port=3306,
user="",
password="",
database="",
connect_factory=lambda: connection,
)
with self.assertRaisesRegex(main.ProcessingError, "already contains newer metric time"):
store.replace_latest([database_row()])
self.assertFalse(connection.committed)
self.assertTrue(connection.rolled_back)
self.assertEqual(connection.cursor_instance.inserted, [])
self.assertTrue(connection.closed)
class RecordingStore:
def __init__(self) -> None:
self.rows: list[dict[str, str]] = []
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]:
self.rows = list(rows)
return {
"enabled": True,
"table": main.DEFAULT_DB_TABLE,
"metric_time": rows[0]["hour_start"],
"inserted_rows": len(rows),
"refreshed_rows": 0,
"old_rows_deleted": 0,
}
class FakeCursor:
def __init__(self, latest_time: datetime | None, same_hour_rows: int, old_rows: int) -> None:
self.latest_time = latest_time
self.same_hour_rows = same_hour_rows
self.old_rows = old_rows
self.rowcount = 0
self.inserted: list[tuple[object, ...]] = []
self.closed = False
def execute(self, query: str, params: tuple[object, ...] | None = None) -> None:
normalized = " ".join(query.split())
if normalized.startswith("DELETE") and "metric_time = %s" in normalized:
self.rowcount = self.same_hour_rows
elif normalized.startswith("DELETE") and "metric_time <> %s" in normalized:
self.rowcount = self.old_rows
def executemany(self, query: str, values: list[tuple[object, ...]]) -> None:
del query
self.inserted = list(values)
self.rowcount = len(values)
def fetchone(self) -> tuple[datetime | None]:
return (self.latest_time,)
def close(self) -> None:
self.closed = True
class FakeConnection:
def __init__(self, latest_time: datetime | None, same_hour_rows: int = 0, old_rows: int = 0) -> None:
self.cursor_instance = FakeCursor(latest_time, same_hour_rows, old_rows)
self.committed = False
self.rolled_back = False
self.closed = False
def cursor(self) -> FakeCursor:
return self.cursor_instance
def commit(self) -> None:
self.committed = True
def rollback(self) -> None:
self.rolled_back = True
def close(self) -> None:
self.closed = True
def database_row() -> dict[str, str]:
return {
"hour_start": "2026-07-31 10:00:00",
"hour_end": "2026-07-31 11:00:00",
"source_type": "5G干扰监控",
"cgi": "46000-100-1",
"cell_name": "测试小区",
"interference_dbm": "-100.5",
"source_path": "/source.zip",
}
def create_archive(root: Path, source_type: str, window: str, bad_header: bool = False) -> None:
date_dir = root / f"{window[:4]}-{window[4:6]}-{window[6:8]}"