feat: 实现干扰指标最新小时入库
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
*
|
||||
!Dockerfile
|
||||
!requirements.txt
|
||||
+2
-1
@@ -1,6 +1,7 @@
|
||||
FROM python:3.13.11-slim
|
||||
|
||||
RUN pip install --no-cache-dir openpyxl==3.1.5
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
RUN pip install --no-cache-dir -r /tmp/requirements.txt
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
|
||||
@@ -22,14 +22,16 @@ The summary columns are:
|
||||
hour_start,hour_end,source_type,cgi,cell_name,interference_dbm,source_path
|
||||
```
|
||||
|
||||
The script never modifies source storage and does not write a database yet. Pending decisions are recorded in `docs/assumptions.md`.
|
||||
The script never modifies or deletes source storage files. By default it writes the complete selected hour to MySQL and keeps only that hour in the target table.
|
||||
|
||||
The database table contains one time column, `metric_time DATETIME`, which is the source KPI start time. A same-hour rerun replaces that whole batch. After a successful insert, rows for every other hour are deleted in the same transaction. The script refuses to replace a newer database hour with an older source hour.
|
||||
|
||||
## Local development
|
||||
|
||||
```powershell
|
||||
python -m pip install -r requirements.txt
|
||||
python -m unittest discover -s tests -v
|
||||
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output
|
||||
python main.py --source-dir C:\path\to\mock-or-exported-tree --output-dir output --no-database
|
||||
```
|
||||
|
||||
Use `--window 2026073110001100` to require one exact source window. Without it, the script selects the newest window containing all seven source types and falls back if the newest observed window is incomplete.
|
||||
@@ -39,19 +41,20 @@ Use `--window 2026073110001100` to require one exact source window. Without it,
|
||||
The target server is offline, so build the runtime image on a connected machine and import it through Metrix Container Management:
|
||||
|
||||
```powershell
|
||||
docker build -t interference-etl-runtime:1.0 .
|
||||
docker build -t interference-etl-runtime:1.1 .
|
||||
New-Item -ItemType Directory -Force dist | Out-Null
|
||||
docker save -o dist\interference-etl-runtime-1.0.tar interference-etl-runtime:1.0
|
||||
docker save -o dist\interference-etl-runtime-1.1.tar interference-etl-runtime:1.1
|
||||
```
|
||||
|
||||
The local `dist/` directory is intentionally excluded from Git. Transfer the generated TAR as a deployment artifact rather than source code.
|
||||
The current `1.1` TAR is 46,944,256 bytes with SHA-256 `BF92BF776D52865CA484D0B7D1B10336C998A07E8310D6F01576C74A65A1FF6A`.
|
||||
|
||||
Create the Metrix script project with:
|
||||
|
||||
```text
|
||||
Name: InterferenceETL
|
||||
Language: python
|
||||
Base image: interference-etl-runtime:1.0
|
||||
Base image: interference-etl-runtime:1.1
|
||||
Network: bridge
|
||||
Run command: python main.py
|
||||
Timeout: 1800 seconds
|
||||
@@ -67,11 +70,17 @@ Configure project environment variables in Metrix instead of committing secrets:
|
||||
"INTERFERENCE_SOURCE_ROOT": "/网优日常优化数据文档/(勿删)干扰定时小时指标",
|
||||
"INTERFERENCE_OUTPUT_DIR": "/workspace/output",
|
||||
"INTERFERENCE_LOOKBACK_DAYS": "3",
|
||||
"INTERFERENCE_PLMN": "460-00"
|
||||
"INTERFERENCE_PLMN": "460-00",
|
||||
"INTERFERENCE_DB_HOST": "<reachable MySQL host>",
|
||||
"INTERFERENCE_DB_PORT": "3306",
|
||||
"INTERFERENCE_DB_USER": "<MySQL user>",
|
||||
"INTERFERENCE_DB_PASSWORD": "<MySQL password>",
|
||||
"INTERFERENCE_DB_NAME": "<database name>",
|
||||
"INTERFERENCE_DB_TABLE": "interference_hourly_summary"
|
||||
}
|
||||
```
|
||||
|
||||
`172.17.0.1` is the current Linux Docker default-bridge gateway used to reach the Metrix host port. Verify it before creating the online schedule because Docker bridge configuration can differ by server.
|
||||
`172.17.0.1` is the current Linux Docker default-bridge gateway used to reach the Metrix host port. Verify both API and MySQL addresses before creating the online schedule because Docker network configuration can differ by server. A container name such as `ShareMySQL` resolves only when the script container joins the same Docker network.
|
||||
|
||||
## CLI options
|
||||
|
||||
@@ -85,4 +94,11 @@ Configure project environment variables in Metrix instead of committing secrets:
|
||||
--storage-id ID Metrix storage connection ID
|
||||
--root PATH Source directory in Metrix storage
|
||||
--plmn PLMN LTE/SDR CGI PLMN prefix, default 460-00
|
||||
--no-database Generate CSV files without writing MySQL
|
||||
--db-host HOST MySQL host or INTERFERENCE_DB_HOST
|
||||
--db-port PORT MySQL port, default 3306
|
||||
--db-user USER MySQL user
|
||||
--db-password VALUE MySQL password; prefer the environment variable
|
||||
--db-name NAME MySQL database name
|
||||
--db-table NAME Table name, default interference_hourly_summary
|
||||
```
|
||||
|
||||
@@ -7,8 +7,15 @@
|
||||
- Each selected ZIP must contain exactly one XLSX. The script strictly validates the known `Sheet0` header, writes one full UTF-8-BOM CSV per source type, and writes one merged CSV containing `hour_start`, `hour_end`, `source_type`, `cgi`, `cell_name`, `interference_dbm`, and `source_path`.
|
||||
- LTE/SDR CGI currently uses `460-00-{node}-{cell}`; NR uses `masterOperatorId` unchanged. This is an explicit assumption pending user confirmation.
|
||||
- Runs are idempotent at the output-window directory level. Generation happens in a scoped temporary directory, then replaces only the same window below the configured output root. Source storage is never modified.
|
||||
- Database persistence is intentionally not implemented because the target schema and credentials are pending. `manifest.json` records input paths, sizes, SHA-256 hashes, row counts, warnings, and generated files for later ingestion auditing.
|
||||
- The Metrix script container needs `openpyxl==3.1.5`, bridge networking, `python main.py`, a reachable `METRIX_API_BASE_URL`, and `METRIX_API_TOKEN` injected through project environment settings. Secrets must not be committed.
|
||||
- `manifest.json` records input paths, sizes, SHA-256 hashes, row counts, warnings, generated files, and the database write result for ingestion auditing.
|
||||
- The Metrix script container needs `openpyxl==3.1.5`, `PyMySQL==1.1.2`, bridge networking, `python main.py`, reachable API/MySQL addresses, and credentials injected through project environment settings. Secrets must not be committed.
|
||||
- Read-only validation against the current Metrix storage selected window `2026073110001100`, processed all seven source ZIP files, and produced 1,831 summary rows. Per-source row counts were `7 / 804 / 45 / 86 / 700 / 164 / 25` in `EXPECTED_TYPES` order; sampled CGI, cell name, and interference values matched the source workbooks.
|
||||
- The Linux/amd64 runtime image is `interference-etl-runtime:1.0`. Its offline archive is generated locally at `dist/interference-etl-runtime-1.0.tar` (46,828,032 bytes, SHA-256 `B65B94941BB304CA955224B885582E36BE05E390CBD76FC4C069494352E77746`) and remains excluded from Git.
|
||||
- Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, and cross-midnight window parsing.
|
||||
- The current Linux/amd64 runtime image is `interference-etl-runtime:1.1`. Its offline archive is generated locally at `dist/interference-etl-runtime-1.1.tar` (46,944,256 bytes, SHA-256 `BF92BF776D52865CA484D0B7D1B10336C998A07E8310D6F01576C74A65A1FF6A`) and remains excluded from Git.
|
||||
- Six Mock tests pass on Windows Python and inside the runtime image. They cover complete-hour fallback, strict schema rejection, summary extraction, cross-midnight window parsing, transactional database replacement, and rollback protection.
|
||||
|
||||
## 2026-07-31: Latest-hour MySQL retention
|
||||
|
||||
- Scheduled runs write to MySQL by default. File-only development runs must explicitly pass `--no-database`; connection values come from `INTERFERENCE_DB_*` environment variables.
|
||||
- The default table is `interference_hourly_summary`. Its only time column is `metric_time DATETIME`, populated from the source KPI `开始时间`, so users can identify the hour represented by every row.
|
||||
- One transaction deletes the selected hour for idempotent refresh, inserts its complete batch, then deletes every other database hour. Any failure rolls back the data changes, and an older selected hour cannot replace a newer hour already stored.
|
||||
- Database retention never deletes or modifies Metrix Storage/SFTP source ZIP or XLSX files. Generated CSV retention remains a separate pending decision.
|
||||
|
||||
+4
-2
@@ -10,10 +10,12 @@
|
||||
- NR CGI uses the source `masterOperatorId` unchanged. Confirm whether the final database should retain this source format or normalize it.
|
||||
- Output CSV files use UTF-8 with BOM so they open correctly in Excel.
|
||||
- Source files are read-only. The script writes only below its configured output directory.
|
||||
- MySQL is the scheduled-run output. The table uses one `metric_time DATETIME` column containing the source KPI start time.
|
||||
- A successful run transactionally refreshes the selected hour and deletes rows for all other hours. A selected hour older than the newest database hour is rejected to prevent data rollback.
|
||||
|
||||
## Pending user decisions
|
||||
|
||||
- Target database connection, database name, table name, and credentials.
|
||||
- Target database host, database name, user, and credentials. The default table name is `interference_hourly_summary` and can be overridden.
|
||||
- Whether the seven converted full CSV files must be retained after successful database import.
|
||||
- Whether the `指标(计数器)` metadata sheet must also be stored.
|
||||
- Final CGI normalization rules, especially for NR `masterOperatorId`.
|
||||
@@ -23,4 +25,4 @@
|
||||
|
||||
## Mock boundary
|
||||
|
||||
Automated tests generate seven in-memory XLSX/ZIP sources plus a newer incomplete hour. The mock verifies complete-hour fallback, schema validation, CSV conversion, CGI extraction, summary merging, and cross-midnight windows. No mock credentials or fake database writes are present in production code.
|
||||
Automated tests generate seven in-memory XLSX/ZIP sources plus a newer incomplete hour. The mock verifies complete-hour fallback, schema validation, CSV conversion, CGI extraction, summary merging, cross-midnight windows, transactional latest-hour replacement, and database rollback protection. No mock credentials or fake database writes are present in production code.
|
||||
|
||||
@@ -14,7 +14,7 @@ import re
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from typing import Protocol
|
||||
from typing import Callable, Protocol
|
||||
from urllib.error import HTTPError, URLError
|
||||
from urllib.parse import quote, urlencode
|
||||
from urllib.request import Request, urlopen
|
||||
@@ -138,6 +138,9 @@ SUMMARY_HEADER = (
|
||||
"source_path",
|
||||
)
|
||||
|
||||
DEFAULT_DB_TABLE = "interference_hourly_summary"
|
||||
DB_IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9_]+$")
|
||||
|
||||
|
||||
class ProcessingError(RuntimeError):
|
||||
pass
|
||||
@@ -157,6 +160,134 @@ class Source(Protocol):
|
||||
def download(self, path: str) -> bytes: ...
|
||||
|
||||
|
||||
class SummaryStore(Protocol):
|
||||
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]: ...
|
||||
|
||||
|
||||
class MySQLSummaryStore:
|
||||
def __init__(
|
||||
self,
|
||||
host: str,
|
||||
port: int,
|
||||
user: str,
|
||||
password: str,
|
||||
database: str,
|
||||
table: str = DEFAULT_DB_TABLE,
|
||||
connect_factory: Callable[[], object] | None = None,
|
||||
) -> None:
|
||||
if not DB_IDENTIFIER_RE.fullmatch(table):
|
||||
raise ProcessingError(f"Invalid database table name: {table!r}")
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.user = user
|
||||
self.password = password
|
||||
self.database = database
|
||||
self.table = table
|
||||
self.connect_factory = connect_factory
|
||||
|
||||
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]:
|
||||
if not rows:
|
||||
raise ProcessingError("Refusing to replace database data with an empty batch")
|
||||
metric_times = {datetime.strptime(row["hour_start"], "%Y-%m-%d %H:%M:%S") for row in rows}
|
||||
if len(metric_times) != 1:
|
||||
raise ProcessingError("Database batch must contain exactly one metric hour")
|
||||
metric_time = next(iter(metric_times))
|
||||
table = f"`{self.table}`"
|
||||
connection = self._connect()
|
||||
cursor = None
|
||||
try:
|
||||
cursor = connection.cursor()
|
||||
cursor.execute(
|
||||
f"""
|
||||
CREATE TABLE IF NOT EXISTS {table} (
|
||||
metric_time DATETIME NOT NULL COMMENT '指标开始时间',
|
||||
source_type VARCHAR(64) NOT NULL,
|
||||
cgi VARCHAR(128) NOT NULL,
|
||||
cell_name VARCHAR(255) NOT NULL,
|
||||
interference_dbm DECIMAL(10,3) NOT NULL,
|
||||
source_path VARCHAR(1024) NOT NULL,
|
||||
PRIMARY KEY (metric_time, source_type, cgi)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
|
||||
"""
|
||||
)
|
||||
cursor.execute(f"SELECT MAX(metric_time) FROM {table}")
|
||||
latest_row = cursor.fetchone()
|
||||
latest_time = latest_row[0] if latest_row else None
|
||||
if isinstance(latest_time, str):
|
||||
latest_time = datetime.fromisoformat(latest_time)
|
||||
if latest_time is not None and latest_time > metric_time:
|
||||
raise ProcessingError(
|
||||
f"Database already contains newer metric time {latest_time:%Y-%m-%d %H:%M:%S}; "
|
||||
f"refusing to replace it with {metric_time:%Y-%m-%d %H:%M:%S}"
|
||||
)
|
||||
|
||||
cursor.execute(f"DELETE FROM {table} WHERE metric_time = %s", (metric_time,))
|
||||
refreshed_rows = cursor.rowcount
|
||||
cursor.executemany(
|
||||
f"""
|
||||
INSERT INTO {table}
|
||||
(metric_time, source_type, cgi, cell_name, interference_dbm, source_path)
|
||||
VALUES (%s, %s, %s, %s, %s, %s)
|
||||
""",
|
||||
[
|
||||
(
|
||||
metric_time,
|
||||
row["source_type"],
|
||||
row["cgi"],
|
||||
row["cell_name"],
|
||||
row["interference_dbm"],
|
||||
row["source_path"],
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
)
|
||||
cursor.execute(f"DELETE FROM {table} WHERE metric_time <> %s", (metric_time,))
|
||||
old_rows_deleted = cursor.rowcount
|
||||
connection.commit()
|
||||
except ProcessingError:
|
||||
connection.rollback()
|
||||
raise
|
||||
except Exception as exc:
|
||||
connection.rollback()
|
||||
raise ProcessingError(f"MySQL write failed: {exc}") from exc
|
||||
finally:
|
||||
if cursor is not None:
|
||||
cursor.close()
|
||||
connection.close()
|
||||
|
||||
return {
|
||||
"enabled": True,
|
||||
"table": self.table,
|
||||
"metric_time": metric_time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"inserted_rows": len(rows),
|
||||
"refreshed_rows": refreshed_rows,
|
||||
"old_rows_deleted": old_rows_deleted,
|
||||
}
|
||||
|
||||
def _connect(self) -> object:
|
||||
if self.connect_factory is not None:
|
||||
return self.connect_factory()
|
||||
try:
|
||||
import pymysql
|
||||
except ImportError as exc:
|
||||
raise ProcessingError("PyMySQL is required for database output") from exc
|
||||
try:
|
||||
return pymysql.connect(
|
||||
host=self.host,
|
||||
port=self.port,
|
||||
user=self.user,
|
||||
password=self.password,
|
||||
database=self.database,
|
||||
charset="utf8mb4",
|
||||
autocommit=False,
|
||||
connect_timeout=15,
|
||||
read_timeout=60,
|
||||
write_timeout=60,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise ProcessingError(f"MySQL connection failed: {exc}") from exc
|
||||
|
||||
|
||||
class ApiSource:
|
||||
def __init__(self, base_url: str, token: str, storage_id: str, root: str) -> None:
|
||||
if not token:
|
||||
@@ -302,7 +433,14 @@ def parse_workbook(raw_zip: bytes, source_type: str) -> tuple[tuple[str, ...], l
|
||||
workbook.close()
|
||||
|
||||
|
||||
def process(source: Source, output_root: Path, lookback_days: int, requested_window: str = "", plmn: str = "460-00") -> Path:
|
||||
def process(
|
||||
source: Source,
|
||||
output_root: Path,
|
||||
lookback_days: int,
|
||||
requested_window: str = "",
|
||||
plmn: str = "460-00",
|
||||
store: SummaryStore | None = None,
|
||||
) -> Path:
|
||||
candidates = source.candidates(lookback_days)
|
||||
window, selected, warnings_out = select_window(candidates, requested_window)
|
||||
temp_dir = output_root.resolve() / f".{window}.tmp-{os.getpid()}"
|
||||
@@ -315,6 +453,7 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
|
||||
|
||||
summary_rows: list[dict[str, str]] = []
|
||||
manifest_files: list[dict[str, object]] = []
|
||||
database_result: dict[str, object] = {"enabled": False}
|
||||
try:
|
||||
for source_type in EXPECTED_TYPES:
|
||||
candidate = selected[source_type]
|
||||
@@ -340,6 +479,8 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
|
||||
summary_rows.sort(key=lambda item: (EXPECTED_TYPES.index(item["source_type"]), item["cgi"], item["cell_name"]))
|
||||
summary_name = f"interference_summary_{window}.csv"
|
||||
write_dict_csv(temp_dir / summary_name, SUMMARY_HEADER, summary_rows)
|
||||
if store is not None:
|
||||
database_result = store.replace_latest(summary_rows)
|
||||
manifest = {
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"window": window,
|
||||
@@ -351,6 +492,7 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
|
||||
"warnings": warnings_out,
|
||||
"files": manifest_files,
|
||||
"summary_csv": summary_name,
|
||||
"database": database_result,
|
||||
}
|
||||
(temp_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
output_root.resolve().mkdir(parents=True, exist_ok=True)
|
||||
@@ -365,6 +507,10 @@ def process(source: Source, output_root: Path, lookback_days: int, requested_win
|
||||
print(f"selected_window={window}")
|
||||
print(f"source_files={len(manifest_files)}")
|
||||
print(f"summary_rows={len(summary_rows)}")
|
||||
if database_result["enabled"]:
|
||||
print(f"database_metric_time={database_result['metric_time']}")
|
||||
print(f"database_inserted_rows={database_result['inserted_rows']}")
|
||||
print(f"database_old_rows_deleted={database_result['old_rows_deleted']}")
|
||||
for message in warnings_out:
|
||||
print(f"warning={message}")
|
||||
print(f"output={final_dir}")
|
||||
@@ -471,15 +617,45 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
parser.add_argument("--storage-id", default=os.getenv("METRIX_STORAGE_ID", DEFAULT_STORAGE_ID))
|
||||
parser.add_argument("--root", default=os.getenv("INTERFERENCE_SOURCE_ROOT", DEFAULT_ROOT))
|
||||
parser.add_argument("--plmn", default=os.getenv("INTERFERENCE_PLMN", "460-00"))
|
||||
parser.add_argument("--no-database", action="store_true", help="Generate files without writing MySQL")
|
||||
parser.add_argument("--db-host", default=os.getenv("INTERFERENCE_DB_HOST", ""))
|
||||
parser.add_argument("--db-port", type=int, default=int(os.getenv("INTERFERENCE_DB_PORT", "3306")))
|
||||
parser.add_argument("--db-user", default=os.getenv("INTERFERENCE_DB_USER", ""))
|
||||
parser.add_argument("--db-password", default=os.getenv("INTERFERENCE_DB_PASSWORD", ""))
|
||||
parser.add_argument("--db-name", default=os.getenv("INTERFERENCE_DB_NAME", ""))
|
||||
parser.add_argument("--db-table", default=os.getenv("INTERFERENCE_DB_TABLE", DEFAULT_DB_TABLE))
|
||||
return parser
|
||||
|
||||
|
||||
def mysql_store_from_args(args: argparse.Namespace) -> MySQLSummaryStore:
|
||||
required_values = {
|
||||
"INTERFERENCE_DB_HOST": args.db_host,
|
||||
"INTERFERENCE_DB_USER": args.db_user,
|
||||
"INTERFERENCE_DB_PASSWORD": args.db_password,
|
||||
"INTERFERENCE_DB_NAME": args.db_name,
|
||||
}
|
||||
missing = [name for name, value in required_values.items() if not value]
|
||||
if missing:
|
||||
raise ProcessingError(f"Missing database settings: {', '.join(missing)}; use --no-database for file-only runs")
|
||||
if not 1 <= args.db_port <= 65535:
|
||||
raise ProcessingError("db-port must be between 1 and 65535")
|
||||
return MySQLSummaryStore(
|
||||
host=args.db_host,
|
||||
port=args.db_port,
|
||||
user=args.db_user,
|
||||
password=args.db_password,
|
||||
database=args.db_name,
|
||||
table=args.db_table,
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
if args.lookback_days < 1:
|
||||
raise ProcessingError("lookback-days must be at least 1")
|
||||
source: Source = LocalSource(args.source_dir) if args.source_dir else ApiSource(args.api_base, args.api_token, args.storage_id, args.root)
|
||||
process(source, args.output_dir, args.lookback_days, args.window, args.plmn)
|
||||
store = None if args.no_database else mysql_store_from_args(args)
|
||||
process(source, args.output_dir, args.lookback_days, args.window, args.plmn, store)
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -1 +1,2 @@
|
||||
openpyxl==3.1.5
|
||||
PyMySQL==1.1.2
|
||||
|
||||
+121
-1
@@ -26,8 +26,9 @@ class PipelineTest(unittest.TestCase):
|
||||
for source_type in main.EXPECTED_TYPES:
|
||||
create_archive(root, source_type, COMPLETE_WINDOW)
|
||||
create_archive(root, main.EXPECTED_TYPES[0], INCOMPLETE_WINDOW)
|
||||
store = RecordingStore()
|
||||
|
||||
result = main.process(main.LocalSource(root), output, lookback_days=3)
|
||||
result = main.process(main.LocalSource(root), output, lookback_days=3, store=store)
|
||||
|
||||
self.assertEqual(result.name, COMPLETE_WINDOW)
|
||||
converted = sorted((result / "converted").glob("*.csv"))
|
||||
@@ -45,6 +46,8 @@ class PipelineTest(unittest.TestCase):
|
||||
self.assertEqual(manifest["summary_rows"], 7)
|
||||
self.assertEqual(len(manifest["warnings"]), 1)
|
||||
self.assertIn(INCOMPLETE_WINDOW, manifest["warnings"][0])
|
||||
self.assertEqual(manifest["database"]["metric_time"], "2026-07-31 10:00:00")
|
||||
self.assertEqual(len(store.rows), 7)
|
||||
|
||||
def test_requested_incomplete_window_is_rejected(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
@@ -69,6 +72,123 @@ class PipelineTest(unittest.TestCase):
|
||||
("2026-07-30 23:00:00", "2026-07-31 00:00:00"),
|
||||
)
|
||||
|
||||
def test_mysql_store_replaces_same_hour_and_deletes_other_hours(self) -> None:
|
||||
connection = FakeConnection(datetime(2026, 7, 31, 10), same_hour_rows=7, old_rows=20)
|
||||
store = main.MySQLSummaryStore(
|
||||
host="",
|
||||
port=3306,
|
||||
user="",
|
||||
password="",
|
||||
database="",
|
||||
connect_factory=lambda: connection,
|
||||
)
|
||||
|
||||
result = store.replace_latest([database_row()])
|
||||
|
||||
self.assertEqual(result["metric_time"], "2026-07-31 10:00:00")
|
||||
self.assertEqual(result["inserted_rows"], 1)
|
||||
self.assertEqual(result["refreshed_rows"], 7)
|
||||
self.assertEqual(result["old_rows_deleted"], 20)
|
||||
self.assertEqual(connection.cursor_instance.inserted[0][0], datetime(2026, 7, 31, 10))
|
||||
self.assertTrue(connection.committed)
|
||||
self.assertFalse(connection.rolled_back)
|
||||
self.assertTrue(connection.closed)
|
||||
|
||||
def test_mysql_store_refuses_to_replace_a_newer_hour(self) -> None:
|
||||
connection = FakeConnection(datetime(2026, 7, 31, 11))
|
||||
store = main.MySQLSummaryStore(
|
||||
host="",
|
||||
port=3306,
|
||||
user="",
|
||||
password="",
|
||||
database="",
|
||||
connect_factory=lambda: connection,
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(main.ProcessingError, "already contains newer metric time"):
|
||||
store.replace_latest([database_row()])
|
||||
|
||||
self.assertFalse(connection.committed)
|
||||
self.assertTrue(connection.rolled_back)
|
||||
self.assertEqual(connection.cursor_instance.inserted, [])
|
||||
self.assertTrue(connection.closed)
|
||||
|
||||
|
||||
class RecordingStore:
|
||||
def __init__(self) -> None:
|
||||
self.rows: list[dict[str, str]] = []
|
||||
|
||||
def replace_latest(self, rows: list[dict[str, str]]) -> dict[str, object]:
|
||||
self.rows = list(rows)
|
||||
return {
|
||||
"enabled": True,
|
||||
"table": main.DEFAULT_DB_TABLE,
|
||||
"metric_time": rows[0]["hour_start"],
|
||||
"inserted_rows": len(rows),
|
||||
"refreshed_rows": 0,
|
||||
"old_rows_deleted": 0,
|
||||
}
|
||||
|
||||
|
||||
class FakeCursor:
|
||||
def __init__(self, latest_time: datetime | None, same_hour_rows: int, old_rows: int) -> None:
|
||||
self.latest_time = latest_time
|
||||
self.same_hour_rows = same_hour_rows
|
||||
self.old_rows = old_rows
|
||||
self.rowcount = 0
|
||||
self.inserted: list[tuple[object, ...]] = []
|
||||
self.closed = False
|
||||
|
||||
def execute(self, query: str, params: tuple[object, ...] | None = None) -> None:
|
||||
normalized = " ".join(query.split())
|
||||
if normalized.startswith("DELETE") and "metric_time = %s" in normalized:
|
||||
self.rowcount = self.same_hour_rows
|
||||
elif normalized.startswith("DELETE") and "metric_time <> %s" in normalized:
|
||||
self.rowcount = self.old_rows
|
||||
|
||||
def executemany(self, query: str, values: list[tuple[object, ...]]) -> None:
|
||||
del query
|
||||
self.inserted = list(values)
|
||||
self.rowcount = len(values)
|
||||
|
||||
def fetchone(self) -> tuple[datetime | None]:
|
||||
return (self.latest_time,)
|
||||
|
||||
def close(self) -> None:
|
||||
self.closed = True
|
||||
|
||||
|
||||
class FakeConnection:
|
||||
def __init__(self, latest_time: datetime | None, same_hour_rows: int = 0, old_rows: int = 0) -> None:
|
||||
self.cursor_instance = FakeCursor(latest_time, same_hour_rows, old_rows)
|
||||
self.committed = False
|
||||
self.rolled_back = False
|
||||
self.closed = False
|
||||
|
||||
def cursor(self) -> FakeCursor:
|
||||
return self.cursor_instance
|
||||
|
||||
def commit(self) -> None:
|
||||
self.committed = True
|
||||
|
||||
def rollback(self) -> None:
|
||||
self.rolled_back = True
|
||||
|
||||
def close(self) -> None:
|
||||
self.closed = True
|
||||
|
||||
|
||||
def database_row() -> dict[str, str]:
|
||||
return {
|
||||
"hour_start": "2026-07-31 10:00:00",
|
||||
"hour_end": "2026-07-31 11:00:00",
|
||||
"source_type": "5G干扰监控",
|
||||
"cgi": "46000-100-1",
|
||||
"cell_name": "测试小区",
|
||||
"interference_dbm": "-100.5",
|
||||
"source_path": "/source.zip",
|
||||
}
|
||||
|
||||
|
||||
def create_archive(root: Path, source_type: str, window: str, bad_header: bool = False) -> None:
|
||||
date_dir = root / f"{window[:4]}-{window[4:6]}-{window[6:8]}"
|
||||
|
||||
Reference in New Issue
Block a user