Safety KB Import — 安全生产法规标准导入工具#

Overview#

This skill provides standardized, safe import workflows for adding regulations, standards, and policy documents into the safety-review knowledge base (SQLite). It handles multi-source text extraction, smart clause splitting, conflict detection, and three-table atomic writes (regulations + clauses + std_registry).

Database location: ~/.openclaw-autoclaw/skills/safety-review/db/knowledge.db

When to Use This Skill#

• User wants to add new standards/regulations to the knowledge base
• User has PDF files that need text extraction before import
• User needs to batch-import multiple standards at once
• User asks about importing courseware-referenced standards that are missing
• Any write operation on the safety-review database

Trigger phrases (Chinese): 导入法规、入库、添加标准、补充知识库、PDF提取、拆分条款、批量导入

Companion skill: Use safety-kb-query first to check what's already in the database before importing.

Prerequisites#

1. Detect Python command: bash

   python --version
   
   

2. Required packages for PDF extraction: bash

   pip install pdfplumber
   
   

   For OCR of scanned PDFs: bash

   pip install pdf2image pytesseract
   # Also requires Tesseract OCR engine installed on system
   
   

Import Workflow (Complete)#

Phase 1: Preparation — Check What's Needed ⭐ Always Do This First#

Before importing anything, use safety-kb-query to identify gaps:

bash

python <kb_query_path>/kb_query.py check "GB 16423" "AQ/T 2033" "AQ 2034"

This prevents duplicates and identifies data quality issues.

Phase 2: Text Extraction#

Option A: From PDF Files

bash

python scripts/kb_import.py extract-pdf /path/to/document.pdf

Response fields:

• success: boolean
• text: extracted full text (empty if scan-only)
• char_count: number of characters extracted
• page_count: total pages
• is_scan_only: true if PDF is image-based (needs OCR)

If is_scan_only is true, the PDF is a scanned/image-based document:

1. Try installing and using tesseract OCR
2. If OCR unavailable, extract content from PPT lecture materials or web sources as fallback
3. Document the source as "PPT整理" or "网络来源" rather than official text

Option B: From Web Sources

Use web_fetch to get full text from government websites, wikisource, etc. Common reliable sources:

• 维基文库 (wikisource.org) — full text of laws/policies
• 政府公报 (gov.cn/gongbao) — official gazette versions
• 部委官网 — original standard publications

Option C: From Existing Documents (.docx, .pptx)

Extract text from these formats using appropriate libraries ( python-docx, python-pptx) or the respective skills.

Phase 3: Create Import Manifest#

Create a JSON manifest file listing all items to import:

json

{
  "items": [
    {
      "title": "金属非金属矿山安全规程",
      "document_number": "GB 16423—2020",
      "issuing_authority": "国家市场监督管理总局",
      "authority_level": "national",
      "effective_date": "2021-09-01",
      "status": "current",
      "domains": "矿山安全",
      "category": "国标",
      "full_text": "<complete extracted text here>",
      "source_url": "",
      "page_count": 70,
      "clause_split_pattern": "standard"
    },
    {
      "title": "国务院关于进一步加强企业安全生产工作的通知",
      "document_number": "国发〔2010〕23号",
      "issuing_authority": "国务院",
      "authority_level": "national",
      "effective_date": "2010-07-23",
      "status": "current",
      "domains": "安全生产",
      "category": "政策文件",
      "full_text": "<complete text>",
      "source_url": "https://zh.wikisource.org/...",
      "page_count": 5,
      "clause_split_pattern": "policy"
    }
  ]
}

Manifest Field Reference#

FieldRequiredDescriptiontitle✅Full title of the regulation/standarddocument_number✅Standard number (GB XXXX, AQ/T XXXX, 国发[X]X号)issuing_authority❌Issuing agency (default: "")authority_level❌One of: national, ministerial, local``effective_date❌ISO date format YYYY-MM-DDstatus❌current (default), superseded, draft, repealed``domains❌Domain category (e.g., "矿山安全")category❌Type: "国标", "行标", "政策文件", "地方文件"full_text✅Complete text content for clause splittingsource_url❌Original source URL for attributionpage_count❌Number of pages (for reference)clause_split_pattern❌standard (default), policy, raw_lines

Clause Splitting Patterns#

The tool supports three splitting strategies — choose based on document type:

PatternBest ForHow It WorksstandardGB/AQ national/industry standardsRecognizes chapters (第X章), sections (N.N), sub-sections (N.N.N), appendixespolicyGovernment notices, State Council documentsRecognizes Chinese numbering (一、二、(一)、1.)raw_linesUnstructured text, fallbackSplits by non-empty lines

Test splitting before full import:

bash

python scripts/kb_import.py split-clauses --text "$SAMPLE_TEXT" --pattern standard

Phase 4: Execute Import#

bash

python scripts/kb_import.py import --json manifest.json

What happens during import:

1. For each item in the manifest:
   • Searches existing regulations by document_number
   • If found → UPDATE (overwrite existing data)
   • If not found → INSERT (create new record)
2. Clause processing:
   • Deletes old clauses (if updating)
   • Re-splits full_text using specified pattern
   • Inserts new clause records linked to regulation ID
3. std_registry registration (automatic):
   • If document_number starts with GB/AQ → auto-registers in std_registry table
   • Skips if already registered

Output includes per-item status:

• created — New record inserted
• updated — Existing record overwritten
• skipped — (reserved for future skip logic)
• error — Database error with message

Phase 5: Post-Import Validation#

Always validate after importing:

bash

# Validate specific imported records
python scripts/kb_import.py validate <regulation_id>

# Check overall data quality
python <kb_query_path>/kb_query.py conflicts

# Verify it's findable
python <kb_query_path>/kb_query.py search "<document_number>"

Handling Special Cases#

Scanned/Image-Based PDFs (No Extractable Text)#

When extract-pdf returns "is_scan_only": true:

1. First choice: Install tesseract and run OCR
2. Second choice: Find text version from web sources (government sites, wikisource)
3. Third choice: Extract from related PPT/lecture materials (document as "PPT整理")
4. Last resort: Skip or note as "待补充官方全文"

Important: When using non-official sources (PPT, web scraping), always note this in the source_url field so data provenance is tracked.

Large Standards (e.g., GB 16423 with 80K+ characters)#

No special handling needed — the tool processes them normally. Clause count may be high (2000+). Consider using --pattern standard for best results.

Batch Imports (10+ Items)#

Split manifests into batches of 5-10 items each. Run sequentially. This makes error isolation easier.

Conflict: Existing Record Has Wrong Data#

The tool will overwrite any existing record matching the document_number. Before overwriting:

1. Use safety-kb-query info <id> to check current data
2. If current data looks correct (different standard sharing similar number?), abort and investigate
3. The conflicts command in safety-kb-query can help identify mismatched records proactively

Complete Example: Importing Courseware-Referenced Standards#

This is the canonical workflow when a user says "the standards referenced in my training material aren't in the database":

text

Step 1: Extract references from user's document
        → List: [GB 16423-2020, AQ/T 2033-2023, AQ 2034, 国发[2010]23号]

Step 2: Gap analysis
        $ python kb_query.py check GB16423 AQ2033 AQ2034 "国发[2010]23号"
        → Found: 1, Missing: 3, Issues: 1 (ID:94 has wrong data)

Step 3: Extract text for missing items
        $ python kb_import.py extract-pdf GB16423-2020.pdf
        → { success: true, text: "...", char_count: 80357 }

Step 4: Create manifest.json with all items

Step 5: Execute import
        $ python kb_import.py import --json manifest.json
        → { imported: 3, updated: 2, skipped: 0 }

Step 6: Validate
        $ python kb_import.py validate 94
        → { is_valid: true, issues: [] }

Step 7: Verify
        $ python kb_query.py check GB16423 AQ2033 AQ2034 "国发[2010]23号"
        → All found ✓

Relationship with Other Skills#

SkillRolesafety-kb-queryQuery/read operations; must use BEFORE import for gap detectionsafety-kb-import ( this one)Import/write operations into the databasepdfAdvanced PDF handling (merge, split, watermark) — use for complex PDF prep workstandard-update-coursewareUpdate courseware after standards change — uses both query & import

Known Limitations#

• No rollback: Import commits immediately. Validate before importing bulk data.
• OCR dependency: Scanned PDF handling requires external tesseract installation.
• Clause granularity: Split patterns are heuristic-based; review output for edge cases.
• Single-user: No locking mechanism for concurrent access.

Troubleshooting#

ErrorCauseSolutionDatabase not foundWrong pathSet KB_PATH env var or update DEFAULT_DB_PATH``no such column: XSchema changedRun schema command to verify columnsUNIQUE constraint failedDuplicate insert attemptTool should handle updates; check manifest has unique doc numbersclause_count: 0 after importText empty or pattern mismatchedTry different clause_split_pattern; verify full_text field isn't emptyGarbled Chinese in outputEncoding issueEnsure script runs with UTF-8 locale; Windows: chcp 65001

Version History#

• 1.0.0 (2026-04-25): Initial release with import, extract-pdf, split-clauses, validate, schema commands