Document Ingestion Engine — SKILL.md#

When to Use This Skill#

Use when a user needs to process raw accounting source documents into standardized transaction records for QBO import. Triggers on:

• "Process these documents / invoices / receipts / bank statements"
• "Ingest docs for [client]"
• "I have PDFs/CSVs to categorize"
• "Batch import these transactions to QBO"
• "Extract data from 1099s / payroll reports"
• Document drop + categorization requests during month-end close

When NOT to Use#

• Not for running bank reconciliation (use bank-reconciliation skill)
• Not for P&L variance analysis (use pl-quick-compare skill)
• Not for single manual journal entries (just post directly in QBO)
• Not for AR collections or aging (use ar-collections-agent skill)

---

What It Does#

Processes 6 document types → standardized records → Excel workbook + QBO import CSV.

Input TypeFormatsExtractsBank StatementsCSV, OFX/QFX, PDFDate, vendor, amountCredit Card StmtsCSV, PDFDate, merchant, amount, categoryInvoicesPDFVendor, total, date, due date, invoice #, line itemsReceiptsPDF, JPG/PNG*Merchant, date, amount1099 / Tax FormsPDFPayer, TIN, form type, box amountsPayroll ReportsCSV, PDFEmployee, gross, taxes, net per employee

*Image OCR requires tesseract installed.

Processing Steps#

1. File type detection — magic bytes + extension fallback
2. Document classification — bank/CC/invoice/receipt/1099/payroll
3. Content extraction — CSV parsing, OFX parsing, PDF text extraction
4. Format normalization — dates (multi-format), amounts (Decimal), vendor names (strip noise)
5. QBO COA pull — fetches live Chart of Accounts from QBO for categorization
6. Duplicate detection — same amount + vendor within ±3 days → flagged
7. Auto-categorization — vendor map → COA keywords → doc-class default
8. Confidence scoring — HIGH (exact match) / MEDIUM (fuzzy) / LOW (needs review)
9. Exception flagging — missing dates, zero amounts, unknown vendors, LOW confidence
10. QBO import CSV — ready for batch import (excludes dups + failed extractions)
11. Excel workbook — 6 tabs (see below)
12. CDC tracking — delta since last run cached in .cache/document-ingestion/{slug}.json

Excel Output Tabs#

TabContentsProcessed TransactionsAll records with category, confidence, dup flag, exception⚠ ExceptionsRecords needing manual review before importDuplicatesFlagged potential duplicates with "Dup Of" referenceCategory MappingUnique vendor → QBO account map with confidenceImport ReadyQBO-format rows (Date, Description, Amount, Account, Memo)CDC LogDelta metrics vs. prior run + this-run stats summary

---

Script Location#

text

scripts/pipelines/document-ingestion.py

Usage#

bash

# Process a directory of mixed documents
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ~/Downloads/month-end-docs

# Single file
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --file ~/Downloads/invoice_march.pdf

# Multiple files + custom output dir
python3 scripts/pipelines/document-ingestion.py \
    --slug glowlabs \
    --file ~/Downloads/stmt.csv \
    --file ~/Downloads/payroll.csv \
    --out ~/Desktop/ingested

# Offline mode (no QBO auth needed)
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ./docs \
    --no-qbo-coa

# QBO sandbox
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ./docs \
    --sandbox

All CLI Flags#

FlagDefaultDescription--slugrequiredCompany slug (QBO + client vendor map)--input-dir—Directory of docs to process--file—Single file (repeatable)--out``~/DesktopOutput directory--no-qbo-coafalseUse built-in COA only (offline)--sandboxfalseQBO sandbox mode

---

Dependencies#

Required (pip)#

bash

pip install openpyxl

Optional (better extraction quality)#

bash

pip install pdfminer.six   # Better PDF text extraction
pip install ofxparse       # Better OFX/QFX parsing
brew install tesseract     # Image receipt OCR (JPG/PNG)

Node.js QBO Client#

text

Node.js QBO client   # Auth token must be configured

---

Categorization Logic#

Priority Chain#

1. Vendor Map exact match → HIGH confidence
2. Vendor Map substring match → HIGH confidence
3. COA keyword index (built from COA account names + keywords) → MEDIUM confidence
4. Doc-class default → LOW confidence

Built-in Vendor Map#

50+ known vendors pre-mapped:

• Stripe/Square/PayPal → Sales Revenue
• Gusto/ADP/Deel/Paychex → Payroll - Salaries & Wages
• Google/Microsoft/Slack/GitHub/Zoom → Software & Subscriptions
• Delta/United/Marriott/Uber → Travel
• FedEx/UPS/USPS → Postage & Delivery
• Chase/BofA service charges → Bank & Merchant Fees
• etc. (see VENDOR_MAP in script)

Client-Specific Overrides#

Auto-loaded by --slug:

• glowlabs → Loads GlowLabs vendor map (Deel, Toptal, Brex, Huellas Labs, etc.)
• sb-paulson / willo → Loads Willo Salons vendor map
• Other clients → Reads clients/{slug}/categorization-map*.md markdown tables

---

Duplicate Detection Rules#

• Window: ±3 days (configurable via DUP_WINDOW_DAYS constant)
• Match criteria: Same amount (exact Decimal) + same vendor key (first 3 meaningful words)
• Action: Flagged as is_duplicate=True, excluded from import file
• Always confirm before deleting — duplicates tab shows "Dup Of Row #" reference

---

Exception Rules (auto-flagged)#

ConditionFlagMissing transaction date"Missing transaction date"Zero amount (non-1099)"Zero amount — verify or skip"Empty/unknown vendor"Vendor name missing or unknown"LOW confidence category"Low categorization confidence — manual review"PDF extraction failed"PDF text extraction failed — manual review required"Image without tesseract"Image OCR not available — manual entry required"

---

QBO Import CSV Format#

Ready-to-import columns:

text

Date | Description | Amount | Vendor/Customer | Account | Class | Memo | Doc Number

• Amount sign: positive = expense (debit), negative = credit/income
• Memo includes source file + doc type for audit trail
• Excludes: duplicates, failed extractions

---

CDC Cache#

Location: .cache/document-ingestion/{slug}.json

Tracks between runs:

• docs_processed, records_extracted, duplicates_caught
• exceptions_flagged, import_ready
• high_confidence, medium_confidence, low_confidence

---

Output File Naming#

text

DocIngestion_{slug}_{YYYYMMDD}.xlsx
DocIngestion_{slug}_{YYYYMMDD}_QBO_Import.csv

---

Agent Instructions#

Standard Run#

1. Collect input files from user (directory path or individual files)
2. Get client slug ( sb-paulson, glowlabs, etc.)
3. Run pipeline. If QBO auth not set, use --no-qbo-coa
4. Deliver summary:
   • Records extracted, dups caught, exceptions
   • HIGH/MED/LOW confidence split
   • Path to Excel + import CSV
5. Walk user through Exceptions tab — those need action before import

Month-End Close Integration#

• Run AFTER bank statement download, BEFORE bank reconciliation
• Use --input-dir pointing to client's document drop folder
• Import CSV goes into QBO → then run bank-reconciliation.py

Exception Handling#

• PDFs with no extractable text → LOW confidence + exception flag → send to client for re-scan
• Image receipts with no tesseract → exception flag → use nano-pdf skill or manual entry
• Unknown vendors → update VENDOR_MAP in script or add to clients/{slug}/categorization-map.md

Adding New Client Vendor Maps#

Edit load_client_vendor_map() in the script:

python

if slug_lower in ("new-client", "nc"):
    client_map.update({
        "vendor name": "QBO Account Name",
    })

Or create clients/{slug}/categorization-map.md with markdown table:

markdown

| Vendor / Memo Keyword | Primary Account | Notes |
|---|---|---|
| Amazon | Office Supplies | |
| Comcast | Utilities | |

---

Financial Math#

All amounts use Python Decimal with ROUND_HALF_UP to 2 decimal places. No float arithmetic.