LLM Chain#

An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.

Commands#

Core AI Operations#

CommandDescriptionllm-chain configure <input>Record a configuration change (or view recent configs with no args)llm-chain benchmark <input>Log a benchmark run and its resultsllm-chain compare <input>Record a model or output comparisonllm-chain prompt <input>Log a prompt template or prompt engineering notellm-chain evaluate <input>Record an evaluation result or metricllm-chain fine-tune <input>Log a fine-tuning session or parametersllm-chain analyze <input>Record an analysis observationllm-chain cost <input>Log cost tracking data (tokens, dollars, etc.)llm-chain usage <input>Record API usage metricsllm-chain optimize <input>Log an optimization attempt and outcomellm-chain test <input>Record a test case or test resultllm-chain report <input>Log a report entry or summary

Utility Commands#

CommandDescriptionllm-chain statsShow summary statistics across all log filesllm-chain export <fmt>Export all data in json, csv, or txt formatllm-chain search <term>Search all entries for a keyword (case-insensitive)llm-chain recentShow the 20 most recent activity log entriesllm-chain statusHealth check: version, entry count, disk usage, last activityllm-chain helpDisplay full command referencellm-chain versionPrint current version (v2.0.0)

How It Works#

Every core command accepts free-text input. When called with arguments, LLM Chain:

1. Timestamps the entry ( YYYY-MM-DD HH:MM)
2. Appends it to the command-specific log file (e.g. benchmark.log, cost.log)
3. Records the action in a central history.log
4. Reports the saved entry and running total

When called with no arguments, each command displays the 20 most recent entries from its log file.

Data Storage#

All data is stored locally in plain-text log files:

text

~/.local/share/llm-chain/
├── configure.log     # Configuration changes
├── benchmark.log     # Benchmark results
├── compare.log       # Model comparisons
├── prompt.log        # Prompt templates & notes
├── evaluate.log      # Evaluation metrics
├── fine-tune.log     # Fine-tuning sessions
├── analyze.log       # Analysis observations
├── cost.log          # Cost tracking
├── usage.log         # API usage metrics
├── optimize.log      # Optimization attempts
├── test.log          # Test cases & results
├── report.log        # Report entries
├── history.log       # Central activity log
└── export.{json,csv,txt}  # Exported snapshots

Each log uses pipe-delimited format: timestamp|value.

Requirements#

• Bash 4.0+ with set -euo pipefail
• Standard Unix utilities: wc, du, grep, tail, date, sed
• No external dependencies — pure bash

When to Use#

1. Tracking LLM experiments — log benchmark results, prompt variations, and evaluation scores as you iterate on model configurations
2. Cost monitoring — record token usage, API costs, and billing data to keep spending under control across multiple models
3. Comparing models side-by-side — use compare and benchmark to log performance differences between GPT-4, Claude, Gemini, etc.
4. Fine-tuning documentation — capture fine-tuning parameters, dataset info, and results for reproducibility
5. Generating operational reports — export all logged data to JSON/CSV for dashboards, audits, or stakeholder reviews

Examples#

bash

# Log a configuration change
llm-chain configure "switched to gpt-4o, temperature=0.7, max_tokens=2048"

# Record a benchmark result
llm-chain benchmark "gpt-4o MMLU=87.2% latency=1.3s cost=$0.012/req"

# Track a cost entry
llm-chain cost "2024-03-18: 142k tokens, $4.26 total (gpt-4o)"

# Compare two models
llm-chain compare "claude-3.5 vs gpt-4o: claude wins on reasoning, gpt wins on speed"

# Log a prompt engineering note
llm-chain prompt "added chain-of-thought prefix: 'Let me think step by step...'"

# Search all logs for a keyword
llm-chain search "gpt-4o"

# Export everything to JSON
llm-chain export json

# Check health and disk usage
llm-chain status

Configuration#

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-chain/

---

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com