Data Analyzer — 数据分析工具#

Load, analyze, and report on structured data from CSV, Excel, and JSON files. Compute statistics, detect anomalies, identify trends, and generate reports with visualization recommendations.

Workflow#

text

1. Load data     → Read the file, inspect structure
2. Profile       → Column types, missing values, basic stats
3. Analyze       → Statistics, trends, anomalies, correlations
4. Report        → Summary with visual recommendations

Step 1 — Data Loading#

Supported Formats#

FormatHow to ReadNotesCSVRead the file directly, parse header row + data rowsCheck delimiter (comma, tab, semicolon). Handle quoted fields.**Excel (.xlsx)**Read via openpyxl or pandas. If unavailable, convert to CSV first.Handle multiple sheets. Note which sheet was used.JSONParse as structured objects. Detect if array-of-objects or object-of-arrays.Flatten nested structures where possible.TSVSame as CSV with tab delimiter.

If Python is available (recommended for large datasets):

bash

pip install pandas openpyxl  # if missing
python3 -c "
import pandas as pd
df = pd.read_csv('data.csv')
print(df.info())
print(df.describe())
print(df.head())
"

If Python is not available, parse manually:

1. Read the file line by line
2. Identify headers (first row)
3. Identify column types (numeric vs text vs date)
4. Store as an array of rows or objects

Initial Inspection#

After loading, always answer these questions:

• Shape: How many rows and columns?
• Column names: What are they and what data types?
• Missing values: Which columns have gaps, and how many?
• Date/time columns: Are they parsed as datetime objects?
• Unique values: For categorical columns, how many unique categories?

Step 2 — Descriptive Statistics#

Numeric Columns#

Compute and report:

StatisticWhat It Tells YouCountNumber of non-null valuesMeanAverage valueMedianMidpoint (50th percentile) — more robust than mean for skewed dataStd DevSpread around the meanMin / MaxFull range25th / 75th PercentileInterquartile range boundsSkewnessSymmetry of the distribution. Positive = right tail, negative = left tail.

Formula reference (manual calculation):

text

Mean       = sum(x) / n
Median     = middle value when sorted
Std Dev    = sqrt(sum((x - mean)^2) / (n-1))
Percentile = sort values, take value at position (p/100 * n)

Categorical Columns#

StatisticWhat It Tells YouCountTotal non-null valuesUniqueNumber of distinct categoriesTopMost frequent categoryFrequencyHow often the top category appearsDistributionShare of each category (as percentages)

Step 3 — Analysis#

3a. Anomaly Detection#

Method: IQR (Interquartile Range)

text

Q1 = 25th percentile
Q3 = 75th percentile
IQR = Q3 - Q1
Lower fence = Q1 - 1.5 * IQR
Upper fence = Q3 + 1.5 * IQR
Anomaly = any value outside [Lower fence, Upper fence]

Method: Z-Score (for approximately normal distributions)

text

z = (x - mean) / std_dev
Anomaly = |z| > 3 (values more than 3 std devs from mean)

Output anomalies: For each detected anomaly, report:

• Row index
• Column name
• Anomalous value
• Distance from expected (how many IQRs or std devs)

3b. Trend Analysis#

For time-series data (data with a date/time column):

1. Identify the time column — Sort by date
2. Aggregate by period — Group by day/week/month/quarter/year
3. Direction — Is the metric increasing, decreasing, or flat?
4. Rate of change — Period-over-period percentage change
5. Seasonality — Recurring patterns (monthly, quarterly, yearly)
6. Breakout — Sudden jumps or drops (potential regime changes)

Output format:

text

📈 Trend: [Metric Name]
Period: [Date Range]
Direction: [Up/Down/Flat] (slope: ±X%)
Key Points:
- [Date]: Value = X (↗/↘/→)
- Highest point: [Date] = X
- Lowest point: [Date] = X

For non-time-series data, analyze rank order and distribution shape:

text

Top 5 by [metric]:
1. [Category] = X
2. [Category] = Y
...
Bottom 5 by [metric]:

3c. Correlation Analysis#

Pearson correlation coefficient (for linear relationships between two numeric variables):

text

r = sum((x - mean_x) * (y - mean_y)) / (n * std_x * std_y)

Interpretation:

r valueStrengthDirection0.7 to 1.0StrongPositive (both rise together)0.3 to 0.7ModeratePositive0 to 0.3WeakPositive-0.3 to 0WeakNegative (one rises, other falls)-0.7 to -0.3ModerateNegative-1.0 to -0.7StrongNegative

Caveats:

• Correlation ≠ causation. Always note this.
• Pearson only captures linear relationships.
• Outliers can distort correlation heavily — check after removing anomalies.

Step 4 — Report Generation#

Visualization Recommendations#

For each finding, recommend the best chart type:

Analysis TypeRecommended ChartWhyDistribution of one variableHistogramShows shape, skew, peaksComparison across categoriesBar chartEasy to compare magnitudesTrend over timeLine chartEmphasizes direction and continuityRelationship between 2 variablesScatter plotShows correlation, clusters, outliersPart of a wholePie / Donut chartUse only for 2-5 categoriesComposition over timeStacked area chartShows both total and partsRank orderHorizontal bar chartEasy to read sorted valuesComparing multiple distributionsBox plotShows median, IQR, outliersHeatmap (correlation matrix)HeatmapQuick visual of many correlations

Full Report Template#

text

# Data Analysis Report: [Dataset Name]
Date: [YYYY-MM-DD]

## 1. Overview
- Rows: X | Columns: Y
- Missing data: X cells (X%)
- Key columns: [list with types]

## 2. Descriptive Statistics
### Numeric Columns
[Table: col_name, count, mean, median, std, min, 25%, 75%, max]

### Categorical Columns
[Table: col_name, unique_count, top_value, frequency%]

## 3. Key Findings

### Finding 1: [Title]
[Description of finding]
📊 Recommended chart: [Chart type]
Supporting data: [stats/view]

### Finding 2: [Title]
...

## 4. Anomalies Detected
[Table: row, column, value, severity]

## 5. Correlations
[Notable correlations >|0.3| or < -|0.3|]

## 6. Recommendations
[Data-driven suggestions based on analysis]

One-Page Summary (Quick)#

For quick results, use this compact format:

text

📊 [Dataset]: [N] rows × [M] cols

📈 Key metrics:
- [metric1]: mean=X, median=Y, range=[min, max]
- [metric2]: ...

🔍 Top findings:
1. [Finding] — [chart recommendation]
2. [Finding] — [chart recommendation]

⚠️ Anomalies: X detected

Python Script (Optional)#

For complex analysis, create and run a Python script:

python

import csv, json, statistics
from collections import Counter

# Load data
with open('data.csv') as f:
    reader = csv.DictReader(f)
    rows = list(reader)

# Get numeric columns
# (column name → list of float values, filtering out blanks)
# Compute mean, median, stdev, percentiles
# Detect outliers via IQR
# Compute correlations between pairs
# Print formatted results

Run with:

bash

python3 analysis.py