Chinese NLP Toolkit#

Process and analyze Chinese text with specialized NLP capabilities.

Core Capabilities#

1. Text Segmentation (分词)#

Chinese has no word boundaries. Segmentation is the foundation of all Chinese NLP.

Approach: Use rule-based heuristics when no library is available:

• Dictionary matching (maximum forward/backward matching)
• Context-aware: "南京市长江大桥" → ["南京市", "长江大桥"] not ["南京", "市长", "江大桥"]
• Domain-specific terms should be added as custom dictionary entries

Common Ambiguities:

TextWrong SplitCorrect Split雨伞雨/伞雨伞 (compound)结婚的和尚未结婚的结婚/的/和尚/未/结婚/的结婚/的/和/尚未/结婚/的项目部项目/部项目部 (compound)

2. Sentiment Analysis (情感分析)#

Beyond positive/negative — Chinese sentiment is nuanced:

Intensity levels: 强烈负面 < 偏负面 < 中性 < 偏正面 < 强烈正面

Chinese-specific signals:

• Rhetorical questions often indicate negative sentiment: "这也算好？"
• Sarcasm markers: "呵呵", "厉害了", "也是醉了", "你开心就好"
• Intensifiers: "非常", "特别", "简直了", "超级"
• Diminishers: "还行吧", "马马虎虎", "凑合"

Emoji contribution (critical for social media):

• 😊👍❤️ = positive amplification
• 😤👎💔 = negative amplification
• 🙄🙄🙄 = sarcasm/disdain (intensity scales with repetition)

3. Keyword Extraction (关键词提取)#

For Chinese text, prioritize:

• Noun phrases (名词短语)
• Domain-specific terminology
• Named entities (人名、地名、机构名)

Method: TF-IDF adapted for Chinese + positional weighting (first/last sentences carry more weight in Chinese writing).

4. Text Summarization (文本摘要)#

Chinese-specific rules:

• Summarize to 20-30% of original length
• Preserve key numbers, names, and claims
• Chinese articles often "bury the lead" — the conclusion may be more important than the introduction
• Extract key sentences using positional + keyword scoring

5. Readability Scoring (可读性评分)#

Rate Chinese text on a 1-10 scale considering:

• Average sentence length (characters per sentence)
• Vocabulary difficulty (HSK level estimate)
• Clause density ( commas per sentence)
• Use of classical Chinese elements
• Technical jargon density

ScoreLevelTarget Audience1-3EasyGeneral public4-6ModerateEducated readers7-8HardDomain experts9-10Very HardAcademic specialists

6. Format Conversion#

ConversionExampleSimplified → Traditional体验 → 體驗Traditional → Simplified體驗 → 体验Chinese → Pinyin你好 → nǐ hǎoChinese → Zhuyin你好 → ㄋㄧˇ ㄏㄠˇ

Workflow#

When Processing Chinese Text:#

1. Detect variant: Simplified (简体) or Traditional (繁体)?
2. Segment: Break into meaningful units
3. Analyze: Apply the requested analysis type(s)
4. Report: Present results with Chinese annotations

Output Format#

text

原文：[original text]
分词：[segmented text with / separators]
关键词：[top 5-10 keywords with relevance scores]
情感：[sentiment label + confidence + key signals]
摘要：[summarized text]
可读性：[score/10 + brief explanation]

Edge Cases#

• Mixed-language text: Handle code-switching naturally ("这个bug太坑了") — don't force Chinese segmentation on English words
• Internet slang: Recognize common abbreviations (yyds, xswl, nbcs, awsl) and expand for formal analysis
• Poetry/classical Chinese: Flag as special case — modern NLP rules don't apply; use classical grammar patterns
• Dialectal text: Flag non-Mandarin text (Cantonese, Shanghainese written forms) — analysis may be unreliable
• Zero-width characters: Chinese text sometimes contains invisible characters (U+200B, U+FEFF) that affect processing

Common Tasks & Prompts#

• "Analyze the sentiment of this Chinese review"
• "Extract keywords from this article"
• "Summarize this Chinese news article in 100 characters"
• "Rate the readability of this document"
• "Convert this to Traditional Chinese with pinyin annotation"
• "Segment this Chinese text and identify named entities"