zhpipe

Chinese text enrichment pipeline — segment sentences into words, annotate with tone-marked pinyin, and look up English definitions from CC-CEDICT.

Two tools:

Binary	Description
zhpipe	Read Chinese text from stdin, emit enriched JSON to stdout
zhpipe-whisper	Transcribe audio with whisper.cpp, then enrich the transcript the same way

Quick Start

# Build everything
make

# Enrich Chinese text from stdin
echo "你好世界" | bin/zhpipe

# Pipe from a file
cat sentences.txt | bin/zhpipe | jq '.[0].words'

# Disable translation lookup
echo "你好世界" | bin/zhpipe -translate=false

Whisper Transcription

# Transcribe a video/audio file
ffmpeg -i video.webm -vn -acodec pcm_s16le -ar 16000 -ac 1 -f wav /tmp/test.wav
bin/zhpipe-whisper -model large-v3-turbo /tmp/test.wav

# Pipe WAV from ffmpeg directly
ffmpeg -i video.webm -vn -acodec pcm_s16le -ar 16000 -ac 1 -f wav - | \
  bin/zhpipe-whisper -model large-v3-turbo

# Specify language (default: zh)
bin/zhpipe-whisper -model base -lang auto audio.wav

Note: When piping from ffmpeg, make sure only the WAV audio data goes into zhpipe-whisper, not ffmpeg's error/verbose log output. If you redirect ffmpeg's stderr (e.g. ffmpeg ... 2>ffmpeg.log or ffmpeg ... 2>&1 | ...), those log lines end up in stdin and corrupt the WAV stream. If you want to see ffmpeg's logs, redirect stderr to a file: ffmpeg ... -loglevel verbose 2>ffmpeg.log | zhpipe-whisper ...

Translation Strategy

Dictionary lookups use CC-CEDICT (source) with two fallback mechanisms:

Variant resolution — CC-CEDICT marks some entries as variant references (e.g. "variant of 吃[chi1]"). These are automatically resolved by looking up the referenced headword and appending its real definition:

喫  →  "variant of 吃[chi1] | to eat; to consume"

Character decomposition — When a word is not found in the dictionary (e.g. a name like "金金"), it is split character-by-character using longest-prefix matching, and the individual translations are concatenated:

金金  →  "gold; chemical element Au | gold; chemical element Au"

Output Format

Both tools emit JSON arrays to stdout:

[
  {
    "text": "你好世界",
    "words": [
      {
        "word": "你好",
        "pinyin": "nǐ hǎo",
        "translation": "hello; hi"
      },
      {
        "word": "世界",
        "pinyin": "shì jiè",
        "translation": "the world; earth; globe"
      }
    ]
  }
]

The whisper tool adds startMs and endMs timestamps to each segment and word.

Building

Prerequisites

• Go 1.26+
• Homebrew (for whisper.cpp CGO dependencies)
• ffmpeg (for audio conversion when using whisper)

Install whisper.cpp and its dependencies via Homebrew:

brew install whisper ggml ffmpeg

Build Commands

# Build both tools
make

# Build only the text CLI (pure Go, no CGO)
make cli

# Build only the whisper CLI (requires CGO + whisper.cpp libraries)
make whisper

# Run tests
make test

# Clean
make clean

Manual Build (whisper)

CGO_CFLAGS="-I/opt/homebrew/include" \
CGO_LDFLAGS="-L/opt/homebrew/lib -L${PWD}/lib -lwhisper -lggml -lggml-base -lm -lstdc++ -framework Accelerate -framework Metal -framework Foundation -framework CoreGraphics" \
go build -o bin/zhpipe-whisper ./cmd/whisper

Whisper Models

The whisper tool downloads models on first use and caches them in ~/.cache/zhpipe/models/. Use -mcp to specify a custom cache directory.

Available models: tiny, base, small, medium, medium.en, large-v1, large-v2, large-v3, large-v3-turbo (plus quantized variants like base-q5_1, large-v3-turbo-q8_0).

# Use a specific model
bin/zhpipe-whisper -model large-v3-turbo audio.wav

# Cache models in a custom directory
bin/zhpipe-whisper -model base -mcp /data/models audio.wav

Project Structure

zhpipe/
├── cmd/
│   ├── cli/          # Text enrichment CLI
│   │   └── main.go
│   └── whisper/      # Audio transcription CLI (CGO)
│       ├── main.go
│       └── cgo.go
├── internal/
│   ├── dict/         # CC-CEDICT dictionary (embedded)
│   ├── model/        # Whisper model downloader/cache
│   └── segmenter/    # Jieba segmentation + pinyin
├── bin/              # Build output (gitignored)
├── Makefile
└── README.md

License

MIT