Normalize and segment
Normalize Khmer Unicode, score dictionary paths by frequency, recover unknown words with deterministic rules, and split long curated forms with runtime word composition.
Khmer language engine
Deterministic word boundaries for Khmer text.
A dictionary-based Viterbi segmenter for NLP preprocessing, search, formal documents, and embedded applications, without a runtime machine-learning model.
Language processing
Predictable results with useful diagnostics.
Explain possible errors
Return whole-span typo diagnostics, Khmer-aware ranked suggestions, and unknown_word flags instead of silently changing input.
Support text layout
Generate safe word-break opportunities and typed token metadata with offsets for downstream systems.
Stable pipeline
From Unicode input to structured tokens.
The same deterministic core can support search indexing, document review, OCR post-processing, or browser applications.
POS values are lexical candidates, not contextual part-of-speech tagging. The project code is MIT licensed, while bundled linguistic data carries separate attribution and noncommercial terms.
Khmer Unicodeinput
↓
Normalizationcanonical text
↓
Viterbi decodingfrequency + rules
↓
Tokens and diagnosticsoffsets + suggestions
# Python 3.10 or newer python -m venv .venv # Activate .venv, then install from PyPI python -m pip install khmer-viterbi-segmenter # Segment Khmer text khmer-segment segment "ខ្ញុំស្រឡាញ់ប្រទេសកម្ពុជា"
Quick start
Segment your first Khmer sentence.
Create a virtual environment and install the published package from PyPI. The bundled runtime data works immediately without a separate download.
- activate with
source .venv/bin/activateon Linux or macOS - activate with
.venv\Scripts\Activate.ps1in Windows PowerShell - import the Python package as
khmer_segmenter
Release 0.3.2
Safer segmentation for editors and layout.
The 0.3.x line adds a reviewed spelling-authority layer, runtime word composition, and a new out-of-vocabulary diagnostic, with identical behavior across the Python, Rust, and WebAssembly ports.
Composition and word breaks
Split long curated forms into smaller accepted words with a reviewed keep-list, and expose safe Khmer word-break opportunities with a lone-consonant guard.
Spelling authority
Choose an official or community spelling authority, with visual accuracy accepted by default so equivalent COENG DA/TA forms both pass.
Out-of-vocabulary diagnostics
Report unsupported fragments as unknown_word without guessing a correction, while genuine typos still return ranked suggestions.
Also in 0.3.x: reviewed phrase-collision exclusions stop common phrases such as
អ្នកគ្រូ from being flagged as typos, a secondary Chuon Nath 1967 frequency source is blended into segmentation costs, and the Python, Rust, WebAssembly, and embedded C ports stay aligned.Published for Python
Integrate deterministic Khmer segmentation.
Install version 0.3.2 from PyPI, or review the API, ports, data terms, benchmarks, and examples in the project repository.