Normalize and segment
Normalize Khmer Unicode, score dictionary paths by frequency, and recover unknown words with deterministic rules.
Khmer language engine
Deterministic word boundaries for Khmer text.
A dictionary-based Viterbi segmenter for NLP preprocessing, search, formal documents, and embedded applications, without a runtime machine-learning model.
Language processing
Predictable results with useful diagnostics.
Explain possible errors
Return whole-span typo diagnostics and Khmer-aware ranked suggestions instead of silently changing input.
Support text layout
Generate safe word-break opportunities and typed token metadata with offsets for downstream systems.
Stable pipeline
From Unicode input to structured tokens.
The same deterministic core can support search indexing, document review, OCR post-processing, or browser applications.
POS values are lexical candidates, not contextual part-of-speech tagging. The project code is MIT licensed, while bundled linguistic data carries separate attribution and noncommercial terms.
Khmer Unicodeinput
↓
Normalizationcanonical text
↓
Viterbi decodingfrequency + rules
↓
Tokens and diagnosticsoffsets + suggestions
# Python 3.10 or newer python -m venv .venv # Activate .venv, then install from PyPI python -m pip install khmer-viterbi-segmenter # Segment Khmer text khmer-segment segment "ខ្ញុំស្រឡាញ់ប្រទេសកម្ពុជា"
Quick start
Segment your first Khmer sentence.
Create a virtual environment and install the published package from PyPI. The bundled runtime data works immediately without a separate download.
- activate with
source .venv/bin/activateon Linux or macOS - activate with
.venv\Scripts\Activate.ps1in Windows PowerShell - import the Python package as
khmer_segmenter
Published for Python
Integrate deterministic Khmer segmentation.
Install version 0.2.0 from PyPI, or review the API, ports, data terms, benchmarks, and examples in the project repository.