Transformersを用いた古典中国語(漢文)文切りモデルの製作

Abstract

古典中国語(漢文)は，単語と単語の間に区切りがなく，文と文の間にも区切りがない．これが，白文と呼ばれる古典中国語の書写形態であり，傍目には，漢字が連続的に並んでいるだけである．それゆえ，白文に対する言語処理は，単語と単語の間を区切る「単語切り」(word tokenization)と，文と文の間を区切る「文切り」(sentencesegmentation) から始まる．本稿では，Transformersを用いた古典中国語の「文切り」手法と，それに伴う古典中国語「文切り」モデルの製作について述べる．これに加え，Transformers を用いた古典中国語の「単語切り」を，品詞付与と同時に行う手法についても述べる．

Classical Chinese texts do not have any spaces or punctuations between words or between sentences. They consist of continuous strings of Chinese characters from the start to the end. The analysis of classical Chinese texts has to begin with finding out word-boundaries and sentence-boundaries, i.e. word tokenization and sentence segmentation. In this paper we discuss the sentence segmentation of classical Chinese texts using Transformers, which is designed for natural language processing with pre-trained language models such as BERT and RoBERTa. In the appendix we discuss the word tokenization (and Part-Of-Speech tagging) of classical Chinese texts using Transformers.