Header

UZH-Logo

Maintenance Infos

The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles


Moran, Steven; Cysouw, Michael (2017). The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles. Berlin: Language Science Press.

Abstract

This text is a practical guide for linguists, and programmers, who work with data in multilingual computational environments. We introduce the basic concepts needed to understand how writing systems and character encodings function, and how they work together at the intersection between the Unicode Standard and the International Phonetic Alphabet. Although these standards are often met with frustration by users, they nevertheless provide language researchers and programmers with a consistent computational architecture needed to process, publish and analyze lexical data from the world's languages. Thus we bring to light common, but not always transparent, pitfalls which researchers face when working with Unicode and IPA. Having identified and overcome these pitfalls involved in making writing systems and character encodings syntactically and semantically interoperable (to the extent that they can be), we created a suite of open-source Python and R tools to work with languages using orthography profiles that describe author- or document-specific orthographic conventions. In this cookbook we describe a formal specification of orthography profiles and provide recipes using open source tools to show how users can segment text, analyze it, identify errors, and to transform it into different written forms for comparative linguistics research.

Abstract

This text is a practical guide for linguists, and programmers, who work with data in multilingual computational environments. We introduce the basic concepts needed to understand how writing systems and character encodings function, and how they work together at the intersection between the Unicode Standard and the International Phonetic Alphabet. Although these standards are often met with frustration by users, they nevertheless provide language researchers and programmers with a consistent computational architecture needed to process, publish and analyze lexical data from the world's languages. Thus we bring to light common, but not always transparent, pitfalls which researchers face when working with Unicode and IPA. Having identified and overcome these pitfalls involved in making writing systems and character encodings syntactically and semantically interoperable (to the extent that they can be), we created a suite of open-source Python and R tools to work with languages using orthography profiles that describe author- or document-specific orthographic conventions. In this cookbook we describe a formal specification of orthography profiles and provide recipes using open source tools to show how users can segment text, analyze it, identify errors, and to transform it into different written forms for comparative linguistics research.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

1401 downloads since deposited on 16 Feb 2017
135 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Monograph
Communities & Collections:06 Faculty of Arts > Department of Comparative Language Science
Dewey Decimal Classification:490 Other languages
890 Other literatures
410 Linguistics
Language:English
Date:2017
Deposited On:16 Feb 2017 10:57
Last Modified:14 Feb 2023 10:42
Publisher:Language Science Press
Series Name:Translation and Multilingual Natural Language Processing
Volume:10
Number of Pages:132
ISBN:978-3-96110-091-0
Additional Information:Moran, Steven, & Cysouw, Michael. (2018). The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles (Version 2.0.1). Zenodo. https://doi.org/10.5281/zenodo.1300528 Version published by language science press.
OA Status:Gold
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.5281/zenodo.290662
Official URL:http://langsci-press.org/catalog/book/176
Related URLs:https://github.com/unicode-cookbook
(Publisher)
Other Identification Number:eISBN 978-3-96110-090-3 / ISSN: 2364-8899
Project Information:
  • : FunderFP7
  • : Grant ID240816
  • : Project TitleQUANTHISTLING - Quantitative modeling of historical-comparative linguistics: Unraveling the phylogeny of native South American languages
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)
  • Content: Published Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)