CLARIN taxonomy for tools and services tasks

Sök i vokabulär

Innehållets språk

Begreppsinformation

analysis > linguistic analysis > orthographic processing > tokenisation
> annotation > linguistic annotation > orthographic processing > tokenisation

Föredragen term

tokenisation  

Definition

  • turning a sequence of characters (a text) into a sequence of occurrences of inflected word forms and interpunction symbols (often called 'tokens')

Överordnat begrepp

URI

https://vocabs.sshopencloud.eu/vocabularies/csd-tool-tasks/0102

Ladda ned detta begrepp:

RDF/XML TURTLE JSON-LD Senast editerad 2024-01-02