Concept information
Preferred term
tokenisation
Definition
- turning a sequence of characters (a text) into a sequence of occurrences of inflected word forms and interpunction symbols (often called 'tokens')
Broader concept
URI
https://vocabs.sshopencloud.eu/vocabularies/csd-tool-tasks/0102
{{label}}
{{#each values }} {{! loop through ConceptPropertyValue objects }}
{{#if prefLabel }}
{{/if}}
{{/each}}
{{#if notation }}{{ notation }} {{/if}}{{ prefLabel }}
{{#ifDifferentLabelLang lang }} ({{ lang }}){{/ifDifferentLabelLang}}
{{#if vocabName }}
{{ vocabName }}
{{/if}}
