- tokenization
-
a) The act or process of tokenizing.
This was an unlikely tokenization of the input string.
b) Something tokenized.
Wikipedia foundation.
tokenization
Look at other dictionaries:
Index (search engine) — Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and… … Wikipedia
Parsing expression grammar — A parsing expression grammar, or PEG, is a type of analytic formal grammar that describes a formal language in terms of a set of rules for recognizing strings in the language. A parsing expression grammar essentially represents a recursive… … Wikipedia
Analyse lexicale — L analyse lexicale se trouve tout au début de la chaîne de compilation. C est la tâche consistant à décomposer une chaîne de caractères en unités lexicales, aussi appelées tokens. Ces tokens, produits à la demande de l analyseur syntaxique, sont… … Wikipédia en Français
Lexical analysis — In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. Programs performing lexical analysis are called lexical analyzers or lexers. A lexer is often organized as separate scanner and … Wikipedia
Search engine technology — Modern web search engines are complex software systems using the technology that has evolved over the years. The largest search engines such as Google and Yahoo! utilize tens or hundreds of thousands of computers to process billions of web pages… … Wikipedia
Dynamic SSL — is an endpoint security technology developed by Daniel McCann and Nima Sharifimehr of NetSecure Technologies Ltd. Dynamic SSL was created to solve the endpoint security problem in public networks by transparently correcting the implementation… … Wikipedia
Programming language — lists Alphabetical Categorical Chronological Generational A programming language is an artificial language designed to communicate instructions to a machine, particularly a computer. Programming languages can be used to create programs that… … Wikipedia
Slot machine — redirects here. For the band, see Slot Machine (band). One armed bandit redirects here. For the album, see One Armed Bandit (album). Slot machines in the Trump Taj Mahal A slot machine (American English), informally fruit machine (British… … Wikipedia
Speech synthesis — Stephen Hawking is one of the most famous people using speech synthesis to communicate Speech synthesis is the artificial production of human speech. A computer system used for this purpose is called a speech synthesizer, and can be implemented… … Wikipedia
GRASS (programming language) — GRASS ( GRAphics Symbiosis System ) was a programming language created to script 2D vector graphics animations. GRASS was similar to BASIC in syntax, but added numerous instructions for specifying 2D object animation, including scaling,… … Wikipedia
