tokenizer
Look at other dictionaries:
Tokenizer — Ein Tokenizer (auch: lexikalischer Scanner, kurz Lexer) ist ein Computerprogramm zur Zerlegung von Plain text (zum Beispiel Quellcode) in Folgen von logisch zusammengehörigen Einheiten, so genannte Token (engl. tokens). Als solcher ist er oft… … Deutsch Wikipedia
Atari BASIC — For the version of BASIC bundled with the Atari ST computer series, see Atari ST BASIC . ATARI BASIC is a ROM resident BASIC interpreter for the Atari 8 bit family of 6502 based home computers. The interpreter originally shipped on an 8 KB… … Wikipedia
Lexical analysis — In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. Programs performing lexical analysis are called lexical analyzers or lexers. A lexer is often organized as separate scanner and … Wikipedia
PHPDoc — is an adaptation of Javadoc for the PHP programming language. It is a formal standard for commenting PHP code. It offers three main advantages to generic or random commenting styles: First, it makes comments readable in a standard method… … Wikipedia
Comparison of layout engines (HTML5) — HTML HTML and HTML5 Dynamic HTML XHTML XHTML Mobile Profile and C HTML Canvas element Character encodings Document Object Model Font family HTML editor HTML element HTML Frames HTML5 video HTML scrip … Wikipedia
Commodore BASIC — Commodore BASIC, also known as PET BASIC, is the dialect of the BASIC programming language used in Commodore International s 8 bit home computer line, stretching from the PET of 1977 to the C128 of 1985. The core was based on 6502 Microsoft BASIC … Wikipedia
ThinBasic — Infobox programming language name = ThinBasic developer = Eros Olmi and Roberto Bianchi latest release version = v1.6.0.10 latest release date = July 5 2008 dialects = BASIC operating system = Windows license = Freeware website =… … Wikipedia
Index (search engine) — Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and… … Wikipedia
General Architecture for Text Engineering — Infobox Software name = GATE caption = General Architecture for Text Engineering. developer = [http://gate.ac.uk/ GATE research team] , Dept. Computer Science, University of Sheffield released = 1996 frequently updated = yes programming language … Wikipedia
CMS Made Simple — Not to be confused with CMSimple. CMS Made Simple CMS Made Simple Admin Portal … Wikipedia
