Package cssutils :: Module tokenize2 :: Class Tokenizer |
[hide private]
[frames] | no frames]

# Class Tokenizer

source code

    object --+
             |
            Tokenizer

generates a list of Token tuples:
  (Tokenname, value, startline, startcolumn)

Instance Methods | [hide private]
__init__(self, macros=None, productions=None) inits tokenizer with given macros and productions which default to cssutils own macros and productions | source code
_expand_macros(self, macros, productions) returns macro expanded productions, order of productions is kept | source code
_compile_productions(self, expanded_productions) compile productions into callable match objects, order is kept | source code
tokenize(self, text, fullsheet=False) Generator: Tokenize text and yield tokens, each token is a tuple of: | source code
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]
| _atkeywords = ``{``u'``@font-face``'``:` `'``FONT_FACE_SYM``'``,` `u'``@import``'``:` `'``IM``...``
| _linesep = ``...``

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, macros=None, productions=None) (Constructor) | source code
inits tokenizer with given macros and productions which default to cssutils own macros and productions Overrides: object.__init__

### tokenize(self, text, fullsheet=False) | source code
Generator: Tokenize text and yield tokens, each token is a tuple of:
    (nname, value, line, col)
The token value will contain a normal string, meaning CSS unicode escapes have been resolved to normal characters. The serializer escapes needed characters back to unicode escapes depending on the stylesheet target encoding. text to be tokenized fullsheet if `True` appends EOF token as last one and completes incomplete COMMENT or INVALID (to STRING) tokens

Class Variable Details | [hide private]

  ### _atkeywords Value:
      {u'@font-face': 'FONT_FACE_SYM',
       u'@import': 'IMPORT_SYM',
       u'@media': 'MEDIA_SYM',
       u'@namespace': 'NAMESPACE_SYM',
       u'@page': 'PAGE_SYM'}

  ### _linesep Value:
      u'''
      '''