Pygments

Architecture

There are four types of components that work together highlighting a piece of code:

• A lexer splits the source into tokens, fragments of the source that have a token type that determines what the text represents semantically (e.g., keyword, string, or comment). There is a lexer for every language or markup format that Pygments supports.
• The token stream can be piped through filters, which usually modify the token types or text fragments, e.g. uppercasing all keywords.
• A formatter then takes the token stream and writes it to an output file, in a format such as HTML, LaTeX or RTF.
• While writing the output, a style determines how to highlight all the different token types. It maps them to attributes like "red and bold".

Example

Here is a small example for highlighting Python code:

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

code = 'print "Hello World"'
print highlight(code, PythonLexer(), HtmlFormatter())

which prints something like this:

<div class="highlight">
<pre><span class="k">print</span> <span class="s">&quot;Hello World&quot;</span></pre>
</div>

As you can see, Pygments uses CSS classes (by default, but you can change that) instead of inline styles in order to avoid outputting redundant style information over and over. A CSS stylesheet that contains all CSS classes possibly used in the output can be produced by:

print HtmlFormatter().get_style_defs('.highlight')

The argument to get_style_defs is used as an additional CSS selector: the output may look like this:

.highlight .k { color: #AA22FF; font-weight: bold }
.highlight .s { color: #BB4444 }
...

Options

The highlight() function supports a fourth argument called outfile, it must be a file object if given. The formatted output will then be written to this file instead of being returned as a string.

Lexers and formatters both support options. They are given to them as keyword arguments either to the class or to the lookup method:

from pygments import highlight
from pygments.lexers import get_lexer_by_name
from pygments.formatters import HtmlFormatter

lexer = get_lexer_by_name("python", stripall=True)
formatter = HtmlFormatter(linenos=True, cssclass="source")
result = highlight(code, lexer, formatter)

This makes the lexer strip all leading and trailing whitespace from the input (stripall option), lets the formatter output line numbers (linenos option), and sets the wrapping <div>'s class to source (instead of highlight).

Important options include:

encoding : for lexers and formatters

Since Pygments uses Unicode strings internally, this determines which encoding will be used to convert to or from byte strings.

style : for formatters

The name of the style to use when writing the output.

For an overview of builtin lexers and formatters and their options, visit the lexer and formatters lists.

For a documentation on filters, see this page.

Lexer and formatter lookup

If you want to lookup a built-in lexer by its alias or a filename, you can use one of the following methods:

>>> from pygments.lexers import (get_lexer_by_name,
...     get_lexer_for_filename, get_lexer_for_mimetype)

>>> get_lexer_by_name('python')
<pygments.lexers.PythonLexer>

>>> get_lexer_for_filename('spam.rb')
<pygments.lexers.RubyLexer>

>>> get_lexer_for_mimetype('text/x-perl')
<pygments.lexers.PerlLexer>

All these functions accept keyword arguments; they will be passed to the lexer as options.

A similar API is available for formatters: use get_formatter_by_name() and get_formatter_for_filename() from the pygments.formatters module for this purpose.

Guessing lexers

If you don't know the content of the file, or you want to highlight a file whose extension is ambiguous, such as .html (which could contain plain HTML or some template tags), use these functions:

>>> from pygments.lexers import guess_lexer, guess_lexer_for_filename

>>> guess_lexer('#!/usr/bin/python\nprint "Hello World!"')
<pygments.lexers.PythonLexer>

>>> guess_lexer_for_filename('test.py', 'print "Hello World!"')
<pygments.lexers.PythonLexer>

guess_lexer() passes the given content to the lexer classes' analyse_text() method and returns the one for which it returns the highest number.

All lexers have two different filename pattern lists: the primary and the secondary one. The get_lexer_for_filename() function only uses the primary list, whose entries are supposed to be unique among all lexers. guess_lexer_for_filename(), however, will first loop through all lexers and look at the primary and secondary filename patterns if the filename matches. If only one lexer matches, it is returned, else the guessing mechanism of guess_lexer() is used with the matching lexers.

As usual, keyword arguments to these functions are given to the created lexer as options.

Command line usage

You can use Pygments from the command line, using the pygmentize script:

$ pygmentize test.py

will highlight the Python file test.py using ANSI escape sequences (a.k.a. terminal colors) and print the result to standard output.

To output HTML, use the -f option:

$ pygmentize -f html -o test.html test.py

to write an HTML-highlighted version of test.py to the file test.html. Note that it will only be a snippet of HTML, if you want a full HTML document, use the "full" option:

$ pygmentize -f html -O full -o test.html test.py

This will produce a full HTML document with included stylesheet.

A style can be selected with -O style=<name>.

If you need a stylesheet for an existing HTML file using Pygments CSS classes, it can be created with:

$ pygmentize -S default -f html > style.css

where default is the style name.

More options and tricks and be found in the command line reference.