|  |  |  | Python/C API Reference Manual |  |  |  | 
 
 
7.3.2.1 Built-in Codecs 
Python provides a set of builtin codecs which are written in C
for speed. All of these codecs are directly usable via the
following functions.
Many of the following APIs take two arguments encoding and
errors. These parameters encoding and errors have the same semantics
as the ones of the builtin unicode() Unicode object constructor.
Setting encoding to NULL causes the default encoding to be used
which is ASCII.  The file system calls should use
Py_FileSystemDefaultEncoding as the encoding for file
names. This variable should be treated as read-only: On some systems,
it will be a pointer to a static string, on others, it will change at
run-time (such as when the application invokes setlocale).
Error handling is set by errors which may also be set to NULL
meaning to use the default handling defined for the codec.  Default
error handling for all builtin codecs is ``strict''
(ValueError is raised).
The codecs all use a similar interface.  Only deviation from the
following generic ones are documented for simplicity.
These are the generic codec APIs:
| PyObject* PyUnicode_Decode( | const char *s,
                                               int size,
                                               const char *encoding,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the encoded
  string s.  encoding and errors have the same
  meaning as the parameters of the same name in the
  unicode() builtin function.  The codec to be used is
  looked up using the Python codec registry.  Return NULL if an
  exception was raised by the codec.
| PyObject* PyUnicode_Encode( | const Py_UNICODE *s,
                                               int size,
                                               const char *encoding,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size and return
  a Python string object.  encoding and errors have the
  same meaning as the parameters of the same name in the Unicode
  encode() method.  The codec to be used is looked up using
  the Python codec registry.  Return NULL if an exception was
  raised by the codec.
| PyObject* PyUnicode_AsEncodedString( | PyObject *unicode,
                                               const char *encoding,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode object and return the result as Python string
  object. encoding and errors have the same meaning as the
  parameters of the same name in the Unicode encode() method.
  The codec to be used is looked up using the Python codec registry.
  Return NULL if an exception was raised by the codec.
These are the UTF-8 codec APIs:
| PyObject* PyUnicode_DecodeUTF8( | const char *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the UTF-8
  encoded string s. Return NULL if an exception was raised
  by the codec.
| PyObject* PyUnicode_DecodeUTF8Stateful( | const char *s,
                                               int size,
                                               const char *errors,
                                               int *consumed) |  
 
- 
  Return value:
  New reference.
 If consumed is NULL, behave like PyUnicode_DecodeUTF8().
  If consumed is not NULL, trailing incomplete UTF-8 byte sequences
  will not be treated as an error. Those bytes will not be decoded and the
  number of bytes that have been decoded will be stored in consumed.
  
New in version 2.4.
| PyObject* PyUnicode_EncodeUTF8( | const Py_UNICODE *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using UTF-8
  and return a Python string object.  Return NULL if an exception
  was raised by the codec.
| PyObject* PyUnicode_AsUTF8String( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using UTF-8 and return the result as
  Python string object.  Error handling is ``strict''.  Return
  NULL if an exception was raised by the codec.
These are the UTF-16 codec APIs:
| PyObject* PyUnicode_DecodeUTF16( | const char *s,
                                               int size,
                                               const char *errors,
                                               int *byteorder) |  
 
- 
  Return value:
  New reference.
 Decode length bytes from a UTF-16 encoded buffer string and
  return the corresponding Unicode object.  errors (if
  non-NULL) defines the error handling. It defaults to ``strict''.
If byteorder is non-NULL, the decoder starts decoding using
  the given byte order:
 
 
   *byteorder == -1: little endian
   *byteorder == 0:  native order
   *byteorder == 1:  big endian
 
and then switches according to all byte order marks (BOM) it finds
  in the input data.  BOMs are not copied into the resulting Unicode
  string.  After completion, *byteorder is set to the current
  byte order at the end of input data.
 
If byteorder is NULL, the codec starts in native order mode.
 
Return NULL if an exception was raised by the codec.
 
| PyObject* PyUnicode_DecodeUTF16Stateful( | const char *s,
                                               int size,
                                               const char *errors,
                                               int *byteorder,
                                               int *consumed) |  
 
- 
  Return value:
  New reference.
 If consumed is NULL, behave like
  PyUnicode_DecodeUTF16(). If consumed is not NULL,
  PyUnicode_DecodeUTF16Stateful() will not treat trailing incomplete
  UTF-16 byte sequences (i.e. an odd number of bytes or a split surrogate pair)
  as an error. Those bytes will not be decoded and the number of bytes that
  have been decoded will be stored in consumed.
  
New in version 2.4.
| PyObject* PyUnicode_EncodeUTF16( | const Py_UNICODE *s,
                                               int size,
                                               const char *errors,
                                               int byteorder) |  
 
- 
  Return value:
  New reference.
 Return a Python string object holding the UTF-16 encoded value of
  the Unicode data in s.  If byteorder is not0,
  output is written according to the following byte order:
 
   byteorder == -1: little endian
   byteorder == 0:  native byte order (writes a BOM mark)
   byteorder == 1:  big endian
 
If byteorder is 0, the output string will always start with
  the Unicode BOM mark (U+FEFF). In the other two modes, no BOM mark
  is prepended.
 
If Py_UNICODE_WIDE is defined, a single Py_UNICODE
  value may get represented as a surrogate pair. If it is not
  defined, each Py_UNICODE values is interpreted as an
  UCS-2 character.
 
Return NULL if an exception was raised by the codec.
 
| PyObject* PyUnicode_AsUTF16String( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Return a Python string using the UTF-16 encoding in native byte
  order. The string always starts with a BOM mark.  Error handling is
  ``strict''.  Return NULL if an exception was raised by the
  codec.
These are the ``Unicode Escape'' codec APIs:
| PyObject* PyUnicode_DecodeUnicodeEscape( | const char *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the
  Unicode-Escape encoded string s.  Return NULL if an
  exception was raised by the codec.
| PyObject* PyUnicode_EncodeUnicodeEscape( | const Py_UNICODE *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using
  Unicode-Escape and return a Python string object.  Return NULL
  if an exception was raised by the codec.
| PyObject* PyUnicode_AsUnicodeEscapeString( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using Unicode-Escape and return the
  result as Python string object.  Error handling is ``strict''.
  Return NULL if an exception was raised by the codec.
These are the ``Raw Unicode Escape'' codec APIs:
| PyObject* PyUnicode_DecodeRawUnicodeEscape( | const char *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the
  Raw-Unicode-Escape encoded string s.  Return NULL if an
  exception was raised by the codec.
| PyObject* PyUnicode_EncodeRawUnicodeEscape( | const Py_UNICODE *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using
  Raw-Unicode-Escape and return a Python string object.  Return
  NULL if an exception was raised by the codec.
| PyObject* PyUnicode_AsRawUnicodeEscapeString( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using Raw-Unicode-Escape and return the
  result as Python string object. Error handling is ``strict''.
  Return NULL if an exception was raised by the codec.
These are the Latin-1 codec APIs:
Latin-1 corresponds to the first 256 Unicode ordinals and only these
are accepted by the codecs during encoding.
| PyObject* PyUnicode_DecodeLatin1( | const char *s,
                                                     int size,
                                                     const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the Latin-1
  encoded string s.  Return NULL if an exception was raised
  by the codec.
| PyObject* PyUnicode_EncodeLatin1( | const Py_UNICODE *s,
                                                     int size,
                                                     const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using
  Latin-1 and return a Python string object.  Return NULL if an
  exception was raised by the codec.
| PyObject* PyUnicode_AsLatin1String( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using Latin-1 and return the result as
  Python string object.  Error handling is ``strict''.  Return
  NULL if an exception was raised by the codec.
These are the ASCII codec APIs.  Only 7-bit ASCII data is
accepted. All other codes generate errors.
| PyObject* PyUnicode_DecodeASCII( | const char *s,
                                                    int size,
                                                    const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the
  ASCII encoded string s.  Return NULL if an exception
  was raised by the codec.
| PyObject* PyUnicode_EncodeASCII( | const Py_UNICODE *s,
                                                    int size,
                                                    const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using
  ASCII and return a Python string object.  Return NULL if an
  exception was raised by the codec.
| PyObject* PyUnicode_AsASCIIString( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using ASCII and return the result as
  Python string object.  Error handling is ``strict''.  Return
  NULL if an exception was raised by the codec.
These are the mapping codec APIs:
This codec is special in that it can be used to implement many
different codecs (and this is in fact what was done to obtain most of
the standard codecs included in the encodings package). The
codec uses mapping to encode and decode characters.
Decoding mappings must map single string characters to single Unicode
characters, integers (which are then interpreted as Unicode ordinals)
or None (meaning "undefined mapping" and causing an error).
Encoding mappings must map single Unicode characters to single string
characters, integers (which are then interpreted as Latin-1 ordinals)
or None (meaning "undefined mapping" and causing an error).
The mapping objects provided must only support the __getitem__ mapping
interface.
If a character lookup fails with a LookupError, the character is
copied as-is meaning that its ordinal value will be interpreted as
Unicode or Latin-1 ordinal resp. Because of this, mappings only need
to contain those mappings which map characters to different code
points.
| PyObject* PyUnicode_DecodeCharmap( | const char *s,
                                               int size,
                                               PyObject *mapping,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the encoded
  string s using the given mapping object.  Return
  NULL if an exception was raised by the codec.
| PyObject* PyUnicode_EncodeCharmap( | const Py_UNICODE *s,
                                               int size,
                                               PyObject *mapping,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using the
  given mapping object and return a Python string object.
  Return NULL if an exception was raised by the codec.
| PyObject* PyUnicode_AsCharmapString( | PyObject *unicode,
                                                        PyObject *mapping) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using the given mapping object and
  return the result as Python string object.  Error handling is
  ``strict''.  Return NULL if an exception was raised by the
  codec.
The following codec API is special in that maps Unicode to Unicode.
| PyObject* PyUnicode_TranslateCharmap( | const Py_UNICODE *s,
                                               int size,
                                               PyObject *table,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Translate a Py_UNICODE buffer of the given length by
  applying a character mapping table to it and return the
  resulting Unicode object.  Return NULL when an exception was
  raised by the codec.
The mapping table must map Unicode ordinal integers to Unicode
  ordinal integers or None (causing deletion of the character).
 
Mapping tables need only provide the __getitem__()
  interface; dictionaries and sequences work well.  Unmapped character
  ordinals (ones which cause a LookupError) are left
  untouched and are copied as-is.
 
These are the MBCS codec APIs. They are currently only available on
Windows and use the Win32 MBCS converters to implement the
conversions.  Note that MBCS (or DBCS) is a class of encodings, not
just one.  The target encoding is defined by the user settings on the
machine running the codec.
| PyObject* PyUnicode_DecodeMBCS( | const char *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Create a Unicode object by decoding size bytes of the MBCS
  encoded string s.  Return NULL if an exception was
  raised by the codec.
| PyObject* PyUnicode_EncodeMBCS( | const Py_UNICODE *s,
                                               int size,
                                               const char *errors) |  
 
- 
  Return value:
  New reference.
 Encode the Py_UNICODE buffer of the given size using MBCS
  and return a Python string object.  Return NULL if an exception
  was raised by the codec.
| PyObject* PyUnicode_AsMBCSString( | PyObject *unicode) |  
 
- 
  Return value:
  New reference.
 Encode a Unicode objects using MBCS and return the result as
  Python string object.  Error handling is ``strict''.  Return
  NULL if an exception was raised by the codec.
Release 2.4.4, documentation updated on 18 October 2006.
 
See About this document... for information on suggesting changes.