What’s New In Python 3.6

Release:2.7.6
Date:August 10, 2016

This article explains the new features in Python 3.6, compared to 3.5.

For full details, see the Misc/NEWS file.

Note

Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.6 moves towards release, so it’s worth checking back even after reading earlier versions.

Summary – Release highlights

New syntax features:

Windows improvements:

  • The py.exe launcher, when used interactively, no longer prefers Python 2 over Python 3 when the user doesn’t specify a version (via command line arguments or a config file). Handling of shebang lines remains unchanged - “python” refers to Python 2 in that case.

New Features

PEP 498: Formatted string literals

Formatted string literals are a new kind of string literal, prefixed with 'f'. They are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format() protocol.

>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'

See PEP 498 and the main documentation at Formatted string literals.

PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows setting the Python memory allocators and/or install debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:

  • Newly allocated memory is filled with the byte 0xCB
  • Freed memory is filled with the byte 0xDB
  • Detect violations of Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
  • Detect write before the start of the buffer (buffer underflow)
  • Detect write after the end of the buffer (buffer overflow)
  • Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.

Checking if the GIL is held is also a new feature of Python 3.6.

See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators.

It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. It helps to use external memory debuggers like Valgrind on a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated.

Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):

Debug memory block at address p=0x7fbcd41666f8: API 'o'
    4 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x02 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #1233329 to debug malloc/realloc.
    Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first):
  File "test/test_bytes.py", line 323
  File "unittest/case.py", line 600
  File "unittest/case.py", line 648
  File "unittest/suite.py", line 122
  File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first):
  File "test/test_bytes.py", line 323 in test_hex
  File "unittest/case.py", line 600 in run
  File "unittest/case.py", line 648 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  ...

(Contributed by Victor Stinner in issue 26516 and issue 26564.)

Other Language Changes

  • None yet.

New Modules

  • None yet.

Improved Modules

asyncio

Since the asyncio module is provisional, all changes introduced in Python 3.6 have also been backported to Python 3.5.x.

Notable changes in the asyncio module since Python 3.5.0:

  • The ensure_future() function and all functions that use it, such as loop.run_until_complete(), now accept all kinds of awaitable objects. (Contributed by Yury Selivanov.)
  • New run_coroutine_threadsafe() function to submit coroutines to event loops from other threads. (Contributed by Vincent Michel.)
  • New Transport.is_closing() method to check if the transport is closing or closed. (Contributed by Yury Selivanov.)
  • The loop.create_server() method can now accept a list of hosts. (Contributed by Yann Sionneau.)
  • New loop.create_future() method to create Future objects. This allows alternative event loop implementations, such as uvloop, to provide a faster asyncio.Future implementation. (Contributed by Yury Selivanov.)
  • New loop.get_exception_handler() method to get the current exception handler. (Contributed by Yury Selivanov.)
  • New timeout() context manager to simplify timeouts handling code. (Contributed by Andrew Svetlov.)
  • New StreamReader.readuntil() method to read data from the stream until a separator bytes sequence appears. (Contributed by Mark Korenberg.)
  • The loop.getaddrinfo() method is optimized to avoid calling the system getaddrinfo function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.)

contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for __enter__() which returns self and leaves __exit__() an abstract method. A matching class has been added to the typing module as typing.ContextManager. (Contributed by Brett Cannon in issue 25609.)

datetime

The datetime.strftime() and date.strftime() methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in issue 12006.)

faulthandler

On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable(). (Contributed by Victor Stinner in issue 23848.)

idlelib and IDLE

The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.

‘Modernizing’ includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in issue 24225. Most idlelib patches since have been and will be part of the process.)

In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.

importlib

importlib.util.LazyLoader now calls create_module() on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter and importlib.machinery.ExtensionFileLoader couldn’t be used with importlib.util.LazyLoader.

os

A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir() iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in issue 25994.)

pickle

Objects that need calling __new__ with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in issue 24164.)

readline

Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in issue 26870.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in issue 25011 and issue 25209.)

Names of most attributes listed by dir() are now completed. Previously, names of properties and slots which were not yet created on an instance were excluded. (Contributed by Martin Panter in issue 25590.)

site

When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in issue 26587).

sqlite3

socket

The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in issue 26536.)

socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now support the context manager protocol. (Contributed by Aviv Palivoda in issue 26404.)

The wfile attribute of StreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling write() is now guaranteed to send the data in full. (Contributed by Martin Panter in issue 26721.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or call explicitly the wait() method to read the exit status of the child process (Contributed by Victor Stinner in issue 26741).

telnetlib

Telnet is now a context manager (contributed by Stéphane Wirtel in issue 25485).

tkinter

Added methods trace_add(), trace_remove() and trace_info() in the tkinter.Variable class. They replace old methods trace_variable(), trace(), trace_vdelete() and trace_vinfo() that use obsolete Tcl commands and might not work in future versions of Tcl. (Contributed by Serhiy Storchaka in issue 22115).

typing

The typing.ContextManager class has been added for representing contextlib.AbstractContextManager. (Contributed by Brett Cannon in issue 25609.)

unittest.mock

The Mock class has the following improvements:

urllib.robotparser

RobotFileParser now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in issue 16099.)

warnings

A new optional source parameter has been added to the warnings.warn_explicit() function: the destroyed object which emitted a ResourceWarning. A source attribute has also been added to warnings.WarningMessage (contributed by Victor Stinner in issue 26568 and issue 26567).

When a ResourceWarning warning is logged, the tracemalloc is now used to try to retrieve the traceback where the detroyed object was allocated.

Example with the script example.py:

import warnings

def func():
    return open(__file__)

f = func()
f = None

Output of the command python3.6 -Wd -X tracemalloc=5 example.py:

example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
  f = None
Object allocated at (most recent call first):
  File "example.py", lineno 4
    return open(__file__)
  File "example.py", lineno 6
    f = func()

The “Object allocated at” traceback is new and only displayed if tracemalloc is tracing Python memory allocations and if the warnings was already imported.

winreg

Added the 64-bit integer type REG_QWORD. (Contributed by Clement Rouault in issue 23026.)

zipfile

A new ZipInfo.from_file() class method allows making a ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() method can be used to check if the ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in issue 26039.)

The ZipFile.open() method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in issue 26039.)

zlib

The compress() function now accepts keyword arguments. (Contributed by Aviv Palivoda in issue 26243.)

fileinput

hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in issue 25788.)

Optimizations

  • The ASCII decoder is now up to 60 times as fast for error handlers surrogateescape, ignore and replace (Contributed by Victor Stinner in issue 24870).
  • The ASCII and the Latin1 encoders are now up to 3 times as fast for the error handler surrogateescape (Contributed by Victor Stinner in issue 25227).
  • The UTF-8 encoder is now up to 75 times as fast for error handlers ignore, replace, surrogateescape, surrogatepass (Contributed by Victor Stinner in issue 25267).
  • The UTF-8 decoder is now up to 15 times as fast for error handlers ignore, replace and surrogateescape (Contributed by Victor Stinner in issue 25301).
  • bytes % args is now up to 2 times faster. (Contributed by Victor Stinner in issue 25349).
  • bytearray % args is now between 2.5 and 5 times faster. (Contributed by Victor Stinner in issue 25399).
  • Optimize bytes.fromhex() and bytearray.fromhex(): they are now between 2x and 3.5x faster. (Contributed by Victor Stinner in issue 25401).
  • Optimize bytes.replace(b'', b'.') and bytearray.replace(b'', b'.'): up to 80% faster. (Contributed by Josh Snider in issue 26574).
  • Allocator functions of the PyMem_Malloc() domain (PYMEM_DOMAIN_MEM) now use the pymalloc memory allocator instead of malloc() function of the C library. The pymalloc allocator is optimized for objects smaller or equal to 512 bytes with a short lifetime, and use malloc() for larger memory blocks. (Contributed by Victor Stinner in issue 26249).
  • pickle.load() and pickle.loads() are now up to 10% faster when deserializing many small objects (Contributed by Victor Stinner in issue 27056).

Build and C API Changes

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7.

Deprecated Python modules, functions and methods

Deprecated functions and types of the C API

  • None yet.

Deprecated features

  • The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in issue 25154.)
  • When performing a relative import, falling back on __name__ and __path__ from the calling module when __spec__ or __package__ are not defined now raises an ImportWarning. (Contributed by Rose Ames in issue 25791.)
  • Unlike to other dbm implementations, the dbm.dumb module creates database in 'r' and 'w' modes if it doesn’t exist and allows modifying database in 'r' mode. This behavior is now deprecated and will be removed in 3.8. (Contributed by Serhiy Storchaka in issue 21708.)

Deprecated Python behavior

Removed

API and Feature Removals

  • inspect.getmoduleinfo() was removed (was deprecated since CPython 3.3). inspect.getmodulename() should be used for obtaining the module name for a given path.
  • traceback.Ignore class and traceback.usage, traceback.modname, traceback.fullmodname, traceback.find_lines_from_code, traceback.find_lines, traceback.find_strings, traceback.find_executable_lines methods were removed from the traceback module. They were undocumented methods deprecated since Python 3.2 and equivalent functionality is available from private methods.
  • The tk_menuBar() and tk_bindForTraversal() dummy methods in tkinter widget classes were removed (corresponding Tk commands were obsolete since Tk 4.0).
  • The open() method of the zipfile.ZipFile class no longer supports the 'U' mode (was deprecated since Python 3.4). Use io.TextIOWrapper for reading compressed text files in universal newlines mode.

Porting to Python 3.6

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in ‘python’ Command Behavior

  • The output of a special Python build with defined COUNT_ALLOCS, SHOW_ALLOC_COUNT or SHOW_TRACK_COUNT macros is now off by default. It can be re-enabled using the -X showalloccount option. It now outputs to stderr instead of stdout. (Contributed by Serhiy Storchaka in issue 23034.)

Changes in the Python API

Changes in the C API