diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 31140555532f8f2..ec37d29ef283ad0 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -891,10 +891,12 @@ many numeric contexts, ``False`` and ``True`` behave like the integers 0 and 1, However, relying on this is discouraged; explicitly convert using :func:`int` instead. +.. _iterator-types: + .. _typeiter: -Iterator Types -============== +Iteration-related types +======================= .. index:: single: iterator protocol @@ -907,6 +909,9 @@ using two distinct methods; these are used to allow user-defined classes to support iteration. Sequences, described below in more detail, always support the iteration methods. +Iterables +--------- + One method needs to be defined for container objects to provide :term:`iterable` support: @@ -923,10 +928,14 @@ support: :c:member:`~PyTypeObject.tp_iter` slot of the type structure for Python objects in the Python/C API. +.. _stdtypes-iterators: + +Iterators +--------- + The iterator objects themselves are required to support the following two methods, which together form the :dfn:`iterator protocol`: - .. method:: iterator.__iter__() Return the :term:`iterator` object itself. This is required to allow both @@ -955,16 +964,278 @@ Implementations that do not obey this property are deemed broken. .. _generator-types: -Generator Types +Generator types --------------- -Python's :term:`generator`\s provide a convenient way to implement the iterator -protocol. If a container object's :meth:`~object.__iter__` method is implemented as a -generator, it will automatically return an iterator object (technically, a -generator object) supplying the :meth:`~iterator.__iter__` and :meth:`~generator.__next__` -methods. -More information about generators can be found in :ref:`the documentation for -the yield expression `. +Python's :term:`generators ` -- or more precisely, +:term:`generator functions ` and +:term:`generator iterators ` -- provide a convenient way +to implement the iterator protocol. + +A function that contains one or more :ref:`yield expressions ` +is a :term:`generator function`. +For example:: + + >>> def count_to_three(): + ... yield 0 + ... yield 1 + ... yield 2 + ... yield 3 + +Generator functions behave as regular +:ref:`user-defined functions ` +(for example, they have the same attributes), except that calling a generator +function returns a :ref:`generator iterator `:: + + >>> count_to_three() + + +Iterating a generator iterator executes code of the underlying +generator function, producing each :keyword:`yield`\ed value in turn:: + + >>> for number in count_to_three(): + ... print(number) + 0 + 1 + 2 + 3 + + >>> list(count_to_three()) + [0, 1, 2, 3] + +One common use for generator functions is implementing the +:meth:`~object.__iter__` method of custom iterable objects. +For example:: + + >>> class CardDeck: + ... def __iter__(self): + ... yield 'three of clubs' + ... yield 'ace of hearts' + + >>> list(CardDeck()) + ['three of clubs', 'ace of hearts'] + + +.. index:: pair: object; generator +.. _generator-methods: + +Generator iterators +^^^^^^^^^^^^^^^^^^^ + +Generator iterators implement the +:ref:`iterator protocol `. +Iterating them drives execution of the underlying generator function. + +.. index:: pair: exception; StopIteration + +.. method:: generator.__next__() + + Starts the execution of a generator function or resumes it at the + :ref:`yield expression ` where the function is currently suspended. + When a generator function is resumed with a :meth:`~generator.__next__` + method, the current yield expression always evaluates to :const:`None`. + The execution then continues to the next yield expression, where the + generator is suspended again, and the value of the expression after the + :keyword:`yield` keyword is returned to :meth:`~generator.__next__`'s + caller. + If the generator exits without yielding another value, + :meth:`~generator.__next__` raises a :exc:`StopIteration` exception, + signalling that iteration has completed. + + This method is normally called implicitly, for example by a :keyword:`for` + loop, or by the built-in :func:`next` function. + +Generator iterators have a few more methods than generic iterators, which +can be used to control the execution of the underlying generator function: + +.. method:: generator.send(value) + + "Sends" a value into the generator function: the *value* argument becomes + the result of the current yield expression. + + Otherwise, this method behaves like :meth:`~generator.__next__`: it resumes + the underlying function and either returns the next yielded value or raises + :exc:`StopIteration`. + + When :meth:`send` is called to start the generator, it must be called + with :const:`None` as the argument, because there is no current yield + expression that could receive the value. + + +.. method:: generator.throw(value) + generator.throw(type[, value[, traceback]]) + + Raises an exception at the point where the generator is currently suspended. + + Otherwise, this method behaves like :meth:`~generator.__next__`: it resumes + the underlying function and either returns the next yielded value or raises + :exc:`StopIteration`. + If the generator function does not catch the passed-in exception, or + raises a different exception, then that exception propagates to the caller. + + When :meth:`throw` is called to start the generator, the generator + immediately exits (that is, subsequent calls to :meth:`~generator.__next__` + will raise :exc:`StopIteration`) and the thrown exception is propagated to + :meth:`throw`'s caller. + + In typical use, this is called with a single argument, an exception instance, + similar to the way the :keyword:`raise` keyword is used. + + For backwards compatibility, however, the second signature is + supported, following a convention from older versions of Python. + The *type* argument should be an exception class, and *value* + should be an exception instance. If the *value* is not provided, the + *type* constructor is called to get an instance. If *traceback* + is provided, it is set on the exception, otherwise any existing + :attr:`~BaseException.__traceback__` attribute stored in *value* may + be cleared. + + .. versionchanged:: 3.12 + + The second signature \(type\[, value\[, traceback\]\]\) is deprecated and + may be removed in a future version of Python. + +.. index:: pair: exception; GeneratorExit + +.. method:: generator.close() + + Raises a :exc:`GeneratorExit` exception at the point where the generator + function is currently suspended (equivalent to calling ``throw(GeneratorExit)``). + + If the generator function has already exited (due to an exception or + normal return), or raises :exc:`GeneratorExit` (by not catching the + exception), :meth:`close` returns :const:`None`. + If the generator yields a value, a :exc:`RuntimeError` is raised. + If the generator raises any other exception, it is propagated to the caller. + If a generator returns a value upon being closed, that value is returned + by :meth:`close`. + + When a generator iterator is garbage collected before it has exited, + :meth:`~generator.close` is called automatically. + + .. versionchanged:: 3.13 + + If a generator returns a value upon being closed, the value is returned + by :meth:`close`. + Previously, it returned ``None``. + + +Calling any of the generator methods (:meth:`~generator.__next__`, +:meth:`~generator.send`, :meth:`~generator.throw`, :meth:`~generator.close`) +while one of these methods is already executing +raises a :exc:`ValueError` exception. + + +.. index:: pair: object; asynchronous-generator +.. _asynchronous-generator-methods: + +Asynchronous generator iterators +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This subsection describes the methods of an asynchronous generator iterator, +which are used to control the execution of an asynchronous generator function. + + +.. index:: pair: exception; StopAsyncIteration + +.. method:: agen.__anext__() + :async: + + Returns an :term:`awaitable` which when run starts to execute the + asynchronous generator function or resumes it at the + :ref:`yield expression ` where the function is currently suspended. + When an asynchronous generator function is resumed with an + :meth:`~agen.__anext__` method, the current yield expression always + evaluates to :const:`None` in the returned awaitable, which when run will + continue to the next yield expression. + The value of the expression after the :keyword:`yield` keyword is the value + of the :exc:`StopIteration` exception raised by the completing coroutine. + If the asynchronous generator exits without yielding another value, the + awaitable instead raises a :exc:`StopAsyncIteration` exception, + signalling that the asynchronous iteration has completed. + + This method is normally called implicitly by an :keyword:`async for` loop, + or by the built-in :func:`anext` function. + + +Asynchronous generator-iterators have a few more methods than generic +asynchronous iterators, which can be used to control the execution of +the underlying generator function: + +.. method:: agen.asend(value) + :async: + + Returns an awaitable which, when run, "sends" a value into the underlying + asynchronous generator function: the *value* argument becomes + the result of the current yield expression. + + Otherwise, this method behaves like :meth:`~agen.__anext__`: when the + returned awaitable runs, it resumes the underlying function and either + returns the next yielded value as the value of the raised + :exc:`StopIteration`, or raises :exc:`StopAsyncIteration`. + + When :meth:`asend` is called to start the asynchronous + generator, it must be called with :const:`None` as the argument, + because there is no yield expression that could receive the value. + + +.. method:: agen.athrow(value) + agen.athrow(type[, value[, traceback]]) + :async: + + Returns an awaitable that, when run, raises an exception at the point where + the underlying asynchronous generator function is currently suspended. + + Otherwise, this method behaves like :meth:`~agen.__anext__`: when the + returned awaitable runs, it resumes the underlying function (with an + exception raised) and either returns the next yielded value as the value of + the raised :exc:`StopIteration`, or raises :exc:`StopAsyncIteration`. + If the underlying function does not catch the passed-in exception, or + raises a different exception, then when the awaitable is run, that + exception propagates to the caller of the awaitable. + + When :meth:`~agen.athrow` is called to start the generator, the generator + exits when the awaitable runs (that is, subsequent results from + :meth:`~agen.__anext__` will raise :exc:`StopAsyncIteration` when run) + and the thrown exception is propagated to the awaitable's caller. + + In typical use, this is called with a single argument, an exception instance, + similar to the way the :keyword:`raise` keyword is used. + + For backwards compatibility, however, the second signature is + supported. + An exception instance is created from three arguments in the same way as in + :meth:`generator.throw`. + + .. versionchanged:: 3.12 + + The second signature \(type\[, value\[, traceback\]\]\) is deprecated and + may be removed in a future version of Python. + + +.. index:: pair: exception; GeneratorExit + +.. method:: agen.aclose() + :async: + + Returns an awaitable that when run will throw a :exc:`GeneratorExit` into + the underlying asynchronous generator function at the point where it is + currently suspended (equivalent to calling ``athrow(GeneratorExit)``). + + If the asynchronous generator function then exits gracefully, is already + closed, or raises :exc:`GeneratorExit` (by not catching the exception), + then the returned awaitable will raise a :exc:`StopIteration` exception. + Any further awaitables returned by subsequent calls to the asynchronous + generator will raise a :exc:`StopAsyncIteration` exception. + + If the asynchronous generator yields a value, a :exc:`RuntimeError` is + raised by the awaitable. + If the asynchronous generator raises any other exception, that exception + is propagated to the caller of the awaitable. + + If the asynchronous generator has already exited due to an exception or + normal exit, then further calls to :meth:`aclose` will return an awaitable + that does nothing. .. _typesseq: diff --git a/Doc/library/tomllib.rst b/Doc/library/tomllib.rst index 55610784362eb84..0a6b9d9d9fad4d0 100644 --- a/Doc/library/tomllib.rst +++ b/Doc/library/tomllib.rst @@ -157,3 +157,48 @@ Conversion Table +------------------+--------------------------------------------------------------------------------------+ | array of tables | list of dicts | +------------------+--------------------------------------------------------------------------------------+ + +Limits and interoperability considerations +------------------------------------------ + +:mod:`!tomllib` places some limits on the documents it can handle, +and it preserves details that other TOML parsers are allowed to ignore. +When writing portable TOML files, only use features that are +guaranteed or recommended by the standard. + +The implementation details listed here may change in future versions of Python. + +Tables/dicts + The TOML spec does not guarantee key/value pairs in TOML documents and + tables to be in any specific order. + + .. impl-detail:: + :mod:`!tomllib` loads dictionary entries in the order they appear in + the source. + +Integers + TOML recommends supporting integers in ``range(−2**63, 2**63)``. + + .. impl-detail:: + :mod:`!tomllib` uses :ref:`Python's limit on integer string conversion + ` (4300 digits by default). + +Floats + TOML recommends supporting at least IEEE 754 binary64 values, + which means that numbers with more than 15 significant decimal digits + are likely to be rounded. + + .. impl-detail:: + :mod:`!tomllib` uses Python :class:`float` by default; + on many common platforms this is the recommended binary64. + See :data:`sys.float_info` for details. + +Nesting limit + TOML 1.1.0 does not recommend a limit on how deeply arrays and tables + may be nested inside one another. + (A limit of 100 has been proposed for a future version of TOML.) + + .. impl-detail:: + In :mod:`!tomllib`, the nesting level is mainly limited by Python's + :func:`recursion limit `. + Note that code that calls :mod:`!tomllib` may contribute to the limit. diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst index 52340ffe92eb857..85841d9d1951b83 100644 --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -6,6 +6,11 @@ **Source code:** :source:`Lib/xml/dom/pulldom.py` +.. The module was written by Paul Prescod and added in Python 2.0. + It is not based on any specification: the implementation is the only + reference. The Java Streaming API for XML (StAX, JSR 173) is based on + it, among other pull parsers. + -------------- The :mod:`!xml.dom.pulldom` module provides a "pull parser" which can also be @@ -48,19 +53,49 @@ Example:: doc.expandNode(node) print(node.toxml()) -``event`` is a constant and can be one of: +``event`` is one of the following constants, +and ``node`` is the node which the event is about. +The nodes implement the :mod:`xml.dom` interfaces; +they are created by the DOM implementation given to :class:`PullDOM`, +which is :mod:`xml.dom.minidom` by default. + + +.. data:: START_DOCUMENT + END_DOCUMENT + + The start and the end of the document. + *node* is the :class:`~xml.dom.Document`. + + +.. data:: START_ELEMENT + END_ELEMENT + + The start tag and the end tag of an element. + *node* is the :class:`~xml.dom.Element`. + + +.. data:: CHARACTERS + + Character data. + *node* is the :class:`~xml.dom.Text` node. + -* :data:`START_ELEMENT` -* :data:`END_ELEMENT` -* :data:`COMMENT` -* :data:`START_DOCUMENT` -* :data:`END_DOCUMENT` -* :data:`CHARACTERS` -* :data:`PROCESSING_INSTRUCTION` -* :data:`IGNORABLE_WHITESPACE` +.. data:: IGNORABLE_WHITESPACE -``node`` is an object of type :class:`xml.dom.minidom.Document`, -:class:`xml.dom.minidom.Element` or :class:`xml.dom.minidom.Text`. + White space in element content, as declared in the DTD. + *node* is the :class:`~xml.dom.Text` node. + + +.. data:: COMMENT + + A comment. + *node* is the :class:`~xml.dom.Comment` node. + + +.. data:: PROCESSING_INSTRUCTION + + A processing instruction. + *node* is the :class:`~xml.dom.ProcessingInstruction` node. Since the document is treated as a "flat" stream of events, the document "tree" is implicitly traversed and the desired elements are found regardless of their @@ -74,12 +109,19 @@ and switch to DOM-related processing. .. class:: PullDOM(documentFactory=None) - Subclass of :class:`xml.sax.handler.ContentHandler`. + Subclass of :class:`xml.sax.handler.ContentHandler` which turns SAX events + into the events of the pull parser. + The nodes are created, but they are not added to the tree, + unless :meth:`~DOMEventStream.expandNode` is called. + *documentFactory*, if given, is a DOM implementation used to create + the document; by default the implementation of :mod:`xml.dom.minidom` + is used. .. class:: SAX2DOM(documentFactory=None) - Subclass of :class:`xml.sax.handler.ContentHandler`. + Subclass of :class:`PullDOM` which also adds every created node + to the tree, so that the complete document is built. .. function:: parse(stream_or_string, parser=None, bufsize=None) @@ -95,7 +137,9 @@ If you have XML in a string, you can use the :func:`parseString` function instea .. function:: parseString(string, parser=None) - Return a :class:`DOMEventStream` that represents the (Unicode) *string*. + Return a :class:`DOMEventStream` that represents the *string*. + *string* must be a :class:`str` instance; + to parse bytes, pass a binary file object to :func:`parse`. .. data:: default_bufsize @@ -111,18 +155,21 @@ DOMEventStream Objects .. class:: DOMEventStream(stream, parser, bufsize) + Produce the events for the data read from the file object *stream* + by the :class:`~xml.sax.xmlreader.XMLReader` *parser*. + The data is read by *bufsize* bytes, or characters for a text stream, + at a time. + .. versionchanged:: 3.11 Support for :meth:`~object.__getitem__` method has been removed. .. method:: getEvent() - Return a tuple containing *event* and the current *node* as - :class:`xml.dom.minidom.Document` if event equals :data:`START_DOCUMENT`, - :class:`xml.dom.minidom.Element` if event equals :data:`START_ELEMENT` or - :data:`END_ELEMENT` or :class:`xml.dom.minidom.Text` if event equals - :data:`CHARACTERS`. + Return the next ``(event, node)`` tuple, + or ``None`` at the end of the document. + See above for the events and the corresponding nodes. The current node does not contain information about its children, unless - :func:`expandNode` is called. + :meth:`expandNode` is called. .. method:: expandNode(node) @@ -140,4 +187,13 @@ DOMEventStream Objects # Following statement prints node with all its children '

Some text

and more

' print(node.toxml()) - .. method:: DOMEventStream.reset() + .. method:: reset() + + Discard the events which are not read yet + and prepare the object for parsing a new document. + + + .. method:: clear() + + Release the parser and the document. + The stream is not closed, and the object can no longer be used. diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index 6f5904fdf35e808..b916e5ad9bd782a 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -31,14 +31,6 @@ The Document Object Model is being defined by the W3C in stages, or "levels" in their terminology. The Python mapping of the API is substantially based on the DOM Level 2 recommendation. -.. What if your needs are somewhere between SAX and the DOM? Perhaps - you cannot afford to load the entire tree in memory but you find the - SAX model somewhat cumbersome and low-level. There is also a module - called xml.dom.pulldom that allows you to build trees of only the - parts of a document that you need structured access to. It also has - features that allow you to find your way around the DOM. - See http://www.prescod.net/python/pulldom - DOM applications typically start by parsing some XML into a DOM. How this is accomplished is not covered at all by DOM Level 1, and Level 2 provides only limited improvements: There is a :class:`DOMImplementation` object class which diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index f9ffb07ada88fd3..ec272a076c902bf 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -551,10 +551,9 @@ Functions .. function:: Comment(text=None) Comment element factory. This factory function creates a special element - that will be serialized as an XML comment by the standard serializer. The - comment string can be either a bytestring or a Unicode string. *text* is a - string containing the comment string. Returns an element instance - representing a comment. + that will be serialized as an XML comment by the standard serializer. + *text* is a string containing the comment string. + Returns an element instance representing a comment. Note that :class:`XMLParser` skips over comments in the input instead of creating comment objects for them. An :class:`ElementTree` will @@ -621,10 +620,11 @@ Functions ``"pi"``, ``"start-ns"`` and ``"end-ns"`` (the "ns" events are used to get detailed namespace information). If *events* is omitted, only ``"end"`` events are reported. - *parser* is an optional parser instance. If not given, the standard - :class:`XMLParser` parser is used. *parser* must be a subclass of - :class:`XMLParser` and can only use the default :class:`TreeBuilder` as a - target. Returns an :term:`iterator` providing ``(event, elem)`` pairs; + *parser* is an optional parser instance. + If not given, the standard :class:`XMLParser` parser is used. + *parser* must be an instance of :class:`XMLParser` or its subclass + and can only use the default :class:`TreeBuilder` as a target. + Returns an :term:`iterator` providing ``(event, elem)`` pairs; it has a ``root`` attribute that references the root element of the resulting XML tree once *source* is fully read. The iterator has the :meth:`!close` method that closes the internal @@ -696,8 +696,7 @@ Functions Subelement factory. This function creates an element instance, and appends it to an existing element. - The element name, attribute names, and attribute values can be either - bytestrings or Unicode strings. *parent* is the parent element. *tag* is + *parent* is the parent element. *tag* is the subelement name. *attrib* is an optional dictionary, containing element attributes. *extra* contains additional attributes, given as keyword arguments. Returns an element instance. @@ -895,11 +894,22 @@ Element Objects Element class. This class defines the Element interface, and provides a reference implementation of this interface. - The element name, attribute names, and attribute values can be either - bytestrings or Unicode strings. *tag* is the element name. *attrib* is + *tag* is the element name. *attrib* is an optional dictionary, containing element attributes. *extra* contains additional attributes, given as keyword arguments. + The element name and the attribute names and values are strings or + :class:`QName` instances, and the text and the tail are strings or + ``None``. + The element name can also be :func:`Comment` or + :func:`ProcessingInstruction`, which are used for special elements. + If it is ``None``, the element itself is not serialized: only its text + and its children are written, and its attributes are ignored. + This can be used for a fragment which contains several elements. + With ``method="html"`` the attribute value can also be ``None``, + which produces an empty attribute (such as ``checked``). + Other objects can be stored in the tree, but they cannot be serialized. + .. versionchanged:: 3.15 *attrib* can now be a :class:`frozendict`. @@ -965,14 +975,12 @@ Element Objects .. method:: items() - Returns the element attributes as a sequence of (name, value) pairs. The - attributes are returned in an arbitrary order. + Returns the element attributes as (name, value) pairs. .. method:: keys() - Returns the elements attribute names as a list. The names are returned - in an arbitrary order. + Returns the element attribute names. .. method:: set(key, value) @@ -1313,8 +1321,7 @@ TreeBuilder Objects .. method:: data(data) - Adds text to the current element. *data* is a string. This should be - either a bytestring, or a Unicode string. + Adds text to the current element. *data* is a string. .. method:: end(tag) @@ -1417,7 +1424,8 @@ XMLParser Objects .. method:: feed(data) - Feeds data to the parser. *data* is encoded data. + Feeds data to the parser. *data* is a string + or encoded data (:class:`bytes` or a :term:`bytes-like object`). .. method:: flush() @@ -1496,7 +1504,8 @@ XMLPullParser Objects .. method:: feed(data) - Feed the given bytes data to the parser. + Feed the given data to the parser. *data* is a string + or encoded data (:class:`bytes` or a :term:`bytes-like object`). .. method:: flush() diff --git a/Doc/library/xml.sax.handler.rst b/Doc/library/xml.sax.handler.rst index 4188debf566d9b4..07477f7d660128e 100644 --- a/Doc/library/xml.sax.handler.rst +++ b/Doc/library/xml.sax.handler.rst @@ -72,12 +72,15 @@ for the feature and property names. optionally do not report original prefixed names (default). | access: (parsing) read-only; (not parsing) read/write + The parser based on :mod:`xml.parsers.expat` does not support this feature. + .. data:: feature_string_interning | value: ``"http://xml.org/sax/features/string-interning"`` | true: All element names, prefixes, attribute names, Namespace URIs, and - local names are interned using the built-in intern function. + local names are interned in a dictionary + (see :data:`property_interning_dict`). | false: Names are not necessarily interned, although they may be (default). | access: (parsing) read-only; (not parsing) read/write @@ -90,6 +93,9 @@ for the feature and property names. | false: Do not report validation errors. | access: (parsing) read-only; (not parsing) read/write + The parser based on :mod:`xml.parsers.expat` does not support this feature, + because Expat is a non-validating parser. + .. data:: feature_external_ges @@ -116,6 +122,8 @@ for the feature and property names. DTD subset. | access: (parsing) read-only; (not parsing) read/write + The parser based on :mod:`xml.parsers.expat` does not support this feature. + .. data:: all_features @@ -125,7 +133,7 @@ for the feature and property names. .. data:: property_lexical_handler | value: ``"http://xml.org/sax/properties/lexical-handler"`` - | data type: xml.sax.handler.LexicalHandler (not supported in Python 2) + | data type: :class:`~xml.sax.handler.LexicalHandler` | description: An optional extension handler for lexical events like comments. | access: read/write @@ -134,20 +142,25 @@ for the feature and property names. .. data:: property_declaration_handler | value: ``"http://xml.org/sax/properties/declaration-handler"`` - | data type: xml.sax.sax2lib.DeclHandler (not supported in Python 2) + | data type: an object implementing the SAX2 ``DeclHandler`` interface | description: An optional extension handler for DTD-related events other than notations and unparsed entities. | access: read/write + No parser in the standard library supports this property, + and the standard library provides no such handler. + .. data:: property_dom_node | value: ``"http://xml.org/sax/properties/dom-node"`` - | data type: org.w3c.dom.Node (not supported in Python 2) + | data type: :class:`xml.dom.Node` | description: When parsing, the current DOM node being visited if this is a DOM iterator; when not parsing, the root DOM node for iteration. | access: (parsing) read-only; (not parsing) read/write + No parser in the standard library supports this property. + .. data:: property_xml_string @@ -155,7 +168,28 @@ for the feature and property names. | data type: Bytes | description: The literal string of characters that was the source for the current event. - | access: read-only + | access: read-only, and only during a handler callback + + +.. data:: property_encoding + + | value: ``"http://www.python.org/sax/properties/encoding"`` + | data type: String + | description: The name of the encoding to assume for input data. + | access: read/write + + No parser in the standard library supports this property. + + +.. data:: property_interning_dict + + | value: ``"http://www.python.org/sax/properties/interning-dict"`` + | data type: Dictionary + | description: The dictionary used to intern names, + or ``None`` if names are not interned. + Setting it enables interning, as does the + :data:`feature_string_interning` feature. + | access: read/write .. data:: all_properties diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 1a5ab6a214f819a..ebbdb24364bfb8c 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -216,6 +216,15 @@ Instances of :class:`IncrementalParser` offer the following additional methods: allocated during parsing. +.. method:: IncrementalParser.prepareParser(source) + + Prepare the parser for parsing *source*, an + :class:`InputSource` instance. + It is called by :meth:`~XMLReader.parse` before feeding the data. + The parser implementation must override this method; + the default implementation raises :exc:`NotImplementedError`. + + .. method:: IncrementalParser.reset() This method is called after close has been called to reset the parser so that it diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index 77234cac5d92add..ab6eb061d357505 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -28,10 +28,10 @@ the SAX API. :meth:`~xml.sax.xmlreader.XMLReader.setFeature` on the parser object and argument :data:`~xml.sax.handler.feature_external_ges`. -The convenience functions are: +The convenience functions and data are: -.. function:: make_parser(parser_list=[]) +.. function:: make_parser(parser_list=()) Create and return a SAX :class:`~xml.sax.xmlreader.XMLReader` object. The first parser found will @@ -43,18 +43,23 @@ The convenience functions are: The *parser_list* argument can be any iterable, not just a list. -.. function:: parse(filename_or_stream, handler, error_handler=handler.ErrorHandler()) +.. function:: parse(filename_or_stream, handler, errorHandler=handler.ErrorHandler()) Create a SAX parser and use it to parse a document. The document, passed in as - *filename_or_stream*, can be a filename or a file object. The *handler* + *filename_or_stream*, can be a system identifier (a string identifying the + input source -- typically a file name or a URL), + a :term:`path-like ` object, or a file object. + A system identifier which does not refer to an existing file + is opened with :func:`urllib.request.urlopen`. + The *handler* parameter needs to be a SAX :class:`~handler.ContentHandler` instance. If - *error_handler* is given, it must be a SAX :class:`~handler.ErrorHandler` + *errorHandler* is given, it must be a SAX :class:`~handler.ErrorHandler` instance; if omitted, :exc:`SAXParseException` will be raised on all errors. There is no return value; all work must be done by the *handler* passed in. -.. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) +.. function:: parseString(string, handler, errorHandler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a parameter. *string* must be a :class:`str` instance or a @@ -63,6 +68,15 @@ The convenience functions are: .. versionchanged:: 3.5 Added support of :class:`str` instances. + +.. data:: default_parser_list + + The list of the names of modules which are tried by :func:`make_parser` + after the modules named in its *parser_list* argument. + It contains ``'xml.sax.expatreader'``, or, if the + :envvar:`!PY_SAX_PARSER` environment variable is set and the environment + is not ignored, the comma-separated list of module names taken from it. + A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some piece of code that reads the bytes or characters from the input source, and @@ -135,6 +149,14 @@ classes. class for similar purposes. +.. exception:: SAXReaderNotAvailable(msg, exception=None) + + Subclass of :exc:`SAXNotSupportedException` raised when no parser is + available. A parser module raises it when it is imported or during + parsing if the parser it provides cannot be used, and :func:`make_parser` + raises it if no module from the tried ones provides a usable parser. + + .. seealso:: `SAX: The Simple API for XML `_ diff --git a/Doc/library/xml.sax.utils.rst b/Doc/library/xml.sax.utils.rst index 2de7ae2bda42602..4e58a8fc1c7be48 100644 --- a/Doc/library/xml.sax.utils.rst +++ b/Doc/library/xml.sax.utils.rst @@ -81,6 +81,15 @@ or as base classes. override specific methods to modify the event stream or the configuration requests as they pass through. + .. method:: getParent() + + Return the parent reader, or ``None`` if it is not set. + + + .. method:: setParent(parent) + + Set the parent reader, which the events are read from. + .. function:: prepare_input_source(source, base='') diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index af313f42f9bff6b..2e0b6621eb11c86 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1179,95 +1179,6 @@ on the right hand side of an assignment statement. The proposal that expanded on :pep:`492` by adding generator capabilities to coroutine functions. -.. index:: pair: object; generator -.. _generator-methods: - -Generator-iterator methods -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This subsection describes the methods of a generator iterator. They can -be used to control the execution of a generator function. - -Note that calling any of the generator methods below when the generator -is already executing raises a :exc:`ValueError` exception. - -.. index:: pair: exception; StopIteration - - -.. method:: generator.__next__() - - Starts the execution of a generator function or resumes it at the last - executed yield expression. When a generator function is resumed with a - :meth:`~generator.__next__` method, the current yield expression always - evaluates to :const:`None`. The execution then continues to the next yield - expression, where the generator is suspended again, and the value of the - :token:`~python-grammar:yield_list` is returned to :meth:`__next__`'s - caller. If the generator exits without yielding another value, a - :exc:`StopIteration` exception is raised. - - This method is normally called implicitly, e.g. by a :keyword:`for` loop, or - by the built-in :func:`next` function. - - -.. method:: generator.send(value) - - Resumes the execution and "sends" a value into the generator function. The - *value* argument becomes the result of the current yield expression. The - :meth:`send` method returns the next value yielded by the generator, or - raises :exc:`StopIteration` if the generator exits without yielding another - value. When :meth:`send` is called to start the generator, it must be called - with :const:`None` as the argument, because there is no yield expression that - could receive the value. - - -.. method:: generator.throw(value) - generator.throw(type[, value[, traceback]]) - - Raises an exception at the point where the generator was paused, - and returns the next value yielded by the generator function. If the generator - exits without yielding another value, a :exc:`StopIteration` exception is - raised. If the generator function does not catch the passed-in exception, or - raises a different exception, then that exception propagates to the caller. - - In typical use, this is called with a single exception instance similar to the - way the :keyword:`raise` keyword is used. - - For backwards compatibility, however, the second signature is - supported, following a convention from older versions of Python. - The *type* argument should be an exception class, and *value* - should be an exception instance. If the *value* is not provided, the - *type* constructor is called to get an instance. If *traceback* - is provided, it is set on the exception, otherwise any existing - :attr:`~BaseException.__traceback__` attribute stored in *value* may - be cleared. - - .. versionchanged:: 3.12 - - The second signature \(type\[, value\[, traceback\]\]\) is deprecated and - may be removed in a future version of Python. - -.. index:: pair: exception; GeneratorExit - - -.. method:: generator.close() - - Raises a :exc:`GeneratorExit` exception at the point where the generator - function was paused (equivalent to calling ``throw(GeneratorExit)``). - The exception is raised by the yield expression where the generator was paused. - If the generator function catches the exception and returns a - value, this value is returned from :meth:`close`. If the generator function - is already closed, or raises :exc:`GeneratorExit` (by not catching the - exception), :meth:`close` returns :const:`None`. If the generator yields a - value, a :exc:`RuntimeError` is raised. If the generator raises any other - exception, it is propagated to the caller. If the generator has already - exited due to an exception or normal exit, :meth:`close` returns - :const:`None` and has no other effect. - - .. versionchanged:: 3.13 - - If a generator returns a value upon being closed, the value is returned - by :meth:`close`. - .. index:: single: yield; examples Examples @@ -1367,90 +1278,6 @@ of a *finalizer* method see the implementation of The expression ``yield from `` is a syntax error when used in an asynchronous generator function. -.. index:: pair: object; asynchronous-generator -.. _asynchronous-generator-methods: - -Asynchronous generator-iterator methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This subsection describes the methods of an asynchronous generator iterator, -which are used to control the execution of a generator function. - - -.. index:: pair: exception; StopAsyncIteration - -.. method:: agen.__anext__() - :async: - - Returns an awaitable which when run starts to execute the asynchronous - generator or resumes it at the last executed yield expression. When an - asynchronous generator function is resumed with an :meth:`~agen.__anext__` - method, the current yield expression always evaluates to :const:`None` in the - returned awaitable, which when run will continue to the next yield - expression. The value of the :token:`~python-grammar:yield_list` of the - yield expression is the value of the :exc:`StopIteration` exception raised by - the completing coroutine. If the asynchronous generator exits without - yielding another value, the awaitable instead raises a - :exc:`StopAsyncIteration` exception, signalling that the asynchronous - iteration has completed. - - This method is normally called implicitly by a :keyword:`async for` loop. - - -.. method:: agen.asend(value) - :async: - - Returns an awaitable which when run resumes the execution of the - asynchronous generator. As with the :meth:`~generator.send` method for a - generator, this "sends" a value into the asynchronous generator function, - and the *value* argument becomes the result of the current yield expression. - The awaitable returned by the :meth:`asend` method will return the next - value yielded by the generator as the value of the raised - :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the - asynchronous generator exits without yielding another value. When - :meth:`asend` is called to start the asynchronous - generator, it must be called with :const:`None` as the argument, - because there is no yield expression that could receive the value. - - -.. method:: agen.athrow(value) - agen.athrow(type[, value[, traceback]]) - :async: - - Returns an awaitable that raises an exception of type ``type`` at the point - where the asynchronous generator was paused, and returns the next value - yielded by the generator function as the value of the raised - :exc:`StopIteration` exception. If the asynchronous generator exits - without yielding another value, a :exc:`StopAsyncIteration` exception is - raised by the awaitable. - If the generator function does not catch the passed-in exception, or - raises a different exception, then when the awaitable is run that exception - propagates to the caller of the awaitable. - - .. versionchanged:: 3.12 - - The second signature \(type\[, value\[, traceback\]\]\) is deprecated and - may be removed in a future version of Python. - -.. index:: pair: exception; GeneratorExit - - -.. method:: agen.aclose() - :async: - - Returns an awaitable that when run will throw a :exc:`GeneratorExit` into - the asynchronous generator function at the point where it was paused. - If the asynchronous generator function then exits gracefully, is already - closed, or raises :exc:`GeneratorExit` (by not catching the exception), - then the returned awaitable will raise a :exc:`StopIteration` exception. - Any further awaitables returned by subsequent calls to the asynchronous - generator will raise a :exc:`StopAsyncIteration` exception. If the - asynchronous generator yields a value, a :exc:`RuntimeError` is raised - by the awaitable. If the asynchronous generator raises any other exception, - it is propagated to the caller of the awaitable. If the asynchronous - generator has already exited due to an exception or normal exit, then - further calls to :meth:`aclose` will return an awaitable that does nothing. - .. _primaries: Primaries diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index 4e7ec83723b344c..3ce7c2f22e12b0f 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -23,7 +23,6 @@ Doc/library/urllib.parse.rst Doc/library/urllib.request.rst Doc/library/wsgiref.rst Doc/library/xml.dom.minidom.rst -Doc/library/xml.dom.pulldom.rst Doc/library/xml.sax.reader.rst Doc/library/xml.sax.rst Doc/library/xmlrpc.client.rst diff --git a/Doc/tools/removed-ids.txt b/Doc/tools/removed-ids.txt index 059cbb9a8465496..20bef00eb891cbe 100644 --- a/Doc/tools/removed-ids.txt +++ b/Doc/tools/removed-ids.txt @@ -68,3 +68,17 @@ using/windows.html: return-codes using/windows.html: the-full-installer-deprecated using/windows.html: virtual-environments using/windows.html: windows-full + +# Moved to library/stdtypes: +reference/expressions.html: agen.__anext__ +reference/expressions.html: agen.aclose +reference/expressions.html: agen.asend +reference/expressions.html: agen.athrow +reference/expressions.html: asynchronous-generator-iterator-methods +reference/expressions.html: asynchronous-generator-methods +reference/expressions.html: generator-iterator-methods +reference/expressions.html: generator-methods +reference/expressions.html: generator.__next__ +reference/expressions.html: generator.close +reference/expressions.html: generator.send +reference/expressions.html: generator.throw diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index dfec7a59b02583b..6de2d07bcbb3b38 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -330,9 +330,6 @@ def get_build_info(): # Get most important configure and build options as a list of strings. # Example: ['debug', 'ASAN+MSAN'] or ['release', 'LTO+PGO']. - config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' - cflags = sysconfig.get_config_var('PY_CFLAGS') or '' - cflags += ' ' + (sysconfig.get_config_var('PY_CFLAGS_NODIST') or '') ldflags_nodist = sysconfig.get_config_var('PY_LDFLAGS_NODIST') or '' build = [] @@ -351,18 +348,16 @@ def get_build_info(): free_threading = f"{free_threading} GIL={int(PYTHON_GIL)}" build.append(free_threading) - if hasattr(sys, 'gettotalrefcount'): + if support.Py_DEBUG: # --with-pydebug build.append('debug') - if '-DNDEBUG' in cflags: + if not support.built_with_c_assertions(): build.append('without_assert') else: build.append('release') - if '--with-assertions' in config_args: - build.append('with_assert') - elif '-DNDEBUG' not in cflags: + if support.built_with_c_assertions(): build.append('with_assert') # --enable-experimental-jit diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index ea7edb798051567..90b32aaaaf9a40a 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -611,14 +611,6 @@ def collect_sysconfig(info_add): value = normalize_text(value) info_add('sysconfig[%s]' % name, value) - PY_CFLAGS = sysconfig.get_config_var('PY_CFLAGS') - NDEBUG = (PY_CFLAGS and '-DNDEBUG' in PY_CFLAGS) - if NDEBUG: - text = 'ignore assertions (macro defined)' - else: - text= 'build assertions (macro not defined)' - info_add('build.NDEBUG',text) - for name in ( 'WITH_DOC_STRINGS', 'WITH_DTRACE', @@ -844,6 +836,8 @@ def collect_support(info_add): support.check_sanitizer(memory=True)) info_add('support.check_sanitizer(ub=True)', support.check_sanitizer(ub=True)) + info_add('support.built_with_c_assertions', + support.built_with_c_assertions()) def collect_support_os_helper(info_add): diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 1c28af08988d9f0..28a0ba6c666629b 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -74,7 +74,7 @@ "run_no_yield_async_fn", "run_yielding_async_fn", "async_yield", "reset_code", "on_github_actions", "requires_root_user", "requires_non_root_user", - "skip_if_double_rounding", + "skip_if_double_rounding", "built_with_c_assertions", ] @@ -3526,3 +3526,18 @@ def check_immutable_type(testcase, type): else: flags = type_getflags(type) testcase.assertTrue(flags & Py_TPFLAGS_IMMUTABLETYPE) + + +def built_with_c_assertions(): + """Check if Python was built with C assertions (assert()).""" + + if MS_WINDOWS: + # On Windows, rely on the Py_DEBUG macro to check for assertions + return Py_DEBUG + + # Check if the NDEBUG macro is defined in C compiler flags + PY_CFLAGS = (sysconfig.get_config_var('PY_CFLAGS') or '') + if '-DNDEBUG' in PY_CFLAGS: + return False + + return True diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py index 4c721c01a34f070..10f2a5dfb505e31 100644 --- a/Lib/test/test_gc.py +++ b/Lib/test/test_gc.py @@ -12,7 +12,6 @@ import gc import sys -import sysconfig import textwrap import threading import time @@ -79,13 +78,6 @@ def __init__(self, partner=None): def __tp_del__(self): pass -if sysconfig.get_config_vars().get('PY_CFLAGS', ''): - BUILD_WITH_NDEBUG = ('-DNDEBUG' in sysconfig.get_config_vars()['PY_CFLAGS']) -else: - # Usually, sys.gettotalrefcount() is only present if Python has been - # compiled in debug mode. If it's missing, expect that Python has - # been released in release mode: with NDEBUG defined. - BUILD_WITH_NDEBUG = (not hasattr(sys, 'gettotalrefcount')) ### Tests ############################################################################### @@ -1422,8 +1414,8 @@ def test_collect_garbage(self): @requires_subprocess() - @unittest.skipIf(BUILD_WITH_NDEBUG, - 'built with -NDEBUG') + @unittest.skipIf(not support.built_with_c_assertions(), + 'built without C assertions') def test_refcount_errors(self): self.preclean() # Verify the "handling" of objects with broken refcounts diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py index 3735a6046891ea9..e204bdc7dc672db 100644 --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -642,6 +642,30 @@ def test_toprettyxml_preserves_content_of_text_node(self): dom.getElementsByTagName('B')[0].childNodes[0].toxml(), dom2.getElementsByTagName('B')[0].childNodes[0].toxml()) + def test_isWhitespaceInElementContent(self): + # only " \t\r\n" are whitespace in XML (see XML 1.0, 2.3) + dom = parseString(']>' + ' x\xa0') + children = dom.documentElement.childNodes + self.assertTrue(children[0].isWhitespaceInElementContent) + self.assertFalse(children[2].isWhitespaceInElementContent) + dom.unlink() + + def test_remove_whitespace_in_element_content(self): + from xml.dom.xmlbuilder import DOMBuilder, DOMInputSource + builder = DOMBuilder() + builder.setFeature("whitespace-in-element-content", False) + source = DOMInputSource() + source.byteStream = io.BytesIO( + b']>' + b' x\xc2\xa0') + dom = builder.parse(source) + children = dom.documentElement.childNodes + # ignorable whitespace is removed, other characters are not + self.assertEqual([node.nodeName for node in children], ['b', '#text']) + self.assertEqual(children[1].data, '\xa0') + dom.unlink() + def testProcessingInstruction(self): dom = parseString('') pi = dom.documentElement.firstChild diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index a0337de58b23ae4..f9ff8c4c3541eda 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -845,6 +845,15 @@ def test_indent_space_caching(self): len({id(el.tail) for el in elem.iter()}), ) + def test_indent_non_xml_whitespace(self): + # only " \t\r\n" are whitespace in XML (see XML 1.0, 2.3) + elem = ET.XML('\xa0

text

\xa0') + ET.indent(elem) + self.assertEqual( + ET.tostring(elem), + b' \n

text

 \n' + ) + def test_indent_level(self): elem = ET.XML("

pre
post

text

") with self.assertRaises(ValueError): @@ -4900,6 +4909,11 @@ def test_simple_roundtrip(self): xml = '' self.assertEqual(c14n_roundtrip(xml), xml) + def test_c14n_strip_non_xml_whitespace(self): + # only " \t\r\n" are whitespace in XML (see XML 1.0, 2.3) + self.assertEqual(c14n_roundtrip(" \xa0x\xa0 ", strip_text=True), + "\xa0x\xa0") + def test_c14n_exclusion(self): xml = textwrap.dedent("""\ diff --git a/Lib/xml/dom/expatbuilder.py b/Lib/xml/dom/expatbuilder.py index d56b2ddfdb25698..e3917c1cc682880 100644 --- a/Lib/xml/dom/expatbuilder.py +++ b/Lib/xml/dom/expatbuilder.py @@ -30,7 +30,8 @@ from xml.dom import xmlbuilder, minidom, Node from xml.dom import EMPTY_NAMESPACE, EMPTY_PREFIX, XMLNS_NAMESPACE from xml.parsers import expat -from xml.dom.minidom import _append_child, _set_attribute_node +from xml.dom.minidom import (_append_child, _set_attribute_node, + _XML_WHITESPACE) from xml.dom.NodeFilter import NodeFilter TEXT_NODE = Node.TEXT_NODE @@ -413,7 +414,8 @@ def _handle_white_text_nodes(self, node, info): # whitespace. L = [] for child in node.childNodes: - if child.nodeType == TEXT_NODE and not child.data.strip(): + if (child.nodeType == TEXT_NODE + and not child.data.strip(_XML_WHITESPACE)): L.append(child) # Remove ignorable whitespace from the tree. diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py index 5fd3911bd3c9ebb..7639fa14c5050fb 100644 --- a/Lib/xml/dom/minidom.py +++ b/Lib/xml/dom/minidom.py @@ -31,6 +31,9 @@ _nodeTypes_with_children = (xml.dom.Node.ELEMENT_NODE, xml.dom.Node.ENTITY_REFERENCE_NODE) +# The white space characters of the XML specification (see XML 1.0, 2.3). +_XML_WHITESPACE = " \t\r\n" + class Node(xml.dom.Node): namespaceURI = None # this is non-null only for elements and attributes @@ -1209,7 +1212,7 @@ def replaceWholeText(self, content): return None def _get_isWhitespaceInElementContent(self): - if self.data.strip(): + if self.data.strip(_XML_WHITESPACE): return False elem = _get_containing_element(self) if elem is None: diff --git a/Lib/xml/dom/pulldom.py b/Lib/xml/dom/pulldom.py index 913141cd7ef3ceb..9d33d83458e39ce 100644 --- a/Lib/xml/dom/pulldom.py +++ b/Lib/xml/dom/pulldom.py @@ -1,3 +1,5 @@ +"""Support for building partial DOM trees from SAX events.""" + import xml.sax import xml.sax.handler @@ -11,6 +13,10 @@ CHARACTERS = "CHARACTERS" class PullDOM(xml.sax.ContentHandler): + """Content handler which turns SAX events into pull parser events. + + The nodes are created, but they are not added to the tree.""" + _locator = None document = None @@ -202,6 +208,8 @@ def fatalError(self, exception): raise exception class DOMEventStream: + """Stream of the pull parser events.""" + def __init__(self, stream, parser, bufsize): self.stream = stream self.parser = parser @@ -211,6 +219,7 @@ def __init__(self, stream, parser, bufsize): self.reset() def reset(self): + """Discard unread events and prepare for parsing a new document.""" self.pulldom = PullDOM() # This content handler relies on namespace support self.parser.setFeature(xml.sax.handler.feature_namespaces, 1) @@ -226,6 +235,7 @@ def __iter__(self): return self def expandNode(self, node): + """Expand all children of the node into the node.""" event = self.getEvent() parents = [node] while event: @@ -241,6 +251,7 @@ def expandNode(self, node): event = self.getEvent() def getEvent(self): + """Return the next (event, node) tuple, or None at the end.""" # use IncrementalParser interface, so we get the desired # pull effect if not self.pulldom.firstEvent[1]: @@ -274,13 +285,14 @@ def _emit(self): return rc def clear(self): - """clear(): Explicitly release parsing objects""" + """Release the parser and the document.""" self.pulldom.clear() del self.pulldom self.parser = None self.stream = None class SAX2DOM(PullDOM): + """PullDOM which also adds every created node to the tree.""" def startElementNS(self, name, tagName , attrs): PullDOM.startElementNS(self, name, tagName, attrs) @@ -316,6 +328,7 @@ def characters(self, chars): default_bufsize = (2 ** 14) - 20 def parse(stream_or_string, parser=None, bufsize=None): + """Return a DOMEventStream for the given file name or file object.""" if bufsize is None: bufsize = default_bufsize if isinstance(stream_or_string, str): @@ -327,6 +340,7 @@ def parse(stream_or_string, parser=None, bufsize=None): return DOMEventStream(stream, parser, bufsize) def parseString(string, parser=None): + """Return a DOMEventStream for the given string.""" from io import StringIO bufsize = len(string) diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py index 1cba3eae7ad629d..bed8c27df5a3845 100644 --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -101,6 +101,9 @@ from . import ElementPath +# The white space characters of the XML specification (see XML 1.0, 2.3). +_XML_WHITESPACE = " \t\r\n" + class ParseError(SyntaxError): """An error when parsing an XML document. @@ -129,9 +132,6 @@ class Element: want to check if an element is truly empty, you should check BOTH its length AND its text attribute. - The element tag, attribute names, and attribute values can be either - bytes or strings. - *tag* is the element name. *attrib* is an optional dictionary containing element attributes. *extra* are additional element attributes given as keyword arguments. @@ -359,21 +359,17 @@ def set(self, key, value): self.attrib[key] = value def keys(self): - """Get list of attribute names. + """Get attribute names. - Names are returned in an arbitrary order, just like an ordinary - Python dict. Equivalent to attrib.keys() + Equivalent to attrib.keys() """ return self.attrib.keys() def items(self): - """Get element attributes as a sequence. - - The attributes are returned in arbitrary order. Equivalent to - attrib.items(). + """Get element attributes as (name, value) pairs. - Return a list of (name, value) tuples. + Equivalent to attrib.items(). """ return self.attrib.items() @@ -1209,17 +1205,17 @@ def _indent_children(elem, level): child_indentation = indentations[level] + space indentations.append(child_indentation) - if not elem.text or not elem.text.strip(): + if not elem.text or not elem.text.strip(_XML_WHITESPACE): elem.text = child_indentation for child in elem: if len(child): _indent_children(child, child_level) - if not child.tail or not child.tail.strip(): + if not child.tail or not child.tail.strip(_XML_WHITESPACE): child.tail = child_indentation # Dedent after the last child by overwriting the previous indentation. - if not child.tail.strip(): + if not child.tail.strip(_XML_WHITESPACE): child.tail = indentations[level] _indent_children(tree, 0) @@ -1724,7 +1720,7 @@ def _default(self, text): if prefix == ">": self._doctype = None return - text = text.strip() + text = text.strip(_XML_WHITESPACE) if not text: return self._doctype.append(text) @@ -1940,7 +1936,7 @@ def _flush(self, _join_text=''.join): data = _join_text(self._data) del self._data[:] if self._strip_text and not self._preserve_space[-1]: - data = data.strip() + data = data.strip(_XML_WHITESPACE) if self._pending_start is not None: args, self._pending_start = self._pending_start, None qname_text = data if data and _looks_like_prefix_name(data) else None diff --git a/Lib/xml/sax/__init__.py b/Lib/xml/sax/__init__.py index fe4582c6f8b758c..45f69c6bc223c5f 100644 --- a/Lib/xml/sax/__init__.py +++ b/Lib/xml/sax/__init__.py @@ -27,12 +27,21 @@ def parse(source, handler, errorHandler=ErrorHandler()): + """Parse an XML document with the default parser. + + source is a system identifier, a path-like object or a file object, + handler is a ContentHandler instance, and errorHandler is an + ErrorHandler instance. All work is done by the handler.""" parser = make_parser() parser.setContentHandler(handler) parser.setErrorHandler(errorHandler) parser.parse(source) def parseString(string, handler, errorHandler=ErrorHandler()): + """Parse an XML document from a string with the default parser. + + string is a str or a bytes-like object, the other arguments are the + same as for parse().""" import io if errorHandler is None: errorHandler = ErrorHandler() diff --git a/Lib/xml/sax/saxutils.py b/Lib/xml/sax/saxutils.py index c1612ea1cebc5d0..77ab9ee2faf0388 100644 --- a/Lib/xml/sax/saxutils.py +++ b/Lib/xml/sax/saxutils.py @@ -110,6 +110,7 @@ def __getattr__(self, name): write_through=True) class XMLGenerator(handler.ContentHandler): + """Content handler which writes the events back as an XML document.""" def __init__(self, out=None, encoding="iso-8859-1", short_empty_elements=False): handler.ContentHandler.__init__(self) diff --git a/Lib/xml/sax/xmlreader.py b/Lib/xml/sax/xmlreader.py index e906121d23b9ef3..98e93113fbb5ab9 100644 --- a/Lib/xml/sax/xmlreader.py +++ b/Lib/xml/sax/xmlreader.py @@ -89,7 +89,7 @@ def setProperty(self, name, value): raise SAXNotRecognizedException("Property '%s' not recognized" % name) class IncrementalParser(XMLReader): - """This interface adds three extra methods to the XMLReader + """This interface adds four extra methods to the XMLReader interface that allow XML parsers to support incremental parsing. Support for this interface is optional, since not all underlying XML parsers support this functionality. @@ -104,7 +104,7 @@ class IncrementalParser(XMLReader): is, after parse has been called and before it returns. By default, the class also implements the parse method of the XMLReader - interface using the feed, close and reset methods of the + interface using the prepareParser, feed and close methods of the IncrementalParser interface as a convenience to SAX 2.0 driver writers.""" @@ -274,6 +274,7 @@ def getCharacterStream(self): # ===== ATTRIBUTESIMPL ===== class AttributesImpl: + """Implementation of the Attributes interface.""" def __init__(self, attrs): """Non-NS-aware implementation. diff --git a/Makefile.pre.in b/Makefile.pre.in index b2bd89039e12303..78a486623181fa8 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -3441,7 +3441,7 @@ MODULE__CTYPES_DEPS=$(srcdir)/Modules/_ctypes/ctypes.h MODULE__CTYPES_TEST_DEPS=$(srcdir)/Modules/_ctypes/_ctypes_test_generated.c.h MODULE__CTYPES_MALLOC_CLOSURE=@MODULE__CTYPES_MALLOC_CLOSURE@ MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/pyexpat.c @LIBEXPAT_INTERNAL@ -MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h $(srcdir)/Modules/_openssl_mem.h MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h MODULE__REMOTE_DEBUGGING_DEPS=$(srcdir)/Modules/_remote_debugging/_remote_debugging.h $(srcdir)/Modules/_remote_debugging/gc_stats.h @@ -3460,7 +3460,7 @@ MODULE__HMAC_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_HMAC_HEADERS) $(LIBHACL_ MODULE__HMAC_LDEPS=$(LIBHACL_HMAC_LIB_@LIBHACL_LDEPS_LIBTYPE@) MODULE__SOCKET_DEPS=$(srcdir)/Modules/socketmodule.h $(srcdir)/Modules/addrinfo.h $(srcdir)/Modules/getaddrinfo.c $(srcdir)/Modules/getnameinfo.c -MODULE__SSL_DEPS=$(srcdir)/Modules/_ssl.h $(srcdir)/Modules/_ssl/cert.c $(srcdir)/Modules/_ssl/debughelpers.c $(srcdir)/Modules/_ssl/misc.c $(srcdir)/Modules/_ssl_data_111.h $(srcdir)/Modules/_ssl_data_300.h $(srcdir)/Modules/socketmodule.h +MODULE__SSL_DEPS=$(srcdir)/Modules/_ssl.h $(srcdir)/Modules/_openssl_mem.h $(srcdir)/Modules/_ssl/cert.c $(srcdir)/Modules/_ssl/debughelpers.c $(srcdir)/Modules/_ssl/misc.c $(srcdir)/Modules/_ssl_data_111.h $(srcdir)/Modules/_ssl_data_300.h $(srcdir)/Modules/socketmodule.h MODULE__TESTCAPI_DEPS=$(srcdir)/Modules/_testcapi/parts.h $(srcdir)/Modules/_testcapi/util.h MODULE__TESTLIMITEDCAPI_DEPS=$(srcdir)/Modules/_testlimitedcapi/testcapi_long.h $(srcdir)/Modules/_testlimitedcapi/parts.h $(srcdir)/Modules/_testlimitedcapi/util.h MODULE__TESTINTERNALCAPI_DEPS=$(srcdir)/Modules/_testinternalcapi/parts.h $(srcdir)/Parser/tokenizer/cursor.h $(srcdir)/Parser/tokenizer/source.h $(srcdir)/Python/ceval.h $(srcdir)/Modules/_testinternalcapi/test_targets.h $(srcdir)/Modules/_testinternalcapi/test_cases.c.h diff --git a/Misc/NEWS.d/next/Library/2026-08-22-10-30-00.gh-issue-156228.k3Rmxw.rst b/Misc/NEWS.d/next/Library/2026-08-22-10-30-00.gh-issue-156228.k3Rmxw.rst new file mode 100644 index 000000000000000..536bcd44c921116 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-08-22-10-30-00.gh-issue-156228.k3Rmxw.rst @@ -0,0 +1,4 @@ +The :mod:`ssl` and :mod:`hashlib` modules now route OpenSSL memory +allocations through the Python raw memory allocators, making OpenSSL memory +usage visible to :mod:`tracemalloc` and to custom allocators installed with +:c:func:`PyMem_SetAllocator`. diff --git a/Misc/NEWS.d/next/Library/2026-08-30-19-30-00.gh-issue-156658.Xq5Nt7.rst b/Misc/NEWS.d/next/Library/2026-08-30-19-30-00.gh-issue-156658.Xq5Nt7.rst new file mode 100644 index 000000000000000..3f2f105116c1838 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2026-08-30-19-30-00.gh-issue-156658.Xq5Nt7.rst @@ -0,0 +1,5 @@ +:mod:`xml.dom` and :mod:`xml.etree.ElementTree` no longer treat characters +which are not white space in XML (such as U+00A0) as white space. Previously +they could be lost in :func:`~xml.etree.ElementTree.indent`, +:func:`~xml.etree.ElementTree.canonicalize` with ``strip_text=True``, and when +parsing with the ``whitespace-in-element-content`` feature turned off. diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c index fa178f53e9b3ffe..18bbbb618c2b18b 100644 --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -3165,7 +3165,6 @@ makeuniversal(XMLParserObject* self, const char* string) necessary */ PyObject* tag; - char* p; Py_ssize_t i; /* look for namespace separator */ @@ -3174,22 +3173,28 @@ makeuniversal(XMLParserObject* self, const char* string) break; if (i != size) { /* convert to universal name */ - tag = PyBytes_FromStringAndSize(NULL, size+1); - if (tag == NULL) { + PyBytesWriter *writer = PyBytesWriter_Create(1 + size); + if (writer == NULL) { Py_DECREF(key); return NULL; } - p = PyBytes_AS_STRING(tag); + char *p = PyBytesWriter_GetData(writer); p[0] = '{'; memcpy(p+1, string, size); size++; + + tag = PyBytesWriter_Finish(writer); + if (tag == NULL) { + Py_DECREF(key); + return NULL; + } } else { /* plain name; use key as tag */ tag = Py_NewRef(key); } /* decode universal name */ - p = PyBytes_AS_STRING(tag); + const char *p = PyBytes_AS_STRING(tag); value = PyUnicode_DecodeUTF8(p, size, "strict"); Py_DECREF(tag); if (!value) { diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c index f895c9037485c43..d23767afeb96cfb 100644 --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -27,6 +27,7 @@ #include "pycore_strhex.h" // _Py_strhex() #include "pycore_pyatomic_ft_wrappers.h" // FT_ATOMIC_LOAD_PTR_RELAXED #include "hashlib.h" +#include "_openssl_mem.h" /* EVP is the preferred interface to hashing in OpenSSL */ #include @@ -2933,5 +2934,6 @@ static struct PyModuleDef _hashlibmodule = { PyMODINIT_FUNC PyInit__hashlib(void) { + _PyOpenSSL_SetupMemFunctions(); return PyModuleDef_Init(&_hashlibmodule); } diff --git a/Modules/_openssl_mem.h b/Modules/_openssl_mem.h new file mode 100644 index 000000000000000..5284d9b5e63361a --- /dev/null +++ b/Modules/_openssl_mem.h @@ -0,0 +1,56 @@ +// Route OpenSSL allocations through the raw memory allocators. +// Shared by the _ssl and _hashlib modules. + +#ifndef Py_OPENSSL_MEM_H +#define Py_OPENSSL_MEM_H + +#include "Python.h" + +#include // CRYPTO_set_mem_functions() + +// LibreSSL stubs out CRYPTO_set_mem_functions() and BoringSSL lacks it. +// AWS-LC has it, but unlike OpenSSL it does not refuse to install hooks +// after the first allocation, so earlier size-prefixed allocations would +// be freed with the wrong allocator. +#if !defined(LIBRESSL_VERSION_NUMBER) && !defined(OPENSSL_IS_BORINGSSL) \ + && !defined(OPENSSL_IS_AWSLC) +# define _Py_OPENSSL_CAN_SET_MEM_FUNCTIONS +#endif + +#ifdef _Py_OPENSSL_CAN_SET_MEM_FUNCTIONS + +static void * +_PyOpenSSL_Malloc(size_t size, const char *Py_UNUSED(file), + int Py_UNUSED(line)) +{ + return PyMem_RawMalloc(size); +} + +static void * +_PyOpenSSL_Realloc(void *ptr, size_t size, const char *Py_UNUSED(file), + int Py_UNUSED(line)) +{ + return PyMem_RawRealloc(ptr, size); +} + +static void +_PyOpenSSL_Free(void *ptr, const char *Py_UNUSED(file), + int Py_UNUSED(line)) +{ + PyMem_RawFree(ptr); +} + +#endif // _Py_OPENSSL_CAN_SET_MEM_FUNCTIONS + +static void +_PyOpenSSL_SetupMemFunctions(void) +{ +#ifdef _Py_OPENSSL_CAN_SET_MEM_FUNCTIONS + // Fails if OpenSSL has already allocated memory (e.g. another + // libcrypto user in the process); it then keeps its current allocator. + (void)CRYPTO_set_mem_functions(_PyOpenSSL_Malloc, _PyOpenSSL_Realloc, + _PyOpenSSL_Free); +#endif +} + +#endif // !Py_OPENSSL_MEM_H diff --git a/Modules/_ssl.c b/Modules/_ssl.c index aadc015405453f3..73b32c1d86c72ec 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -41,6 +41,7 @@ #endif #include "_ssl.h" +#include "_openssl_mem.h" /* Redefined below for Windows debug builds after important #includes */ #define _PySSL_FIX_ERRNO @@ -7447,5 +7448,6 @@ static struct PyModuleDef _sslmodule_def = { PyMODINIT_FUNC PyInit__ssl(void) { + _PyOpenSSL_SetupMemFunctions(); return PyModuleDef_Init(&_sslmodule_def); } diff --git a/PCbuild/_hashlib.vcxproj b/PCbuild/_hashlib.vcxproj index 2cd205224bc0891..e0110f32f201f16 100644 --- a/PCbuild/_hashlib.vcxproj +++ b/PCbuild/_hashlib.vcxproj @@ -97,6 +97,9 @@ ws2_32.lib;%(AdditionalDependencies) + + + diff --git a/PCbuild/_hashlib.vcxproj.filters b/PCbuild/_hashlib.vcxproj.filters index 7a0700c007f6442..d26954116bb5edc 100644 --- a/PCbuild/_hashlib.vcxproj.filters +++ b/PCbuild/_hashlib.vcxproj.filters @@ -4,6 +4,9 @@ {cc45963d-bd25-4eb8-bdba-a5507090bca4} + + {5abcdd3e-a8bc-4833-949c-9477609092b9} + {67630fa4-76e4-4035-bced-043a6df1e2e0} @@ -13,6 +16,11 @@ Source Files + + + Header Files + + Resource Files diff --git a/PCbuild/_ssl.vcxproj b/PCbuild/_ssl.vcxproj index ce21f992ff8510e..127b76fe11e3290 100644 --- a/PCbuild/_ssl.vcxproj +++ b/PCbuild/_ssl.vcxproj @@ -97,6 +97,9 @@ ws2_32.lib;crypt32.lib;%(AdditionalDependencies) + + + diff --git a/PCbuild/_ssl.vcxproj.filters b/PCbuild/_ssl.vcxproj.filters index 8aef9e03fcc429a..e96da0580e01b44 100644 --- a/PCbuild/_ssl.vcxproj.filters +++ b/PCbuild/_ssl.vcxproj.filters @@ -4,6 +4,9 @@ {695348f7-e9f6-4fe1-bc03-5f08ffc8095b} + + {7c1bd5da-8912-4107-b6ac-f3930f2d90c7} + {1b18a2e6-040d-46c7-a9ac-ac2ec64fb5d6} @@ -13,6 +16,11 @@ Source Files + + + Header Files + + Resource Files diff --git a/configure b/configure index e0ad04b036aeb13..a2ca40de07d54ec 100755 --- a/configure +++ b/configure @@ -16745,12 +16745,14 @@ else case e in #( e) cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ +#include int main(void) { _Float16 val = 1.0f16; + int test = isinf(val) || isnan(val); /* basic support from libm */ double d = 3.14; val = d; - return 0; + return test; } _ACEOF diff --git a/configure.ac b/configure.ac index 2b8a5b052878c91..e1333230fefa688 100644 --- a/configure.ac +++ b/configure.ac @@ -4474,12 +4474,14 @@ AC_CACHE_CHECK([for _Float16 support], [ac_cv_float16_supported], WITH_SAVE_ENV([ CFLAGS="$CFLAGS -O0" AC_RUN_IFELSE([AC_LANG_SOURCE([[ +#include int main(void) { _Float16 val = 1.0f16; + int test = isinf(val) || isnan(val); /* basic support from libm */ double d = 3.14; val = d; - return 0; + return test; } ]])], [ac_cv_float16_supported=yes], [ac_cv_float16_supported=no],