PyNUTClient: decode NUT values and escape authentication arguments

NEWS.adoc

@@ -520,10 +520,16 @@ https://github.com/networkupstools/nut/milestone/13
       the overflows it addressed) in normal device interactions. [#3588]
 
  - NUT client libraries:
+    * `PyNUTClient`: fixed parsing of escaped descriptions and values in
+      server replies, including `GetUPSList()` and its consumers. Shared
+      parsing also handles `nutauth.conf` values, comments and included
+      filenames using NUT syntax. Authentication arguments are escaped
+      before transmission. Existing response types are preserved; single
+      quotes in configuration values are now literal. [issue #3620, PR #3629]
     * The Perl `UPS::Nut` module now decodes quoted values and backslash
       escapes in query/list replies and `nutauth.conf` entries, and quotes
       authentication arguments. Single quotes in configuration are literal.
-      Command descriptions now accept the server's `CMDDESC` response.
+      Command descriptions now accept the server's `CMDDESC` response. [PR #3634]
     * Complete support for actions documented in `docs/net-protocol.txt`
       was implemented in C++, Python and PERL bindings in-tree, and for Java
       in link:https://github.com/networkupstools/jNut[jNut] nearby. Among
@@ -660,6 +666,9 @@ https://github.com/networkupstools/nut/milestone/13
     * Added a `clean_exit()` handler similar to that in `upsmon`. [PR #3499]
 
  - `upsd` data server updates:
+    * Escape variable and command descriptions from `cmdvartab` when sending
+      `DESC` and `CMDDESC` responses, so embedded quotes, backslashes and
+      hashes are decoded correctly by clients. [PR #3629]
     * Validate complete numeric values and conversion ranges for `MAXAGE`,
       `TRACKINGDELAY`, `MAXCONN` and `CERTREQUEST` in `upsd.conf`, retaining
       the previous setting when conversion fails. Preserve public types

UPGRADING.adoc

@@ -31,15 +31,28 @@ command line, in order to quickly pick up any other removed option flags.
 Changes from 2.8.5 to 2.8.6
 ---------------------------
 
+- PLANNED: Keep track of any further API clean-up?
+
+- `PyNUTClient` now decodes NUT escape sequences in server descriptions and
+  values exactly once. Applications which worked around the old behavior by
+  removing escape characters themselves should stop doing so. The existing
+  bytes/string return types are unchanged. [PR #3629]
+
+- Python's `nutauth.conf` reader now follows NUT quoting rules: single
+  quotes are literal characters, not string delimiters. Replace single
+  quotes used for grouping with double quotes, and quote or escape spaces
+  within values. Backslashes introduce literal characters rather than
+  being retained, and unquoted hashes begin comments. Pass unescaped
+  usernames and passwords to `PyNUTClient`; the client handles their wire
+  escaping. [PR #3629]
+
 - The Perl `UPS::Nut` module now decodes NUT quoting and backslash escapes
   in query/list values and `nutauth.conf`, including include paths. Remove
   manual decoding of these returned values and pass literal credentials
   to `Authenticate()`. Single quotes in configuration are now literal, as
   in the C parser; replace single-quoted values with double-quoted values
   where grouping was intended. The legacy `Set()` argument contract is
-  unchanged.
-
-- PLANNED: Keep track of any further API clean-up?
+  unchanged. [PR #3634]
 
 - The common configuration and protocol parser now accepts literal `#`
   characters inside double quotes. An unescaped `#` outside double quotes

docs/man/nutauth.conf.txt

@@ -90,6 +90,13 @@ should be double-quoted and/or use escape sequences, like in other NUT files.
 
 Blank lines and characters after an un-quoted hash (`#`) are ignored.
 
+Single quotes are literal characters; only double quotes group a value
+containing spaces. A backslash escapes the following character once,
+including a space, quote, backslash or hash. It does not introduce C or
+Python escape sequences. These rules also apply to `INCLUDE` filenames.
+For compatibility with older NUT parsers, escape a hash with a backslash
+even inside double quotes.
+
 Example:
 
 	# Global defaults

docs/nut.dict

@@ -1,4 +1,4 @@
-personal_ws-1.1 en 3833 utf-8
+personal_ws-1.1 en 3842 utf-8
 AAC
 AAS
 ABI
@@ -480,11 +480,14 @@ Gathman
 Geerling
 Gembe
 Gert
+GetEnumList
 GetRWVars
+GetRangeList
 GetUPSCommands
 GetUPSList
 GetUPSNames
 GetUPSVars
+GetVariableDescription
 Ghali
 Giese
 Gigabit
@@ -2230,6 +2233,7 @@ filenames
 filesystem
 filesystems
 firewalling
+firmware's
 firmwares
 fixNSS
 fmt
@@ -2516,6 +2520,7 @@ kVA
 kadets
 kaminski
 kde
+kernel's
 kex
 kext
 keychain

scripts/python/module/Makefile.am

@@ -13,7 +13,19 @@
 # (having only shell scripting as a prerequisite suffices for that)
 all: PyNUTClient
 
-check-local:
+check-local: check-pynut-parser check-pynut-disconnect
+
+check-pynut-parser:
+	@if test -n "$(PYTHON_DEFAULT)" && test "$(PYTHON_DEFAULT)" != no ; then \
+		for TEST in test_upslist.py test_protocol.py ; do \
+			PYTHONPATH="$(builddir)" $(PYTHON_DEFAULT) "$(srcdir)/$$TEST" || exit $$? ; \
+		done ; \
+	else \
+		echo "SKIP: PyNUT parser tests require a configured Python interpreter"; \
+	fi
+	@echo "You may want to set up a NUT data server and run 'make tox' here: `pwd`"
+
+check-pynut-disconnect:
 	@if test -n "$(PYTHON_DEFAULT)" && test "$(PYTHON_DEFAULT)" != no; then \
 		$(PYTHON_DEFAULT) "$(builddir)/test_nutclient.py" --disconnect ; \
 	else \
@@ -25,7 +37,7 @@ check-local:
 tox: dist .pypi-tools-tox
 	tox
 
-EXTRA_DIST = tox.ini MANIFEST.in
+EXTRA_DIST = tox.ini MANIFEST.in test_upslist.py test_protocol.py
 
 NUT_SOURCE_GITREV_NUMERIC = @NUT_SOURCE_GITREV_NUMERIC@
 PYTHON_DEFAULT = @PYTHON_DEFAULT@

scripts/python/module/PyNUT.py.in

@@ -75,6 +75,82 @@ import re
 import os
 import sys
 
+
+def _nut_parse(lines):
+    """Yield logical lines of NUT tokens, preserving the input string type.
+
+    Like parseconf, quotes only open at a token boundary, single quotes
+    are literal, and a backslash consumes the next character exactly once.
+    Keep protocol bytes intact; this is not a Unicode or shell decoder.
+    """
+    fields = []
+    word = []
+    started = quoted = escaped = comment = False
+    for line in lines:
+        empty = line[:0]
+        for index in range(len(line)):
+            char = line[index:index + 1]
+            code = ord(char)
+            if comment:
+                if code == 10:
+                    yield fields
+                    fields = []
+                    comment = False
+                continue
+            if escaped:
+                if code != 10:
+                    word.append(char)
+                escaped = False
+                continue
+            if quoted:
+                if code == 34:
+                    fields.append(empty.join(word))
+                    word = []
+                    started = quoted = False
+                elif code == 92:
+                    escaped = True
+                elif code != 10:
+                    word.append(char)
+                continue
+            if code == 92:
+                started = escaped = True
+            elif code == 34 and not started:
+                started = quoted = True
+            elif code in (9, 10, 11, 12, 13, 32, 35, 61):
+                if started:
+                    fields.append(empty.join(word))
+                    word = []
+                    started = False
+                if code == 35:
+                    comment = True
+                elif code == 61:
+                    fields.append(char)
+                elif code == 10:
+                    yield fields
+                    fields = []
+            else:
+                word.append(char)
+                started = True
+    if quoted or escaped:
+        raise ValueError("Incomplete NUT quoted string or escape")
+    if started:
+        fields.append(empty.join(word))
+    if fields:
+        yield fields
+
+
+def _nut_tokens(line):
+    """Parse a single NUT response line or field."""
+    return next(_nut_parse([line]), [])
+
+
+def _nut_quote(value):
+    """Encode one ASCII protocol argument, including for older NUT peers."""
+    if any(ord(char) < 32 or ord(char) == 127 for char in value):
+        raise ValueError("Control character in NUT argument")
+    return '"' + re.sub(r'([\\\\"#])', r'\\\1', value) + '"'
+
+
 ssl_available = False
 try:
     import ssl
@@ -281,16 +357,12 @@ class AuthConf:
         try:
             AuthConf.printDebug( "readAuthConfFile(): Reading NUT AuthConf data from '%s'" % (filename) )
             with open(filename, 'r') as f:
-                for line in f:
-                    line = line.strip()
-                    if not line or line.startswith('#'):
+                for fields in _nut_parse(f):
+                    if not fields:
                         continue
+                    line = fields[0]
 
                     if line.startswith('['):
-                        # Chomp any trailing comments:
-                        if '#' in line:
-                            line = line[:line.index('#')].strip()
-
                         if not line.endswith(']'):
                             raise PyNUTError("Invalid section header in '%s': '%s'" % (filename, line))
 
@@ -321,30 +393,19 @@ class AuthConf:
                         continue
 
                     # INCLUDE support
-                    m = re.match(r'^(INCLUDE(?:_REQUIRED)?)\s+(.*)$', line, re.I)
-                    if m:
-                        inc_type = m.group(1).upper()
-                        inc_file = m.group(2).strip()
-                        if (inc_file.startswith('"') and inc_file.endswith('"')) or \
-                           (inc_file.startswith("'") and inc_file.endswith("'")):
-                            inc_file = inc_file[1:-1]
+                    inc_type = fields[0].upper()
+                    if inc_type in ('INCLUDE', 'INCLUDE_REQUIRED') and len(fields) >= 2:
+                        inc_file = fields[1]
 
                         is_required = (inc_type == "INCLUDE_REQUIRED")
                         AuthConf.printDebug( "readAuthConfFile(): INCLUDE '%s'" % (inc_file) )
                         AuthConf.readAuthConfFile(inc_file, is_required, (current_ac is None or current_ac == AuthConf.__global_defaults))
                         continue
 
-                    if '=' in line:
-                        key, value = line.split('=', 1)
-                        key = key.strip()
-                        keyUC = key.strip().upper()
-                        value = value.strip()
-
-                        # FIXME: NUT parseconf for possibly escaped values is a bit more complicated than this:
-                        # Remove quotes if present
-                        if (value.startswith('"') and value.endswith('"')) or \
-                           (value.startswith("'") and value.endswith("'")):
-                            value = value[1:-1]
+                    if len(fields) >= 2 and fields[1] == '=':
+                        key = fields[0]
+                        keyUC = key.upper()
+                        value = fields[2] if len(fields) >= 3 else ''
 
                         if current_ac is None:
                             if global_scope:
@@ -970,24 +1031,16 @@ if something goes wrong.
                     self.__starttls_retry_budget = self.__starttls_retry_budget_default
 
         if self.__login != None :
-            self.__send( ("USERNAME %s\n" % self.__login).encode('ascii') )
+            self.__send( ("USERNAME %s\n" % _nut_quote(self.__login)).encode('ascii') )
             result = self.__read_until( b"\n" )
             if result[:2] != b"OK" :
                 raise PyNUTError( result.replace( b"\n", b"" ).decode('ascii') )
 
         if self.__password != None :
-            self.__send( ("PASSWORD %s\n" % self.__password).encode('ascii') )
+            self.__send( ("PASSWORD %s\n" % _nut_quote(self.__password)).encode('ascii') )
             result = self.__read_until( b"\n" )
             if result[:2] != b"OK" :
-                if result == b"ERR INVALID-ARGUMENT\n" :
-                    # Quote the password (if it has whitespace etc)
-                    # TODO: Escape special chard like NUT does?
-                    self.__send( ("PASSWORD \"%s\"\n" % self.__password).encode('ascii') )
-                    result = self.__read_until( b"\n" )
-                    if result[:2] != b"OK" :
-                        raise PyNUTError( result.replace( b"\n", b"" ).decode('ascii') )
-                else:
-                    raise PyNUTError( result.replace( b"\n", b"" ).decode('ascii') )
+                raise PyNUTError( result.replace( b"\n", b"" ).decode('ascii') )
 
         # NOTE: no-op if "None"
         self.__tracking = self.SetTrackingMode(self.__tracking_wanted)
@@ -1146,7 +1199,7 @@ if something goes wrong.
         if result[:4] == b"DESC" :
             # DESC <ups> <var> "<description>"
             off = len( ("DESC %s %s " % ( ups, var )).encode('ascii') )
-            return result[off:-1].split(b'"')[1].decode('ascii')
+            return _nut_tokens(result[off:-1])[0].decode('ascii')
         else :
             raise PyNUTError( result.replace( b"\n", b"" ).decode('ascii') )
 
@@ -1167,7 +1220,7 @@ if something goes wrong.
         end_offset = 0 - ( len( ("END LIST ENUM %s %s\n" % ( ups, var )).encode('ascii') ) + 1 )
 
         for current in result[:end_offset].split( b"\n" ) :
-            enum_list.append( current[offset:].split( b'"' )[1].decode('ascii') )
+            enum_list.append( _nut_tokens(current[offset:])[0].decode('ascii') )
 
         return enum_list
 
@@ -1189,8 +1242,8 @@ if something goes wrong.
 
         for current in result[:end_offset].split( b"\n" ) :
             # RANGE <ups> <var> "<min>" "<max>"
-            ranges = current[offset:].split( b'"' )
-            range_list.append( { 'min' : ranges[1].decode('ascii'), 'max' : ranges[3].decode('ascii') } )
+            ranges = _nut_tokens(current[offset:])
+            range_list.append( { 'min' : ranges[0].decode('ascii'), 'max' : ranges[1].decode('ascii') } )
 
         return range_list
 
@@ -1230,7 +1283,11 @@ which is of little concern for Python2 but is important in Python3
 
         for line in result.split( b"\n" ) :
             if line[:3] == b"UPS" :
-                ups, desc = line[4:-1].split( b'"' )
+                fields = re.match( b'([^"]*)("(?:[^"\\\\]|\\\\.)*")$', line[4:] )
+                if fields is None :
+                    raise ValueError( "Invalid UPS list entry" )
+                ups, desc = fields.groups()
+                desc = _nut_tokens(desc)[0]
                 ups_list[ ups.replace( b" ", b"" ) ] = desc
 
         return( ups_list )
@@ -1271,8 +1328,7 @@ available vars.
         end_offset = 0 - ( len( ("END LIST VAR %s\n" % ups).encode('ascii') ) + 1 )
 
         for current in result[:end_offset].split( b"\n" ) :
-            var  = current[ offset: ].split( b'"' )[0].replace( b" ", b"" )
-            data = current[ offset: ].split( b'"' )[1]
+            var, data = _nut_tokens(current[offset:])
             ups_vars[ var ] = data
 
         return( ups_vars )
@@ -1314,7 +1370,7 @@ of the command as value
         end_offset = 0 - ( len( ("END LIST CMD %s\n" % ups).encode('ascii') ) + 1 )
 
         for current in result[:end_offset].split( b"\n" ) :
-            var  = current[ offset: ].split( b'"' )[0].replace( b" ", b"" )
+            var = _nut_tokens(current[offset:])[0]
 
             # For each var we try to get the available description
             try :
@@ -1325,7 +1381,10 @@ of the command as value
                     raise PyNUTError
                 else :
                     off  = len( ("CMDDESC %s %s " % ( ups, command )).encode('ascii') )
-                    desc = temp[off:-1].split(b'"')[1]
+                    if temp[off:off + 1] != b'"':
+                        raise PyNUTError
+                    desc = _nut_tokens(temp[off:-1])[0]
+                    # OLDER: # desc = temp[off:-1].split(b'"')[1]
             except :
                 desc = var
 
@@ -1353,8 +1412,7 @@ The result is presented as a dictionary containing 'key->val' pairs
 
         try :
             for current in result[:end_offset].split( b"\n" ) :
-                var  = current[ offset: ].split( b'"' )[0].replace( b" ", b"" )
-                data = current[ offset: ].split( b'"' )[1]
+                var, data = _nut_tokens(current[offset:])
                 rw_vars[ var ] = data
 
         except :