PCAP-NG timestamp_epoch is shifted by the host timezone, not UTC #361

PCAPNG._read_timestamp promises a UTC epoch and returns a value shifted by a timezone offset. When the capture carries no if_tzone option — which the pcapng draft says should be the normal case — the offset comes from the machine running pcapkit, so the same file parses to different absolute timestamps on different hosts.

file:line

• pcapkit/protocols/misc/pcapng.py:1230-1231 — the shift
• pcapkit/protocols/misc/pcapng.py:1187-1188 — the host-timezone fallback
• pcapkit/protocols/misc/pcapng.py:1218-1221 — the docstring it contradicts
• propagates to pcapkit/toolkit/pcapng.py:217 and :234

Offending expressions

# pcapkit/protocols/misc/pcapng.py:1230-1231
            ts_decimal = timestamp_epoch + decimal.Decimal(
                tzone.utcoffset(None).total_seconds())

# pcapkit/protocols/misc/pcapng.py:1187-1188  (_get_timezone)
        if tzone is None:
            return self._get_local_timezone()

while the docstring at :1218-1221 says the second element is a decimal.Decimal object "since UNIX-Epoch in UTC timezone".

Observable symptom

examples/captures/dhcp.pcapng frame 1. Parsing the raw block bytes directly (little-endian section, IDB carries only if_tsresol=6, no if_tzone and no if_tsoffset, EPB ts_high=256643 ts_low=892570125) gives raw 1102274184317453 / 10**6 = 1102274184.317453 = 2004-12-05T19:16:24.317453Z.

What pcapkit returns:

Two further symptoms:

1. The two return values contradict each other. ts_datetime is built from the unshifted timestamp_epoch at :1235, so frame.info.timestamp.astimezone(timezone.utc) is 2004-12-05T19:16:24.317453+00:00 under all three TZ values — correct — while the returned decimal is the shifted ts_decimal. One call, one correct value and one wrong one.

2. Read→write drifts by the same amount. _make_timestamp (:1283-1285) applies if_tsoffset but not the timezone, although its docstring at :1268-1269 promises "offset and timezone conversion". Under TZ=Asia/Shanghai:

raw ts field in file = 1102274184317453
_make_timestamp(timestamp_epoch) -> 1102302984317453   (drift +28800000000 units = +28800 s)

Also worth noting: _get_local_timezone snapshots today's offset via datetime.now(utc).astimezone().tzinfo, so America/New_York contributes a fixed -04:00 (EDT) to a 2004-12-05 timestamp that was actually in EST — the fallback is not even the host's offset at the capture's own instant.

Plain PCAP is not affected in the epoch, as suspected: pcapkit/protocols/misc/pcap/frame.py:220-224 computes ts_sec + ts_usec/1e6 with no offset, and test.pcap frame 1 gives time_epoch = 1500000000.000509 under all three TZ values. Its time field does vary (2017-07-14T10:40:00.000509 vs 2017-07-13T22:40:00.000509) because :228 calls datetime.fromtimestamp(...) with no tz argument — that is a naive local rendering of the correct instant, not a wrong instant.

Minimal reproduction

Sample captures are not tracked in git; generate them with python examples/generators/make_samples.py first.

import datetime
import os
import struct
import sys
import time

# pcapkit reads its timezone at parse time; set TZ before importing it
os.environ['TZ'] = sys.argv[1]
time.tzset()

import pcapkit

PATH = 'examples/captures/dhcp.pcapng'

# --- independent ground truth, straight out of the file bytes
buf = open(PATH, 'rb').read()
assert buf[8:12] == b'\x4d\x3c\x2b\x1a'          # little-endian section
shb_len = struct.unpack_from('<I', buf, 4)[0]
idb_len = struct.unpack_from('<I', buf, shb_len + 4)[0]
epb = shb_len + idb_len                           # first Enhanced Packet Block
hi, lo = struct.unpack_from('<II', buf, epb + 12)
true_epoch = ((hi << 32) | lo) / 10 ** 6          # if_tsresol=6, no if_tsoffset

blk = pcapkit.extract(fin=PATH, store=True, nofile=True).frame[0].info
print('TZ=%-20s local utcoffset=%s'
      % (sys.argv[1], datetime.datetime.now().astimezone().utcoffset()))
print('  true epoch      = %.6f  (%s)'
      % (true_epoch,
         datetime.datetime.fromtimestamp(true_epoch, datetime.timezone.utc).isoformat()))
print('  timestamp_epoch = %s' % blk.timestamp_epoch)
print('  DELTA           = %+.0f s' % (float(blk.timestamp_epoch) - true_epoch))
print('  timestamp.astimezone(utc) = %s  <- correct, disagrees with the epoch'
      % blk.timestamp.astimezone(datetime.timezone.utc).isoformat())

Output:

$ python repro.py Asia/Shanghai
TZ=Asia/Shanghai        local utcoffset=8:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102302984.317453
  DELTA           = +28800 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

$ python repro.py America/New_York
TZ=America/New_York     local utcoffset=-1 day, 20:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102259784.317453
  DELTA           = -14400 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

$ python repro.py UTC
TZ=UTC                  local utcoffset=0:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102274184.317453
  DELTA           = +0 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

What the spec says

draft-ietf-opsawg-pcapng-02:

• §4.3 (Enhanced Packet Block): the 64-bit timestamp, with if_tsoffset added, "represents the number of units of time that have elapsed since 1970-01-01 00:00:00 UTC". There is no timezone term.
• §4.2 (IDB options, Table 3, type 10) on if_tzone: it "has never been specified in greater detail" and would be "insufficient for converting between UTC and local time"; "Therefore, it SHOULD NOT be used" — "instead, the if_iana_tzname option SHOULD be used if time zone information is to be specified".
• §4.2 (type 14) on if_tsoffset: "This offset is not intended to be used as an offset between local time and UTC."

So the epoch is UTC by definition, if_tzone should not drive it, and deriving it from the host timezone when if_tzone is absent has no basis in the format at all.

What a fix would need to touch

• PCAPNG._read_timestamp (pcapkit/protocols/misc/pcapng.py:1209-1241) — drop the utcoffset term from ts_decimal; keep tzone only as the tzinfo handed to fromtimestamp for the display datetime.
• PCAPNG._get_timezone (:1168-1189) — return datetime.timezone.utc when if_tzone is absent, rather than self._get_local_timezone().
• PCAPNG._make_timestamp (:1259-1286) — either apply the same convention or fix the docstring at :1268-1269, so read and write agree.
• Two existing tests assert the current behaviour and must be updated: tests/protocols/misc/test_pcapng_unit.py:235 (ts_decimal == decimal.Decimal(3612), i.e. 12 + 3600) and :354 (epoch == decimal.Decimal(7200) + decimal.Decimal(2) + decimal.Decimal('0.000000001')).

Observed vs inferred

• Observed by running code: every number in this report — the raw-byte ground truth, all three timestamp_epoch values and their deltas, the astimezone(utc) agreement, the block2frame ts_sec values, the +28800 s _make_timestamp round-trip drift, and the test.pcap comparison.
• Inferred by reading code: behaviour on a capture that actually carries if_tzone — every measurement above exercises the fallback path, since dhcp.pcapng has no such option; that the two named unit tests will need editing under a fix (read, not run); and the suggested fix shape.