availability

See also https://github.com/hapi-server/data-specification/wiki/file-listings

Recommended dataset naming is to end it with "DATASET/filelist", e.g., "MyDataset/filelist" (not "filelist/MyDataset"), and also add "x_parameterSchema="filelist" in the info response.

We start with a base class which is just a "Listing of Times":

• time isotime

"Availability" required elements:

• startDate - start date/time of coverage (isotime, required)
• stopDate - (isotime) stop date/time of coverage; exclusive (like HAPI request end times)

Optional and recommended elements:

• fractionalCoverage (double) Defined as fraction (0.0 to 1.0) of interval which has useful data. This is ambiguous, so here are some examples for several scenarios. In the description for this parameter, please explain how this value was computed.

Scenario 1: For data with a uniform cadence (such as a 1 hour averages), the fractionalCoverage would represent the number of possible records (out of the 24 per day) where at least one parameter of significance has a non fill value. (Key point: availability is for an entire dataset, which has many parameters, each of which may be independently present or absent.) As an example, if you have ground magnetometer data with support parameters (geomag index, station location, ambient temperature, etc) in addition to mag values themselves, then the fractionalCoverage should apply to the mag values since reporting availability for just the auxiliary parameters is not what most users are expecting. If any of the main values of interest (mag values) are present, this should count towards the availability fraction. If all of the key values are absent or fill, this would not count towards the availabilityFraction.

Scenario 2: For data that alternates between a slow cadence survey mode and high cadence burst mode, or for data that is collected occasionally / episodically, a set of daily (or fixed duration) availability fractions can be tricky. One suggestion is to have multiple cadences of availability information where users can get a large scale overview, or zoom in. Another option is to use non-uniform start / stop times for the availability ranges (i.e., decouple the availability records from the file cadence and match it more to the episodes of actual data collection).

Scenario 3: If you want to compute a simple availability estimate for a file-based dataset with daily files, a simple way to get rough availability would be to tally which days have a file present, and just use that for a 0 or 1 value of availability for each day. If there is a possibility that there are files containing entirely fill values, this might not be a great way to indicate availability, since it would report availability for days where there was actually no data. It would be better to scan through each file once and keep track of what fraction of records contain non-fill for values of interest, and then using the duration of each record to determine the fraction of each day that has actual content.

Because of the different ways to interpret availability, it is very important to explain the precise meaning of the values provided.

We need a few examples:

1. Burst mode, start/stop bounds when instrument on. (fractionalCoverage always 100%)
2. start/stop always fixed length. (fractionalCoverage is what provider wants to tell user about how good interval is)

For future discussion

• numberValidRecords (int) count of the records in the interval where there is at least one non-FILL value in the record; this is easy an unambiguous to compute on the server side; the interpretation by clients is harder, esp. for non-uniform cadence (clients want to know - is this interval good or not)
• numberOfRecords (int) - the number of records (valid data or FILL values) that are present within the startDate and stopDate
• estimatedFractionOfIntervalFilledWithValidRecords - easy to use for clients, but hard to know how to uniformly implement on servers
• version (str)
• PARAMETER_NAME/numberValidRecords ???