Error unconverting Rdr

[rwSTM]

Hi Mark,
The RDR unconvertor seems to have a problem. If NUM_BINS = 0 then REC_LEN is set to zero. I think it ought to be set to two.

Regards
Rob

[marklio]

Nice to hear from you again, Rob. This behavior is based on an admittedly assumptive reading of the STDF spec where 0 is the "assumed" value for missing length fields, and that fields with their default values do not need to be present in the record. (for instance, a zero-length string at the end of a record need not be included in the actual record. I can see why this particular case is interesting since 0 has special semantics (everything is being retested). Is there some specific software that doesn't understand that a zero-length RDR implicitly means NUM_BINS=0? A custom unconverter is a completely reasonable fix for this. It is unfortunate that record registration can't be "overridden", or that there isn't a convention for hand-crafted unconverters.

[rwSTM]

Hi Mark, For the RDR, the problem isn’t that the array following the record is zero length it is that the size is always present (as NUM_BINS) as a ushort following the header but this is not reflected in the record size. I agree that a custom unconvertor is probably the solution. From: Mark Miller <notifications@github.com> Sent: Saturday, October 06, 2018 7:17 PM To: marklio/LinqToStdf <LinqToStdf@noreply.github.com> Cc: Rob WADSWORTH <rob.wadsworth@st.com>; Author <author@noreply.github.com> Subject: Re: [marklio/LinqToStdf] Error unconverting Rdr (#9) Nice to hear from you again, Rob. This behavior is based on an admittedly assumptive reading of the STDF spec where 0 is the "assumed" value for missing length fields, and that fields with their default values do not need to be present in the record. (for instance, a zero-length string at the end of a record need not be included in the actual record. I can see why this particular case is interesting since 0 has special semantics (everything is being retested). Is there some specific software that doesn't understand that a zero-length RDR implicitly means NUM_BINS=0? A custom unconverter is a completely reasonable fix for this. It is unfortunate that record registration can't be "overridden", or that there isn't a convention for hand-crafted unconverters. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUSJK-Mku53BTct-g70IpmsojpKSLvmqks5uiUf2gaJpZM4XLjoc>.

[marklio]

Oh, if the size is written but the record length is zero, that's a bug. I'll try to take some time to see what could be wrong

[marklio]

Rob, do you have a small repro that shows the RDR behavior? I could not reproduce it. After some looking, it shouldn't be possible for the length to mismatch the content. UnknownRecord doesn't keep a length separate from the array of bytes. "unconverting" an RDR with a zero-length array does produce 2 bytes for the count. Given the code, I'm not sure how StdfFileWriter could produce a record that claims a zero-length record but actually has contents:

writer.WriteHeader(new RecordHeader((ushort)ur.Content.Length, ur.RecordType));
_Stream.Write(ur.Content, 0, ur.Content.Length);

I did notice that a "null" array will produce a truly empty record.

[rwSTM]

I’ll see if I can generate a test case. The RDR is injected via a post-processing step in our production facility so I need to track down the person who does that. I can’t send a complete STDF at the moment as the data is company confidential. However, for a little more information here is a record dump using Galaxy Examinator. The first is the dump of the record in the original STDF. ** Retest Data Record (RDR) ** RDR.rec_len = 2 RDR.rec_typ = 1 RDR.rec_sub = 70 RDR.num_bins = 0 RDR.rtst_bin = This is the dump of the record in the STDF after it has been read/written via LinqToStdf ** Retest Data Record (RDR) ** RDR.rec_len = 0 RDR.rec_typ = 1 RDR.rec_sub = 70 ** ERROR: unexpected end of record!! From: Mark Miller <notifications@github.com> Sent: Sunday, October 07, 2018 7:19 PM To: marklio/LinqToStdf <LinqToStdf@noreply.github.com> Cc: Rob WADSWORTH <rob.wadsworth@st.com>; Author <author@noreply.github.com> Subject: Re: [marklio/LinqToStdf] Error unconverting Rdr (#9) Rob, do you have a small repro that shows the RDR behavior? I could not reproduce it. After some looking, it shouldn't be possible for the length to mismatch the content. UnknownRecord doesn't keep a length separate from the array of bytes. "unconverting" an RDR with a zero-length array does produce 2 bytes for the count. Given the code, I'm not sure how StdfFileWriter could produce a record that claims a zero-length record but actually has contents: writer.WriteHeader(new RecordHeader((ushort)ur.Content.Length, ur.RecordType)); _Stream.Write(ur.Content, 0, ur.Content.Length); I did notice that a "null" array will produce a truly empty record. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUSJK-KE5YqR6RI0omoinPJQVrXJzLlPks5uipnygaJpZM4XLjoc>.

[rwSTM]

I did notice that a "null" array will produce a truly empty record.

I suspect that is the problem. NUM_BINS is not an optional field.

[marklio]

removing the MissingValue = ushort.MinValue should fix that, although if the array is getting set to null as part of a round-trip operation, I suspect there's something else going on as well. it does seem interesting that there is a difference between empty and null in the behavior of NUM_BINS.

[rwSTM]

I've just noticed a similar(ish) thing with the SDR Unconvertor. In the spec, all the fields in the SDR records are required. The char arrays are tagged as invalid by having the length byte set to zero. The LinqToStdf Unconvertor doesn't emit fields that are 'missing/invalid'. i.e. Zero length strings do not have a length emitted.

[marklio]

Yeah, it seems we should revisit those markings. A few years back, Paul went through and applied a somewhat liberal interpretation of the spec where the notion of "present in the record" is separate from the notion of missing/invalid. I believe this allowed the most permissive parsing of records, but this doesn't translate well to writing well-formed records. It would be nice to support permissive parsing of records, while enforcing strict conformance for writing.

[marklio]

Rob, are you on the branch with the extra indexing stuff in it?

[rwSTM]

Yes I am. From: Mark Miller <notifications@github.com> Sent: Wednesday, October 10, 2018 6:18 PM To: marklio/LinqToStdf <LinqToStdf@noreply.github.com> Cc: Rob WADSWORTH <rob.wadsworth@st.com>; Author <author@noreply.github.com> Subject: Re: [marklio/LinqToStdf] Error unconverting Rdr (#9) Rob, are you on the branch with the extra indexing stuff in it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUSJKxY_Z_teGTyXI5dFzD9rV6XmNnkBks5ujoApgaJpZM4XLjoc>.

[marklio]

Rob, are you still having trouble here? I've got some motivation to do some work here in the next few months, and I've trying to close some lingering issues.

[rwSTM]

Apologies for the late reply. Yes, this is still technically a problem. I say technically as I have had no need for retest data records so I have simply excluded them when writing STDF records. However, it is something that ought to be addressed. From: Mark Miller ***@***.***> Sent: Thursday, July 15, 2021 8:12 PM To: marklio/LinqToStdf ***@***.***> Cc: Rob WADSWORTH ***@***.***>; Author ***@***.***> Subject: Re: [marklio/LinqToStdf] Error unconverting Rdr (#9) Rob, are you still having trouble here? I've got some motivation to do some work here in the next few months, and I've trying to close some lingering issues. - You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFCISKZMSUUU3GJT7JAC2Y3TX6BPBANCNFSM4FZOHIOA>. ST Restricted