Tokenizer manipulates leading and padding zeros on decimal numbers

[beheybird]

In Applesoft you can drop the leading zero or add zeros to a decimal number and it will store this as typed. It does not affect the value or operations on that variable by math functions. The output will be the same but the source listing is modified by bastools.

Why would anyone care? Typing in Applesoft programs from a magazine such as Nibble and verifying with a checksum tool such as KeyPerfect will give incorrect values. If possible, it would be nice if what you type is what you get in the conversion tool. I already hit this on the second program I tried because there is no fixed rule for entering numbers in Applesoft.

Example:

APPLESOFT (VALID CODE, SHOWN USING LIST)
10 PRINT "MATHING"
30 A = .4
40 B = 0.6000
50 C = -.250
60 D = -0.70
70 PRINT "A=";A
80 PRINT "B=";B
90 PRINT "C=";C
95 PRINT "D=";D

TOKENIZER
10 PRINT "MATHING"
30 A = 0.4
40 B = 0.6
50 C = - 0.25
60 D = - 0.7
70 PRINT "A=";A
80 PRINT "B=";B
90 PRINT "C=";C
95 PRINT "D=";D

SAME OUTPUT:
A=.4
B=.6
C=-.25
D=-.7

[a2geek]

The general premise was to give a "better" or "more modern" experience for writing Applesoft, not necessarily to replicate the Applesoft experience. So there are differences.

Apparently, this is one. I looked at the underlying Java StreamTokenizer and I don't see a way to capture the original form of the number. I just get a number. (Which, I think, based on your tokenizer output, could potentially be optimized to remove that unneeded 0 digit.) But since I don't see the original input, I can't replicate it.

[beheybird]

I figured it was not trivial and I get it. I'm aware now what to look for. What was throwing me for a loop. You can also do C = 0000.6000. Absurd but applesoft doesn't care, it's still just evaluated as .6.

For example verifying the non-existent problem
100 CI = 11 * I1 + .4 / I8 - 16 * .50
100 CI = 11 * I1 + 0.4 / I8 - 16 * 0.5

Looking at a not good pdf scan and entering the code. Is that 11, or I1, 18, or I8 ? You need to understand the program variables and what is going on. It compiles I push it into a dsk and then run keyperfect and it fails and I stare at what I typed in my editor vs the pdf. I had to actually load the applesoft and go line by line in the checksum block and then see what was going on.

The program runs but when the checksum is off, is it really working. The only way is to double check the failed sections.

It's a nice ask but I understand.

[a2geek]

I understand where you're coming from. I've been picking up books with code in them for when I just feel like typing in a program!

For now, let's leave this. I'm kind of curious what it would take to write a parser/tokenizer that is configurable for these differing scenarios. And I'd also like to do some work to bump the Java version (etc), and this wold be a legit capability improvement.

[a2geek]

Hey @beheybird -- I have added a tokenizer that allows for the number preservation. The new switch is --preserve to select that tokenizer configuration.

Sample: (using the first set of source)

$ $ bt --preserve --list ticket-49.bas 
10  PRINT "MATHING"
30 A = .4
40 B = 0.6000
50 C = -.250
60 D = -0.70
70  PRINT "A=";A
80  PRINT "B=";B
90  PRINT "C=";C
95  PRINT "D=";D

The default is --modern (which doesn't support number preservation):

$ bt --list ticket-49.bas 
10  PRINT "MATHING"
30 A = 0.4
40 B = 0.6
50 C =  - 0.25
60 D =  - 0.7
70  PRINT "A=";A
80  PRINT "B=";B
90  PRINT "C=";C
95  PRINT "D=";D

Hoping you have some time to run it through the paces? It's a hand-coded tokenizer, and I'm not spotting anything, but I also don't have a large set of source to try it against.

Also of note, the new --classic tokenizer (the base for --preserve) will take the DATA statement exactly as-is as well as the REM. So hopefully that helps as well.

[a2geek]

The README has been updated with the help page, just in case that's useful. I'll try to keep it up to date instead of pasting multiple copies in here. 😉

https://github.com/AppleCommander/bastools/blob/feature/new-old-tokenizer/tools/bt/README.md

[a2geek]

Another update. I realized that there are non-alphabetic tokens (=, +, -, *, /, <, >, etc) that needed to be parsed.

Also, all of this is now Java 21. Sorry if that complicates things!

bastools-tools-bt-5.0-SNAPSHOT.zip

[a2geek]

I happened to come across an old disk of mine that had Nibble Checkit code, so tried adding it. It's not perfect (if the source has multiple colon separating statements, I think that gets lost, but... it might be useful.

$ bt --checkit checkit-example.bas 
Nibble Checkit, Copyright 1988, Microsparc Inc.
37|10 REM RING THE BELL
54|20  FOR J = 1 TO 5: PRINT  CHR$ (7): NEXT J
91|30  END 
TOTAL: 1CB9

Curious if you have Key Perfect somewhere? I'm poking around and found that a list (partial, maybe) is here: https://github.com/22samurai/computer-magazine-proofreaders
... so this site has code for Compute! Magazine's proofreader, and I found a copy of "Apple Checker 3.0" (but I can't figure out how to use it). I just haven't found Key Perfect yet.

[beheybird]

Thanks for all of this. I too went looking for the Key Perfect source or algorithm as I thought they must have published it but now it may have just been the earlier Apple Checker source. MicroSparc sold Key Perfect so unless someone disassembles it, it's a black box. Apple Checker 3 source code was published in Nibble Express Volume III.

How it's used is you
BRUN APPLE CHECKER 3.0
BLOAD / LOAD (APPLESOFT) YOUR PROGRAM
CALL 25
THEN FP TO CLEAR THINGS AND START OVER

ex.
BRUN APPLE CHECKER 3.0
LOAD OTHELLO
CALL 25

CHECK CODE 3.0

ON: OTHELLO
TYPE: A

LENGTH: 0D89
CHECKSUM: 2E

[a2geek]

Thanks! I did find a copy of Apple Checker (somebody actually had a PDF of Nibble Express III), so I've added that as well.

I did find some disk images with Key Perfect (hypothetically versions 2.0, 4.0, and 5.0). I spent most of the day figuring out SourceGen and working with the 2.0 version... only to find it appears to be broken. So now I'm working through 4.0 (after verifying it does work). It's really slow going. No promises, but it's worth a shot!

Are there others you're aware of? Thanks!

[beheybird]

The most useful is Key Perfect 5.0 as it generates both 4 & 5 checksums which Nibble used for about 10 years. There is a program call AccuType by Mindcraft that had a line editor that added checkit checksums for line and program total. Not really that useful as Nibble only showed the program total and not the line numbers. Probably to get people to buy Key Perfect.

The .zip can be found from this to a web archive link
http://fileformats.archiveteam.org/wiki/AccuType

[a2geek]

Ok, thanks! I'll shift over to KP 5 then.

Interesting. I might be able to use the AccuType manual for some more samples.

[a2geek]

Finding I'm really slow when it's a relatively large assembly program!

However, I just committed code for Key Perfect 4.0. I'll dive into 5.0, and I hope that's relatively quick. The 4.0 look pulls calls out to the 5.0 code but with special REM handling (as far as I can figure).

Interestingly (well, somewhat), the 4.0 code is a vague bit of shifting on one byte and then everything else becomes a counter of something or another... but 5.0 has 4 different 256 byte tables that get EOR'd against the data. So, that'll be fun to pull over... assuming I'm following everything correctly!

[a2geek]

I just committed Key Perfect 5.0 as well. From the sample above (1st one) using the new --preserve and --kp5 flags:

$ bt --preserve --list --kp5 ticket-49a.bas 
10  PRINT "MATHING"
30 A = .4
40 B = 0.6000
50 C =  - .250
60 D =  - 0.70
70  PRINT "A=";A
80  PRINT "B=";B
90  PRINT "C=";C
95  PRINT "D=";D
Line# - Line#   CODE-5.0
-------------   --------
   10 -    95   E6BF208D
PROGRAM TOTAL   E6BF208D

Attaching the JAR file (in a Zip ... because): bastools-tools-bt-5.0-SNAPSHOT.zip