performance issue

[ziliangpeng]

As is noted in the project description, this library is way slower than the standard json library.

Is there plan to improve performance? What's the design and timeline?

[dpranke]

Hi!

Great question (and sorry for the delay in replying).

This project is mostly a test case for parser generator I'm working on, and so raw speed is less important to me than being able to actually generate a correct parser cleanly from a grammar.
That said, a generated grammar this slow is almost useless, so I do need to make sure the generator generates much better code.

Unsurprisingly, the builtin json module has both been tuned to be as fast as possible in pure Python, and has a C extension to be even faster. While I might eventually be able to generate a C extension using the parser generator, that's a ways off on the roadmap.

The pure python builtin JSON parser is 6-10x slower than the C parser; currently, I think my generated code is about 160x - 200x slower than the builtin pure python version.

I think I can get the generated code to be a lot better, and the current dev version I have on the parser can produce a JSON parser that is only 10x slower than the pure python version; this appears to be comparable to what you'd get with PLY and significantly faster than an ANTLR-generated JSON parser. I expect I should be able to match this for JSON5 without too much more work.

That's a long way of saying I expect I can get this 10-20x faster fairly soon now, maybe another week or two depending on how much free time I can find.

I don't know how hard it will be to get that last 10x, to match the pure python implementation, or even if it's doable at all, and I have no ETA or even a real expectation that I'll be able to match the speed of the native JSON implementation, but I do plan to keep working on it and I will post updates with my progress.

Hopefully that answers your question, probably in much more detail than you were hoping for. I hope the answer is not too disappointing.

[ziliangpeng]

That's a really informative read. Thanks for sharing.

I'm quite into the performance improvement and am doing deep reading of your code. @dpranke

[mostealth]

Hi dpranke

Any progress on this?. It would be really useful to have a fast enc/dec

[dpranke]

Hi,

Sorry for the delay in replying. Sheesh it's been a long time since I've updated this. As noted originally, this project was originally intended as a test case for a parser generator, and work on that parser generator has been long-derailed because I couldn't figure out how to get a particular feature to work. I've finally resolved that, though, so now I hope to get back to working on the performance improvements I had and merging them into the main branch.

So, hopefully we'll see some real improvements soon, though I can't say how much faster things will be at the moment. I will update this again soon once I dust off that branch and re-run some benchmarks.

[Greedquest]

FWIW https://github.com/Kijewski/pyjson5 looks like a good alternative which is written with Cython and so is pretty fast (3 orders of magnitude on loads for a small document). Although I've only come across either of them today so I have no idea if their APIs are equivalent or anything