SBJson v5.0.0 Release Notes

Release Date: 2016-11-15 // about 6 years ago
  • I certainly didn't a month ago.

    ๐Ÿš€ This is the second release motivated by Nicholas Seriot's Parsing JSON is a
    ๐Ÿ“œ Minefield

    Targeting RFC 7159

    ๐Ÿš€ This release allows scalar values at the top level; as recommended by RFC
    7159, which obsoletes the original RFC 4627. Since it is a change in
    behaviour I chose to bump the major version to 5.

    Please note : When parsing numbers at the top level there is no way to
    differentiate 42 on its own from 4200 truncated to just the first two
    digits. This problem affects SBJson 5 because it expects to receive input
    ๐Ÿ‘€ bit-by-bit. When SBJson 5 sees "42" on its own it returns
    ๐Ÿ‘€ SBJson5WaitingForData, since cannot be sure it has seen the full token
    โ†ช yet, and needs more data to make sure. A workaround for this issue could be
    to append a space or newline to your input if you intend to give SBJson 5
    the whole input in one go. This is not an issue with any of the other JSON
    ๐Ÿ›  datatypes because they are either fixed length (true, false, null) or
    have unambigous delimiters at both ends ([], {}, "").

    ๐Ÿ“‡ Rename all classes & public symbols

    Because the class names contains the major version number a major-version
    โฌ†๏ธ bump necessitates renaming all the classes & enums. The upshoot of this is
    that you can use SBJson 3, 4 and 5 in the same application without
    problems. (Though why you would want to I cannot even begin to guess at.)

    โœ‚ Remove the processBlock: API

    ๐Ÿš€ This release removes the untested processBlock: interface. I believe it
    ๐Ÿ“œ was a distraction from SBJson's core purpose: to parse & generate JSON.
    โž• Additionally this API had no tests, and the code had a lot of special case
    ๐Ÿ“œ hooks all over the SBJson*Parser class to do its work.

    ๐Ÿ“œ SBJson actually has two parsers: the low-level SBJson5StreamParser and the
    ๐Ÿ“œ higher-level SBJson5Parser providing a block interface. I believe it's
    ๐Ÿ‘ better to just do what the processBlock interface did in SBJson5Parser's
    ๐Ÿ“œ value block. However, you could also use the stream parser to implement the
    ๐Ÿ–จ processBlock interface yourself.

    ๐Ÿ“œ Constructor changes for parsers + writers

    Since I decided to bump the major version number anyway, I took the
    ๐Ÿ’ป opportunity to iron out some UI niggles that's been bothering me for a
    while. Now we take options as constructor parameters rather than as
    ๐Ÿ“œ properties for boh the parsers and writers, to avoid the impression that
    you can (and that it might make sense!) to change these settings during
    ๐Ÿ“œ parse/generation. It is absolutely not supported, and that should be more
    clear now.

    โž• Add a sbjson binary for reformatting JSON

    This can be useful from a sort of what would SBJson do? point of view. It
    takes some options. Here's the result of invoking it with --help:

    Usage: sbjson [OPTIONS] [FILES]
      --help, -h
        This message.
      --verbose, -v
        Be verbose about which arguments are used
      --multi-root, -m
        Accept multiple top-level JSON inputs
      --unwrap-root, -u
        Unwrap top-level arrays
      --max-depth INT, -m INT
        Change the max recursion limit to INT (default: 32)
      --sort-keys, -s
        Sort dictionary keys in output
      --human-readable, -r
        Format the JSON output with linebreaks and indents
    If no FILES are provided, the program reads standard input.

    โš™ Run sbjson under American Fuzzy Lop

    To try and shake out any new crashes, I've run the sbjson binary alluded
    to above under American Fuzzy Lop. I didn't find any more crashes in the
    ๐Ÿ“œ parser after fixing the bugs that went into v4.0.4, but wanted to share
    this with you to show I tried to find more bugs before releasing v5.

    โœ… Here's a snapshot of the latest session I've run:

                           american fuzzy lop 2.35b (master)
    โ”Œโ”€ process timing โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€ overall results โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚ run time : 1 days, 12 hrs, 36 min, 22 sec โ”‚ cycles done : 11 โ”‚
    โ”‚ last new path : 0 days, 0 hrs, 34 min, 26 sec โ”‚ total paths : 583 โ”‚
    โ”‚ last uniq crash : none seen yet โ”‚ uniq crashes : 0 โ”‚
    โ”‚ last uniq hang : 0 days, 2 hrs, 10 min, 54 sec โ”‚ uniq hangs : 47 โ”‚
    โ”œโ”€ cycle progress โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€ map coverage โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚ now processing : 170 (29.16%) โ”‚ map density : 0.39% / 1.49% โ”‚
    โ”‚ paths timed out : 0 (0.00%) โ”‚ count coverage : 5.02 bits/tuple โ”‚
    โ”œโ”€ stage progress โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€ findings in depth โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚ now trying : splice 7 โ”‚ favored paths : 93 (15.95%) โ”‚
    โ”‚ stage execs : 5/32 (15.62%) โ”‚ new edges on : 142 (24.36%) โ”‚
    โ”‚ total execs : 18.1M โ”‚ total crashes : 0 (0 unique) โ”‚
    โ”‚ exec speed : 282.7/sec โ”‚ total hangs : 297 (47 unique) โ”‚
    โ”œโ”€ fuzzing strategy yields โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€ path geometry โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚ bit flips : 0/678k, 4/677k, 0/677k โ”‚ levels : 15 โ”‚
    โ”‚ byte flips : 0/84.8k, 0/84.5k, 0/83.9k โ”‚ pending : 31 โ”‚
    โ”‚ arithmetics : 0/4.72M, 0/16.6k, 0/307 โ”‚ pend fav : 0 โ”‚
    โ”‚ known ints : 0/480k, 0/2.35M, 0/3.69M โ”‚ own finds : 40 โ”‚
    โ”‚ dictionary : 0/0, 0/0, 2/2.49M โ”‚ imported : 3 โ”‚
    โ”‚ havoc : 29/1.25M, 5/753k โ”‚ stability : 100.00% โ”‚
    โ”‚ trim : 11.02%/43.6k, 0.00% โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
    ^Cโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ [cpu: 69%]
    +++ Testing aborted by user +++
    [+] We're done here. Have a nice day!

    ๐Ÿ›  Fix bug in unwrapper code that caused arrays to be skipped

    ๐Ÿ›  Whilst playing with AFL I accidentally found (and fixed) a bug where the
    ๐Ÿ“œ unwrapRootArray parser would break on any arrays at the next-to-outermost

    ๐Ÿ‘Œ Improved documentation

    ๐Ÿ“š I've tried to improve the documentation a little, both in README and the API
    ๐Ÿ“š documentation in the header files.