Comparisons of fuzzer

[arcuri82]

Hi all @grebnetiew @ThomasTNO @Stranger6667 @en-milie ,

writing this here to simplify the discussions, and to have it open.

At a certain point, I would like to make a "fair" and "sound" comparison of fuzzers (@Stranger6667 this is somehow related to the discussion we had on Reddit), where all compared fuzzers' authors are directly involved.
This would lead to a possible joint academic publication among us (for whom is interested).

However, what is the status of supporting WFC Authentication? If I understood correctly, WuppieFuzz now supports it. What about Schemathesis and CATS?
This would be needed to easily run experiments on WFD, by using the already existing auth info in WFC format.

This is a very busy period, unfortunately (work and personal reasons). So, don't think anything would happen any time soon before some weeks (there would be several technical details to discuss and agree among us). However, I am writing it now to test out if there is any interest in it.

Let me know.

[grebnetiew]

It would be a nice test case for our WFC auth support to start with :D

We would be particularly interested if this leads to a benchmark framework that is considered 'sound' and/or 'fair' by the broader scientific community. In the past, we have had trouble converging (with article reviewers) on benchmarks that sufficiently represent real world bug-finding performance. In the future, we would like to use such benchmarks as integration tests for new features, fixes etc.

[Stranger6667]

From the Schemathesis side I have a WIP branch with the WFC Authentication support for a few months already, but didn't get to finish it just yet. So, I am going to bump its priority - probably will take a few weeks, so I'd aim to have it implemented by the end of July :)

This would lead to a possible joint academic publication among us (for whom is interested).

You can count on me here :)

P.S. I took some inspiration from WFC Authentication in Schemathesis's own declarative dynamic auth support

[arcuri82]

@Stranger6667 nice ;)

personally i be ll off on vacation most of July... so something starting around August/September would good ;)

meanwhile, to test WFC support, I would suggest to try with the auth configuration files for the APIs needing auth in WFD.

[ThomasTNO]

August also seems to fit nicely in my schedule. 👌

[arcuri82]

hi @Stranger6667 @ThomasTNO @grebnetiew @en-milie ,

I am resuming the work related to this task. I ll share it once I have something more concrete (it is on top of WFD in a new repository). However, while starting to setup things, I have a few questions per fuzzer:

1. @ThomasTNO @grebnetiew in WuppieFuzz, is there any CLI option to override the base url of the target API? eg, the equivalent of --server in CATS, --base in EvoMaster and --url in SchemaThesis. this is needed as OpenAPI schemas on file system would not have the right port info when running experiments in parallel, and, when running in docker, might need to use host.docker.internal instead of localhost. This is kind of a showstopper, as cannot trying out how to integrate WuppieFuzz in the experiment scaffolding

2. @Stranger6667 does Schemathesis have any "timeout" option? ie, something like running the fuzzer for up to 10 minutes, and stop it if reaching limit, without having to kill the process via a signal (eg, using a timeout shell wrapper). Eg, the equivalent of --maxTime in EvoMaster and --timeout in WuppieFuzz.

3. @en-milie, similar to point (2), I see CATS has a --stopAfterTimeInSec option, but that is only for when fuzzing single endpoints in isolation? is there any global setting?

Once I have a basic updated setup for experiments, then we can discuss details of how we want to carry out the experiments and comparisons.

[arcuri82]

thx! somehow I missed it when looking at the documentation... do you have a link with a list of all available CLI options?

[ThomasTNO]

I do not have a link, I think. You can get them through wuppiefuzz fuzz --help.

Usage: wuppiefuzz fuzz [OPTIONS] [OPENAPI_SPEC.YAML]

Arguments:
  [OPENAPI_SPEC.YAML]
          The path to the open api specification of the target. The specification must also contain the "server"-field at which the target is hosted

Options:
      --config <CONFIG_FILE.YAML>
          The path to a configuration file. If present, the configuration file is used to configure the fuzzer. Arguments given on the command line take precedence over the configuration file

  -i, --initial-corpus <CORPUS_DIRECTORY>
          The path to an initial corpus given as a directory with yaml files

      --target <TARGET>
          The URL of the server to fuzz. This is usually specified in the OpenAPI specification, but you can use this option to override it

      --coverage-host <COVERAGE_HOST>
          The host address of the coverage agent from which the coverage map can be obtained. Can be either a hostname or an IP address, and must include a port

      --coverage-format <COVERAGE_FORMAT>
          The format in which your instrumentation provides coverage information. Must be one of {'jacoco', 'lcov', 'coverband'}. If omitted, the fuzzer will use endpoint coverage only

          [possible values: jacoco, lcov, coverband]

      --timeout <TIMEOUT>
          Total fuzzing time-out in seconds. If present, the fuzzer exits after the timeout has passed

      --request-timeout <REQUEST_TIMEOUT>
          Per-request time-out in milliseconds. Defaults to DEFAULT_REQUEST_TIMEOUT milliseconds

      --power-schedule <POWER_SCHEDULE>
          Set the power schedule to use. Defaults to FAST

          Possible values:
          - explore: The `explore` power schedule
          - exploit: The `exploit` power schedule
          - fast:    The `fast` power schedule
          - coe:     The `coe` power schedule
          - lin:     The `lin` power schedule
          - quad:    The `quad` power schedule

      --crash-criteria <CRASH_CRITERIA>
          Which errors the fuzzer considers a bug.

          The possible errors are:
          - "OperationNotInSpec": The operation does not exist in the spec, which incidentally means it should not have been executed by the fuzzer to begin with.
          - "Status5xxNotSpecified": The HTTP status code 5xx returned from the API is not one of the status codes mentioned for this path in the specification.
          - "OtherStatusNotSpecified": The HTTP status code 1xx - 4xx returned from the API is not one of the status codes mentioned for this path in the specification.
          - "ResponseReferenceBroken":  The specification calls for an object to be returned, and refers to the correct structure of this object using a reference (`#/example/reference`). However, the reference path is not present in the specification or contains circular references.
          - "ResponseObjectIncorrect": The response body returned by the API does not match the structure specified in the API specification.
          - "ResponseEnumIncorrect": A field in the response body object is specified as an enumeration, but the returned value is not one of the possible variants.
          - "ResponseMalformedJSON": The response body returned by the API can not be parsed as JSON.
          - "UnexpectedContent": The API returned a response body, but no response is specified.
          - "MediaTypeContainsNoSchema": The API contains a media type "application/json" with no schema for the data inside the json object. We can't validate the response if no model is given.
          - "SchemaIsAny": The schema can be anything (occurs e.g. when it does not specify a type) we cannot validate schemas that are this flexible.

          By default, the value for this option is a list of all of the above. By specifying a subset of the above errors, you can configure what behaviour is considered a bug by the fuzzer.

      --report[=<REPORT>]
          If present, ask the coverage monitor to generate a report after the time-out passes

          [possible values: true, false]

      --method-mutation-strategy <METHOD_MUTATION_STRATEGY>
          If present, determine with which HTTP methods to mutate. Must be one of {'follow-spec', 'common5', 'common7'}, follow-spec: only mutate with methods from api specification common5: mutate with one of [get, post, put, patch, delete] common7: mutate with one of [get, post, put, patch, delete, head, trace] If omitted, only mutate with methods from api specification

          [possible values: follow-spec, common5, common7]

      --jacoco-class-dir <JACOCO_CLASS_DIR>
          When generating a Jacoco coverage report, look for class files in this directory

      --source-dir <SOURCE_DIR>
          When generating a coverage report, look for source files in this directory

      --output-format <OUTPUT_FORMAT>
          Output to stdout can be formatted in human readable format or json

          [possible values: json, human-readable]

      --authentication <AUTH.YAML>
          How to log in to the API server, if applicable. The value should be the name of a YAML file that contains the login configuration. See login.md for information on how to build one

      --header <STATIC_HEADERS.YAML>
          Custom (static) headers that should be added to each request. These header parameters will not be mutated, contrary to the usual header parameters passed through an API specification

      --log-level <LOG_LEVEL>
          Log level to output. This flag takes precedence over the environment variable. [possible values: off, error, warn, debug, info, trace]

          [env: LOG_LEVEL=]

      --jacoco-class-prefix <JACOCO_CLASS_PREFIX>
          Prefix used to filter the classes returned from the jacoco coverage. The class name can be found in the source code of the software under test. The class name returned from jacoco is in the form of "org/example/software/class". If no coverage is obtained anymore please check if the prefix is correct. If you use the trace debug level all skipped segment names are logged

  -h, --help
          Print help (see a summary with '-h')

[Stranger6667]

@arcuri82

There is --max-time only for st fuzz - a semi-experimental runner, which I'd skip for now. At the moment I'd use something like:

while true; do
  st run <FILE> \
    --url <URL> \
    --max-examples 750 \
    --continue-on-failure \
    --suppress-health-check all \
    --no-shrink || true
done

over timeout.

Also, I've been working on a new scheduler that supports --max-time for st run natively. So if there is still some time, it would be nice to use this, once it lands in schemathesis 4.26 (my ETA is ~1 week) :) otherwise, the loop above should work :)

[arcuri82]

@Stranger6667 great to hear ;) we have all the time we need, there is no hard deadline, but it is just best to not postpone things for too long :) Worst case we can run Schemathesis in a loop, but best if it could directly support a timeout

i have an infrastructure to run large scale experiments, it is just that i m in the middle of updating/improving it. I will share it hopefully in the next coming days (also when I write some more documentation for it), so you can debug any issue with your fuzzer before we finalize how we want to run and evaluate the experiments (I guess something within 1-2 months from now).

meanwhile, I am already using it to try out Schemathesis and other tools. There is one big problem though, which is the amount of data generated as output. Currently running 5 repetitions per fuzzer on 36 APIs, and WuppieFuzz generates 37GB, CATS 87GB, Schemathis 0, other fuzzers all less than 1GB.

@grebnetiew @ThomasTNO I am currently running with:

 "mkdir -p $OUTPUT_FOLDER \
     && cd $OUTPUT_FOLDER \
     && wuppiefuzz fuzz  \
        ${SCHEMA_FILE_PATH} \
        --target http://${HOST}:${PORT} \
        --timeout ${TIME_SECONDS} \
        ${EXTRA_CONFIGS} \
    "

anyway to reduce the amount of data generated in $OUTPUT_FOLDER? Still want to get data (ie generated test cases, and possibly summary reports, to then host on Zenodo when experiments are finalized), but 37GB feels a bit too much :)

@Stranger6667 by default Schemathesis just creates its tests on the console logs (which I save) without generating test files, isn't it?

[arcuri82]

hi @Stranger6667 @grebnetiew @ThomasTNO I created a new repository, with current work-in-progress experiment scaffolding (updated from WFD). I sent you invitations for it, as it is currently a private repository (we can make it public once our study is finalized). We can continue our discussion there.

You will see there a first, short set of experiments, with results. Logs are in the release page (as it is a few MBs file).
You should check if there is any API in which your fuzzer crashes for any reason, and fix those bugs before we finalize the experiments.
Also, in current results, only EvoMaster is configured with auth configuration in WFC format. You should make a PR with changes to your /tools/<name> Docker file to enable WFC auth support. You will see some documentation there.

I ll create a new discussion thread in that repository.

[en-milie]

@arcuri82 In order to reduce the output of CATS, for example ignoring success cases i.e. what is not considered an issue, you can use: --skipReportingForSuccess.

[en-milie]

@arcuri82 Regarding limiting the execution time, there is no global command that will time execution to a specific duration. It's mostly filtering for paths, methods, operations, etc. But let me add some stopping conditions that will be available in the next release.