Add parameter for setting failure threshold

[kyrstenkelly]

I need to be able to set some sort of threshold for a # of violations that are allowed.
I tried to get this working by setting skipFailures = true, and using the violationsCallback to run my own assertions, but running assertions in that callback makes the tests behave weirdly.

I have currently forked this repo so I can add this parameter, but I would love if it got included here so I could keep using this repo instead!

[dwightjack]

I'd need something like this as well. Is there any blocker from merging the PR?

[sapegin]

Thanks for the pull request!

I was thinking about adding an option to define which severities should fail tests, and this one could be a good addition too.

1. I'm thinking about a new API for the next major release, and probably we'll need to replace all positional arguments of checkA11y with an object because we already have way too many and they are all optional.

2. The name of the parameter. Maybe maxViolations similar to ESLint's maxWarnings would be more clear?

I'll keep this pull request as a reference, and either include in the new version or ask you to rebase it on top of it.

[sapegin]

@kyrstenkelly I drafted my ideas on the new API: #75. I hope this covers your use case — let me know what you think.

[kyrstenkelly]

@sapegin I'm loving the new API proposal (also 👍🏻 TS). In the mean time, we already long ago forked this repo and made the changes I had suggested here. Once the new API is done, I'll be excited to get rid of our fork and switch back to this lib. :)