Re: BDD testing framework without dependencies

[Coda Highland <chighland@...>]

On Tue, Aug 23, 2016 at 10:33 AM, steve donovan
<steve.j.donovan@gmail.com> wrote:
> On Tue, Aug 23, 2016 at 7:13 PM, Coda Highland <chighland@gmail.com> wrote:
>> But there comes a time when you want to run just one test out of your
>> whole suite, or when you want to be able to have your test suite
>> continue running after one test has failed, or when you want to be
>> able to format your test output in a useful way, or when you want to
>> improve your test isolation, or when you want to dummy in some
>> components that can't be autotested hermetically, or when you want to
>> monitor performance regressions, or when you want to collect
>> statistics
>
> I suppose (like Sean) I don't appear to want these ;)  Look, I'm not
> knocking the idea but I like tests simple - they don't talk, unless
> they break. If they break I sort it out immediately. I don't gain any
> satisfaction in seeing pages of pretty green output with the
> occasional outbreak of red. The statistics which are ultimately
> meaningful are about _coverage_ and that's outside the scope of a test
> framework.

There are statistics beyond coverage statistics. :P Especially in
large multi-developer projects, it's really useful to have historical
information about which tests fail most often, especially if your test
framework allows automatically retrying failed tests a couple times,
which can illustrate that the code under test (or the test itself)
isn't deterministic.

Like Peter said: Not everyone needs all of these features. If you're
working alone or in a small, tightly-knit group, or if your project is
structurally simple enough that it's easy to keep things deterministic
and hermetic (among other things, this means your project never uses
the network) and well-organized, then you can get away without this
functionality. But as projects and groups grow, the importance of
these things grows simply because of their collaborative benefit.

That said, compiler authors spent years working on being able to
proceed past the first error so that you can get more useful
information at a time instead of iterating over potentially lengthy
operations one bump at a time. Any project of significant size can
benefit from having a test framework that catches errors and
assertions, reports on them, and moves on to the next test.

(On the other hand, I certainly agree that green and red status
indicators are mostly vanity.)

/s/ Adam