I've used this in a few systems, using (in my case) nbconvert.
As I write more code, I increasingly find the most important thing about tests early on is that they are easy to write and maintain. The help that, I find one of the best 'quick test suites' is "run program, save output, run 'git diff' to see if anything changes".
This has several advantages. If you have lots of small programs it's trivial to parallelise. It's easy to see what outputs have changed. It's very easy to write weird one-off special tests that need to do something a bit unusual.
Yes, eventually you will probably want some nicer test framework, but even then I often keep this framework around, as there will still often be a few tests that don't fit nicely in whatever fancy testing library I'm trying to use (for example, checking a program's front end produces correct error messages when given invalid input).
As I write more code, I increasingly find the most important thing about tests early on is that they are easy to write and maintain. The help that, I find one of the best 'quick test suites' is "run program, save output, run 'git diff' to see if anything changes".
This has several advantages. If you have lots of small programs it's trivial to parallelise. It's easy to see what outputs have changed. It's very easy to write weird one-off special tests that need to do something a bit unusual.
Yes, eventually you will probably want some nicer test framework, but even then I often keep this framework around, as there will still often be a few tests that don't fit nicely in whatever fancy testing library I'm trying to use (for example, checking a program's front end produces correct error messages when given invalid input).