Add todo reporting - #67
Conversation
|
Hello! Are you still working on this? |
|
Sorry, I thought I had marked it ready for review already. |
lpil
left a comment
There was a problem hiding this comment.
Thank you!! I've left a comment inline.
|
|
||
| handle_cancel(_test_or_group, Data, State) -> | ||
| ?reporting:test_failed(State, <<"gleeunit">>, <<"main">>, Data). | ||
| handle_cancel(test, Data, State) -> |
There was a problem hiding this comment.
What's this new code for? It seems unrelated to todo?
I think it might be code for another feature that got included by mistake?
There was a problem hiding this comment.
I added it to support a case when the todo was hit inside a spawned process ( i.e. OTP ), it wasn't reporting it correctly as a todo. If you prefer to leave that case unsupported this can be removed, and they'll continue to fail via error as current behaviour.
There was a problem hiding this comment.
I couldn't really add a good test case to demonstrate / cover this within the tests without bringing in dependencies that wouldn't be wanted in this repo.
There was a problem hiding this comment.
Yes please, only one feature per PR 🙏
There was a problem hiding this comment.
I have now rebased to cut out the OTP todo handling change.
1033bb7 to
9051777
Compare
lpil
left a comment
There was a problem hiding this comment.
Instead of mutating implicit state could you pass in the print function as an argument please 🙏
We always prefer simple functional solutions!
|
@lpil Thank you for that feedback, I have reworked it to inject print which has indeed made it a lot cleaner. Sorry, I'm not as used to writing functional code, it's purely a hobby, the day job demands OOP. While testing this, I realised it was frustrating to have the test output fill up with the same todo warning over and over, a single todo could get written dozens of times in the test output, after having already been output in the compilation log. I've changed that so it cleanly prints a yellow dot, in place of the green dot, to signal a test that is in the "todo" state:
I can change that back if it's not wanted, but the alternative is a messy progress log. The neatest solution might involve collecting the unique todo errors and then printing them before the summary, to give an overview of which were hit, but that kind of full test run summary would really be an entire different extra feature, perhaps better served by a different reporter. |

Satisfies #61
This adds todo reporting as a separate category in the final summary.
When there are only failing todos and no failures from asserts or panics, then this will show yellow, but still exit code 1, so will still fail in CI environments, to maintain compatibility with current behaviour.
There are a number of accomodations that had to be made to get this tested within this repository. I had to suppress the raw test output, so that it is not confused with the tests being asserted in the test module. Then in order to that in a compatible way, I had to special case Node and Deno handling.