acknowledgement of confusing nature of tests

I think I have a problem.  Someone stop me.

9unit.c

@@ -350,7 +350,7 @@ print_log(TestState *s)
 	TestLogEntry *e = s->first_log;
 	while(e)
 	{
-		if(e->text) print("%s\n", e->text);
+		if(e->text) print("\n%s", e->text);
 		else print("(empty message)\n");
 		e = e->next;
 	}
@@ -389,7 +389,8 @@ reindex(TestState *s)
 	else if (s->last_log) // we have a last log but no first?
 	{
 		s->first_log = s->last_log;
-		fprint(2, "potential memory leak in test log\n");
+		log_test_context(s);
+		append_test_log(s, "potential memory leak in test log");
 	}
 }
 

README.md

@@ -1,16 +1,314 @@
 # 9unit
 Copyright (C) 2023  Jonathan Lamothe <jonathan@jlamothe.net>
 
-This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Lesser General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.
 
-This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more details.
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Lesser General Public License for more details.
 
-You should have received a copy of the GNU Lesser General Public License along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-## WARNING
-
-This library is experimental and should be considered subject to change at any time.
+You should have received a copy of the GNU Lesser General Public
+License along with this program.  If not, see
+<http://www.gnu.org/licenses/>.
 
 ## Summary
 
-A simple C testing framework for Plan9
+A simple unit testing framework for C programs in Plan9
+
+This provides the library file `9unit.a` and the header `9unit.h`.
+The header is relatively well commented and can provide a fairly
+comprehensive breakdown of the API.  This document will however
+provide a basic overview below.
+
+This library is used to test itself, consequently the `test` directory
+contains a relitively decent real-world (if somewhat confusing)
+example of how it can be used.
+
+## `TestState`
+
+The entire testing framework is centred around the `TestState` data
+structure.  As its name would imply, it contains the current state of
+the tests in progress, however it should almost never be necessary to
+interact with it directly.  With the exception of `run_tests()`
+(described below), all functions provided by the library will take
+take a pointer to the current `TestState` value as their first
+argument.
+
+## `run_tests()`
+
+This will typically be the first function called.  It creates an
+initial `TestState` value, runs the tests, and displays a test log and
+summary at the end.  If any of the tests fail, it will cause the test
+process to exit with a status of `"test(s) failed"`.  Its prototype
+follows:
+
+```C
+void run_tests(void (*)(TestState *));
+```
+
+Its only argument is a pointer to a function which is then
+responsible for actually running the tests.  A pointer to the newly
+created `TestState` value will be passed to this function.
+
+## Simple Tests
+
+The simplest form of test can be represented by a function resembling
+the follwoing:
+
+```C
+TestResult my_test(TestState *s)
+{
+	// test code goes here...
+}
+```
+
+This function should return a `TestResult` value representing (perhaps
+unsurprisingly) the result of the test.  The options are as follows:
+
+- `test_success`: the test was completed successfully
+- `test_failure`: the test failed
+- `test_pending`: the test is pending, and should be ignored for now
+
+Tests of this type can be run by passing a pointer to them to the
+`run_test()` function which has the following prototype:
+
+```C
+void run_test(
+	TestState *,
+	TestResult (*)(TestState *)
+)
+```
+
+This function will call the provided test function, and update the
+provided `TestState` to reflect the result.  Thus, the above
+hypothetical test could by run as follows:
+
+```C
+void
+tests(TestState *s)
+{
+	run_test(s, my_test);
+}
+
+void
+main()
+{
+	run_tests(tests);
+	exits(0);
+}
+```
+
+Passing a null `TestState` pointer will cause nothing to happen.  This
+is true of all functions in this library.  (This behaviour might be
+reconsidered later, so don't count on it.) Passing a null function
+pointer to `run_test()` is interpreted as a pending test.
+
+## Passing Values to Tests
+
+Since C supports neither lambdas nor closures, this would leave one
+with little choice but to come up with a unique name for each
+individual test function.  This, while possible, would definitely be
+rather inconvenient.  To combat this shortcoming, it is helpful to be
+able to pass data into a generic test function so that it can be
+reused multiple times.
+
+### The `ptr` Value
+
+The `TestState` struct has a value called `ptr` which is a `void`
+pointer that can be set prior to calling `run_test()` (or any other
+function, really).  This value can then be referenced by the test
+function, giving you the ability to essentially pass in (or out) *any*
+type of data you may need.  While not ideal, it's *a* solution.
+
+The library does not perform any kind of validation or automatic
+memory management on the `ptr` value (this is C after all), so the
+responsibility for this falls to the programmer implementing the
+tests.
+
+### Convenience Functions
+
+As the test suite becomes more and more complex, managing a single
+`ptr` value can become increasingly burdensome.  For this reason,
+there are a few convenience functions that provide an alternate
+mechanism for passing data into a function without altering the `ptr`
+value.  (They actually do alter it internally, but they restore the
+original value before passing the state on.) Two such functions are:
+`run_test_with()`, and `run_test_compare()`.
+
+`run_test_with()` has the following prototype:
+
+```C
+void run_test_with(
+	TestState *,
+	TestResult (*)(TestState *, void *),
+	void *
+);
+```
+
+The first argument points to the current test state.  The second
+points to a test function much like the simple test function described
+above, but that takes a void pointer as a second argument.  Finally,
+the third argument is the pointer that gets passed into the test
+function.
+
+`run_test_compare()` is similar, but it allows *two* pointers to be
+passed into the test function.  This is useful for comparing the
+actual output of a function to an expected value, for instance.
+
+The prototype for `run_test_compare()` follows:
+
+```C
+void run_test_compare(
+	TestState *,
+	TestResult (*)(TestState *, void *, void *),
+	void *,
+	void *
+);
+```
+
+The pointers will be passed into the test function in the same order
+they are passed into `run_test_compare()`.
+
+## Test Contexts
+
+It is useful to document what your tests are doing.  This can be
+achieved using contexts.  Contexts are essentially labelled
+collections of related tests.  Contexts can be nested to create
+hierarchies.  This is useful both for organization purposes as well as
+creating reusable test code.  There are several functions written for
+managing these contexts.  Each of these functions takes as its first
+two arguments: a pointer to the current `TestState`, and a pointer to
+a string describing the context it defines.  If the pointer to the
+string is null, the tests are run as a part of the existing context.
+
+### `test_context()`
+
+```C
+void test_context(
+	TestState *,
+	const char *,
+	void (*)(TestState *)
+);
+```
+
+This function takes a pointer to the current `TestState`, a string
+describing the context, and a function pointer that is used the same
+way as the one passed to `run_tests()`.  This function will be called
+and its tests will be run within the newly defined context.  Nothing
+prevents this function from being called again in a different context.
+
+### `test_context_with()`
+
+```C
+void test_context_with(
+	TestState *,
+	const char *,
+	void (*)(TestState *, void *),
+	void *
+);
+```
+
+This funciton works similarly to `test_context()`, but allows for the
+passing of a `void` pointer into the test function in much the same
+way as the `run_test_with()` function.  Its arguments are (in order),
+a pointer to the current state, the context description, a pointer to
+the test function, and the pointer to be passed into that function.
+
+### `test_context_compare()`
+
+```C
+void test_context_compare(
+	TestState *,
+	const char *,
+	void (*)(TestState *, void *, void *),
+	void *,
+	void *
+);
+```
+
+This funciton allows the passing to two `void` pointers into a context
+in a manner similar to `run_test_compare()`.
+
+### `single_test_context()`
+
+```C
+void single_test_context(
+	TestState *,
+	const char *,
+	TestState (*)(TestState *)
+);
+```
+
+This function applies the context label to a *single* test.  The
+function passed in is expected to operate in the same way as the one
+passed to `run_test()`.
+
+### `single_test_context_with()`
+
+```C
+void single_test_context_with(
+	TestState *,
+	const char *,
+	TestState (*)(TestState *, void *),
+	void *
+);
+```
+
+This is similar to `single_test_context()` but allows a `void` pointer
+to be passed as in `run_test_with()`.
+
+### `single_test_context_compare()`
+
+```C
+void single_test_context_compare(
+	TestState *,
+	const char *,
+	TestResult (*)(TestState *, void *, void *),
+	void *,
+	void *
+);
+```
+
+I assume you get the idea at this point.
+
+## Logging
+
+When `run_tests()` finishes running the tests, it displays a log and
+summary.  The summary is simply a tally of the number of tests run,
+passed, failed, and pending.  While this is useful (and probably all
+you need to know when all the tests pass) it is likely desirable to
+have more detail when something goes wrong.  To facilitate this, tests
+can append to the test log, which is automatically displayed just
+before the summary.  There are two functions for doing this.
+
+### `append_test_log()`
+
+```C
+void append_test_log(
+	TestState *,
+	const char *
+);
+```
+
+This appends an arbitrary string to the end of the test log.  The
+contents of the string are copied into the log, so the value pointed
+to by the second argument does not need to persist in memory beyond
+the end of the call to the function.  Log entries are expected to be
+single lines.  No trailing newline should be present (but the trailing
+NUL character should (obviously)).
+
+### `log_test_context()`
+
+```C
+void log_test_context(TestState *);
+```
+
+This function appends an entry to the log indicating the test's
+current *full* context.  If no context is defined, the log entry will
+be `"<no context>"`.  If the test is inside of a context labeled
+`"foo"` which is inside of another context labeled `"bar"`, the
+resulting log entry will read `"bar: foo"`.

test/append-test-log.c

@@ -44,8 +44,15 @@ static void missing_last(TestState *);
 static void missing_first(TestState *);
 static void null_message(TestState *);
 static TestResult null_state(TestState *);
+static void chk_empty_first(TestState *, void *);
+static void chk_first(TestState *, void *);
+static void chk_second(TestState *, void *);
 static void chk_last(TestState *, void *);
+static void mf_chk_first(TestState *, void *);
+static void mf_chk_second(TestState *, void *);
+static void mf_chk_third(TestState*, void *);
 static void mk_log_data(LogData *);
+static void clean_entry(TestLogEntry *);
 
 static void chk_ptr_chg(
 	TestState *,
@@ -77,8 +84,16 @@ append_to_empty(TestState *s)
 	TestState test;
 	mk_sample_state(&test);
 	append_test_log(&test, "foo");
-	chk_ptr_ne(s, "first_log", test.first_log, 0);
-	chk_str_eq(s, "first_log->text", test.first_log->text, "foo");
+
+	// first_log
+	test_context_with(
+		s,
+		"first_log",
+		chk_empty_first,
+		&test
+	);
+
+	// last_log
 	chk_ptr_eq(s, "last_log", test.last_log, test.first_log);
 }
 
@@ -88,7 +103,8 @@ append_to_existing(TestState *s)
 	LogData ld;
 	mk_log_data(&ld);
 	append_test_log(&ld.state, "baz");
-	chk_ptr_eq(s, "first_log", ld.state.first_log, &ld.first);
+	test_context_with(s, "first_log", chk_first, &ld);
+	test_context_with(s, "second log", chk_second, &ld);
 	test_context_with(s, "last_log", chk_last, &ld);
 }
 
@@ -99,7 +115,8 @@ missing_last(TestState *s)
 	mk_log_data(&ld);
 	ld.state.last_log = 0;
 	append_test_log(&ld.state, "baz");
-	chk_ptr_eq(s, "first_log", ld.state.first_log, &ld.first);
+	test_context_with(s, "first_log", chk_first, &ld);
+	test_context_with(s, "second log", chk_second, &ld);
 	test_context_with(s, "last_log", chk_last, &ld);
 }
 
@@ -110,8 +127,13 @@ missing_first(TestState *s)
 	mk_log_data(&ld);
 	ld.state.first_log = 0;
 	append_test_log(&ld.state, "baz");
-	chk_ptr_eq(s, "first_log", ld.state.first_log, &ld.second);
+	test_context_with(s, "first_log", mf_chk_first, &ld);
 	test_context_with(s, "last_log", chk_last, &ld);
+	TestLogEntry
+		*context = ld.second.next,
+		*warning = context->next;
+	clean_entry(context);
+	clean_entry(warning);
 }
 
 static void
@@ -132,6 +154,48 @@ null_state(TestState *)
 	return test_success;
 }
 
+static void
+chk_empty_first(TestState *s, void *ptr)
+{
+	TestState *test = ptr;
+	chk_str_eq(s, "text", test->first_log->text, "foo");
+	chk_ptr_eq(s, "next", test->first_log->next, 0);
+}
+
+static void
+chk_first(TestState *s, void *ptr)
+{
+	LogData *ld = ptr;
+
+	// first_log
+	chk_ptr_eq(
+		s,
+		0,
+		ld->state.first_log,
+		&ld->first
+	);
+
+	// next
+	chk_ptr_eq(
+		s,
+		"next",
+		ld->state.first_log->next,
+		&ld->second
+	);
+}
+
+static void
+chk_second(TestState *s, void *ptr)
+{
+	LogData *ld = ptr;
+	chk_ptr_eq(
+		s,
+		"next",
+		ld->second.next,
+		ld->state.last_log
+	);
+}
+
 static void
 chk_last(TestState *s, void *ptr)
 {
@@ -139,8 +203,38 @@ chk_last(TestState *s, void *ptr)
 	chk_ptr_chg(s, 0, ld->state.last_log, &ld->second);
 	chk_str_eq(s, "text", ld->state.last_log->text, "baz");
 	chk_ptr_eq(s, "next", ld->state.last_log->next, 0);
-	free(ld->state.last_log->text);
-	free(ld->state.last_log);
+	clean_entry(ld->state.last_log);
+}
+
+static void
+mf_chk_first(TestState *s, void *ptr)
+{
+	LogData *ld = ptr;
+	chk_ptr_eq(s, 0, ld->state.first_log, &ld->second);
+	test_context_with(s, "next", mf_chk_second, ptr);
+}
+
+static void
+mf_chk_second(TestState *s, void *ptr)
+{
+	LogData *ld = ptr;
+	TestLogEntry *e = ld->second.next;
+	chk_str_eq(s, "text", e->text, "<no context>");
+	test_context_with(s, "next", mf_chk_third, ptr);
+}
+
+static void
+mf_chk_third(TestState *s, void *ptr)
+{
+	LogData *ld = ptr;
+	TestLogEntry *e = ld->second.next->next;
+	chk_str_eq(
+		s,
+		"text",
+		e->text,
+		"potential memory leak in test log"
+	);
+	chk_ptr_eq(s, "next", e->next, ld->state.last_log);
 }
 
 static void
@@ -183,4 +277,11 @@ chk_ptr_chg_test(
 	chk_ptr_ne(s, "not null", actual, 0);
 }
 
+static void
+clean_entry(TestLogEntry *e)
+{
+	free(e->text);
+	free(e);
+}
+
 //jl

test/mkfile

@@ -22,6 +22,8 @@ HFILES=\
 	util.h\
 	initial-state.h\
 	run-test.h\
+	run-test-with.h\
+	run-test-compare.h\
 	append-test-log.h\
 	test-context.h
 
@@ -29,6 +31,8 @@ OFILES=\
 	util.$O\
 	initial-state.$O\
 	run-test.$O\
+	run-test-with.$O\
+	run-test-compare.$O\
 	append-test-log.$O\
 	test-context.$O
 

test/run-test-compare.c

@@ -0,0 +1,87 @@
+/*
+
+	9unit
+	Copyright (C) Jonathan Lamothe <jonathan@jlamothe.net>
+
+	This program is free software: you can redistribute it and/or modify
+	it under the terms of the GNU Lesser General Public License as
+	published by the Free Software Foundation, either version 3 of the
+	License, or (at your option) any later version.
+
+	This program is distributed in the hope that it will be useful, but
+	WITHOUT ANY WARRANTY; without even the implied warranty of
+	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+	Lesser General Public License for more details.
+
+	You should have received a copy of the GNU Lesser General Public
+	License along with this program.  If not, see
+	<http://www.gnu.org/licenses/>.
+
+*/
+
+#include <u.h>
+#include <libc.h>
+
+#include "../9unit.h"
+#include "util.h"
+#include "run-test-compare.h"
+
+// Internal Prototypes
+
+static TestResult
+test_run_test_compare_test(TestState *, void *, void *);
+
+// Public Functions
+
+void
+test_run_test_compare(TestState *s)
+{
+	TestState actual, expected;
+
+	// build initial state
+	mk_sample_state(&actual);
+	void
+		*pass_in1 = malloc(1),
+		*pass_in2 = malloc(1),
+		*passed_in[2];
+	for (int i = 0; i < 2; i++)
+		passed_in[i] = 0;
+	actual.ptr = passed_in;
+
+	// run it and see what was passed
+	run_test_compare(
+		&actual,
+		test_run_test_compare_test,
+		pass_in1,
+		pass_in2
+	);
+	chk_ptr_eq(s, "first value passed", passed_in[0], pass_in1);
+	chk_ptr_eq(s, "second value passed", passed_in[1], pass_in2);
+
+	// ensure the state was updated correctly
+	mk_sample_state(&expected);
+	expected.passed = 2;
+	expected.run = 7;
+	expected.ptr = &passed_in;
+	chk_TestState_eq(s, "resulting state", &actual, &expected);
+
+	free(pass_in1);
+	free(pass_in2);
+}
+
+// Internal Functions
+
+static TestResult
+test_run_test_compare_test(
+	TestState *s,
+	void *ptr1,
+	void *ptr2
+)
+{
+	void **passed_in = s->ptr;
+	passed_in[0] = ptr1;
+	passed_in[1] = ptr2;
+	return test_success;
+}
+
+//jl

test/run-test-compare.h

@@ -0,0 +1,24 @@
+/*
+
+	9unit
+	Copyright (C) Jonathan Lamothe <jonathan@jlamothe.net>
+
+	This program is free software: you can redistribute it and/or modify
+	it under the terms of the GNU Lesser General Public License as
+	published by the Free Software Foundation, either version 3 of the
+	License, or (at your option) any later version.
+
+	This program is distributed in the hope that it will be useful, but
+	WITHOUT ANY WARRANTY; without even the implied warranty of
+	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+	Lesser General Public License for more details.
+
+	You should have received a copy of the GNU Lesser General Public
+	License along with this program.  If not, see
+	<http://www.gnu.org/licenses/>.
+
+*/
+
+extern void test_run_test_compare(TestState *);
+
+//jl

test/run-test-with.c

@@ -0,0 +1,71 @@
+/*
+
+	9unit
+	Copyright (C) Jonathan Lamothe <jonathan@jlamothe.net>
+
+	This program is free software: you can redistribute it and/or modify
+	it under the terms of the GNU Lesser General Public License as
+	published by the Free Software Foundation, either version 3 of the
+	License, or (at your option) any later version.
+
+	This program is distributed in the hope that it will be useful, but
+	WITHOUT ANY WARRANTY; without even the implied warranty of
+	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+	Lesser General Public License for more details.
+
+	You should have received a copy of the GNU Lesser General Public
+	License along with this program.  If not, see
+	<http://www.gnu.org/licenses/>.
+
+*/
+
+#include <u.h>
+#include <libc.h>
+
+#include "../9unit.h"
+#include "util.h"
+#include "run-test-with.h"
+
+// Internal Prototypes
+
+static TestResult test_run_test_with_test(TestState *, void *);
+
+// Public Functions
+
+void
+test_run_test_with(TestState *s)
+{
+	TestState actual, expected;
+
+	// build initial state
+	mk_sample_state(&actual);
+	void
+		*pass_in = malloc(0),
+		*passed_in = 0;
+	actual.ptr = &passed_in;
+
+	// run it and see what was passed
+	run_test_with(&actual, test_run_test_with_test, pass_in);
+	chk_ptr_eq(s, "value passed", passed_in, pass_in);
+
+	// ensure the state was updated correctly
+	mk_sample_state(&expected);
+	expected.passed = 2;
+	expected.run = 7;
+	expected.ptr = &passed_in;
+	chk_TestState_eq(s, "resulting state", &actual, &expected);
+
+	free(pass_in);
+}
+
+// Internal Functions
+
+static TestResult
+test_run_test_with_test(TestState *s, void *ptr)
+{
+	void **passed_in = s->ptr;
+	*passed_in = ptr;
+	return test_success;
+}
+
+//jl

test/run-test-with.h

@@ -0,0 +1,24 @@
+/*
+
+	9unit
+	Copyright (C) Jonathan Lamothe <jonathan@jlamothe.net>
+
+	This program is free software: you can redistribute it and/or modify
+	it under the terms of the GNU Lesser General Public License as
+	published by the Free Software Foundation, either version 3 of the
+	License, or (at your option) any later version.
+
+	This program is distributed in the hope that it will be useful, but
+	WITHOUT ANY WARRANTY; without even the implied warranty of
+	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+	Lesser General Public License for more details.
+
+	You should have received a copy of the GNU Lesser General Public
+	License along with this program.  If not, see
+	<http://www.gnu.org/licenses/>.
+
+*/
+
+extern void test_run_test_with(TestState *);
+
+//jl

test/tests.c

@@ -25,6 +25,8 @@
 #include "../9unit.h"
 #include "initial-state.h"
 #include "run-test.h"
+#include "run-test-with.h"
+#include "run-test-compare.h"
 #include "append-test-log.h"
 #include "test-context.h"
 
@@ -54,6 +56,8 @@ tests(TestState *s)
 
 	test_context_with(s, "initial state", test_initial_state, &scpy);
 	test_context(s, "run_test()", test_run_test);
+	test_context(s, "run_test_with()", test_run_test_with);
+	test_context(s, "run_test_compare()", test_run_test_compare);
 	test_context(s, "append_test_log()", test_append_test_log);
 	test_context(s, "test_context()", test_test_context);
 }