diff --git a/app/docs/dev.md b/app/docs/dev.md index dbb1f6e..c731629 100644 --- a/app/docs/dev.md +++ b/app/docs/dev.md @@ -8,15 +8,29 @@ Valid params include `absolute_tolerance` and `relative_tolerance`, which can be is_correct = abs(res - ans) <= (absolute_tolerance + relative_tolerance*abs(ans)) ``` +Alternatively, `sig_figs` (or `significant_figures`) can be supplied to require both a numeric match and a precision match to N significant figures: + +```python +is_correct = ( + round_to_sig_figs(res, sig_figs) == round_to_sig_figs(ans, sig_figs) + and count_sig_figs(res) == sig_figs +) +``` + +`sig_figs` cannot be combined with `atol`/`rtol` — supplying both raises an exception. + +In `sig_figs` mode, `response` must be a **string** (e.g. `"92.00"`), not a pre-parsed `int`/`float` — significant-figure counting depends on the exact digits written (trailing zeros, decimal point placement), which a parsed number can't preserve (`92.0 == 92.00 == 92`). `_split_numeric_string` in `evaluation.py` validates and decomposes the string (sign / integer part / fractional part / exponent) using `str.isdigit()` on each part; if `response` isn't a string, or isn't a valid numeral, the sig-figs check returns the standard "Please enter a number." result rather than raising. `_count_sig_figs` then applies the standard significant-figures counting rules (non-zero digits always count; zeros between digits count; leading zeros don't; trailing zeros only count if a decimal point was written) to the parsed parts. + ## Inputs ```json { - "response": "", + "response": "", "answer": "", "params": { "absolute_tolerance": "", - "relative_tolerance": "" + "relative_tolerance": "", + "sig_figs": "" } } ``` @@ -29,12 +43,16 @@ Absolute tolerance parameter Relative tolerance parameter +### `sig_figs` + +Significant-figures parameter. Mutually exclusive with `atol`/`rtol`. + ## Outputs ```json { "is_correct": "", "real_diff": "", - "allowed_diff": "", + "allowed_diff": "" } ``` diff --git a/app/docs/user.md b/app/docs/user.md index 81372f3..3018298 100644 --- a/app/docs/user.md +++ b/app/docs/user.md @@ -12,7 +12,7 @@ The left-hand side is the absolute difference between the student's response and ## Parameters -Both parameters default to `0` (exact match required) and can be used individually or together. +`atol` and `rtol` both default to `0` (exact match required) and can be used individually or together. `sig_figs` is an alternative comparison mode and cannot be used together with `atol`/`rtol`. ### `absolute_tolerance` — Absolute tolerance @@ -22,6 +22,20 @@ Specifies a fixed margin around the answer, regardless of its magnitude. Use thi Specifies an acceptable error as a fraction of the answer's magnitude. Use this when the answer is very large or very small and a percentage-based margin makes more sense than a fixed one. +### `sig_figs` — Significant figures + +Checks two things: that the response's numeric value rounds to the same value as the answer at the given number of significant figures, and that the response was itself *written* with exactly that many significant figures. Use this when correctness is defined in terms of precision (e.g. "correct to 3 significant figures") rather than a fixed or proportional margin. Cannot be combined with `atol`/`rtol` — supplying both raises an error. + +In `sig_figs` mode, **`response` must be supplied as a string** (e.g. `"92.00"`, not `92.0`), so that trailing zeros and decimal points are preserved exactly as written — a parsed number can't distinguish `92` from `92.00`. If `response` isn't a string, or isn't a valid number, the student is asked to enter a number, the same as any other non-numeric response. + +Significant figures are counted as follows: +- All non-zero digits are significant. +- Zeros between non-zero digits are significant (e.g. `2051` has 4). +- Leading zeros are not significant (e.g. `0.0032` has 2). +- Trailing zeros after a decimal point are significant (e.g. `92.00` has 4). +- Trailing zeros in a whole number are only significant if a decimal point is shown (e.g. `540` has 2, but `540.` has 3). +- In scientific notation, only the digits before the exponent count (e.g. `5.02e4` has 3). + ## Examples ### Exact match (default) @@ -52,8 +66,18 @@ With answer `6.674e-11`, accepts any response within **1%** of the answer. Good Both tolerances contribute: with answer `9.81`, the allowed difference is `0.01 + 0.005 × 9.81 ≈ 0.059`. Useful when you want a minimum floor (`absolute_tolerance`) plus a proportional allowance (`relative_tolerance`). +### Significant figures + +```json +{ "sig_figs": 3 } +``` + +With answer `3.14159`, the response `"3.14"` is accepted (correct value, 3 significant figures). `"3.1"` is rejected for having too few significant figures, and `"3.14159"` is rejected for having too many — even though both are numerically close to the answer. Unlike `atol`/`rtol`, this cannot be combined with tolerance params — `{ "sig_figs": 3, "atol": 0.01 }` will raise an error rather than be evaluated. + ## Notes **Note:** If the answer is not a number, all responses will generate an error. **Note:** If the response is not a number, a feedback message asking the student to submit a number will be returned. + +**Note:** `sig_figs` and `atol`/`rtol` are mutually exclusive; supplying both will generate an error. diff --git a/app/evaluation.py b/app/evaluation.py index 871f314..a83fa16 100644 --- a/app/evaluation.py +++ b/app/evaluation.py @@ -1,5 +1,90 @@ from numpy import spacing + +def _is_number(value): + return isinstance(value, int) or isinstance(value, float) + + +def _round_to_sig_figs(value, sig_figs): + if value == 0: + return 0.0 + return float(f"{value:.{sig_figs}g}") + + +def _split_numeric_string(value): + if not isinstance(value, str): + return None + stripped = value.strip() + body = stripped[1:] if stripped[:1] in ("+", "-") else stripped + + mantissa, sep, exponent = body.partition("e") if "e" in body else body.partition("E") + if sep and not exponent.lstrip("+-").isdigit(): + return None + + has_decimal = "." in mantissa + int_part, _, frac_part = mantissa.partition(".") + if not (int_part.isdigit() or frac_part.isdigit()): + return None + if int_part and not int_part.isdigit(): + return None + if frac_part and not frac_part.isdigit(): + return None + + return int_part, frac_part, has_decimal + + +def _count_sig_figs(int_part, frac_part, has_decimal): + digits = int_part + frac_part + first_nonzero = next((i for i, d in enumerate(digits) if d != "0"), None) + if first_nonzero is None: + return 1 + + trimmed = digits[first_nonzero:] + if has_decimal: + return len(trimmed) + return len(trimmed.rstrip("0")) or 1 + + +def _result(is_correct, real_diff, allowed_diff, feedback=""): + return { + "is_correct": bool(is_correct), + "real_diff": real_diff, + "allowed_diff": allowed_diff, + "feedback": feedback, + } + + +def _evaluate_sig_figs(response, answer, sig_figs): + parts = _split_numeric_string(response) + if parts is None: + return _result(False, None, None, "Please enter a number.") + + response_value = float(response) + rounded_answer = _round_to_sig_figs(answer, sig_figs) + rounded_response = _round_to_sig_figs(response_value, sig_figs) + numeric_correct = abs(rounded_response - rounded_answer) <= spacing(abs(rounded_answer)) + + precision_correct = response_value == 0 or _count_sig_figs(*parts) == sig_figs + is_correct = numeric_correct and precision_correct + + feedback = "" + if numeric_correct and not precision_correct: + feedback = f"Please give your answer to {sig_figs} significant figures." + + return _result(is_correct, abs(response_value - answer), None, feedback) + + +def _evaluate_tolerance(response, answer, relative_tolerance, absolute_tolerance): + allowed_diff = absolute_tolerance + relative_tolerance * abs(answer) + spacing(answer) + + if not _is_number(response): + return _result(False, None, allowed_diff, "Please enter a number.") + + real_diff = abs(response - answer) + + return _result(real_diff <= allowed_diff, real_diff, allowed_diff) + + def evaluation_function(response, answer, params) -> dict: """ Function used to grade a student response. @@ -21,30 +106,27 @@ def evaluation_function(response, answer, params) -> dict: to output the grading response. """ + sig_figs = params.get("significant_figures", params.get("sig_figs")) relative_tolerance = params.get("relative_tolerance", params.get("rtol", 0)) absolute_tolerance = params.get("absolute_tolerance", params.get("atol", 0)) - if not (isinstance(answer, int) or isinstance(answer, float)): - raise Exception("Answer must be a number.") - - allowed_diff = absolute_tolerance + relative_tolerance * abs(answer) - allowed_diff += spacing(answer) + uses_tolerance = any( + key in params + for key in ("relative_tolerance", "rtol", "absolute_tolerance", "atol") + ) - if not (isinstance(response, int) or isinstance(response, float)): - return { - "is_correct": False, - "real_diff": None, - "allowed_diff": allowed_diff, - "feedback": "Please enter a number.", - } + if sig_figs is not None and uses_tolerance: + raise Exception( + "significant_figures/sig_figs cannot be used together with " + "relative_tolerance/rtol or absolute_tolerance/atol." + ) + if sig_figs is not None and (not isinstance(sig_figs, int) or sig_figs < 1): + raise Exception("significant_figures must be a positive integer.") - real_diff = abs(response - answer) - is_correct = bool(real_diff <= allowed_diff) + if not _is_number(answer): + raise Exception("Answer must be a number.") - return { - "is_correct": is_correct, - "real_diff": real_diff, - "allowed_diff": allowed_diff, - "feedback": "", - } + if sig_figs is not None: + return _evaluate_sig_figs(response, answer, sig_figs) + return _evaluate_tolerance(response, answer, relative_tolerance, absolute_tolerance) diff --git a/app/evaluation_test.py b/app/evaluation_test.py index fda062e..b69dfd6 100644 --- a/app/evaluation_test.py +++ b/app/evaluation_test.py @@ -274,5 +274,323 @@ def test_full_param_incorrect(self): self.assertEqual(response.get("is_correct"), False) + def test_sig_figs_correct(self): + body = { + "response": "3.14", + "answer": 3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_incorrect(self): + body = { + "response": "3.15", + "answer": 3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + + def test_sig_figs_too_many_digits(self): + body = { + "response": "3.14159", + "answer": 3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + self.assertEqual( + "significant figures" in response.get("feedback"), True) + + def test_sig_figs_too_few_digits(self): + body = { + "response": "3.1", + "answer": 3.10, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + self.assertEqual( + "significant figures" in response.get("feedback"), True) + + def test_sig_figs_negative_numbers(self): + body = { + "response": "-3.14", + "answer": -3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_zero_answer(self): + body = { + "response": "0", + "answer": 0, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_trailing_decimal_zeros_significant(self): + body = { + "response": "92.00", + "answer": 92, + "params": { + "sig_figs": 4 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_leading_zeros_not_significant(self): + body = { + "response": "0.0032", + "answer": 0.0032, + "params": { + "sig_figs": 2 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_whole_number_trailing_zeros_not_significant(self): + body = { + "response": "540", + "answer": 540, + "params": { + "sig_figs": 2 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_whole_number_trailing_zeros_not_significant_wrong_count(self): + body = { + "response": "540", + "answer": 540, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + + def test_sig_figs_explicit_decimal_point_trailing_zeros_significant(self): + body = { + "response": "540.", + "answer": 540, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_scientific_notation(self): + body = { + "response": "5.02e4", + "answer": 50200, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_response_not_string(self): + body = { + "response": 3.14, + "answer": 3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + self.assertEqual("Please enter a number." in response.get("feedback"), True) + + def test_sig_figs_dangling_exponent_marker(self): + body = { + "response": "3.14e", + "answer": 3.14159, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + self.assertEqual("Please enter a number." in response.get("feedback"), True) + + def test_sig_figs_invalid_value(self): + body = { + "response": 3.14, + "answer": 3.14159, + "params": { + "sig_figs": 0 + }, + } + + self.assertRaises( + Exception, + evaluation_function, + body["response"], + body["answer"], + body["params"], + ) + + def test_sig_figs_invalid_type(self): + body = { + "response": 3.14, + "answer": 3.14159, + "params": { + "sig_figs": 3.5 + }, + } + + self.assertRaises( + Exception, + evaluation_function, + body["response"], + body["answer"], + body["params"], + ) + + def test_sig_figs_and_tolerance_mutually_exclusive(self): + body = { + "response": 3.14, + "answer": 3.14159, + "params": { + "sig_figs": 3, + "atol": 0.1 + }, + } + + self.assertRaises( + Exception, + evaluation_function, + body["response"], + body["answer"], + body["params"], + ) + + def test_sig_figs_and_tolerance_mutually_exclusive_long_names(self): + body = { + "response": 3.14, + "answer": 3.14159, + "params": { + "significant_figures": 3, + "relative_tolerance": 0.1 + }, + } + + self.assertRaises( + Exception, + evaluation_function, + body["response"], + body["answer"], + body["params"], + ) + + def test_sig_figs_synonym_key(self): + body = { + "response": "3.14", + "answer": 3.14159, + "params": { + "significant_figures": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), True) + + def test_sig_figs_non_numeric_response(self): + body = { + "response": "two", + "answer": 3.14, + "params": { + "sig_figs": 3 + }, + } + + response = evaluation_function(body['response'], body['answer'], + body.get('params', {})) + + self.assertEqual(response.get("is_correct"), False) + self.assertEqual("Please enter a number." in response.get("feedback"), True) + + def test_sig_figs_non_numeric_answer(self): + body = { + "response": 3.14, + "answer": "pi", + "params": { + "sig_figs": 3 + }, + } + + self.assertRaises( + Exception, + evaluation_function, + body["response"], + body["answer"], + body["params"], + ) + if __name__ == "__main__": unittest.main() diff --git a/readme.md b/readme.md index 6d08dbb..b8d5c53 100644 --- a/readme.md +++ b/readme.md @@ -1,6 +1,6 @@ # IsSimilar -This function checks whether a student's numeric response is within an acceptable tolerance of the correct answer, using absolute (`absolute_tolerance`) and relative (`relative_tolerance`) tolerance parameters. The comparison follows the formula: `|response - answer| ≤ absolute_tolerance + relative_tolerance × |answer|`. By default both tolerances are 0, requiring an exact match (within floating-point precision). +This function checks whether a student's numeric response matches the correct answer, using either absolute (`absolute_tolerance`) and relative (`relative_tolerance`) tolerance parameters, or a `sig_figs` significant-figures check. The tolerance comparison follows the formula: `|response - answer| ≤ absolute_tolerance + relative_tolerance × |answer|`. By default both tolerances are 0, requiring an exact match (within floating-point precision). Alternatively, `sig_figs` requires both the response's value and its written precision to match the answer to N significant figures (the response must be given as a string, e.g. `"92.00"`, so trailing zeros and decimal points are preserved); it cannot be combined with `absolute_tolerance`/`relative_tolerance`. For more information, look at the docs in `app/docs/`.