Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent behavior with mcq questions allowing multiple correct answers #2587

Open
Incogdino opened this issue Jan 24, 2025 · 6 comments · May be fixed by #2591
Open

Inconsistent behavior with mcq questions allowing multiple correct answers #2587

Incogdino opened this issue Jan 24, 2025 · 6 comments · May be fixed by #2591

Comments

@Incogdino
Copy link
Contributor

Please confirm that you have searched existing issues in the repo

Yes, I have searched the existing issues

Any related issues?

No response

Tell us about your environment

Windows 11 Pro

MarkBind version

5.5.3

Describe the bug and the steps to reproduce it

When browsing through the user guide for adding mcq quesitons, I chanced upon a section that allows for multiple correct mcq questions.

Image

The source code for the question is reproduced below:

<question type="mcq" header="Which of these **contradicts** the heuristics recommended when creating test cases with multiple inputs?">
  <!-- Insert the reason for the option using the reason attribute -->
  <q-option reason="This is **correct**. We need to figure out if a positive test case works!">
    Each valid test input should appear at least once in a test case that doesn’t have any invalid inputs.
  </q-option>
  <q-option>
    It is ok to combine valid values for different inputs.
  </q-option>
  <q-option>
    No more than one invalid test input should be in a given test case.
  </q-option>
  <!-- Use the 'correct' attribute to indicate an option as correct. -->
  <q-option correct>
    All invalid test inputs must be tested together.
    <!-- Optionally, you may use a reason slot instead of a reason attribute. -->
    <div slot="reason">
    If you test all invalid test inputs together, you will not know if each one of the invalid inputs are handled
    correctly by the SUT.
    This is because most SUTs return an error message upon encountering the first invalid input.
    </div>
  </q-option>
  <div slot="hint">
  How do you figure out which inputs are wrong? (or correct)
  </div>
</question>

The first option is indicated as a correct option but lacks correct as a parameter

Expected behavior

According to the tip below, mcqs can have multiple correct answers. Does that mean that the first option would need to be correct as well?

I noticed in the source code above that the first option is not indicated as a correct option.

A sample of how to the correct output is shown below (ignore the qn in the heading):

Image

Anything else?

No response

@damithc
Copy link
Contributor

damithc commented Jan 24, 2025

@Incogdino The bug is in the UG example, or in the feature itself?

@lhw-1
Copy link
Contributor

lhw-1 commented Jan 25, 2025

After reading this section through, it seems that the feature itself is not the issue, rather the way we provide the examples is slightly confusing.

The first option is indicated as a correct option but lacks correct as a parameter

The question for this example is asking for: which of the options contradicts the heuristics recommended?

The idea is that the first option is correct, meaning that it does not contradict the heuristics recommended, hence not the correct option to be chosen here. The question is intended to be single-answer, with the 4th option being the correct answer.

The tip (for multiple correct answers) that follows the question is standalone, and there is no example provided for this.

So the UG example is also actually correct. The issue is not in the correctness here, but in the wording. I think we just need to make the following documentation changes:

  1. Re-word the reasoning for the first option from "correct" to "does not contradict the heuristics recommended".
  2. Provide another example of a question with multiple correct options.

@Incogdino
Copy link
Contributor Author

@Incogdino The bug is in the UG example, or in the feature itself?

I was referring to the UG example sorry for the confusion. The feature works as expected here.

@Incogdino
Copy link
Contributor Author

@lhw-1

  1. Re-word the reasoning for the first option from "correct" to "does not contradict the heuristics recommended".

I agree that this would be better if it was rephrased.

  1. Provide another example of a question with multiple correct options.

Would it be better if this was left as a tip instead? Though allowed, according to proper heuristics, if there are multiple correct answers, users should choose to use a checkbox question instead of a multiple choice question.

@lhw-1
Copy link
Contributor

lhw-1 commented Jan 27, 2025

Would it be better if this was left as a tip instead? Though allowed, according to proper heuristics, if there are multiple correct answers, users should choose to use a checkbox question instead of a multiple choice question.

I don't think this is true for all cases - I would consider an MCQ question with multiple correct answers vs. a Checkbox question with multiple correct answers to be two different use cases. (The first case might be useful if a question setter wants to be more "lenient" in a textbook quiz)

Leaving it as a tip is likely not going to cause huge problems per se, but given that (1) this issue was raised since the tip was initially thought to refer to the example above the tip, and (2) we can have more examples as needed (we are doing it for the Fill-in-the-Blanks Questions section), it might be worthwhile to at least clear up the potential confusion.

If providing an example seems to be overkill, we can expand / reword the tip so that there is no ambiguity, i.e. make it clear that the example shown is meant to be a single-answer question.

@Incogdino
Copy link
Contributor Author

Would it be better if this was left as a tip instead? Though allowed, according to proper heuristics, if there are multiple correct answers, users should choose to use a checkbox question instead of a multiple choice question.

I don't think this is true for all cases - I would consider an MCQ question with multiple correct answers vs. a Checkbox question with multiple correct answers to be two different use cases. (The first case might be useful if a question setter wants to be more "lenient" in a textbook quiz)

Leaving it as a tip is likely not going to cause huge problems per se, but given that (1) this issue was raised since the tip was initially thought to refer to the example above the tip, and (2) we can have more examples as needed (we are doing it for the Fill-in-the-Blanks Questions section), it might be worthwhile to at least clear up the potential confusion.

If providing an example seems to be overkill, we can expand / reword the tip so that there is no ambiguity, i.e. make it clear that the example shown is meant to be a single-answer question.

Thanks for sharing your perspective. I see your point about MCQ and Checkbox questions serving different purposes, especially in cases where flexibility in grading is desired.

Given that the original concern stemmed from potential ambiguity in the tip, I agree that we should ensure clarity and should go along with adding an example to avoid ambiguity while still showing that it is possible to do so.

@Incogdino Incogdino linked a pull request Jan 29, 2025 that will close this issue
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants