Solution/answer template

[arcrose]

One of the big features the quiz system could have is a simple interface for describing questions and answers. @Gudahtt talked about building a feature that would generate new questions automatically given some specification. Before that, we should discuss the design of a simple format for specifying questions and answers. An obvious starting place is to use JSON, but we still need to think about what kind of information we need to keep track of.

[Manouchehri]

That's part of the reason I was suggesting python, that would allow a lot of freedom for creating new questions. In my example of MATH 1000, SymPy could be used create equations that are solvable. For entering an answer, we would need an HTML5/JS based form. (Flash isn't really an option anymore, a lot of us do these quizzes in study rooms on our mobile devices.) Anyone know of something that's already available? Obviously forcing the user to enter in a LaTeX format for equations would be the easiest, but we can't expect students to know that (useful skill though).

A fun improvement with python is that would could generate random graphs, something that isn't commonly done in the current off the shelf systems. Heck, we could even create interactive 3D models on the spot.

I know some instructors might not know python (which in STEM you really should know..), but those aren't the people who are going to go the extra mile to create any questions except the default anyway. They could simply select from previously created questions.

[arcrose]

There are a lot of ways we could allow students to answer questions once we know how we are to represent the questions and answers.
For example, we can:

• Allow for multiple-choice questions so that students don't have to write anything.
• Provide a specification for answer components that would translate to images/buttons that students could press to add that component to their answer (e.g. an integral button would add an integral symbol)

And so on...

However, before thinking about the UI, we need to know how this is going to work behind the scenes. The question/answer format spec needs to be generic but also easy to understand so that new data can be created and managed easily. That's what this issue exists to discuss.

[Gudahtt]

@redwire I can't recall discussing the "automatically generate questions from a specification" part. That sounds ambitious. But we definitely need question/answer templates.

I suppose, for the 'question' part, we'd just use a blank text area, with support for some sort of markup (i.e. a way to add images, lists, mathematical equations, etc.). And each question would have a number, letter, or some sort of identifier.

Then the quiz author would need to specify what type of question it was (i.e. multiple choice, short/long answer, fill in the blanks (with or without "word bank"), etc.). They would need to supply additional information for some of those question types. And we'd have the option of specifying the valid input types and ranges whenever possible, to do some validation (e.g. they could specify certain blanks need to be integers, strings, floats, etc., or that it must be positive).

[Manouchehri]

We may be overestimating the involvement that an instructor would want.
Probably the largest reason people use Pearson Mastering and similar tools
is because they don't have the time to create questions themselves.

In reality, it's going to be a small group of people who create questions
for the masses. I feel like the feature set should be more important than
the usability here (within reason), because those who are willing to create
questions are probably willing to learn new tricks too. Ideally a TA with
programming experience would be the one to set up a course (might be
dreaming for too much here).

That's why I mentioned python earlier, I was thinking we could give the
creator access to program their own questions. Obviously we'd provide
templates and suggestions, but this would give them in theory the ability
to create any course specific question.

On Mon, Oct 27, 2014 at 2:22 PM, Mark Stacey [email protected]
wrote:

@redwire https://github.com/redwire I can't recall discussing the
"automatically generate questions from a specification" part. That sounds
ambitious. But we definitely need question/answer templates.

I suppose, for the 'question' part, we'd just use a blank text area, with
support for some sort of markup (i.e. a way to add images, lists,
mathematical equations, etc.). And each question would have a number,
letter, or some sort of identifier.

Then the quiz author would need to specify what type of question it was
(i.e. multiple choice, short/long answer, fill in the blanks (with or
without "word bank"), etc.). They would need to supply additional
information for some of those question types. And we'd have the option of
specifying the valid input types and ranges whenever possible, to do some
validation (e.g. they could specify certain blanks need to be integers,
strings, floats, etc., or that it must be positive).

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[Gudahtt]

Ideally a TA with programming experience would be the one to set up a course (might be dreaming for too much here).

Definitely dreaming for too much. There's no reason to require quizes to be written programmatically; it wouldn't be easier for us or the quiz authors. That sounds painful. It might be a nice feature... but definitely not the typical case.

The users of this software would be quiz authors (whether they're TA's, profs, volunteers, etc.) and quiz takers. We don't care who the instructor is or how involved they are; only that the quiz author is able to author a quiz.

Though, the 'program your own question' suggestion brings up a few interesting questions...

Lets say that a quiz is a series of questions and answers. We know the responsibility of the quiz is to asses the competency of the quiz taker and assign a grade determined by the number of questions correctly answered. What are the resposibilities of the questions and answers?

My first thought would be that a question has to have a description, an id, and... a format, I guess? i.e. whether it's multiple choice, fill-in-blank, etc. And then it needs a way to verify the correctness of the answer given. It also needs a weigh, i.e. the number of 'points' it's worth compared to the other questions.

[arcrose]

@Gudahtt My bad, it was actually Andrew whom I was talking to in the lab one time that suggested a feature for automatically generating questions and solutions. I don't think it's as feasible as he seems to think it is, but I don't think he was really thinking about the general case either.

[Manouchehri]

I don't see why that wouldn't be possible, isn't that how all the current
systems work?

On Sat, Nov 1, 2014 at 7:10 PM, Zack Mullaly [email protected]
wrote:

@Gudahtt https://github.com/Gudahtt My bad, it was actually Andrew whom
I was talking to in the lab one time that suggested a feature for
automatically generating questions and solutions. I don't think it's as
feasible as he seems to think it is, but I don't think he was really
thinking about the general case either.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[arcrose]

No. They currently select a random subset of questions from a pool of questions.

[Manouchehri]

That sounds time consuming to create.

On Mon, Nov 3, 2014 at 7:25 PM, Zack Mullaly [email protected]
wrote:

No. They currently select a random subset of questions from a pool of
questions.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[Gudahtt]

It's pretty straightforward though.

Automatically generating questions? Not so much.

For starters, what would they be generated from?

[arcrose]

Yeah I don't think generating problems is really feasible. Creating the pool manually is certainly not the easiest thing to do, either. I suppose we could just grab a bunch of problems for different courses from various textbooks, though.

[Manouchehri]

It wouldn't be that bad with a math/physics course, or anything number
related. Just put some restrictions on the number range, and generate
questions based off that.

On Mon, Nov 3, 2014 at 8:03 PM, Zack Mullaly [email protected]
wrote:

Yeah I don't think generating problems is really feasible. Creating the
pool manually is certainly not the easiest thing to do, either. I suppose
we could just grab a bunch of problems for different courses from various
textbooks, though.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[arcrose]

You'd still have to go through the process of specifying a generic format for each type of question and writing a solver to check generated solutions.

[Manouchehri]

Indeed. I'm getting more off track on a MOOC idea than a quiz system here.

On Mon, Nov 3, 2014 at 8:07 PM, Zack Mullaly [email protected]
wrote:

You'd still have to go through the process of specifying a generic format
for each type of question and writing a solver to check generated solutions.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

[Gudahtt]

It wouldn't be that bad with a math/physics course, or anything number related. Just put some restrictions on the number range, and generate questions based off that.

That doesn't sound too bad actually. I was thinking of something more general than that.

A simple question-generator that deals with numerical ranges like that would be a nice feature, and could probably be worked on independantly from the rest of the project. Sounds promising.