Item Development

With the objectives finalized, the next stage is to write the questions that will appear on the exams (note that within the world of exam development, a question is referred to as an item).

Unlike the steps in the other major developmental phases, the steps within item development were conducted in parallel. That is, items flowed from one step to another as they came in.

Security is a major concern in item development. It is important the entire item pool be kept as confidential as possible. As a result, everyone participating in this phase was required to agree not to disclose item content to anyone and to sign non-disclosure agreements. Additional security precautions were taken as well.

Item Writing

The procedure used to develop the items for most IT certification exams is to fly a group of subject-matter experts (SMEs) into a certain location for a week or more, give them training in how to write items, and then just have them work very intensely to create the necessary pool of items.

Owing to the large expense of doing this, as well as a desire to be as inclusive as possible, we elected not to work this way for the Level 1 exams. Instead, a public call was put out across the Internet for item writers in August, 1999. Everyone who was interested and knowledgeable was encouraged to help with item writing. A web-based interface called TIPS was used to collect most items. New items developed since then for exam rotation have been developed in-house by SMEs, contributed online by volunteers, or developed during special item-writing workshops.

During that initial exam development phase, item writers submitted items for each objective. The number of items needed for each objective was determined by the weight value assigned to the objective.

While this method of item collection was effective, and we had over 70 people submit items for consideration, it did significantly lengthen this stage of exam development.

Item Screening

Once the items were submitted, all items were screened by exam development supervisors. Screening focused on three criteria:

• Redundancy: Items that were substantially identical to previously submitted items were rejected. The purpose, in general, was to ensure that each item covers somewhat different content.
• Phrasing: Items that were phrased in confusing or otherwise inappropriate ways were rejected or reworded. Attention was paid to ensuring that items can be understood by non-native English speakers.
• Accuracy: Supervisors were not experts in all areas of Linux, but they were allowed to reject or reword items that were obviously not technically accurate.

At the item screening stage, each item was:

• Rejected,
• Accepted as is, or,
• Accepted after rewording.

Item Technical Review

Items that were accepted in the item screening stage were submitted to a technical review by a panel of Linux experts. We contracted a volunteer group of about 10 experts in Linux to review the items.

The primary criteria on which items were evaluated at the review stage were:

• Correctness: Reviewers ensured that the keyed correct answer is indeed correct.
• Appropriateness of Distractors (for multiple-choice items): Reviewers ensured that the distractor answer choices are incorrect but that they are reasonably plausible.
• Phrasing: Reviewers ensured that the items are worded in appropriate language.
• Relevance to Objective: Reviewers ensured that the item is sufficiently related to the objective it is intended to measure.
• Expected Difficulty

Each item was reviewed by at least two experts. Each expert classified each item as:

• Approved,
• Rejected, or
• Other: Reviewer might suggest rewording the item or might otherwise decline to evaluate it.

Exam Development supervisors collected the reviews. At this stage, each item was:

• Accepted based on consensus: If the reviewers agreed that the item should be approved, the supervisor usually accepted it.
• Rejected based on consensus: If the reviewers agreed that the item should be rejected, the supervisor usually rejected it.
• Accepted after further review: If the reviewers did not agree on the item, the supervisor might accept it, perhaps based on the opinion of an additional reviewer.
• Rejected after further review: If the reviewers did not agree on the item, the supervisor might reject it, perhaps based on the opinion of an additional reviewer.
• Accepted after revision: In some cases, reviewers might suggest rewording the item and the supervisor might accept the item after rewording it.