Experience Evaluation Process

Before conducting an expert evaluation, we can organize the process into four key stages: preparation, expert selection, evaluation, and summary.

Preparation Phase: First, clearly define the evaluation content and complete the evaluation checklist. The evaluation points are not fixed, and we need to adjust the checklist based on the current project’s needs and progress, including adding, removing, or splitting content. If we plan to present the evaluation results in a table format, we can also add a scoring function to quantify the evaluation of each item.

Finding Relevant Experts: The evaluation subjects should have substantial experience in the relevant field, so finding the right experts is crucial. We can contact professionals from the industry or game developers whose work we admire. Ideally, the number of experts should be between 2 and 6. Too few experts may result in overly subjective results, while too many may affect the evaluation efficiency and disrupt the process.

Evaluation Phase: Start with individual evaluations. Experts can be placed in separate rooms to conduct independent evaluations, which reduces mutual influence and allows for feedback from different perspectives. If it’s not possible to provide separate rooms, the experts can be asked not to communicate during the evaluation to minimise interference.After individual evaluations, a group discussion phase follows.

At this stage, the experts will exchange their views on each evaluation point. Since each expert makes judgments based on their experience, disagreements are common. When disagreements occur, as the organiser, we can ask the experts to elaborate on their points and ensure the discussion continues smoothly without excessive arguments or emotional disagreements.

Summary Phase: After the experts leave, we need to summarise their opinions and suggestions and formulate a reasonable plan for subsequent modifications. For many experimental games, developers sometimes intentionally step outside the experience evaluation framework to create a “non-traditional” uncomfortable experience, which could be one of the unique features of the game.