It is common to end an assessment with establishing the level of certainty in the conclusion. This can be formulated as a expert judgement of a categorical statement expressed as the subjective probability of the statement being true. Although this sounds simple, there are few evaluated protocols for expert elicitation on categorical questions. In practice, there are protocols combing verbal and quantitative expressions of certainty, using summaries of confidence and likelihood with an unclear relationship, allowing for experts to express their probability judgement with bounds, and procedures to aggregate expert judgements. This year I will start a research project with the aim to develop and test a protocol for Expert Knowledge Elicitation (EKE) of categorical questions that can be used for panel-based assessments. The project is funded by the Swedish research council FORMAS.
Project page https://portal.research.lu.se/en/projects/development-of-a-protocol-for-expert-elicitation-of-conclusions-i/
I currently have a call out for a PhD student to work with expert elicitation and Bayesian modelling as part of this project. https://lu.varbi.com/what:job/jobID:936946/