Can we publish our build model in the challenge competition in a scientific journal ?
Are we allowed to ‘use’ the model in previous challenges by other groups? I presume of course we cannot just replicate the best model in previous year as our model in this year, but how about using similar techniques etc? Let’s say I build a decision model that decides whether the subject should use the 1st model or 2nd model in the last challenge to generate a prediction, would it be acceptable? What would be the rule on this?
Can we use any other predictor/data that is not in the provided data?
I didn’t quite understand what happened when a group submitted more than 1 model, even after listening to the zoom talk.
I remember when there are multiple models for one task, only the best model would count?
Can a group build a single model for each prediction task? If so, how would it be evaluated?
Thank you for your great questions. Below are our responses:
Publishing your model: Yes, after the third challenge concludes on November 22nd and the results are publicly announced, you are free to publish your developed model and findings in a scientific journal. We encourage participants to share their models and insights with the broader scientific community.
Using models from previous challenges: Contestants can use models from the first and second challenges as inspiration or apply similar techniques, provided they do not simply replicate them. Developing a new model that builds on past methods or includes decision-making processes, such as selecting between previous models based on specific conditions, is permissible. While no strict rules exist for reusing these models, we encourage innovation rather than exact replication.
Contestants may use external datasets or predictors when building and evaluating their models. However, since it is important to disclose these details in the third challenge manuscript, we kindly request that they describe their approach to incorporating external data and predictors to ensure clarity for others. We will email all contestants at the end of the challenge while drafting the third challenge manuscript to request details of their models.
Regarding multiple models for prediction tasks: a) If you prefer to build an independent model for each task, that is perfectly acceptable. For instance, if you attempt all six tasks, you will have six models, each selected by your team based on certain criteria. These models would then be combined into one submission file and submitted on the CMI-PB website. The CMI-PB team will evaluate the most recent submission file you provide before the challenge deadline. Each model’s performance will be evaluated individually, and your overall score will be based on how well your models perform across all tasks. b) If you employ multiple modeling approaches and wish to make distinct submissions for each one, we recommend creating separate CMI-PB accounts to manage these entries. For those using more than three modeling strategies, which may complicate submission management, we encourage coordinating with the CMI-PB team to submit entries offline.
I hope this answers your questions. Feel free to reply to this thread if you need further clarification.