I think Google Forms might be the best option in terms of being able to shuffle the questions or options in order to mitigate primacy and recency effects. (People won’t have anything to compare the first car to, so it’s more likely to get an average score)
Idk if I’m taking it too seriously but, seeing the amount and time people poured into designing their cars, I believe everyone deserves a fair evaluation. I’m not saying this for my own design, because there are far more beautiful cars and I don’t expect to be competitive on this front. However, if we only have a photo gallery where people can rate the cars they like, simple designs with muted colors might be overlooked due to not having so striking thumbnails.
About processing, I’m sure some people are willing to help you out by volunteering their time. (If you are willing to cooperate / accept help from the participants. Maybe there are non-participants in the wider community (discord or the forum) who’d like to help, who knows)
I can help with the survey design and uploading the images to Google Forms.
Also a bit about the survey design and questions
1 - Ranking each design from 1-10:
Pro: A simple solution that can be done on Google Forms with question order randomization.
Con: It will be hard to distinguish mid-rated designs since the answers will likely follow a normal distribution, many mid range designs will share the same score
2 - Preference order ranking (participants rank the entire list from 1 to 245 in the order of their preference)
Pro: Will represent the communities common preference more accurately
Con: Google Forms doesn’t support it. I’m not sure how well the free tiers of other platforms are suitable for this. I don’t think people will sort and fine tune their preference across 245 cars
3 - Having an online gallery where people rate as many or as few images as they want
Pro: Least effort intensive solution
Cons: Number of the votes per entry might skew towards best and worst perceived designs, having fewer votes per mid range entries. Also not having the same amount of votes per entry might mean that designs with fewer votes can over or underperform significantly if we’re comparing averages. Access restriction might be an issue, people inside or outside the community can spam ratings.
Other solutions have their own issues
- sub-set preference (e.g. where users pick their top 10) will leave many images unranked while having the same issue as the gallery one. Designs that are included in top tens fewer times can over or underperform significantly if we’re comparing averages
- pair-wise preference (where users pick one preference out of two displayed, for many pairs, like a sports league). This is impossible due to number of combinations (29K+
) If we do a sub-set like having people pair-rank 20-50 pairs, might result in a similar problem like gallery or sub-set preference - Tournament / knock-off / bracket voting (where choices are randomly matched and participants select their preference from each pair) requires the voting to be done in several steps (if we have 256 entries, 8 steps) Plus if we don’t have 2^n entries, some would have to start from the second round.
Sorry for a very long post. I tried to present some ideas based on my survey/data gathering experience and issues I encountered irl, with the hopes of helping Der Bayer/community to decide. If I had to pick one I guess I’d go with 1-10 scale ranking, although not ideal, it seems to present a balance between feasibility/ease of application, accuracy and fairness.
I can volunteer some time regardless of which option is picked.
