Smarter Judging in the Smart Cities Challenge: Briging Rigour to Major Award Processes
Using manual tools to evaluate large amounts of feedback can be time-consuming and prone to errors. Infrastructure Canada used Ethelo to reduce staff evaluation time and provide a place where evaluators could interact with one another during the review process for the Smart Cities Challenge. Ethelo allowed the project’s 13 jury members to quickly and easily log comments on each proposal and gave Infrastructure Canada a solid, quantified foundation for its deliberation meeting to decide which proposals would advance in the competition.
$75M
in Grants Awarded
17 different
award criteria
Manager, Smart Cities Canada
The Smart Cities Challenge is a new way the Government of Canada is helping communities become healthier, more inclusive and livable. By challenging communities to use data and connected technologies to address their most pressing problems, the Smart Cities Challenge encourages innovation through competition.
The competition awarded one top prize of up to $50 million, two prizes of up to $10 million each, and one prize of up to $5 million. All cities could apply to the $50 million category, but the other two were restricted by city size.
Of the 130 proposals received, 46 cities applied for $5 million, 71 applied for $10 million, and 13 applied for $50 million. Each group was evaluated by a panel of government officials and other professionals with experience in infrastructure and public projects.
Each proposal needed to be evaluated quickly, but thoroughly, with a goal of selecting 20 to advance to the next round of competition. Ethelo provided the framework necessary to make this happen in an innovative way — much like the ideas being evaluated in the Smart Cities Challenge.
Following a brief initial consultation, Ethelo and the Smart Cities Canada team worked to set up an evaluation platform for 13 jury members based on 17 criteria, including the potential for economic growth and commitment to a healthy environment.
Kathleen Bouchard, a senior analyst on the Smart Cities Challenge, said the team did not know how many proposals they would receive and wanted a tool that could handle 20 just as easily as it could handle 200. The team also wanted a tool that was flexible enough to accommodate all 17 criteria. They found that flexibility in Ethelo.
Using Ethelo, jury members viewed each proposal in their assigned category and made comments on how well it adhered to each of the criteria. Rather than working individually, Ethelo allowed reviewers to see each other’s comments and provided the opportunity for group interaction.
The Smart Cities Canada team monitored progress throughout the evaluation period and communicated with evaluators if their scores and comments appeared to be out of line with the others.
“We were able to quickly calibrate who judged low and who judged high,” Bouchard said. “If we noticed a discrepancy, we could call [the specific judge] back about rating scale. It was definitely a time saver on our side.
Based on feedback gathered in Ethelo, the jury members held a meeting to choose 20 finalists, who each received $250,000 to develop final proposals that outlined the design, planning, and other elements of their proposed ideas. The winners in each category were then announced in May 2019.
Bouchard said Ethelo saved the project team many, many hours of work compiling individual feedback into a combined evaluation. It also greatly reduced the potential for error and allowed the team to make sure everyone completed the evaluation before the deliberation meeting.
“It would have been hours of work trying to compile this information if we used a spreadsheet or some other method of evaluation,” Bouchard said. “We also eliminated the error rate of tabulating with spreadsheets and could keep track of who was doing the work.”
Infrastructure Canada will be running two additional competitions in the future. Bouchard said they are very likely to use Ethelo again for evaluations based on the success of the Smart Cities Challenge.
“Ethelo proved to be a valuable tool in the initial evaluation of applications to the Smart Cities Challenge in 2018. Using Ethelo’s straightforward and user-friendly platform as part of their evaluation toolbox, the independent Jury was able to systematically apply assessment criteria to a diverse range of submissions and helped them to work toward consensus on their recommendation of program finalists.”
Amanda Aizlewood, Manager, Smart Cities Canada
