Daisy Thomas
Vicki Woolgar
The Department for Business and Trade (DBT) has released its final evaluation report on Microsoft 365 (M365) Copilot. This comprehensive report examines the risks and benefits of deploying M365 Copilot across the department, identifying user groups who may benefit more than others.
How it all began: taking part in the M365 Copilot trial
In October 2024, DBT took part in a cross-government trial of M365 Copilot. With 1,000 licences allocated to UK-based colleagues for a 3-month period, the trial aimed to understand:
- use: what DBT staff used M365 Copilot for and how effective the tool was for these tasks
- time savings: if users experienced time savings from M365 Copilot
- satisfaction: how satisfied users were with M365 Copilot
- accuracy: if M365 Copilot was able to produce high quality, accurate outputs
The Monitoring and Evaluation team in Digital, Data and Technology (DDaT) designed an evaluation methodology to capture data throughout the trial. The aim was to increase the department’s understanding of the risks and benefits of M365 Copilot.
The primary evaluation method was a diary study to collect detailed data on use cases, satisfaction, accuracy and time savings. The Monitoring and Evaluation team also gathered data through qualitative interviews, observed tasks and built a live usage dashboard to support the diary study findings.
Key insights at-a-glance
- Overall satisfaction: 72% of respondents to a diary study reporting being satisfied with M365 Copilot.
- Time savings: M365 Copilot reduced the time taken for drafting documents, summarising research, transcribing meetings, searching for information and brainstorming ideas. Most other tasks had small time savings, but generating images, scheduling and producing PowerPoints with M365 Copilot did not save time.
- Most popular use cases: transcribing and summarising Teams meetings, writing and summarising emails, using the M365 Copilot Chat and drafting or editing written documents.
- Satisfaction across use cases: users were most satisfied with M365 Copilot when reviewing code, summarising research, drafting documents and transcribing and summarising Teams meetings. In contrast, users were less satisfied using M365 Copilot to produce PowerPoints and for scheduling.
- Benefits for specific user groups: the evaluation identified benefits for neurodiverse colleagues, colleagues with other disabilities, such as hearing and vision impairments, and non-native English speakers.
- Benefits for specific roles: Colleagues in Human Resources and Commercial may benefit more from M365 Copilot due to it being appropriate for many tasks in these roles. Those in Policy and Legal roles may find M365 Copilot less useful.
Concerns and limiting factors: some users were concerned about the environmental impacts of AI, limiting their use. As a result, the monitoring and evaluation team is conducting environmental impact analysis. Uncertainty about security and acceptable use policies also made users cautious.
Conclusion
The evaluation demonstrated that M365 Copilot has the potential to save time and achieve high satisfaction among users, particularly those with specialised needs. However, the evaluation identified some limitations and the need for training and support to maximise M365 Copilot benefits and mitigate risks. The evaluation findings will feed into a series of internal recommendations and next steps. Further evaluation is being conducted to assess the value for money and environmental impact of M365 Copilot usage in the department.
You can read the full report if you would like to gain a deeper understanding of the findings and implications for the future of AI in DBT.
Read more: Understanding the evaluations role in measuring the impact of AI interventions across government.
Leave a comment