In developing the Orla service model we understood the imperative of actually demonstrating that a change to the Orla would deliver value to both the people who adopt it and the organizations they work for. We therefore commissioned Phil Graves of www.philipgraves.net
to develop a stand-up survey methodology which is used during the Phase 1 Pilot part of our service delivery model.
The Orla survey methodology involves questionnaires that cover workers' own definitions of effectiveness. The aim is to understand some of the more subjective measures that people use to gauge their own performance in the key areas of efficiency and effectiveness.
To ensure the statistical integrity of this measurement exercise, qualitative research was undertaken, with two intentionally broad objectives:
How do Outlook and email impact on workers' efficiency and effectiveness, and the demand for their time? Put another way: If a product were to make their email life easier, how would they know it had worked?
Whilst the aim of the research was to develop questions that would help us to measure the impact of Orla, our approach was to analyse people's behaviour, thoughts and feelings rather than to question respondents more directly (since their post-rationalised answers are rarely accurate).
'Efficiency' and 'effectiveness' are defined as follows:
Face-to-face in-depth interviews were conducted with people who worked in the offices of large or medium-sized organisations. All were users of email and worked in environments where email communication was frequent. The sample included a cross-section of ages (from 23 to 50), and both sexes were represented. By necessity, all interviewees were based in the South-east of England.
In addition, the behaviour and thoughts of people were solicited by email (again using very open questions) to explore the extent to which the attitudes identified in the in-depth interviews were prevalent more widely. These surveys included respondents in the UK, the US and Canada.
Not surprisingly, since the computer, the Internet, Outlook and even many corporations are global entities, no significant differences across regions were found. Consequently, whilst cultural differences in communication undoubtedly exist between countries (as they do between organisations), the results are believed to be applicable in all English-speaking countries.
A number of themes emerged consistently in the research:
An 'efficient day' almost always involves a visual indicator of work that has been completed: either an empty inbox or a completed task list (the latter being considered more satisfying, since the inbox will refill outside of their control).
Most people felt that email controls their working day (with work and social emails having a similar impact) – it is distracting and demanding.
For most people, being in control of emails was synonymous with good time-management generally. When offered a 'magic wish' for email, a frequent request was that emails would organise themselves.
People also frequently retained emails for the purpose of covering their own back.
Some respondents reported that there was an expectation within their organisations that emails would be met with an almost instant response, and that it was annoying when this did not occur. Typically, these people made no use of priority flags on the emails they sent.
Based on the lessons we learned from this research, we devised the Orla Survey System, as follows:
- Day 1 Survey Before Orla Essentials Learning, taking in:
- Efficiency
- Email Volume
- Frequency
- Learning Development
- Day 1 Survey Immediately After Orla Essentials Learning, taking in:
- Feedback on Learning
- Feedback on Orla
- Learning More
- Worker Comments
- Learning Folio, taking in the 14-day Orla Essentials Learning Experience:
- The extent to which the three key organised workload characteristics were covered
- The extent to which the 4D decision-making methodology has been mastered
- Learning time engaged
- Change considerations generally
- Orla change considerations specifically
- Worker Comments
- Orla Advanced programmes favoured
- Day 15 Survey After Orla Essential Learning, taking in:
- Efficiency
- Email Volume
- Frequency
- Using Orla
- Value
In a future post (soon) I shall share with you some of the outcomes of Orla using these metrics. Very provocative stuff!
Posts
