CASE STUDY (MAY 2004)
Evaluating Caloundra City Council’s EDMS classification
Caloundra City Council recently selected an Electronic Document Management System (EDMS) to meet their document and records management needs. Early on in the project, the Council recognised that the success of the EDMS would largely be determined by the usability of the classification scheme (or file structure) they implemented.
The Council knew that deploying a classification with a high level of usability would enable staff to efficiently and effectively retrieve as well as file documents. On the other hand, if the classification did not match the way staff think about their jobs and the documents they produce, the result would be inconsistent filing of documents and a high level of frustration when staff try to search for them. In short, the effectiveness of the EDMS as a business and information management tool would be in jeopardy.
At the time of the EDMS implementation, we were assisting Caloundra City Council with their intranet, this lead to us being asked to coach the project team in evaluating the usability of the EDMS classification. The Council had selected Keyword for Councils as a starting point, as it was developed specifically for local government and is used by Queensland State Archives as the basis for the appraisal of records.
This case study sets out the approach taken to evaluate the Keyword for Councils classification for Caloundra City Council, as well as the findings and recommendations made. The case study also provides a brief definition of Electronic Document Management Systems and an overview of Keyword for Councils.
A usable classification is crucial to the success of an EDMS
About Caloundra City Council
Caloundra City Council is responsible for the management of the Caloundra City district, located in the southeast corner of Queensland and the southern gateway to the Sunshine Coast, one of Queensland’s leading tourist destinations.
Caloundra City has a population of approximately 85,000, and like many local councils in Australia, the Council is responsible for a vast array of activities, everything from water management to eco-tourism and building certification. The Council itself has over 600 staff who are located within the heart of the city as well as throughout the district.
As part of the Council’s corporate plan, there are significant initiatives to improve the Council’s information management systems and processes. Part of this includes the implementation of an EDMS.
Records management at the Council
Caloundra City Council has a well established records management team that centrally manage Council records.
Paper documents are provided to Records Management, who convert the documents to microfiche. The details of the documents are then entered into a records management system, by attaching documents to appropriate files. This process involves classifying each document so that it can be retrieved at a later date.
The records management system has a large subject based classification, running into hundreds of classifications. This is workable when there is a central team classifying documents, but poses problems if some of the document management responsibilities are to be decentralised.
Currently staff are able to access the records management system to look up documents they wish to request. Some staff are frequent users of the system and therefore have a level of familiarity with the current classification. However, there are many staff who do not use the system and rely on Records Management to do the searching for them.
In addition to the records management system, Caloundra City Council uses other software solutions to manage their human resource functions, properties and main council activities. As these systems are core to their business they hold documents that are not in the records management system.
Faced with documents held in multiple systems and no way to manage documents electronically, the Council saw the opportunity to integrate an EDMS with their core business systems. For example, if a customer service officer was to enter an application into the front-office system, a record would automatically be created in the EDMS and populated with the appropriate metadata.
This integration would address the reluctance of staff to enter data into multiple systems and the loss of productivity and data integrity that such double-handling presents.
So, what is an EDMS?
Electronic Document Management Systems enables organisations to store documents electronically, including scanned images and documents created on computers, such as word processing files, spread sheets, and graphics.
An EDMS allows documents to be retrieved, shared, tracked, revised and distributed. It can include workflow encapsulating business rules, and has the facility to capture metadata about documents.
A complete EDMS includes various capabilities, such as document imaging, optical character recognition, and text retrieval.
What is Keyword for Councils?
Keyword for Councils is a thesaurus for local government, designed for use in classifying, titling and indexing council records independent of the technological environment.
It was developed in 2000 by State Records New South Wales and contains a business classification scheme. The classification describes and groups all of the business activities performed by councils, including general administrative activities.
As the business classification scheme is used as the basis for the disposal and retention of records for local government, it has been adopted by Queensland State Archives.
Keyword for Councils classification is hierarchical in nature and contains three levels. The first level of the classification lists council functions, then council activities related to those functions, and finally the subjects related to the activities. The levels are explained more
fully in Table 1 below.
Table 1: Keyword for Councils classification
Lists functions of local government, such as Council Properties.
Lists activities performed within the function.
For example, acquisition and disposal is an activity related to the function of Council Properties.
Defines the subject content for activities.
For example, under acquisition and disposal, the subjects are dedications of land, easements, title deeds and so on.
The subject listings are not designed to include every possible subject descriptor. Rather it is offered as a guide to councils.
When subject descriptors are insufficient for a particular activity, Keyword for Councils recommends that councils add their own. Likewise, subjects tied to an activity can be used with other activities considered appropriate by the council implementing the classification.
Given the three tier classification, a title deed related to the acquisition of a council property would be filed under Council properties/acquisition and disposal/title deeds.
For more information on Keyword for Councils, see http://www.records.nsw.gov.au/publicsector/rk/KFC/kfc.htm
Evaluate the classification with real end-users
Why evaluate Keyword for Councils?
Given that Keyword for Councils was developed specifically for local government, why would a council need to evaluate its suitability for their EDMS? Why not implement the classification as is?
The answer lies in an appreciation for how the classification will be used and by whom. A function based classification is useful and usable for a team of specialised records management staff, but how will staff with no records management experience respond to functions? This is especially the case when the Council is looking at moving from a subject to a function based classification.
With this understanding, the Council had the foresight to evaluate the classification with real end-users to identify any problems staff would have using the classification. Once armed with this knowledge, the Council would be in a position to make informed decisions about modifying the classification in the way that would best meet their needs.
Modifying the classification could involve:
- changing terminology
- adding levels to the classification
- adding or removing activities and subjects, and to a lesser extent keywords.
The Council would also gain insight into whether the setup or design of the EDMS needed to change to support the use of the classification, as well as the type of metadata needed to support the integration of the EDMS with other corporate systems.
Coaching and mentoring role
Rather than the Council engaging us to plan and conduct the evaluation process for them, we worked in a coaching and mentoring role with the project team.
As our expertise lies in evaluating a range of information systems from the user perspective, rather than records management, we worked closely with the Council’s Information Management Unit to come up with the best evaluation approach. We then worked with the Records Manager and a member of the project team to coach them in the approach selected, so that they could conduct the majority of the evaluation internally and have a technique they could apply on future projects.
Finally, we worked with the team to analyse the results of the evaluation and then conducted a final review of their findings and recommendations report.
Using a coaching model was important to the success of the evaluation, as it ensured that Caloundra City Council owned the process and was comfortable with the decisions made at the conclusion of the research.
Look for patterns in staff mental models
In consultation with the project team, we identified the objectives of the evaluation. The objectives were to:
- Validate if Keyword for Councils is both suitable and usable for Caloundra City Council’s needs.
- If Keyword for Councils is appropriate, identify areas of the classification that require modification.
- Identify whether a fourth level of classification is required beyond the three levels of Keyword for Councils. Although Keyword for Councils only allows for three levels, the EDMS selected by the Council allows for a fourth level.
- Identify training and performance support requirements to aid in the implementation of the classification.
It was decided that, during the evaluation, particular attention needed to be paid to the mental models staff employ when retrieving and filing documents. For example, if a staff person is looking to retrieve the Employee Code of Conduct, do they first think of the subject (code of conduct), of the activity (employee conditions), or of the council function (personnel).
Identifying if there is a common pattern in the way staff think would:
- Determine whether Keyword for Councils aligns with staff mental models of filing and retrieving documents.
- Determine the best way for staff to use the EDMS classification. For example, are staff more successful at filing and retrieving documents using a search facility, backed by rich document metadata, or drilling down from the top level of the classification.
- Provide insight into the type of metadata that needs to be captured at the point of entering records into a corporate system so it can be replicated in the EDMS.
Involve staff with varied experience
To assess the suitability of Keyword for Councils, a usability test was selected as the most appropriate evaluation method. A usability test involves end users attempting scenarios using a system, or in this case a classification, while being observed by a facilitator and other stakeholders. Actions and issues are analysed and changes are made as a result (if required).
In the case of the Council, the usability test involved participants being asked for the types of words they would use to file and retrieve certain documents, as well as where they would file the same documents using level one of Keyword for Councils (functions).
The usability test did not involve staff using the actual EDMS system, as it was not yet available to the project team and we wanted the evaluation to focus on the classification rather than the design of the EDMS.
Detailed below is the approach taken to recruit participants, create scenarios and conduct the usability testing sessions.
Approximately 20 staff from various areas of the Council were recruited to participate in the usability test. Although the majority of participants worked in an administration role, around 25% of participants were managers and there was a small representation of records management staff.
Every attempt was made to include staff with varied experience of the current records management system, ranging from never having seen the system to staff who use the system extensively in their job.
Finally, the project team was aware that a number of staff throughout the Council had seen Keyword for Councils. Given this, 50% of the participants recruited had seen the classification, ranging from a quick glance to reviewing the full classification to determine where their documents would be stored.
By recruiting staff with varied experience of the current records management system and Keyword for Councils, we could determine if their prior experience had a positive effect on using the Keyword for Councils classification.
Once the participants were recruited, they were asked to attend a separate 30-minute session, during which they would be asked to complete a number of scenarios based around retrieving and filing documents.
Create scenarios about filing and retrieving common documents
In preparation for the usability test, six scenarios where created for each participant. They consisted of three scenarios given to all participants and three scenarios related to the participant’s job. That way, participants were asked to retrieve and file documents they regularly use as well as documents they are less familiar with.
We also attempted to cover as much of the classification as possible with the scenarios, so that not all the scenarios where concentrated within one or two functions or activity types.
Each scenario was written on a small filing card and allocated a unique letter. The letter was used to identify the scenario during the analysis of the testing results.
Examples of scenarios include:
- An election is coming up and the Electoral Office has sent a letter reminding the Council to submit their boundary figures. What words would you use to find the boundary figures? (Scenario A)
- You have received a letter complaining that the park over the road has not been mowed for six months. What would you file this letter under? (Scenario B)
- Someone has lodged a request to be married at Moffat Beach on a certain date. They also want undercover facilities in case of rain. What would you file the request under? (Scenario C)
Conducting the usability testing
As mentioned, each participant was asked to attend a separate 30-minute session. Sessions were broken into two parts:
- Participants were asked for the types of words they would use to file or retrieve documents listed in the scenarios. This was based on the knowledge that the EDMS has a search facility that can be used to retrieve document as well as search for the correct place to file them.
- Using the same scenarios, participants were asked where they would file or retrieve the documents using the first level of Keyword for Councils. This was with a view to evaluating how easy or difficult it is to understand the functional aspect of the classification. See Table 2 below for the list of council functions shown to participants.
To demonstrate how to conduct the sessions, we facilitated the first few, and then handed over to a Council project member to facilitate the remaining sessions. Whoever was not facilitating recorded the responses given by the participants.
The sessions were also attended by the Records Manager, as business owner of the EDMS, who was able to provide background to the project and answer participant questions about the project.
As with all usability testing, participants were told that there is no right or wrong answer and that their responses would help to determine the suitability of Keyword for Councils for the EDMS.
As each scenario was presented to the participant, the words they used to file or retrieve the document were written down. If they mentioned more than one word or phrase, we asked them to state their first choice. Likewise, in the second half of the session, if they could not decide between a number of functions, we asked them for their first choice.
Analysing the results
After all the usability testing sessions were completed, we collated the participants’ responses and calculated their success rates at retrieving and filing documents based on implementing the Keyword for Councils classification without changes.
Calculate participant success rates in retrieving and filing documents
Success rates when searching
For the first half of the sessions, where participants were asked what words they would use to file and retrieve documents, success rates were calculated by scoring:
- a full point (1) where the participant’s first response is contained in Keyword for Councils, and would be likely to retrieve a match using a keyword based search feature
- half a point (0.5) where a subsequent response is contained in Keyword for Councils
- no points (0) where none of the responses are contained in Keyword for Councils.
During the analysis, we were acutely aware that it is somewhat subjective whether participants would find successful matches, given they were not entering search terms into a live system. However, we felt that the ratings at least provided an indication of success sufficient to assess the suitability of Keyword for Councils.
Once we scored the responses, percentage success rates where calculated for each participant, the average across all participants, and for each of the three common scenarios. For example, the first participant was successful at retrieving or filing documents in five out of the six scenarios, and therefore had a success rate of 83%.
Levels of entry
The results from the first half of the sessions were also analysed to see what level of the classification participants were using as their entry point. That is, were the words they mentioned a function, activity or subject.
Success rates for functions
For the second half of the sessions, where participants were asked where they would file or retrieve documents using the first level of Keyword for Councils (refer to Table 2), success rates were calculated by scoring:
- a full point (1) where the participant’s first response matched the Keyword for Councils function
- half a point (0.5) where a subsequent response matched the Keyword for Councils function
- no points (0) where none of the responses matched.
Once again, percentage success rates were calculated for each participant, the average across all participants, and for each of the three common scenarios.
Impact of participant experience
Success rates were then analysed to see if they were impacted by participants’ use of the current records management classification or the fact that some participants had viewed Keyword for Councils prior to the usability test.
After analysing the usability testing results, the following findings were made.
Staff did not find the functions to be intuitive
Usability of functions is low
If staff were required to file and retrieve documents by knowing the first level of Keyword for Councils (functions), their level of successful document retrieval and filing would be low.
In terms of the calculated success rates, in only 52% of cases did participants successfully identify the correct function for a document.
- had trouble distinguishing between some of the functions, such as Information Management and Information Technology
- did not know what types of documents they would find under certain functions, such as Governance and Commercial Activities
- did not associate certain subjects with a function, for example, very few participants worked out that a marriage permit would be filed under Recreational and Cultural Services.
Staff think at the subject level, rather than function or activity
Staff think at subject level
When participants were asked what words or phrases they would use to search for, or file, a document, close to half of all attempts (43%) were at the subject level. This compares to only 8% of responses at the function level and 20% of responses at the activity level.
Searching via word or phrase increases success
When participants searched for a document using a word or phrase, as opposed to navigating down from the function, their success rates generally increased.
Across all participants, the average success rate was 75% compared to a success rate of 52% when identifying the function.
Experience of records management system had no impact on success
Regular and extensive users of the current records management system faired no better in successfully retrieving or filing documents.
Prior viewing of Keyword for Councils had no impact on success
Those participants who had seen Keyword for Councils before the usability test faired no better than staff who had never seen the classification. This demonstrates that providing staff with the complete classification and expecting them to read the document (amid all their other work) is not a feasible approach.
Staff appreciated consultation
A number of staff mentioned that they were pleased to be involved in evaluating Keyword for Councils. In fact, the user-centred approach adopted by Caloundra City Council will help establish an ongoing dialogue with staff, with the view to improving the EDMS classification over time.
Based on the findings, the following recommendations were made.
Rely on the subject level of Keyword for Councils
Based on the results of the usability test, Keyword for Councils will meet the needs of Caloundra City Council provided that staff do not have to understand level one of the classification (functions).
As the majority of the participants used subject to retrieve or file a document, the EDMS implementation must focus on allowing staff to search by subject.
Modify the subject level
Although participants were reasonably successful when searching using subject descriptors, the average success rate (75%) is still insufficient to guarantee a high level of satisfaction with the EDMS.
To improve the overall success rate, it was recommended that the subject level of Keyword for Councils is modified to more fully support the way the Council does business. This is to be done in close consultation with staff in each of the functional areas, and involves adding new subjects and possibly modifying the terminology of some existing subjects.
Modify the classification in close consultation with staff
Investigate the EDMS searching capabilities
As participants largely searched by subject, the success of the EDMS will depend upon its searching capabilities.
As the EDMS was not available during the planning of the usability test, it is vital that its searching capabilities are assessed. For example, does the EDMS support synonyms? Are the search results easy to understand? What advanced searching capabilities are provided and are they easy to use?
Implement synonym functionality
It was recommended that the EDMS caters for synonyms within its search feature. The need for synonym functionality was highlighted by Scenario C (marriage at Moffat Beach) when only one participant searched on the Keyword for Councils preferred term marriage. Nine other participants used the
Training is an adjunct, not the whole solution
Do not implement a fourth level
Although the EDMS is capable of implementing a fourth level to the classification, there is no perceived need to do so. Few participants mentioned words that would be candidates for a fourth level, such as the names of banks listed under a credit card subject descriptor.
A fourth level is likely to add complexity to the maintenance of the classification as well as the filing and retrieving of documents.
Usability test once EDMS is set up
Once a working prototype of the EDMS is available, it was recommended that a usability test is conducted prior to launch. This will identify:
- any modification that needs to be made to the EDMS to improve usability
- whether the classification needs to be tweaked further to work effectively within the EDMS
- any issues that need to be included in the training and / or performance support materials.
The usability test will also continue with the user-centred nature of the project to date by involving staff in the next level of the implementation.
Provide training and performance support materials
If the use of the EDMS requires an understanding of the functions, it is critical that this understanding is provided in training and performance support materials, such as cheat sheets.
This understanding will also be important for the small percentage of staff that think at higher levels within the classification.
However, training will not guarantee the success of the EDMS, as a successful implementation is not a training issue. Rather, training is an adjunct to improving the usability of the classification through user-centred design and evaluation techniques.
Given this, possible training and performance support materials include:
- Handout providing a high level description of what documents are filed within which functions.
- Cheat sheets or laminated handouts showing the path used to file common documents. These cheat sheets need to be tailored to different groups within the Council.
Review the classification on an ongoing basis
An EDMS classification is a living entity. Therefore, it was recommended that the classification is reviewed at regular intervals to improve its match to the way the Council does business and accommodate any changes to the way Council works over time.
Suggested ways of reviewing the classification include:
- Investigate whether search reports can be run from the EDMS, highlighting the most common successful searches and the most common failed searches. Failed searches indicate words or phrases that need to be added as subject descriptors or included in metadata.
- Six months after launch review the file structure and look for files paths that are full as well as those that are empty. What does this tell the Council about the classification? Are the empty files that way because the file path (function/activity/subject) is not relevant to the Council or because staff do not understand the terminology? Have the full files become dumping grounds because staff do not understand how to correctly categorise their documents?
Review the classification on an ongoing basis
The results of the research can be used to continually improve the classification. This will ensure the EDMS does not become stagnant and continues to support the needs of the Council in their day-to-day operations.
How the report was used
Once the findings and recommendations report was finalised, the Council used the report to:
- hold discussions with the EDMS vendor to ensure that the EDMS could support the recommendations made
- form the basis for an approach to training as well as the development of performance support materials
- plan future sessions with staff to determine how to best modify the subject level of the classification for their needs
- inform the metadata that needs to be captured as part of integrating the EDMS with other corporate systems.
This case study demonstrates how user-centred evaluation techniques can be applied to the world of document and records management.
In the case of Caloundra City Council, a usability test was conducted to assess the suitability of Keyword for Councils for their EDMS classification. The usability test involved asking participants for the words they would use to file and retrieve documents, as well as where they would place certain documents when faced with the list of 32 council functions.
By calculating success rates for document filing and retrieval, we determined that Keyword for Councils would be suitable for the Council on the premise that staff did not have to understand level one of the classification (functions). As staff largely think at the subject level, the usability of the EDMS search feature was identified as paramount.
The Council has used the results of the usability test to inform many areas of the EDMS implementation, such as the set up of the EDMS itself, modification of the classification, training approach to be adopted, and the integration of the EDMS with other corporate systems.
Finally, the exposure to user-centred techniques has prompted Caloundra City Council to continue consulting with their staff to develop a classification that meets their day-to-day needs. This consultation will not only improve the usability of the classification but build buy in to the concept of an EDMS.