Fidelity review of the IPS job coaching service

Who are the instructions for?

This is a guideline for professionals or experts who carry out a fidelity review of IPS job coaching.

Fidelity reviewing IPS job coaching is beneficial for all

The IPS operating model includes an annual fidelity review, in which the IPS service of the unit being evaluated is compared to IPS's international 25-item fidelity scale. Fidelity reviews can be used to indicate whether the service is IPS job coaching, to identify the level of methodological adherence of the service, and to identify the key strengths and areas for development. At its best, a fidelity review is fruitful co-creation that benefits both the subject of the assessment and the assessor.   

The unit to be assessed is usually one psychiatric outpatient clinic that provides IPS job coaching. Studies show that units with a high fidelity review score produce better employment outcomes. 

The quality assessor can be a national body, such as the Finnish Institute for Health and Welfare (THL) and its IPS experts, or an IPS team leader and an IPS employment specialist from another organisation implementing the IPS service (peer review). Usually there are 2 people assessing. If these methods of assessment are not possible, the fidelity review is carried out as a self-assessment. 

In the unit under evaluation, the IPS team leader is responsible for the practical arrangements. If you're an IPS team leader, set aside a few days for preparations, such as gathering documents and scheduling interviews. 

The fidelity review visit itself lasts two days and includes interviews with the IPS team, psychiatric care provider and clients, review of documents, observation of meetings and a visit to the premises. If an on-site review visit is not possible, the fidelity review can also be carried out remotely, eliminating the possibility for observation of the premises. 

After the visit, the evaluators prepare a fidelity report on what they should reserve time for. Based on the fidelity report, the management of the unit draws up an action plan for the development of the service. 

These instructions are based on the original IPS Fidelity Manual of the IPS Employment Center (USA). Please find the manual in English at the IPS Employment Center Library by typing in the library's search field "IPS fidelity manual".
IPS Employment Center, Library

Fidelity scale and scores

The implementation of IPS job coaching is based on an international fidelity scale with 25 criterion. The fidelity scale applies to the personnel, organisation and content of the service. You will find the original English version of the fidelity scale in the IPS Employment Center Library by typing in the library's search field "IPS Fidelity Scale – English".
IPS Employment Center, Library

Each criterion can receive 1–5 points in the evaluation. 
Overall score:

• 115–125 = exemplary level
• 100–114 = good level
• 74–99 = moderate level
• 73 or less = the service is not an IPS service for supported employment

Data collection

The most important method of data collection for the fidelity review is interviewing. 

Information is also collected from documents requested in advance from the IPS team leader. If you work as an IPS team leader, it is essential that you use the team leader's monitoring table to support your daily work, which also collects information needed for fidelity review. The information includes, for example, the number of clients, the number of clients employed and contacts with employers. You will find the English version of the table in the IPS Employment Center Library by typing in the library's search field "Sample Supervisor Data Table".
IPS Employment Center, Library

If the fidelity review visit is carried out by travelling to the unit to be evaluated on site, information is also obtained by observation. 

Interviews

The most important way of data collecting is interviews.

• Client (4 clients will be interviewed) 
• Family member (1–2 client’s family members will be interviewed) 
• IPS employment specialist (2 IPS specialists will be interviewed) 
• IPS team leader 
• Primary nurse (3 primary nurses will be interviewed) 
• Benefit counsellor (the unit's social worker or equivalent will be interviewed) 
• Supervisor of the care team (head nurse or similar will be interviewed) 
• Psychiatric management (two people will be interviewed, one of whom represents the management and the other is a senior psychiatrist, for example) 

You will find the English interview schedules in the English IPS Fidelity Manual by typing in the IPS Employment Center Library search field "IPS fidelity manual".
IPS Employment Center, Library

Documents requested

• Action plan or development plan to improve the quality of the IPS service (to be prepared after the first fidelity review)  
• Minutes of the organization's IPS steering group meetings
• Following information that the team leader gathers from the teams leader's monitoring table:  
  • number of clients per IPS employment specialist at the moment
  • total number of clients in the last 6 months
  • number of clients employed in the open labour market in the last 6 months
  • number of clients having headed to studies in the last 6 months
  • number of clients having headed to a work trial, traineeship or volunteer work in the last 6 months
  • number of clients having headed to social rehabilitation, rehabilitative work activities or similar in the last 6 months
  • number of days between the start of the IPS service and the first employer contact for each client in the last 6 months
  • tasks for which the clients have been employed in the last 6 months (including work trials and traineeships)
  • names of organisations of employers for whom clients have been employed in the last 6 months (including work trials and traineeships)
  • number of face-to-face employer contacts per IPS employment specialist in the past two months
• A list of employers (organizations without personal names) that the IPS specialists have contacted in the last two months (including phone calls and emails)  
  • information on the number of times each employer has been contacted

Observation 

Visiting the unit's premises

• Are there brochures or similar customer material about the IPS service visible in the client premises?
• How close are the working and break rooms of the IPS team to the premises of the psychiatric team?

Attending a psychiatric team meeting

• What does the IPS team's cooperation with the care provider look like?
• Are IPS matters discussed at the meeting?
• Are IPS team members present throughout the meeting?

Attending an IPS team meeting

• How is the IPS team doing, what kind of spirit?
• Are client cases discussed at the meeting?
• What issues does the IPS team leader raise?
• Are the needs for support from IPS employment specialists taken up?

Observing two IPS employment specialists when they meet with employers  

• Are employer connections created with the wishes and strengths of the customers in mind?
• Does the IPS specialist map out the needs of employers? 

Preparation of the fidelity report 

After data collection, the evaluators prepare a fidelity report. The report evaluates each of the 25 points of the quality scale verbally and numerically with a score of 1–5. The report also includes a summary of the unit's strengths on the one hand and development needs on the other. You will find the English version of the fidelity report model in the IPS Fidelity Manual by typing in the IPS Employment Center Library search field "IPS fidelity manual".
IPS Employment Center, Library

Based on the fidelity report, the management of the unit prepares an action plan for the development of its IPS service.