Methodology for Continuous Improvement Process (CIP3.0)

The BSc. Electronics Engineering program offered by the University of the West Indies, Mona uses the assessment and evaluation framework shown in Figure 1, to drive its continuous improvement processes so that graduates are able to achieve its stated program educational objectives (PEOs) in the workplace. Each program’s improvement plan defines an ongoing process of assessment and evaluation to identify program strengths and weaknesses, and to implement opportunities for improvement.

Figure 1     Framework for UWI Mona Continuous Improvement Process

The processes used for continuous Improvement by the Mona School of Engineering, UWI contain continuous improvements for Program Education Objectives (PEOs), Student Outcomes (SO) and course learning outcomes, so that both ABET Criteria 2 and 4 are achieved. Each course is assessed and evaluated to ensure alignment with the course learning outcomes, which in turn aligns to the Student Outcome and the Program Educational Outcome so that the mission of the University is achieved. The goal is to enhance student learning through improvement to courses, curriculum, program and processes.

1.0          Alignment of PEOs and curriculum (courses) to ABET Student Outcomes

Once the PEOs are defined the program curriculum is developed in alignment with the stated PEOs, University mission and ABET program criterion. The courses are developed to enable the curriculum to achieve its stated objectives of alignment to PEOs and to achieve ABET student outcomes, while ensuring a balance of the curriculum content for math and science, core engineering and general education are achieved. These alignments are shown in tables 1 and 2, where the PEOs are mapped to SOs and courses are mapped to PEOs and SOs.

Table 1: Mapping of PEOs to SOs

Table 2: Mapping of courses in curriculum to Student Outcomes

2.0       Assessments and Evaluations

2.1       Assessment of PEOs Continuous Improvement

The flow chart process shown in Figure 1 is used to ensure that the courses and the curriculum continue to meet the requirements of the PEOs and the SOs. Rubrics and score cards for performance indicators are developed and used to perform continuous assessment and evaluation.  Figure 2 summarizes the process of continuous assessment and evaluation of a set of defined PEOs as indicated by the performance indicators developed for the program curriculum.

Figure 2 Continuous Improvement Process for PEOs

Since Mona Engineering is seeking its initial ABET accreditation it is important to have annual reviews of PEOs and Curriculum. In the long term, these reviews may be spaced out to once every 6 and 3 years, respectively. Course reviews are done at the end of every semester and usually involve faculty and students. The assessment and review of the program’s educational objectives ensures that the program remains consistent with the institutional mission, the program constituents’ needs, and ABET’s Criteria.

A quantitative assessment method to show if PEOs are expected to be achieved in 3 to 5 years after graduating uses the rubrics and performance indicators measurements for each SOs, as shown in Table 1. The ranking may then be averaged to show how well the curriculum is expected to achieve the stated PEO.

2.2       Evaluation of PEOs Continuous Improvement

The results of the assessment are evaluated to determine how well the PEOs have been achieved. The Mona School of Engineering has established an Assessment and Evaluation Committee consisting of faculty members. This committee meets once per year to review assessment surveys/rubrics, meeting minutes from Advisory Board meetings and results of the assessment done. The Advisory Board consists of representatives from each constituent/stakeholder and approves and ratifies the instruments that are normally used to gather data, including alumni survey, industry, advisory board, employer survey and evaluation of internships (where applicable). The results of the direct and indirect assessment tools are discussed at the Schools assessment and Evaluation committee meeting, where the next steps are decided and timelines for improvement are determined.

2.3        Assessment of Student Outcomes Continuous Improvement

Figures 3a and 3b summarize the process for continuous assessment and evaluation of how well the student outcomes have been achieved and what adjustments to the courses and curriculum may be required.

Figure 3(a) Continuous improvement process for alignment of curriculum to student outcomes

As shown in Table 2, the courses are mapped to one or more ABET student outcomes. Each course consists of various components that are assessed over the period that the course is offered (a semester or year) and may be assigned to separate SO. Depending on the course, these components range from assignments, design projects, case studies, field trips, written examinations, quizzes, laboratory exercises and designs, simulations, oral and written presentations, surveys, paper analysis, and so on.

Figure 3(b) Continuous Improvement process for alignment of courses to student outcomes

The following steps summarize the creation of performance indicators and rubrics for directly or indirectly assessing how well outcomes are achieved from performances in each component of a course.

1. Select course(s) from Table 2 that aligned with SO
2. Assign SO to course component and/or individual exam questions
   • A course may have several components that align with different SOs (a-k)
   • Assess each course component

3. Create a Rubric to assess the performance criteria for each outcome. Some outcomes may be broken down into sub-performance criteria. In such cases, the following may apply:
   • Create rankings or levels that are aligned to grades for each question/assignment/paper, etc. that assess the outcome
   • Create verbal descriptors for each ranking. This may vary from course to course for each outcome

4. Assign ranking to verbal descriptors and align with the University’s (UWI, Mona) pass rate requirement (75% of class should achieve grades of 50% (grade C) or higher). Then align range to a number category with associated descriptor; as shown in Table 3 below:

Table 3: UWI Mona performance standard and descriptor ranking

5. Assess the performance indicators by determining (calculate) the ranking score for each rubric. For direct assessment of grades for a course component, an Excel spreadsheet may be used to extract quantitative range of values and assigned to specific performance descriptors, as follows:

1. Insert the range of grades (in percentages for the component) for the students registered for the course
2. Categorize the grades in accordance with Table 3 and extract the number of students that achieved each ranking
3. Express the ranking as a percentage of the number of students registered for the course component
4. Calculate the weighted average for the class
5. Average ranking indicates the level of achievement of the class for the particular outcome that is assessed

6. Use the average ranking for evaluation purposes

2.4        Evaluation of Student Outcomes Continuous Improvement

The curriculum committee meets twice a year (January and August), to evaluate the performances of the previous semester and to make recommendations for improvements. The following steps summarizes the processes used:

1. Determine the class performance against the ranking processes (used for assessment)
   • If more than 25% of class is assessed as poor/needs improvement (unsatisfactory) for a particular outcome in a course then an improvement mechanism must be put in place to strengthen that particular outcome
   • In general, performance criteria of less than Proficient (3.0) for any outcome will require a planned intervention for improvement
2. Determine the desired action that may be taken to improve performance
3. Indicate the basis for the action(s) recommended
4. Implement he action within a stated date and timeline
5. Assess the results of the action and determine if improvements occurred

2.5        Summary for Continuous Improvement Process of the Mona School of Engineering

The goal of the Mona School of Engineering, UWI, is to enhance student learning through improvements to courses, curriculum, program, and processes, which in turn drives program-level perspective in course design focused on student attainment of prescribed ABET outcomes. The process for assessments and evaluations are continuous and are meant to guide a student-centered approach that will allow our engineering programs to go beyond assessing students to assessing how well our programs prepare students to enter a dynamic, competitive, global workplace.  We intend to achieve this via effective assessments that use appropriate performance indicators and measurement rubrics to foster continuous improvement in programs, curriculum, and courses.

Each program’s assessment plan defines an ongoing process of assessment and evaluation to identify program strengths and opportunities for improvement. Assessment methods provide evidence of student knowledge, skills and behavior to achieve performance indicators and student outcomes. Performance indicators are specific, measurable statements that describe the performance required to achieve each student outcome and reflect the level of student understanding and associated performance, using appropriate verbs and instructional content. Rubrics are used to articulate expectations for student performance and improve objectivity when scoring direct assessments.