Program Self-Study Report for Mechanical Engineering

3. Program Outcomes and Assessment in .pdf format

This section presents our systematic outcome assessment process including detailed discussion of the process definition, its actual implementation, and improvements made with regard to the process.

Continuous assessment and improvement of our curriculum has always been very important for our program and has led to a major revision of our undergraduate curriculum that was adopted in the fall 1997 semester. Most of our efforts during a three-year transition period to full implementation of the revised program were concentrated on continuous assessment, improvement, and "fine-tuning" of the curriculum. During this period both our Program Educational Objectives (PEOs) and POs were broadly stated. Although our POs, in general, encompassed the ABET a-k outcomes, the correspondence was not very evident. It was clear that both our POs and assessment process used at that time also required revision.

Consequently, new procedures to methodically evaluate the performance and learning outcomes of our current students and recent graduates have been developed. In addition, with the participation of all constituents of the program, clear program objectives as well as systematic evaluation and improvement plans have been identified to make certain that the overall assessment scheme is robust and sustainable. The following is a summary of our outcome assessment process.

3.1 Program Outcomes

The Department of Mechanical Engineering has developed eleven Program Outcomes as listed below. Corresponding ABET EC2000 criteria (a-k) and ME program specific requirements (ME-1 to ME-4) are identified inside the brackets at the end of each program outcome.

After completing the mechanical engineering program, graduates should have the following attributes:

1. An ability to apply knowledge of mathematics, calculus based science and engineering to mechanical engineering problems [ABET 3a, ME-1 & ME-2].
2. An ability to design and conduct experiments, as well as to analyze and interpret data [ABET 3b].
3. An ability to design thermal and mechanical systems, components, or processes to meet desired needs [ABET 3c, ME-4].
4. An ability to function on multi-disciplinary teams [ABET 3d].
5. An ability to identify, formulate, and solve engineering problems [ABET 3e].
6. An understanding of professional and ethical responsibility [ABET 3f].
7. An ability to communicate effectively with written, oral, and visual means [ABET 3g].
8. The broad education necessary to understand the impact of engineering solutions in a global and societal context [ABET 3h], and a knowledge of contemporary issues [ABET 3j].
9. A recognition of the need for, and an ability to engage in life-long learning [ABET 3i].
10. An ability to use modern engineering techniques, skills, and computing tools necessary for engineering practice [ABET 3k].
11. Familiarity with statistics and linear algebra [ME-3].

3.2 Program Educational Objectives (PEOs) and Their Relationship with the Program Outcomes (POs)

A summary of the relationship between the established ME program outcomes and the program educational objectives, as discussed in section 2, is shown in Table 3-1. In order to quantify this relationship, three correlation levels are used. A solid square indicates the strongest correlation between a specific outcome and a given educational objective. The next level of correlation, marked by a half-filled square, represents moderate correlation. An open square indicates the lowest level of correlation. Every PEO is seen to be correlated with at least two POs with varying degrees of correlation. This table also lists the PEOs and the POs for convenient reference.

Currently, our PEOs are being reevaluated because of criticism by key constituents that our PEO statements appear to correspond too closely with our POs rather than being statements that describe more broadly the expected accomplishments of our graduates during their first several years after graduation. A new set of PEOs has been drafted and is undergoing evaluation by all constituents. It will be adopted officially in Fall 2003. However, we expect no significant change in the POs even with the release of the new PEOs. Consequently, we will base our discussion of POs on the current PEOs.

After completing the mechanical engineering program graduates should have the following attributes:

1. An ability to apply knowledge of mathematics, calculus based science and engineering to mechanical engineering problems [ABET: 3a, ME-1, ME-2].
2. An ability to design and conduct experiments, as well as to analyze and interpret data [ABET 3b].
3. An ability to design thermal and mechanical systems, components, or processes to meet desired needs [ABET 3c, ME-4].
4. An ability to function on multi-disciplinary teams [ABET 3d].
5. An ability to identify, formulate, and solve engineering problems [ABET 3e].
   
6. An understanding of professional and ethical responsibility [ABET 3f].
7. An ability to communicate effectively with written, oral, and visual means [ABET 3g].
8. The broad education necessary to understand the impact of engineering solutions in a global and societal context [ABET 3h], and a knowledge of contemporary issues [ABET 3j].
9. A recognition of the need for, and an ability to engage in life-long learning [ABET 3i].
10. An ability to use modern engineering techniques, skills, and computing tools necessary for engineering practice [ABET 3k].
11. Familiarity with statistics and linear algebra [ME-3].

3.3 Program Outcome Assessment Structure

The general framework of an ideal outcome assessment plan would involve the participation of all constituents of the program at all levels. However, we feel that it is inefficient, if not impossible, to directly involve all constituents in all assessment processes. We believe that an effective assessment plan should have a hierarchical structure with a bottom-up approach that incorporates appropriate delegation of responsibilities to maximize the interaction from all program constituents. Based on this concept, the ME assessment process is categorized into three different levels: individual course, curriculum, and ME program.

For each level, constituents and coordinators who are responsible for the assessment process within that level were identified. Within each level a comprehensive assessment loop was devised with the participation of all core constituents. The loop consists of assessment objectives, strategies and tools, qualitative and quantitative outcome matrices, and an improvement plan. Every faculty member participates in the course-level assessment through teaching and coordination of either core or technical elective courses. The curriculum level serves as the intermediate coordination stage between the program-level and the course-level. The program-level deals with assessment issues involving overall educational objectives and outcomes, as well as interaction with constituents external to the program such as alumni, employers and Mechanical Engineering Advisory Council (MEAC). Each assessment level is described in more detail in the following sections.

3.3.1. Course-Level Assessment

Based on our previous experience in developing the integrated curriculum, the communication between the teaching faculty and the program administration is often far from flawless. This prevents timely adjustments from being made to the program to accommodate its frequently changing needs. This miscommunication is more serious if either a number of inter-related courses or a single course taught by several faculty members is involved. Inconsistency in the teaching of core subjects can lead to students being unprepared for higher-level classes or graduates not satisfying specific program outcomes.

Therefore, we decided to delegate the responsibility for each core ME course to one faculty member (designated as the responsible faculty) who is an expert in the respective field. The responsible faculty is in charge of the development of objectives and measurable outcomes for the course, with final approval by the curriculum committee. The responsible faculty for each course is identified in the outcome-specific course syllabus.

The learning objectives for each course define the overall goals for the course in the context of the ME curriculum. Specific measurable outcomes for each course reflect the knowledge and capabilities that each student should have upon completion of the course. Every faculty member who teaches the course must coordinate with the responsible faculty to ensure that educational content does not significantly deviate from the documented objectives/outcomes of the course. The coordination takes place both informally through private communication and formally through curriculum meetings with respective area coordinator and/or program review meetings. Therefore, at the course level, both the responsible faculty and the teaching faculty member are held responsible for the development of a specific course.

Table 3-2 outlines the major components in the course-level assessment process. As stated earlier, course-level assessment is reported directly to the curriculum area coordinator, personally or formally in regularly scheduled curriculum meetings. A Course Portfolio is prepared for each ME core and technical elective course.

The course-level assessment is the core of the overall assessment process. Although each individual class does not satisfy all POs, the collective outcomes from all these courses do satisfy all the program outcomes. A more detailed discussion about the course-level assessment process can be found in the Course Portfolio available in ME conference room for review. Included in the portfolio are the assessment plan, samples of objective/outcome specific syllabi, course assessment reports, and SUSSAI student assessment reports, all done at the course-level.

3.3.2. Curriculum-Level Assessment

As a result of revamping the curriculum starting in 1996, the undergraduate program has been categorized into five major academic sub-curricula (areas): (1) Thermal and Fluids; (2) Dynamic Systems; (3) Mechanics and Materials; (4) Mechanical Systems; and (5) Design. The Design category is the culmination of all acquired knowledge and skills in the preceding four areas and their implementation in engineering practice. It is classified separately for the purpose of efficient coordination and assessment. In a 1997 faculty meeting, coordinators were assigned to each of these areas as follows: (1) Thermal and Fluids: C. Shih; (2) Dynamics Systems: E. Collins; (3) Mechanics and Materials: H. Garmestani (who was replaced by N. Chandra in 2003); (4) Mechanical Systems: P. Hollis; (5) Design: C. Luongo. These five area coordinators, along with the undergraduate coordinator (Dr. Hollis) and the Associate Chair (Dr. Buzyna), constitute the ME curriculum committee. This committee has the primary responsibility for the design, implementation, assessment, and improvement of the curriculum.

All major components involved in the curriculum-level assessment process are listed in Table 3-3. Working with the involved faculty members, the area coordinators are responsible for the planning, assessment, and improvement of their respective area. As members of the curriculum committee, the area coordinators are responsible for the efficient inter-area coordination of relevant courses, common requirements, and activities. The curriculum committee reports all assessment results/improvement plans to the program administration and the full faculty during regularly scheduled curriculum and faculty meetings.

To ensure that these efforts are properly documented, a Curriculum Portfolio is prepared by each area coordinator and is available for review in ME conference room. The portfolio includes, at a minimum, the following items:

1. Learning objectives and measurable outcomes consistent with the departmental POs;

2. The area curriculum structure and its relationship to the other areas;
3. A self-assessment curriculum report to the undergraduate coordinator;
4. Course syllabi for all area courses, including cores and technical electives. It is required that all area faculty review the contents of their respective curriculum portfolios during each assessment cycle.

The curriculum-level assessment process guides the overall development of the ME curriculum and, in particular, the program outcomes (ABET criterion 3), the professional components (ABET criterion 4) and the program criteria (ABET criterion 8). Informal and formal area curriculum meetings are regularly conducted to coordinate within the area. Every semester, at least one curriculum meeting is held for all coordinators to review their respective areas in addition to examining the entire ME curriculum. A curriculum review meeting is conducted annually by the curriculum committee to provide a comprehensive overview of the ME curriculum to the entire faculty. All proposals for changes and improvements are discussed and decided in the meeting for future implementation. A more detailed discussion about the curriculum-level assessment process can be found in the Curriculum Portfolio.

Verification that all of the POs are addressed in the curriculum is provided by Table 3-4. This shows that, collectively, the ME required and core courses thoroughly cover all POs and, consequently, fulfill the PEOs.

Table 3-4 Program Outcomes and Individual Course Outcomes

Course Description ME Program Outcomes

Course #

Title

Credit

1

2

3

4

5

6

7

8

9

10

11

EML 3002

ME Tools

4

EML 3004

Intro to ME

4

EML 3011

Mechanics & Materials I

4

EML 3012

Mechanics & Materials II

3

EML 3234

Materials Science & Eng

3

EML 3013

Dynamics Systems I

4

EML 3014

Dynamics Systems II

4

EML 3015

Thermal-Fluids I

4

EML 3016

Thermal-Fluids II

4

EML 4304L

Thermal-Fluids Lab

2 (3)

EML 3017

Mechanical Systems I

4

EML 3018

Mechanical Systems II

4

EML 4551

Senior Design I

4

EML 4552

Senior Design II

4

EML 4930

Senior Seminar

0 (1)

EGN 1004L

First Year Engineering Lab

1

ENC ****

English I & II

3 + 3

Liberal Education

6 + 3

MAC ****

Calculus I, II, III

4 + 4 + 5

MAP ****

Engineering Math I & II

3 + 3

PHYS ****

Physics w/Lab I & II

5 + 5

CHM ****

Chemistry w/Lab

4

3.3.3. Program-Level Assessments

The program-level assessment loop is the ultimate process to evaluate the overall outcomes of the program. Its scope encompasses the definition, assessment, feedback and improvements of all delineated ABET criteria, including students (criterion 1), educational objectives (criterion 2), faculty (criterion 5), facilities (criterion 6), institutional support (criterion 7), and curriculum (criteria 4 and 8). The program outcomes and assessment process itself constitutes the present criterion (criterion 3). Program administrators, including the Chair, Associate Chair, undergraduate program coordinator, and advising coordinator, are primarily responsible for carrying out the program-level assessment process. However, all constituents contribute significantly to the effort by participating in the continuous feedback and improvement of the process itself. The following table summarizes the process developed for the assessment of the total program. More information concerning the program-level assessment can be found in the Program Assessment Portfolio (also placed permanently in the ME conference for review)

Working with the college through a grant supported by the NSF SUCCEED Coalition, the department developed a general framework for the assessment plan to evaluate the performance of students at the program-level. The following guidelines were proposed:

• Develop assessment and improvement plans specifically tailored to the curriculum and program. The ABET criteria will be considered as guidelines but not as dictating constraints. This allows us to develop a flexible overall assessment strategy to guarantee long-lasting success.
   
• Involve students in the assessment process at all levels
   
• Implement outcome-specific assessments in all classes; coordinate assessed results at curriculum- and program-levels.
   
• Engage MEAC in the assessment loop; in particular, solicit their participation in the senior design evaluation panel and program review.
   
• Extend the use of the graduating senior survey/interview to allow students to self-evaluate their own academic performance as related to the specific program outcomes.
   
• Restructure a systematic alumni survey in place of the previously ad-hoc process.
   
• Discuss the possible development of new assessment tools for systematic integration into the overall assessment loop, including student portfolio, professional/educational outreach assignment, standardized test, etc.
   

The faculty approved this holistic assessment plan during the faculty summer retreat meeting, and the program administration and curriculum committee took steps to implement the plan immediately. The plan was subsequently presented to the MEAC and received their full endorsement. As a result, they established a MEAC curriculum sub-committee. Several MEAC committee members agreed to participate in a design project review and to provide feedback to the department. This MEAC review process has been conducted annually for the past two years.

The following is a complete presentation of all activities in the assessment process. Details of the implementation, results and action plan of each event are discussed separately.

3.4. Program Assessment Plan Implementation

To date, the complete assessment plan has been through two complete assessment cycles over a period of two years. Although it is expected that the process will evolve continuously, the following procedure is representative of the general process put in place for the assessment of the program. Details and documented results of these review practices are included in the Program Assessment Portfolio available in the ME conference room for all constituents to review.

The following is a list of general assessment tools/schemes either already adopted in our process or currently under development

1. Undergraduate Program Review Meeting
    
   This annual meeting provides a comprehensive review of the undergraduate program. This meeting concludes the yearly program assessment cycle by evaluating the effectiveness of the program in achieving its stated PEOss and POs and, at the same time, summarizing efforts made toward the improvement of the program. Program and curriculum suggestions and improvement plans are proposed and discussed and future implementation is decided on. In addition, strategies used in the previous assessment cycle are examined thoroughly to ensure that the process is sustainable and efficient. Detailed reports of the 2002 and 2003 meetings, including meeting agenda, assessed results, recommendations and actions taken and planned are included in the Program Assessment Portfolio for review.
    
2. MEAC Senior Design Fair/Program Review Meeting
    
   One of the new activities developed for the program-level assessment is the direct involvement of MEAC in the undergraduate program outcomes assessment process. The MEAC is an advisory board consisting of engineering professionals and engineering faculty/administrators from other institutions who are familiar with engineering profession and practice. The FAMU-FSU MEAC was established in 1998 and its current members are listed in the Appendix I E-1. It meets twice a year during the Fall and Spring semesters to provide input and feedback to the program on issues of curriculum, research development and industrial outreach.
    
   During the spring semester, MEAC conducts an annual program review meeting in three successive stages:
    
   1. MEAC curriculum committee members attend the senior capstone project presentation and open-house exhibition to assess the design curriculum and the graduating seniors' overall design experience.
   2. On following day, there is a program review meeting attended by the MEAC curriculum committee, the departmental administrators, key area coordinators, and the senior design coordinator. All issues related to the curriculum and, in particular, the design experience, are discussed. Comments and suggestions for potential improvements and modifications in other elements of the program, such as the PEOs, POs, and industry collaboration efforts, are also discussed.
   3. A summary report is presented to the entire council by the MEAC curriculum committee chair and further recommendations and suggested action items are discussed in the council and relayed back to the department in the annual MEAC report.
       
3. Student-Faculty Council Meeting
    
   The objective of the council is to seek effective communication between students and the program administrators about all departmental issues, especially those directly affecting students such as POs, PEOs, and advising among others. The council members consist of the departmental administrators and student representatives selected from all significant groups (both universities, key professional societies, each academic level, etc.). At least one forum is scheduled at the end of every semester and additional meetings can be arranged as needed. Student representatives are asked to arrange separate meetings with their respective constituents to collect their comments/concerns before the council meeting. Much useful feedback was obtained during the 2002 & 2003 meetings and summary reports are included in the Program Assessment Portfolio.
    
   
4. Senior Exit Interview/Questionnaire
    
   For several years, senior questionnaire forms have been used to assess graduating seniors' perception of the program and to solicit their comments about possible improvements and modifications. Two years ago, we decided to add individual senior interviews to the process in order to come face-to-face with every graduating senior. We felt that we could receive a more complete picture than by simply using the questionnaire alone. The use of both senior exit interviews and questionnaires provides a comprehensive overview of the program from the graduating senior class. We can assess POs as seen from the students' perspective just before their graduation from the program. Assessment of the perception of preparedness of the student for career entry and professional development at this stage is useful as it can be used to compare to alumni survey data given by graduates who have been in the workforce for a few years. We also ask them to honestly assess the advising practice, school infrastructure, etc. and comment on how we can improve the process. Summary reports are also included in the Program Assessment Portfolio.
    
   
5. Alumni Survey
    
   A total of 96 (out of a total of more than 200) students graduated between 1998 and 2002 were surveyed in 2001 & 2003. We decided to only survey alumni who have graduated in the last five years as our starting point. An alumni survey form was generated and could be filled out on-line via the Web. So far, 54 (>20%) alumni have responded to the survey and a summary report of the surveyed data is included in the Assessment Portfolio. The alumni survey will be conducted every two years.
    
   
6. Student Portfolio (under development)
    
   The student portfolio can be used to establish a longitudinal record of a student's progression over his/her entire college career. It can be used to assess a student's skill development in areas such as the familiarity of graphical/design tools such as ProE, and proficiency in communication. Most important, it is an effective means to track a student's progress toward the culmination of their design capability as they begin to combine all relevant subjects in engineering, science, and math into practical applications and design synthesis.
    
   A preliminary discussion of student portfolios took place at the Fall 2001 program review meeting, however the full implementation of this assessment scheme was postponed due to the lack of an efficient inventory/evaluation mechanism. However, this constraint was removed when Florida State University started its own electronic portfolio program in 2003. A proposed plan to use the student portfolio as a program assessment measure has been drafted and will be presented in the 2003 summer faculty retreat.
    
   In short, the "career portfolio" for each student consists of major course projects/artifacts from each of the ME core course that are electronically deposited into the FSU portfolio site. For example, in the Thermal/Fluids course sequence, both written and oral reports (PowerPoint presentation files at this stage) for the design project of thermal systems will be included to showcase a student's capability to design thermal systems as well as his/her communication skills. A proposed list of all portfolio materials is included in the Program Assessment Portfolio.
    
   The collection of the portfolio will be initiated in the Intro. to ME class during a student's sophomore year. Responsible faculty will emphasize the importance of keeping a professional development record and provide necessary guidance for all students. The curriculum committee will evaluate the portfolios periodically. A yearly review will provide a timely evaluation of the curriculum at each level. An end-of-career review will be done during the senior year in the senior capstone class so that it can provide a comprehensive, longitudinal overview of a student's work over his/her entire career. MEAC curriculum committee members will be asked to participate in the review as external evaluators. A reasonable sample size of at least 10 or 10%, whichever is larger, of the student population will be used initially on a voluntary basis. However, all students will be encouraged to participate in the program. A detailed plan of the proposal can be found in the Program Assessment Portfolio.
    
   
7. Employer Survey (ongoing improvement)
    
   This is one of the least successful assessment schemes we have attempted. In 2002, employer survey forms (included in Program Assessment Portfolio) were developed and sent out to several companies for feedback but only a few were returned. Departmental administrators have discovered several reasons for the low return rate through telephone interviews and personal contacts. Due to the relatively young age of our program, we have not produced a large group of graduates. With only one or, at most, a few graduates from our program, employers considered the survey either an alumni survey or a personal performance review. In the latter case, some companies refuse to respond on the basis of privacy concerns. We have since tried other approaches to solicit employers' opinions. For example, we interact closely with both MEAC and College Advisory Council members (during Industry Day activities in 2002 and 2003) since some of them employ our graduates. Also, we interviewed campus recruiters for their opinions regarding the quality of our graduates in general. A summary report of these interviews is included in the Program Assessment Portfolio.
    
   A summary of all these assessment tools/schemes and their use in assessing each specific program outcome is included in Table 3 6. Again, three levels of effectiveness are chosen for clearer classification. A completely filled square means extremely useful; a half-filled square represents the next level of usefulness; and an open square indicates the item is only somewhat useful in assessment and should not be used as primary assessment tool for a given outcome.

1. An ability to apply knowledge of mathematics, calculus based science and engineering to mechanical engineering problems [ABET: 3a, ME-1, ME-2].
2. An ability to design and conduct experiments, as well as to analyze and interpret data [ABET 3b].
3. An ability to design thermal and mechanical systems, components, or processes to meet desired needs [ABET 3c, ME-4].
4. An ability to function on multi-disciplinary teams [ABET 3d].
5. An ability to identify, formulate, and solve engineering problems [ABET 3e].
6. An understanding of professional and ethical responsibility [ABET 3f].
7. An ability to communicate effectively with written, oral, and visual means [ABET 3g].
8. The broad education necessary to understand the impact of engineering solutions in a global and societal context [ABET 3h], and a knowledge of contemporary issues [ABET 3j].
9. A recognition of the need for, and an ability to engage in life-long learning [ABET 3i].
10. An ability to use modern engineering techniques, skills, and computing tools necessary for engineering practice [ABET 3k].
11. Familiarity with statistics and linear algebra [ME-3].

3.5. Plan for Continuous Assessment/Feedback/Improvement

To complete the assessment process, a plan for continuous and sustainable improvement of the program has been devised. The assessment-feedback-improvement process of the ME program is presented schematically in Figure 3-1. Although some elements have been used for sometime, the entire assessment package was not fully implemented until Spring 2001. To date, we have completed two complete assessment cycles. The figure illustrates the annual timeline, assessment tools/strategies and the implementation process for the overall assessment of ME program outcomes. It should be noted that the assessment process, along with all its essential elements, is being constantly evaluated to make sure that process itself can be optimized as well.

The assessment cycle starts every Fall semester when curriculum improvements and other reform actions suggested from the previous cycle will be implemented. For example, every teaching faculty member, in consultation with respective responsible faculty and area coordinators, should have prepared/reviewed an outcome-specific syllabus and conducted outcome assessment to evaluate the development of every ME core and elective course. Area curriculum meetings and faculty meetings will be arranged during the semester to coordinate the assessment activities within the course- and curriculum-levels. This provides an opportunity for all faculty members to directly discuss and evaluate courses and curriculum, to make timely adjustments to immediate problems and to also provide suggestions for improvements to the program that will receive more comprehensive consideration and long-term resolution in later faculty meetings. These activities are illustrated inside the top group (dotted box) in Figure 3-1.

Indications of problems, successes and improvements to the overall curriculum will be highlighted and discussed in the Annual ME Program Review Meeting, scheduled at the end of every Spring semester. In this meeting, each area coordinator will provide a detailed report of their respective area describing deficiencies and achievements observed during the current cycle. In addition, development efforts such as direct results steming from the previous assessment period will be evaluated and discussed. The curriculum committee will develop a plan summarizing the overall progress of the ME curriculum and recommending appropriate actions in response to the assessment outcome in the meeting.

Program administration is responsible, with assistance from relevant faculty members and the curriculum committee, for collecting all program-level assessment results from all sources including MEAC program review, senior design project review, senior exit interview and questionnaire, alumni survey, student-faculty council results, among others. A summary of all these activities is reported in the annual program review meeting (refer to the middle group in Figure 3-1). Recommendations for potential program improvements and curriculum reforms based on these assessment results areproposed by the program administration in the annual meeting so that a preliminary action plan can be discussed, debated and voted on by the full faculty body.

Finally, based on the preliminary proposal, a detailed implementation plan is prepared by the program administration, in consultation with the curriculum committee if necessary, during the summer period immediately following the Annual Program Review Meeting. The final comprehensive plan is presented and approved in the annual ME faculty retreat scheduled at the end of the summer for immediate implementation (bottom group in Figure 3-1.) This concludes one complete assessment-feedback-improvement cycle over the period of one year. Currently, we have just completed our second complete cycle, which began starting Fall 2002 and are completing in Summer 2003.

Figure 3-1 ME Continuous Assessment-Feedback-Improvement Process

3.6. Qualitative and Quantitative Assessment Artifacts Available For Review

In this section, we briefly summarize all quantitative and qualitative artifacts that have been used in the ME continuous outcome assessment process. Since this set of assessment artifacts is used to assess all POs, we refer to it as the "Common Assessment Packet" to distinguish it from a few specialized assessment schemes designed to evaluate specific outcomes. All summarized assessment results are included in the Program Assessment Portfolio and will be available for review during the ABET visit.

1. Course Portfolio: This is a comprehensive folder prepared for each ME core course to ensure that the teaching of the course is coordinated throughout the curriculum and that an individual faculty cannot make uninformed changes. This folder is placed permanently in ME conference room for all interested parties to review. Included in the portfolio are a course overview, a course assessment plan, an objectives/outcomes specific syllabus, course assessment reports, SUSSAI student assessment reports, and examples of student work such as exams, homework, lab reports, and design reports. Some of these assessment tools are explained in the following:
    
   • Objectives/outcome specific syllabus (see an example in Appendix I E-2): For each course, the responsible faculty and the area coordinator develop a set of learning objectives for the course. These objectives are used to identify critical topics, remove out-of-date materials, prioritize the teaching schedule, facilitate effective learning activities, and coordinate efficiently with other faculty.
      
     For example, one of the objectives (objective 1 as highlighted in the sample syllabus) for Analytical Tools in ME is "To familiarize students with mathematical and numerical tools commonly used in ME." Each of these course objectives is then directly linked to their corresponding POs defined by responsible faculty along with the area coordinator. For example, the successful accomplishment of the above example, learning objectives1, can demonstrate the achievement of the ME PO 10 (an ability to use modern engineering techniques, skills, and computational tools.) The number 10 is then shown in the bracket next to the objective 1 indicating a direct link from this course objective to the PO 10.
      
     A set of measurable course outcomes describing the directly observable results that demonstrate the achievement of the course objectives, have been defined. Using the same example, one of the outcomes (outcome 6 in the sample syllabus, also highlighted) for the previously defined objective 1 is that "Students be able to use ALGOR FEM package to analyze simple physical systems, such as trusses, beams…" This is a clearly defined course outcome can be measured exactly through the use of homework, quizzes, tests, projects, and student surveys. The successful fulfillment of this outcome can be used to show the partial satisfaction of the course objective 1.
      
     In summary, each ME core course has clearly defined course objectives, which also correlate directly to the POs. In addition, measurable course outcomes are identified for each of these course objectives so that appropriate assessment schemes can be arranged to measure the effectiveness of a student's learning. In each ME core course, the course/outcome-specific syllabus is distributed and discussed with students at the beginning of the semester so that they know exactly the expectations of their achievements going through the class.
      
     
   • SUSSAI outcome assessment report: SUSSAI stands for State University System Student Assessment of Instruction. It is a state-mandated student evaluation procedure providing a general assessment of classroom teaching by asking students to evaluate such factors as "description of course objectives and assignments" and "overall assessment of instructor". It is administered at the end of each semester by a student proctor without the direct participation of the instructor. Results are tabulated by the university and returned to the instructor and program administration.
      
     Optional items can be added to the SUSSAI form and tabulated results of these optional items are furnished to the instructor for the self-evaluation purpose.
      
     Appendix I E-3 shows the SUSSAI optional items used for the class, "Analytical Tools in ME". Students are asked to rate each item from 1 (excellent) to 5 (poor) regarding their capability in specific course outcomes. The average response, referred to as the Course Outcome Index (COI), is a quantitative measure of the students' collective perception on their mastery of the particular course outcome. Take note that this index has only been used as a reference and will not be used to replace other traditional measures such as homeworks, quizzes, tests, and projects. However, a chronological monitoring of the variation of the index can be an effective measure for tracking the progression of the students' learning effectiveness of specific course materials. Individual instructors and area coordinators could use these values, at their discretion, to modify and improve the teaching of the class accordingly.
      
     
   • Course Outcome Index (COI): One of the quantitative assessment tools used is the COI for each individual ME core course. As described earlier, this index alone in one class might not be able to provide meaningful information to the overall program outcomes. However, a collection of this index for all core courses could be used as an indicator of the program outcomes. Each COI is directly linked to specific course learning objectives, which also have been mapped to POs. Therefore, a COI can be perceived as a quantitative measure of how a course satisfies some elements of the POs.
      
     Using the same example above (also see Appendices E-2 and E-4), a COI of 1.4 assigned to course outcome 6 will also be mapped into course objectives 1 and 7 and then to all corresponding POs. In the present example, a 1.4 index will be assigned to both PO 10 (via objective 1) and POs 1, 5 & 10 (via objective 7) based on the COI assessment on course outcome 6. Collectively, a number of course outcomes might map into a number of program educational outcomes, and a COI worksheet (see Appendix E-4 for an example) can be produced for each core course. The averaged index values for all correlated POs are entered into the ME summarized COI table (in Appendix I E-5). The averaged program COI for a specific PO can then be determined by averaging COI, weighted by their respective course credit hours, for all ME core courses. For a more detailed discussion on the use of COI on ME PO assessment, please refer to Appendix I E-5.
      
2. Curriculum Portfolios for the five areas (Thermal/Fluids, Dynamic Systems, Mechanical Systems, Mechanics and Materials, and Design): These area folders include objectives and measurable outcomes of the area, the area curriculum structure within the program, a self-assessment curriculum report, syllabi and assessment reports for all area courses. Any major modifications done on the specific area are documented chronologically in the portfolio for coordination purposes.
    
3. Program Assessment Portfolio: This is a summary folder for all assessment materials at the program-level. It includes the definition of the assessment process, the implementation plan for continuous improvement process, objectives/outcomes of all scheduled assessment activities, selected data from program assessment results, and a chronological documentation of the improvements/modifications made. Other materials included in this portfolio include:
    
   
   • Senior exit interview/questionnaire results: A summary report of qualitative comments from graduating seniors from 2000 through 2003, including perceptions of the program and faculty, problems with courses and labs, and evaluation of advising experiences. In addition, quantitative data of the graduates' perception of the quality of program and outcome assessment are tabulated.
      
   • MEAC Program Review Meeting reports: Discussion and interaction with the MEAC curriculum committee, both in the senior design fair and the follow-up meeting, and recommendations proposed by the full council are documented in these reports. Both 2002 and 2003 reports are included. Archived MEAC reports from 1998 through 2001 are also available.
      
   • Senior design project review: In addition to the MEAC design review, a summary report of the 2001-2002 and 2002-2003 senior design activities is included.
      
   • Alumni survey statistical data: Two alumni surveys have been conducted; one for students who graduated before 2001 and another for those who graduated in 2001 and 2002. All relevant data and recommendations/actions are summarized in the portfolio.
      
   • Student-Faculty council meeting records: A council meeting between student representatives and program administration is conducted every semester to establish communication between students and the program. Meeting records are available for review.
      
   • Employer survey results.
      
   • Graduate placement statistics.
      
   • Curriculum, program review meeting records.
      

3.7. Assessment of Individual ME Program Outcomes (POs)

In the following section, a brief summary of each individual PO is provided. For each outcome, its assessment results are first discussed, followed by an action plan for improvements/modifications, if necessary.

In order to track our progress toward the satisfaction of all POs, the curriculum committee meets every summer to discuss the level of accomplishment of our students based on assessment results collected in the current cycle. Each member then rates these outcomes separately using the standard letter grade system to provide a quantitative measure of the specific outcome. A consensus grade for each outcome is decided by consolidating every member's opinions through deliberation (individual grades of curriculum committee members are included in the Program Assessment Portfolio.) This subjective grade on a specific outcome is considered a baseline measure for our self-assessment and the tracking of the future development. Following the traditional grading system, the following scale is used for this purpose: A - excellent, B - Good, C - Average, D - Poor, F - Failure. A grade of B range or better is considered acceptable. A grade of C or lower warrants immediate consideration of an improvement plan. A plus or minus scheme is added to provide additional flexibility in the grading system. The self-assessed grades are presented in the annual ME summer retreat along with other quantifiable assessment results.

Please note that the numerical results on alumni and senior exit surveys reported in this section are based on a response scale from 1 to 5 such that: 1 - Excellent, 2 - Very good, 3 - Good, 4 - Fair, and 5 -Poor. A value less than 2.5 is considered acceptable, while any higher value needs discussion. The third column of the outcome table is the percentage of respondents who answered either excellent (1) or very good (2). A 75% or better is an acceptable goal in this category. The last column represents the percentage who answered fair (4) or poor (5). Any number higher than 20% is considered unacceptable and an immediate action is needed.

3.7.1. Individual PO Assessment Report

1. An ability to apply knowledge of mathematics, calculus based science and engineering to mechanical engineering problems.
    
   

   	Ability (averaged score)	Excellent or very good (%) (1/2)	Fair or Poor (%) (4/5)
   Senior exit questionnaire (2002)	1.8	8.80	2.3
   Senior exit questionnaire (2003)	1.5	92.6	2.4
   Alumni survey (pre 2001)	2.0	73.1	3.8
   Alumni survey (2001-02)	1.7	92.5	0.0
   Curriculum committee grade and comments	A-: ME curriculum has included strong analytical components in math, science and engineering emphasizing problem solving. Although improvements through continuous assessment are sought; no major changes are considered at this time.

   
   

   In general, this is considered as one of the strongest educational outcomes in our program and a vast majority of students (88.0% & 92.6%) and alumni (73.1 % & 92.5%) agree with this observation. The curriculum committee gives a grade of "A-" to this outcome and no major changes are recommended at this time. The data indicate an improvement in the alumni perception over a three-year period. As will be seen, this improvement is common to all outcomes except the teaching of statistics (outcome 11). We believe that is due to the implementation of the integrated curriculum in 1997 since the first group of ME graduates under the new curriculum graduated in 2001. This aspect will be further monitored for additional assessment.

2. An ability to design and conduct experiments, as well as to analyze and interpret data.
    
   

   	Ability (averaged score)	Excellent or very good (%)	Fair or Poor (%)
   Senior exit questionnaire (2002)	2.1	79.0	4.6
   Senior exit questionnaire (2003)	1.6	90.2	2.4
   Alumni survey (pre 2001)	2.3	53.8	7.7
   Alumni survey (2001-02)	2.0	77.8	3.7
   Curriculum committee grade and comments	B+: new lab components are being added to Dynamic Systems II. Mechanical Systems course sequence needs to establish visualization-enhanced lab. General observation is that students are capable of planning, executing, and analyzing an experiment. All lab courses should emphasize "design" experiments.

   
   

   This is also one of the strongest outcomes in ME curriculum. Most graduating seniors (79% & 90.2%) and alumni (53.8% & 77.8%) either agreed or strongly agreed that they had been prepared well in this outcome. A negligible percentage of them (< 8%) either disagreed or strongly disagreed. Significant upgrades of the laboratories in recent years have improved the instructional quality of the lab courses, consistent with improving survey results. For example, new laboratory components have been integrated into both Dynamics Systems I & II courses, a visualization-enhanced laboratory has been added to the Thermal and Fluids lab class, and the Mechanics and Materials lab has also been modernized to include several digital data acquisition stations. The "B+" grade is assigned with the understanding that more emphasis will be placed on the teaching students on how to design experiments. In addition, the development of a Mechanical Systems laboratory is currently being considered.

   Senior design projects are selected for their value as a teaching tool for the 'engineering design cycle', so projects that involve just an experiment and data analysis are typically not accepted (we try to avoid 'research' projects and favor 'product design' projects). So even if all projects have a hands-on component, they do not always involve an experiment. However, in many projects, an experiment is needed somewhere in the design cycle, or as part of prototype implementation. The following table summarizes the major experiments that have been carried out as part of the senior design projects in the past few years. A detailed list of these projects is included in the Program Assessment Portfolio. It is obvious that a significant number of capstone projects (40%) involve the design and conduct of experiments.

   Year	Number of projects involving experiments
   2002-03	6 of 13
   2001-02	5 of 18
   2000-01	6 of 12
   1999-00	7 of 17
   Total	24/60 or 40%

   
   
3. An ability to design thermal and mechanical systems, components, or processes to meet desired needs.
    
   

   	Ability (averaged score)	Excellent or very good (%)	Fair or Poor (%)
   Senior exit questionnaire (2002)	2.0	77.4	2.4
   Senior exit questionnaire (2003)	1.7	91.3	2.4
   Alumni survey (pre 2001)	2.4	57.6	19.2
   Alumni survey (2001-02)	2.1	74.1	3.7
   Curriculum committee grade and comments	B+: Integrated thermal and mechanical curriculum enables faculty to teach the thermal and mechanical topics from a systematic perspective, thus allowing students to work on more open-ended problems and design projects. This paradigm change in 1997 is extremely important for the program to satisfy this outcome.

   
   

   Although still considered mainly positive, pre-2001 alumni give a relatively low average of 2.4 (19.2% do not feel comfortable with this outcome). On the other hand, 2001-2002 alumni and graduating seniors survey data show a much more optimistic view on this aspect (74.1%, 77.4% & 91.3%) indicating a confidence in their design abilities. This improved perception is most likely the direct result of our recent emphasis on placing more design content in core courses and enhancing the overall design experience. All ME graduates are now required to pass a sequence of core ME engineering classes with emphasis on the design of thermal and mechanical systems, components, or processes.

   Design experience is culminated in a senior capstone design course which provides students with the design of a mechanical or thermal system (sometimes both). So, as seen on the tables below, almost all of the projects involve the design of mechanical or thermal systems (the few that did not, were involved either with a material selection problem, a control system, or some other aspect of mechanical engineering practice). Again, the detailed list of these projects is available in the Program Assessment Portfolio.

   Year	Number of projects involving experiments
   2002-03	11 of 13
   2001-02	17 of 18
   2000-01	10 of 12
   1999-00	17 of 17
   Total	55/60 or 91.7%

   
   

   It is safe to say that, without the full implementation of the integrated curriculum starting 1997, this particular outcome will be difficult, if not impossible, to achieve. For example, traditionally, the thermal area was covered in three separate subjects: Thermodynamics, Fluid Mechanics, and Heat Transfer. Without a comprehensive understanding of all three disciplines together, it is not possible to design a complete thermal system. By integrating these subjects into one course, the new thermal/fluids course allows students to better understand the inter-relationship between these subjects from a system perspective. In addition, it is possible to assign year-long design projects in the thermal/fluids. (Please refer to the course portfolio for a more detailed description of the thermal projects.) Similarly, in Mechanical Systems courses, open-ended problems are assigned emphasizing on materials, stress analysis, shaft design, bearings, joints, flywheels, among other design considerations. In summary, a "B+" grade is given to this outcome.

4. An ability to function on multi-disciplinary teams.
    
   

   	Ability (averaged score)	Excellent or very good (%)	Fair or Poor (%) (4/5)
   Senior exit questionnaire (2002)	2.2	66.7	7.1
   Senior exit questionnaire (2003)	2.4	55.0	20.0
   Alumni survey (pre 2001)	3.0	23.1	23.1
   Alumni survey (2001-02)	2.2	62.9	3.7
   Curriculum committee grade and comments	C+: We have only done an average job in preparing our students for multi-disciplinary (MD) collaboration. Continued efforts will be made to introduce more MD projects into senior design class. In addition, the present curriculum does not have enough "non-ME" courses to provide the breadth of exposure needed to be truly MD.

   
   

   A low rating of 3.0 has been given by pre-2001 alumni (only 23.1% positive while 23.1% not so favorable) on their ability to function on MD teams. Better but not great ratings (62.9%, 66.7 % and 55.0% positive) are given by 2001-02 alumni and graduating seniors in the past two years. This correlates closely to the recent increase of MD collaborations in senior capstone projects.

   To further assess this outcome, this outcome can be separated into two parts: (1) the students' ability to work on a team; and (2) their ability to function on MD collaborative tasks. We believe most of our graduates have problems with the second part of the outcome instead of the first one. Team building has actually been one of the strongest components in our curriculum since we encourage team projects and team building in all ME core courses. A three-hour weekly laboratory session is included in every core course for group work and collaborative learning (see individual course portfolios for more detailed description of this activity.)

   The second part of the outcome is more difficult to answer. As a matter of fact, the ME program is convinced that MD cooperation is important for the training of the next-generation engineers and is actively pursuing collaboration with other programs. In recent years, we have solicited MD student projects from the other engineering programs as well as from organizations outside the college such as the business school and the physics departments. However, this is still one of the most difficult attributes to establish since it requires the full commitment from all parties. This is usually not automatic and most of the time it is out of our control. A proposal to initiate a college-wide senior design project by ME in 2002 has not been adopted due to lack of interest from other departments. Our current approach is to actively seek external sponsored projects on our own that require MD collaboration and use this incentive to recruit students from other programs. In addition, we have been actively involved in the college Multi Disciplinary Technical Clinic (MDTC) program by participating in more than 80% of the MDTC projects in recent years.

   All things considered, we have made significant progress in increasing the fraction of ME projects that include students from other disciplines, as summarized in the tables below. (Note: we define a true MD collaboration as a project involving the active participation of disciplines outside ME. The integration of several sub-disciplines within the ME program, as most of our projects have already done, does not qualify as a MD project.) A detailed list of MD projects is included in the Program Assessment Portfolio.

   Year	Number of projects involving experiments
   2002-03	5 of 13
   2001-02	7 of 18
   2000-01	1 of 12
   1999-00	1 of 17
   Total	14/60 or 23.3%

   
   
   • One of our goals in satisfying this outcome is to have more than 50% of our projects have significant MD collaboration within two years.
      
5. An ability to identify, formulate, and solve engineering problems.
    
   

   	Ability (averaged score)	Excellent or very good (%)	Fair or Poor (%) (4/5)
   Senior exit questionnaire (2002)	2.0	83.0	2.4
   Senior exit questionnaire (2003)	1.4	95.1	2.4
   Alumni survey (pre 2001)	2.1	76.9	3.8
   Alumni survey (2001-02)	1.9	88.9	3.7
   Curriculum committee grade and comments	B+: A strong component in the program. Collectively, the integrated curriculum and the 3-hour lab format strengthen students' problem solving capability. However, they are still weak on both identifying and formulating problems. More open-ended problems will be introduced at the junior and senior levels.

   
   

   This is another strong component in our program. The integrated ME curriculum allows us to present engineering subjects from a system perspective thus enhancing students capability in dealing with engineering problems. In addition, the weekly three-hour lab periods in all ME core courses provide the learning atmosphere for all students to work together on more comprehensive engineering problems. An overall "B+" grade is given.

6. An understanding of professional and ethical responsibility.
    
   

   	Ability (averaged score)	Excellent or very good (%)	Fair or Poor (%) (4/5)
   Senior exit questionnaire (2002)	2.1	76.2	9.5
   Senior exit questionnaire (2003)	2.2	62.5	12.5
   Alumni survey (pre 2001)	2.8	38.4	23.1
   Alumni survey (2001-02)	2.3	66.7	18.5
   Curriculum committee grade and comments	C+: Should have done a better job in promoting professional and ethical responsibility. The added emphasis in Intro. to ME and Senior Design courses as well as the requirement of professional outreach participation should improve the situation.

   
   

   Several actions are currently being taken to improve this PO. We expect positive assessment data to demonstrate the improvement soon.

• Emphasis on ME as a profession early in the curriculum; openly discuss ethics and professional activities in Intro. to ME class (reformatted in Fall 2002). The term profession is defined in terms of public safety, moral and ethical responsibilities and the need to constantly update the knowledge through the life-long learning process. Various professional organizations are introduced and their roles in shaping the professions are outlined. The students are strongly encouraged to join professional societies and take the FE exam.
   
• Invite engineering practitioners to address students on ethics and other professional considerations in Senior Seminar course.

• Encourage students to participate in professional activities such as ASME, SAE regional and national conferences, and competitions. A good example is the participation of many ASME students in two recent ASME Town Hall meetings (2002 & 2003) on the discussion of many engineering and technology-related issues facing the state of Florida.
   
• Invite engineering practitioners to address students on ethics and other professional considerations in Senior Seminar course.

• Require students to participate in educational outreach activities such as Industry Day design competition, ASME Town Hall meeting, assume professional society leadership positions, etc. Students must fulfill specified outreach requirements during their college career before they can graduate (a detailed plan will be proposed in the Summer Retreat and the deliberation results will be available for review).
   
• Invite engineering practitioners to address students on ethics and other professional considerations in Senior Seminar course.

• Place more emphasis on professionalism and ethics in senior capstone class: Two full lectures of the senior design project are devoted to these topics. One of the lectures (in the context of e