CS 3240 - Specification Grading

Specifcation Grading

Specification Grading Chart: http://cs3240.cs.virginia.edu/specgrading

This course uses a points-free, competency-based assessment system that differs from traditional points-based systems in some very important ways. Under this system, the grade you earn in the course and the workload that you take on are a result of your choices, rather than the outcome of a statistical calculation. It is designed to provide you with control over the grading process, transparency as to your progress toward a course grade, and a final course grade that truly reflects your actual mastery of course concepts. In other words the grade you earn in the course is entirely up to you.

Balance Patches

Patch 1.1 - 10/12/2018

Now that we are well into the semester, it has become more apparent as to the appropriate time committment needed for the various Knowledge Areas. To address this, along with realigning the KAs with learning objectives, we have made the following changes:

  • Guided Practice percentages have decreased: With the number of missed opportunities for Guided Practice up to this point, we are reducing the overall percentage needed for a Pass and there is a committment to have roughly one a week for the remainder of the semester, providing around 8 total Guided Practice opportunities.
  • Requirements KA: The take home user story assessment has been removed. In discussions with the entire staff, it appeared that the elicitation activity was truly a team effort and is sufficient to cover this area.
  • Verification and Validation KA: Now that the Test Plan assignment has been created and released, the learning objectives from examining another team's Test Plan are not as obvious. The main learning objective from this KA is the basic knowledge of testing techniques and their application. Thus measuring your contributions to the team project seem fitting.

If you have any questions about these balance changes, please talk to Prof. Sherriff directly.

Theory Behind Using Specification Grading

In nearly every other course you have taken, there is a notion of partial credit on many assessments. You may have not gotten a solution exactly right, but you followed most of the right algorithm. Or perhaps you created a program that works for most of the possible inputs, but not all.

In industry, however, that "partial credit" distinction fades quickly. If your software fails, it's impact could be anywhere from minor convenience to a literal catastrophe. The purpose of this course is to teach you about software engineering, often defined similarly to the following:

"...the systematic and regular application of scientific and mathematical knowledge to the design, construction, and operation of machines, systems, and so on of practical use and, hence, of economic value. Particular characteristic of engineers is that they take seriously their responsibility for correctness, suitability, and safety of the results of their efforts. In this regard they consider themselves to be responsible to their customer (including their employers where relevant), to the users of their machines and systems, and to the public at large..."

Therefore, in this course, we will eschew the uses of traditional points-based, percentage-based grading, and instead focus directly on you showing competency in the core learning objectives and material of the course.

Establishing Your Grade

Each letter grade level has a set of criteria you must meet in order to earn that grade. Plus (+) levels are for students who earn a particular level grade, but excel beyond that level in a few key areas. Minus (-) levels are reserved for issues of professionalism lapses by the student (e.g. excessive computer use in class to the point of disruption, excessive absences, rude behavior, etc.). Minus levels will not be assigned as a part of the normal assessment of the course.

In general, students who attain a particular grade could be described as such:

  • Students earning an A in this course will have shown compentency in all major knowledge areas of the course. These students will have good to excellent attendance in both lecture and lab, participating in class activities. These students will also be a contributing part of a successful project team, completing nearly all of the project sprints on time and creating a final product that is not only acceptable to the customer, but shows a level of care about the creation of the prodcut.
  • Students earning a B in this course will have shown compentency in all but one of the major knowledge areas of the course. These students will have overall good attendance in both lecture and lab. These students will be a contributing part of a successful project team. However, the completed project may not have the level of polish expected by the course staff.
  • Students earning a C in this course will have shown compentency in two-thirds of the major knowledge areas of the course. These students may be lacking in attendance. These students may also may have not been a contributing memeber of their project team. Failure to be a participating member of a project team, when the course is built around the idea of learning to work in a professional software development environment, carries significant consequences.
  • Students earning a D in this course will have shown compentency in half of the major knowledge areas of the course. Any fewer than this is not considered to be a passing grade. These students most likely have poor attendance and/or were not major contributors on the project team. This is the highest level that any student on a team that does not complete the final course project can attain.

Specification Grading Chart

The chart outlining the criteria for the letter grades can be found at: http://cs3240.cs.virginia.edu/specgrading

To attain your desired level on the chart, you must meet all of the criteria identified across that level. Failure to meet even one aspect will move a student down to the next level.

Knowledge Area (KA) Mastery

One of the primary criteria on the Specification Grading Chart is the Knowledge Area (KA) Mastery criteria. This criteria represents competency in the six different content areas of the course:

  • Software Process: understanding how the different software development methodologies work and how to implement them in a moderate-sized software project;
  • Requirements: how to elicite requirements from potential customers and model those requirements as actionable specifications for a development team;
  • Verification and Validation: understand the various ways we ensure that the software we build not only works, but also does what the customer wants;
  • Architecture and Design: the ability to take a set of specifications and requirements and create a design that will address the customers' needs and generate documents and artifacts that effectively communicate those design decisions to other team members;
  • Security and Maintenence: what can we do during the development process to ensure we are building software that is sturdy against attack and can live into the future as other developers make modifications; and,
  • Professioanl Issues: what are your ethical and professional obligations as a software engineering.

To show competency in any given area, a set of assessments can be found on the Specifcation Grading Chart. These assessments will be made available throughout the semester.

Attendance and Participation

Attendance and participation in the course is essential for learning the material and being a successful, contributing member to your project team. Therefore, we will assess your attendance and participation as follows:

  • Lab attendance will be taken each week electronically, using the RFID chip in your UVA student ID. Failure to register your attendance in a timely fashion at the beginning of lab will result in a recorded absence. Coming to lab over 15 minutes late without a legitimate reason will also be recorded as an absence.
  • During lecture on certain days, we will have Guided Practice activities, which include such things as partnered worksheets on current course material or guest speakers. These will occur throughout the semester, and while some will be announced, some will not.

Team Project and Performance

The team project is a major component of this course and is the vehicle by which students can put into practice the material that is covered in lecture regarding the software devleopment process. We expect any student that is successful in this course will participate fully in the development of the project along, while also being a good teammate. We will assess a student's participation in the project as follows:

  • On a scale of 0-10, students will recieve a number of evaluations throughout the semester from teammates and the course staff, based on the following chart:
    • 10: Student was a solid, contributing member throughout the project, often going above and beyond what was needed. Student fulfilled their team role exceptionally well.
    • 9: Student was a solid, contributing member throughout the project. Role with the team may have been lacking slightly.
    • 8: Student was a solid, contributing member throughout the project. Student may have had to be reminded once or twice on a few minor things. Overall, the student was responsible for thier work.
    • 7: Student was a contributing member throughout the project, but fell short in a few instances. Overall, though, the student was seen as a reasonable member of the team.
    • 6: Student was a contributing member, but fell short, and another team member had to make up their work.
    • 5: Student contributed to the project, but was seen as unreliable for some tasks.
    • 4: Student contributed, but required excessive reminders and/or was often late. Any contributions created required rework by others.
    • 3: Student contributed at least something, but was largely absent.
    • 2: Student did not contribute any code, but did come to meetings and was present in lab for discussion.
    • 1: Student was not present.
    • 0: Student was actively a hinderance to the team, preventing overall progress, regardless of the students's individual contributions.
  • Each week during the team project, a team will present their current sprint progress to course staff. Based on a detailed specification made available to the team ahead of time, the course staff (a TA, group of TAs, and/or the instructor) will determine whether the sprint is rated as a Pass or No Pass.
  • As the team project is a major portion of the class, we expect students to work together to build something that not only "works," but shows a level of care, polish, and attention that a student would be proud to add to their professional portfolio. Therefore, the team project will be assessed as one of the following:
    • Completed and Well-Reviewed: A project at this level will meet all customer requirements, is well-recevied by the customers, and, in the opinion of the course staff, meets a reasonable level of polish and care.
    • Completed: A project at this level will meet all customer requirements, but, in the opinion of the course staff, is obviously lacking at attention to detail or is overall sloppy. At this level, the project may be rejected by a manager that a developer is working for.
    • Insufficient: A project at this level does not meet all of the customer requirements, or simply does not work.

Assessment Evaluation

Each assessment, sprint, or activity will come with a detailed grading specfication and will be graded as Pass, No Pass, or Not Yet. Only the team project will have a different set of levels. For all assessments in the course, a Pass is only awarded for work that meets the following criteria:

  • Correctness: Any software or documents you create should be free from all but minor errors. Any error that detracts from the purpose of the document or major functionality of the software will be seen as lacking and will not earn a Pass.
  • Professionalism: Any software or documents you create should be professionaly formatted and presented. This includes, but is not limited to:
    • Commenting code / self-documenting code: Headers on files are a must, with some in-line comments where necessary
    • Formatting: All code and documents should follow a clean, professional format that is easy to follow. Code should follow all appropriate coding standards.
    • Language: Professional language should be used at all time, which includes grammar, sentence stucture, and clarity of thought.
  • Care and Detail: As UVA students, we expect a level of attention to detail and pride in the work you create. Any work that is seen as "rushed" or otherwise sloppy in some regard, will not earn a Pass.

A Not Yet will be given in the circumstances where an assessment meets all of the above criteria except for some minor aspects that, in the staff member's opinion, would be reasonable to fix without significant effort. Knowledge Area Core Assessments will not use the Not Yet level.

Resubmission of Assessments

Because of the nature of specification grading, it is likely that all students will receive at least one No Pass at some point during the semester. The purpose of this grading system is not to penalize students, but to establish a level of quality and competency that matches course expectations. It is up to the student to determine which level of competency they want to work toward based on the Specification Grading Chart.

Thus, the ability to revise and resubmit your work is a key component to this grading system, and all assessments have some mechanism by which you can take the comments from the staff, improve upon your submission, and have it reevaluated.

However, there needs to be a balance between this concept and simply turning every assessment in twice, because that also is not realistic for professional work. You need to become competent in assessing your own work to determine if it meets specifications and is of high-enough quality.

There are three modes of assessment, as outlined on the Specification Grading Chart: In-Class, Take Home, and Project.

  • In-Class assessments will either be Knowledge Area Core Assessments, as outlined in the section below, or Guided Practice. Guided Practice activities are assessed by participation and are thus not eligible for resubmission.
  • Take Home assessments happen infrequently during the semester. Every student has two "tokens" that they can spend to resubmit a Take Home assessment. Once these tokens are used, there are no opportunities for resubmission.
  • Sprints for the project can be reevaluated during that week's office hours. More guidance on what is necessary for a reevaluation can be found with the sprint specfications. Each team has two "tokens" that they can spend to have a sprint reevaluated. Once these tokens are used, there are no opportunities for reevaluation.
  • Major project activities (such as the requirements document and design document) that fall specifically to a particular member of a team (such as the requirements manager) can be reevaluated by the responsible member using one of their personal "tokens" under the same rules as the other Take Home assessments.

Any assessment that is given a Not Yet does not require the use of a "token" for reevaluation. However, it will not be converted to a Pass until all requested changes have been made.

Knowledge Area Core Assessments

Each of the six Knowledge Areas has a corresponding Core Assessment that covers the key material or concept in that Knowledge Area. During the semester, there will be three Core Assessment Periods in addition to the final exam period in which students can take any of the six Core Assessments. As noted on the Specification Grading Chart, most students will complete at least 5 of the 6 Core Assessments during the semester.

If a No Pass is given for any Core Assessment, a student can attempt a different version of the Core Assessment during a later Core Assessment Period. The second version of the Core Assessment covers the same material, but would differ in the exact question or scenario (e.g. a different set of test cases is presented for analysis for the Verification and Validation Core Assessment). Attempting a Core Assessment for a second time does not use any of a student's "tokens."

Students who earn two No Pass evaluations on the same Core Assessment cannot attempt that assessment again.

Some language and information courtesy Prof. Robert Talbert, Grand Valley State University

more ...