To give a bit of scope to our efforts, let me break down the hierarchy of the process as it relates to learning and assessment. The institution has its mission. For DSU it looks like this:
It follows, then, that this mission drives what happens on a college or department level. For our department it currently looks like this:
.jpg)
This needs to be worked a bit to evolve specific goals of the department, but it's evident we're here to create competencies in communication theory, content creation and ethics.
These goals then drive outcomes, what we're calling Program Learning Outcomes. Ours look like this:
.jpg)
The Program Learning Outcomes help frame the most important part of the program, its curriculum and how it is assessed. The curriculum is developed from course learning objectives in each class offering in the degree. These are the objectives for just one course with our program and I'm using them just as an example:
.jpg)
Understand that not every outcome applies to every class and not all are assessed on the same level.
For example, for any given class offering there are a number of learning levels, both academic and experiential, that you will want to achieve with students ranging from simple knowledge, conceptualization, comprehension, and application, to beyond application, as your Class Learning Objectives relate to the Program Learning Outcomes.
That’s the purpose here, to understand and design that link between the Program Learning Outcome and the Class Learning Objective so that valid assessment approaches can be created to measure course effectiveness and teaching efficacy. It is based on this efficacy that we measure and evaluate students and grant to them the credentials of the program not to mention how the program gains and retains accreditation.
Let’s define a couple of terms. Outcome is the word we’re using to create assessment criteria on the degree level. Objective is the term that denotes what is evaluated and assessed on a class level.
For demonstration purposes I'm using an old course, COMM 3650, TV Field Production and Reporting, to show how the course may be driven by departmental Program Learning Outcomes that frame the Course Learning Objectives.
Program Learning Outcomes by Theme
Students who graduate with a Bachelor of Science in Communication will be able to demonstrate competency in:
For demonstration purposes I'm using an old course, COMM 3650, TV Field Production and Reporting, to show how the course may be driven by departmental Program Learning Outcomes that frame the Course Learning Objectives.
Program Learning Outcomes by Theme
Students who graduate with a Bachelor of Science in Communication will be able to demonstrate competency in:
Theory - The conceptual understanding of mediated communication theories from various traditions and practical understanding of media production, management and consumption.
Critical Thinking - The ability to apply good reasoning, critical thinking, ethics and problem solving skills to critically evaluate media production and consumption.
Analysis - The ability to conduct professional research to evaluate efficacy of mediated messages integrating media analytics and metrics into qualitative and quantitative approaches.
Research - The ability to conduct academic research, explore conceptual issues in mediated communication, and make academic arguments with established research methods.
Content Creation - The skills to create and promote mediated messages in compliance with technical and aesthetic principles for a variety of purposes based on in-depth analysis of the audience and situation.
Critical Thinking - The ability to apply good reasoning, critical thinking, ethics and problem solving skills to critically evaluate media production and consumption.
Analysis - The ability to conduct professional research to evaluate efficacy of mediated messages integrating media analytics and metrics into qualitative and quantitative approaches.
Global Perspective - Demonstrate effective cross-cultural communication knowledge and skills in achieving a global perspective.
Next, let’s identify the Course Learning Objectives for TV Field Production and Reporting.
Course Learning Objectives Identified by Theme
Through regular attendance and active participation you will be able to:
Baseline Research - Evaluate current ENG production on local, regional and national levels,
explain basic terminology and concepts of EFP and ENG.
Through regular attendance and active participation you will be able to:
explain basic terminology and concepts of EFP and ENG.
Investigate - Gather information through effective interviewing, traditional research methods, email and other electronic query, and world-wide web sources.
Write - Effectively write for different applications in ENG including stand-ups, voice-overs, and on-camera narratives.
Produce - Shoot and engineer quality video for stand-ups, interviews and multi-camera production for broadcast, acquire and mix audio for EFP.
Perform - Effectively deliver and record stand-ups, voice-overs and narratives for stories and packages.
Edit - Produce media for broadcast including B-roll, SOT, VO-SOT, stand-ups and packages,
Competency - Function in production positions such as field producer, photographer, and reporter, and studio positions such as producer, director, technical director, replay, graphics, audio, operator, and anchor.
Deliver - Distribute news content over the air and via the Internet.
Analyze - Gather data on content distribution, consumption and feedback via media analytics.
In the hierarchy of the pedagogy of the department as it relates to field news production, framing the objectives based on the outcomes might look something like this:
Outcomes - In Bold
Objectives - In Italics
Theory
All Objectives
Research
Baseline, Investigate, Analyze
Content Creation
Investigate, Write, Field Produce, Perform, Edit
Critical Thinking
Investigate, Field Produce, Competency, Analyze
Analysis
Analyze
Global Perspective
Investigate, Write, Field Produce, Perform, Edit
I’ve framed these relationships between Outcomes and Objectives based on two things; desired cognitive levels of learning (listed below), and weight or the priority of the objective as I’ve determined as a teacher based on the needs of the students and the class (in the matrix below). This is my stewardship as a teacher.
Desired Cognitive Levels of Learning
These levels are drawn and interpreted from instructional design theory and from experience in teaching. These levels are simple knowledge, comprehension, application, conceptualization, and beyond application.
Simple Knowledge - Discusses and distinguish ideas
Comprehension - Relates and critique concepts
Application - Implements and practices ideas, concepts and theories
Conceptualization - Creates original thinking and approaches
Beyond Application - Synthesizes and teaches concepts
Again, these are my determined cognitive levels. Yours may be different, but it's important that your levels inspire the levels of the Program Learning Outcomes for the curriculum map. These are:
Introduce
Develop
Mastery
Here's how it comes together (and this is the most important aspect to understand in developing valid assessment tools).
In my curriculum development, class discussion and lecture target simple knowledge under the outcome to introduce. Planned activities lead to comprehension and application under the outcome to develop. Student-driven projects or small group activities inspire conceptualization and beyond application competencies under the outcome to master.
Assessment activities are designed to comprehensively measure achievement on all or any targeted learning level.
Weight and Priority
In framing objectives and designing curriculum, I take the relationships now created between the outcomes and the objectives, identify my desired cognitive levels of learning I want to reach with my students and then weigh and prioritize those objectives accordingly. To do so I let the main verb of the objective relate to the desired cognitive level and weight given to the objective.
It looks something like this:
Baseline
|
Simple Knowledge
|
4
|
10%
|
Investigate
|
Conceptualization
|
1
|
14%
|
Write
|
Application
|
1
|
12%
|
Produce
|
Conceptualization
|
1
|
14%
|
Perform
|
Application
|
2
|
12%
|
Edit
|
Conceptualization
|
2
|
10%
|
Competency
|
Beyond Application
|
2
|
10%
|
Deliver
|
Application
|
3
|
6%
|
Analyze
|
Beyond Application
|
3
|
8%
|
With content weighed and prioritized, learning levels identified, and course objectives defined, I can now develop the curriculum. According to the priorities, it looks like I want students to come away from the class knowing how to investigate, write, and produce, the foundation of electronic news gathering. A close second is their ability to perform and edit with a degree of general competency in all areas of production. Third, they should be able to deliver content and analyze its metrics, all with having performed some kind of initial baseline (fourth) against which I can measure their achievement.
The weight assigned to each might help me mete out the quantity and type of assignments, activities and assessments. Should I decide to be comprehensive in assessment, the instrument should reflect the weights of each covered objective in the number of stems and/or in the number of points assigned each measure.
The item pool I developed for this class looks like this (taken from the syllabus):
Item Pool
These activities are designed to get you discovering different aspects of real-world electronic news and content gathering and dissemination, developing a working knowledge of specific production protocols, and qualifying to use production equipment in this program. While much emphasis is placed on technical skills related to this end, high expectations exist in writing for this medium.
1. News Gathering
Develop a local story with regional or national interest. Gather facts from at least four stratified sources; interviewing, electronic correspondence, the Internet, and traditional publication. Document your information gathering and qualify your sources. Present your findings in an editorial meeting and submit your written work by the deadline indicted on the schedule.
2. Write to be Heard
Once approved by editorial, write your story. Choose your style; inverted pyramid, chronological, news narrative, or focus structure.
3. Shooting B-roll
Shoot B-roll to support your story. Execute news-photography techniques shooting general and wide, then paring down focus to transitional and detail shots. All aspects of videography should be practiced including proper set-up with a tripod, white-balancing, rolled bars for :30, and good composition. B-roll should ALWAYS be supplemented with natural (NAT) sound.
4. Stand and Deliver
Demonstrate your on-camera credibility and charisma in recording a stand-up and/or voice-over (VO) for your story package using proper microphone technique.
5. The Package
Produce your package and relevant media. Initially you will need to find an editor to assist you if you don't have non-linear editing editing practice. If you do, you'll be expected to help and teach other reporters in your cohort. Your package should be no longer that 2:00 and should include a stand-up, an on-camera interview, B-roll with NAT sound, and a toss. Then cut a VO-SOT and a NAT-SOT of the same story.
7. Produce
Collaborate with peers to produce at least six news broadcasts and engage in a minimum of four studio production positions including producer, director, technical director, floor director, reader, audio mixer, camera operator, and all field production positions including field producer, reporter and photographer.
Assessments
All assessments are designed to give me an idea of how you’re doing in understanding and applying the objectives of this course, and how I’m doing in teaching them. Assessments may be a combination of multiple choice, true/false fill-in-the-blank, and short answer. You will be accountable for and assessed on principles discussed in class and in the book.
1. News Gathering
Develop a local story with regional or national interest. Gather facts from at least four stratified sources; interviewing, electronic correspondence, the Internet, and traditional publication. Document your information gathering and qualify your sources. Present your findings in an editorial meeting and submit your written work by the deadline indicted on the schedule.
2. Write to be Heard
Once approved by editorial, write your story. Choose your style; inverted pyramid, chronological, news narrative, or focus structure.
3. Shooting B-roll
Shoot B-roll to support your story. Execute news-photography techniques shooting general and wide, then paring down focus to transitional and detail shots. All aspects of videography should be practiced including proper set-up with a tripod, white-balancing, rolled bars for :30, and good composition. B-roll should ALWAYS be supplemented with natural (NAT) sound.
4. Stand and Deliver
Demonstrate your on-camera credibility and charisma in recording a stand-up and/or voice-over (VO) for your story package using proper microphone technique.
5. The Package
Produce your package and relevant media. Initially you will need to find an editor to assist you if you don't have non-linear editing editing practice. If you do, you'll be expected to help and teach other reporters in your cohort. Your package should be no longer that 2:00 and should include a stand-up, an on-camera interview, B-roll with NAT sound, and a toss. Then cut a VO-SOT and a NAT-SOT of the same story.
7. Produce
Collaborate with peers to produce at least six news broadcasts and engage in a minimum of four studio production positions including producer, director, technical director, floor director, reader, audio mixer, camera operator, and all field production positions including field producer, reporter and photographer.
Assessments
All assessments are designed to give me an idea of how you’re doing in understanding and applying the objectives of this course, and how I’m doing in teaching them. Assessments may be a combination of multiple choice, true/false fill-in-the-blank, and short answer. You will be accountable for and assessed on principles discussed in class and in the book.
What’s the point of all this? Simply to clarify the learning. These priorities and weights now help me design activity, evaluation and assessment tools. These items allow me to measure achievement based on the amount of time and evaluation I’ve dedicated to each objective, which then relates to my efficacy in reaching the related Program Learning Objectives.
But here's the critical part - having followed this pedagogy, my assessment instruments (rubrics) almost write themselves. As I follow the clarified learning path I can quantify achievement through point values and qualify efforts through descriptors.
One of the rubrics from the item pool looks like this:
News Gathering Rubric
Objective:
Gather information through effective interviewing, traditional research methods, email and other electronic query, and world-wide web sources.
Point Value: 40
Activity Description
Develop a local story with regional or national interest. Gather facts from at least four stratified sources; interviewing, electronic correspondence, the Internet, and traditional publication. Document your information gathering and qualify your sources. Present your findings in an editorial meeting and submit your written work by the deadline indicted on the schedule.
Rubric
1. The reporter develops a local story lead that has regional and/or national interest.
2. The reporter gathers information from at least two interview sources.
3. The reporter gathers information via electronic correspondence, the Internet and from traditional publication such as local newspapers.
4. The reporter documents and qualifies sources and presents their story and findings to editorial.
Gather information through effective interviewing, traditional research methods, email and other electronic query, and world-wide web sources.
Point Value: 40
Activity Description
Develop a local story with regional or national interest. Gather facts from at least four stratified sources; interviewing, electronic correspondence, the Internet, and traditional publication. Document your information gathering and qualify your sources. Present your findings in an editorial meeting and submit your written work by the deadline indicted on the schedule.
Rubric
1. The reporter develops a local story lead that has regional and/or national interest.
Wanting (0-2) Developing (2-3) Accomplished (4-5)
2. The reporter gathers information from at least two interview sources.
Wanting (0-4) Developing (5-9) Accomplished (10-13)
3. The reporter gathers information via electronic correspondence, the Internet and from traditional publication such as local newspapers.
Wanting (0-4) Developing (5-9) Accomplished (10-12)
4. The reporter documents and qualifies sources and presents their story and findings to editorial.
Wanting (0-3) Developing (4-7) Accomplished (8-10)
Rubrics afford many advantages in curriculum development, not the least of which is that students know (assuming they read them) what is expected from their engagement. A critical advantage is they create data through which I can a) see how they're doing, and b) see how I'm doing.
What needs to happen on a course level, for every class in the program, is a clarified learning approach that validates your curriculum and measurement practices, advancing data back to the institution that demonstrates the function of the program.
