🗂️ Reimagining Assessments in the Age of AI
Explore new ways educators are designing assessments to gauge student understanding and skill mastery
The rise of AI in education sparks a fundamental reevaluation of assessments. Rather than regurgitating information, the priority of assessments could instead shift to the higher-order skills that will enable students to apply their knowledge and skills to solve complex problems, innovate, and contribute meaningfully in real-world contexts. While the integration of AI tools in classrooms remains relatively new, how frequently do you permit students to utilize Generative AI tools during assessments?
Here is an overview of today’s email:
“Performance-based Assessments” targeting real-world applications of knowledge and skills
“Two-lane” approach to assessment at the University of Sydney
Student and educator perspectives on the role of AI in education
AI usage aimed to provide individualized and high-quality feedback to students
Google’s new AI tool, Gemini, and its potential implications and impact in the field of education
If you have any suggestions on how we can improve our future newsletters, please submit them below!
🚀 Practical AI Usage and Policies
The history of standardized testing dates back as early as 1838, but after over 175 years of use, research shows these tests are often “inaccurate, inequitable, and ineffective” at assessing true student understanding and their ability to apply their knowledge in other contexts. Rather than relying solely on standardized tests, the National Education Association advocates for more performance-based assessments “that allow students to demonstrate knowledge and skill through critical thinking, problem-solving, collaboration, and the application of knowledge to real-world situations”.
Performance-based assessments can take many forms - conducting science investigations, curating portfolios that highlight growth over time, producing original research papers, creating art or poems that demonstrate conceptual understanding. This focus on application, analysis and creation aligns with the upper tiers of Bloom's Taxonomy.
While current forms of assessment often focus merely on facts and rote memorization — which students can typically get by with last-minute cramming and quickly forget — this age of advanced AI makes fostering originality and critical thinking in students more crucial than ever.
By harnessing Generative AI, educators can design assessments that prompt students to demonstrate their understanding through tasks that focus on creativity, originality, and deeper conceptual mastery. These can be assessments where students generate hypotheses, design experiments, compose music, or develop innovative solutions to real-world problems. These tasks not only assess their comprehension but also foster skills essential for success beyond the classroom.
Moreover, Generative AI can facilitate personalized assessments, catering to individual learning styles and paces. It can adapt prompts and challenges based on each student's progress, providing tailored evaluations that focus on their strengths and areas needing improvement.
In fact, institutions such as the University of Sydney have recently revised their assessment guidelines to stay up to date with the integrations of AI. The University of Sydney has developed a “two-lane” approach to assessment that prioritizes students’ “assessment of learning”, which are supervised assessments authenticating students’ mastery of specific knowledge and skills, and “assessment for learning”, unsupervised assessments that encourage students to engage responsibly with AI in their learning process. You can learn more about their approach here: https://educational-innovation.sydney.edu.au/teaching@sydney/embracing-the-future-of-assessment-at-the-university-of-sydney/
This blog from the University of Wisconsin Green Bay Center for the Advancement of Teaching and Learning also offers some practical strategies and tips for creating “Generative AI-Resistant” assessments: https://blog.uwgb.edu/catl/strategies-for-creating-generative-ai-resistant-assessments/
For educators looking into various methods of alternative assessments, The Colorado School of Mines offers a helpful compilation of resources in this bibliography: https://docs.google.com/document/d/1TqpoJqlrIv0sQmvxY8v5yWVvoztFNWIGd7Yg5mwdXiI/edit#heading=h.xdw6pnrb6mv7
By transforming assessments through the power of Generative AI, we can create a culture of open-ended exploration where students are motivated to continually refine their understanding. For example, asking students to formulate new test questions or design simulations allows them to synthesize knowledge in creative ways. Coupling this with conversational AI tutors that adaptively respond to the learner’s needs provides responsive scaffolding. Assessment becomes not an endpoint, but a constructive dialogue where critical thinking and innovation are intrinsically rewarded.
By embracing these alternative models, we can shift focus from rote memorization to directly nurturing the skills and mindsets needed for success in a rapidly changing world. The goal becomes not just scoring well on a test, but developing learners’ capacity to pose ambitious questions, undertake self-directed research, and think systematically - all while sustaining their innate curiosity to understand the world around them.
Long, Cindy. “Standardized Testing Is Still Failing Students.” NEA, 30 Mar. 2023, www.nea.org/nea-today/all-news-articles/standardized-testing-still-failing-students#:~:text=Key%20Takeaways,engaging%20for%20students%20and%20teachers. 📣 Student Voices and Use Cases
Vox recently released a video examining the evolving landscape of AI integration in education, featuring interviews with students and educators about their experiences with AI technology in the classroom. Check out the video here:
We also wanted to learn firsthand from students to gain insight into their perspectives on the role of AI in education. This week, we had a chance to speak with Jiwang, a first-year student at Rice University studying Sports Medicine and Exercise Physiology. In the following, we present select highlights from these conversations, which have been slightly edited for enhanced clarity and precision:
Q: In what ways have your teachers integrated AI tools into the classroom, and could you offer a specific example of how you interacted with these tools?
Several of my professors have slowly integrated AI tools into their curriculum. Most notably, AI technology was often used in my first-year writing seminar course: Water and Cities. Given the specific nature of the course’s content, using ChatGPT to produce full papers would be of little benefit. As a result, my professor openly encouraged students to utilize ChatGPT in several assignments to assist the research process. In fact, consulting with ChatGPT was a required element of our final project, which addressed a major water issue in a city of our choice. My work revolved around Dallas and the Trinity River, though my conversation with ChatGPT demonstrated the tool’s limitations. When asked about the interplay between the two, it produced a highly generic answer that lacked the nuance that from pertinent research studies and historical documents.
Q: How has using AI tools impacted your learning habits, critical thinking, or problem-solving skills, if at all?
ChatGPT can seamlessly complete most of my homework questions or problem sets, though I extract the most benefit when I ask the AI questions about holes in my knowledge. As a result, ChatGPT has been incredibly helpful when learning new concepts or during the studying process. The AI can read my thought process incredibly well, and I can ask it highly niche questions that my professors would likely struggle to offer a straightforward answer to.
Q: From your perspective, how might AI impact the future of assessments and evaluations in education? What advice would you give educators or institutions looking to implement AI in evaluating students' understanding of different topics?
With the rapid proliferation of AI tools, I believe the future of assessments and evaluations will look dramatically different from past examinations. Gone will be the days of evaluations focused on applied memorization. Rather, I feel that future assessments will test students’ ability to communicate information obtained from AI tools in an effective, concise manner. Additionally, students may be tested on their understanding of certain concepts directly with AI tools. Students could be asked to debate an AI on a given topic, hold an engaging conversation, or recognize programmed faults in an AI-produced text. To educators and institutions looking to implement AI tools into their curricula, I would advise them to lean into the near-limitless abilities of AI tools. These technologies may be used to expedite current processes, but I think this overlooks the tremendous capabilities of these tools. AI may be able to create individualized tests for each student, dynamically adjusting difficulty and wording as each student progresses.
📝 Latest Research in AI + Education
International Society of the Learning Sciences
The study is set in a semester-long digital fabrication course at a graduate school, where 19 students used standard makerspace tools for personal projects. The research focuses on whether AI can assist in managing the demanding task of monitoring students and providing tailored feedback.
The authors describe the development of a platform that utilizes GPT-3 to generate encouraging messages and summarize student blog posts, aiming to ease the feedback process for instructors. They found that GPT-3 was adept at summarizing work and offering positive reinforcement. However, it struggled to provide appropriate feedback to students facing difficulties, often generating insensitive or off-target responses.
Key findings include the necessity of human oversight in refining AI-generated feedback, especially for nuanced situations. The study notes that while AI can streamline the feedback process, it requires a balance of human expertise and AI capabilities. The research highlights the importance of understanding AI's limitations and integrating human inputs effectively. Future work will focus on enhancing AI feedback quality, incorporating diverse data sources, and involving instructors and students in a co-design process to improve the system's effectiveness.
Sung, G., Guillain, L., & Schneider, B. "Can AI Help Teachers Write Higher Quality Feedback? Lessons Learned from Using the GPT-3 Engine in a Makerspace Course." Proceedings of the 17th International Conference of the Learning Sciences - ICLS 2023, edited by P. Blikstein, J. Van Aalst, R. Kizito, & K. Brennan, 2023, pp. 2093-2094.Canva
Canva’s AI in Education Study: Teachers embrace power of AI ↗️
Canva's AI in Education Study reveals that teachers are eager to embrace artificial intelligence (AI) in the classroom but face challenges in knowing how to effectively integrate it. The study, which surveyed 1,000 U.S. teachers, found that while 78% of educators are keen on incorporating AI into their teaching, 93% admit uncertainty about where to start. This indicates a significant knowledge gap and a need for more support in adapting to rapidly advancing technology.
Despite this, teachers recognize the benefits of AI in boosting productivity, creativity, and reducing administrative workloads. The study shows that 60% of teachers find AI sparks innovative ideas for student productivity, 59% believe it fosters student creativity, and 56% see it as a tool to reduce administrative burdens. Additionally, a notable majority see AI as beneficial for students with different learning needs, aiding in language learning and universal accessibility.
Canva's commitment to integrating AI into education focuses on safety, accessibility, and transformation. The study highlights the potential of AI in personalizing education and empowering students, aiming to bridge the AI knowledge gap and make AI a transformative force in global classrooms. The company is dedicated to being a responsible first point of contact for teachers engaging with AI technology, envisioning a dynamic future for education.
Barrett, Alex, and Austin Pack. "Not Quite Eye to A.I.: Student and Teacher Perspectives on the Use of Generative Artificial Intelligence in the Writing Process." International Journal of Educational Technology in Higher Education, vol. 20, no. 1, 2023, pp. 1-24.📰 In the News
The Conversation
Australia Has Its First Framework for AI Use in Schools but We Need to Proceed With Caution ↗️
Key takeaways:
AI Framework in Schools: Australia introduces a framework for generative AI in schools, guiding safe and effective use, with emphasis on potential benefits and risks like bias and data misuse.
Framework Details: It includes six principles and 25 guiding statements focusing on understanding AI, promoting well-being, ensuring fairness and privacy, and maintaining security.
Expert Concerns: Experts caution the framework's high demands on schools and teachers, citing challenges in comprehending AI complexities and adapting educational practices.
Improvement Suggestions: Recommendations for the framework include recognizing AI biases, demanding evidence of educational benefits, addressing chatbot dangers, ensuring transparency, and valuing teachers' expertise.
Teacher Roles and Global Context: Amid a US$300 billion global education tech market, the framework should prioritize preserving teachers' roles and expertise in the face of increasing AI integration.
The Guardian
Google says new AI model Gemini outperforms ChatGPT in most tests ↗️
Key takeaways:
New AI Model: Google has introduced Gemini, a new AI model that reportedly surpasses ChatGPT in most tests, showcasing advanced reasoning abilities and the capacity to handle various formats, such as grading physics homework.
Versions and Features: Gemini is available in three versions, including Ultra, Pro, and Nano. Ultra has outperformed human experts and other state-of-the-art AI models in various benchmarks, including reasoning and image understanding. The model is multimodal, capable of processing text, audio, images, video, and code.
Integration and Release Plans: Gemini will be integrated into Google products like its search engine and Bard chatbot. The most powerful Ultra version is slated for public release in early 2024.
High Score on Subject Tests: Ultra is the first AI model to outperform human experts, with a score of 90%, on a multitasking test called MMLU, which covers 57 subjects including math, physics, law, medicine, and ethics.
Challenges and Capabilities: Despite its advanced capabilities, Google acknowledges ongoing challenges with the model, such as resolving "hallucinations" or false answers. Promotional demonstrations highlight Gemini's proficiency in tasks like interpreting handwritten physics homework and identifying objects in images, pointing towards its potential in diverse applications.
“Chatgpt.” ChatGPT, OpenAI (GPT-4), openai.com/chatgpt. Accessed 11 Dec. 2023. And that’s a wrap for this week’s newsletter! Based on the results from our previous newsletter poll, we discovered that the majority of educators who took our survey felt that AI may influence students’ analytical reasoning abilities the most.
If you enjoyed our newsletter and found it helpful, please consider sharing this free resource with your colleagues, educators, administrators, and more.
I think Sydney University is on the right track by having two types of assessment, one of which is supervised and the other unsupervised and for which use of AI is encouraged.
In any profession or life pursuit from now on AI will be an essential tool, so we should assess how well our students have learned to use it.
Perfect time. I was about to start writing a section of an article on AI and assessment today. I really love what you folks are doing. Some of the best AI/Education analysis in the business.