Recently, a student came into my office. They were having a problem getting their assignment to work. I asked what they had tried, and they said "I tried Claude, I tried Gemini, and I tried ChatGPT." This was a very concerning interaction to me. What bothered me was not the AI use (which is allowed by course policy), but that the student didn't have a good process for making progress once they were stuck. As we worked through this student's problem, it became clear that they were not developing a mental model of what the problem was, or trying to break the problem up into smaller problems, but instead were trying to solve the problem all at once with a single prompt to an LLM. I call this anti-pattern "slot machine programming," where the student just kept re-using the prompt hoping the next answer would be correct. Similar to playing slot machines, where each play is statistically independent from the last, running the same prompt through different AI tools doesn't help a student make progress towards a solution.
Unfortunately, this is a pattern that I am noticing more and more among students. While I have heard professors assert that students use AI because they are "lazy," that is often not the case for many of my students. In fact, I would say sometimes the students who are (mis)using AI are working harder than the students who are not, because they are not working efficiently, and therefore are spending a bunch of time and effort not making meaningful progress towards their goal.
Slot Machine Programming is one example of a general concern that I am increasingly worried about. Historically, there were a lot of skills that we as educators expected implicitly along the way as they progressed throughout a CS degree. This is sometimes referred to as the "Hidden Curriculum". Slot machine programming shows a lack of problem solving skills. Other examples of important but often hidden skills include debugging, working with large codebases, version control, command line use, teamwork, software design, technical interviews, documentation writing, and more. While some of these skills were taught in some places (e.g., MIT's missing semester class, or How to Design Programs), they were not widely taught directly to most students, and yet we often expect students to pick them up along their learning journey. However, now that the process of learning computer science looks so different, we cannot rely on students learning these skills implicitly along the way, as the experiences that lead to this learning are no longer a part of the CS learning process.
As an example, let's look at problem solving. The student mentioned above was not able to perform problem decomposition. Historically, even if they are not consciously engaging in problem decomposition, the very nature of writing the code by hand makes it impossible for a human to write hundreds of lines simultaneously, implicitly forcing them into at least some decomposition. Now a student can easily generate hundreds (or even thousands) of lines of code in one prompt, without ever being forced to make any decomposition by the process.
Why does this matter? While AI has completely changed the game when it comes to writing code, it is not at all clear that Software Engineering (i.e., building computer systems on time, on budget, of the correct quality) is a solved problem. In fact, as the price of writing code approaches zero, what is left is the work of software engineering. And that work is becoming a larger and larger part of the average developer's day. Getting to impressive results with AI tools is only through the application of software engineering principles and skills to the process.
So how should we approach the hidden curriculum? Instead of just giving up, we should instead find ways to uncover the hidden curriculum and explicitly teach students the skills that they will need to be successful going forward. This is a double challenge, because we both are needing to teach skills that previously we didn't explicitly teach, but also simultaneously reevaluate the curriculum, and decide which skills still matter, and which ones are less relevant in the age of AI.
Research shows problem solving can be taught. Briana B. Morrison, Lauren E. Margulieux, and Mark Guzdial found that students who worked with subgoal labels outperformed students who didn't. Dastyni Loksa, Amy Ko, Margaret Burnett and others taught a six-stage process:
- Reinterpret problem prompt.
- Search for analogous problems.
- Search for solutions.
- Evaluate a potential solution.
- Implement a solution.
- Evaluate implemented solution.
When they developed this process, students would have spent a very large percentage of their time in the "Implement a solution" phase. Now that LLMs have reduced the cost of that to almost zero, anyone who is looking to follow this process is going to be spending the majority of their time on the remaining steps outside of "implement a solution."
As educators we should be particularly aware of the fact that there is a hidden curriculum and evaluating which parts we should ensure students are still being taught. I strongly believe that problem solving will be useful for students as long as there are problems to solve. The process of writing this essay has led me to the conclusion that I should explicitly teach problem solving in my software engineering course. I encourage instructors to look for anti-patterns such as slot machine programming, particularly as a way to uncover skills and knowledge that students are no longer acquiring, and find ways to ensure students are still being taught what they need to be successful.