Did these developers not have experience with AI?

This is from the article

But Rush and Becker have shied away from making sweeping claims about what the results of their study mean for the future of AI. For one, the study’s sample was small and non-generalizable, including only a specialized group of people to whom these AI tools were brand new.

I’m not sure focusing on one aspect to scope a reasonable and doable study automatically makes it “really low effort”.

You are right, but I believe they should at least have chosen another use case, to make it interesting. I wouldn’t have needed a study to know that an AI performs worse than a developer in a project the developer most likely built them self. The existing project might have some really weird code smells and work arounds that only the developer on the project knows about and understand. There might be relevant context external to the solution. The AI have to be a mind reader in these cases.

But, if you gave the AI and the developer a blank canvas a clear defined task, I just believe it would be a more interesting study. *

It kind of sounds like they were just handed a tool they knew nothing about and were asked to perform better with it. A mitter saw is way better and faster than a regular saw, if you know how to use it.

*edit

To make my point more clear, I don’t mean the developer needed to solve an issue that’s not related to his daily work, but a task that’s not dependent on years of tech debt or context that is not provided to the AI. And yes, by that, I don’t believe code generation from an AI have a big use case in scenarios where the project have too many dependencies and touches on niche solutions, but you can still use it for other purposes than building features.

I get the agenda of the study and I also agree with it, but the study itself, is really low effort.

Obviously, an experienced developer working on a highly specialized project, where the software developer already have all the needed context, and have no experience with using AI, will beat a clueless AI.

How would the results look like, if the software developer had experience with AI, and were to start on a new project, without any existing context? A lot different, i would imagine. AI is also not only for code generation. After a year of working as a software developer, I could no longer gain much experience from my senior colleagues (says much more about them, than me or AI) and I kinda was forced to look for sparring elsewhere. I feel like I have been speed running my experience and career, by using AI. I have never used code generation that much, but instead I’ve used it to learn about things i don’t know i don’t know about. That have been an accelerator.

Today, I’m using code generation much more; when starting a new project, or when i need to prototype something, complete mundane tasks on existing projects, make some none-critical python scripts, get useful bash scripts, spin up internal UI projects, etc…

Sometimes, i naturally waste time, as it takes time for an AI to produce code, and then it takes time to review the code, but in general I feel my productivity have gained by using AI.

I have nothing to compare to, but I recently bought a Dell OptiPlex 9020 for $15/£13. It works wonders. I run a handful docker containers and a VM and haven’t experienced any issue since I bought it. It’s my first time experimenting with a home lab setup.

Torrentio have been unstable for me the last couple of days and i couldn’t get it to work a few hours ago.