Anecdotal observation - experience matters with effective prompting

Pianomahnn

Pianomahnn

[H]ard|Gawd
2FA
Joined
May 22, 2001
Messages
2,025
Background: Been working in the industry since ~2004. Solid two decades in all sorts of things orbiting data.

The past few months I've been leveraging VSCode/Claude Code to handle myriad data modeling, architecture, engineering, bi developer, documentation author, tasks. Some large projects, others one off. I think it's gone very well, quite successful with outcomes.

What's running through my head, as organizations roll these things out, is who is best suited to take real advantage of LLMs. My anecdotal observation is that experiences, seasoned people are more effective.

It makes sense - but I don't have a sense my own organization, and others, are looking at it like this. A company rolls out an LLM, it can go wide. Massive platform in the hands of people who don't yet have the capacity to understand the broader implications of their work. They don't have the experience of managing teams, structuring large chunks of work, prioritizing steps in order to achieve good results. All the things we pick up along the way that influence our ability to identify problems, work through and implement solutions.

edit: The other facet of this is ensuring a good pipeline of juniors, succession planning and the like. And I'm in this state of uncertainty on what is truly necessary for someone to learn and understand in order to do their work. And my comparison would be when I started off, the tools I had, being abstracted away from what people in the 70s or 80s would need to do. To write software, work with a database, it was much lower-level. We abstracted that away over time, and people didn't become incapable of understanding data, how it relates or drawing conclusions from it. The how changed.

So is it okay if someone is an effective prompter but doesn't understand the fundamentals of a star schema? I don't know.

There's no point here other than opening up for thoughts. I want to solidify something on this though, as it impacts some decisions I'm considering across the business. So your discussion here would be helpful.

Happy Friday!
 
I've put LLMs in the hands of a couple people at my work. What caught me off guard were the novice users who would blindly follow it down really dumb paths, wasting untold time. It's an amazing tool but you need sufficient knowledge about you're working on to really reap the benefits.
 
For me, I would compare it to the "Search engine"

Many of us know people who claim "I Google'd that but couldn't find anything" , and then we go and Google it and first few results give us exactly what we were looking for.

LLM's are the same, sadly they have been sold as this magic solution to ALL your problems and too many people believe it.

For those of us who know how to use tools, analyze our usage, correct behavior to get better results, it is a god send!

One of my 2 recent uses that saved me a lot of time, was in 2 situations:
- Using Claude to help me architect a secured deployment for MCP hosting with our our existing Azure tenant to allow more integrations with some of our products while retaining control of usage and access, down to the very requirements for network isolation to keep it internal, products we can use and will use, and everything in between. I am of course vetting everything it said, but so far it has been 100% dead on.
- User account compromise - Fable and Opus , the amount of audit logs I have to pull and review, Claude made quick work of it, parsing, grouping, creating accurate timelines of events, confirming my audits I pulled from specific queries, including some items I missed in one audit KeyQL query I was using.

What often can take me several days (when other things come up of course), took me 2.5 days exactly and that was me being overly meticulous myself not wanting to miss anything!

philb2 said:
retarded ???
Click to expand...
"Please fix this excel file for me" - while not including what is wrong or what they want fixed, then getting upset when it doesn't know what to do, or just goes off and does something different.

This is due to the marketing and over hyping of the product, aswell as some people not wanting to learn how to prompt effectively.

We have some Teams channels for CoPilot and Claude, and I am always posting guides on usage, better prompting and how to try and get the most of the tools, but the questions some people still ask...

1791047480184.png
 
MrGuvernment said:
For me, I would compare it to the "Search engine"

Many of us know people who claim "I Google'd that but couldn't find anything" , and then we go and Google it and first few results give us exactly what we were looking for.

LLM's are the same, sadly they have been sold as this magic solution to ALL your problems and too many people believe it.

For those of us who know how to use tools, analyze our usage, correct behavior to get better results, it is a god send!

One of my 2 recent uses that saved me a lot of time, was in 2 situations:
- Using Claude to help me architect a secured deployment for MCP hosting with our our existing Azure tenant to allow more integrations with some of our products while retaining control of usage and access, down to the very requirements for network isolation to keep it internal, products we can use and will use, and everything in between. I am of course vetting everything it said, but so far it has been 100% dead on.
- User account compromise - Fable and Opus , the amount of audit logs I have to pull and review, Claude made quick work of it, parsing, grouping, creating accurate timelines of events, confirming my audits I pulled from specific queries, including some items I missed in one audit KeyQL query I was using.

What often can take me several days (when other things come up of course), took me 2.5 days exactly and that was me being overly meticulous myself not wanting to miss anything!


"Please fix this excel file for me" - while not including what is wrong or what they want fixed, then getting upset when it doesn't know what to do, or just goes off and does something different.

This is due to the marketing and over hyping of the product, aswell as some people not wanting to learn how to prompt effectively.

We have some Teams channels for CoPilot and Claude, and I am always posting guides on usage, better prompting and how to try and get the most of the tools, but the questions some people still ask...

View attachment 829342
Click to expand...
I would say it's more like a higher-level language. Because I've seen the same type of cognitive decline as well throughout my career and technology has lowered the bar of entry.

For example, Python vs C++. You can get a lot more done in Python than C++, and it has a ton of convenient features that that don't exist in C++. And it simplifies programming so that more people can do it and you don't have to worry about the myriads of intricacies of C++ which can lead to bugs. At the end of the day though, there's stuff C++ can do that Python just cannot. But for most people, the tradeoff of significantly longer development time is not worth it.

It's like the indie game scene. Today, many indie games come out which require hundreds of megabytes of storage and RAM, emulating an NES game which ran in 30KB. And the game is 1000x slower. That just doesn't matter for most people when your hard drive is over a TB in size and the initial time is 50 µs. It's like, I remember when MS Word took 6 minutes to even open compared to Word today which opens in a couple seconds and has vastly more features. Hardware improvements have allowed this tradeoff. For most projects, you probably don't need the technical knowledge.

The downside is, if you do need the technical knowledge. We're trading the removal of technical debt for an ever-increasing cognitive debt. As the other thread about NYT article about the studies of cognitive decline pointed out, it's not necessarily that the LLMs make us worse, rather, it's when you don't learn the fundamentals to begin with. AI there is like giving kids a calculator for math, but they've never learned how to add, subtract, multiply and divide in the first place. The problem here is that's becoming more pervasive in society. And companies are there to make money, expecting people to already know the fundamentals. Because at the end of the day, as good as LLMs are, reading code doesn't give you the same level of understanding as writing code. And now we're removing the latter while expecting people to do 100x as much reading.
 
A while back I spoke to someone studying cg animation at a university. The training started by doing traditional cel animations to get the basics of motion and keyframing etc.

This is the only way going forward if we don't want things to collapse. Coding should be taught and in some cases even done strictly by hand depending on the application. AI should only be used for vulnerability scanning, refactoring, and other tedious tasks that don't make sense for a human operator to do. Or when the human operator has enough hands-on experience to know if the output is usable or not.
 
myzz said:
A while back I spoke to someone studying cg animation at a university. The training started by doing traditional cel animations to get the basics of motion and keyframing etc.

This is the only way going forward if we don't want things to collapse. Coding should be taught and in some cases even done strictly by hand depending on the application. AI should only be used for vulnerability scanning, refactoring, and other tedious tasks that don't make sense for a human operator to do. Or when the human operator has enough hands-on experience to know if the output is usable or not.
Click to expand...
I think a decent analogy is mathematics. You will always have the nerds at the very top of the field pushing it forward who knows the fundamentals and beyond of how it all works. But for the vast majority of people who need to do some math, they are better off letting the calculator or spreadsheet handle it for them. They don't need to know how why a formula works the way it does, just that they can depend on the output being correct.

I know AI coding isn't quite there yet, but based on the rapid speed of improvement even from just a year ago, I'm betting we'll get there.
 
Good analogy actually - because when you do math with a calculator you have known inputs and expected outputs and it is deterministic. AI coding is none of those things. Not yet at least.

The big issue is that (some, maybe most) software is made for others to use. Some who even pay. They expect it to work. It's like vibe coding the calculator or spreadsheet itself and selling it.

But yes, of course not all software needs to be handcrafted. If you're making tools to use for yourself at home, knock yourself out. Unless those tools help prepare your taxes maybe? Can you tell the IRS that your AI messed up?
 
M76 said:
There are no surprises there, even doing an effective google search is a skill, so of course AI prompting requires some too.
Click to expand...
Not only the prompting but being able to tell if the output is complete garbage or not.
 
myzz said:
Good analogy actually - because when you do math with a calculator you have known inputs and expected outputs and it is deterministic. AI coding is none of those things. Not yet at least.

The big issue is that (some, maybe most) software is made for others to use. Some who even pay. They expect it to work. It's like vibe coding the calculator or spreadsheet itself and selling it.

But yes, of course not all software needs to be handcrafted. If you're making tools to use for yourself at home, knock yourself out. Unless those tools help prepare your taxes maybe? Can you tell the IRS that your AI messed up?
Click to expand...
I do wonder, and certainly hope (as a consumer of software where everything is SAAS and ridiculous in price) that the expected output of AI code becomes deterministic and trustworthy to the point that everyone including businesses are using it to develop custom solutions. At the very least I hope it becomes a downward pressure on the price that companies can charge for their software. It's gotten a bit out of hand imo

As to the topic at hand, it will always be a tool. You still need to know how to effectively use a spreadsheet or a calculator, same with AI, and perhaps we'll see a new career evolve as more and more companies what their own internal software engineers.
 
You must log in or register to reply here.
Back
Top