#223: AI Answers - AI Washing, Flatter Org Charts, Advice for Students, Agent Security & the AI Writing Gap

A

Six months ago, these tools were not where they are today. And like putting a standard AI assistant in place that people could talk to and build some documents with and things, the risk surface area for that was relatively low. We're now entering a phase where agents are becoming very real. They have autonomy in select areas. Reliability is still a question mark. Welcome to AI Answers, a special Q and A series from the Artificial Intelligence Show. I'm Paul Raitzer, founder and CEO of SmartRx and marketing AI institute. Every time we host our live virtual events and online classes, we get dozens of great questions from business leaders and practitioners who are navigating this fast moving world of AI. But we never have enough time to get to all of them. So we created the AI Answers series to address more of these questions and share real time insights into the topics and challenges professionals like you are facing. Whether you're just starting your AI journey or already putting it to work in your organization, these are the practical insights. Use cases and strategies you need to grow smarter. Let's explore AI together. Welcome to episode 223 of the Artificial Intelligence Show. I'm your host, Paul Reitzer along with my co host, Kathy McPhillips. Welcome back to the show, Kathy.

B

Thank you. It's been a little while.

A

It has been a little while. So if you're newer to the Artificial Intelligence show, we have our weekly episode that comes out on Tuesdays. That's with me and Mike. And then once or twice a month, Kathy and I team up for what we call our AI answers series. So this is the 20th episode in that series. These are based on actual questions from attendees of our Intro to AI and Scaling AI classes, as well as some of our virtual summits and events that we run. So today's episode is a curated list of questions from our May 13th and June 18th classes for Intro to AI. Right. Does that sound right, Kathy? Because we had to Skip a scaling AI in there. So intro to AI we have done 59 times. We started teaching that class every month live in 2021 and fall of 2021. Pre ChatGPT. So we have literally fielded thousands of questions. And one of the cool things for us is to see how the questions evolve over time and the kinds of things that people are interested in. It gives us a really good pulse into what's going on. So hopefully today's episode not only answers some of the questions you may share, but it gives you insight into what other people are thinking about right now and the kinds of things that they're asking us when they attend our classes.

B

It was interesting when I was putting. Because we use AI to help us curate the questions, get them in a, in a sequence, so it makes. So they flow. And is Paul going to answer this question in question two? So I want to ask it as question three. So there's a whole process that Claire and I go through. And it actually said, are you sure these are from your intro class, not your scaling class?

A

Oh, they're getting more advanced.

B

They are getting more advanced.

A

I believe that I felt that when we were doing those because we take questions live also. And they are definitely just getting more advanced. It's starting, you know, fewer people, I would say, are at the very beginning stage of their journey where they're truly just trying to understand what, what is it? What does it do? They're now getting deeper into the, what are the implications to me, to my business, to my career, things like that. What's the impact on the environment? What's the. So a lot of these people probably listen to the podcast regularly and so they have just much more advanced questions, which is cool. All right, so this episode is brought to us by AI for Business Boot Camp. This is a new event that we are running. So we spent a lot of time on the podcast talking about what's possible with AI. The AI for Business bootcamp is where we will help put those ideas into practice. So you can join me and Mike Kaput, my regular podcast co host and our Chief content officer, on July 16th in Columbus, Ohio for an intensive one day workshop that's designed to help you move beyond experimentation and start applying AI across your organization. So the day starts with a State of AI for Business keynote that I will give. That's then followed by two hands on workshops. Mike leads an AI for Productivity workshop that's focused on helping you transform how you work. I will then lead an AI for Innovation workshop where we'll explore practical frameworks for identifying new growth opportunities and competitive advantages. We've also built in plenty of time to connect with other business leaders who are navigating many of the same challenges. Attendance is limited to only 180 people, so get in early, get those tickets going now. AI mastery members through our AI Academy get special pricing. You can go to SmartRx AI events, and you can actually use promo code POD100 to take $100 off. So again, a one day event, you're going to learn a ton, you're going to build things, you're going to come away ready to apply AI the next day and think bigger picture about innovation. So SmartRx AI forward slash events use pod 100. And again, if you're an AI mastery member, you get an additional savings on that event. All right, Kathy, if I missed anything about today or the AI for Business bootcamp, I'm going to turn it over to you and kind of walk us through how this all works.

B

No, I was going to say that for the boot camp. We took that for a little test drive earlier in the year when we did a team retreat and going through both of those workshops really set the stage for us for the year. So it's, they were both super helpful. So I highly recommend if you have questions, reach out to me. I'd love to tell you more about it.

A

And it's an excellent just to get in the room with other people who are in the same boat and like get inspiration from them. Not just learning from me and Mike, but learning from everybody else. And there's tons of times set aside for sharing and like other ideas. So yeah, it's a great environment. Great.

B

Okay, let's jump in. We have 15 really, really good questions from our Intro to AI classes. So I'm going to start with number one. How do organizations encourage bottom up AI experimentation while ensuring transformation is still driven by CEO level strategy, governance and accountability?

A

I think the again, if you're new to these, I don't prep on these. I actually haven't even seen these questions in advance. So when I do answer things like I'm literally just sort of answering top of mind and like what, what comes to me at. So what I would say here is the bottom up experimentation. One, you have to enable the technology, they have to have access to the things to experiment with. ChatGPT, Claude Gemini, co pilot, whatever that may be. Ideally you've given them some guardrails. So there's AI policies in place that tell them how to experiment responsibly in a even better world. There's an AI Council or some similar body that's helping kind of govern the responsible use of this. The part where we often see this breaking down is that second part, Cathy, the CEO level strategy that is often the thing that's missing. So most organizations that we talk to, most of our business accounts within AI Academy, which there's hundreds of, almost all of them are coming into AI Academy from the ground up where the innovation and experimentation is being driven by a small group of people within a team or a department within an enterprise. They rarely have CEO level strategy governance in place is like what's being developed. But most of the Time like in, in a best case scenario you do have that macro level. Here's what the future of work look like at our company. Here's how we're evolving as an organization. Here's how we're going to empower you to do this in a responsible way that's happening at the C Suite. And then you have the ability at the the ground up to go out and experiment in a responsible way. Knowing what data you're allowed to use, knowing what tools you're allowed to use, knowing what third party software you're allowed to connect these AI systems to and agents to. It's just rarely that far along. Honestly, like most organizations again, and I'm Talking like Fortune 100 companies there, the experimentation is still being driven from the ground up. It is not being guided by the C Suite.

B

And I would even say, you know, I've been working a long time and I've been in leadership roles for a long time. I've never been a CEO. So for me to use the CEO, the co CEO GPT that I built for me of my Paul GPT, I run things through there because I'm like, what am I not thinking of? So even people that have been doing this a long time or in our strategic, if they haven't been in that role, there are some elements that they just don't know to ask.

A

Yeah, like we're. I just actually approved two days ago, I had a meeting with our COO and our Director of Operations to go through our updated AI policy. There are things in that policy that I can guarantee you. Most people in our company and most people in other companies would never even consider as risks associated with what we're doing or why. We can't just connect Claude to everything like as the professional, as the person doing the work, or even the manager, director, VP level. Sometimes you just want to go like, does this drive the innovation? Let's do the thing. But the C Suite has to worry about all kinds of other things and those don't often come as just obvious to everyone else. And that's why it is so important to have that transparency from the top down where you not only put the policies in place but you explain them to where everyone else buys in. It's like, oh, okay, that's interesting. I would have actually never thought about the relation of that risk to what we're doing each day.

B

Maybe we need to build a co CEO GPT so we can put in here. What would a footwear Tracy or COO be asking us?

A

Well, we will have A notebook LM of the policy that will enable people to like be able to go in and have these conversations. But yes, I agree.

B

Okay, number two, many organizations are stuck in policy mode. How can leaders bring it legal and business stakeholders together to move from restricting AI access to enabling responsible enterprise wide experimentation?

A

This, you know, this obviously just kind of builds on the first question and the response I gave. I do think this is where the AI councils start to come more into play, where you're bringing together stakeholders from different areas of the company. You are understanding risks, concerns, opportunities across these different areas and you're aligning on that and trying to move the organization forward. The reality is six months ago these tools were not where they are today. And like putting a standard AI assistant in place that people could talk to and build some documents with and things, the risk surface area for that was relatively low. We're now entering a phase where agents are becoming very real. They have autonomy in select areas. Reliability is still a question mark. How it works when you give them access to different areas, different knowledge bases, people's emails, like everyone's still learning there. And so it's, you know, I think in some ways a little bit more conservative approach when it comes to the more advanced uses of AI. Like the agentic stuff is probably advisable in most companies. You know, you want people to race forward and experiment, you know, in the areas where the risk profile is lower. But when you start connecting it to outside, you know, connecting these outside tools to proprietary data, confidential information, personally identifiable information for your customer base, like you got to move with caution and you have to have the right people in the room who can think about all the different areas that need to be considered.

B

Okay, number three, for organizations facing significant budget constraints such as community colleges, how should leaders evaluate and choose just two or three AI models to best prepare students for the future? And I think you could swap out community colleges for small businesses students for, you know, so I think this could go across the board in many different industries and examples.

A

I mean all the different models and AI labs have different strengths. There's a bit of a leapfrog game going on where, you know, every three to five months, you know, Anthropic was in the lead today and maybe it's Google tomorrow and then maybe it's OpenAI. It's like what I generally guide people is if you just optimize the use of one of them, you're going to be so far ahead of where you were without any of them. And so while you may make a Bet on Gemini today and then like six months from now realize, wow, like OpenAI anthropic are just better now. You're still going to be getting so much value from having just doubled down on that platform and maximizing your use of it. So I think if you're talking about community colleges like education and non profits, I would first look and see do any of these labs have special programs with special pricing for those groups? Many of them do, especially in the education side. And so I would you again, you can't go wrong with any of the three major labs or mix Microsoft in there as well. Like you're, it's, it's going to be really valuable technology no matter what. So find the best partner for the, the area you're in now. If you're a small business and you're just going to spend your 20 to $25 a month per license, then you're just probably looking at, well, who makes the most sense to align. If you're already a Google workspace, workspace customer or Google Enterprise customer, then Gemini might make the most sense. If you've been on chat GPT for the last two years and everybody's got a license, then just go with ChatGPT. Like you can't really go wrong yet. That could change as the government gets more involved with these labs. Like I get a little nervous right now about for example, how much the government seems to hate anthropic. I think that'll evolve and they'll kind of come to agreements about their differences. But that makes me nervous. Like as someone who's thinking about building an organization on select platforms. Six months ago we weren't really factoring in that the government could step in and throw its weight around and kind of pick who the winners are going to be. And right now we're entering a kind of a slippery slope where the government may decide who the winners are. And so that's the kind of stuff you got to think a little bit about. But long story short, you can't go wrong with any of, I would say like the four major platforms between Copilot, Gemini, Claude, ChatGPT. You know, just if that's all you can do is the one, just pick one of them and just, just go hard at it and make sure you, you know, personalized training and learning for everybody on that platform.

B

Okay, number four. With AI washing becoming more common, what questions should organizations ask to effectively evaluate vendors, AI capabilities, security practices and underlying model architecture?

A

We're definitely talking about more technical analysis here. So my first reaction is get the people who are comfortable evaluating a more technical way involved. Like even at SmartRx, we are now reliant relying much more heavily on our outside IT partners and our legal counsel for major decisions around any of this stuff. So we do our own work. You know, the operations team does their research. We've developed frameworks to assess technologies. One of the things we're putting in is literally like the criteria we have to make decisions around. So for example, if someone on the team says, can I connect Claude to HubSpot as an example that there is a five step process that we have to go through to verify are we or are we not going to allow this connection to occur? That process we may devise internally, but that will be then vetted with our outside partners to say yes, this is a thorough process. And here are the two steps where we, we as the outside IT consultants will step in and analyze it. And here's the two steps you will do and here's the step legal will do. So we are definitely at the point where we have to have these more structured frameworks and you have to know whether it's outside partners or inside experts that are going to get involved at the different stages that allow this to, you know, be done in a responsible way.

B

How would you recommend folks know that a legal team or an outside IT team could help that they're the right people to analyze such things?

A

Well, I mean one way would literally just be, you know, if you're confident with Claude or chat GPT, use the most advanced thinking version of IT and say, hey, I'm the director of operations at a, you know, retail company and I'm trying to decide how to do this. Can you help me build a framework and help me know when should I turn to outside counsel? And like I would just ask those questions and it'll give you like a starting point if you, if you literally have no idea the answer to that question when you should and shouldn't, I would just like start with a conversation there. Most organizations are going to have some outside trusted partner that can refer them to somebody. Like for example, you know, maybe you have a trusted relationship with an attorney and you might go to your turn and say, okay, I need to find a trusted outside partner for my it. Who do you guys know? Who should I talk to kind of thing. Because what I've often found is whether it's my banker, my attorney, my IT firm, they're all working together at some point. And so like if I have a trusted advisor, I'll often Say, who do you trust to collaborate with? Because you will be working with these people too.

B

So great. Number five, as organizations weigh the cost of deploying frontier models at scale, when should they prioritize local task specific small language models or edge AI instead?

A

Yeah, I'll kind of zoom out a little bit to explain the context of this question in case, because it's a really good, more advanced question. I would say this getting asked on an entered AI class is a really good example of what we're saying. We're getting these really advanced questions. So at the most basic level, an organization can do a deal with OpenAI or anthropic or Microsoft or Google. They can get those their standard models and then they can pay their monthly fee or pay through the API to access that intelligence and to give it to their people. What this question is saying is, hey, at what point do you maybe use like an open source model that you can have more control of, that you can run locally that doesn't have to go to the cloud and like burn tokens to do things to do, like smaller scale things that maybe don't need the most advanced model or aren't the most expensive, like best of the best kind of thing. I think this is going to be increasingly become a topic of discussion. You know, on the podcast lately we've talked a lot about the cost of intelligence that these tokens or people are burning more and more of them as they're doing more agentic stuff as these. They're using the reasoning capabilities, as they're using the coding capabilities. We're just all using more intelligence to do the things that these models enable. And so the more advanced companies are starting to look at and say, okay, well just assume there's 100 tasks that we're going to use AI for this month. How many of those really need the smartest model? How many of those could we just do with last year's best model and could we just have an open source version of that that our T team sets up and we're just good to go? So I don't think most organizations are at that level of sophistication, but the ones who are asking these questions and the even more advanced ones are saying these models out of China, they're not so bad, they're 10x cheaper than the US models. And so maybe we'll just go get deep, seek to do some of this stuff. Now I mentioned on the podcast last week, I will, I'll have talked about it again by the time you hear this one, because we're recording this on June 26th, by the way. I didn't say that, but this will come out probably six days after we record this. I had theorized on episode 221 that the US government would outlaw Chinese models from US firms. We are four days from when I said that. I will again talk about this again into. I am more convinced than ever that I may have been right, that we may very quickly see the US government step in and outlaw the use of Chinese models by US companies and that then changes the dynamic again. So this question, really smart question. I would imagine maybe less than 5% of companies are really thinking deeply about this right now. I think most companies are still just how do we maximize the use of the standard models that we can get access to from these major labs?

B

Yeah. So maybe if listen to 2:22, I

A

will have expanded on people already.

B

You would change your answer.

A

Yeah. And by the time I record 222 on Monday, I think I will be even more convinced than I am at the moment. There's been some things in the last 12 hours that have happened that have me. Yeah, I would not be surprised if that happens by the end of July.

B

So if you're a new listener, news podcasts drop on Tuesdays. AI answers drop on Thursday. So we're recording this fixed days prior because of Paul and my schedules. And here we are. Okay, number six. What are the biggest security risks of deploying autonomous AI agents in highly sensitive environments? And what safeguards should organizations put in place before adopting them?

A

Again, these are so beyond intro to AI questions. The biggest security risks are the unknowns, honestly, like they're. We're just supposed to trust these labs, that these things work. So one of the things we will talk about on episode 222. So you may have already heard me talk about this again. We're kind of going back in time. Here is Claude. Tags. Tags get access to Slack and supposedly, according to anthropic, they're like walled gardens. So if you give Claude access to your marketing channel in Slack, it has no idea what happens in the operations channel or the HR channel. So let's say the head of hr, the CEO and a few other people are in an HR channel where they talk about sensitive information related to people's comp and all these things. Things supposedly Claude marketing never sees anything that happens in Claude HR and has no knowledge base of it whatsoever. That may be 100% true, but you have zero way to verify that. So you are literally just trusting that it does that you similar scenario would be you give Claude access to your Gmail and your calendar. There's a whole lot of stuff that happens in your Gmail that maybe other people in the company shouldn't read. And so again, let's just stand the HR example. Say I'm having a conversation back and forth with the head of HR about an employee that is private between the two of us. But Claude's seeing it and learning it. And the assumption is the information that Claude gains from those email exchanges never leaks into the company knowledge base and is never discoverable by anyone else in the company. Again, may 100% be true. You are taking a leap of faith that that is accurate, that whatever you do and should be governed by the perimeter, the parameters you've put in place to govern data hold when you give language models access to them. I will say I have intentionally slow played many connectors in our company because I don't necessarily trust that that is true. And so that's when you get into agents and giving them access to all these things to get full value from them, they need to be connected to data, they need to be connected to your other software elements of your stack. That's where the real value gets unlocked. But the confidence level I have that the safeguards we're putting in place will hold are lower by the day. They're not going in the opposite direction. There's a reason the government forced Anthropic to withhold their smartest models. It's because they can't control them. They, they, the guardrails they put on them don't work. And so like, we're just supposed to trust that the guardrails work within sensitive corporate environments. I, I just don't. And so that, that's the challenge we face right now is how to put guardrails in place on something that doesn't work like traditional software. You don't just write rules that it follows. It sometimes does whatever the hell it wants to do because they're language models. It's not traditional software.

B

But even if it was traditional software, there's still breaches in human error.

A

Totally, yes. There's always going to be breaches, 100%. There's just new possibilities that it just does whatever it wants to do because we don't know they're still learning. It's like understanding how the brain works. Even now we have so little understanding of how the human brain works. That's where we're at with language models. We just don't really know why they do what they do sometimes. And so that creates all kinds of uncertainty around scaling. Agentic stuff.

B

Yep. Okay, Number seven. Recent reports suggest advanced AI models like Anthropic's Mythos can exhibit seemingly human like behaviors such as fatigue, user preferences, and sensitivity to tone. If AI performance increasingly depends on how employees interact with AI, how should organizations redesign workflows and train employees to ensure consistent, reliable outcomes at scale?

A

These are so not intro questions. This is absurd. All right, let me take this in two parts. I'm going to tackle the first part of this. So recent reports suggest advanced AI models can exhib human like behaviors such as fatigue, user preferences, sensitivity, tone. Okay, so I'll just give you the two sides of the debate on this first point then gives us context for the second point. This is true. So research does continually show that these models seemingly behave more and more like people. There was, there was this weird study back in like early 2025, I think it was, that showed that Claude got lazy during the summer months in the uk and one of the theories was that people in the UK tend to take more holidays during the summer and so they don't work as much. And Claude learned that from its training data. And so it would actually not work as hard during the summer months. Like that's how weird these things are. It's like we, they, they exhibit a behavior and it's like, what the heck is it doing? Like, why would it behave in this way? So the two camps are that it's just a simulation machine. It learns from data, and within that data it learns that humans get tired. And so it simulates being tired sometimes or that it, it, it responds as though it's like fatigued, but it's only because it, it just learned that in training data. And it's not actually fatigued, it's just simulating something and it's training data. The other camp would be that these things are actually conscious that they actually function more like the human brain than people believe or want to believe, and that they will inherently start to exhibit actual behaviors, not just because it lived in the training data, but because they are a form of intelligence just like the human mind. And so that sets up a lot of weird debates and research paths and things like that. So that's kind of the setup on the behavior thing. It's either it does happen, they do exhibit these behaviors, and they either are actually behaving that way, or they're simulating that behavior because they learned it somewhere in their training data. So the second part, if AI performance increasingly depends on how employees interact with AI. How should organizations redesign workflows and train employees to ensure consistent, reliable outcomes at scale? Yeah, I don't know. I mean, I think at a very high level, I just think the future of all work is human plus machine. And we have to do. Everything we do has to account for that, from how we train our people, from how the people interact with the machines, to how the people learn from the machines. It's a symbiotic thing. Like, we train the machines and everything we do like they're learning from us because we like something, we don't like something. We pick the best output and we give it back to the machine and say, that was a good output. I was. It's fun. I was actually like, in my car today. I took over the full self driving from the Tesla and the thing pops up. Now it says, why did you take over? And you have to pick critical navigation. I figured that. And then the other one's other. And so my son's friend said, why is it asking you that? I was like, well, it's reinforcement learning. And I explained how reinforcement learning works with an AI model. So as the human, I'm training the AI. It didn't do something correct. But in exchange, we can learn from the machine. It's like, hey, that output Claude, you just wrote was really good. Like, what did you do different than I was doing? And it'll say, well, listen, here's how you wrote it. Here's how I wrote. And it's like, okay, like, I just had a learning moment where I learned from the machine. And so I think you have to look holistically at jobs and people's career paths and how you teach the future of work. And you have to make it a symbiotic thing, because a lot of people are resistant to AI and they don't think they have anything to learn from it or that it isn't a learning tool. But it can be if it's taught well. And this is critical within schools, too, that they teach that it's a learning aid, not just a replacement for thinking right.

B

Was one of the options that you just wanted to drive?

A

No, you can. But what you can do is gives you four options. And it drives me nuts because they won't go away until you pick one. So if you just take over and drive the next 20 minutes, it'll stay there on your screen. Drives me nuts. The other option is you can actually press the button and tell Grok by voice why you took over. So if it's not One of the four things, or if you want to give additional context, you hit the voice button and then it gives up to 15 seconds, it records it, and then anonymously, supposedly anonymously sends your feedback. And then that actually goes to Grok. GROK analyzes it. Sometimes a human gets involved and they try and figure out why you take over, why you intervene.

B

Interesting. Okay.

A

Which by the way, is I think, how agents will work in the. In the future. I think business agents will actually do what Tesla does, which is why I find it so fascinating. Why did you take over from the agent sending the email? Well, because it was sending it to the wrong people and it'll learn from that.

B

Number eight, if you work in a highly regulated industry and only have access to basic tools like Microsoft Copilot, what are the highest value ways to demonstrate AI, to demonstrate to AI leadership today?

A

Well, first I would understand the full capabilities of whatever version of Copilot you have. While they can be neutered, for sure, it can have less capabilities than, say, if you brought your own device to work and shadow AI, like we're using Claude on the side, which people do all the time, I would fully understand what is actually possible within copilot. I would then push on those capabilities and like, use it to the best of your ability. If there are capabilities, you know, exist in other platforms or are possible within Copilot that you're just not being given access to, then I would try and say, can I get permission to build a business case to be allowed to do the thing? Like, so, like Agent Studio, for example, like, if you wanted to try and mess around with building some agents in Copilot and that's not currently turned on for you, go to whomever it is that controls that decision and say, I've been doing research, I think we could save 20 hours a week as a team. If I could build an agent to help with the weekly newsletter or to help with the podcast each week, can I get permission to run that experiment? Here's the data I would need access to. Here's how we would work it. Here's how we would set a baseline of previous performance versus what happens. And if I can get permission that I'd love to have that. And then you just do that one at a time until they trust you enough to say, okay, you get full access to Agent Studio.

B

Okay, number nine. For professionals and relationship driven industries, what's the most practical way to build a 24.7ai powered virtual twin that engages prospects and customers without sacrificing authenticity?

A

These Are seriously like the most advanced questions I've ever seen on an intro. There is no way. I mean, so like I talked about this on a recent episode with Saster. So Saster, Jason from Saster, and I forget the lady's name. That's really heading up a lot of this for them. But they had a podcast episode where they talked about building all their agents and one of the ones they talked about was like an sdr, like an agent that was designed to go communicate people. And now their sdr, which is traditionally considered a sales development rep, they were actually using for customer communications as well. So it was really like mixing both. And they talked about how they had like 50 iterations of this thing before it became reliable enough to actually work in a trusted way. And so my general response to this is this is maybe possible in some businesses, but it is going to take a massive amount of effort to build it to reliable point where you trust it to actually do this kind of interaction. And you are likely looking at an ongoing role that is overseeing this thing. Like I don't see us at any point in the next six to 12 months at least being to where you just go and you buy an agent through a third party or you build your own agent and you give it a knowledge base and you say, hey, I want you to engage with customers based on this information. And here's a bunch of examples. Go do your thing. And then you go on and you go do the next thing. And you never monitor that agent. It is likely that someone is monitoring in real time to start every interaction it's having. And then it might be like a daily check in where you're like, okay, what happened yesterday? We do this with our, our chatbot on our website, right, Kathy, like we will. We have trained this thing. It has this massive knowledge base, is trained on hundreds of FAQs. But like someone on our team actually looks at those engagements and says, okay, was there a gap? Did it say something wrong to someone? That we actually need to reach out to that person and say, hey, actually, sorry, but it gave you the wrong information. So this is one of those where we say, well, what are the new roles that are going to exist? I could absolutely see someone's full time job just being management of agents. And it's quality control. It's continued reinforcement learning where it's got improving, it's improving the knowledge base it has access to, it's stepping in when a human is needed. So yeah, I would say it's doable, but this is not a Buy a thing, pay 20 bucks a month, and you're good to go, and you move on and build the next one. And you just have five agents and you never have to hire a customer service rep. Like, that's just not going to be the case anytime soon.

B

And the thing with our chatbot on the website is it answers a lot of questions that people ask us over and over and over again. And it's not pretending to be a human. We're not pretending that someone is talking to Noah or someone else on our team. It's clear that it's sabbat. Then you can.

A

I had to laugh. Go ahead.

B

Then you get too human if you want to, but nine times out of ten, someone's just saying, like, when's Macon? When? How do I log in here? Like, they don't need a human for that. They don't expect a human for that. They just want a quick answer.

A

I retweeted somebody last night and I thought it was hilarious. She said, three years from now, people are going to be asking their humans if they can talk to the AI agent, because they're just going to come to trust the agents more and be like, it's just way easier. Can you just give me the agent? Get away from me.

B

I did think that, like, is there going to be a point in time where it's socially acceptable for someone to be knowing they're talking to an agent via a sales process?

A

Totally. I think it'll be actually assumed, like, I. Very quickly, I believe we'll get to that point where it's like, like, why am I talking to you, Kathy? Like, why don't you have your agent doing this initial phase? I don't need you yet. We're going to start to just have this internal filter where it's like, I don't need the human yet. This is like an obvious thing. Why aren't you guys built this yet?

B

Right. We talked about this earlier this week, actually, about sometimes humans don't like talking to humans. And not that I'm saying we shouldn't and we need to know how to communicate, but a lot of times it's easier for folks to get the information they need if they know that they don't have to talk to somebody. Well, yeah, I'm not encouraging that.

A

Yeah. And there's so many times, though, where you're like, if you're in need of a customer support agent, it's not like that happens neatly in the eight to five hours or like. Or when it's not busy for the Call center. So the reality is like, you just want answers most of the time. You just want a quick resolution. And if it's 11 o' clock at night and I can finally like deal with the item I hadn't returned yet or ask the question about pricing for the SaaS product, I'm thinking about like your personal and your business lives blend and like you just want answers, you want quick resolutions. So I just feel like people are going to prefer to just deal with agents and assume that's what they're dealing with on most reasons that they would reach out to a company.

B

Yep. Okay. Number 10, my own writing consistently outperforms AI generated drafts. What prompting techniques or workflows are most effective for closing that quality gap and producing more authentic human sounding content?

A

Well, one, there's the side of like, if you're the better writer, like and you enjoy writing, why make AI sound human sounding? Just be the human that writes it. And that's like for me, most of the time I default to, I'm not that interested in training an agent to sound like me. I actually just want to write the thing. Like I, you know, if it means less volume, fine. Like it's the part of the thing I enjoy. So let's assume you don't enjoy the writing or that you just like want to do way more and you want to create more stuff and so you need the AI to sound like you and elements of it. The way you do it is give it more examples. Like this goes back to this training thing. So let's say you build a GPT that or a skill and claw, whatever that's going to help you write newsletter copy or blog posts or LinkedIn shares or whatever it is. Give it your 10 favorite examples. Say, I want you to mirror my tone, my style. Here's 10 examples. Read these and then ex and then give me a summary of my voice, like of how you see that I write. Mike shared a great example of this on the AI for B2B marketers summit. He had like a training a voice agent thing where he said like, here's what I want you to do. So a lot of it literally just comes down to that. I want you to mirror my tone and style. I want you to write in my Voice and here's 10 examples and then come back to me with an analysis of my voice and how I generally write and then say, oh, that actually looks great. You nailed it. Write me a sample. It writes a sample. That's a great sample. That's it. Let's lock that in from now forward, I want you to write in that style. And my guess is you're going to get like 95% of the way there with that simple process that you could probably do in 20 minutes.

B

So what I noticed in this one is it's outperforming AI. So that means that it's, you know, converting or doing whatever it needs it to do. It may not be as good of a writer. I mean, like, I can write something I think is really strong, but then I can run it through something saying, are we going to get results from this? And that's really what I want. So there's a difference between good writing and good writing that converts.

A

But yeah, I mean, like, you're a great example, Kathy, because you still write a lot of our, especially like our event emails and stuff. And I know that there's time where our marketing team will use it to assist with like subject lines or maybe even writing a draft. And then there's other times I know you will spend like four hours personally writing like a 400 word email because like it's deep and it's personal and you want to like connect with the audience and like AI is not going to write that one. Like it doesn't know you went on the walk last week and that that walk inspired something. Like that's the human element. And that's what I'm saying. Like sometimes the human should write the thing and other times it's just information based or it's just like we just got to get this thing out and like, let's just do this. And so that's why I always go back to like, just because AI can write doesn't mean it should in all instances. If it's an if, authenticity is what matters. I actually am a huge proponent that AI should have a very limited role in it.

B

Okay, number 11. As AI shifts from assisting knowledge workers to completing entire workflows, what skills will become significantly more valuable over the next five years or 12 months?

A

Yeah, I don't know. We say like the words like taste and judgment matter a lot. Meaning, you know, the taste of like, what to have it work on, what to have it. Right? Like it's that, hey, I can do these different things. What workflows should I have it applied to? And then the judgment is like, is it any good? Is this something I would use? Do I need to make edits to this? So I think more and more you need the domain expertise and experience to know what to ask the AI to do and then to know what to do with the output outputs. Like, a lot of getting the most value out of AI is question asking. It's like, what do you know? What, what do I want it to think about? What do I want it to do? Like, I was working on something this morning and it's been a grind. Like, I've, I've been trying to get this brief together that was. I was really struggling with the alignment between two ideas. And they're complimentary, but they, every time I would read what I wrote, I was like, this isn't working. Like, it's still confusing to me. And so I finally got to the point where Claude just wasn't helping me get there. Like, it kept outputting things and I felt like I was in this loop, like this death loop of like, this isn't the answer. And so I actually, while I was on my drive, I opened up voice mode and chat GPT and I was like, all right, blank slate. I'm struggling with these two ideas. I'm trying to make them align and be complimentary, but every time I do it, it just keeps coming back, like, confusing to me. And so, like, what do you think? And I had this amazing 15 minute conversation with ChatGPT and actually landed on the thing I was trying to say that Claude just couldn't get me to. And so, like, that's an example of just knowing to go ask something else and to frame the question differently. And so my ability to have the domain expertise to know I need to make this cleaner. So as a writer, as a communicator by trade, I knew my message wasn't there yet, that it wasn't going to land, but I was like stuck. It was like writer's block of how to say it better. And then I knew Claude wasn't doing it, like, it just wasn't getting me to the end game. And so then like, I knew to go ask the question of ChatGPT, like, and so that to me is the future of work. It's like you have access to these tools, you have access to agentic tools and knowledge tools and you gotta know how to use them and how to talk to them, how to get something out of them and then to know if what you got was any good.

B

Absolutely. Okay, number 12, is there a practical framework leaders can use to determine which work should be automated, augmented or remain fully human?

A

Well, the fully human thing. My initial instinct on this one is to go back to the authenticity thing. Like, more and more. If it, if it needs to be authentic human voice, you know, it's got to probably be human, if it's high risk, if, like the output requires, like it's a really important thing. Like the example I just gave. Like, it is what, what I just explained and maybe I'll come back and give the context, like a few months from now. It is literally foundational to the future of our company. Like, it is that important that I nail this thing. And so AI is assisting me in many ways, but it's a thought partner. The end decision has to be me and I have to be fully committed to it to then take it to the team and say, this is the direction I want to go and then get feedback from other humans. Now Kathy may say, hey, Paul's got this idea, like she may bounce it around in her co, CEO or wherever she goes to have a thought. But everybody, at the end of the day, the humans are going to make the decision. Now when we look at all the other ways across our workflows and the tasks that people do every day, we look at things that are repetitive, that are data driven, you know, that are kind of following the same process over over again, where we're generating images or video or text. Those are all really good candidates for, for advanced forms of automation. I would say that we don't really need the human in the loop much beyond setting the objective, monitoring the outcome, and then making the final decisions and maybe approving the thing. So there's just these different degrees. This human to machine scale is the thing I created where we actually kind of go through and assign, is this all human or is it all machine? And I think increasingly AI is going to be able to get us to an advanced level where it can be mostly machine, mostly AI with human oversight. But some things like writing, you may choose to still do predominantly human, even though the AI can get you there. So some of this is technical capabilities and some of it is going to be personal and brand preference as to how and when to use AI.

B

Right? And I think, you know, just thinking about, say, our event strategy for, from a marketing standpoint for the year, if I use AI to help me with something and I can't explain it to you without looking at my notes, without going through what the AI told me, I shouldn't be using AI for that. And especially for something as critical as that, I need to be like locked in, you know. So I think it's important to make sure that AI can give you really good stuff. But if you don't understand it or you can't talk to someone about it, then you're doing it wrong. Yeah.

A

It's why I don't use AI the vast majority of time on the podcast. Like everything I do on our podcast, I am usually reading the entire thing, copy and pasting key excerpts, making my notes from them, because my retention of the information and my confidence level to discuss it is so dramatically different than if I just gave it to ChatGPT said, Summarize this article, give me five bullet points that I can regurgitate on the podcast. I would not retain that information. And so that, that to me is like, yeah, of course AI can summarize 50 page reports and turn it into five bullet points in five seconds. But doesn't mean I understand the report deeply to be able to talk about it and answer questions about it. So sometimes you just got to do the hard work. Like, there's no replacement for that.

B

Right. If you think about our director of research, Taylor, and the report that she built in a half a day, you know, she talked about it on our B2B summit. She knows that thing backwards and forwards. You could take that report and not have it in front of her and ask for a question and she's going to know it. So there is this whole learn, learning element that she took into account, even though AI helped her for a large portion of it.

A

Yep. Yeah. The AI confidence gap is what we talked about it as. Like, you just, if you don't do the work, you're not going to have confidence in the material, you're not going to answer questions about it, give a presentation about it, retain the information beyond, like, cramming for a final. Like, that's kind of how I equate it. It's like we all in college, like, you know that one long night where you drank a bunch of coffee and stayed up and crammed for the final, and then two days later you're like, I didn't retain any of that stuff like that. It's just like you're literally just cramming the information in there. And that's kind of how I feel about using AI to summarize articles and research reports. Like, it's just, just it doesn't stick in my brain if I don't do the hard work.

B

Right. Okay, number 13. Companies like Coinbase are flattening organizations in response to AI. Why are so many leaders eliminating management layers instead of using AI to make experienced managers more effective?

A

This is one of the unknowns, is like, which levels of organizational structures are going to be the most impacted the fastest. And entry level is certainly a Candidate, you know, it's like a lot of the tactical work entry level people did is certainly able to be done by the AI. And so, you know, you might see some disruption and displacement and, you know, lower employment rates for, you know, the younger generation, the 25 and under the management layer. My, my instinct here is if I think about the people who are going to get the most value out of an AI assistant or the ability to build and manage agents, it is the people with the most institutional knowledge, the most domain expertise, who are experts in their field, who now have almost unlimited intelligence and assistance that they can spin up to help them do things. So they did all the work of the entry level people, so they know the tactical work and now they have access to a tool that they can just put to work to do all that. So Kathy can come up with an idea to launch a product on a Friday night. Well, let's do a Tuesday night. I want you working on Friday nights. She random idea on a Tuesday night. She can spin up the idea. She can develop a three page brief and then she could say, build me a plan to launch it, write me the email copy, write me the landing page, do the thing, and then she can wake up Wednesday morning and she can edit everything she just created. Two years ago, she would have needed three people on the team, the email marketer, the SEO person, like, and she would have come in the next morning and said, I got this idea, I'm going to spend the next five days writing the brief about it, then I'm going to turn it over to you all and you're going to go do all these things. That doesn't happen anymore. And so the entry level is difficult, but the manager often doesn't have that advanced experience expertise. You develop from failing sometimes and having enough campaigns you ran that didn't work and spending years looking at the data and like all the things that the seasoned leaders have, they don't have that yet. They don't have the full taste and judgment to know what's good and what's not good. And like, so that is my theory. I haven't dug into it deeply, but I even think about our own organization. And like, I mean, if you look around Kathy, like we have a lot of people with 12, 15 plus years experience who are crushing it, who are just like, like 10Xing what they were capable of three years ago because now they have all that expertise. They always have, but now they have unlimited access to intelligence and assistance. And so those are the people where you're like man, like let's just get more of those people because they're just going to crush it. And we don't probably need as many middle managers one, because there's probably going to be fewer entry level people to manage. I don't know, like this is, this is an unknown. So I'm just theorizing at the moment because no one really has the answer to this yet. This is a, an area where you're going to see a lot of research done as we start to see the impact where people stop playing the game of are they just AI washing these like, you know, layoffs and stuff and they realize, oh wait, AI is actually replacing the need to have as many humans working at places. Then we'll start to see, well, what layers are actually getting reduced fastest.

B

Which brings us to question number 14. As AI begins displacing both entry level and knowledge workers, who is actually responsible for addressing the broader economic consequences? And are any serious various long term solutions emerging?

A

Well, I mean it's traditionally in a democracy the government would be, I mean, I guess really any form of government. The government would be responsible for this. I mean at a micro level, each business has a responsibility, but businesses have fiduciary responsibilities to their shareholders. So you know, take any enterprise and yeah, I mean they want to be good corporate citizens. They want to be doing positive things in the economy and in society and in their community. But they also have an earnings call next month and their main responsibility is shareholder value. Unless they're a public, you know, a public good organization or corporation, then public benefit Corporation, I guess the total pbc. Their responsibility is shareholder value. So the only way you stop that is through unfortunately government regulation and government, you know, and laws. So the broader economic consequences really fall to the government. And no, there are no serious long term solutions emerging. There's ideas we're going to have. Andrew Yang is going to present at Macon this year, Macon 2026. He ran for president in 2020 and he was running on the idea of universal basic income, that we needed some economic stimulus provided to people who, who don't necessarily have the best job prospects. Now that's one idea. There's ideas like universal basic services. There's, you know, Elon Musk is worth a trillion dollars. Let's just have him give everybody some money. I, I mean like there's the idea that these AI labs need to be better corporate citizens and when they go in and build data centers in a community, they should, you know, subsidize education and energy bills and stuff. Like that. But no, there is. There is no plan. And this current administration is not on a path to have a plan. I would say so right now it comes down to each of us trying to move the needle forward in the best way we can in our communities and our companies.

B

Yep. Okay. Number 15. Given the rapid disappearance of traditional entry level jobs, what would you tell a college student today? Gaining experience, building skills, and launching a successful career in an AI first economy.

A

Be the best at AI in your discipline. I don't care if you're an arts major. Coming out as an economist, a doctor, a lawyer, an HR professional, a marketer, sales professional, whatever it is, be the best at AI at that thing. So I don't like, my kids are. My daughter will be a freshman in high school. My son will be in eighth grade. I know, Kathy, yours are in the professional world now. I wouldn't tell them to change their major or their areas of interest, even given everything I know about AI. So whatever my kids want to go into, I will support them. I will highly encourage them to do it at a school that has a very AI Forward vision for the future of work and is providing the curriculum and experiences needed to prepare them for that future. And if they end up somewhere that isn't, then I will strongly encourage self learning to go figure that stuff out. So you got to just get into the professional world. You have to do everything you can to gain experience. There's no excuse like 20 bucks a month to get a tool to do this. I mean, I know, I remember, you know, being in college and hoping there was five bucks in my bank account that night to go to the bar. Like, I know that money is hard to come by, especially at that age, but find the 20 bucks. Like, go do something and earn the $20 a month and pay for that license and test it every day. Like, work with it, get comfortable with it. So when you go to a job interview, you can demonstrate your competency even if you're not learning it in the classroom. Use AI to learn about the people that are going to interview you. Use AI to build an app to demonstrate like, your capabilities for the job you're interviewing for. Build an agent. You can show them, like, just do something. You cannot show up to a job interview and not like AI or at least tell them you don't like AI or that you don't know what you're doing with it. That is the fastest way to remain unemployed.

B

But underlying the point of you don't need to major in AI.

A

No, you don't and on the positive side, because we always like to end on the positive note in the Q and A. If you are the AI forward person, if you are the one in your class or in your discipline that has advanced knowledge and capabilities with these tools, your job prospects are probably going to be incredible.

B

Right? All right, let's end on the word incredible.

A

There you go.

B

Okay. Thanks, Paul. And if you haven't attended an Intro to AI class, our next one, our 60th, is July 29th. You can visit IntroClass AI to learn more. Just to reiterate, as A reminder, our AI for Business Boot Camp takes place July 16th in Columbus, Ohio. We're making the two and a half hour drive down to Columbus to see everybody. You can visit smartrx.com events and don't forget Pod 100 saves $100. So thank you Paul for a nice hour together.

A

Thank you Kathy. Thanks everyone for joining us. Thank you.

B

Thanks.

A

Thanks for listening to AI answers. To keep learning, visit SmarterX AI where you'll find on demand courses, upcoming classes and practical resources to guide your AI journey. And if you've got a question question for a future episode, we'd love to hear it. That's it for now. Continue exploring and keep asking great questions about AI.