The perils of just turning it on.

The perils of just turning it on

Amanda picked this week’s topic after hearing about a firm that signed for an AI product, provisioned every license, and called it done. A COO’s story made the stakes real: enterprise licenses bought for the whole firm at year end, and six months later only five people had ever turned theirs on, because there was no plan for governance, policy, or training behind the purchase.

The reasons game

Rob, Amanda, and Debbie traded reasons in turn: skipped training breeds either abandonment or over-reliance, bad habits form that are hard to unlearn, every role in a firm has a different use case nobody hands them, unmonitored spend turns into a token bonfire, support staff cannot back a tool they were never trained on, and files end up scattered everywhere but the document management system. Client risk closed the list: work product that reads like ChatGPT wrote it tells a client their lawyer was not really lawyering.

Activation is not adoption

Only 16% of midsize firm leaders have a hard metric for AI ROI, and more than half are spending out of competitive fear rather than a modeled return. Debbie’s read: usage does not equal proficiency, and the AI game inside a firm gets won person by person, not by a dashboard.

Does pasting into ChatGPT break privilege?

A live audience question landed the panel on real legal ground: active court cases already treat sharing a client’s documents into a public AI tool as giving up attorney-client privilege. The harder problem is that most firms, including their own, have not told clients that yet, and the rules of professional conduct are being rewritten state by state as they speak.

The loudest voice is not the rollout plan

44% of firms have no formal way to tell a partner no, and litigation departments show up in 68% of the ones pushing hardest for a tool. Debbie’s story from two weeks earlier: an equity partner dropping a personal credit card on an associate’s desk and telling him to just sign up. It works, but it is not governance.

Transcript

Back for week two, from two different offices

Rob: All right, Debbie and Amanda, we are back for our second week of Sidebar Live. You’re back in your office, Amanda? Debbie is somewhere in some state, maybe even some country. I think you’re on the West Coast today.

Amanda: We’re back. I’m super excited, I’m back in office.

Debbie: Yeah, I’m in California. I’m in Irvine, California. But this crowd is going to have to get used to seeing me from random spots around the country.

Rob: I love it. Last week, we had your hotel room.

What last week’s Centerbase Affinity webinar revealed

Rob: But this has been a big week. Debbie, you and I just did a webinar yesterday for Centerbase Affinity and NetDocuments. I know we have a topic we want to talk about, but anything that surprised you in that webinar, or anything you think we want to share with the community?

Debbie: I was so impressed with how many people. We had 255 people registered to come to that event. And we did some live polling, and the answers did surprise me. We had 65% of the people in that audience who either hadn’t even begun to think about policies and strategies, or were actively thinking about it but didn’t have anything yet. 65% of that audience. So for everyone who feels like they’re behind, welcome to the club.

Rob: Welcome to reality. Because I actually shared that with Amanda before you jumped on, Debbie, and I said I was shocked. Amanda, what’d you say?

Amanda: I said, I’ve been telling you this over and over again, right? Like we talked about this last, there’s nowhere we’re getting help from. It’s all, you know, it’s funny, I had dinner on Monday night and an attorney said, we’re so far behind, like all these other firms are doing this. I’m like, no, no, we’re not far behind, I promise you. So anybody listening who thinks you’re so, so, so far behind at your firm, you’re not.

Rob: So today, the topic that we want to really dig into, Amanda, actually, you recommended it. Instead of me kicking it off, why did you recommend it? What’s the topic?

This week’s topic: the perils of just turning it on

Amanda: So the topic that I chose today, or recommended, is called the perils of just turning it on. And this came from another conversation I had, where a contract was signed to purchase an AI product, all the licenses were provisioned, and it’s, let’s just turn it on for everyone. And the activating of those seats is not adoption of the product. I kind of was lost for words to explain why that’s not a great thing. And then I’m like, okay, how fast can I get to Claude to give the answer to this? I’m sitting at dinner like this. But then I thought to myself, you know what, I’m going to have two great experts on LinkedIn Live with me this week, Debbie Foster and Rob Joyner. Debbie could tell us the answer to what are the perils of just turning it on, and why can’t you just put it on everybody’s computer. Let’s look at Copilot, gen AI, Copilot, Claude, OpenAI. Why can’t you just turn it on and let everybody play with it?

The reasons game: why flipping the switch is not enough

Rob: Well, do you want me to read a quote real quick? I love this quote, it makes a lot of sense, and I feel like a lot of people will resonate with it. The quote is from a COO. They wrote: at the end of last year, in an effort to spend cash before year end, and out of fear of not making a decision while being paralyzed by indecision, we purchased Copilot enterprise licenses for everybody at the firm. Six months later, we have activated five of them, because we didn’t have a plan for governance, policy, training, or anything.

Debbie: So I think we should make this a little bit of a game. I’ll give a reason, then Rob will give a reason, then Amanda will give a reason, and we’ll keep going until we run out of time. Okay, my first reason: without the right strategy and training, people are either going to think it doesn’t work and give up on AI altogether, or they’re going to become over-reliant and not understand what guardrails they should be using when they’re using the tool.

Rob: Training is continuous. If you don’t even start training, you’ve got a problem. What we really recommend is continuously training on a product. When you first roll it out, you’re only going to retain so much. There’s a stat out there that you usually only use about 20% of a piece of software. You have to keep training to see the full ROI and really get the value out of it.

Amanda: My first reason is the analogy I gave, that it’s like learning to drive a car. Anybody who’s driven a car before can drive a car, so if you’ve used technology, you can use technology. But if you’re getting into a brand new car and you’ve never been taught how cruise control works, you don’t know how to hook up your Bluetooth, you don’t know how to use manual versus, say, your car is a stick shift, just because you know a tech doesn’t mean you know that specific tech. And figuring it out is a learning curve you don’t want many of your employees doing on their own.

Debbie: That leads into my next one: bad habits, developing bad habits that are so hard to unlearn.

Rob: You often hear too that law firms are a bunch of small businesses, everybody kind of works on their own. When you’re rolling out these tools, there’s a lot of value in the memory and the learning across the whole firm. You see a lot of the larger firms rolling out their own systems to really try to capture that and personalize it to their IP. If you’re working in silos, you’re typically not working together well, and you’re not capturing that knowledge. So you don’t get the efficiency gain you’re looking for.

Amanda: Okay, my next one is personalization. In a law firm there are a million different roles: legal assistants, paralegals, accounting. With any AI product, and if we’re speaking just about gen AI, every single one of those people has a different use case. But not all of them will immediately see what their use case could be. I could walk up to a paralegal and probably name one, but I don’t do their job every day, so I don’t necessarily know what a use case could be. Sometimes it takes thinking that big picture. If you’re just turning it on and expecting someone to come up with their own use case, but they don’t even know where to start, you’re going to have a lot of people who aren’t using it for the best. It just turns into Google, and it’s not building things that work for them. It’s that specific use case that you need someone to help guide.

Rob: I’m out. Debbie probably has another one.

Debbie: No, I have another one. Token costs. Not being able to understand, like, how do we not let people have a token bonfire every day, just burning through tokens without really understanding the tool, the practical application, what it means for the work you’re doing, the firm, and using the right model for the right job.

Amanda: Yeah, I got a lot. Come on now.

Rob: Do you have any more? Oh my gosh, keep going.

Amanda: Bandwidth for support and ongoing support. We talked a little bit about how training is important, but support is equally important. Your support personnel, whether that’s your help desk or your MSP, kind of have to be trained first, and they have to know the product inside and out. All these products are brand new, so none of us know them any better than anybody else, but it’s hard to support something you don’t know and understand. So just turning it on means the people who have to support it are learning at the same time, and you might not get the best support you can have. So mine is to make sure you have support, but also team leadership, internal champions, internal experts.

Rob: Internal experts, people in the office that somebody who’s frustrated can walk down the hallway and go talk to, and not wait on support.

Debbie: Okay, my next one is governance. People are creating messes. Their downloads folders are full of artifacts the tool has created. They start working in a document, they save it, they never put it where it’s supposed to go. There’s just too much room to save more stuff in more places, it’s the wild, wild west. And we already have an issue in law firms with people saving stuff wherever they want instead of the document management system.

Rob: I think the best part, Debbie, is as you’re telling this story, you’ve got background music going on, it sounds like a piano playing, and it puts a nice little effect on what you’re saying. Amanda, sorry, go ahead.

Amanda: Oh my gosh. I think about it from a leadership standpoint. If I’ve got a bunch of people playing with gen AI, I don’t know what they don’t know until we find a way to measure it. Somebody comes in and says, I’m absolutely an expert in, say, marketing, writing press releases, and the next thing I know everything that comes out is press releases, or all my social media starts looking like AI slop, because they’re using it and haven’t been trained to use it well. So from a leadership standpoint I need to know what they know and don’t know, and I want to help them figure out how to use it best. That’s probably my last one, Rob. Debbie, I’m sure you have a thousand more.

Rob: Well, Amanda, one of the early things we did at Centerbase that I love, and you can do this in different areas, is we rolled out different skills, the first being a branding skill. If you’re producing things, let’s make it consistent. There’s low-hanging fruit like that, that starts governing what’s coming out.

Amanda: Well, I have a skill I’ve been thinking about. One is just email to the JA. Any legal assistant who needs to communicate with the court’s office, with the JA, there’s certain language you use and you don’t use. If I could write a skill called letter to JA, they could just type in plain language and it would turn it into something I’d want to send to a court. So I’ve been thinking about that. But the perils of turning it on are things leadership has thought through that I don’t have time to write or implement. And people, like Debbie started with, have stopped playing with it because it didn’t work, and they won’t come back to some of the things we’re building.

Debbie: My next one is client risk. We fall into the trap of thinking AI can help us with communication, and then we start sending emails to our clients that sound like we’re not humans, or we start sending work product to clients that makes it sound like they could have done this themselves with ChatGPT. It might not be that they could have, but our clients are getting smarter, and they recognize the signals of AI. As soon as they see that in your work product, they assume you aren’t lawyering, that you’re just ChatGPTing. I think that’s a real risk for firms.

Rob: That’s a good one. Amanda, are you out?

Amanda: No, it just goes all the way back to, we still have to figure out what we’re telling clients about our AI use. We have to figure out what we’re going to tell clients about how they’re blowing client privilege by taking our documents and uploading them into the free version of ChatGPT to get a better answer than we gave them. Client privilege is just out the door. It’s the biggest threat to the attorney-client privilege relationship we’ve ever seen.

Five numbers from inside Sidebar

Rob: So I have something fun I want to do. I have five talking points, each backed by themes we’re hearing within Sidebar. If you haven’t joined Sidebar, please do, go to gosidebar.ai.

Amanda: Can you drop a cute link in, like you did last time?

Rob: I can. I did it last time and then forgot how. I’m looking, but I’ll figure it out after I read this. Okay, so there’s five talking points, and I just want comments on them from the experts. You like that, Amanda? The experts, the two of you. Okay, five talking points. Number one: activation and adoption are different metrics, and firms are only tracking the wrong one. Only 16% of midsize law firm leaders have a hard metric for AI ROI. More than half are spending out of competitive fear, not a modeled return.

Amanda: FOMO’s a real thing. Like I said, I had that conversation with an attorney, we went to a bar event and all anyone’s talking about is how great their firm is using AI, and we’re the last ones who’ve done it.

Rob: There it is.

Amanda: We’re not behind. But hard metrics, very hard to find as a law firm manager. So I was not surprised by this 16% number. Debbie, were you surprised?

Debbie: Not even remotely. I think that even the metrics we have tell a tiny bit of the story. If I see that, say, thirty-six people are in and this particular person spent two hundred and twenty-eight dollars in Claude tokens, I don’t know if they did amazing work with that, or if they just burned through tokens. Usage does not equal proficiency. I agree we’re not measuring the right things, but I don’t know that we have access to the right things. I don’t like to use the word war, because it’s controversial, but this AI game in a law firm is only going to be won by fighting it desk to desk, person by person, and then going back to the first person and doing it again. It’s hard for me to understand when the metrics are going to give us really valuable information about effective tool use instead of just tool use.

Rob: I think that’s the next wave of technology you’re going to see, some of that governance, that ROI. But you’re right, it’s hard to tell right now who’s using what.

Does pasting into ChatGPT break privilege?

Amanda: So, Rob, we had a question come up in the comments. Can I backtrack us really quick? The question was, if you take something your attorney gave you and put it into ChatGPT or OpenAI, have you broken attorney-client privilege? Yes. There are active court cases where they’ve said that a client is giving up their right to attorney-client privilege by sharing documents from their attorney into a publicly available forum. So, yes. And that’s where, if we’re not telling our clients that, and it’s not in our engagement agreements, how are we communicating that to them? We tell our clients don’t talk about your case to your neighbor, don’t post about it on social media. But are we telling them don’t put this into a publicly available model that trains large language models? No, we’re not telling them that yet. Our firm isn’t. We will be, though, very soon.

Debbie: And if I could make one comment on that: the laws are being written as we’re speaking. Judges are making decisions that contradict other opinions as we speak. And a general counsel in a law firm, everybody has that lawyer who’s named general counsel, is that person actually keeping up with these rulings, to be able to help their lawyers understand what today’s rules say, how the rules of professional conduct are getting updated in your state and with the ABA? Are we staying up to date on that stuff? Because it’s not reliable for more than a day, a week, or a month, whatever your level of comfort is with paying attention to the news cycle on this.

Rob: Debbie, you also had a little tip about being on-site with a client and what you had them do.

Debbie: I don’t remember this one, just tell my story.

Rob: The setting in Claude. You had them go in and click the button, they hadn’t even turned off the setting.

Debbie: Yes, that button. They were saying, we can put anything into ChatGPT because we have a paid license, and then we went and looked, and they didn’t have “don’t use my information to train the model” turned on. It was turned off. They were like, what? So if you take away one thing, go check your settings.

Amanda: That takes us full circle back around the perils of just turning it on. You need to know what you paid for, and if you have a commercial or enterprise license, it has to be properly configured by a partner who has done this before, a company like Affinity or a company like your MSP. If you haven’t had somebody go through it word for word for you, you’re going to be sharing information, burning through tokens.

Rob: One takeaway: if you take one thing away, go back and turn it off.

A rocky first impression compounds

Rob: Okay, we have four more, let’s get through these real quick. With nine minutes, eight minutes left, I don’t even have to read much on the second one. A rocky first impression doesn’t reset. It compounds, especially with attorneys.

Amanda: Debbie started with that. That’s what was her first example, I think.

Debbie: Every time they tell the story about how it failed them, it grows into how badly it failed them. The more they tell it, the more they believe it, and the more they believe it, the less likely they are to try it again.

Training on the platform is not training on the problem

Rob: Bad apple. The third one: training on the platform is not training on the problem.

Amanda: That’s what I started with, is if you aren’t mapping the task individually for a person, figuring out their pain points and mapping it individually, then building the training around solving those problems, no one’s adopting the tool because it’s not designed for them. If we fix something that was annoying yesterday, that’s when adoption occurs.

Debbie: I agree. And I’d just say, there’s one of my favorite quotes, from Peter Drucker: there’s nothing so useless as doing efficiently that which should not be done at all. Sometimes you get these fancy tools and think, I’m going to make this better, and I’m like, hold the phone, why are we even doing that? It’s a good opportunity to examine all the things we think are mission critical.

Ownership on paper is not ownership in practice

Rob: I find myself doing that sometimes, solving a problem that’s not even there, just doing it to do it, and then wondering what I’m doing it for. Okay, ownership on paper is not ownership in practice. This is a lot about the 37% of firms with a formal AI committee or task force who admit no one is actually willing to make the hard calls when it counts.

Debbie: I feel like that’s Amanda’s life.

Rob: We’re live, Debbie.

Amanda: It is, but we’ve actually come up with a solution to this, and I’m really excited about what we’re doing. One of the whole reasons we started Sidebar is to help administrators know they’re not alone. We’ve been having to make the hard calls, or a single partner or managing partner is absorbing this decision, and by default it’s not designed that way. Your IT director might be more focused on break-fix, and that makes the strategy behind this more difficult. So yeah, ownership in practice is extremely important, and 37% is a huge chunk of people.

Debbie: So can I just say one thing about the committee thing? Because it applies broadly to lots of firms, and Amanda doesn’t have to say whether it applies at her firm. We don’t know what to do about AI, so instead of one person who doesn’t know what to do, we put six people who don’t know what to do in a room. I don’t know why we think adding six people who don’t know what to do suddenly makes it easier to decide. The reality is it’s a very challenging decision, and it’s in motion. We’re trying to decide about something that’s changing as we’re talking about it, and time is not on our side.

The loudest voice is not the right rollout order

Rob: I like that, it’s your stuck word. Okay, last one. The loudest voice is not the same as the right rollout order.

Amanda: What’s your stat here?

Rob: 44% of firms have no formal way to say no to a partner pushing hardest for a tool.

Amanda: And the next one is my favorite. Litigation departments show up in 68% of the ones pushing for AI and chasing after tools. I love litigators, they’re my favorite. That’s all I’m going to say. Not having a formal way to say no to the loudest voice isn’t AI-specific, that’s been happening in the twenty-one and a half years I’ve been in law firms. But in these areas where there isn’t a lot of shared knowledge, you need a formal way, and having the committee be the default isn’t always the right way.

Debbie: I can tell you that two weeks ago I was at a firm where an associate told me the firm hadn’t made an AI decision yet, so the equity partner, the big rainmaker, that person’s boss, walked into his office and put his personal credit card down on the desk and said, just sign up and put it on my personal credit card.

Rob: That’s one way to solve problems.

Debbie: It is one way, and it’s hard to say no. I think what we have to do is say yes, but, or yes, and. We have to make sure we dot the i’s and cross the t’s. This can’t be a wild, wild west decision, we have to do this methodically.

Why join Sidebar this week

Rob: So we have two minutes left. Why should people join Sidebar this week?

Amanda: First of all, their information feeds these conversations. We’re quoting exact percentages that come out of conversations we’re having on Sidebar. The bigger the data set, the more relevant our numbers are going to be. That’s number one. Number two, at that webinar, how many people said they needed help, they needed peers, they needed a group? Almost everybody.

Rob: Almost everybody. One of the most interesting things was how many attendees wanted a version of the presentation for their managing partners.

Amanda: A Sidebar email can be read by a managing partner on their mobile phone in a few quick minutes. So if you’re answering the question, you’ll get the response back that week, and the managing partner can read it and see what’s up to date. Next week, we have another question coming. We’re going to ask people to share what they’re personally doing with AI, what they’ve individually built, and we’re going to share some of the things administrators are actually building. We’d love to have someone join and do a show and tell. So sign up, especially if you want to share, brag about what you’ve built.

Rob: Shoot us a note on LinkedIn. Nothing’s too small, I’m always surprised by something you think is common knowledge that might be exactly the solution the next person was looking for.

Debbie: I’ll give you my twenty seconds. I think the world is changing so fast now. We used to be able to get survey results that lasted us the whole year. We’ve got to be talking to each other as a community every week to stay up to date on this stuff, and that’s what Sidebar is all about.

Rob: I’m going to take that and put it on the website. All right guys, thanks for joining us this week. We’ll see you again next week, same time, same place. Send us your questions. Debbie will be somewhere different.

Amanda: It was great to see you all. You and I will be at the same place, Debbie, who knows where she will be.

Debbie: I’ll be home. Next week I’ll be home. It’ll be amazing.