School for School Counselors Podcast

[Bonus Ep.] Are Yours Legit? The Uncomfortable Truth About School Counseling Resources

• School for School Counselors • Episode 114

Share episode

In the 114th episode of the School for School Counselors Podcast, host Steph Johnson discusses the critical issue of subpar materials flooding the market, especially with the rise of AI-generated resources. She differentiates between "evidence-based" and "evidence-aligned" materials, emphasizing the importance of using thoroughly vetted, peer-reviewed resources, particularly in high-stakes situations. Steph also offers guidance on where to find reliable, evidence-backed resources and advises on how to evaluate the quality and effectiveness of counseling tools. 

00:00 Welcome and Podcast Celebration

00:50 The Epidemic of Subpar Counseling Materials

03:47 Understanding Evidence-Based vs. Evidence-Aligned

08:56 High Stakes and Tier Two Interventions

11:48 Becoming a Discerning Consumer

18:26 Finding Reliable Resources

24:09 Conclusion and Invitation


*********************************

References/Resources:

American School Counselor Association. (n.d.). Professional School Counseling Journal. Retrieved from https://www.schoolcounselor.org/Publications-Research/Professional-School-Counseling-Journal

Collaborative for Academic, Social, and Emotional Learning. (n.d.). SEL programs. CASEL. Retrieved from https://casel.org/

Education Resources Information Center (ERIC). (n.d.). ERIC - Institute of Education Sciences. Retrieved from https://eric.ed.gov/

U.S. Department of Education. (n.d.). What Works Clearinghouse. Retrieved from https://ies.ed.gov/ncee/wwc/

**********************************


Our goal at School for School Counselors is to help school counselors stay on fire, make huge impacts for students, and catalyze change for our roles through grassroots advocacy and collaboration. Listen to get to know more about us and our mission, feel empowered and inspired, and set yourself up for success in the wonderful world of school counseling.

Hang out in our Facebook group
Jump in, ask questions, share your ideas and become a part of the most empowering school counseling group on the planet! (Join us to see if we're right.)

Join the School for School Counselors Mastermind
The Mastermind is packed with all the things your grad program never taught you IN ADDITION TO unparalleled support and consultation. No more feeling alone, invisible, unappreciated, or like you just don't know what to do next. We've got you!

Did someone share this podcast with you? Be sure to subscribe for all the new episodes!!


Speaker 1:

Hey, hey there. School counselor, welcome back to the School for School Counselors podcast. I'm Steph Johnson, a full-time school counselor just like you, delighted to be back here with you for the third episode of our podcast celebration power pack. We are celebrating the rise of the School for School Counselors podcast as the number one school counseling podcast on the planet, and it is all thanks to you, all of our listeners out there, who are willing to not only listen but to subscribe and rate and review. We can't thank you enough in our journey on the way to an official number one status on our chart. We made it to number three not too long ago, but we're aiming for that number one spot and you can help us get there. Thank you so much for being here with me Now.

Speaker 1:

In this episode, I want to talk about an epidemic that's going on in our school counseling world right now and it's a little concerning. This opinion, however, is probably also going to buy me several enemies. It happens from time to time. Some of our other folks in our school counseling world don't appreciate my views or opinions, and that's okay. We can have some adult conversations about them, but I'll call them like I see them, and that's part of this real world school counseling vibe we've got going on in School for School Counselors. We want to get down and dirty in the trenches with you and support you in real time and with your real issues. So what the heck am I talking about in this episode? What is the big deal about? What's going to make everybody mad? It is the epidemic of subpar materials that are currently flooding our school counseling marketplace. What do I mean by that? With the advent of AI and all of the artificial intelligence platforms, we have started to see, and will see, exponentially more resources, curriculums I always use air quotes around that or other counseling activities hitting the marketplace that are absolute garbage. They're going to be absolute crap. Unfortunately, that is just a side effect and it's something that a lot of industries are seeing.

Speaker 1:

When it's so easy just to type in a prompt and feel like you get a magical answer, a lot of folks all of a sudden become experts through their AI generation and sort of believe that they know more about their content than maybe they actually do, and they start pumping this stuff out, thinking they're going to make a couple bucks, or sometimes, you know, they just want to share it out of the goodness of their heart, they're doing it from the right place, but the information is not accurate, it's not appropriate for the situation. Sometimes AI just plain makes it up, and so we have to be discerning and understand that. But I will tell you, even in some of my own personal experiments with AI in the non-school-for-school counselor side of my life, I have run across things where chat GPT just straight up makes up information. It's telling me things and trying to convince me of things that do not actually exist in the real world, and that's scary. That's scary, I think too, when we're looking at school counseling resources, there has been this huge push, probably in the last six or seven years, that I've noticed where people are talking about years, that I've noticed where people are talking about this is evidence-based material, this curriculum is evidence-based, this protocol is evidence-based, and they don't have a clear understanding of what that really means. We're using those words to try to lend more weight and power to things than maybe they actually deserve, and so what I want to talk about in this podcast episode is how do we weed those out? How can we be discerning between evidence-based and evidence-aligned? Because those are two very distinct categories, and if you're not able to pick out the nuances of each. You're probably picking a lot of materials or supports that aren't showing you a lot of results and you're thinking why isn't this working? It should have worked. Everybody else says it works. Or maybe even worse case scenario, you take it on yourself. I must not have done a good job with that. I must need to up my game. I didn't know something. I should have done something better, when in reality it was the resource. So let's talk about evidence-based versus evidence-aligned.

Speaker 1:

Evidence-based means that it's been evaluated, that there has been some sort of study or research on the effectiveness of that thing. Evidence-based also means it has proven effectiveness, so it has been shown through trials to accomplish the job that it sets out to do. It means that it has something that's been peer-reviewed. It's not just the author's opinion. That is awesome. Other people think it works too, and typically these kinds of resources, if they're evidence-based, have a specific implementation. There is a specific way it has to be done. There is a protocol, there is a manual, there is a series of steps and a procedure in order to ensure a best outcome that is evidence-based, and so a lot of the things that people are trying to give you that they say are evidence-based are not.

Speaker 1:

They come from the second category, which is evidence-aligned. Evidence-aligned is different. It's drawing on other initiatives, other resources, other points of peer-reviewed literature to say this kind of approach works. We did it differently, and so there is that foundation of evidence, but it doesn't belong to the material itself, if that makes sense. When people put out evidence-aligned materials, what they're doing is they're incorporating elements that align with what the research says works. So they're moving in that general direction. They're not specifically implementing a true blue protocol. They're saying this has been known to work, so we're going to try that. We're going to give you this because we know in these instances it's worked before. That's not a terrible thing, right? That's how we learn and grow and develop practices. That's how we deepen our understanding of our interventions. That's not always bad. But the distinction I'm trying to make is if somebody's telling you that they have an evidence-based material, is it really or is it evidence-aligned? New programs fit into the evidence-aligned category until they become evidence-based.

Speaker 1:

There is a very popular social-emotional curriculum out there right now that everybody just seems to be raving about. That when it was first released had to be recalled. They provided it for school counselors to review, to give their constructive criticisms and insights, and what they found was that some of the activities that were built into that curriculum were not developmentally appropriate. They were counter to what we know to be true in the school counseling world. Some of them were even, I would say, borderline damaging to students. They recalled that curriculum and reworked it. They put it back out in the marketplace again. It didn't have its own evidence base yet. They were working on it, but they didn't have it. So it was evidence aligned yet they were working on it, but they didn't have it. So it was evidence aligned and it stayed there for many years and, I think just recently, was able to acquire its evidence base, where now it can stand on its own. Now it has its peer-reviewed data and literature to say this approach works, the way we put this specific thing together in this specific way gives these specific outcomes.

Speaker 1:

Are you seeing the difference? This is important for us to realize in our work. So when does it matter? You're like Steph, like I have a million things to do. I don't have time to go read all these journals and select all these interventions. What does this really mean to me? Most of the small group curriculums on TPT look all like the same. What's the difference? Why does it matter?

Speaker 1:

I think if it's a high stakes situation, you need to go evidence-based. I think, at the end of the day, if you have a high-stakes situation with a student and you're not using evidence-based materials, in my opinion that could be considered borderline unethical, because you do have access to information about what works. You have access to treatment planners, you have access to protocols, you have access to journal articles. You should have access to the Professional School Counseling Journal through the American School Counselor Association and you can access others as well on your own. But you need to make sure you're doing your due diligence. If this is a high-stakes situation, if this is a kid that's known to have high levels of crisis, if they're a threat to themselves or others, if it's extreme behaviors, those kinds of things, that calls for evidence-based. We don't want to mess around with that. We don't want to give them the anger management hippopotamus and hope that it works. It may be evidence aligned somewhat, but is it evidence-based? Let's pull out the big guns when it's high stakes and if you need help identifying those programs, we can help you with that, all right.

Speaker 1:

I think another area that we really need to be mindful about, evidence-based materials. Some of y'all are not going to want to hear this, but it's in our Tier 2 interventions, because if a student has demonstrated that Tier 1 is not adequate for their needs, if they still have some additional needs in the schoolhouse, if they have some unmet needs, if they have some lagging skills whatever the case may be if they have demonstrated that your tier one efforts are not enough, we don't want to start throwing spaghetti at the wall for tier two. We want to make sure that we have something that's intentional and that's targeted, and here's where I've made a lot of enemies, because there are a lot of people out there who do not like me talking about this. They have built empires on top of the resources they're creating, and some of them are really good. There are a few over there that I will still follow and sometimes use things from, but I do want you to get some thought to this.

Speaker 1:

I want you to be thinking about what am I using that's evidence-based, what am I using that's evidence-aligned? And then, third, what am I using that isn't evidence, anything? And, y'all, there's a lot of that out there too. So I will say this If you pick up a resource and you feel like evidence, aligned, is adequate for your purposes, and as a professional with mental health training, you are qualified to make that decision. I don't want you to feel like I'm talking down to you or telling you you must or you should, or you don't know what you're doing. That is not my message here. I'm just trying to expand your understanding a little bit. I want to challenge your thinking because it makes us all better.

Speaker 1:

Look at the resources that you're using and try to vet the methodology behind it. Vet the seller. See who they are, what is their area of expertise, what is the mechanism behind the change that this thing is going to facilitate? It's going to take a little bit of reflection. It's going to take a little bit of reflection. It's going to take a few minutes of really investigating this thing. And the sad part is you really can't investigate all those nuances until you've already purchased and downloaded it, which I think is a major shortcoming of teachers pay teachers, but that is a whole other podcast episode.

Speaker 1:

Here's what we need to be looking at. Does the resource have data? Can the creator provide some kind of a data to show that it works. It blows my mind that we still have people in the marketplace selling initiatives and interventions with no data behind them. We're supposed to take them at face value. Sure, it works. It's cute, isn't it? Research shows us and we have empirical evidence of this that typically the cuter a resource is on Teachers Pay Teachers the less effective it is. See where I'm making some school counseling enemies right now. Yeah, because these are hard things to talk about, but they're true.

Speaker 1:

So does it have data behind it? Can you identify a clinical rationale? Can you look at that resource and identify the underlying mechanism? And I'm not saying, look at it and slap on a general label of oh, that's CBT, that's not what I'm talking about. What specific aspects of CBT is it employing? What is the mechanism that's facilitating change? Can you identify it? If it's a bunch of fun activities and we're talking about our feelings and we're talking about different ways to approach things in the future, that's all well and good. Under what umbrella? What's the mechanism of change? Is it developmentally appropriate? That's the other thing.

Speaker 1:

We have to be able to look at these resources and really identify if they are what we're being told they are. I think also, it's important to keep this evidence-based versus evidence-aligned situation in our minds, because it's important for us to evidence-based versus evidence-aligned situation in our minds, because it's important for us to practice what we preach. You see my hand going if you're watching the video, because I just get so invested in this idea. If you were called to explain the resources that you're using in a public forum, could you justify them? Could you justify them beyond the price? And it looked really cute and it seemed fun. If those are your only three criteria, you might need to look a little bit deeper.

Speaker 1:

If we are in our industry, harping on the advocacy angle of things and what we should be allowed to do and comprehensive school counseling programs and national models, and we're shooting all over everybody all the time Go back to episode 102 if you want a good reference on this and then we expect people to invest in ASCA's best practices, then we need to build up ourselves by utilizing best practices as well. We can't have it both ways. We can't say our industry standard is this and this is what is suggested for the best outcome for students, but we're going to do our own thing over here. Pay no attention to what's happening behind the curtain. If we want to be able to make these arguments, we've got to be able to walk the walk while we talk the talk. This is going to hurt a little bit. This is going to challenge you, because that means no more print and pray lessons. Print them and pray they work. Print them and wonder why they didn't work. And it's really not hard. I'm not saying that everything on Teachers Pay Teachers is not worthwhile, but what I'm saying is they have enough of a customer base now that it ought to be relatively easy to gather that data. So why aren't we doing that? Why aren't we publicly vetting our resources? It would be so easy to do.

Speaker 1:

If you're a resource creator, I would urge you to provide the rationale. Explain the methodology behind it. This is not a trade secret. Your scoot game and your match the picture to the word and your drawing activities those are not trade secrets. So identify for your colleagues what the mechanisms are. What are you aiming to do and through what methods? How is this developmentally appropriate for kids? In what other situations has it been presented and been effective? Make your customers informed consumers. Don't expect them just to take your word at it, because you have really cute graphics and the lettering's fun and you have some really great ads and you have a big, booming Facebook group. None of those make an expert. So show us your expertise. Talk through these things, add a video in your listing.

Speaker 1:

Hey y'all, I just want to walk you through the components of this. This is lesson one. We're doing this and this is why we're doing it. We're following this protocol for an intervention in this specific issue. Protocol says we're going to do this. We're doing it in this way. The second part of the protocol says this we're doing it this way. This is the rationale for why we're doing it. At the end, you should have this outcome. Other school counselors that have used this resource have reported X, y and Z. That's not a perfect system, but it's a start, and so I would really encourage anybody. If you're thinking about doing that in the future, think that direction. It's going to put you head and shoulders above everybody else.

Speaker 1:

Okay, so then, where do we find these things? You're like Steph, I'm motivated, I am bought in. I want to look at evidence-based, evidence-aligned. I want to be able to tell the difference. I want to make sure that I'm using resources that have some sort of evidence behind them. Where the heck do I find them? Here's where we're going to go.

Speaker 1:

Number one we're going to go to CASEL. Some of your favorite things that you love using you're not going to find in CASEL and it's going to break your heart. There are some very well-known curriculums that do not have a place on the CASEL list of social-emotional learning programs. That's for a reason, my friends. So be thoughtful about these things. Go look them up on the CASEL list and see if they're available there. Second, the what Works Clearinghouse. This is an oldie but a goodie. It's not a fancy website. It's not flashy, but it has good, solid information. So look that direction. It's going to give you ideas based on topic of evidence-aligned and evidence-based interventions.

Speaker 1:

Third, utilize your journal access. You have it through your ASCA membership. Log in, click the menu to the left and then click on the right to access the journal. I do it about 200 times a week. I'm constantly looking for literature, I'm constantly looking at the new information that's coming out, because I wanna better my craft and I wanna help support you along that same journey. So utilize your journals. And fourth, use the ERIC database. That's the Educational Resources Information Center. If you Google ERIC database, a whole bunch of resources are going to be at your fingertips and, again, they're all going to be evidence-based. They're going to be giving you the lowdown on the data, the populations, the situations, what worked, what didn't, and that's the kind of information that you're looking for.

Speaker 1:

You can also look for evidence-based treatment planners, and, while I realize we're not therapists, we don't generally use the word treatment in our work. There are some that are designed for school counselors, for brief, targeted interventions, so why not use those? Why not pull out the evidence-based pieces within those treatment planners and utilize those For evidence-aligned materials? The first thing you've got to do is look for a logic model. What is the structure? What is the intent? I've talked about this at length in this episode, so I'm not going to go back and repeat all that, but make sure you can identify the logic model. Look at the author's credentials. Are they a school counselor or are they an influencer? Those two things do not have to be mutually exclusive. We have a lot of school counselors turned influencers that just expect you to take them at their word because they have a big name in the marketplace. Some of those folks are really awesome. Some of those folks are turning out trash, and so you need to be aware and really thoughtful about the people that you're buying from.

Speaker 1:

Is the material plagiarized? You would be amazed at the amount of plagiarism on Teachers Pay Teachers, and if you look at the resources closely enough, you'll be able to see who started the trail of resources of that thing. You'll be able to identify them backwards. Well, first, this person did this thing. Everybody really liked it, so this person copied it. That person copied it. That person copied it. They changed it from a pirate to a hippopotamus, put some new words on it, slapped a fun little song at the end and called it their own Evidence base tells us that that is a very common occurrence.

Speaker 1:

So look and see. Is it their own workers that have been plagiarized? Who came first? And then lean on your training and your expertise. Can you identify the mechanism? Can you pick out what feels like best practice? Because again, with the GPT garbage that we have only begun to start seeing in our marketplace, it's a trickle now and it's about to be a tsunami. You mark my words.

Speaker 1:

We are going to have to be able to up-level our ability to utilize our professional discernment and to be able to make great choices, looking at evidence-based versus evidence-aligned materials and non-evidence-based or aligned materials is going to be so important going forward. Let's stake my reputation on this. I think it's going to be make or break in our school counseling industry. I think that if a principal comes to you and says where are you using this intervention, if you're not able to explain the mechanism behind it and what you're doing here, in the very near future a lot of jobs are going to be at stake. We have to assert our authority and expertise.

Speaker 1:

People already think that we only sit and talk about feelings and give kids a piece of candy and send them to class, right, or that we just fix schedules. And if we don't change that perception and we ride this tide of subpar materials, subpar resources, we're only going to have ourselves to blame when people start going. Well, that's easy. Anybody could do that. Let the media coordinator do that. Let the English teacher teach that. Heck, the PE teacher could teach that. That's a whole other conversation. You know if you can download it and the PE teacher could teach it without you being there, is it really a school counseling resource? It's something to think about.

Speaker 1:

All right, this episode has gone on a while, but I hope it really challenged your thoughts about some of the things you're doing in your program. It doesn't mean you need to go throw everything out. It doesn't mean that if you've been using resources that don't have an evidence base or aren't evidence aligned, that you're a bad counselor, that you are not good at your job, that you're making a ton of mistakes. It doesn't mean any of that. It means that now you've listened to this, you've expanded your understanding. Right, we don't know what we don't know until we understand that we didn't know it. And that's okay. That's part of growth, that's part of learning and developing as a professional. And so I hope that's what you take away from this conversation Not any finger pointing, not condemnation, but just expanding your understanding and your intent to better utilize the evidence base that we have at our disposal, because if you can do that, you will be winning 1000% winning.

Speaker 1:

All right, if you need some support in identifying some of those resources or in ways to identify those resources on your own, come join us in our School for School Counselors Mastermind. It's open all the time because we want you to be able to get the help you need when you need it. We have weekly support and consultation chats where we can have these kinds of conversations and identify these kinds of resources. Often we put them right there up on your screen and we walk through them together to identify the mechanisms, the rationale behind them, if they're going to work, what populations they're appropriate for. But we also do case consultation and other related things. We have a monthly master class. We have a monthly data discussions where we're working on school counseling program data, and then we have a monthly 15-minute challenge that is really going to move the needle and give you an empirically based product to give to your administrator at the end of every month. It's phenomenal, it's mastermind, it is just the coolest community ever and I would love for you to come join us.

Speaker 1:

Schoolforschoolcounselorscom slash mastermind. All right, so this was the third installment of our podcast power pack. I'm going to be back soon with our regular episode of the School for School Counselors podcast, so stay tuned, but in the meantime, I hope you have the best week. Take care.