TPT's Dirty Truths & Why You Need an Evidence-Base

Ever wondered why some school counseling programs deliver long-lasting results while others fall flat? In this episode of the School for School Counselors podcast, host Steph Johnson addresses the critical need for evidence-based practices in the field of school counseling.

Though counselor-created resources are popular, Johnson emphasizes the importance of moving beyond these kinds of resources commonly found online. She discusses the insufficiency of peer-reviewed research in current school counseling interventions and offers reliable alternatives such as cognitive behavioral therapy (CBT), mindfulness-based interventions, and established curriculums like Second Step and MindUP.

Additionally, Johnson highlights the potential harm of inadequately researched resources found on platforms like Teachers Pay Teachers. She provides practical advice on where to find validated interventions and underscores the value of collecting long-term data to assess their effectiveness.

If you truly want to be a better school counselor, listen in for actionable insights to elevate your school counseling practices, backed by the power of evidence-based approaches that truly make a difference in students' lives.


Mentioned in This Episode:
School for School Counselors Mastermind


Resources:

American School Counselor Association (2019). ASCA school
counselor professional standards & competencies. Alexandria,
VA: Author. Retrieved from https://www.schoolcounselor.org/getmedia/a8d59c2c-51de-4ec3-a565-a3235f3b93c3/SC-Competencies.pdf

American School Counselor Association. (2021). ASCA student standards: Mindsets & behaviors for student success. Retrieved from https://www.schoolcounselor.org/getmedia/7428a787-a452-4abb-afec-d78ec77870cd/mindsets-behaviors.pdf

Collaborative for Academic, Social, and Emotional Learning (CASEL). (n.d.). Retrieved from https://casel.org/

Committee for Children. (n.d.). Second Step program. Retrieved from https://www.secondstep.org/

Gallagher, C., Swalwell, K., & Bellows, L. (2019). Editorial processes and quality control in educational materials. Educational Studies, 48(1), 112-128.

Kendall, P. C. (1990). Coping Cat program for anxiety disorders. Journal of Clinical Child Psychology, 19(2), 99-107.

MacArthur, J., Harris, L., Archonbald, N., & Shelton, K. (2021). Findings on the quality of activities in educational resources. Journal of Educational Research, 54(3), 245-262.

National Association of School Psychologists. (n.d.). Professional development. Retrieved from https://www.nasponline.org/professional-development

Sawyer, R., Dick, M., & Sutherland, J. (2020). The correlation between resource aesthetics and cognitive demand. Mathematics Education Review, 62(4), 301-318.

The Goldie Hawn Foundation. (n.d.). MindUP curriculum. Retrieved from https://mindup.org/

U.S. Department of Education. (n.d.). What Works Clearinghouse. Retrieved from https://ies.ed.gov/ncee/wwc/

Transcript

Speaker 1: 0:06

Hey there, school counselor, welcome back to the School for School Counselors podcast. I'm Steph Johnson, your host, and I'm going to tell you right away today I'm expecting to get a lot of hate mail from this podcast episode. Oh my gosh, what a way to start right. But we're going to be tackling some taboo subjects in our school counseling world right now. I know there are many of my colleagues who do not agree with me and are not going to be happy with my choice of topic, but I really feel like this kind of stuff needs to be talked about. We need to be having more open conversations about this and we really need to be assessing where we stand in the school counseling world. I'll tell you all about what I mean coming up here in just a minute, but before we jump in, I want to share with you a sweet review that we received from one of our listeners very recently. This comes from Reba1976, and they say beyond grateful. The review goes on to say this podcast and all of the resources School for School Counselors offers are game-changing.

Speaker 1: 1:17

I'm in my second year of getting back in the field of school counseling, 15 years after graduating. Needless to say, I've had a lot to learn and, for a while, got lost in all of the noise out there. But since finding this podcast, steph is my number one guide. Everything is explained so clearly and with such realistic perspective and tons of empathy. I'm just incredibly grateful to be able to absorb all the knowledge and insight. Oh my gosh, thank you so much. That means so much to know that we're hitting the mark with exactly what we've aimed to do, which is to give you some real-world perspectives, cut through all of the baloney out there and guide you toward your realization of your best school counseling practice. So thank you, reva1976. That was so, so kind for you to take a few moments to leave that review and y'all. If you haven't left us a review yet, you got to go do it. Just hop on into Apple Podcasts, leave a review for us, letting other folks know If you find our podcast helpful, and I'll tell you it makes a huge, huge difference for us each time one of those comes through. We just reached 10,000 downloads per month, very, very proud of that, because each and every download means that we're being of service, and that's what this is all about, all right, so this week I want to talk about evidence-based practices for school counselors.

Speaker 1: 2:52

This is a super important topic and one that I don't see acknowledged a lot. I think it's because it is difficult to put your finger on evidence-based practices in school counseling. There's not a ton of research on this stuff. It's difficult to find peer-reviewed resources on it. There's a whole bunch of other stuff out there, and so it's hard to know if the interventions that you're providing are going to be effective, if they're going to work for the students you have in mind, and sometimes, if you're not diligent in collecting your data, you don't really know if it ever worked. To be honest, and it's so funny because I see people doing some pre and post assessments in their small group curriculums and again I'm using air quotes when I say that but seeing people do these pre and post assessments for these small group curriculums, but you never hear people talking about administering those post-assessments several weeks after the group to see if they retained effectiveness. Short-term effectiveness is easy to grab. It's the long-term stuff that we really need to be looking toward, and we see almost none of that in the school counseling field.

Speaker 1: 4:14

Evidence-based interventions have been shown to drive positive growth for students. If we're giving anything else, it's a shot in the dark. It's throwing spaghetti at the wall to see what sticks. And I do think a large part of what I'm seeing in the school counseling world is just that People finding these random resources, utilizing them with students and then feeling surprised when they don't see the level of change they expected or they're left wondering why they didn't work or, worse yet, feeling like they're an incompetent counselor because they're not able to fix these problems. So what is an evidence-based intervention? What are some specific evidence-based interventions that you could be using with students? Sad story, they're not the flashy, cutesy, sexy resources that you see advertised on the internet. It's almost like I tell my kids here at home if they have to give you a commercial or an advertisement for it, it's probably something you don't need.

Speaker 1: 5:24

Good examples of evidence-based interventions would be cognitive behavioral therapy. That has a lot of evidence behind it. There are components of CBT that we can pull out and use with our students. There are some evidence-based curriculums that are built on the back of CBT, and so it's worth mentioning. Other evidence-based interventions might be mindfulness-based interventions those mindfulness-based activities, protocols, formal curriculums. A lot of them have a lot of evidence base behind them.

Speaker 1: 5:59

If you spend much time in our School for School Counselors Facebook group. You hear me talk a lot about the Mind Up curriculum, which I absolutely adore and recommend to folks any chance that I get. It's low cost. It has a wealth of evidence base behind it. They publish all of their research resources there, what they've drawn on, how they've tested the curriculum. It's all there on the website and the best part is you can buy one guidebook that covers roughly a span of four to five grade levels per book and you can get it for 30 bucks. It's run by the Goldie Hawn Foundation. It's just an amazing curriculum and one of the reasons that I love it so much other than I've used it and I've seen it work, I continue to use it is that it has a huge evidence base behind it.

Speaker 1: 6:52

Another example of evidence-based interventions would be certain bullying prevention programs, formalized programs with evidence base behind them. You keep hearing me say this over and over again, but it's something that we're just not picking up in the school counseling world. We seem to have this idea that we can pick up pretty much any resource. We want to present it to a kid and it's going to solve a problem, and unfortunately that's just not true, and here's where I make a lot of enemies, because a lot of my colleagues in the school counseling world don't want you thinking about that.

Speaker 1: 7:35

There is a huge push right now towards this online freedom, side, hustle, lifestyle, entrepreneur mentality. Have you guys seen or heard these words floating around? And about twice a year there's a big influencer who releases a program and convinces everybody that they can start selling their materials on Teachers, pay Teachers and become a millionaire virtually overnight, with almost no work. I can tell you firsthand that that could not be any further from the truth. To me, that feels like snake oil selling, but that's a whole other podcast topic, point being, if you don't know what you don't know, then you don't know. And a lot of these folks don't realize or haven't thought through the fact that these interventions have little to no evidence base behind them. The things that you're seeing advertised on Pinterest, in blogs on Teachers, pay Teachers, in social media ads, typically do not have an evidence base, and that's troubling.

Speaker 1: 8:46

Now am I saying you should never, ever use those resources, that you should stay away from them, and that you're being reckless and irresponsible if you're using them? No, I'm not. I think for certain aspects of our work they're dynamite, but when we're talking about implementing an 8 or 12 week program for students based on aggression or anger management or anxiety, we better make darn tootin' that we have a good evidence base behind what we're doing. Let me tell you why I say that there's been a lot of research into the online world, and I'm going to use teachers pay teachers as an example, because I think that's what most of you guys are relying on for your supplemental materials and resources. So a study by MacArthur, harris, archambault and Shelton in 2021 said and I'll quote them here findings show that activities were aesthetically well designed, but 70% were of moderate or low overall quality. According to our rubric that assessed areas such as higher-order thinking, connections to student experiences and representations of multiple perspectives, 30% of the activities this is still a quote were found to be potentially harmful to students. That's a sobering statistic, isn't it? And I will say this is not specifically about school counseling, but this movement for people creating these resources, getting them up as fast as they can, I think is pretty prevalent across all subject matter.

Speaker 1: 10:34

We have lots of people super invested in their content but maybe not realizing that it's not being presented in the best way. And if you don't have folks engaging in research on what you're putting out there. You don't truly know if it's working. You may have anecdotal evidence right, you might have what we used to call action research which we use that term very loosely way back when I think they've tightened it up a little bit since but you can't just try it yourself and then say everyone should use it. That's not how that works.

Speaker 1: 11:08

Gallagher, swalwell and Bellows in 2019 state that with no editorial board vetting process or quality control, these sites can act as turbocharged conduits for bad ideas disguised as cute lessons. Man, they are not mincing words there. They're letting you know exactly how they feel about this. It's interesting that they bring up cute lessons and they have that in quotation marks. It's cute lessons. Let me tell you why it's stated that way. Sawyer Dick in Sutherland in 2020.

Speaker 1: 11:48

When they were looking at mathematics resources, stated that they found a correlation between cute, non-functional images and lower-level cognitive demand tasks, meaning that the cuter the resource was, the less effective it was to state. Other researchers like Hertel, wesman and Inziger found similar results in terms of lower quality resources on websites like Teachers, pay Teachers and Pinterest. Less than one percent of the resources that these folks looked at could be categorized as the highest level of cognitive demand and only 38 percent were at the second highest level of demand. And the really interesting part about this is that they found that when they surveyed the creators of these materials, the largest proportion of those that responded explained their popularity on Teachers Pay Teachers as the time they took creating and revamping items for their customers. So in these folks' minds these products are valuable because they're spending all this time. They're revising them when needed, but they really don't realize that their materials aren't making the grade. And so I bring this up not to bash people who are creating these types of materials.

Speaker 1: 13:15

Again, I have a select few in my office that I rely on that I really feel do a good job in certain specific, isolated components of my school counseling work, but very, very rarely do I ever print out a so-called curriculum or anything like that, because I know from my research base that I can achieve the same end from an evidence-based practice. Aggression does not necessarily require its own curriculum. Anger does not necessarily require its own curriculum. There may be different aspects of that curriculum that I'm going to focus on more, but at the end of the day, the majority of concerns we're looking at in schools revolve around some sort of regulation, and that's where our CBT, our mindfulness, those kinds of things come in.

Speaker 1: 14:11

Interestingly enough, talking about cute images Sorry if I'm jumping all over the place, I'm just literally talking off the cuff without a script here. I don't script these podcasts. I don't think it helps anybody when I do that. I want to be very, very real with you and my thoughts as they come to mind. As we're talking about these cute resources and about making them aesthetically pleasing, right, being able to be commercially driven. You wouldn't want to put a lot of the evidence-based materials that we have available to us on an advertisement because they wouldn't be pretty, and our culture very much is driven by pretty and flashy, and so things like Coping Cat. If you've ever seen a Coping Cat book, it's a little sketchy looking when you first look at it, or at least mine is. It looks like it was run on a mimeograph machine back in 1985, but I don't care, because it works. It's evidence-based. I know where it's going to take me. Same thing with Second Step. They stepped up their game a little bit with their aesthetics here lately.

Speaker 1: 15:14

A lot of folks don't want to do second step. As a matter of fact, I often see people in social media say recommend a curriculum anything but second step Because this perception that it's slow, it's out of touch, it doesn't work when it has one of the largest bases of evidence behind it of any curriculum out there. But on the other hand, we have colleagues recommending all this other stuff that has no evidence base behind it whatsoever. I know there's one big name curriculum out there right now that has a celebrity behind it that has been trying to get CASEL approval for some time and can't acquire it because they can't meet the standard. So I want to really, really encourage you as you're looking at resources and materials for your students keep an eye toward this evidence base. It's so, so, super important.

Speaker 1: 16:11

So where do we find this stuff? Where do we decide where our evidence-based interventions are going to come from? There are a few places that you can look. The first one I'm going to offer with a caveat, and that is this If you want to look toward the ASCUS side of things, they publish their standards and competencies, mindsets and behaviors all this information about what we should be tackling in the school counseling world and they do have a little bit of an evidence base behind it, from what I've been able to find.

Speaker 1: 16:46

For example, when I look at the ASCA student standards that were developed in 2021, they have let's see, I'm looking at it right now 10 resources listed, if you want to call them that. The problem is there are no citations for any of them and no links for them, so I don't know if that really qualifies as a resource. It would be difficult to go find these documents. There's no way. We've been given no means to do it. It would take some research to find them. Category 1 has 6 mindsets. Category 2 has 27 listed behavior standards listed. Out of all those, we have 10 resources. So interesting and again, I don't know how these were devised. I haven't sat down with anybody that does this kind of stuff. It would be an interesting question to ask, but I feel like even a little old podcast like us will cite our research sources, so I'm interested to see if they ever start listing them in ASCA.

Speaker 1: 18:03

And then we look at the ASCA Standards Aligned Curriculum. It goes on, for I'm scrolling, I'm scrolling. My PDF reader is showing me 34 pages. Hang on a minute. It's taking a minute to catch up. 30 printed pages for the standards aligned curriculum. We have thanked the people that have helped develop that.

Speaker 1: 18:29

I do not see any research base for these. Not saying that they need to have them, because these are just standards that we're touching on, but it would be interesting to see where all these came from. But there's no documentation to that at all. Let me make sure I've not misspoken here. Yes, and even the category for research and program evaluation is very sparse. On critically examining the published research, there's not a lot there. So again, not bashing anybody, but just bringing to mind some holes we need to start filling in our school counseling world. We need more of an evidence-based in things we're doing. So ASCA is an interesting source. Use it, but be thoughtful about it if you go down that road.

Speaker 1: 19:19

Just because an activity states that it's ASCA, aligned or based on CASEL competencies, doesn't mean a whole lot. I can base my life around healthy living and still do some really unhealthy things, so it's no guarantee. So be careful with that phrasing and wording. I think it's meant to impart a status that maybe necessarily shouldn't be ascribed to that resource. We're trying to make it look good and, yes, it's targeted toward the areas that we need to be talking about, but is there any evidence-based that? Your little scoot game, your put the worries in the monster's mouth those kinds of things are going to be effective in your situation. We need some more evidence-based to that. We're not just going to take somebody's word for it because they've decided they don't want to be a counselor anymore. They'd rather create curriculum.

Speaker 1: 20:11

Okay, so where do I find my evidence-based practices? Number one and I don't talk about this often, but as a licensed therapist, I am very, very invested in evidence-based practices. I want to make sure that I'm bringing the best empirically validated content to my students that I possibly can, just because that's the mindset out of which I operate. That doesn't mean I'm giving therapy at school. It doesn't mean I'm trying to be a therapist on my school campus. I'm not. I don't even claim that distinction in trainings in my email signature, anything like that, but that's very much the way I've been trained to look at things. So I'm going to be looking at things like the what Works Clearinghouse. That's going to give me a good rundown of evidence-based interventions. That is sponsored by the US Department of Education. We'll leave a link to that one in the show notes. Another good one would be your academic research, your journals, peer-reviewed literature, what is working? What are the programs that have been identified to move the needle for students?

Speaker 1: 21:18

Some of the online training platforms can be helpful, depending on who they're coming from. Again, I'll give you a caveat here, because it's super easy now to create a website and create an organization to do training. There are a lot of folks out there doing that that, in my opinion, aren't qualified to be doing that. But if you're sticking with something like NASP, I love NASP. The information that they put out is amazing. They are usually at the forefront of big policy statements, legislative pushes and all that kind of stuff. I just have always been super impressed with them. They do some great online training, some great webinars and things like that. That's naspcenterorg. That's a great one to go to, and they're backing those trainings up with their own peer-reviewed literature.

Speaker 1: 22:09

Say it louder for the people in the back, because that's what we're missing in the school counseling world right now. In our mastermind program, when we're providing information for folks, you will constantly see citations running at the bottom of our slides. You will see citations on our printables. You guys constantly see citations in our show notes for our podcast. We are very, very driven by that type of empirical validation, and we really hope that the rest of our field is going to start looking in that direction too, because I think it's going to be the best outcome for school counselors at large as well for the students and the families that we serve, and so I've given you some resources for finding your own. You may be thinking, steph, what are some of the evidence-based interventions that you use? You might be interested to know what I'm personally using in my school counseling program.

Speaker 1: 23:05

Tier one we're using Second Step Again has a huge research base behind it. A lot of people aren't in love with that curriculum it. A lot of people aren't in love with that curriculum, and I get it. It's not flashy by any means, but you know what it is. It's solid, it's vetted, and a lot of the effectiveness of Second Step, in my opinion, is the dedication and the enthusiasm with which it is presented. I can look at a lesson in second step and I can immediately identify exactly what those folks are targeting in on, and I'm not saying like a generic skill label like this week. We're working on identifying feelings. Anybody can look at it and see that, but it's the specific mechanisms of how they're doing that Because I am so invested in the research, I can look at it and say this is what's happening.

Speaker 1: 24:00

I often give this example to my teachers. There's in the online version of Second Step. There is an exercise where the students listen and there's some sort of a sound they're listening for and they have to raise their hand when they hear it. There's another one where it's a clapping and they have to raise their hand when they hear it. There's another one where it's a clapping pattern that they have to emulate and it's such a simple pattern and when we show it to teachers in training, you can almost feel them rolling their eyes, like it is just like oh, my goodness, give me a break.

Speaker 1: 24:30

And it does look lame. It looks really, really silly and lame. But if you know the evidence base behind it, if you know what they're doing, you know there's a mechanism for that and you know there's a purpose for that game. They're not just filling time. They're not just having kids clap their hands for the fun of it or to bore them to tears. However you were looking at it, it really serves a purpose in that lesson. It's teaching kids attentional control. It's teaching them focus. It's teaching them mindfulness in all of those little activities, and so don't take all of that stuff at face value. I think that's where a lot of this evidence-based stuff gets discredited by folks is it just looks silly and simple and lame, but it most definitely is not. So we use second step.

Speaker 1: 25:18

Personally, again, I love the MindUp curriculum. I think it is appropriate for so many things. Anger, aggression, anxiety can all circle back to mindfulness being a component of addressing that. And in the school setting, where we're very short term, we don't have time to get super therapeutic, that's a great place to start and if the student isn't able to better regulate after working through a program like that, then we know we need to be looking at referring on to additional services, other evidence-based protocols. I have been known to not use this one with Fidelity. It's a little long for the school setting.

Speaker 1: 26:01

I really like Coping Cat. I think it's super helpful for specific instances of student concerns. Sometimes we have some severe separation anxieties, anxiety at school, those kinds of things. Coping Cat is really great with that. It has a great evidence base behind it. Castle is another one. They're a clearinghouse of all of their list, of all the social emotional programming, the audiences in which it was tested, what the outcomes were, whether or not it reaches the point of the pinnacle of their suggestion, or whether it's one that's a promising program that's a great resource to look at. We'll link to that one as well.

Speaker 1: 26:42

And then any of your I hesitate to use this word, but this is what they call them treatment planners or school counseling treatment planners out there. Look toward those with evidence-based practices embedded in them and they will guide you toward the right direction. Now, the cons of this type of approach is time, and I think that's where these cute little resources have maybe taken hold is because it's so much easier to search for. You know a small group curriculum on anger management I'll just use that one, because I hear that one a lot. Find one that looks good, looks cute, looks like the kids will like it, print it off and roll with it, and that's just about all the time. We feel like we have to do this. But as you get more invested in your evidence-based practice, you'll find that you don't have to seek as many resources, because you can look at a situation and go oh, this is what we need. This is the direction we need to be headed. We need to have a little dialectical behavior therapy going on. We need to have a little bit of mindfulness going on. We need to have these evidence-based practices in place. You'll be able to pick those out really quickly as you use them more and more and more.

Speaker 1: 27:55

So, again, I do expect to get a lot of hate mail for this episode. There are a lot of folks out in the school counseling world who feel very threatened by my views on this. This is their livelihoods that I'm talking about. They're making quite a nice living selling resources online, but the question is are we able to empirically validate them? And if the answer is no, then we need to look further into can we see components of empirically validated practices? If you can't see either one of those pretty readily, I'm going to suggest that you move on to something a little more vetted. But again, this is up to you.

Speaker 1: 28:35

If you have any questions about evidence-based practices or where to find them or what you want to do with them once you have them, I know a great place you can go to start. It's called the School for School Counselors Mastermind. You hear me talk about it every week. That's the only thing I talk about, because that's what I love. I love getting school counselors in a group each and every week to talk through challenges, case consultation and to provide support, because our school counseling world is messy and sometimes we just need to know there's other people that get it. This past week we had an extensive case consultation with one of our members. We were able to provide them with multiple action plans for a situation on campus and we are eagerly awaiting an update when we have our next chat to see how she fared with these action plans. What needs to happen next? They're a little bit nervous about things that are going on and I don't blame them, but we're walking with them through it every step of the way. That's what we do in the School for School Counselors Mastermind. You can find out more about it at schoolforschoolcounselorscom.

Speaker 1: 29:41

Slash mastermind.

Speaker 1: 29:44

All right, my friends, I'm going to let you go for this week, but just keep an eye toward what you're doing.

Speaker 1: 29:49

You don't need to scrap everything that you've been presenting to students. If you have some resources that look like they're working well, continue using them, but capture that data on those and capture data two or three or four weeks out to see if you're seeing sustained benefits. I would really love to see our TPT resources. Start publishing that information, crowdsourcing that type of data, instead of relying on these starred reviews that may or may not be legit. Guys, I'm not saying that your favorite school counselor authors do it, but there are a lot of folks on TPT that are paying for fake reviews. So be very, very careful. Vet your resources either independently, on your own or go to your literature and your evidence-based practices, and our mastermind would love to be a support for you as you seek to elevate your school counseling work with evidence-based interventions. Have a great week. Don't forget to go rate the podcast for us, please, and I'll be back soon with another episode. In the meantime, take care, my friend.