In Episode 23 of the Trades Secret Podcast hosts Devon Hayes and Amanda Joyce help you identify what web crawlers look for on a website and offer tips to optimize your own website for peak performance:
- How web crawlers work
- How Google reads your site
- What quick changes you can make to optimize your site
- A real time walk through of a website to point out the good and the bad
Send us your questions, comments, feedback, praise! We can’t wait to hear from you!
Transcription:
Devon Hayes: Hi, I’m Devon Hayes.
Amanda Joyce: And I’m Amanda Joyce. And today’s topic is Crawler Confusion, How Whack Site Structure Can Kill Your Search Rankings. And here’s why you should care. If search engine crawlers are confused by your site or they can’t figure out what it is that you do within half a second, your rankings will suffer.
Okay. Another great topic. This is a topic that we cover a lot of times in discovery calls with people, and we’ve talked about it at a lot of conferences and things. And Devon, in all of her brilliance always does such a great job of explaining this. She’s obviously the brains behind all of our SEO strategy here at Elevation Marketing, and I love hearing her go through this exercise. So I think it’s going to be really good for all of our listeners to really get a good idea about what exactly Google sees when they hit their website.
Devon Hayes: Yes. Absolutely. It is actually really interesting to understand how search engines do read your website. So we actually love to go over this with clients and it helps them kind of understand some of the magic behind SEO. But before we kind of dive in today, we are going to do a screen share, and we haven’t done that yet on the Trade Secrets Podcast.
So if you normally just listen to us, I’m going to try and be as descriptive as possible, but this might be one where you actually want to go to our YouTube channel and check it out, because then you’ll kind of be able to exercise these same practices on your website and take a look at some of the things that you either, you’re kind of checking the box or maybe you’re missing, and maybe that’s something that’s hindering your performance on Google and your rankings.
So when we type in a query to any search engine, there’s called a crawler, and the crawler kind of dives into its index, which is just basically a big database of stored information on every URL out there. And that stores information on files, images, media files, text, and it also stores the context, not just the content. That’s what Google’s AI crawlers can do. So it’s not just the exact verbiage on the page, it’s also the implied context that Google has kind of decided what your page is about through these 200 checkpoints.
And there you are, sitting and living in their database and that is for every single URL. So there’s your primary domain for your website, but then every single URL is also indexed and stored, unless you tell Google not to. And it also will arbitrarily stop storing that information or stop indexing a page. So that’s something to also keep up with that we’ve talked about on other podcasts.
So in understanding how Google is reading your site, you can better understand how to make some quick changes to your site structure to help the users, because ultimately that’s Google’s goal, having for us users to have a good experience when we land on your page. And maybe if your rankings aren’t doing great and you don’t really understand why, some of this information can help. So with that, am I getting too in the weeds here, Amanda?
Amanda Joyce: No. I think it’s perfect. I think that’s exactly, you teed it up well. So let’s take a look at a website and talk about if Google would swipe right or left and check out this website.
Devon Hayes: Okay. So let’s see here. Let me, and we found this website a long time ago. I think years ago. We put a proposal into them and they never picked us. And so, we always peek back at them to be like, “Did they go with another agency? What did they do?” So this is a roofing company out of Tennessee. Bluebird Roofing.
Devon Hayes: But let’s start on the site. So first and foremost, there is information stored on your website that Google, that we don’t see as users, but that crawlers see. And some of that is what’s called your meta descriptions and meta titles, which actually as users, we see those, but sometimes we don’t see the meta descriptions.
And then, that’s a whole other rabbit hole where I’m going to get caught in the weeds here. You can see the meta descriptions if Google decides to serve that on the search engine results page, but sometimes it chooses content that’s on the URL versus what we’ve asked it to store. What I meant to say was schema is something that we can’t see as users, but it’s sort of is an advanced SEO technique that tells a crawler what your website is about in the most concise way possible.
So pretending that your schema is fully optimized, now we’re just looking on page at your website. And this is what we kind of do when we’re on a discovery call and we’re looking through someone’s site. We just started the top like a crawler would and work its way down. So the first thing you see here is the URL. You have the lock right here so you know it’s a secure site. So that’s massively important. If you don’t have this, pause this and go buy an SSL certificate and secure your website. It’s just a trust factor and part of the core vitals and part of your rankings in Google.
The next thing we do, we’re looking at the URL. They’ve got a strong URL, because they have the name of the service that they provide in their URL. Sometimes it’s really challenging when we have contractors that provide a… Hold on. Sorry. My dog is going fucking crazy right now. Oh god. Shit. Let me pause this. Hang on. Hang on. Will you put up your flag?
Devon Hayes: Okay. So the first thing you want to do is, you start at the top. If you see this little lock is closed, that means your site is secure. This is a great trust factor and a ranking factor. So if this is red or shows that it’s unlocked, you want to go out and right now drop what you’re doing, get an SSL certificate installed on your website.
The next thing we’re looking at is your URL. This one, this is great. It has the name of their primary service in the URL. This doesn’t always have to happen. If you think of Google, it doesn’t say googlesearchengine.com, but it’s, especially if you have a younger domain and you’re a newer business, you really want to have your primary service in the URL. Sometimes switching from even just construction to roofing, if that’s your primary, if you do both, or if you do gas fitting and plumbing, but plumbing is your primary service, you would want to have that in the URL.
So if there’s a way to get your primary service in the URL, that’s a great checkpoint. Not always necessary, but something else that we look at what that could potentially be hindering your positioning. So next thing we do, if you’re a crawler, this is you’re starting from the top and you’re kind of moving your way down. So we just see, this just says… Oh, it takes us to their review page. All right. So already we have a link to their page. It just says, if it’s reading the text, we see 229 our reviews. Sees images for stars, and then 5.0.
Then, we go over here, and this is just a logo. This isn’t text. So Google is reading the image. It can pick up text on an image, but it takes more time than if it were just text. So we’re taking up a lot of time just to tell it Bluebird Roofing as an image. Then, we come over here, we’re reading the navigation, home, reviews, about us, our services, gallery, contact.
So far we haven’t actually told Google what we do, unless it took the time to read the image. So we come down here and we see this kind of video, but this, I can’t highlight it. This is also just text on an image. Down here, this is more just text on an image. And so far, we haven’t actually even been told what action this roofer would like us to take. There’s no phone number up here, there’s no button call out, there’s no chatbot, there’s no pop out form. So far, they’re telling us about themselves. They’re a Bluebird. They’re even telling us where we are.
Amanda Joyce: Yeah. Devon, real quickly. One of the things that I know people always ask us about when you’re going through this exercise, in the navigation, it’s really common to just have our services there. I know that’s one of the first things you really kind of coach people away from. Will you talk about, because a lot of people are like, “Oh, if you click on our services and then look down, there you can see what we do.” Or in this case you can’t. But a lot of times it’ll say our services underneath it [inaudible 00:14:10]
Devon Hayes: Yeah. For sure. So if you’re a crawler and you’re reading this, you’re really missing an opportunity to tell them, to tell it what your services are. And if you think about it, your navigation is on every page of your website. So you’re repeatedly from the top telling a crawler, roofing services, roofing services, roofing services versus here it says our services.
This could be literally any service. This could be, I don’t know, a law practice, this could be an auto body shop. This could be anything really in terms of the text that the crawler has read. So yeah, thank you for calling that out. All of these are so generic, nothing specifically speaks to the service that these guys provide.
So again, and then this is just a giant image, and I think it was even like a video. Yeah. It was like a slow animation, which video, while we love, it’s probably a time suck. I haven’t even done a speed test on the site, but we’re just going to focus kind of on page for now. So you have a nice video. I wouldn’t… I don’t know. It does nothing for the customer. So far, we’re not telling anybody what kind of service they provide. We’re not telling them what kind of outcome or benefits come with choosing this roofing company.
A lot of that, while it has to do with more marketing language and more CRO or conversion rate optimization, this is kind of old-timey, narcissistic, me, me, me, me. This is me, this is my logo, this is my website, this is my address, but so far, we don’t really know why we would choose this roofing company. We don’t know what outcome we can expect by choosing them, the benefits of what to expect by going with them.
And furthermore, so far they don’t really care how we get in touch with them. I mean this phone number isn’t even HTML coded. This is a missed opportunity for local SEO to have a map embed right here that links out to their map. And again, user experience, if I’m on mobile, which let me shrink my screen down. Now we’re on mobile. At least, we’ve got the clickable phone number on up top on mobile. And it gets rid of I guess the address and the header, or sorry, and this hero image. Okay. So we’ll go back to desktop view.
And now we’re down here and we finally get the first text. See, none of this, I couldn’t highlight. So the first text we’re getting… Oops. This is the first opportunity for this website to tell Google what it’s about. I haven’t taken a look at their schema markup. But in any case, great company. Finally, this review says the name of the company, the text is on here. Lots of reviews that they’re pushing here.
And then, we finally get to the H1. H1’s are typically above the fold. Above the fold means before you scroll, before you hit the scroll. And this one’s kind of down below here. So we finally understand in very concise terms that they’re a roofing contractor in Mount Juliet, Tennessee. Typically, this H1, your first header tells Google, this is the most important thing about your website or what they can expect to find on this page of your website.
So it’s great. It’s a great SEO headline, but it doesn’t tell the customer what they’re going to do for them. This is a very SEO focused title. And sometimes when you’re starting out with a website, you do have to have that SEO focus and kind of avoid some of the flowery marketing language. But there should be… You have to remember that the goal is conversions. The goal of SEO is conversions. So you should find an agency where you can have that happy marriage and you can connect the two where you can combine what we call a benefit outcome statement with some of this roofing contractor in Mount Juliet, Tennessee.
This could even be a subhead, in any case, so we finally are below the fold where we see what this is about. We get a little bit of text. Best practices, you should have at least, at least, at least 600 words on your homepage. Google’s gotten a little bit more lax with that. And you’ll see homepages that they rank really well and they only have 400 characters on the site. But this is really your best opportunity to tell a customer what you’re about because you don’t know if they’re going to click through to the next page, especially a site like this where you’re not really guiding them, you’re not really telling them what you want to do next.
So I would imagine their bounce rate is probably pretty high. And then, we get down here. Here we start to say lightning fast response. That’s a benefit to the customer. But you’re kind of way down here. Insurance claims experts. So I’m sure this is another service, but there’s no further information. These aren’t buttons, even though they look like it. That’s poor UX.
And then, exceptional work. Again, these credentials are great. They help to build trust, but there’s a missed opportunity by not linking out, having an external link to this trusted manufacturer. So that’s kind of a miss there too. There’s internal and external links that provide some value. It’s kind of a mystery how much weight is placed on them, but having external links to trusted sources helps your positioning and your ranking as well. So you want to make sure you’re ranking out to any of these trust logos that you might have on your website.
And then, coming down here, request your roofing and exterior inspection. We finally get a call to action, but they just want you to call them. They don’t… We have a form down here, a missed opportunity here for a request an estimate. A lot of roofers provide like a free roof inspection, requesting an estimate, I know estimates are a lot more, they’re kind of a little bit more intensive, so maybe that’s why they went with that language, but it’s still, I don’t know, it’s just kind of weak in terms of a call to action.
And then, over here, I’m glad to see they have a map embed on their homepage. I would be curious to see if they’ve got kind of the code. There’s some fun work you can do on the back end with that. But then, anyways, we come down here to the bottom. We have the address here, but again, this is the footer. So your footer is carried through to each page of your website.
So this should actually, if it were optimized properly, link out to their maps listing or even their Google business profile listing. So that’s kind of just a big miss here. And it’s great that they have some other internal links listed within the footer, and their office hours. And then, some external links here. So sometimes those are missed, but that’s good that they have those.
I would hope these hours match whatever you have on your Google business profile, these hours should match that. Let’s see here. If we click through, they’re really, the reviews are kind of the most prominent thing throughout the site, which is understandable. They have what looks like 229 of them. But this, it’s interesting that you can’t click out to look at, to look at their reviews on their Google business profile.
And then, they have a lead form here to, I guess to submit reviews right here. And actually, I’m going to go back to that. If we look at this title, Reviews of Bluebird Roofing. You could optimize this title a bit more, right? I mean you could say roofing contractor reviews in Mount Juliet, Tennessee. And then, have a subhead saying, Bluebird Roofing. So titles like this assume that customers know your brand.
So when you go more broad and generic, then you are targeting a larger audience, because they’re wanting to investigate reviews of roofing contractors in the area versus customers who maybe are brand aware. So this is making a big assumption and kind of limiting yourself in terms of search for who you could potentially reach. And then, about us, I’m a sucker for a good team photo, so that’s nice. But again, these elements are not clickable and they’re just part of an image.
And the content, it needs to be 600 pictures. I love that they’re sharing photos of their sales team. But again, the call to action comes way, way, way, way, way down the page. Quickly our services, it’s just a big missed opportunity to not have the header above the fold. I have to scroll down before I find out that their headline of this page is roofing services from Bluebird Roofing.
So I’m a crawler, I’m still seeing our services in the URL, so it’s another missed opportunity to say roof replacement or whatever primary services or roofing services or roofing company. Each market is different. But the URL, again, thinking that a crawler starts at the top of a website and moves its way down, it’s kind of just a missed opportunity there. And then, again, not having a headline above the fold, that’s a big miss. And then, just how the shortage of content.
Amanda Joyce: Yeah. And it’s really surprising to see all of the services just there kind of lumped into one page. We don’t often see that.
Devon Hayes: Yeah. Everything just… And you can see an overview page and I get that. But if you’re a crawler, you just see roofing services, you don’t… But each focus key or each service should have its own page. So there should be a roof installation page, there should be a roof repair page. I love that they, like installation, they’ve pulled as different from replacement. That’s interesting.
And then, storm damage. But they don’t tell us whether this is commercial or residential or both. They don’t pull out the services. And there’s different keywords and different keyword focuses for those particular things. So when you lump it all into one page, what do you want this page to rank for? What are you trying to rank for? It’s confusing.
And then, a gallery is always great for a user to take a look at. I like a gallery that tells me more about the shingle that I’m looking at. In local search there’s a missed opportunity here because you’re not saying where this house is. It’s a beautiful house and you’re not saying the type of shingle. And you could tell a little bit about the project. So there’s just some missed opportunities in terms of local SEO on this page. They’re just like checking a box, a little bit of their minimum.
And then, we click through to the contact page. And just the contact page doesn’t have to be anything crazy, but there’s no language really telling us about what their unique selling proposition is or USP anywhere throughout the site. They’re not talking about how maybe they have a simplified process, a manufacturer’s warranty, a labor warranty, a referral rewards program. They don’t even really brag about their reviews. They just keep pushing you off to their reviews page. They don’t highlight, maybe in their reviews a common theme is customer service or great communication. They don’t even tell you anything great or why to choose them.
So in terms of search, they’re dismissing the box on content, headlines, calls to action, their navigation, their URL structure, their footer optimization, header optimization. On desktop, you should see a phone number up top, even though I’m not going to sit here and FaceTime you from my desktop, it doesn’t mean that a crawler isn’t reading that and it can’t grab that local area code and associate your roofing company with the local area to help you get ranking in that map pack at the top.
And I like they have cars wrapped. I mean they’re doing the stuff, but it’s a digital miss. And I think these are great contractors. They do really quality work. They do great job site marketing. These guys are really doing their social media. I love them on social. And I literally, they’re not a client, we just met them. A guy who’s been long gone from that company for a long time now. And so, we always just peek in on them. Anytime we don’t win a contract, we’re like-
Amanda Joyce: What are they doing? Are doing all right?
Devon Hayes: I mean they’re doing a lot of stuff really great, but just their website is not working for them. And this is your asset that can be working for you 24/7 when you’re sleeping and you’re not doing anything. There’s just so many like SEO layups on this site where if they understood how a crawler was starting from the top and going to the bottom, that helps you look at your website through a different lens. Instead of like, “Oh, I know I need an update.”
But even if they did changed it to roofing services versus our services, even these small changes will help you and start to help you gain at least some market share depending on… I don’t know how optimized our competitors are, I haven’t even looked at that but just-
Amanda Joyce: Absolutely. And I think this is just really great feedback for people listening to just go through the same exercise on your website. Take a look at it, think about the way Devon was just looking at this website. Take that same lens and just go assess maybe some of these misses that she’s pointed out are misses on your website that you could easily tweak on your own or ask your agency to make improvements.
Devon Hayes: Yes. That is some great advice. And yeah, I didn’t comb over this ahead of time. We just kind of picked them out, because we’re like, “Oh yeah, whatever happened to that company? We should at least go through it.” But we didn’t even look at their off page stuff and there’s other things there. But the focus of this podcast was just to share with you how a crawler is reading your website from the top down, from the backend to the front end, and things that you can do and these slight tweaks that you can make without throwing a whole bunch of money at it. And just make those quick wins, kind of website layups for the user experience on your site and for crawlers.
Amanda Joyce: Brilliant.
Devon Hayes: Well, I got a little long-winded there, but thank you all for listening. If you made it through this far, you’re the best. And we really appreciate your support. Let us know what questions you have. What did we miss? Do you want us to do your website on one of these things and just kind of record it? And we’d be happy to talk you through it. And I don’t know, that might be a fun one too, to have a recurring podcast where we kind of just have someone on the line with us and go over their site and maybe some things that we’re seeing, but anyways.
Amanda Joyce: That would be brilliant. All right. Awesome. Well, great job partner.
Devon Hayes: Thank you. Thanks, everyone. And we’ll see you next time.
