It's likely that if you've ever worked in SEO, you've experienced many times when you felt at Google's mercy. It almost seems as if Google takes some sort of sadistic pleasure in relegating your website to the lowest reaches of its search results, despite the fact that you are certain you've followed all of Google's instructions and gone through them with a fine-tooth comb.
Let's beginning by making one thing straight: Google wants you to be successful as long as you put in the effort. This is evident in the way it increases the transparency of its internal workings, especially when it comes to how it crawls and indexes websites. The URL Inspection Tool, which essentially dissects Google's indexing standards and aids you in identifying any flaws that might be hurting your website's search engine visibility, was developed by Google in 2018.
The fact that Google's bots are always out there scouring the largest portions of the internet sets it apart from other search engines, which it has done since the beginning of time. After a website is crawled, it is added to Google's "index," an ever-growing library. To consistently deliver the most pertinent and high-quality web content as easily and rapidly as possible, our index is constantly updating and improving. Indexing is the process through which Google crawls websites, extracts all the important data, and stores it in its library.
Sounds very simple, doesn't it? Well, not quite. Making it as simple as possible for Google to efficiently crawl and index websites is an essential component of SEO. This includes figuring out a variety of typical technological problems that serve as obstacles for all of those little bots scuttling about the more intricate areas of your website.
This phrase is used to describe connections that are essentially dead ends. This can be due to wrongly entering a URL, an HTML code error, or the fact that the page is no longer accessible.
This is the page's allocated URL, which serves as a signal to Google that it should be given precedence over other factors (e.g. HTTP vs. HTTPS). By doing this, the dangers of duplicate content are reduced. Duplicate content occurs when two pages are so similar that Google is unable to decide which one to prioritise in search results.
To prevent webspam or duplicate content, these tags inform Google that a page shouldn't be indexed. Even though these are frequently included on purpose, you should check for them if Google isn't indexing your page.
Broken links are usually fixed using a 301 redirect, although occasionally a page may be redirected to another and then link back to the original URL. As a result, a page keeps redirecting in a closed loop.
This file is added to a website so that it can essentially serve as a reference point for Google's bots. By doing this, you can be certain that only pages that are visible to users of your website will be crawled and indexed. Check your robots.txt file to make sure that no significant pages have unintentionally managed to make it there.
As you are probably aware, Google uses mobile-first indexing to accommodate the extreme shift in web usage from desktop to mobile. Although this has long been the case, certain websites have not yet caught up. Google isn't exactly compelled to index or rank your page highly if it loads slowly or doesn't have a responsive design for mobile consumers.
Using Google's URL Inspection Tool, you should be able to determine which of the problems identified are hurting your website. This 2018 tool is a great illustration of Google's efforts to make its operations more transparent. The components of the URL Inspection Tool are broken down in this part, along with how they function as a whole to guarantee that your pages are properly indexed.
If your URL is marked as on Google, you have been indexed! This is one of Google's five indexability categories; the others are listed below:
URL is on Google, but has issues: Technically, the URL has been indexed, however the improvements that are supposed to make the URL more visible to search engines have issues. The website might not be considered mobile-friendly, or the structured data—often referred to as the "language" of browsers and search engines—might be wrong.
URL is not on Google: Google has paid attention to warnings that some pages shouldn't be crawled, such as those that are listed in a website's robots.txt file. Less obvious signs that a page shouldn't be indexed may also exist. It might be an orphan page, for instance, which means that no other internal links on the website point to it.
URL is not on Google (Indexing): One of the indexability problems described earlier in this article, such as a broken link or a no-index tag, has been discovered by Google.
URL is an alternate version: When Google recognises your website as being mobile-first, it will mark that the URL you have supplied is either the AMP version or possibly the desktop version of a URL.
You may learn about the three primary methods Google uses to crawl your website and assess its overall quality and user experience in this section of Google Search Console. The 'HTML' tab, which first displays the displayed page code, enables you to identify problems such incorrect canonical URLs or misplaced structured data. Second, you may visualise any potential issues with mobile usability by looking at a screenshot of the crawled webpage as it would appear on a smartphone. Last but not least, a convenient "More Info" section enables you to delve into the finer points of a webpage's technological architecture, such as the precise sort of material that has been crawled (often text and HTML) and HTTP status codes.
Have you updated a URL that you want Google to crawl recently? The Request Indexing tool tells Google to crawl and re-index your page if you've added fresh material, possibly optimised it for a high-priority keyword, or repaired any urgent technical problems. Although submitting URLs for crawling is a straightforward process, there are a few important factors to take into account:
Before you start acting hastily, you should be aware that repeatedly requesting that the same page be indexed will not result in a quicker re-indexing of your URL by Google.
A maximum of 10–12 URLs from the same domain may be submitted for indexing per day.
No of how many times a page is asked to be indexed, Google won't ignore problems like missing index tags or improperly used canonical tags.
If you're curious in how Google was able to crawl and index a webpage, you may read more about it in this section. First, the Discovery tab shows how Google's dependable bots first came across the URL, such as whether it was through a recommendation from another page or by crawling a sitemap. Second, the Crawl section provides information on the most recent successful crawl of a page, including the crawl date and the user-agent (e.g. a smartphone or desktop). The "user-declared canonical," which is typically supplied using a canonical tag, is distinguished from the canonical URL that Google was able to determine in the Indexing section.
As SEO specialists, this is where we get to demonstrate the value of all the small flourishes that contribute to a website's maximum crawlability and search engine visibility. This portion of the URL Inspection Tool, for instance, pulls in any structured data you have supplied, underlines if your website is mobile-friendly, and displays whether or not Google has indexed important elements of your website's brand identity, such as the logo or any user reviews.
This tool enables you to receive a real-time update on the indexability status of a URL by asking Google to retrieve the most recent version of the URL. It is frequently used to verify technical fixes so users can determine whether their efforts directly affected whether a page is indexable or not.
Google is frequently extremely busy, as you might probably guess! If there are no problems that might limit the website's ability to be indexed, it could take anything from a few days to a few weeks for a page to be indexed. Although we advise checking frequently, it is not worthwhile to submit URLs to be re-crawled until an indexability issue has been resolved.
Diginow is a top London SEO agency, with a team of technical specialists who specialize in supporting clients in achieving long-term success in organic search.
Do you need assistance increasing your organic search traffic, rankings, and conversions? To learn more, Get in touch with our skilled professionals.
Whether it’s a new venture or existing brand. Let us know your ideas below using our project planner and we’ll see how we can help.