These frameworks are JS driven ways of creating webpages, which allow for a greater amount of dynamic content. Although Google insists they are quite capable of indexing JS the reality is a little different. This largely comes down to the inflexibility of JS rendering compared to HTML and CSS. In other words, the likelihood of a page not being able to be read in JS by Google is higher than a page in HTML / CSS.
For example a large part of Wordtracker is built with Angular (although we have in the past used other libraries such as React and Coffee.JS for different elements). This allows us to have the the keyword tool as part of the same application as the homepage. This gives us far greater flexibility in how these different parts of Wordtracker interact.
Yes, but how does it actually work?
This is where the DOM comes in. The Document Object Model is the representation of what you see on page as it is displayed in your browser. This might be very different from the page code, such as the HTML/CSS behind the page, as scripts and other dynamically updating content affect it. The DOM sits in the middle between the page code and what gets displayed.
Within your browser you can use the ‘Inspect Element’ function (found in Chrome, Firefox, Edge) to see what the DOM looks like and change it yourself. Making changes here are reflected in the rendered version of the page you are shown.
The biggest fundamental here is whether the page code looks different to the rendered page. For an easy example here is the Wordtracker homepage code:
Comparing that to the actual page and using the Inspect Element feature to view the DOM shows that there is a lot of additional information here, which has to be coming from somewhere. We can also see that there is a .js script being run:
Google doesn’t use the latest, greatest version of Chrome to render pages, it actually uses Chrome 41. We’re currently on version 70.X so it’s a little behind the rest of us. This means there may be functionality supported in a modern browser that Google’s older rendering technology cannot decipher.
“Googlebot uses a web rendering service (WRS) that is based on Chrome 41 (M41). Generally, WRS supports the same web platform features and capabilities that the Chrome version it uses”
You can use the ‘Fetch as Google’ tool in your search console to see what the page looks like to Google and make sure that content is rendering as it should:
Critical rendering path
The critical rendering path is the way pages are loaded so that the user has the lowest possible page loading time. This means critical assets are loaded first, so content above the fold appears the most quickly.
Use tools such as Google Page Speed Insights to analyse the page and highlight errors such as this:
Google wants you only to load the vital stuff that’s needed for the above the fold content first, so the user ends up with the quickest possible experience.
5 seconds to go...
You also have to consider your ‘crawl budget’, the total amount of time and resource Google will allocate to crawling your site. All websites are not equal. The more popular your site the more generous Google will be with its crawl budget. If your server is slower to respond it will crawl fewer pages.
So, if you have lots of slow loading scripts and assets you will eat up your allocated budget more quickly resulting in fewer pages being crawled. Most likely the lower down or deeper pages, which may contain highly valuable long tail content, will be missed.
We all know the classic mistake of a website not getting indexed as someone has set the robots.txt to:
What is the DOM: