Screaming Frog is a UK marketing agency. The company has two very useful tools for those working with Digital Marketing: SEO Spider and Log File Analyzer. In the following, we will better understand how they serve your strategies and what their key features are.
Search Engine Optimization enlists the help of tools. They are for SEO professionals to capture valuable data and automate actions that would not be possible manually. And Screaming Frog has created two ideal tools for this.
SEO Spider and Log Fyle Analyzer are the two company tools that allow to perform complete complete SEO technical audits. It is with this information gathered that you can begin to outline optimization strategies , knowing what is working, what needs to be fixed, and what should be improved.
That’s better than having features that do full scans and bring reports with processed information to you, right?
So, let’s get to know Screaming Frog now, one of the names best known by SEO professionals worldwide. Follow this text and understand:
- What is Screaming Frog?
- How can Screaming Frog help with SEO strategies?
- How does SEO Spider, Screaming Frog’s main tool, work?
- How does the Log File Analyzer work?
What is the Screaming Frog?
Screaming Frog is a marketing agency in the United Kingdom. Founded in 2011, the company was born from its best known tool: SEO Spider. It began developing in 2010 to collect data, identify technical issues on websites, and support SEO recommendations.
After the founding of the company, SEO Spider was released to the public in 2011, with a free version and paid plans. From there, the tool began to become popular among the SEO community and became indispensable for many professionals.
Over the years, it has gained new updates. It is currently in version 12.0, released October 22, 2019.
In 2016, Screaming Frog released yet another tool widely used by SEO professionals: Log File Analyzer. It is currently at version 3.0 and allows for technical analysis of page crawl and the behavior of the Google robot on the site.
Next, let’s explain further how each of these tools can help with your SEO strategies.
Download this post by entering your email below
Powered by Rock Convert
How can Screaming Frog help with SEO strategies?
Screaming Frog has become one of the most recognized names in the SEO universe. The tools the agency has created help professionals with valuable data about websites and digital marketing strategies Digital Marketing .
SEO Spider is your best known tool . It is an ideal website crawler to perform technical audits and support SEO decisions. Using this tool you can:
- find broken links (404 error) and server errors;
- audit redirects;
- analyze titles and meta descriptions (length, absences, duplications);
- identify duplicate content;
- identify pages blocked by robots.txt and “noindex”, “nofollow” and canonical tags;
- generate xml sitemaps ;
- view site architecture in tree charts and interactive diagrams;
- integrate with google analytics and search console (paid version);
- validate structured data (paid version).
Log File Analyzer can also be very useful for your strategies, but it’s a bit more technical and not as popular as SEO Spider. With it you can:
- identify which urls the robot can track and which cannot;
- find out the tracking frequency by robots;
- discover broken links and crawl errors;
- analyze the most and least traced urls;
- identify the heaviest and slowest pages.
How does SEO Spider, Screaming Frog’s main tool, work?
Screaming Frog SEO Spider is a website crawler that simulates Google’s robot action in crawling pages.
With this tool, you can identify how the search engine is viewing your pages and what problems they are encountering so that you can fix them and improve your SEO results.
To use SEO Spider, you need to download a program to your computer – it’s different from other tools, such as SEMrush and SimilarWeb , which run in the cloud. The software works on Windows, Mac and Ubuntu.
The program can be downloaded for free, but has limitations: some features are not available, you can only analyze 500 URLs and have no technical support.
In the paid version, you can track as many URLs as you like, technical support is included, and advanced functionality such as structured data validation and Accelerated Mobile Pages (AMPs).
If you’re interested in the paid version, there are plans starting at £ 149 a year. This covers up to 4 licenses, and there are discounts if you purchase more licenses for your company.
Screaming Frog SEO Spider Tutorial: How to get started with the tool?
Once downloaded and installed on your computer, the program is ready to use. This is the first view you will get of him:
To get started, then, you must enter the site you want to analyze in the field above, next to the name “Screaming Frog”. As an example, let’s crawl Screaming Frog’s own website.
In this field, you can also enter some site specific directory, such as the blog. In that case, we could enter “https://www.screamingfrog.co.uk/blog/” and we would review all the pages included within the blog.
Note, however, that the free version only crawls 500 pages . If you want to analyze more pages (up to 10,000!), You can insert them in List mode, which is available in the top menu.
In this case, it is not the tool crawler that fetches URLs – it is you who must submit them manually via a file or sitemap.
With this search, you will already receive a report of all crawled pages , with which you can perform a full site audit.
In this report you can see information such as page title, meta description, heading tags, canonical tags, number of internal and external links, response time, last updated, and other data. If you want to save this data or take it to another program, you can export the report to CSV or XLS.
Browse the tabs (Internal, External, Protocol, etc.) and use the filters to find exactly the information you want. You can also customize the tabs to work with just what matters at the moment.
In the right column there is also a general report on crawled pages, with the total number of URLs found, the number of pages that are too long title or have no meta description, the number of images without [19459059 ] alt text , among other information.
If you click on any item in this report, you can see the detailed information – which are the pages with duplicate titles, for example – and already make the necessary corrections.
Useful Features of Screaming Frog SEO Spider
With these reports alone, you can already gather a wealth of information about your pages and define where your SEO actions should begin. But let’s highlight some interesting features of the tool that can further help your work.
In a site or domain migration, it is common to use 301 redirect so that users do not encounter a 404 error when visiting an old page. This feature is also widely used when the site changes its URL structure.
However, it is also common for several errors to occur in these redirects, as they are often made in bulk.
Therefore, SEO Spider can help check if everything is right with redirects . To do this, you must use the program in List mode, as shown above. Upload the old URLs and check the “Always Follow Redirects” field in the settings.
Then start tracking. When all URLs are crawled, go to the Reports menu and click Redirects> All Redirects. This will give you a complete report of all redirected URLs as well as page status.
Edit titles and meta descriptions
Those who work with content production know the importance of the title and the meta description of a page. These elements not only help Google understand your content, but also help you decide if you click the link in the search results.
Screaming Frog SEO Spider knows the importance of this and has developed a feature for editing the page title and description directly in the program .
To do this, click on any URL that is in the crawled pages report. In the bottom window you will see the information of the selected page. Between the lower window tabs, look for “SERP Snippet”. This will allow you to see the page snippet exactly as it appears in Google search results.
A little further down you will see the edit fields for the title and meta description. As you edit these fields, the program tells you how many characters you can still use.
Once you’ve made all the edits you want, simply export the Page Titles report and upload it to your site. That makes it a lot easier, doesn’t it?
Also note that you can select the type of device to view. By the way, did you know that the search results snippet appears differently on desktop and mobile ? See this example below:
If you don’t have a sitemap yet, it’s time to create one for your site and submit it to Google. This file tells the search engine all pages of the site to crawl, as well as their frequency of updating and other important information.
Screaming Frog SEO Spider offers sitemap export functionality in XML format . After crawling the site pages, you must go to the Sitemaps menu and select XML Sitemap. You will then see the page below where you can select the fields you want to include in this file.
Then submit the sitemap to Google. This can be done via Google Search Console in the specific tool for this.
View Site Architecture
Want to know if your site is well organized, if page categorization makes sense, and if hierarchy is intuitive to the user? Viewing a tree-shaped site can help with this understanding .
First, crawl the URLs. Then, on the SEO Spider menu, go to Visualization and click Crawl Tree Graph. Then the tool will open a page like the one below, which shows the connections between the website URLs.
Did you notice that in this article we reviewed the Screaming Frog website without owning it? This means that you can crawl the URLs of any website – including your competitors !
We have already talked about the importance of keeping an eye on the competition . You need to know how competitors are organizing their digital strategies to realize flaws and successes and get inspired to improve their site. In addition, you also have a better perception of your market position – whether you are ahead of or behind them.
In Screaming Frog SEO Spider, you can analyze how the competitor organized the site, how it prepares titles and meta descriptions, whether it has pages in other languages, uses structured data, and other valuable information.
And you would never get all this data without a tool for it! So, take the opportunity to spy on the competition and gain insights into your web performance.
How does the Log File Analyzer work?
Now, let’s talk about another Screaming Frog tool. As its name proposes, the Log File Analyzer analyzes log files. These files carry data that shows exactly what happened when a search robot visited the site .
Whenever a site is requested by both a human visitor and a robot, a new line with the visit information is recorded in the logs. These entries are generated continuously and in just one second, multiple rows can be created.
However, these files are extremely heavy, technical, and difficult to understand for those with no programming experience. Therefore, the function of the tool is to make this task less laborious by processing data for the user and generating easier-to-understand reports .
In the Log File Analyzer, it is the user who must send the files to the program for analysis. To handle these files, it is ideal to have specialized personnel. In these links you will find tutorials for accessing the logs of Apache (Linux) , NGINX (Linux) and IIS (Windows) .
To use the program you also need to download from the site. It is available for Windows, Mac and Ubuntu.
The paid and free versions include the same features. But as a premium, you have no limit on analysis and design and still have technical support included. Paid plans start at £ 99 a year, with discounts starting at 4 licenses.
How to Use Screaming Frog Log File Analyzer
When you download Screaming Frog Log File Analyzer, you will come across a screen like this:
So, to start an analysis, you must upload your site logs. Log File Analyzer supports Apache, W3C Extended, and Amazon Elastic Load Balancing file formats.
For an example, let’s use the demo file that the tool offers. After uploading the files, you already see a screen with lots of information and graphics:
In the first window you see a summary of collected URLs and logged events per day, as well as server responses to requests.
In the second window there is a graph with the server responses. They are represented by the following codes:
- 2xx: success;
- 3xx: Redirect;
- 4xx: Customer Error;
- 5xx: Server error.
This way you can identify which pages are in error and why they might be able to fix them.
In the Events window you can see which robots have visited your site and how often they visit. Realize that it’s not just Googlebot that is in action – Bing and Baidu, for example, may be watching too!
Finally, in the last window is the number of URLs that were crawled by each robot. If you only want to view information from one of the agents, you can select only what matters.
You can still browse the tabs in the top menu (Overview, URLs, Response Codes, etc.). In the URLs tab, for example, you can find out interesting information.
Clicking this tab displays a Number of Events column, which refers to the number of events related to a given page. These events concern the number of times the robot has visited the site.
So with this column you can find out which pages are more or less crawled . This could be related to how often the URL is updated, but also how relevant Google is to that page. If the robot visits multiple times, that page is probably important to the searcher, and you should pay attention to it.
This same information can be found on the “Directories” tab. In this case, the number of events refers to the robot’s visits to site directories, such as the blog or page categories.
Anyway, these are the two tools Screaming Frog offers to the SEO market. The information they provide is valuable for auditing websites . But you need to know how to go beyond basic reporting and explore the plethora of data, know all features, access all tabs, and crosscheck information to inform your strategic decisions.
Now, read our article on making SEO reports from data collected in an audit to generate insights and results.