Table of Contents
As websites and web applications grow larger, often crawling it to investigate technical SEO issues can be too much for your local computer to handle. James Finlayson came up with a great idea.
“Why don’t we just put screaming frog on the cloud?”
This guide will help you easily setup Screaming Frog on a virtual machine instance on Google Cloud Platform. We will also setup a remote desktop and a graphical interface in order to see the Screaming Frog GUI.
Let’s get to it.
Firstly, you’ll need to have registered for a Google Cloud Platform account, then create a Google Cloud Project. If you need help with the setup, you can refer to this guide.
Click on the hamburger menu in the top left, then find the Compute Engine and Click on VM Instances. If this is the first time that you’ve setup a compute instance for this Google Project, you’ll likely need to wait several minutes. Then click create!
Then click the create button at the bottom of the screen.
After your virtual machine has finished setting up, click on the SSH button. This will create a browser SSH session with the virtual machine. It will take 1 – 2.5 minutes to connect so please be patient.
Now that you are inside the VM, copy and paste the entire code block of commands in one go into your browser based terminal.
These scripts will perform the following actions:
☕ Copy And Paste All Of These Scripts In One Go, Then Grab A Coffee! ☕
# Upgrading the system; \ sudo apt-get update; \ yes y | sudo apt-get upgrade; \ yes y | sudo apt-get install wget; \ \ # Installing Google Chrome & Package Dependencies; \ yes y | wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb; \ yes y | sudo apt --fix-broken install ./google-chrome-stable_current_amd64.deb; \ yes y | sudo apt-get install cabextract xfonts-utils; \ yes y | wget http://ftp.de.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.6_all.deb; \ yes y | sudo dpkg -i ttf-mscorefonts-installer_3.6_all.deb; \ yes y | sudo apt-get install xdg-utils zenity libgconf-2-4 fonts-wqy-zenhei; \ \ # Installing Screaming Frog; \ yes y | wget https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_12.6_all.deb; \ yes y | sudo apt --fix-broken install ./screamingfrogseospider_12.6_all.deb; \ \ # Installing Ubuntu Desktop & The vncserver Package; \ yes y | sudo apt-get install gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal; \ yes y | sudo apt --fix-broken install tightvncserver;
Note: Remember to hit enter on the last tightvncserver installation too!
In your browser SSH enter the following two commands: (You will need to enter a password and skip the view only password here)
Basically what we’re doing with the above script is just creating a VNC server, then deleting it so that the start-up script is generated.
We’ll create custom start-up script using the Nano text editor:
Then add the lines at the bottom of your ~/.vnc/xstartup script, and hit CTRL + X to save it.
gnome-panel & gnome-settings-daemon
You will need to install the Google Cloud Software Development (SDK) Kit on your local machine. This provides you with a command line interface for accessing and controlling your Google Cloud Platform resources.
Make sure you have Python 3x installed on your computer. If you haven’t installed Python, follow this tutorial on how to install Python Anaconda.
cd ~ && nano .bash_profile
Then copy and paste following this code at the bottom and hit CTRL + X to and type y to save it.
(You will need to replace jamesaphoenix with your current username, which you easily find by doing: pwd).
Note: If you’re using a newer Mac, then you’ll be using a zsh shell. So please also export the same PATH in the following file:
cd ~; nano .zshrc
Now go to either your terminal or command prompt and type:
gcloud init gcloud login
In order to access the remote desktop which is currently running a vncserver on port 5901 by default, we will need to create an SSH tunnel.
Make sure to gather all of the relevant information:
gcloud compute ssh insertvirtualinstanncenamehere –project=insertprojectidhere –zone=insertzoneidhere –ssh-flag “-5901:localhost:5901″
For example my command looks like this:
gcloud compute ssh screaming-frog-crawler --project=savvy-motif-278215 --zone=us-central1-a --ssh-flag "-L 5901:localhost:5901"
Now that you have the following two things running:
We can setup a remote desktop session.
After connecting to your remote desktop, click on applications in the top left –> Internet, then you’ll see Screaming Frog!
Now you can run your Screaming Frog crawls in the Cloud instead of on your local computer!
Key Things To Remember:
- When you connect to port 8783 on your local system, that connection is tunneled through your ssh link to the ssh server on server.com. From there, the ssh server makes TCP connection to localhost port 8783 and relays data between the tunneled connection and the connection to target of the tunnel. - The "connection refused" error is coming from the ssh server on server.com when it tries to make the TCP connection to the target of the tunnel. "Connection refused" means that a connection attempt was rejected. The simplest explanation for the rejection is that, on server.com, there's nothing listening for connections on localhost port 8783. In other words, the server software that you were trying to tunnel to isn't running, or else it is running but it's not listening on that port.
connection refused on the VNC remote viewer : You will need to double check that you’re SSH tunnel and vncserver are the same (i.e 5901 and 5901). If you launch multiple vncservers, then the port number will increase by 1 (i.e. 5901 –> 5902 –> 5903). Therefore double check that you’re only running on vncserver!