137
Free
Add to project

Product support

Get help in the plugin support forum.

Categories

Renick.Spider

Just a simple OctoberCMS plugin for crawling your local installation. Laravel 10+ is required.

  • [x] crawl all pages to warmup your cache and resizer images
  • [x] find 404 not found errors in log file
  • [x] OctoberCMS MultiSite support
  • [x] Reset all caches before crawling
  • [x] Setup Spider as cronjob to re-create the caches frequently
  • [x] heavily customizable through env params
  • [x] new backend ui for displaying results

What can the spider plugin do useful?

We have several OctoberCMS sites which have several caches in place. Like Redis and Quicksilver. Also we heavily resize images for blog posts, products and many more. All of these sites have the Spider installed, and once a day all content gets crawled.

At first, we can see all 404 not found errors in the log. Second it clears and re-generates all caches once a day, so we have quick loading times and prevent outdated content.

Feel free to ask, raise ideas, mind bugs or contribute on the Github repository. Please create a new issue or pull request.

To install this plugin, you can just run the following command:

php artisan plugin:install Renick.Spider

We have two modes the plug-in can be used at the moment. We try to keep our plug-ins really basic. You can use the command-line tool, or the cronjob. The plug-in creates a spider.log file containing all the successful and failed crawls. We also plan to create an backend ui for it. 

php artisan spider:crawl

You can set up the behavior of this plug-in heavily with environment variables. Here are the options:

SPIDER_CRON_ENABLED=true

SPIDER_LOG_MODE=single|append|failed
SPIDER_LOG_FILE=logs/spider.log

SPIDER_CONCURRENCY=10
SPIDER_CACHE_CLEAR=true
SPIDER_CRON_SCHEDULE="1 1 * * *"
SPIDER_CACHE_CLEAR_RESIZER=false
SPIDER_USER_AGENT="Mozilla/5.0 (compatible; Renick.Spider/1.0;)"

For Developers

This plugin also provides some APIs and a library for crawling the site. Feel free to get inspired by the command-line script.

1.1.2

Make the backend overview page more robust

Apr 01, 2024
1.1.1

Fix a Type-Error in the backend overview page

Apr 01, 2024
1.1.0

Added backend overview page

Mar 31, 2024
1.0.1

Minor bug fixes

Mar 31, 2024
1.0.0

Basic crawler and cache warming functionalities

Mar 10, 2024