AddSearch Crawl – Documentation

Requirements

• ExpressionEngine 7.2 or later.
• PHP 8.1 or later with the curl and openssl extensions (standard on typical EE hosts). The installer refuses to proceed if either is missing.
• An AddSearch account with API access, and its Site Key and secret API Key (AddSearch dashboard → Setup → Keys and installation).
• Your site must be reachable by AddSearch's crawler at the URLs ExpressionEngine knows for your entries (see How the URL is chosen under Usage).

Installation

1. Unzip the release and copy the addsearch_crawl/ folder to system/user/addons/addsearch_crawl/.
2. In the Control Panel go to Developer → Add-ons and install AddSearch Crawl.
3. Open the add-on's Settings page, enter your Site Key and API Key, and click Test connection. Save.

Upgrading

Replace the addsearch_crawl/ folder and run the add-on's Update from the Add-ons page. The update registers the hooks this release listens to, removes the ones earlier releases used, widens the settings table where needed, and re-encrypts an API key saved by an earlier release into the current format.

Run the Update as soon as the files are in place. Until it has run, crawls are not reliable: depending on the release you upgraded from, saves may not be crawled at all, or an entry that has just been published may be crawled as an update. If the add-on's settings table is missing, the Settings page says so and asks for the Update.

Uninstalling

Uninstalling removes the add-on's settings and hooks. Nothing on AddSearch is touched.

Usage

Settings

The settings live under Developer → Add-ons → AddSearch Crawl → Settings and apply to the whole installation (one AddSearch index per site; under Multi-Site Manager the channel list names each channel's site). Only members ExpressionEngine lets into this add-on's control panel reach the page (see Security notes).

• AddSearch account — the Site Key (the public key of your index) and the API Key (your secret key). Both are needed before the settings can be saved. The API key is stored encrypted, shown only as its last four characters afterwards, and never sent to the browser again; leave the box blank on later saves to keep it. Test connection checks both keys with a read-only call before you save (a blank box tests the stored key), and Save itself refuses credentials AddSearch rejects (a save while AddSearch is unreachable still goes through, with a warning).
• Channels — All channels crawls entries from every channel. Turn it off to pick the channels whose entries should be crawled; with none picked, nothing is crawled and the page says so.
• When to crawl — Crawl on publish fires when an entry first becomes public: a new entry saved with the status Open, or an existing entry whose status changes to Open. Crawl on update fires when an entry that is already Open is saved again. Entries in any other status (Closed, Draft, a custom status) are never crawled.
• Logging — with Developer log on (the default after install), every crawl request, skip and error is written to Developer → Logs → Developer, prefixed [AddSearch Crawl]. Keys are never logged.

What triggers a crawl

The add-on listens to ExpressionEngine's entry-save hooks, so it sees every save that goes through the entry model: the publish form, bulk status changes, Channel Form, and add-ons such as MCP Server For EE. Autosaves and Live Preview do not save the entry and never trigger a crawl. A save that changes nothing fires no hooks.

The crawl request is made during the save and waits at most fifteen seconds for AddSearch; a slow or failed request is logged and never blocks the save.

How the URL is chosen

AddSearch is asked to fetch the entry's public URL, chosen in this order:

1. The site's page map (the Pages module and Structure both maintain it). Structure pages, Structure listing entries and Pages entries resolve to exactly the URI their templates link to, under the site URL. If Structure is set to add a trailing slash, the URL carries one.
2. Otherwise the channel's Channel URL (Developer → Channels → the channel → Settings) followed by the entry's url_title. {base_url} and {site_url} in the Channel URL are replaced by the site URL.

An entry that is in neither place is logged as an error ("Could not determine the URL") and skipped. Give its channel a Channel URL, or add the entry to Structure or Pages.

What AddSearch does with it

The add-on calls AddSearch's Crawl API (POST https://api.addsearch.com/v1/crawler with the action FETCH, your Site Key and the URL, authenticated with your Site Key and API Key). AddSearch answers "Scheduled" with the document id (the md5 of the URL) and fetches the page, normally within a minute. The add-on does not push content itself: whatever your page serves to AddSearch's crawler is what gets indexed, so a page behind a login or blocked by a firewall will not change in the index.

Closing or deleting an entry does not remove it from the index; AddSearch drops pages that answer 404 on its own schedule, or you can remove the document from the AddSearch dashboard.

Troubleshooting

Turn Developer log on, publish or update a test entry, and read Developer → Logs → Developer.

• "Not configured" — the Site Key or API Key is missing. Enter both on the Settings page.
• "Could not determine the URL of entry …" — see How the URL is chosen under Usage. Give the channel a Channel URL, or add the entry to Structure or Pages.
• "Channel … is not selected for crawling" / "Crawl on … is turned off" — the settings excluded this save; adjust Channels or When to crawl.
• "AddSearch answered HTTP 401/403" — the Site Key or API Key is wrong or was rotated. Re-enter them and use Test connection.
• "No response from api.addsearch.com" — the server could not reach AddSearch (DNS, firewall, outbound HTTPS blocked, or a missing CA bundle for curl). The curl error text is in the log line.
• "A key is stored but can no longer be read" on the Settings page — the site's encryption_key changed since the key was saved. Enter the API key again.
• "The add-on needs its update run" on the Settings page — the add-on's settings table does not exist. Run the add-on's Update from the Add-ons page.
• Nothing is logged at all — Developer log is off, the entry's save changed nothing, or the add-on's Update has not been run after upgrading.

Security notes

• The API key is encrypted with ExpressionEngine's own Encrypt service (keyed from the site's encryption_key), displayed masked, and never sent to the browser; every AddSearch request is made server-side.
• The Settings page and the Test connection endpoint are reachable only by members ExpressionEngine lets into this add-on's control panel (the Can access add-ons permission with the add-on assigned to the role, or a Super Admin). Test connection accepts POST only, with the control panel's CSRF token.
• Keys never appear in the Developer Log, the PHP error log, or any page.

Support

Turn on the Developer log, reproduce the issue, and send the relevant entries to [email protected].

Disclaimer

AddSearch Crawl is developed and maintained by Multimediality and is not affiliated with or endorsed by AddSearch. AddSearch is a trademark of its owner.

Changelog

• 1.2.2 License terms gain the term-and-renewal section every Multimediality add-on now shares; the changelog is grouped by kind of change. No code changes.
• 1.2.1 New icon and manual updates (upgrade guidance, a Troubleshooting line for the update notice, a License section). No code changes.
• 1.2.0 Settings page rebuilt with ExpressionEngine 7 toggles, a masked API key box and a clearer Test connection; crawl on publish now means "became public" (a status change to Open counts, a save with a backdated entry date no longer counts as an update); entry URLs come from the site's page map (Structure pages and listings, Pages), with Structure's trailing-slash setting honoured; the API key is encrypted with ExpressionEngine's own Encrypt service and re-encrypted on update; a crawl request waits at most fifteen seconds and never blocks the save; requirements declared to the Add-on Manager (EE 7.2+, PHP 8.1+); PHP 8.4 clean.
• 1.1.0 Modernised for ExpressionEngine 7 (namespace, setup metadata, compatibility).
• 1.0.0 First release: crawl on publish and update, channel selection, Structure-aware URLs, Test connection, developer logging.

The full changelog is at multimediality.com/add-ons/addsearch-crawl/changelog.

License

AddSearch Crawl is commercial software. One license covers a single production site plus one related staging or development site; each additional production site needs its own license. The full terms are on the license page.