Sites that meet the "Crawlable Conditions" below are crawlable. 

The information listed in the "Cases that cannot be handled" is only an example.


■Crawlable conditions are as follows

If a site does not meet the following conditions, it may be difficult to crawl the site.

・ Multiple logins are possible.

・ Sites with Basic Authentication, Digest Authentication, and Form Authentication (authentication method using ID and password from website forms)

・ The site must be accessible via the Internet.

・ Access should be possible with a browser (latest version of Chrome) with no special settings required for access.

・ No specific order of access or input of specific values is required.

・ The form must be in Japanese or English.


■Typical examples of cases where both automatic and manual crawling cannot be handled are as follows

・ Sites where multiple logins are not possible.  (sites where sessions are won first and double logins are prohibited)

・ Sites requiring multi-factor authentication. 

・ Sites with browsers other than Chrome. 

・ Sites that require authentication other than Basic Authentication or Form Authentication. 

・ Sites that cannot be accessed from the Internet.  (e.g., intranets and private networks)

・ Sites that require authentication that requires human perception (e.g., capture authentication (CAPTCHA)).

・ Sites that require WebSocket communication. 

・ Sites that are inaccessible with TLS 1.2 or higher. 

・ Cases where the URL of the site becomes invalid once it is processed, such as campaign sites. 

・ Sites with a content size of 8 MB or more per page. 


■Typical examples of cases that cannot be handled by automatic crawling and can be handled by manual crawling are as follows

・Requires the entry of a specific value, such as the entry of a specific agency code, in an insurance system, for example. (*Note1)

・ The input of a specific range of amounts (numerical values) is required for securities systems, auctions, etc. (*Note1)

・ In financial systems, for example, the terms and conditions document is displayed in a frame and all the contents need to be scrolled. (*Note1)

・ Downloading PDF files of terms and conditions is required for financial systems, etc. (*Note1)

・ The input of non-duplicated values is required for continuous access in the new registration function, etc. (*Note1)

・ Workflow functions, for example, require multiple accounts to reach the page. (*Note1)

・ The number of inventory items, etc., has a fixed data registration limit and a fixed access limit. (*Note1)

・ Statistical systems, search functions, etc., require complex combinations of check boxes, radio buttons, etc. (*Note1)

・ When the file upload function has restrictions on file size or file extension (*Note2)


(*Note1)

Crawling may be possible by changing the value to be sent from the Form Input Value Change and Re-crawl (snail icon) of the crawl results.


(*Note2)

AeyeScan uploads and transitions image files if the file upload function is available during automatic patrol. Therefore, if there are restrictions on file size or file extension, manual patrol is the only way to handle this.


Please go to Scan List > Hamburger Menu > Crawl Results > Page List.