Place Your Ad Here

XenForo Prevent crawlers from accessing deep forum pages

S

sajal

Guest
We have got a forum site that has a lot of threads/posts. Thus, if we go to the forum listing page, it has a "lot of" pages (In paging) of threads.

Now some crawlers access these deep forum pages, like 60000th page, 70000th page etc.

"GET /forum/latest-news/page-64099 HTTP/1.1"
"GET /forum/latest-news/page-65000 HTTP/1.1"

This creates a heavy load on the db because MySQL offset queries will be heavy to fetch this kind of content. And eventually, the site goes down in front-end for a...

Read more

Continue reading...
 
You must log in or register to reply here.

Similar threads

M
Ques/Help prevent the use of particular urls in the forum
Replies
0
Views
29
XenForo General Discussions
MediaFolSupport
M
O
Ques/Help Prevent bots from downloading resources
Replies
0
Views
28
XenForo General Discussions
Old Nick
O
A
XenForo Prevent fa icons from being duplicated
Replies
0
Views
33
XenForo Development & Modifications
acdcadicto
A
S
Ques/Help How to prevent auto links from appearing in notices?
Replies
0
Views
47
XenForo General Discussions
Sal Collaziano
S
T
XenForo How to prevent spam auto-registering account for XenForo 2.
Replies
0
Views
64
XenForo 2
Thangvip9x
T
Top Bottom
Back