So Outside Technology Security CMS
AnQiCMS
Online Demonstration

Is there any advertisement on the page, and will it change whether the AI crawler can read the content?

📅 2026-11-01 👁️ 0

Is there any advertisement on the page, will it change? Can AI spiders read the content on this page? Yes. The new platform's default value treats 'whether the page displays advertisements' as one of the judgment criteria: spiders classified as training and proxy types are blocked by default on pages with advertisements, while search types continue to be allowed. That is to say, whether the same article can be read may be two results, and this level of judgment is not written in the robots file.

Why is there an advertisement in the judgment criteria?

This rule is to treat content as an asset for monetization: the page earns money through advertisements, while training type crawlers no longer bring back visitors after collecting materials in batches, the platform will block by default. The search type is different, the content it retrieves will ultimately point to the source, belonging to another type of use, so it is allowed by default. The determination occurs at the classification layer on the platform side, which does not use text matching, but rather a combination of page features and crawler classification conditions to decide.

The direct impact on the site owner is that the visibility of content is no longer solely determined by the rules written by themselves, but also depends on whether there is an ad slot in the page template. Sites using two sets of templates for document pages and article pages are most likely to appear in two months with 'some pages read and some not read.'

Default handling of three types of crawlers

Training material collection is a one-time large-scale grab, default block; agency class is the batch that retrieves content in real-time for users, also default block; retrieval class is responsible for organizing pages into referenceable indexes, default allow. The default values of the three types only take effect on new domains connected, and the currently effective values of existing domains need to be confirmed by themselves.

One point to note: There is no such rule in the robots file. Text guidance cannot express conditions like 'block with ads, allow without ads', and expressing this intention through robots will not take effect; the actual handling occurs at the classification level.

Will there be two sets of results within the same site?

Will, and this is the easiest to overlook. Common three types of inconsistencies: column templates with ads while special topic templates without; landing pages without ads, while the main content pages have; static pages and dynamic pages use different templates. In all three cases, the visibility of the same content to different crawlers is different, and the report will show that the request volume of a particular crawler suddenly stratifies.

When troubleshooting, group by template instead of by category. First, look at which types of pages have ad slots, and then look at the return status of these pages for the three types of crawlers respectively. Only when the two sets of information match can it be confirmed whether the default value is in effect.

To ensure that the content is read, check which part it should be matched with.

Firstly, it is the classification switch on the platform side, confirming the current effective values of training, proxy, and retrieval classes, as well as whether this default value has taken effect on your own domain. Secondly, the robots configuration should align with the disposal direction of the classification layer, avoiding the situation of allowing and blocking at the same time. Thirdly, the exit for large language models, the visibility criteria of llms.txt related to GEO should be confirmed separately, and it should be judged whether the content is organized and referenced as expected.

After making changes once, only move one place at a time, and then review the crawling records: by counting the request volume and return status of the three types of crawlers separately, you can confirm which layer the changes have fallen on.

Common questions

Ask: Can the training class read it by turning off the advertisement? Answer: It is valid in terms of direction, but you need to confirm whether this default value is effective on your own domain and what the current effective value of the classification layer is.

Ask: Can the robots file express this condition? Answer: No. The judgment of whether there is advertisement or not occurs at the platform classification layer, and the robots cannot express this condition.

Ask: How can we locate the inconsistency in visibility between the same site and different pages? Answer: Check the return status by grouping by template, count once for each template with and without advertisement, which is faster than grouping by column.

Related articles

Why would the proportion of different AI crawlers intercepted by the same site differ by a few points?

The same batch of sites intercepts AI crawlers from different manufacturers, and the intercept ratio differs by several points, mostly due to the rules being written according to the user group name, while each proxy's identifier is not just one: omitting one will result in a difference in the ratio. When checking, first distinguish between the three meanings of no file, empty file, and release, and then rewrite the rules by grouping for use, and finally verify the returned status of each one by using the site map entry and real crawling records instead of just looking at the configuration text.

2026-11-01

The proportion of AI crawler interception changed between two months, is it changed by the default value of the platform or the webmaster?

The same batch of domain names has changed the proportion of AI crawler interception in two months, most of which come from the launch of platform default values, not from webmasters changing configurations one by one. By looking at the direction of change, it can be distinguished: the displacement is concentrated on the training and proxy sides, while the search class has not moved much. When comparing, first lock the same caliber and the same industry level, and then confirm the configuration of robots files, platform classification switches, and in-site exit one by one, changing one and measuring one to attribute.

2026-11-01

After a website is attacked, it is necessary to restore the backup, should the vulnerability be patched first or the data be restored first?

The order of disposal is to first cut off the entry that can be reused again, install the affected programs and extensions, and then recover from a known good backup. Restoring data will bring back the time points with backdoors, and the repeated infections are mostly caused by this. After recovery, it is also necessary to check the accounts, scheduled tasks, and modified files, and confirm whether the deleted content still stays in the recycle bin.

2026-11-01

Should the new article be submitted to the search engine actively, and which companies does this protocol cover?

The proactive submission resolves the section of

2026-11-01

The new default intercept value only manages new domains, old domains need to be moved manually.

The new default blocking value on the platform only applies to newly connected domains, while the old domains already on the platform need to manually fetch and control this layer: first confirm the current effective values for training, proxy, and search categories separately, then merge them with the existing robots rules on the site, and finally verify with the real return status. The default value will not automatically overwrite existing configurations, assuming it has already taken effect is the most common judgment error.

2026-11-01

Why do we need to tighten the registration entry first and then patch the multi-site user registration open?

The reason why a multi-site with open user registration needs to tighten the registration entry first and then patch it is because the triggering conditions of this kind of risk are written together: registration is open, and ordinary users can create new sites. Closing the registration during the patch window period and restricting the permission to create new sites can directly cut off the attack entry; after patching, it is also necessary to check back on the sites that have been created and to synchronize the backup and permission review.

2026-11-01

Why is it necessary to limit server access to internal network addresses in the place where external links can be filled in the background?

The place where external links can be filled in the background needs to limit the server's access to the internal network address because the server will replace the user to retrieve this address: Once the link verification is bypassed, the request can be sent to the link-local address range, exposing the internal network service to external inputs. The verification should be done both at the time of writing and before the request, and then supplemented by the outgoing network limit, especially for entry points such as importing and remote image retrieval, which should be separately controlled.

2026-11-01

The alias of the article address can be listed externally, where should the list interface receive it?

The article address alias can be enumerated externally, usually requiring two conditions to be met: the alias can be guessed according to a regular pattern, and the list interface allows bulk pagination return. Therefore, the bottleneck is on the interface side rather than the page side, and it is necessary to limit the scale of single returns, set the exposure range according to the domain, and tighten the access to unpublic content. This article explains the conditions for enumeration, what list is exposed, and the location to be checked for internal interface.

2026-11-01