0

There's this question robots.txt vs language path prefix.

OP finds out that Drupal's default robots.txt doesn't actually cover his multi-language setup, which they configured to use path prefixes. An interesting question indeed. But is this really a Drupal-specific problem?

robots.txt is a customizable and generic file to communicate with crawlers, independently of what app, CMS or framework you are using. And multi-language is an optional feature of Drupal. Nothing the default robots.txt needs to cover from the start.

The same you wouldn't consider Drupal's default theme Bartik to cover all possible visual needs, you can't consider Drupal's default robots.txt to cover all possible path configurations. Talking about robots.txt in specific, there's http://www.robotstxt.org/robotstxt.html (and as SE network probably Webmasters) to get help configuring this generic file. It's also commented in the default robots.txt:

# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

robots.txt is SEO and I'd consider the mentioned question to be off-topic. Different it would be though if OP comes up with a question like "How to dynamically generate robots.txt to take language path prefixes into account" or similar, with a feature they built and a specific point where they are stuck. This can clearly considered to be on-topic.

3
  • 4
    Not going to answer, but I think that question is on-topic. As worded, it is about how multilingual works, and requires Drupal-specific knowledge to answer.
    – mpdonadio Mod
    Commented Jan 9, 2019 at 13:14
  • "The same you wouldn't consider Drupal's default theme Bartik to cover all possible visual needs" Actually Bartik does provide a functionally complete theme experience. Everything that core produces, every common option, is supported by Bartik, in that you get something functionally sufficient. Of course it does not match with the design spec of a specific site, but as said it works.
    – donquixote
    Commented Jan 9, 2019 at 14:17
  • @donquixote – Yeah, the comparison lags when thinking it out fully. But you get what it means. :)
    – leymannx
    Commented Jan 9, 2019 at 14:24

1 Answer 1

Reset to default
7

OP finds out that Drupal's default robots.txt doesn't actually cover his multi-language setup, which they configured to use path prefixes. An interesting question indeed. But is this really a Drupal-specific problem?

Very much so. In my opinion, there's no scenario where this question is off-topic here. It's not "his" multi-language setup, it's the multi-language setup, that Drupal core provides. Questions around that are welcome here.

And multi-language is an optional feature of Drupal. Nothing the default robots.txt needs to cover from the start.

That's fine, and I can agree with your opinion on that. My opinion extends further though, and is that as soon as language features are turned on, a Drupal-specific mechanism would kick in to solve the problem.

The same you wouldn't consider Drupal's default theme Bartik to cover all possible visual needs, you can't consider Drupal's default robots.txt to cover all possible path configurations.

No, but we're not considering whether Drupal's robots.txt covers "all possible path configurations". We're considering whether it covers the path configurations for the multi-language system that Drupal specifically provides out of the box.

If Bartik didn't work with the core multi-language setup, that would be a very Drupal-specific problem wouldn't it? In the same way, so is this.

Talking about robots.txt in specific there's http://www.robotstxt.org/robotstxt.html (and as SE network probably Webmasters) to get help configuring this generic file

Generic help isn't being sought, though - Drupal-specific help is being sought. That's why we have this site. I would imagine that's why you yourself answered the question, providing a Drupal-specific way for the OP to configure the very same "generic file".

robots.txt is SEO

Well no, not really. Some of its usage has some knock-on effect on SEO, sure, but that isn't what it's for. Its purpose is to instruct crawlers how to behave. Notice that the terms "SEO" and "Search engine optimisation" do not appear on http://www.robotstxt.org.

The only argument I can see for it being off topic in any way is that maybe it would be better served as a feature request to the core project. But as there are ways to solve the problem, I think that's reaching.

In my opinion, you're probably trying to apply the letter of the law too strictly here, without giving thought to the nuances of the question/subject, and how applicable and useful this is and will be to future users of Drupal.

5
  • It's not the multi-language setup Drupal provides. It's just one. You can simply configure Drupal to play content based on domain rather than on language prefix. No path prefixes then. Having path prefixes or not, is nothing which is specifically provided only by Drupal. That's actually a quite generic feature.
    – leymannx
    Commented Jan 9, 2019 at 13:18
  • But yeah, +1 I'm just applying the rules very strictly. Whereas at the same time I find this a very interesting question. And would love to help developing a tool to solve the problem of generating robots.txt dynamically based on the configured language setup. But(!) this is not what OP is currently having a problem with.
    – leymannx
    Commented Jan 9, 2019 at 13:27
  • I didn't mean to make it sound like I was concentrating on one particular language detection method (like path prefixes), I'm approaching this from the higher level that Drupal provides a multi-language environment at all. The fact that there's a robots.txt in Drupal at all means that the core devs have acknowledged that there are some Drupal-specific paths that need to be catered for. Once you have translation enabled, new versions of those paths exist, so logically they need to be catered for as well. Just because you can do it manually, with no Drupal-specific knowledge...
    – Clive Mod
    Commented Jan 9, 2019 at 13:37
  • 1
    ...doesn't mean that you have to do that. IMO the question can be answered both completely generically, if that's how one wants to look at it, or in a very Drupal-specific manner. And I personally think that the latter automatically makes it on topic here
    – Clive Mod
    Commented Jan 9, 2019 at 13:37
  • 1
    After I've found drupal.org/project/drupal/issues/347515 and its successor issue I totally have to accept your answer. Thanks a lot to everybody for the fruitful discussion.
    – leymannx
    Commented Jan 9, 2019 at 16:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .