There's this question [robots.txt vs language path prefix][1].

OP finds out that Drupal's default `robots.txt` doesn't actually cover his multi-language setup, which they configured to use path prefixes. An interesting question indeed. But is this really a Drupal-specific problem?

`robots.txt` is a customizable and generic file to communicate with crawlers, independently of what app, CMS or framework you are using. And mutli-language is an optional feature of Drupal. Nothing the default `robots.txt` needs to cover from the start.

The same you wouldn't consider Drupal's default theme Bartik to cover all possible visual needs, you can't consider Drupal's default `robots.txt` to cover all possible path configurations. Talking about `robots.txt` in specific there's http://www.robotstxt.org/robotstxt.html (and as SE network probably [Webmasters][2]) to get help configuring this generic file. It's also commented in the default `robots.txt`:

    # For more information about the robots.txt standard, see:
    # http://www.robotstxt.org/robotstxt.html

`robots.txt` is SEO and I'd consider the mentioned question to be off-topic. Different it would be though if OP comes up with a question like "How to dynamically generate robots.txt to take language path prefixes into account" or similar, with a feature they built and a specific point where they are stuck. This can clearly considered to be on-topic.

  [1]: https://drupal.stackexchange.com/q/274833/15055
  [2]: https://webmasters.stackexchange.com