diff options
| author | Jinwei Zhao <[email protected]> | 2016-01-12 18:25:26 +0800 |
|---|---|---|
| committer | Jinwei Zhao <[email protected]> | 2016-01-12 18:25:26 +0800 |
| commit | 40a347e5c2cc38b28d725bd7af1d9a230bacfc7c (patch) | |
| tree | 6b4371d629432ca3d2e7c2ea46729d69ee086a2f | |
| parent | 53347875073a252f20d2d417c80ec5a25f000b8b (diff) | |
| download | ring0.xyz-40a347e5c2cc38b28d725bd7af1d9a230bacfc7c.tar.gz | |
Update robots.txt
| -rw-r--r-- | robots.txt | 43 |
1 files changed, 42 insertions, 1 deletions
| @@ -1,12 +1,53 @@ | |||
| 1 | User-agent: Googlebot | 1 | User-agent: Googlebot |
| 2 | Disallow: | 2 | Disallow: |
| 3 | |||
| 3 | User-agent: googlebot-image | 4 | User-agent: googlebot-image |
| 4 | Disallow: | 5 | Disallow: |
| 6 | |||
| 5 | User-agent: googlebot-mobile | 7 | User-agent: googlebot-mobile |
| 6 | Disallow: | 8 | Disallow: |
| 9 | |||
| 10 | User-agent: MSNBot | ||
| 11 | Disallow: / | ||
| 12 | |||
| 7 | User-agent: Baiduspider | 13 | User-agent: Baiduspider |
| 8 | Disallow: / | 14 | Disallow: / |
| 9 | User-agent: baiduspider | 15 | |
| 16 | User-agent: Baiduspider-image | ||
| 17 | Disallow: / | ||
| 18 | |||
| 19 | User-agent: YoudaoBot | ||
| 20 | Disallow: / | ||
| 21 | |||
| 22 | User-agent: Sogou web spider | ||
| 23 | Disallow: / | ||
| 24 | |||
| 25 | User-agent: Sogou inst spider | ||
| 26 | Disallow: / | ||
| 27 | |||
| 28 | User-agent: Sogou spider2 | ||
| 29 | Disallow: / | ||
| 30 | |||
| 31 | User-agent: Sogou blog | ||
| 32 | Disallow: / | ||
| 33 | |||
| 34 | User-agent: Sogou News Spider | ||
| 35 | Disallow: / | ||
| 36 | |||
| 37 | User-agent: Sogou Orion spider | ||
| 38 | Disallow: / | ||
| 39 | |||
| 40 | User-agent: ChinasoSpider | ||
| 41 | Disallow: / | ||
| 42 | |||
| 43 | User-agent: Sosospider | ||
| 44 | Disallow: / | ||
| 45 | |||
| 46 | User-agent: yisouspider | ||
| 47 | Disallow: / | ||
| 48 | |||
| 49 | User-agent: EasouSpider | ||
| 10 | Disallow: / | 50 | Disallow: / |
| 51 | |||
| 11 | User-agent: * | 52 | User-agent: * |
| 12 | Disallow: | 53 | Disallow: |