Update bots.md (#2458)

added crawlers to the doc

docs/feature-flags/bots.md

@@ -33,7 +33,7 @@ You might, however, want to purposefully restrict what features bots see. For ex
     Once created, add a new rule to the segment. Set Criteria to "Browser Name". Leave Operator as "Any Of". In the Values field, copy + paste the following string in its entirety. (There is a copy button to the right.) When pasting, Statsig console will take care of splitting the bots up into individual names.
 
     ```
-    TSMbot, Googlebot, FacebookBot, TwitterBot, AdsBot-Google, ImagesiftBot, Dragonbot, bingbot, YandexRenderResourcesBot, startmebot, SeznamBot, Better Uptime Bot, SeekportBot, Slackbot-LinkExpanding, googlebot, harsilbot, AhrefsBot, Applebot, PetalBot, Pokey_Bot, Preview Service; bot, Googlebot-Image, WRTNBot, GPTBot, MJ12bot, YandexBot, Slackbot, trendictionbot0, SmarshBot, Amazonbot, VirusTotalBot, GooglePlusBot, com/bot, DuckDuckBot, Discordbot, ; bot, TermlyBot, YandexAccessibilityBot, UptimeRobot, CCBot, BitSightBot, AwarioSmartBot, SiteAuditBot, pingbot, ; Bot, Pinterestbot, idealo-bot, net/bot, AwarioBot, BLEXBot, adsbot, Linkbot, AcademicBotRTU, MojeekBot, LinkedInBot, BeeperBot, robot, PopeTech-ScanBot, FullStoryBot, Storebot, Zoombot, coccocbot, Mattermost-Bot, Nigooutbot, EzoicBot, com/bots, FreshpingBot, Mail.RU_Bot, Mediumbot, tkbot, triptease-bot, KlaxoonBot, CriteoBot, SMTBot, JobboerseBot, Mediatoolkitbot, iCjobs Stellenangebote, StatusCakeBot, AppleNewsBot, ClaudeBot, AmazonAdBot, GoogleBot, SemrushBot, SynologyChatBot, YandexMobileBot, bitlybot, ) Bot, redditbot, Radius Compliance Bot, Twitterbot, PingdomBot, DotBot, amazonproductbot, iASD_SpiderBot, XBot, WallabyupBot, XBot_Senior, HatenaBlog-bot, Exabot, AASA-Bot, DuckAssistBot, StractBot, wpbot, YextBot, chatbot, Pharosbot, ClarityBot, Monsidobot, AndersPinkBot, DataForSeoBot, Robot, Quora-Bot, notebot, HyperMegaBotGettingOnlyHTMLsFromYourWebsite, bountybot, WincherBot, Leikibot, ExtendedStayBot, Caliperbot, keys-so-bot, aixnew_aibot, SeobilityBot, Synapse (bot, UOrgTestingBot, serpstatbot, Diffbot, SummalyBot, rogerbot, bidswitchbot, PiBot, aka-bot, node CCBot, seesawbot, SearchAtlas Bot, MetaJobBot, archive.org_bot, 48_safeAreaBottom, ZoominfoBot, TurnitinBot, Googlebot-Video, fixbot, taboolabot, yacybot, Plesk screenshot bot, traq-ogp-fetcher-curl-bot, Space Unfurl Bot, Gensparkbot, by fynd.bot, adbeat_bot, SurdotlyBot, Spider_Bot, DiffeoBot, Rhobot, Cookiebot, online-webceo-bot, dataforseobot, Google-Display-Ads-Bot, Timpibot, msnbot, AnytypeBot, com feedbot, Morningscore Bot, Magus Bot, Snap-URL-Preview (bot, BublupBot, DiscourseBot, policy adbeat_bot, htc_botdugls, RyteBot, SaberBot, fr_bot, node FullStoryBot, bot, turbotime, OtherwebBot, TypetalkBot, clever tech bot, compatible; botify, Rankabot, AspiegelBot, Wire LinkPreview Bot, oBot, Amazon-Advertising-ad-standards-bot, es_bot, Ocarinabot, dbot, tyseobot, WebExplorerSearchBot, DF Bot, WebwikiBot, DropboxPreviewBot, NetpeakCheckerBot, Sidetrade indexer bot, Dubbotbot, Senutobot, Veoozbot, Fedicabot, Nextdoorbot, ZumBot, Streamline3Bot, vebidoobot, cXensebot, YodaoBot, Scrapbox Bot, 47_safeAreaBottom, SemanticScholarBot, SiteCheckerBotCrawler, WalluBot, iAskBot, Scomplerbot, ViberBot, GnowitNewsbot, Letianpai_Robot, WellKnownBot, am a bot, OcelotBot, //boteden, TimeTreeBot, seo-audit-check-bot, SEBot, LoomlyBot, StrapBot, QualifiedBot, Swiftbot, uk_ldfc_renderbot, Jones Searchbot, tyseobotmobile, ahrefsbot, ChannelBot, emulate-seobots, twitterbot, SpeechifyBot, PhaverBot, Xbot, node DuckAssistBot, semaltbot, Bawaab_bot, net-Robot, SuperBot, KStandBot, Facebot, node AppleNewsBot, robots, com bot, siteauditbot, co Bot, SerendeputyBot, PartnerOptimizer-bot, http-spiders-bot, telegrambot, RepoLookoutBot, slackbot, LivelapBot, uk_ldfc_bot, Your robot, eu bot, nerdybot, TiggeritoBot, GraphiteBot, BLP_bbot, domainsbot, node ZoominfoBot, LinkArchiver twitter bot, 2ip bot, COIBotParser, exabot, Googlebot-Mobile, DuckDuckGo-Favicons-Bot, PerplexityBot, yoozBot, BadooBot, discobot, web-bot, SEMrushBot, Open Graph Bot, PaperLiBot, Blog Rssbot, MotoMinerBot, eventseekerBot, ResearchBot, MixrankBot, node bitlybot, SpringserveBot, Firefox superpagesbot2, DingTalkBot, MoodleBot, Brightbot, reurl-bot, osapon ) bot, foundeebot, petalbot, //botsin, SeoCherryBot, SemjiBot, TZUnfurlBot, TesseractBotAgent, yandexbot, 5) bot, PubMatic Crawler Bot, archiver/3.1.1 +http://www.archive.org/details/archive.org_bot, msnbot-media, bountybotttt, Googlebot-News, semrushbot, Gulper Web Bot, Google-bot, Superfeedr bot, node AwarioSmartBot, GG PeekBot, playwright-bot, Clickagy Intelligence Bot, x28-job-bot, Catrobatbot, VelaBot, pinterestbot, hstspreload-bot, FandomOpenGraphBot, spbot, Paqlebot, Summalybot, //botim, aiHitBot, JobBot, InsytfulBot, Taboolabot, gptbot, edansbot, KeybaseBot, GroupMeBot, macox bot, Xing Bot, uipbot, Dcard-link-preview-bot, ezoicbot, like Gecko) bot, mj12bot, applebot, jbot, LineBotWebhook, HearsayPDFBot, DatoCmsSearchBot, Quantcastbot, ActiveComplyBot, Parser Robot, amazon-product-discovery-bot, AppsFlyerBot, TwitterCommerceBot, obot, ID bot, GetLocalBot, CapterraBot, BugBountyBot, PlurkBot, preview service; bot, BacklinksExtendedBot, Testomatobot,
+    TSMbot, Googlebot, FacebookBot, TwitterBot, AdsBot-Google, ImagesiftBot, Dragonbot, bingbot, YandexRenderResourcesBot, startmebot, SeznamBot, Better Uptime Bot, SeekportBot, Slackbot-LinkExpanding, googlebot, harsilbot, AhrefsBot, Applebot, PetalBot, Pokey_Bot, Preview Service; bot, Googlebot-Image, WRTNBot, GPTBot, MJ12bot, YandexBot, Slackbot, trendictionbot0, SmarshBot, Amazonbot, VirusTotalBot, GooglePlusBot, com/bot, DuckDuckBot, Discordbot, ; bot, TermlyBot, YandexAccessibilityBot, UptimeRobot, CCBot, BitSightBot, AwarioSmartBot, SiteAuditBot, pingbot, ; Bot, Pinterestbot, idealo-bot, net/bot, AwarioBot, BLEXBot, adsbot, Linkbot, AcademicBotRTU, MojeekBot, LinkedInBot, BeeperBot, robot, PopeTech-ScanBot, FullStoryBot, Storebot, Zoombot, coccocbot, Mattermost-Bot, Nigooutbot, EzoicBot, com/bots, FreshpingBot, Mail.RU_Bot, Mediumbot, tkbot, triptease-bot, KlaxoonBot, CriteoBot, SMTBot, JobboerseBot, Mediatoolkitbot, iCjobs Stellenangebote, StatusCakeBot, AppleNewsBot, ClaudeBot, AmazonAdBot, GoogleBot, SemrushBot, SynologyChatBot, YandexMobileBot, bitlybot, ) Bot, redditbot, Radius Compliance Bot, Twitterbot, PingdomBot, DotBot, amazonproductbot, iASD_SpiderBot,  XBot, WallabyupBot,  XBot_Senior, HatenaBlog-bot, Exabot, AASA-Bot, DuckAssistBot, StractBot, wpbot, YextBot, chatbot, Pharosbot, ClarityBot, Monsidobot, AndersPinkBot, DataForSeoBot, Robot, Quora-Bot, notebot, HyperMegaBotGettingOnlyHTMLsFromYourWebsite, bountybot, WincherBot, Leikibot, ExtendedStayBot, Caliperbot, keys-so-bot, aixnew_aibot, SeobilityBot, Synapse (bot, UOrgTestingBot, serpstatbot, Diffbot, SummalyBot, rogerbot, bidswitchbot, PiBot, aka-bot, node CCBot, seesawbot, SearchAtlas Bot, MetaJobBot, archive.org_bot, 48_safeAreaBottom, ZoominfoBot, TurnitinBot, Googlebot-Video, fixbot, taboolabot, yacybot, Plesk screenshot bot, traq-ogp-fetcher-curl-bot, Space Unfurl Bot, Gensparkbot, by fynd.bot, adbeat_bot, SurdotlyBot, Spider_Bot, DiffeoBot, Rhobot, Cookiebot, online-webceo-bot, dataforseobot, Google-Display-Ads-Bot, Timpibot, msnbot, AnytypeBot, com feedbot, Morningscore Bot, Magus Bot, Snap-URL-Preview (bot, BublupBot, DiscourseBot, policy adbeat_bot, htc_botdugls, RyteBot, SaberBot, fr_bot, node FullStoryBot, bot, turbotime, OtherwebBot, TypetalkBot, clever tech bot, compatible; botify, Rankabot, AspiegelBot, Wire LinkPreview Bot, oBot, Amazon-Advertising-ad-standards-bot, es_bot, Ocarinabot,  dbot, tyseobot, WebExplorerSearchBot, DF Bot, WebwikiBot, DropboxPreviewBot, NetpeakCheckerBot, Sidetrade indexer bot, Dubbotbot, Senutobot, Veoozbot, Fedicabot, Nextdoorbot, ZumBot, Streamline3Bot, vebidoobot, cXensebot, YodaoBot, Scrapbox Bot, 47_safeAreaBottom, SemanticScholarBot, SiteCheckerBotCrawler, WalluBot, iAskBot, Scomplerbot, ViberBot, GnowitNewsbot, Letianpai_Robot, WellKnownBot, am a bot, OcelotBot, //boteden, TimeTreeBot, seo-audit-check-bot, SEBot, LoomlyBot, StrapBot, QualifiedBot, Swiftbot, uk_ldfc_renderbot, Jones Searchbot, tyseobotmobile, ahrefsbot, ChannelBot, emulate-seobots, twitterbot, SpeechifyBot, PhaverBot, Xbot, node DuckAssistBot, semaltbot, Bawaab_bot, net-Robot, SuperBot, KStandBot, Facebot, node AppleNewsBot, robots, com bot, siteauditbot, co Bot, SerendeputyBot, PartnerOptimizer-bot, http-spiders-bot, telegrambot, RepoLookoutBot, slackbot, LivelapBot, uk_ldfc_bot, Your robot, eu bot, nerdybot, TiggeritoBot, GraphiteBot, BLP_bbot, domainsbot, node ZoominfoBot, LinkArchiver twitter bot, 2ip bot, COIBotParser, exabot, Googlebot-Mobile, DuckDuckGo-Favicons-Bot, PerplexityBot, yoozBot, BadooBot, discobot, web-bot, SEMrushBot, Open Graph Bot, PaperLiBot, Blog Rssbot, MotoMinerBot, eventseekerBot, ResearchBot, MixrankBot, node bitlybot, SpringserveBot, Firefox superpagesbot2, DingTalkBot, MoodleBot, Brightbot, reurl-bot, osapon ) bot, foundeebot, petalbot, //botsin, SeoCherryBot, SemjiBot, TZUnfurlBot, TesseractBotAgent, yandexbot, 5) bot, PubMatic Crawler Bot, archiver/3.1.1 +http://www.archive.org/details/archive.org_bot, msnbot-media, bountybotttt, Googlebot-News, semrushbot, Gulper Web Bot, Google-bot, Superfeedr bot, node AwarioSmartBot, GG PeekBot, playwright-bot, Clickagy Intelligence Bot, x28-job-bot, Catrobatbot, VelaBot, pinterestbot, hstspreload-bot, FandomOpenGraphBot, spbot, Paqlebot, Summalybot, //botim, aiHitBot, JobBot, InsytfulBot, Taboolabot, gptbot, edansbot, KeybaseBot, GroupMeBot, macox bot, Xing Bot, uipbot, Dcard-link-preview-bot, ezoicbot, like Gecko) bot, mj12bot, applebot, jbot, LineBotWebhook, HearsayPDFBot, DatoCmsSearchBot, Quantcastbot, ActiveComplyBot, Parser Robot, amazon-product-discovery-bot, AppsFlyerBot, TwitterCommerceBot, obot, ID bot, GetLocalBot, CapterraBot, BugBountyBot, PlurkBot, preview service; bot, BacklinksExtendedBot, Testomatobot, crawlers, NE Crawler, scoopit-crawler, Impressumscrawler, node Screaming Frog SEO Spider, Testcrawler, PAGEFREEZER CRAWLER, peer39_crawler, com crawler, Screaming Frog SEO Spider, BrightEdge Crawler, XoviOnpageCrawler, NetSeer crawler, managr-webcrawler, captify-crawler, IAS Crawler, crawler_eb_germany_2, IBM-Crawler, Elastic-Crawler, Screaming Frog Wise SEO Spider, seobilitybot, fedistatsCrawler, ISSCyberRiskCrawler, AdsTxtCrawlerTP, HubSpot Crawler, Jugendschutzprogramm-Crawler, screaming frog seo spider, about-crawlers, RavenCrawler, Atomseobot, sap-search-web-crawler, wowLink Crawler, our-crawler, Light Crawler, SiteGuruCrawler, Rightlander Crawler, web-crawler, hubspot crawler, Audisto Crawler, Automattic Analytics Crawler, crawler, Web-Crawler, pagefreezer crawler, WazzupCrawler, ecoresearchCrawler, VelenPublicWebCrawler, node GrowSEOBot, rc-crawler, ClineCrawler, naverbookmarkcrawler, IVW-Crawler, AdkernelTopicCrawler, InfobipCrawler, WebCrawler, GrapeshotCrawler, SISTRIX Crawler, Greppr Web Crawler, Server Crawler, DocBase Crawler, Web Crawler, find-seo-bot, PulsePoint-Crawler, GenomeCrawlerd, MSIECrawler, txt Crawler, ev-crawler, DocSearch Crawler, crawler4j, UCMore Crawler, FAST-WebCrawler,
     ```
   
     ![Screenshot 2024-08-09 at 1 08 50 PM](https://github.com/user-attachments/assets/97c1cdfe-9d65-42ae-acc7-ed47b583fce6)