feeds.py (and surprisingly many other plugins like ping.py, and more) do not sanitize user input from commands and lead to crashes and other errors..... OR WORSE | .help leads to excess flood IRC quits

[hazeyez]

As the title suggests, feeds.py for the RSS feeds will gladly accept any format URL as it does not sanitize user input, and in this case it was found that .rss file:///dev/zero floods the bot and crashes it with null chars.

There were other errors with feeds.py as well, such as

• Some custom url request being refused with error HTTP/403 - I found that whatever is being used to make HTTP(S) requests does not specify a User-agent, which leads to some servers refusing the request. So I added a common, passable User-agent.
• feed = feedparser.parse(addr) forced the request to parse the actual feed address? So the .rss <url> request in chan would just have the bot return the URL address. addr was changed so it parsed the actual content.

I took time to fix the plugin - which is below. Feel free to use the code, or don't. I think at this point after seeing how easily anyone can have a plugin pushed to this repo without any solid coding or basic input sanitizing - I've been deterred from continuing to use Cloudbot on IRC. I'm seeking alternatives, sadly - because I'd have to probably modify a substantial mount of plugins. It was a decent bot with good plugin options, but this type of stuff is purely unacceptable.

Someone else also found an issue that leads to more crashes (or Remote code execution????) in ping.py - here's his initial suggestion, which we quit trying to fix after the above realization:

<user> plugins/ping.py i think has a backdoor i am trying to bust,         args = ["ping", "-n", str(count), host]
<user> plugins/ping.py
<user> pingcmd = subprocess.check_output(args).decode("utf-8")
<user> no sanatizing going on in this process
<user> but it does seem to excluse some characters from somewhere, but i think a unicode conversion may kick it
-
<user> plugins/ping.py, replace line 35 with host = re.sub(r'[^a-zA-Z0-9:/._-]', '', args[0])
<user> double check to make sure i am not messing with it in a bad way
<user> we can probably refine it hurther since it shouldn't be a full url anyway
<user> host = re.sub(r'[^a-zA-Z0-9.-]', '', args[0])
<user> that may be better
<user> the file already imports re so it is very weird they didn't do this to that arg

Finally, .help leads to "Excess Flood" server quits by the bot.. I don't know if it's the newest version which I installed, but it was not a problem in the past with an older version...

* bot_nick ([email protected]) Quit (Excess Flood)
* bot_nick ([email protected]) has joined #cIRCuit
* bot_nick ([email protected]) Quit (Excess Flood)
* bot_nick ([email protected]) has joined #cIRCuit
* bot_nick ([email protected]) Quit (Excess Flood)
<hazeyez> okay how is that happening?
* bot_nick ([email protected]) has joined #cIRCuit
<hazeyez> not with .rss
* bot_nick ([email protected]) Quit (Excess Flood)
* bot_nick ([email protected]) has joined #cIRCuit
<user> nope
<user> .help

---

Fixed feeds.py:

import feedparser
import requests
import logging
from cloudbot import hook
from cloudbot.util import formatting, web
from urllib.parse import urlparse

# ''' NOTE: anything with this `''' '''` comment format are things added or customized by hazeyez '''
# NOTE: anything with this comment format OR NO COMMENT are part of the original script

class FeedAlias:
    def __init__(self, url, limit=3):
        self.url = url
        self.limit = limit

# predefined RSS feed aliases
# ''' need to figure out how to get the user a list of these that are available for their use - perhaps add it to the "help" response when only `.rss` command is entered and bot sends /notice to user. Also, can add/remove some feed aliases. ''' 
ALIASES = {
    "xkcd": FeedAlias("http://xkcd.com/rss.xml"),
    "ars": FeedAlias("http://feeds.arstechnica.com/arstechnica/index"),
    "pip": FeedAlias("https://pypi.python.org/pypi?%3Aaction=rss", 6),
    "pypi": FeedAlias("https://pypi.python.org/pypi?%3Aaction=rss", 6),
    "py": FeedAlias("https://pypi.python.org/pypi?%3Aaction=rss", 6),
    "pipnew": FeedAlias(
        "https://pypi.python.org/pypi?%3Aaction=packages_rss", 5
    ),
    "pypinew": FeedAlias(
        "https://pypi.python.org/pypi?%3Aaction=packages_rss", 5
    ),
    "pynew": FeedAlias(
        "https://pypi.python.org/pypi?%3Aaction=packages_rss", 5
    ),
    "world": FeedAlias(
        "https://news.google.com/news?cf=all&ned=us&hl=en&topic=w&output=rss"
    ),
    "us": FeedAlias(
        "https://news.google.com/news?cf=all&ned=us&hl=en&topic=n&output=rss"
    ),
    "usa": FeedAlias(
        "https://news.google.com/news?cf=all&ned=us&hl=en&topic=n&output=rss"
    ),
    "nz": FeedAlias(
        "https://news.google.com/news?pz=1&cf=all&ned=nz&hl=en&topic=n&output=rss"
    ),
    "anand": FeedAlias("http://www.anandtech.com/rss/"),
    "anandtech": FeedAlias("http://www.anandtech.com/rss/"),
}

# ''' added some logging features to coincide with the existing debugging and logging system in the bot '''
logger = logging.getLogger(__name__)

# formats feed results into shortened URL with title
def format_item(item):
    url = web.try_shorten(item.link)
    title = formatting.strip_html(item.title)
    return f"{title} ({url})"

# ''' verify all custom feed links (non-aliases) have HTTP/HTTPS prefix - sanitize URL input, where `.rss file:///dev/zero` command crashed bot with null chars previously '''
def is_valid_url(url):
    parsed_url = urlparse(url)
    return parsed_url.scheme in ["http", "https"]

# gets first 3 items in RSS feed to return. can be modified to return more in `limit =`
@hook.command("feed", "rss", "news")
def rss(text):
    t = text.lower().strip()
    if t in ALIASES:
        alias = ALIASES[t]
        addr = alias.url
        limit = alias.limit
    else:
        addr = text
        limit = 3

    # ''' validate URL - more sanitizing input to avoid bot being crashed with file:///dev/zero null chars vuln '''
    if not is_valid_url(addr):
        return "Invalid URL. Only HTTP and HTTPS protocols are supported."

    try:
        # ''' had to add custom user-agent to requests as most feeds return HTTP/403 error for certain user-agents or non-browser requests '''
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
        }
        response = requests.get(addr, headers=headers, timeout=10)
        response.raise_for_status()
        content = response.content

        # ''' log the content fetched for debugging purposes - to terminal STDOUT and also existing logging system '''
        logger.debug(f"Fetched content: {content[:1000]}") 

        feed = feedparser.parse(content) # ''' "(content)" replaced "(addr)" because "addr" forced it to parse the URL itself instead of the feed content '''
        if not feed.entries:
            return "Feed not found."

        out = []
        for item in feed.entries[:limit]:
            logger.debug(f"Feed item: {item}") # ''' more debugging logging added for each feed item parsed '''
            out.append(format_item(item))

        if "title" in feed.feed:
            start = f"\x02{feed.feed.title}\x02: "
        else:
            start = ""

        return start + ", ".join(out)

    # ''' more error logging, and also returns "not found" error to user if applicable '''
    except requests.exceptions.RequestException as e:
        logger.error(f"Error fetching RSS feed: {e}")
        return f"Error fetching RSS feed: {e}"