ResourceWarning: unclosed <socket.socket...

[ctnguyenvn]

I using elastalert2 on docker, I have created rules with the website's login test (the website has many login turns). I used Enhancements to check some conditions and use Dropmatchexception() to remove most of them. Then I received a warning ...

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51034), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51078), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51124), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51178), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51198), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:320: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51226), raddr=('172.10.0.3', 9200)>
  if len(name) >= min(len(n) for n in self._months.keys()):

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51232), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:320: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51252), raddr=('172.10.0.3', 9200)>
  if len(name) >= min(len(n) for n in self._months.keys()):

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51262), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51314), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51328), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51356), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51362), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51386), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51392), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:192: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51450), raddr=('172.10.0.3', 9200)>
  return list(cls(s))

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51468), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:320: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51514), raddr=('172.10.0.3', 9200)>
  if len(name) >= min(len(n) for n in self._months.keys()):

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51520), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:183: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51540), raddr=('172.10.0.3', 9200)>
  raise StopIteration

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:183: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51568), raddr=('172.10.0.3', 9200)>
  raise StopIteration

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51580), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51626), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:183: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51646), raddr=('172.10.0.3', 9200)>
  raise StopIteration

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51652), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:57: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51680), raddr=('172.10.0.3', 9200)>
  instream = StringIO(instream)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51686), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:183: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51712), raddr=('172.10.0.3', 9200)>
  raise StopIteration

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51776), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51788), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)

WARNING:py.warnings:/usr/local/lib/python3.7/site-packages/python_dateutil-2.6.1-py3.7.egg/dateutil/parser.py:320: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51864), raddr=('172.10.0.3', 9200)>
  if len(name) >= min(len(n) for n in self._months.keys()):

WARNING:py.warnings:/usr/local/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.10.0.9', 51880), raddr=('172.10.0.3', 9200)>
  obj, end = self.scan_once(s, idx)
...

Is Dropmatchexception() in Enhancements did not close socket when match rule (Terminate in Enhancements Class) or large amount of alert makes Elastalert unable to handle the requests? i see discussions/887 but not helpful.
Do I need to close the socket when using Dropmatchexception()?

[jertel]

If you are opening connections in your enhancement class then yes you are responsible for ensuring they are properly closed. However, I suspect what you're observing is an artifact of Python's garbage collection combined with urllib3's connection pooling.

Another user posted something similar earlier this year: #887

And elastic discussions on this topic also exist: elastic/elasticsearch-py#1533

This AWS issue has some useful background, even though it's specific to boto3, but the premise is similar: boto/boto3#454

Based on the public information like that in the links above, it appears that this warning may not be cause for concern. If you experiencing connection errors or crashing I would be interested in seeing a reproducible test case to look into it. If you're not experiencing any problems, you can consider filtering out the warnings, as is described in several discussions like the ones above.

[ctnguyenvn]

Thank for your reply.
The warning may be ignored but as mentioned match rule a lot (but almost DropMatchException()), the log on the file /var/log/elastalert.log is very much. It is good to solve it.
My simple case look like below:

class CheckUserLogin(BaseEnhancement):
    def process(self, match):
        # add field if username == "admin" && uri == "/api/user/admin/x" else drop
        if match['username'] ==  "admin" && match['uri'] == "/api/user/admin/x":
            match["severity"] = "warning"
        else:
            # Do I need Close Socket here? how do it?
            raise DropMatchException()