Multiple protocols on one port: example with TLS1.1 & TLS1.2

[Frky]

This PR aims to add the possibility to retry with another protocol when the first one failed. The implemented example is for TLS: if we have an alert when trying TLS 1.1, we want to retry on the same host, port with TLS 1.2.

The result is that the x509 certificate of a server that only supports TLS1.2 will now be successfully grabbed by masscanned (while before it would not have been):

$ sudo ./bin/masscan --banners -p443 -n $targetip
[+] resolving router 192.168.1.1 with ARP (may take some time)...
Starting masscan 1.3.2 (http://bit.ly/14GZzcT) at 2022-08-18 17:33:30 GMT
Initiating SYN Stealth Scan
Scanning 1 hosts [1 port/host]
Discovered open port 443/tcp on [...]                                  
Discovered open port 443/tcp on [...]                                  
Banner on port 443/tcp on [...]: [ssl] TLS/1.2 cipher:0xc030, [...]
Banner on port 443/tcp on [...]: [X509] MIIEPjCCAyagAwIB[...]
Banner on port 443/tcp on [...]: [X509] MIIFLTCCBBWgAwIB[...]
Banner on port 443/tcp on [...]: [X509] MIIHUDCCBjigAwIB[...]
Banner on port 443/tcp on [...]: [ssl]  ALERT(0x0246) 

However, I am not 100% confident this PR does not have important drawbacks on the performances or even if it breaks things, so any review/testing/feedback/suggestion is welcome.

Here is an overview on how it is implemented:

• using the already existing (but unused?) field next in the ProtocolParserStream struct, we can already link several protocols in one port (see code):

    /* When multiple items are registered for a port. When one
     * connection is closed, the next will be opened.*/
    struct ProtocolParserStream *next;

• we add a field in the ProtocolState state, called try_next, which is set by a protocol parser when it fails (this tells that the next one should be tried) -- for instance, the TLS 1.1 parsing function set try_next to 1 (true) when it parses an SSL alert (see diff):
• in the receiving thread, we add a new SYN packet in the sending queue when the field try_next of the protocol state is set to 1 (see diff):

                if (tcb->banner1_state.try_next) {
                    /* create a new TCP SYN packet in the sending queue */
                    [...]
                    stack_transmit_packetbuffer(tcpcon->stack, response);
                }

• because the transmitting and receiving threads are stateless (to avoid locks), we use the syn-cookie to distinguish the first from the second attempt (to know which protocol to test when we receive the SYN-ACK from the server). What we suggest here is to compute the syn-cookie of the second attempt with entropy + 1 as a key instead of entropy. Of course, this implies to try both keys when we receive a packet, but this allows for the receiving thread to know the index of the protocol to try in the linked-list of protocols:

            /* compute all possible cookies for other protocols */
            for (unsigned int i = 0; i < MAX_ITER; i++)
                cookie[i] = syn_cookie(ip_them, port_them, ip_me, port_me, entropy + i) & 0xFFFFFFFF;

            [...]

                /* check seqno against all possible cookies */
                for (iter = 0; iter < MAX_ITER; iter++) {
                    if (cookie[iter] == seqno_me-1)
                       break;

            [...]

                if (tcb == NULL) {
                    tcb = tcpcon_create_tcb(tcpcon,
                                    ip_me, ip_them,
                                    port_me, port_them,
                                    seqno_me, seqno_them+1,
                                    parsed.ip_ttl, iter);

[robertdavidgraham]

This looks really good. I'd like to see this integrated. Right now I'm going through a phase of accepting the 40+ outstanding pull requests, so I don't have time to look at something this complex.

They way I'd intended to solve this was a little different, maybe a linked-list of protocols to iterate over rather than an array. I'm going to have to look at your design more.

[Frky]

I agree, a linked list would be way nicer. I will adapt my PR.

Also, one problem I encountered using this multi-protocols feature is that once we have multiple probes for one port, it is then needed to have a way in the output file (XML/bin/etc.) to know which probe led to which answer. I have an other draft PR (not pushed yet) to add an option --output-probes which adds a field probe=... in the output file. @p-l- has prepared an update of Ivre to support this, but it could break other tools parsing masscan output (although without the --output-probes flag the output would not change). What did you have in mind for that?

[robertdavidgraham]

I have vastly changed the TCP stack in the latest push. It now makes this much easier. To find all the changes, such search for reconnect in the code. I integrated your first proposal from the IVRE branch, and then rewrote most of it.

First of all, a new SYN-cookie isn't needed, so I removed all those parts. Once a new port is selected, a new SYN-cookie is generated automatically. I now default to using 16 source ports, so there should always be a source port available.

I cleaned up the TCP state-machine, so that when you do a "send" of the SYN rather than a "xmit". This means it's been queued up in the TCB, and if it doesn't get a reply, it'll resend the SYN a few times.

I'm setting this up to integrate Lua scripting, so that scripts can likewise open a new TCP connection whenever they like.

I'm going to close this pull request since I've completely redone the TCP stack in the last week. Also, request has actually been accepted, since I pulled it down from the IVRE fork, but it doesn't have anything you've added in the last week.

Go search for "reconnect", especially the _do_reconnect() function.