Skip to content

Commit

Permalink
WIP: Support HAProxy PROXY protocol V1 on MySQL frontends
Browse files Browse the repository at this point in the history
This adds basic support for the HAProxy PROXY protocol V1 on MySQL frontend
client connections.  This allows the true client IP to be seen in the output
of `SHOW FULL PROCESSLIST` in ProxySQL's admin frontend when the ProxySQL
servers sit behind a load balancer such as HAProxy or AWS classic ELBs with
the PROXY protocol feature enabled.

I believe this should resolve sysown#2497 (Support for proxy_protocol for proxysql behind aws load balancer).

The patch adds a Proxy_Protocol class which handles:

- parsing of PROXY protocol headers (only the V1 header supported for now)
- matching client IPs against a list of configured network CIDRs

A new option, `proxy_protocol_frontend_nets` (list of network CIDRs), is
introduced to allow the PROXY protocol feature to be selectively turned on for
specific subnets where e.g. load balancers are running.  This is similar to how
the PROXY protocol is implemented in MariaDB.

This variable can be set either in the config file or via the ProxySQL Admin
interface.  Configuration file example:

    mysql_variables=
    {
        ...
        proxy_protocol_frontend_nets="10.42.0.0/28 127.10.0.0/16"
        ...
    }

ProxySQL Admin interface example:

    UPDATE global_variables
    SET variable_value="192.168.42.0/24"
    WHERE variable_name="mysql-proxy_protocol_frontend_nets";
    LOAD MYSQL VARIABLES TO RUNTIME;

The network CIDRs are stored as a thread-local variable in the MySQL_Thread
class.  It's only updated by MySQL_Thread::refresh_variables().

A new MySQL Data Stream state (DSS), `STATE_PROXY_PROTOCOL`, is introduced to
handle the initial parsing of the PROXY protocol header.  Once this has
completed, the state is reset back to `STATE_SERVER_HANDSHAKE`.

The patch hooks into the MySQL_Thread's main loop where `accept()` is called
to handle incoming connections.  Next, the actual parsing of the PROXY protocol
header is invoked from the `MySQL_Data_Stream::read_pkts()` method, which is
also called from MySQL_Thread's main loop (where incoming data is handled).

I've only tested this with a HAProxy configuration block like:

    listen proxysql-with-proxy
        bind *:6030
        option tcp-check
        mode tcp
        timeout server 6h
        timeout client 6h
        balance roundrobin
        server proxysql-backend-1 127.0.0.1:6033 check send-proxy
    listen proxysql-without-proxy
        bind *:6031
        option tcp-check
        mode tcp
        timeout server 6h
        timeout client 6h
        balance roundrobin
        server proxysql-backend-2 192.168.1.123:6033 check

NOTE: I took the liberty to reindent a block of code in MySQL_Thread.cpp
so please review with whitespace changes hidden.
  • Loading branch information
noahwilliamsson committed Jul 20, 2020
1 parent ca733a7 commit cffb9f6
Show file tree
Hide file tree
Showing 11 changed files with 424 additions and 139 deletions.
1 change: 1 addition & 0 deletions include/MySQL_Data_Stream.h
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ class MySQL_Data_Stream
int array2buffer_full();
void init(); // initialize the data stream
void init(enum MySQL_DS_type, MySQL_Session *, int); // initialize with arguments
void update_client_addr(struct sockaddr *);
void shut_soft();
void shut_hard();
int read_from_net();
Expand Down
8 changes: 8 additions & 0 deletions include/MySQL_Thread.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
#include "proxysql.h"
#include "cpp.h"
#include "MySQL_Variables.h"
#include "Proxy_Protocol.h"

#ifdef IDLE_THREADS
#include <sys/epoll.h>
#endif // IDLE_THREADS
Expand Down Expand Up @@ -305,6 +307,12 @@ class MySQL_Threads_Handler
bool sessions_sort;
char *default_schema;
char *interfaces;
/**
* List (",; ") of CIDR blocks for which PROXY protocol headers
* should be parsed for incoming MySQL frontend connections
* (mysql-proxy_protocol_frontend_nets)
*/
char *proxy_protocol_frontend_nets;
char *server_version;
char *keep_multiplexing_variables;
//unsigned int default_charset; // removed in 2.0.13 . Obsoleted previously using MySQL_Variables instead
Expand Down
22 changes: 22 additions & 0 deletions include/Proxy_Protocol.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#ifndef __CLASS_PROXY_PROTOCOL_H
#define __CLASS_PROXY_PROTOCOL_H

#include <vector>
#include <sys/types.h>
#include <netinet/in.h>
#include <sys/socket.h>


class Proxy_Protocol {
private:
static bool add_subnet(const char *cidr, std::vector<proxy_protocol_subnet_t> &subnets);

public:
Proxy_Protocol() {}
~Proxy_Protocol() {}
static bool parse_subnets(const char *list, std::vector<proxy_protocol_subnet_t> &subnets);
static bool match_subnet(struct sockaddr *addr, socklen_t addrlen, std::vector<proxy_protocol_subnet_t> &subnets);
static bool parse_header(unsigned char *, size_t n, struct sockaddr_storage *out);
};

#endif /* __CLASS_PROXY_PROTOCOL_H */
8 changes: 8 additions & 0 deletions include/proxysql_structs.h
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,8 @@ typedef struct {
enum mysql_data_stream_status {
STATE_NOT_INITIALIZED,
STATE_NOT_CONNECTED,
/* parse HAProxy PROXY protocol header before server handshake */
STATE_PROXY_PROTOCOL,
STATE_SERVER_HANDSHAKE,
STATE_CLIENT_HANDSHAKE,
STATE_CLIENT_AUTH_OK,
Expand Down Expand Up @@ -406,6 +408,11 @@ typedef struct _PtrSize_t PtrSize_t;
typedef struct _proxysql_mysql_thread_t proxysql_mysql_thread_t;
typedef struct { char * table_name; char * table_def; } table_def_t;
typedef struct __SQP_query_parser_t SQP_par_t;
typedef struct {
unsigned short family;
unsigned short bits;
unsigned char addr[16];
} proxy_protocol_subnet_t;
#endif /* PROXYSQL_TYPEDEFS */

//#ifdef __cplusplus
Expand Down Expand Up @@ -961,6 +968,7 @@ extern __thread char * mysql_thread___monitor_replication_lag_use_percona_heartb

extern __thread char * mysql_thread___add_ldap_user_comment;


#ifdef DEBUG
extern __thread bool mysql_thread___session_debug;
#endif /* DEBUG */
Expand Down
2 changes: 1 addition & 1 deletion lib/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ default: libproxysql.a

_OBJ = c_tokenizer.o
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
_OBJ_CXX = ProxySQL_GloVars.oo network.oo debug.oo configfile.oo Query_Cache.oo SpookyV2.oo MySQL_Authentication.oo gen_utils.oo sqlite3db.oo mysql_connection.oo MySQL_HostGroups_Manager.oo mysql_data_stream.oo MySQL_Thread.oo MySQL_Session.oo MySQL_Protocol.oo mysql_backend.oo Query_Processor.oo ProxySQL_Admin.oo ProxySQL_Config.oo ProxySQL_Restapi.oo MySQL_Monitor.oo MySQL_Logger.oo thread.oo MySQL_PreparedStatement.oo ProxySQL_Cluster.oo ClickHouse_Authentication.oo ClickHouse_Server.oo ProxySQL_Statistics.oo Chart_bundle_js.oo ProxySQL_HTTP_Server.oo ProxySQL_RESTAPI_Server.oo font-awesome.min.css.oo main-bundle.min.css.oo set_parser.oo MySQL_Variables.oo
_OBJ_CXX = ProxySQL_GloVars.oo network.oo debug.oo configfile.oo Query_Cache.oo SpookyV2.oo MySQL_Authentication.oo gen_utils.oo sqlite3db.oo mysql_connection.oo MySQL_HostGroups_Manager.oo mysql_data_stream.oo MySQL_Thread.oo MySQL_Session.oo MySQL_Protocol.oo mysql_backend.oo Query_Processor.oo ProxySQL_Admin.oo ProxySQL_Config.oo ProxySQL_Restapi.oo MySQL_Monitor.oo MySQL_Logger.oo thread.oo MySQL_PreparedStatement.oo ProxySQL_Cluster.oo ClickHouse_Authentication.oo ClickHouse_Server.oo ProxySQL_Statistics.oo Chart_bundle_js.oo ProxySQL_HTTP_Server.oo ProxySQL_RESTAPI_Server.oo font-awesome.min.css.oo main-bundle.min.css.oo set_parser.oo MySQL_Variables.oo Proxy_Protocol.oo
OBJ_CXX = $(patsubst %,$(ODIR)/%,$(_OBJ_CXX))
HEADERS = ../include/*.h ../include/*.hpp

Expand Down
4 changes: 4 additions & 0 deletions lib/MySQL_Protocol.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#include <openssl/rand.h>
#include "proxysql.h"
#if !defined(__FreeBSD__) && !defined(__APPLE__)
#define HAVE_BOOL
#endif
#include "ma_global.h"
#include "cpp.h"

#include "MySQL_PreparedStatement.h"
Expand Down
2 changes: 2 additions & 0 deletions lib/MySQL_Session.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2809,6 +2809,8 @@ int MySQL_Session::handler() {

case CONNECTING_CLIENT:
switch (client_myds->DSS) {
case STATE_PROXY_PROTOCOL:
break;
case STATE_SERVER_HANDSHAKE:
handler___status_CONNECTING_CLIENT___STATE_SERVER_HANDSHAKE(&pkt, &wrong_pass);
break;
Expand Down
Loading

0 comments on commit cffb9f6

Please sign in to comment.