build: add FTP and SFTP servers in docker-compose

[adamdecaf]

We should launch our docker-compose setup with an FTP and SFTP (ssh file transfer protocol) servers running so paygate can show the async file transfer logic. Instead paygate currently spams the log with errors:

{"caller":"file_transfer_async.go:215","startPeriodicFileOperations":"Starting periodic file operations","ts":"2019-07-06T00:47:05.870453375Z"}
...
{"caller":"file_transfer_async.go:271","downloadAndProcessIncomingFiles":"error downloading files into /storage/downloaded469047369","error":"downloadAllFiles: problem with 323075822 file transfer agent init: EOF","ts":"2019-07-06T00:49:05.879605236Z"}

There are commands in the makefile, make start-ftp-server and make start-sftp-server we can use that as a starting point for the docker-compose file. The only complication is that inbound and returned files are deleted, so you’d only see paygate process files in the first iteration of its sftp loop.

This also ties in with #147 as to how we would configure this inside the docker-compose file.

[wadearnold]

Should this be a part of a larger... "run as a developer"?

For production, you would want to know that an SFTP endpoint was properly configured. The error that I see in the log is what I would want to have for production. I also think that a "system check" script would be useful for production to ensure that the system was properly configured.

[adamdecaf]

Maybe the issue wasn't clear, but paygate defaults to checking the FTP/SFTP server from make start-[s]ftp-server -- even if that server isn't running. See:

paygate/file_transfer_configs.go

Lines 174 to 183 in 533fdf8

	func (r *localFileTransferRepository) getSFTPConfigs() ([]*sftpConfig, error) {
	return []*sftpConfig{
	{
	RoutingNumber: "121042882", // from 'go run ./cmd/server' in Accounts
	Hostname: "localhost:2121", // below configs for moov/fsftp:v0.1.0
	Username: "admin",
	Password: "123456",
	},
	}, nil
	}

The error logs would still appear on connection errors in production. We don't log differently for production vs development.

We could add each server to :9092/live, which Kubernetes uses to monitor paygate's health.

$ curl -s localhost:9092/live | jq . 
{
  "accounts": "accounts ping failed: Get http://localhost:8085/ping: dial tcp [::1]:8085: connect: connection refused",
  "ach": "error getting /ping from ACH service: ACH GET requestId=50d190562c0e1b8423c2d0b521052984fad41c72 : Get http://localhost:8080/ping: dial tcp [::1]:8080: connect: connection refused",
  "fed": "FED ping failed: Get http://localhost:8086/ping: dial tcp [::1]:8086: connect: connection refused",
  "ofac": "ofac.Ping: failed: Get http://localhost:8084/ping: dial tcp [::1]:8084: connect: connection refused"
}

Adding fields like "sftp-localhost:2121": null (or an error string).