-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathREADME.proxy.txt
211 lines (147 loc) · 7.64 KB
/
README.proxy.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
NAME
flexnbd-proxy - A simple NBD proxy
SYNOPSIS
flexnbd-proxy --addr ADDR [--port PORT] --conn-addr ADDR
--conn-port PORT [--bind ADDR] [--cache[=CACHE_BYTES]]
[--help] [--verbose] [--quiet]
DESCRIPTION
flexnbd-proxy is a simple NBD proxy server that implements resilient
connection logic for the client. It connects to an upstream NBD server
and allows a single client to connect to it. All server properties are
proxied to the client, and the client connection is kept alive across
reconnections to the upstream server. If the upstream goes away while
an NBD request is in-flight then the proxy (silently, from the point
of view of the client) reconnects and retransmits the request, before
returning the response to the client.
USAGE
Proxy requests from an NBD client to an NBD server, resiliently. Only one
client can be connected at a time, and ACLs cannot be applied to the client, as they
can be to clients connecting directly to a flexnbd in serve mode.
On starting up, the proxy will attempt to connect to the server specified by
--conn-addr and --conn-port (from the address specified by --bind, if given). If
it fails, then the process will die with an error exit status.
Assuming a successful connection to the `upstream` server is made, the proxy
will then start listening on the address specified by --addr and --port, waiting
for `downstream` to connect to it (this will be your NBD client). The client
will be given the same hello message as the proxy was given by the server.
When connected, any request the client makes will be read by the proxy and sent
to the server. If the server goes away for any reason, the proxy will remember
the request and regularly (~ every 5 seconds) try to reconnect to the server.
Upon reconnection, the request is sent and a reply is waited for. When a reply
is received, it is sent back to the client.
When the client disconnects, cleanly or otherwise, the proxy goes back to
waiting for a new client to connect. The connection to the server is maintained
at that point, in case it is needed again.
Only one request may be in-flight at a time under the current architecture; that
doesn't seem to slow things down much relative to alternative options, but may
be changed in the future if it becomes an issue.
OPTIONS
--addr, -l ADDR
The address to listen on. If this begins with a '/', it is assumed to be
a UNIX domain socket to create. Otherwise, it should be an IPv4 or IPv6
address.
--port, -p PORT
The port to listen on, if --addr is not a UNIX socket.
--conn-addr, -C ADDR
The address of the NBD server to connect to. Required.
--conn-port, -P PORT
The port of the NBD server to connect to. Required.
--cache, -c=CACHE_BYTES
If given, the size in bytes of read cache to use. CACHE_BYTES
defaults to 4096.
--help, -h
Show command or global help.
--verbose, -v
Output all available log information to STDERR.
--quiet, -q
Output as little log information as possible to STDERR.
LOGGING
Log output is sent to STDERR. If --quiet is set, no output will be
seen unless the program termintes abnormally. If neither --quiet nor
--verbose are set, no output will be seen unless something goes wrong
with a specific request. If --verbose is given, every available log
message will be seen (which, for a debug build, is many). It is not an
error to set both --verbose and --quiet. The last one wins.
The log line format is:
<TIMESTAMP>:<LEVEL>:<PID> <THREAD> <SOURCEFILE>:<SOURCELINE>: <MSG>
<TIMESTAMP>
Time the log entry was made. This is expressed in terms of monotonic ms
<LEVEL>
This will be one of 'D', 'I', 'W', 'E', 'F' in increasing order of
severity. If flexnbd is started with the --quiet flag, only 'F' will
be seen. If it is started with the --verbose flag, any from 'I'
upwards will be seen. Only if you have a debug build and start it
with --verbose will you see 'D' entries.
<PID>
This is the process ID.
<THREAD>
flexnbd-proxy is currently single-threaded, so this should be the
same for all lines. That may not be the case in the future.
<SOURCEFILE:SOURCELINE>
Identifies where in the source code this log line can be found.
<MSG>
A short message describing what's happening, how it's being done, or
if you're very lucky why it's going on.
EXAMPLES
The main point of the proxy mode is to allow clients that would otherwise break
when the NBD server goes away (during a migration, for instance) to see a
persistent TCP connection throughout the process, instead of needing its own
reconnection logic.
For maximum reliability, the proxy process would be run on the same machine as
the actual NBD client; an example might look like:
nbd-server-1$ flexnbd serve -l 10.0.0.1 -p 4777 myfile [...]
nbd-client-1$ flexnbd-proxy -l 127.0.0.1 -p 4777 -C 10.0.0.1 -P 4777
nbd-client-1$ nbd-client -c 127.0.0.1 4777 /dev/nbd0
nbd-server-2$ flexnbd listen -l 10.0.0.2 -p 4777 -f myfile [...]
nbd-server-1$ flexnbd mirror --addr 10.0.0.2 -p 4777 [...]
Upon completing the migration, the mirroring and listening flexnbd servers will
both exit. With the proxy mediating requests, this does not break the TCP
connection that nbd-client is holding open. If no requests are in-flight, it
will not notice anything at all; if requests are in-flight, then the reply may
take longer than usual to be returned.
When flexnbd is restarted in serve mode on the second server:
nbd-server-2$ flexnbd serve -l 10.0.0.1 -p 4777 -f myfile [...]
The proxy notices and reconnects, fulfiling any request it has in its buffer.
The data in myfile has been moved between physical servers without the nbd
client process having to be disturbed at all.
READ CACHE
If the --cache option is given at the command line, either without an
argument or with an argument greater than 0, flexnbd-proxy will use a
read-ahead cache. The cache as currently implemented doubles each read
request size, up to a maximum of 2xCACHE_BYTES, and retains the latter
half in a buffer. If the next read request from the client exactly
matches the region held in the buffer, flexnbd-proxy responds from the
cache without making a request to the server.
This pattern is designed to match sequential reads, such as those
performed by a booting virtual machine.
Note: If specifying a cache size, you must use this form:
nbd-client$ flexnbd-proxy --cache=XXXX
That is, the '=' is required. This is a limitation of getopt-long.
If no cache size is given, a size of 4096 bytes is assumed. Caching can
be explicitly disabled by setting a size of 0.
ENVIRONMENT
FLEXNBD_UPSTREAM_TIMEOUT The timeout in seconds for the proxy communicating
with the upstream server defaults to 30 seconds.
BUGS
Should be reported via GitHub.
* https://github.com/BytemarkHosting/flexnbd-c/issues
Current issues include:
* only old-style NBD negotiation is supported;
* only one request may be in-flight at a time;
* all I/O is blocking, and signals terminate the process immediately;
* UNIX socket support is limited to the listen address;
* FLUSH and TRIM commands, and the FUA flag, are not supported;
* DISCONNECT requests do not get passed through to the NBD server;
* no active timeout-retry of requests - we trust the kernel's idea of
failure.
AUTHOR
Originally written by Alex Young <[email protected]>.
Original concept and core code by Matthew Bloch <[email protected]>.
Proxy mode written by Nick Thomas <[email protected]>.
The full commit history is available on GitHub.
SEE ALSO
flexnbd(1), nbd-client(8), xnbd-server(8), xnbd-client(8)
COPYRIGHT
Copyright (c) 2012-2016 Bytemark Hosting Ltd. Free use of this
software is granted under the terms of the GNU General Public License
version 3 or later.