Eric Wong [Sun, 7 Apr 2024 10:52:38 +0000 (7 10:52 +0000)]
t/lib.perl: fix Perl integration tests w/o installation
We can't rely on `ruby -I $PATH' args being passed to
subprocesses, so rely on the RUBYLIB environment instead.
Eric Wong [Thu, 8 Feb 2024 12:16:31 +0000 (8 12:16 +0000)]
treewide: future-proof frozen_string_literal changes
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/
74d7689/s/?b=add-fsl.perl
Eric Wong [Mon, 1 Jan 2024 10:43:13 +0000 (1 10:43 +0000)]
doc: various updates and disclaimers
Covering my ass from draconian legislation.
Eric Wong [Mon, 6 Nov 2023 05:45:29 +0000 (6 05:45 +0000)]
tests: port back-out-of-upgrade to Perl 5
Another place where we can be faster without adding more
dependencies on Ruby maintaining stable behavior.
Eric Wong [Tue, 9 Jan 2024 21:35:08 +0000 (9 21:35 +0000)]
t/integration: disable proxies when running curl(1)
This was also done in t/test-lib.sh, but using '*' is more
encompassing.
Eric Wong [Tue, 5 Sep 2023 09:14:11 +0000 (5 09:14 +0000)]
update dependency to Ruby 2.5+
We actually need Ruby 2.3+ for `accept_nonblock(exception: false)';
and (AFAIK) we can't easily use a subclass of `Socket' while using
Socket#accept_nonblock to inject WriteSplat support for
`IO#write(*multi_args)'
So just depend on Ruby 2.5+ since all my Ruby is already on the
already-ancient Ruby 2.7+ anyways.
Eric Wong [Tue, 5 Sep 2023 06:43:20 +0000 (5 06:43 +0000)]
kill off remaining kgio uses
kgio is an extra download and shared object which costs users
bandwidth, disk space, startup time and memory. Ruby 2.3+
provides `Socket#accept_nonblock(exception: false)' support
in addition to `exception: false' support in IO#*_nonblock
methods from Ruby 2.1.
We no longer distinguish between TCPServer and UNIXServer as
separate classes internally; instead favoring the `Socket' class
of Ruby for both. This allows us to use `Socket#accept_nonblock'
and get a populated `Addrinfo' object off accept4(2)/accept(2)
without resorting to a getpeername(2) syscall (kgio avoided
getpeername(2) in the same way).
The downside is there's more Ruby-level argument passing and
stack usage on our end with HttpRequest#read_headers (formerly
HttpRequest#read). I chose this tradeoff since advancements in
Ruby itself can theoretically mitigate the cost of argument
passing, while syscalls are a high fixed cost given modern CPU
vulnerability mitigations.
Note: no benchmarks have been run since I don't have a suitable
system.
Eric Wong [Sun, 26 May 2019 22:15:44 +0000 (26 22:15 +0000)]
remove kgio from all read(2) and write(2) wrappers
It's fairly easy given unicorn was designed with synchronous I/O
in mind. The overhead of backtraces from EOFError on
readpartial should be rare given our requirement to only accept
requests from fast, reliable clients on LAN (e.g. nginx or
yet-another-horribly-named-server).
Eric Wong [Sat, 30 Sep 2023 23:50:35 +0000 (30 23:50 +0000)]
README: fix wording
Reported-by: <ideal.water4095@fastmail.com>
Eric Wong [Sun, 10 Sep 2023 20:14:15 +0000 (10 20:14 +0000)]
doc: various updates ahead of the release
The damage unicorn has done to the entire Ruby, Rack and Rails
ecosystems with its ability to tolerate buggy code is
unforgivable. Update the documentation to further discourage
its use and clarify a few wordings noticed along the way.
Eric Wong [Sun, 10 Sep 2023 19:52:03 +0000 (10 19:52 +0000)]
tests: fold early shutdown() tests into t/integration.t
This means fewer redundant tests and more chances to notice
Ruby incompatibilities.
Eric Wong [Sun, 10 Sep 2023 19:37:32 +0000 (10 19:37 +0000)]
tests: move broken app test to Perl 5 integration test
Less Ruby means fewer incompatibilities to worry about with
every new version.
Eric Wong [Sun, 10 Sep 2023 19:21:05 +0000 (10 19:21 +0000)]
tests: fold SO_KEEPALIVE check to Perl 5 integration
No need to startup more processes than necessary.
Eric Wong [Sun, 10 Sep 2023 09:35:09 +0000 (10 09:35 +0000)]
tests: use Time::HiRes `sleep' and `time' everywhere
The time(2) syscall use by CORE::time is inaccurate[1].
It's also easier to read `sleep 0.01' rather than the
longer `select' equivalent.
[1]
a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
Eric Wong [Sun, 10 Sep 2023 09:31:44 +0000 (10 09:31 +0000)]
tests: use more common variable names between tests
Stuff like $u_conf, $daemon_pid, $pid_file, etc. will
reduce cognitive overhead.
Eric Wong [Sun, 10 Sep 2023 09:15:16 +0000 (10 09:15 +0000)]
tests: introduce `do_req' helper sub
While early tests required fine-grained control in trickling
requests, many of our later tests can use a short one-liner
w/o having to spawn curl.
Eric Wong [Sun, 10 Sep 2023 08:27:04 +0000 (10 08:27 +0000)]
tests: rewrite SIGWINCH && SIGTTIN test in Perl 5
No need to deal with full second sleeps, here.
Eric Wong [Sun, 10 Sep 2023 07:13:11 +0000 (10 07:13 +0000)]
tests: port reopen logs test over to Perl 5
Being able to do subsecond sleeps is one welcome advantage
over POSIX (not GNU) sleep(1) in portable Bourne sh.
Eric Wong [Thu, 7 Sep 2023 23:53:58 +0000 (7 23:53 +0000)]
tests: port t/heartbeat-timeout to Perl 5
I absolutely detest and regret adding this feature,
but I'm hell bent on supporting it until the end of days
because we don't break compatibility.
Eric Wong [Thu, 7 Sep 2023 23:18:16 +0000 (7 23:18 +0000)]
tests: port working_directory tests to Perl 5
We can fold a bunch of them into one test to save startup
time, inodes, and FS activity.
Eric Wong [Thu, 7 Sep 2023 22:55:09 +0000 (7 22:55 +0000)]
tests: port some bad config tests to Perl 5
We can fold some tests into one test to save on Perl startup
time (but Ruby startup time is a lost cause).
EW [Tue, 20 Jun 2023 10:46:20 +0000 (20 10:46 +0000)]
unicorn_http_common.rl: use only ASCII spaces for compatibility
Ragel 6.10 on FreeBSD 12.4 amd64 complains and fails on this, yet the same
Ragel version on Debian 11.x i386 and amd64 never has. I suspect this can
fix compatibility on s390x, arm64, armel, and armhf Debian builds:
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=s390x&ver=6.1.0-1&stamp=
1687156375&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=arm64&ver=6.1.0-1&stamp=
1687156478&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armel&ver=6.1.0-1&stamp=
1687156619&file=log
https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armhf&ver=6.1.0-1&stamp=
1687156807&file=log
Fixes:
d5fbbf547203061b (Add some tolerance (RFC2616 sec. 19.3), 2016-10-20)
Eric Wong [Tue, 6 Jun 2023 11:44:29 +0000 (6 11:44 +0000)]
tests: ensure t/random_blob exists before Perl tests
Allow overriding `PROVE=' while we're at it, too; since
development installations of Perl5 may name it `prove5.$MINOR'
or similar.
Eric Wong [Tue, 6 Jun 2023 11:02:29 +0000 (6 11:02 +0000)]
tests: handle $/ assignment deprecation
...by testing less. `env["rack.input"].gets' users are out-of-luck
if they want anything other than "\n" or `nil', I suppose...
`$/' is non-thread-local and thus non-thread-safe, which doesn't
affect unicorn itself, but Ruby deprecates it for
single-threaded code, too, unfortunately.
Rack::Lint doesn't allow separator arguments for #gets, either,
so we can't support that, either...
EW [Tue, 6 Jun 2023 10:09:25 +0000 (6 10:09 +0000)]
t/active-unix-socket: sleep for init(8) to reap worker
Unfortunately, we need a sleep loop here since kill(2) succeeds
on zombies and init(8) doesn't reap the worker soon enough on
a FreeBSD VM.
EW [Tue, 6 Jun 2023 10:09:24 +0000 (6 10:09 +0000)]
t/lib.perl: FreeBSD: ignore accf_* messages
Testers may not have accf_http loaded nor the permissions
to run `kldload accf_http', thus we must ignore these messages.
Eric Wong [Sun, 11 Jun 2023 22:41:21 +0000 (11 22:41 +0000)]
epollexclusive: handle future rb_io_t deprecation
It looks like Ruby 3.3+ will hide rb_io_t internals and
get rid of the venerable `GetOpenFile' macro in favor of
`rb_io_descriptor'. `rb_io_descriptor' has been public
API since Ruby 3.1 and should be safe to use, and
is necessary for `raindrops' (a dependency of ours):
https://yhbt.net/raindrops-public/
20230609104805.39022-1-samuel.williams@oriontransfer.co.nz/
https://bugs.ruby-lang.org/issues/19057#note-17
We'll also avoid an unnecessary call to `rb_io_get_io' in
`get_readers' since `epio' (aka `self') can only be of
the Unicorn::Waiter IO subclass. However, we must still use
`rb_io_get_io' when handling non-self args in `prep_readers'.
Eric Wong [Mon, 5 Jun 2023 10:12:52 +0000 (5 10:12 +0000)]
LISTEN_FDS-inherited sockets are immortal across SIGHUP
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
Eric Wong [Mon, 5 Jun 2023 10:12:51 +0000 (5 10:12 +0000)]
drop redundant IO#close_on_exec=false calls
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
Eric Wong [Mon, 5 Jun 2023 10:12:50 +0000 (5 10:12 +0000)]
tests: move test_upload.rb tests to t/integration.t
The overread tests are ported over, and checksumming alone
is enough to guard against data corruption.
Randomizing the size of `read' calls on the client side will
shake out any boundary bugs on the server side.
Eric Wong [Mon, 5 Jun 2023 10:12:49 +0000 (5 10:12 +0000)]
t/integration.t: switch PUT tests to MD5, reuse buffers
MD5 is faster, and these tests aren't meant to be secure,
they're just for checking for data corruption.
Furthermore, Content-MD5 is a supported HTTP trailer and
we can verify that here to obsolete other tests.
Furthermore, we can reuse buffers on env['rack.input'].read
calls to avoid malloc(3) and GC overhead.
Combined, these give roughly a 3% speedup for t/integration.t
on my system.
Eric Wong [Mon, 5 Jun 2023 10:12:48 +0000 (5 10:12 +0000)]
test_server: drop early_hints test
t/integration.t already is more complete in that it tests
both Rack 2 and 3 along with both possible values of
check_client_connection.
Eric Wong [Mon, 5 Jun 2023 10:12:47 +0000 (5 10:12 +0000)]
early_hints supports Rack 3 array headers
We can hoist out append_headers into a new method and use it in
both e103_response_write and http_response_write.
t/integration.t now tests early_hints with both possible
values of check_client_connection.
Eric Wong [Mon, 5 Jun 2023 10:12:46 +0000 (5 10:12 +0000)]
tests: get rid of sha1sum.rb and rsha1() sh function
These are no longer needed since Perl has long included
Digest::SHA
Eric Wong [Mon, 5 Jun 2023 10:12:45 +0000 (5 10:12 +0000)]
port t/t0116-client_body_buffer_size.sh to Perl 5
While I'm fine with depending on curl for certain things,
there's no need for it here since unicorn has had lazy
rack.input for over a decade, at this point.
Eric Wong [Mon, 5 Jun 2023 10:12:44 +0000 (5 10:12 +0000)]
port t9000-preread-input.sh to Perl 5
Stuffing it into t/integration.t for now so we can save on
startup costs.
Eric Wong [Mon, 5 Jun 2023 10:12:43 +0000 (5 10:12 +0000)]
tests: consistent tcp_start and unix_start across Perl 5 tests
I'll be using Unix sockets more in tests since there's no
risk of system-wide conflicts with TCP port allocation.
Furthermore, curl supports `--unix-socket' nowadays; so
there's little reason to rely on TCP sockets and the conflicts
they bring in tests.
Eric Wong [Mon, 5 Jun 2023 10:12:42 +0000 (5 10:12 +0000)]
tests: check_stderr consistently in Perl 5 tests
The Bourne shell tests did, so lets not let stuff sneak past us.
Eric Wong [Mon, 5 Jun 2023 10:12:41 +0000 (5 10:12 +0000)]
test_exec: drop test_basic and test_config_ru_alt_path
We already have coverage for these basic things elsewhere.
Eric Wong [Mon, 5 Jun 2023 10:12:40 +0000 (5 10:12 +0000)]
test_exec: drop sd_listen_fds emulation test
The Perl 5 tests already rely on this implicitly, and there was
never a point when Perl 5 couldn't emulate systemd behavior.
Eric Wong [Mon, 5 Jun 2023 10:12:39 +0000 (5 10:12 +0000)]
port t0019-max_header_len.sh to Perl 5
This was the final socat requirement for integration tests.
I think curl will remain an optional dependency for tests
since it's probably the most widely-installed HTTP client.
Eric Wong [Mon, 5 Jun 2023 10:12:38 +0000 (5 10:12 +0000)]
tests: use autodie to simplify error checking
autodie is bundled with Perl 5.10+ and simplifies error
checking in most cases. Some subroutines aren't perfectly
translatable and their call sites had to be tweaked, but
most of them are.
Eric Wong [Mon, 5 Jun 2023 10:12:37 +0000 (5 10:12 +0000)]
port t0100-rack-input-tests.sh to Perl 5
Yet another socat dependency gone \o/
Eric Wong [Mon, 5 Jun 2023 10:12:36 +0000 (5 10:12 +0000)]
port t0011-active-unix-socket.sh to Perl 5
Another socat dependency down... I've also started turning
FD_CLOEXEC off on a pipe as a mechanism to detect daemonized
process death in tests.
Eric Wong [Mon, 5 Jun 2023 10:12:35 +0000 (5 10:12 +0000)]
t/integration.t: use start_req to simplify test slighly
Less code is usually better.
Eric Wong [Mon, 5 Jun 2023 10:12:34 +0000 (5 10:12 +0000)]
port t0002-parser-error.sh to Perl 5
Another socat dependency down...
Eric Wong [Mon, 5 Jun 2023 10:12:33 +0000 (5 10:12 +0000)]
port t0000-http-basic.sh to Perl 5
One more socat dependency down...
Eric Wong [Mon, 5 Jun 2023 10:12:32 +0000 (5 10:12 +0000)]
port t0018-write-on-close.sh to Perl 5
This doesn't require restarting, so it's a perfect candidate.
Eric Wong [Mon, 5 Jun 2023 10:12:31 +0000 (5 10:12 +0000)]
support rack 3 multi-value headers
The first step in adding Rack 3 support. Rack supports
multi-value headers via array rather than newlines.
Tested-by: Martin Posthumus <martin.posthumus@gmail.com>
Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/
Eric Wong [Mon, 5 Jun 2023 10:12:30 +0000 (5 10:12 +0000)]
switch unit/test_response.rb to Perl 5 integration test
http_response_write may benefit from API changes for Rack 3
support.
Since there's no benefit I can see from using a unit test,
switch to an integration test to avoid having to maintain the
unit test if our internal http_response_write method changes.
Of course, I can't trust tests written in Ruby since I've had to
put up with a constant stream of incompatibilities over the past
two decades :< Perl is more widely installed than socat[1], and
nearly all the Perl I wrote 20 years ago still works
unmodified today.
[1] the rarest dependency of the Bourne shell integration tests
Eric Wong [Mon, 5 Jun 2023 09:26:53 +0000 (5 09:26 +0000)]
httpdate: fix build with Ruby 2.7 (at least)
<time.h> is still required for gmtime_r(3), and not all versions
of <ruby.h> include <time.h>, already.
Fixes:
a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01)
Eric Wong [Thu, 1 Jun 2023 23:58:13 +0000 (1 23:58 +0000)]
httpdate: favor gettimeofday(2) over time(2) for correctness
While scanning the git@vger.kernel.org mailing list, I've
learned time(2) may return the wrong value in the first 1 to 2.5
ms of every second. While I'm not sure if the Date: response
header matters to anyone, returning the correct time seems
prudent.
Link: https://lore.kernel.org/git/20230320230507.3932018-1-gitster@pobox.com/
Link: https://inbox.sourceware.org/libc-alpha/20230306160321.2942372-1-adhemerval.zanella@linaro.org/T/
Link: https://sourceware.org/bugzilla/show_bug.cgi?id=30200
Eric Wong [Thu, 1 Jun 2023 23:58:11 +0000 (1 23:58 +0000)]
ext: remove a vestige of ruby <2.0 support
The actual `id_clear' declaration was removed last year, but
I missed it's (unused) initialization :x
Fixes:
c56eb04d683e ("drop Ruby 1.9.3 support, require 2.0+ for now")
Eric Wong [Thu, 1 Jun 2023 21:19:35 +0000 (1 21:19 +0000)]
chunk unterminated HTTP/1.1 responses
Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.
We still need to support Ruby 2.4, at least, since Rack 3.0
does. So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
Jeremy Evans [Thu, 1 Jun 2023 15:55:14 +0000 (1 08:55 -0700)]
Support Rack 3 and fix tests on Rack 3
Most changes are to the tests to avoid uppercase characters in header
keys, which are no longer allowed in rack 3 (to allow for O(1) access).
This also changes a few places where an array of headers was used to
switch to a hash, as a hash is requierd in Rack 3.
Newer versions of curl use a 000 http_code for invalid http codes,
so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will
always raise a 500 error.
There is one test that fails on OpenBSD when opening a fifo. This is
unrelated to unicorn as far as I can see, so skip the remaining part
of the test in that case on OpenBSD.
Tests still pass on Rack 2, and presumably Rack 1 as well, though
I didn't test Rack 1.
Co-authored-by: Eric Wong <bofh@yhbt.net>
Eric Wong [Tue, 28 Mar 2023 12:24:37 +0000 (28 12:24 +0000)]
epollexclusive: use maxevents=1 for epoll_wait
This allows us to avoid both malloc (slow) and alloca
(unpredictable stack usage) at the cost of needing to make more
epoll_wait syscalls in a rare case.
In unicorn (and most servers), I expect the most frequent setup
is to have one active listener serving the majority of the
connections, so the most frequent epoll_wait return value would
be 1.
Even with >1 events, any syscall overhead saved by having
epoll_wait retrieve multiple events is dwarfed by Rack app
processing overhead.
Worse yet, if a worker retrieves an event sooner than it can
process it, the kernel (regardless of EPOLLEXCLUSIVE or not) is
able to enqueue another new event to that worker. In this
example where `a' and `b' are both listeners:
U=userspace, K=kernel
K: client hits `a' and `b', enqueues them both (events #1 and #2)
U: epoll_wait(maxevents: 2) => [ a, b ]
K: enqueues another event for `b' (event #3)
U: process_client(a.accept) # this takes a long time
While process_client(a.accept) is happening, `b' can have two
clients pending on a given worker. It's actually better to
leave the first `b' event unretrieved so the second `b'
event can go to the ep->rdllist of another worker.
The kernel is only capable of enqueuing an item if it hasn't
been enqueued. Meaning, it's impossible for epoll_wait to ever
retrieve `[ b, b ]' in one call.
Eric Wong [Thu, 1 Jun 2023 23:58:10 +0000 (1 23:58 +0000)]
http_server: remove close_sockets_on_exec
This has been irrelevant since Ruby 1.9.0+, actually.
Eric Wong [Sat, 1 Oct 2022 22:57:40 +0000 (1 22:57 +0000)]
http_server: detect disk-full when writing PID file
While unlikely, it's possible for write(2) to return a truncated
value or ENOSPC error if the device is full when writing a tiny
PID file.
As optimism has no place in engineering, use IO#write instead of
IO#syswrite since the former will retry on truncation and raise
a exception on ENOSPC.
Eric Wong [Tue, 16 Aug 2022 18:37:22 +0000 (16 18:37 +0000)]
doc: add POP3 archive link, favor AUTH=ANONYMOUS for IMAP
POP3 access is less straightforward since it requires a UUID
cookie embedded into the username to track deletes, and also
requires embedding the mailbox name into the username (since
POP3 has a 1:1 user:folder mapping). Of course, a Tor .onion
endpoint is supported for anonymity.
AUTH=ANONYMOUS is recommended for IMAP since bots were attempting
to use compromised logins to scrape the public mail archives for
private info and burning up my CPU.
Eric Wong [Sat, 25 Dec 2021 17:45:22 +0000 (25 17:45 +0000)]
unicorn 6.1.0
This release reduces CPU usage for Linux 4.5+ in most cases.
See "[PATCH 6/6] use EPOLLEXCLUSIVE on Linux 4.5+" for more details:
https://yhbt.net/unicorn-public/
20211001030923.26705-7-bofh@yhbt.net/
There's a couple of updates for Ruby 3.1, but we've finally
started relying on Ruby 2.0.0 features after 9 years :P
(so Ruby 1.9.3 users are stuck with older versions).
And the usual round of doc updates and some build speedups.
13 changes by the Bozo Doofus maintainer since v6.0.0:
test_util: less excessive encoding tests
drop Ruby 1.9.3 support, require 2.0+ for now
drop unnecessary IO#close_on_exec=true assignment
extconf.rb: get rid of unnecessary checks
makefile: reduce unnecessary rebuilds
HACKING: drop outdated information about pandoc
http_server: get rid of Process.ppid check
worker_loop: get rid of select() avoidance hack
use EPOLLEXCLUSIVE on Linux 4.5+
allow Ruby to deduplicate remaining globals
epollexclusive: remove rb_gc_force_recycle call
drop Ruby version warning, fix speling errer
doc: v3 .onion updates, nntp => nntps, minor wording changes
Eric Wong [Sat, 25 Dec 2021 17:41:43 +0000 (25 17:41 +0000)]
doc: v3 .onion updates, nntp => nntps, minor wording changes
Tor v2 .onion support is deprecated, so we're stuck with longer
(but more secure :P) v3 .onion names. Use NNTPS where available
instead of NNTP to reduce the likelyhood of MiTM censorship and
injection.
Eric Wong [Sat, 25 Dec 2021 17:41:42 +0000 (25 17:41 +0000)]
drop Ruby version warning, fix speling errer
The warning was probably lost in the noise of the build, anyways.
Eric Wong [Sat, 25 Dec 2021 17:41:41 +0000 (25 17:41 +0000)]
epollexclusive: remove rb_gc_force_recycle call
It's deprecated in Ruby 3.1+, and probably not relevant for
past versions.
Eric Wong [Fri, 22 Oct 2021 04:05:16 +0000 (22 04:05 +0000)]
allow Ruby to deduplicate remaining globals
Most of these are bound to be used in Rack/Rails/apps/gems,
(though possibly with different encodings). Give Ruby
a chance to deduplicate them, at least.
Eric Wong [Fri, 1 Oct 2021 03:09:23 +0000 (1 03:09 +0000)]
use EPOLLEXCLUSIVE on Linux 4.5+
While the capabilities of epoll cannot be fully exploited given
our primitive design; avoiding thundering herd wakeups on larger
SMP machines while below 100% utilization is possible with
Linux 4.5+.
With this change, only one worker wakes up per-connect(2)
(instead of all of them via select(2)), avoiding the thundering
herd effect when the system is mostly idle.
Saturated instances should not notice the difference if they
rarely had multiple workers sleeping in select(2). This change
benefits non-saturated instances.
With 2 parallel clients and 8 workers on a nominally (:P)
8-core CPU (AMD FX-8320), the uconnect.perl test script
invocation showed a reduction from ~3.4s to ~2.5s when
reading an 11-byte response body:
echo worker_processes 8 >u.conf.rb
bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \
test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb
time perl -I lib -w test/benchmark/uconnect.perl \
-n 100000 -c 2 /tmp/u.sock
Times improve less as "-c" increases for uconnect.perl (system
noise and timings are inconsistent). The benefit of this change
should be more noticeable on systems with more workers (and
more cores).
I wanted to use EPOLLET (Edge-Triggered) to further reduce
syscalls, here, (similar to the old select()-avoidance bet) but
that would've either added too much complexity to deduplicate
wakeup sources, or run into the same starvation problem we
solved in April 2020[1].
Since the kernel already has the complexity and deduplication
built-in for Level-Triggered epoll support, we'll just let the
kernel deal with it.
Note: do NOT take this as an example of how epoll should be used
in a sophisticated server. unicorn is primitive by design and
cannot use threads nor handle multiple clients at once, thus it
it only uses epoll in this extremely limited manner.
Linux 4.5+ users will notice a regression of one extra epoll FD
per-worker and at least two epoll watches, so
/proc/sys/fs/epoll/max_user_watches may need to be changed along
with RLIMIT_NOFILE.
This change has also been tested on Linux 3.10.x (CentOS 7.x)
and FreeBSD 11.x to ensure compatibility with systems without
EPOLLEXCLUSIVE.
Various EPOLLEXCLUSIVE discussions over the years:
https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..
20211001&x=t&o=-1
[1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
Eric Wong [Fri, 1 Oct 2021 03:09:22 +0000 (1 03:09 +0000)]
worker_loop: get rid of select() avoidance hack
It doesn't seem to do anything since commit
221340c4ebc15666
(prevent single listener from monopolizing a worker, 2020-04-16).
Eric Wong [Fri, 1 Oct 2021 03:09:21 +0000 (1 03:09 +0000)]
http_server: get rid of Process.ppid check
It's actually been unnecessary since commit
6f6e4115b4bb03e5
(rework master-to-worker signaling to use a pipe, 2013-12-09)
Eric Wong [Fri, 1 Oct 2021 03:09:20 +0000 (1 03:09 +0000)]
HACKING: drop outdated information about pandoc
It's been outdated since
d9b5943af26d5df5 (doc: replace
pandoc-"Markdown" with real manpages, 2019-12-15)
Eric Wong [Fri, 1 Oct 2021 03:09:19 +0000 (1 03:09 +0000)]
makefile: reduce unnecessary rebuilds
For the most part, header changes shouldn't trigger extconf.rb
reruns.
Eric Wong [Fri, 1 Oct 2021 03:09:18 +0000 (1 03:09 +0000)]
extconf.rb: get rid of unnecessary checks
SIZEOF_*, *2NUM and NUM2* should all be defined by ruby.h and
dependencies it pulls in since Ruby 2.0 and possibly earlier.
INT_MAX and LLONG_MAX are in limits.h which is POSIX.
HAVE_GMTIME_R is already defined by ruby/config.h, so we
shouldn't have to check for it, either.
Combined, these changes speed up extconf.rb by several seconds.
Eric Wong [Tue, 14 Sep 2021 23:39:23 +0000 (14 23:39 +0000)]
drop unnecessary IO#close_on_exec=true assignment
Ruby 2.0+ sets FD_CLOEXEC by default on all FDs.
Eric Wong [Tue, 14 Sep 2021 23:39:22 +0000 (14 23:39 +0000)]
drop Ruby 1.9.3 support, require 2.0+ for now
Ruby 1.9.3 was released nearly a decade ago, so there's probably
few (if any) legacy users left, and they can continue using old
versions of unicorn. We'll be able to take advantage of some
Ruby 2.0+-only features down the road (and hopefully 2.3+).
Also, I no longer have a installation of Ruby 1.8 and getting it
working probably isn't worth the effort, so 4.x support is gone.
Eric Wong [Fri, 30 Apr 2021 18:17:33 +0000 (30 18:17 +0000)]
test_util: less excessive encoding tests
Ruby's handling of encodings hasn't changed much in over
a decade and these tests haven't failed for me since August 2013:
https://yhbt.net/unicorn-public/
9af083d7f6b97c0f5ebbdd9a42b58478a6f874b7/s/
So lets take a small step in reducing energy consumption
and save potential developers over 10s of CPU time.
Eric Wong [Wed, 17 Mar 2021 06:15:10 +0000 (17 02:15 -0400)]
unicorn 6.0.0 - no more recycling Rack env
This release allocates a new Rack `env' hash for every request.
This is done for safety with internally-(thread|event)-using Rack
apps which expect to use `env' after the normal Rack response is
complete, but without relying on rack.hijack[1]. Thanks to
Dirkjan Bussink <d.bussink@gmail.com> for the patch:
https://yhbt.net/unicorn-public/
66A68DD8-83EF-4C7A-80E8-
3F1F7AB31670@github.com/
The major version is bumped since:
1) there are performance regressions for some simple Rack apps
2) unsupported 3rd-party monkey patches which previously
relied on this behavior may be broken (our version of
OobGC was).
The test suite is also more reliable on multi-core systems
and Ruby 3.x.
[1] thread from 2017 around rack.hijack safety:
https://yhbt.net/unicorn-public/CAAtdryPG3nLuyo0jxfYW1YHu1Q+ZpkLkd4KdWC8vA46B5haZxw@mail.gmail.com/
Eric Wong [Mon, 15 Mar 2021 06:03:50 +0000 (15 02:03 -0400)]
Merge remote-tracking branch 'origin/master' into v6-wip
* origin/master:
tests: force blocking I/O for Ruby 3.x
bofh@yhbt.net [Sun, 14 Mar 2021 23:17:24 +0000 (14 23:17 +0000)]
tests: force blocking I/O for Ruby 3.x
Otherwise we get test failures since we use sysread
and syswrite in many places
Eric Wong [Sat, 13 Mar 2021 02:14:16 +0000 (12 22:14 -0400)]
http_request: drop unnecessary #clear call
Since we allocate a new request object for each request,
the #clear call is now unnecessary
Dirkjan Bussink [Mon, 8 Mar 2021 08:51:09 +0000 (8 09:51 +0100)]
Allocate a new request for each client
This removes the reuse of the parser between requests. Reusing these is
risky in the context of running any other threads within the unicorn
process, also for threads that run background tasks.
If any other thread accidentally grabs hold of the request it can modify
things for the next request in flight.
The downside here is that we allocate more for each request, but that is
worth the trade off here and the security risk we otherwise would carry
to leaking wrong and incorrect data.
Eric Wong [Sat, 13 Mar 2021 03:00:28 +0000 (13 00:00 -0300)]
bump version for 6.x development
While no incompatible changes to any supported configuration
files are planned, some deep internal data structures will
change and there are likely some 3rd-party monkey patches broken
by this.
Eric Wong [Sat, 13 Mar 2021 02:08:46 +0000 (12 22:08 -0400)]
test/test_helper: only unlink redirected logs from parent
We don't want at_exit firing in child processes and never wanted
it. This is apparently a long standing bug in the tests that
only started causing test_worker_dies_on_dead_master failures
for me. I assume it's only showing up now for me due to kernel
scheduler changes, since I've been using the same 4-core CPU for
~11 years, now.
Eric Wong [Thu, 24 Dec 2020 20:28:58 +0000 (24 20:28 +0000)]
unicorn 5.8.0 - rack.after_reply support
This release supports env['rack.after_reply'] which allows
rack middleware to pass lambdas to be executed after the client
connection is closed, matching functionality in Puma.
Thanks to Blake Williams for this patch:
https://yhbt.net/unicorn-public/
9873E53C-04D3-4759-9678-
CA17DBAEF7B7@blakewilliams.me/
The top-level of our website is now simpler and no longer
redundant with the contents of https://yhbt.net/unicorn/README.html
(which contains the old content)
Eric Wong [Thu, 24 Dec 2020 20:21:02 +0000 (24 20:21 +0000)]
build: publish_doc: remove created.rid and index.html from site
created.rid has no business being published anyways, and
index.html is redundant given README.html. Avoid confusing
search engines with identical documents at different URLs.
Our "homepage" is just a directory listing, now, which should
help discoverability of non-HTML docs. Old content is at
https://yhbt.net/unicorn/README.html
Blake Williams [Tue, 8 Dec 2020 21:47:16 +0000 (8 16:47 -0500)]
Add rack.after_reply functionality
This adds `rack.after_reply` functionality which allows rack middleware
to pass lambdas that will be executed after the client connection has
been closed.
This was driven by a need to perform actions in a request that shouldn't
block the request from completing but also don't make sense as
background jobs.
There is prior art of this being supported found in a few gems, as well
as this functionality existing in other rack based servers (e.g. Puma).
[ew: check if `env' is set in ensure statement]
Acked-by: Eric Wong <e@80x24.org>
Eric Wong [Tue, 8 Sep 2020 02:07:18 +0000 (8 02:07 +0000)]
unicorn 5.7.0
Relaxed Ruby version requirements for Ruby 3.0.0dev.
Thanks to Jean Boussier for testing
Eric Wong [Tue, 8 Sep 2020 02:02:26 +0000 (8 02:02 +0000)]
PATCH: doc: add IMAP/IMAPS mailbox info
Apparently, an IMAP server happened at some point and somehow
stays running. Maybe more users have IMAP clients than NNTP
clients...
Jean Boussier [Fri, 4 Sep 2020 12:34:37 +0000 (4 14:34 +0200)]
Update ruby_version requirement to allow ruby 3.0
Ruby just recently bump the master version to 3.0.
This requirement bump is necessary to test unicorn
against ruby master.
[ew: wrap at <80 columns for hackers with poor eyesight]
Acked-by: Eric Wong <bofh@yhbt.net>
Eric Wong [Sun, 26 Jul 2020 01:57:59 +0000 (26 01:57 +0000)]
build: revamp and avoid unnecessary rebuilds
We can limit the amount of Ruby-version-specific code to
just the stuff in ext/* and bin/*, reducing I/O traffic
and FS + page cache footprint.
Furthermore, rely on GNU make behavior to copy all the necessary
files so we don't trigger unnecessary extconf.rb invocations
just by touching a .rb file in lib.
Eric Wong [Sun, 26 Jul 2020 01:57:58 +0000 (26 01:57 +0000)]
test_helper: support TAIL= env for watching tests
This can be useful for diagnosing failures, especially since
GNU tail supports inotify these days.
Eric Wong [Sun, 26 Jul 2020 01:47:37 +0000 (26 01:47 +0000)]
unicorn 5.6.0 - early_hints support
This release adds support for the early_hints configurator
directive for the 'rack.early_hints' API used by Rails 5.2+.
Thanks to Jean Boussier for the patch.
Link: https://yhbt.net/unicorn-public/242F0859-0F83-4F14-A0FF-5BE392BB01E6@shopify.com/
Eric Wong [Thu, 23 Jul 2020 04:17:35 +0000 (23 04:17 +0000)]
configurator: SIGHUP resets early_hints if unset
If a user removes "early_hints" entirely from the config file, a
SIGHUP needs to restore the default value. This is consistent
with the behavior of all the other configuration variables.
Cc: Jean Boussier <jean.boussier@gmail.com>
Eric Wong [Thu, 23 Jul 2020 04:17:34 +0000 (23 04:17 +0000)]
test_server: test_early_hints: fix test reliability
IO#sysread may only capture the 103 response and return before
the server can send the 200. Since we don't support persistent
connections, we can just use IO#read to rely on the server
giving us an EOF after the 200 is sent.
Cc: Jean Boussier <jean.boussier@gmail.com>
Jean Boussier [Thu, 16 Jul 2020 09:39:30 +0000 (16 11:39 +0200)]
Add early hints support
While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.
The 103 Early Hints response code is specified in RFC 8297.
Eric Wong [Mon, 27 Apr 2020 02:46:26 +0000 (27 02:46 +0000)]
unicorn 5.5.5
This release fixes a bug for users of multiple listeners setups
where a busy listen socket could starve other listeners.
Thanks to Stan Hu for reporting and testing.
No need to upgrade if you're using a single listen socket.
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
Eric Wong [Thu, 16 Apr 2020 09:24:57 +0000 (16 09:24 +0000)]
prevent single listener from monopolizing a worker
In setups with multiple listeners, it's possible for our greedy
select(2)-avoidance optimization to get pinned on a single, busy
listener and starve the other listener(s).
Prevent starvation by retrying the select(2)-avoidance
optimization if and only if all listeners were active. This
should have no effect on the majority of deployments with only a
single listener.
Thanks to Stan Hu for reporting and testing.
Reported-by: Stan Hu <stanhu@gmail.com>
Tested-by: Stan Hu <stanhu@gmail.com>
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
Eric Wong [Tue, 24 Mar 2020 22:07:41 +0000 (24 22:07 +0000)]
unicorn 5.5.4
One change to improve RFC 7230 conformance in the HTTP parser:
https://yhbt.net/unicorn-public/
20200319022823.32472-1-bofh@yhbt.net/
Eric Wong [Thu, 19 Mar 2020 02:28:23 +0000 (19 02:28 +0000)]
http: improve RFC 7230 conformance
We need to favor "Transfer-Encoding: chunked" over
"Content-Length" in the request header if they both exist.
Furthermore, we now reject redundant chunking and cases where
"chunked" is not the final encoding.
We currently do not and have no plans to decode "gzip",
"deflate", or "compress" encoding as described by RFC 7230.
That's a job more appropriate for middleware, anyways.
cf. https://tools.ietf.org/html/rfc7230
https://www.rfc-editor.org/errata_search.php?rfc=7230
Eric Wong [Fri, 31 Jan 2020 20:38:10 +0000 (31 20:38 +0000)]
unicorn 5.5.3
Documentation updates to switch bogomips.org to yhbt.net since
the .org TLD won't be affordable in the near future.
There's also a few minor test cleanups.
Eric Wong [Mon, 20 Jan 2020 09:54:21 +0000 (20 09:54 +0000)]
test_upload: use spawn to simplify redirects
We can start using Ruby 1.9 APIs, nowadays
Eric Wong [Mon, 20 Jan 2020 09:08:14 +0000 (20 09:08 +0000)]
test_helper: remove unused `chunked_spawn'
It was added nearly 11 years ago in commit
6945342a1f0a4caa
("Transfer-Encoding: chunked streaming input support") but
never used.
Eric Wong [Mon, 20 Jan 2020 08:57:02 +0000 (20 08:57 +0000)]
test/exec/test_exec: bring worker_processes down to 2
My hardware gets worse and worse every year :<