Slow /federation/actors/<username> endpoint makes requests to remote pods slow or unreliable
I'm on Funkwhale 1.2.8 with the multi-container setup. I instrumented the Nginx config with the extra time fields shown at https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#setting-up-the-access-log to see this issue better.
Even when there's not enough activity on the server to seem to be able to tie up the 8 API workers I set up, I'm seeing that requests to /federation/actors/<username>
can take a long time. That time all goes to waiting for the API server to send the headers back to nginx.
Since the flow for a lot of federated actions is my browser -> my FW nginx -> my FW API server -> remote Funkwhale -> back to my FW nginx to fetch /federation/actors/<username>
-> back to my FW API (and then the appropriate returns), this ends up making the original operation slow, or even causing it to time out if the remote pod doesn't want to wait for my pod to give it my actor information.
Here's an example of it being oddly slow:
fw-api_1 | 2022-12-07 22:57:35,132 funkwhale_api.music.views INFO Refetching music.Artist:6726 at https://buzzworkers.com/federation/music/artists/38acc7a0-04db-4059-bd77-0f5394ef3c18…
fw-nginx_1 | 172.16.238.2 - - [07/Dec/2022:22:57:45 +0000] "GET /api/v1/artists/6726/?refresh=true HTTP/1.1" 200 1311 "https://music.novak.network/library/artists/6726/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" rt="10.719" uct="0.002" uht="10.717" urt="10.719"
fw-nginx_1 | 172.16.238.2 - - [07/Dec/2022:22:57:45 +0000] "GET /federation/actors/service HTTP/1.1" 200 1469 "-" "python-requests (funkwhale/1.2.8; +https://buzzworkers.com)" rt="9.652" uct="0.001" uht="9.649" urt="9.651"
Nginx logs requests when they finish, and rt
here is how long ago they came in, in seconds. The GET my browser made to /api/v1/artists/6726/?refresh=true
arrived, then immediately my API server logged an INFO line, then within a second the request to /federation/actors/service
from the remote pod came in. That request sat for around 9 seconds waiting for my pod's API server to answer it, which it did (with a 200 OK).
Nginx seems to have the outer refresh request finishing first, somehow, given the log line order, but I'm pretty sure that my server is waiting on the remote server, which is waiting again on my server, for that 9 seconds.
I'll see if I can catch this actually causing a federated action to fail; I suspect this is the cause of my having to repeatedly try to play any federated music to get it to actually make it over to my pod. Is there any federated music I can try that I should be able to play, to test with? I usually pull from tanukitunes.com, but its API seems to be down today.
Is it safe to make nginx cache the /federation/actors/<username>
response for a few hours, to try and speed this up? And why would it be so slow in the first place?