: Rate limiting (!877) · Merge Requests · funkwhale / funkwhale

: Rate limiting

This implements a rate-limit feature

Documentation (or see below)
Customizable limits per endpoints / type of operation / client (anonymous/authenticated)
/api/v1/rate-limit endpoint to expose limits and current usage of the client
X-RateLimit-* headers in response to expose limits and current usage of the client
Handle 429 answer in the front-end and display a proper error message

Default limits as found in common.py are rather high to avoid disruption on existing pods. We can tweak them depending on the feedback we receive, and admins can also customize these individually, by adding a THROTTLING_RATES=<scope1>=<rate1>,<scope2>=<rate2> environment variables. E.g, to to increase the number of reports anonymous users can submit: THROTTLING_RATES=anonymous-reports=100/h.

cc @funkwhale/reviewers-python @funkwhale/reviewers-front

Front-end result

Peek_2019-09-11_10-41

Swagger Documentation

Depending on server configuration, pods running Funkwhale 0.20 and higher may rate-limit incoming requests to prevent abuse and improve the stability of service. Requests that are dropped because of rate-limiting receive a 429 HTTP response.

The limits themselves vary depending on:

The client: anonymous requests are subject to lower limits than authenticated requests
The operation being performed: Write and delete operations, as performed with DELETE, POST, PUT and PATCH HTTP methods are subject to lower limits

Those conditions are used to determine the scope of the request, which in turns determine the limit that is applied. For instance, authenticated POST requests are bound to the authenticated-create scope, with a default limit of 1000 requests/hour, but anonymous POST requests are bound to the anonymous-create scope, with a lower limit of 1000 requests/day.

A full list of scopes with their corresponding description, and the current usage data for the client performing the request is available via the /api/v1/rate-limit endpoint.

Additionally, we include HTTP headers on all API response to ensure API clients can understand:

what scope was bound to a given request
what is the corresponding limit
how much similar requests can be sent before being limited
and how much time they should wait if they have been limited

Rate limiting headers
Header	Example value	Description value
`X-RateLimit-Limit`	50	The number of allowed requests whithin a given period
`X-RateLimit-Duration`	3600	The time window, in seconds, during which those requests are accounted for.
`X-RateLimit-Scope`	login	The name of the scope as computed for the request
`X-RateLimit-Remaining`	42	How many requests can be sent with the same scope before the limit applies
`X-RateLimit-Available` (if `X-RateLimit-Remaining` is 0)	1568126189	A timestamp indicating when the client can retry
`X-RateLimit-AvailableSeconds` (if `X-RateLimit-Remaining` is 0)	3543	How many seconds to wait before a retry
`X-RateLimit-Reset`	1568126089	A timestamp indicating when `X-RateLimit-Remaining` will return to its higher possible value
`X-RateLimit-ResetSeconds`	3599	How many seconds to wait before `X-RateLimit-Remaining` returns to its higher possible value

Closes #261

This implements a rate-limit feature

- [x] [Documentation](http://funkwhale.pages.funkwhale.audio/-/funkwhale/-/jobs/28688/artifacts/docs-review/swagger/index.html) (or see below)
- [x] Customizable limits per endpoints / type of operation / client (anonymous/authenticated)
- [x] `/api/v1/rate-limit` endpoint to expose limits and current usage of the client
- [x] `X-RateLimit-*` headers in response to expose limits and current usage of the client
- [x] Handle 429 answer in the front-end and display a proper error message

Default limits as found in `common.py` are rather high to avoid disruption on existing pods. We can tweak them depending on the feedback we receive, and admins can also customize these individually, by adding a `THROTTLING_RATES=<scope1>=<rate1>,<scope2>=<rate2>` environment variables. E.g, to to increase the number of reports anonymous users can submit: `THROTTLING_RATES=anonymous-reports=100/h`.

cc @funkwhale/reviewers-python @funkwhale/reviewers-front

Front-end result
----------------

![Peek_2019-09-11_10-41](/uploads/20d05fc58df779fd98a829560a48c4f3/Peek_2019-09-11_10-41.mp4)

Swagger Documentation
---------------------

Depending on server configuration, pods running Funkwhale 0.20 and higher may rate-limit incoming
requests to prevent abuse and improve the stability of service. Requests that are dropped because of rate-limiting
receive a 429 HTTP response.

The limits themselves vary depending on:

- The client: anonymous requests are subject to lower limits than authenticated requests
- The operation being performed: Write and delete operations, as performed with DELETE, POST, PUT and PATCH HTTP methods are subject to lower limits

Those conditions are used to determine the scope of the request, which in turns determine the limit that is applied.
For instance, authenticated POST requests are bound to the `authenticated-create` scope, with a default limit of
1000 requests/hour, but anonymous POST requests are bound to the `anonymous-create` scope, with a lower limit of 1000 requests/day.

A full list of scopes with their corresponding description, and the current usage data for the client performing the request
is available via the `/api/v1/rate-limit` endpoint.

Additionally, we include HTTP headers on all API response to ensure API clients can understand:

- what scope was bound to a given request
- what is the corresponding limit
- how much similar requests can be sent before being limited
- and how much time they should wait if they have been limited

<table>
    <caption>Rate limiting headers</caption>
    <thead>
    <th>Header</th>
    <th>Example value</th>
    <th>Description value</th>
    </thead>
    <tbody>
    <tr>
        <td><code>X-RateLimit-Limit</code></td>
        <td>50</td>
        <td>The number of allowed requests whithin a given period</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-Duration</code></td>
        <td>3600</td>
        <td>The time window, in seconds, during which those requests are accounted for.</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-Scope</code></td>
        <td>login</td>
        <td>The name of the scope as computed for the request</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-Remaining</code></td>
        <td>42</td>
        <td>How many requests can be sent with the same scope before the limit applies</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-Available</code> (if <code>X-RateLimit-Remaining</code> is 0)</td>
        <td>1568126189</td>
        <td>A timestamp indicating when the client can retry</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-AvailableSeconds</code> (if <code>X-RateLimit-Remaining</code> is 0)</td>
        <td>3543</td>
        <td>How many seconds to wait before a retry</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-Reset</code></td>
        <td>1568126089</td>
        <td>A timestamp indicating when <code>X-RateLimit-Remaining</code> will return to its higher possible value</td>
    </tr>
    <tr>
        <td><code>X-RateLimit-ResetSeconds</code></td>
        <td>3599</td>
        <td>How many seconds to wait before <code>X-RateLimit-Remaining</code> returns to its higher possible value</td>
    </tr>
    </tbody>
</table>

Edited Sep 11, 2019 by Agate

: Rate limiting

Front-end result

Swagger Documentation

Revert this merge request

Cherry-pick this merge request