# Transfer adapters for resumable upload / download

## Concept

To allow the uploading and downloading of LFS content to be implemented in more
ways than the current simple HTTP GET/PUT approach. Features that could be
supported by opening this up to other protocols might include:

  - Resumable transfers
  - Block-level de-duplication
  - Delegation to 3rd party services like Dropbox / Google Drive / OneDrive
  - Non-HTTP services

## API extensions

See the [API documentation](../http-v1-batch.md) for specifics. All changes
are optional extras so there are no breaking changes to the API.

The current HTTP GET/PUT system will remain the default. When a version of the
git-lfs client supports alternative transfer mechanisms, it notifies the server
in the API request using the `accept-transfers` field.

If the server also supports one of the mechanisms the client advertised, it may
select one and alter the upload / download URLs to point at resources
compatible with this transfer mechanism. It must also indicate the chosen
transfer mechanism in the response using the `transfer` field.

The URLs provided in this case may not be HTTP, they may be custom protocols.
It is up to each individual transfer mechanism to define how URLs are used.

## Client extensions

### Phase 1: refactoring & abstraction

1. Introduce a new concept of 'transfer adapter'.
2. Adapters can provide either upload or download support, or both. This is
   necessary because some mechanisms are unidirectional, e.g. HTTP Content-Range
   is download only, tus.io is upload only.
3. Refactor our current HTTP GET/PUT mechanism to be the default implementation
   for both upload & download
4. The LFS core will pass oids to transfer to this adapter in bulk, and receive
   events back from the adapter for transfer progress, and file completion.
5. Each adapter is responsible for its own parallelism, but should respect the
   `lfs.concurrenttransfers` setting. For example the default (current) approach
   will parallelise on files (oids), but others may parallelise in other ways
   e.g. downloading multiple parts of the same file at once
6. Each adapter should store its own temporary files. On file completion it must
   notify the core which in the case of a download is then responsible for
   moving a completed file into permanent storage.
7. Update the core to have a registry of available transfer mechanisms which it
   passes to the API, and can recognise a chosen one in the response. Default
   to our refactored original.

### Phase 2: basic resumable downloads

1. Add a client transfer adapter for [HTTP Range headers](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35)
2. Add a range request reference implementation to our integration test server

### Phase 3: basic resumable uploads

1. Add a client transfer adapter for [tus.io](http://tus.io) (upload only)
2. Add a tus.io reference implementation to our integration test server

### Phase 4: external transfer adapters

Ideally we should allow people to add other transfer implementations so that
we don't have to implement everything, or bloat the git-lfs binary with every
custom system possible.

Because Go is statically linked it's not possible to extend client functionality
at runtime through loading libraries, so instead I propose allowing an external
process to be invoked, and communicated with via a defined stream protocol. This
protocol will be logically identical to the internal adapters; the core passing
oids and receiving back progress and completion notifications; just that the
implementation will be in an external process and the messages will be
serialised over streams.

Only one process will be launched and will remain for the entire period of all
transfers. Like internal adapters, the external process will be responsible for
its own parallelism and temporary storage, so internally they can (should) do
multiple transfers at once.

1. Build a generic 'external' adapter which can invoke a named process and
   communicate with it using the standard stream protocol (probably just over
   stdout / stdin)
2. Establish a configuration for external adapters; minimum is an identifier
   (client and server must agree on what that is) and a path to invoke
3. Implement a small test process in Go which simply wraps the default HTTP
   mechanism in an external process, to prove the approach (not in release)


