dom/docs/streams.md

   1 # Implementing specifications using WHATWG Streams API
   2
   3 [Streams API](https://streams.spec.whatwg.org/) is [a modern way to produce and consume data progressively and asynchronously](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API). Multiple specifications are starting to use it, namely [Fetch](https://fetch.spec.whatwg.org/), [File Stream](https://fs.spec.whatwg.org/), [WebTransport](https://w3c.github.io/webtransport/), and so on. This documentation will briefly explain how to implement such specifications in Gecko.
   4
   5 ## Calling functions on stream objects
   6
   7 You can mostly follow the steps in a given spec as-is, as the implementation in Gecko is deliberately written in a way that a given spec prose can match 1:1 to a function call. Let's say the spec says:
   8
   9 > [Enqueue](https://streams.spec.whatwg.org/#readablestream-enqueue) `view` into `stream`.
  10
  11 The prose can be written in C++ as:
  12
  13 ```cpp
  14 stream->EnqueueNative(cx, view, rv);
  15 ```
  16
  17 Note that the function name ends with `Native` to disambiguate itself from the Web IDL `enqueue` method. See the [list below](#mapping-spec-terms-to-functions) for the complete mapping between spec terms and functions.
  18
  19 ## Creating a stream
  20
  21 The stream creation can be generally done by calling `CreateNative()`. You may need to call something else if the spec:
  22
  23 * Wants a byte stream and uses the term "Set up with byte reading support". In that case you need to call `ByteNative` variant.
  24 * Defines a new interface that inherits the base stream interfaces. In this case you need to define a subclass and call `SetUpNative()` inside its init method.
  25    * To make the cycle collection happy, you need to pass `HoldDropJSObjectsCaller::Explicit` to the superclass constructor and call `mozilla::HoldJSObjects(this)`/`mozilla::DropJSObjects(this)` respectively in the constructor/destructor.
  26
  27 Both `CreateNative()`/`SetUpNative()` functions require an argument to implement custom algorithms for callbacks, whose corresponding spec phrases could be:
  28
  29 > 1. Let `readable` be a [new](https://webidl.spec.whatwg.org/#new) [`ReadableStream`](https://streams.spec.whatwg.org/#readablestream).
  30 > 1. Let `pullAlgorithm` be the following steps:
  31 >    1. (...)
  32 > 1. Set up `stream` with `pullAlgorithm` set to `pullAlgorithm`.
  33
  34 This can roughly translate to the following C++:
  35
  36 ```cpp
  37 class MySourceAlgorithms : UnderlyingSourceAlgorithmsWrapper {
  38    already_AddRefed<Promise> PullCallbackImpl(
  39       JSContext* aCx, ReadableStreamController& aController,
  40       ErrorResult& aRv) override;
  41 };
  42
  43 already_AddRefed<ReadableStream> CreateMyReadableStream(
  44    JSContext* aCx, nsIGlobalObject* aGlobal, ErrorResult& aRv) {
  45    // Step 2: Let pullAlgorithm be the following steps:
  46    auto algorithms = MakeRefPtr<MySourceAlgorithms>();
  47
  48    // Step 1: Let readable be a new ReadableStream.
  49    // Step 3: Set up stream with pullAlgorithm set to pullAlgorithm.
  50    RefPtr<ReadableStream> readable = ReadableStream::CreateNative(
  51       aCx,
  52       aGlobal,
  53       *algorithms,
  54       /* aHighWaterMark */ Nothing(),
  55       /* aSizeAlgorithm */ nullptr,
  56       aRv
  57    );
  58 }
  59 ```
  60
  61 Note that the `new ReadableStream()` and "Set up" steps are done together inside `CreateNative()` for convenience. For subclasses this needs to be split again:
  62
  63 ```cpp
  64 class MyReadableStream : public ReadableStream {
  65  public:
  66    MyReadableStream(nsIGlobalObject* aGlobal)
  67       : ReadableStream(aGlobal, ReadableStream::HoldDropJSObjectsCaller::Explicit) {
  68       mozilla::HoldJSObjects(this);
  69    }
  70
  71    ~MyReadableStream() {
  72       mozilla::DropJSObjects(this);
  73    }
  74
  75    void Init(ErrorResult& aRv) {
  76       // Step 2: Let pullAlgorithm be the following steps:
  77       auto algorithms = MakeRefPtr<MySourceAlgorithms>();
  78
  79       // Step 3: Set up stream with pullAlgorithm set to pullAlgorithm.
  80       //
  81       // NOTE:
  82       // For now there's no SetUpNative but only SetUpByteNative.
  83       // File a bug on DOM: Streams if you need to create a subclass
  84       // for non-byte ReadableStream.
  85       SetUpNative(aCx, *algorithms, Nothing(), nullptr, aRv);
  86    }
  87 }
  88 ```
  89
  90 After creating the stream with the algorithms, the rough flow will look like this:
  91
  92 ```{mermaid}
  93 sequenceDiagram
  94    JavaScript->>ReadableStream: await reader.read()
  95    ReadableStream->>UnderlyingSourceAlgorithmsWrapper: PullCallback()
  96    UnderlyingSourceAlgorithmsWrapper->>(Data source): (implementation detail)
  97    NOTE left of (Data source): (Can be file IO, network IO, etc.)
  98    (Data source)->>UnderlyingSourceAlgorithmsWrapper: (notifies back)
  99    UnderlyingSourceAlgorithmsWrapper->>ReadableStream: EnqueueNative()
 100    ReadableStream->>JavaScript: Resolves reader.read()
 101 ```
 102
 103 ### Implementing the callbacks
 104
 105 As the flow says, the real implementation will be done inside the algorithms, in this case PullCallbackImpl(). Let's say there's a spec term:
 106
 107 > 1. Let `pullAlgorithm` be the following steps:
 108 >    1. [Enqueue](https://streams.spec.whatwg.org/#readablestream-enqueue) a JavaScript string value "Hello Fox!".
 109
 110 This can translate to the following C++:
 111
 112 ```cpp
 113 class MySourceAlgorithms : UnderlyingSourceAlgorithmsWrapper {
 114    // Step 1: Let `pullAlgorithm` be the following steps:
 115    already_AddRefed<Promise> PullCallbackImpl(
 116       JSContext* aCx, ReadableStreamController& aController, ErrorResult& aRv) {
 117       RefPtr<ReadableStream> stream = aController.Stream();
 118
 119       // Step 1.1: Enqueue a JavaScript string value "Hello Fox!".
 120       JS::Rooted<JSString*> hello(aCx, JS_NewStringCopyZ(aCx, "Hello Fox!"));
 121       stream->EnqueueNative(aCx, JS::StringValue(hello), aRv);
 122
 123       // Return a promise if the task is asynchronous, or nullptr if not.
 124       return nullptr;
 125
 126       // NOTE:
 127       // Please don't use aController directly, as it's more for JavaScript.
 128       // The *Native() functions are safer with additional assertions and more
 129       // automatic state management.
 130       // Please file a bug if there's no *Native() function that fulfills your need.
 131       // In the future this function should receive a ReadableStream instead.
 132
 133       // Also note that you'll need to touch JS APIs frequently as the functions
 134       // often expect JS::Value.
 135    };
 136 };
 137 ```
 138
 139 Note that `PullCallbackImpl` returns a promise. The function will not be called again until the promise resolves. The call sequence would be roughly look like the following with repeated read requests:
 140
 141 1. `await read()` from JS
 142 1. `PullCallbackImpl()` call, which returns a Promise
 143 1. The second `await read()` from JS
 144 1. (Time flies)
 145 1. The promise resolves
 146 1. The second `PullCallbackImpl()` call
 147
 148 The same applies to write and transform callbacks in `WritableStream` and `TransformStream`, except they use `UnderlyingSinkAlgorithmsWrapper` and `TransformerAlgorithmsWrapper` respectively.
 149
 150 ## Exposing existing XPCOM streams as WHATWG Streams
 151
 152 You may simply want to expose an existing XPCOM stream to JavaScript without any more customization. Fortunately there are some helper functions for this. You can use:
 153
 154 * `InputToReadableStreamAlgorithms` to send data from nsIAsyncInputStream to ReadableStream
 155 * `WritableStreamToOutputAlgorithms` to receive data from WritableStream to nsIAsyncOutputStream
 156
 157 The usage would look like the following:
 158
 159 ```cpp
 160 // For nsIAsyncInputStream:
 161 already_AddRefed<ReadableStream> ConvertInputStreamToReadableStream(
 162    JSContext* aCx, nsIGlobalObject* aGlobal, nsIAsyncInputStream* aInput,
 163    ErrorResult& aRv) {
 164    auto algorithms = MakeRefPtr<InputToReadableStreamAlgorithms>(
 165          stream->GetParentObject(), aInput);
 166    return do_AddRef(ReadableStream::CreateNative(aCx, aGlobal, *algorithms,
 167                                                  Nothing(), nullptr, aRv));
 168 }
 169
 170 // For nsIAsyncOutputStream
 171 already_AddRefed<ReadableStream> ConvertOutputStreamToWritableStream(
 172    JSContext* aCx, nsIGlobalObject* aGlobal, nsIAsyncOutputStream* aInput,
 173    ErrorResult& aRv) {
 174    auto algorithms = MakeRefPtr<WritableStreamToOutputAlgorithms>(
 175          stream->GetParentObject(), aInput);
 176    return do_AddRef(WritableStream::CreateNative(aCx, aGlobal, *algorithms,
 177                                                  Nothing(), nullptr, aRv));
 178 }
 179 ```
 180
 181 ## Mapping spec terms to functions
 182
 183 1. [ReadableStream](https://streams.spec.whatwg.org/#other-specs-rs)
 184    * [Set up](https://streams.spec.whatwg.org/#readablestream-set-up): [`CreateNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#132)
 185    * [Set up with byte reading support](https://streams.spec.whatwg.org/#readablestream-set-up-with-byte-reading-support):
 186       - [`CreateByteNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#143): You can call this when the spec uses the term with `new ReadableStream`.
 187       - [`SetUpByteNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#150): You need to use this instead when the spec uses the term with a subclass of `ReadableStream`. Call this inside the constructor of the subclass.
 188    * [Close](https://streams.spec.whatwg.org/#readablestream-close): [`CloseNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#160)
 189    * [Error](https://streams.spec.whatwg.org/#readablestream-error): [`ErrorNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#163)
 190    * [Enqueue](https://streams.spec.whatwg.org/#readablestream-enqueue): [`EnqueueNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#167)
 191    * [Get a reader](https://streams.spec.whatwg.org/#readablestream-get-a-reader): [`GetReader()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStream.h#177)
 192       * [Read a chunk](https://streams.spec.whatwg.org/#readablestreamdefaultreader-read-a-chunk) on reader: [`ReadChunk()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/ReadableStreamDefaultReader.h#81) on ReadableStreamDefaultReader
 193 2. [WritableStream](https://streams.spec.whatwg.org/#other-specs-ws)
 194    * [Set up](https://streams.spec.whatwg.org/#writablestream-set-up):
 195       - [`CreateNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/WritableStream.h#182): You can call this when the spec uses the term with `new WritableStream`.
 196       - [`SetUpNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/WritableStream.h#174): You need to use this instead when the spec uses the term with a subclass of `WritableStream`. Call this inside the constructor of the subclass.
 197    * [Error](https://streams.spec.whatwg.org/#writablestream-error): [`ErrorNative()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/WritableStream.h#192)
 198 3. [TransformStream](https://streams.spec.whatwg.org/#other-specs-ts): For now this just uses the functions in TransfromStreamDefaultController, which will be provided as an argument of transform or flush algorithms.
 199    * [Enqueue](https://streams.spec.whatwg.org/#transformstream-enqueue): [`Enqueue()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/TransformStreamDefaultController.h#47) on TransformStreamDefaultController
 200    * [Terminate](https://streams.spec.whatwg.org/#transformstream-terminate): [`Terminate()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/TransformStreamDefaultController.h#51) on TransformStreamDefaultController
 201    * [Error](https://streams.spec.whatwg.org/#transformstream-error): [`Error()`](https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/dom/streams/TransformStreamDefaultController.h#49) on TransformStreamDefaultController
 202
 203 The mapping is only implemented on demand and does not cover every function in the spec. Please file a bug on [DOM: Streams](https://bugzilla.mozilla.org/describecomponents.cgi?product=Core&component=DOM%3A%20Streams#DOM%3A%20Streams) component in Bugzilla if you need something that is missing here.