IF Commons GitHub

net

Written to: 1 requirement: 1 proved, 0 partly, 0 failing, 0 with no test recorded. 0001-net FR-001

Downloads over a network that fails now and works in a minute (commons 0001-net FR-001), on the web standard fetch and node:fs. Taken from the machine setup's net.ts: a failure is classified (the name, the connection, a busy server, a certificate), retried only where another try can help, with a longer wait each time (or as long as the server's Retry-After asks, up to the cap) and every retry said; a download that stops part way is resumed with Range; and a file is kept only where its SHA-256 is the one expected. A download that fails is an expected outcome, so these return a Result.

import { downloadVerified, Retries } from "./net.ts";
import { sha256Of } from "@intellectual-frontiers/crypto/sha256";
const sum = sha256Of("…64 hex digits…");
if (sum !== undefined) {
  const r = await downloadVerified(new Retries({ tries: 5, base: 2, cap: 30 }, console.error), "https://example.com/tool.tgz", sum, "/tmp/cache");
  if (r.kind === "ok") console.log("kept at", r.value);
}

Source: commons/net/src/net.ts

10 of 26 exports are imported by a test of the run. A test that imports an export may not call it or check what it returns.

netnamespace
namespace net
net.NetFailuretypeno test imports it
type NetFailure =
  | { readonly kind: "dns" }
  | { readonly kind: "connect" }
  | { readonly kind: "broken" }
  | { readonly kind: "timeout" }
  | { readonly kind: "tls" }
  | { readonly kind: "certificate" }
  | { readonly kind: "http"; readonly status: number; readonly retryAfterMs?: number }
  | { readonly kind: "other"; readonly message: string };

Why a request failed, as far as the network is concerned.

net.describeFailurefunctionno test imports it
function describeFailure(f: NetFailure): string

A failure as a person reads it.

net.retryablefunctionno test imports it
function retryable(f: NetFailure): boolean

Whether another try may help: the name, the connection, a busy server; never a refusal or a certificate.

net.classifyfunctionno test imports it
function classify(e: unknown): NetFailure

What an error of fetch says, as a failure of the network: by its code (or its cause's) where it has one, else by its words.

net.RetryPolicytypeno test imports it
type RetryPolicy = { readonly tries: number; readonly base: number; readonly cap: number };

How often and how patiently to try: at most tries attempts, waiting base seconds after the first failure, doubled each time up to cap.

net.RetryNotetypeno test imports it
type RetryNote = { readonly what: string; readonly why: string; readonly attempt: number };

One retry: what was retried, why, and which attempt comes next.

net.Retriesclassno test imports it
class Retries

The retries of a run: how long to wait, each retry said through report and kept for a summary.

MemberTypeWhat it is
notesRetryNote[]The retries so far, in order.
policyRetryPolicyThe policy.
report(line: string) => voidWhere each retry is said.
jitter() => numberSeconds added to each wait so that many computers do not try together (0 to 2 by default).
pause(seconds: number) => Promise<void>How a wait is spent (a timer by default; a test passes one that returns at once).
wait(attempt: number) => numberThe seconds to wait after attempt attempt failed: the base doubled each time, no more than the cap, plus the jitter.
again(what: string, why: string, attempt: number, retryAfterSeconds?: number | undefined) => Promise<void>Say that an attempt failed and when the next one is, keep it, and wait; a server's Retry-After (in seconds, up to the cap) wins.
top(n?: number) => stringWhat the network cost, most retried first: github.com (3), example.org (1).
net.FetchOptionstypeno test imports it
type FetchOptions = {
  /** No answer within this many seconds fails the try (20 by default). */
  readonly answerSeconds?: number;
  /** Less than 1 KB a second over this many seconds fails the try (60 by default). */
  readonly stallSeconds?: number;
  /** Headers to send. */
  readonly headers?: Readonly<Record<string, string>>;
};

Limits of one try.

net.fetchOncefunctionno test imports it
function fetchOnce(url: string, dest: string, o?: FetchOptions): Promise<Result<void, NetFailure>>

One try: the file at url written to dest; a dest ending .part that already has bytes is resumed where the server can.

net.hostOffunctionno test imports it
function hostOf(url: string): string

The host a URL names, for saying what was retried.

net.fetchTofunctionno test imports it
function fetchTo(retries: Retries, url: string, dest: string, o?: FetchOptions): Promise<Result<void, string>>

The file at url written to dest, tried again where the network failed and another try can help; or why it was not.

net.downloadVerifiedfunctionno test imports it
function downloadVerified(retries: Retries, url: string, sum: Sha256, cache: string, o?: FetchOptions): Promise<Result<string, string>>

The file in cache/downloads/<sha256>, fetched where it is not there, and kept only where its SHA-256 is sum. A download that stops part way is kept as .part and resumed by the next try or the next run; a file that is not the one expected is removed, never kept.

net.reachfunctionno test imports it
function reach(retries: Retries, host: string, scheme?: "http" | "https"): Promise<Result<void, NetFailure>>

Whether host answers over HTTPS (any answer counts, even a refusal), tried again where it does not, at most 5 times.

NetFailuretype
type NetFailure =
  | { readonly kind: "dns" }
  | { readonly kind: "connect" }
  | { readonly kind: "broken" }
  | { readonly kind: "timeout" }
  | { readonly kind: "tls" }
  | { readonly kind: "certificate" }
  | { readonly kind: "http"; readonly status: number; readonly retryAfterMs?: number }
  | { readonly kind: "other"; readonly message: string };

Why a request failed, as far as the network is concerned.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

describeFailurefunction
function describeFailure(f: NetFailure): string

A failure as a person reads it.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

retryablefunction
function retryable(f: NetFailure): boolean

Whether another try may help: the name, the connection, a busy server; never a refusal or a certificate.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

classifyfunction
function classify(e: unknown): NetFailure

What an error of fetch says, as a failure of the network: by its code (or its cause's) where it has one, else by its words.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

RetryPolicytypeno test imports it
type RetryPolicy = { readonly tries: number; readonly base: number; readonly cap: number };

How often and how patiently to try: at most tries attempts, waiting base seconds after the first failure, doubled each time up to cap.

RetryNotetypeno test imports it
type RetryNote = { readonly what: string; readonly why: string; readonly attempt: number };

One retry: what was retried, why, and which attempt comes next.

Retriesclass
class Retries

The retries of a run: how long to wait, each retry said through report and kept for a summary.

MemberTypeWhat it is
notesRetryNote[]The retries so far, in order.
policyRetryPolicyThe policy.
report(line: string) => voidWhere each retry is said.
jitter() => numberSeconds added to each wait so that many computers do not try together (0 to 2 by default).
pause(seconds: number) => Promise<void>How a wait is spent (a timer by default; a test passes one that returns at once).
wait(attempt: number) => numberThe seconds to wait after attempt attempt failed: the base doubled each time, no more than the cap, plus the jitter.
again(what: string, why: string, attempt: number, retryAfterSeconds?: number | undefined) => Promise<void>Say that an attempt failed and when the next one is, keep it, and wait; a server's Retry-After (in seconds, up to the cap) wins.
top(n?: number) => stringWhat the network cost, most retried first: github.com (3), example.org (1).

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

FetchOptionstypeno test imports it
type FetchOptions = {
  /** No answer within this many seconds fails the try (20 by default). */
  readonly answerSeconds?: number;
  /** Less than 1 KB a second over this many seconds fails the try (60 by default). */
  readonly stallSeconds?: number;
  /** Headers to send. */
  readonly headers?: Readonly<Record<string, string>>;
};

Limits of one try.

fetchOncefunction
function fetchOnce(url: string, dest: string, o?: FetchOptions): Promise<Result<void, NetFailure>>

One try: the file at url written to dest; a dest ending .part that already has bytes is resumed where the server can.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

hostOffunction
function hostOf(url: string): string

The host a URL names, for saying what was retried.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

fetchTofunction
function fetchTo(retries: Retries, url: string, dest: string, o?: FetchOptions): Promise<Result<void, string>>

The file at url written to dest, tried again where the network failed and another try can help; or why it was not.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

downloadVerifiedfunction
function downloadVerified(retries: Retries, url: string, sum: Sha256, cache: string, o?: FetchOptions): Promise<Result<string, string>>

The file in cache/downloads/<sha256>, fetched where it is not there, and kept only where its SHA-256 is sum. A download that stops part way is kept as .part and resumed by the next try or the next run; a file that is not the one expected is removed, never kept.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

reachfunction
function reach(retries: Retries, host: string, scheme?: "http" | "https"): Promise<Result<void, NetFailure>>

Whether host answers over HTTPS (any answer counts, even a refusal), tried again where it does not, at most 5 times.

Imported by 6 tests of the run (6 passed, 0 failed): the library's results

Titles, headings, exports, tables, columns and requirements. Esc closes.