netnamespacenamespace netDownloads over a network that fails now and works in a minute (commons 0001-net FR-001), on the web standard fetch and node:fs. Taken from the machine setup's net.ts: a failure is classified (the name, the connection, a busy server, a certificate), retried only where another try can help, with a longer wait each time (or as long as the server's Retry-After asks, up to the cap) and every retry said; a download that stops part way is resumed with Range; and a file is kept only where its SHA-256 is the one expected. A download that fails is an expected outcome, so these return a Result.
import { downloadVerified, Retries } from "./net.ts";
import { sha256Of } from "@intellectual-frontiers/crypto/sha256";
const sum = sha256Of("…64 hex digits…");
if (sum !== undefined) {
const r = await downloadVerified(new Retries({ tries: 5, base: 2, cap: 30 }, console.error), "https://example.com/tool.tgz", sum, "/tmp/cache");
if (r.kind === "ok") console.log("kept at", r.value);
}netnamespacenamespace netnet.NetFailuretypeno test imports ittype NetFailure =
| { readonly kind: "dns" }
| { readonly kind: "connect" }
| { readonly kind: "broken" }
| { readonly kind: "timeout" }
| { readonly kind: "tls" }
| { readonly kind: "certificate" }
| { readonly kind: "http"; readonly status: number; readonly retryAfterMs?: number }
| { readonly kind: "other"; readonly message: string };Why a request failed, as far as the network is concerned.
net.describeFailurefunctionno test imports itfunction describeFailure(f: NetFailure): stringA failure as a person reads it.
net.retryablefunctionno test imports itfunction retryable(f: NetFailure): booleanWhether another try may help: the name, the connection, a busy server; never a refusal or a certificate.
net.classifyfunctionno test imports itfunction classify(e: unknown): NetFailureWhat an error of fetch says, as a failure of the network: by its code (or its cause's) where it has one, else by its words.
net.RetryPolicytypeno test imports ittype RetryPolicy = { readonly tries: number; readonly base: number; readonly cap: number };How often and how patiently to try: at most tries attempts, waiting base seconds after the first failure, doubled each time up to cap.
net.RetryNotetypeno test imports ittype RetryNote = { readonly what: string; readonly why: string; readonly attempt: number };One retry: what was retried, why, and which attempt comes next.
net.Retriesclassno test imports itclass RetriesThe retries of a run: how long to wait, each retry said through report and kept for a summary.
| Member | Type | What it is |
|---|---|---|
notes | RetryNote[] | The retries so far, in order. |
policy | RetryPolicy | The policy. |
report | (line: string) => void | Where each retry is said. |
jitter | () => number | Seconds added to each wait so that many computers do not try together (0 to 2 by default). |
pause | (seconds: number) => Promise<void> | How a wait is spent (a timer by default; a test passes one that returns at once). |
wait | (attempt: number) => number | The seconds to wait after attempt attempt failed: the base doubled each time, no more than the cap, plus the jitter. |
again | (what: string, why: string, attempt: number, retryAfterSeconds?: number | undefined) => Promise<void> | Say that an attempt failed and when the next one is, keep it, and wait; a server's Retry-After (in seconds, up to the cap) wins. |
top | (n?: number) => string | What the network cost, most retried first: github.com (3), example.org (1). |
net.FetchOptionstypeno test imports ittype FetchOptions = {
/** No answer within this many seconds fails the try (20 by default). */
readonly answerSeconds?: number;
/** Less than 1 KB a second over this many seconds fails the try (60 by default). */
readonly stallSeconds?: number;
/** Headers to send. */
readonly headers?: Readonly<Record<string, string>>;
};Limits of one try.
net.fetchOncefunctionno test imports itfunction fetchOnce(url: string, dest: string, o?: FetchOptions): Promise<Result<void, NetFailure>>One try: the file at url written to dest; a dest ending .part that already has bytes is resumed where the server can.
net.hostOffunctionno test imports itfunction hostOf(url: string): stringThe host a URL names, for saying what was retried.
net.fetchTofunctionno test imports itfunction fetchTo(retries: Retries, url: string, dest: string, o?: FetchOptions): Promise<Result<void, string>>The file at url written to dest, tried again where the network failed and another try can help; or why it was not.
net.downloadVerifiedfunctionno test imports itfunction downloadVerified(retries: Retries, url: string, sum: Sha256, cache: string, o?: FetchOptions): Promise<Result<string, string>>The file in cache/downloads/<sha256>, fetched where it is not there, and kept only where its SHA-256 is sum. A download that stops part way is kept as .part and resumed by the next try or the next run; a file that is not the one expected is removed, never kept.
net.reachfunctionno test imports itfunction reach(retries: Retries, host: string, scheme?: "http" | "https"): Promise<Result<void, NetFailure>>Whether host answers over HTTPS (any answer counts, even a refusal), tried again where it does not, at most 5 times.
NetFailuretypetype NetFailure =
| { readonly kind: "dns" }
| { readonly kind: "connect" }
| { readonly kind: "broken" }
| { readonly kind: "timeout" }
| { readonly kind: "tls" }
| { readonly kind: "certificate" }
| { readonly kind: "http"; readonly status: number; readonly retryAfterMs?: number }
| { readonly kind: "other"; readonly message: string };Why a request failed, as far as the network is concerned.
describeFailurefunctionfunction describeFailure(f: NetFailure): stringA failure as a person reads it.
retryablefunctionfunction retryable(f: NetFailure): booleanWhether another try may help: the name, the connection, a busy server; never a refusal or a certificate.
classifyfunctionfunction classify(e: unknown): NetFailureWhat an error of fetch says, as a failure of the network: by its code (or its cause's) where it has one, else by its words.
RetryPolicytypeno test imports ittype RetryPolicy = { readonly tries: number; readonly base: number; readonly cap: number };How often and how patiently to try: at most tries attempts, waiting base seconds after the first failure, doubled each time up to cap.
RetryNotetypeno test imports ittype RetryNote = { readonly what: string; readonly why: string; readonly attempt: number };One retry: what was retried, why, and which attempt comes next.
Retriesclassclass RetriesThe retries of a run: how long to wait, each retry said through report and kept for a summary.
| Member | Type | What it is |
|---|---|---|
notes | RetryNote[] | The retries so far, in order. |
policy | RetryPolicy | The policy. |
report | (line: string) => void | Where each retry is said. |
jitter | () => number | Seconds added to each wait so that many computers do not try together (0 to 2 by default). |
pause | (seconds: number) => Promise<void> | How a wait is spent (a timer by default; a test passes one that returns at once). |
wait | (attempt: number) => number | The seconds to wait after attempt attempt failed: the base doubled each time, no more than the cap, plus the jitter. |
again | (what: string, why: string, attempt: number, retryAfterSeconds?: number | undefined) => Promise<void> | Say that an attempt failed and when the next one is, keep it, and wait; a server's Retry-After (in seconds, up to the cap) wins. |
top | (n?: number) => string | What the network cost, most retried first: github.com (3), example.org (1). |
FetchOptionstypeno test imports ittype FetchOptions = {
/** No answer within this many seconds fails the try (20 by default). */
readonly answerSeconds?: number;
/** Less than 1 KB a second over this many seconds fails the try (60 by default). */
readonly stallSeconds?: number;
/** Headers to send. */
readonly headers?: Readonly<Record<string, string>>;
};Limits of one try.
fetchOncefunctionfunction fetchOnce(url: string, dest: string, o?: FetchOptions): Promise<Result<void, NetFailure>>One try: the file at url written to dest; a dest ending .part that already has bytes is resumed where the server can.
hostOffunctionfunction hostOf(url: string): stringThe host a URL names, for saying what was retried.
fetchTofunctionfunction fetchTo(retries: Retries, url: string, dest: string, o?: FetchOptions): Promise<Result<void, string>>The file at url written to dest, tried again where the network failed and another try can help; or why it was not.
downloadVerifiedfunctionfunction downloadVerified(retries: Retries, url: string, sum: Sha256, cache: string, o?: FetchOptions): Promise<Result<string, string>>The file in cache/downloads/<sha256>, fetched where it is not there, and kept only where its SHA-256 is sum. A download that stops part way is kept as .part and resumed by the next try or the next run; a file that is not the one expected is removed, never kept.
reachfunctionfunction reach(retries: Retries, host: string, scheme?: "http" | "https"): Promise<Result<void, NetFailure>>Whether host answers over HTTPS (any answer counts, even a refusal), tried again where it does not, at most 5 times.