Add comprehensive installation and setup documentation

- Add GETTING_STARTED.md with quick start guide and development modes
- Add INSTALL.sh automated installation script
- Add INSTALLATION_CHECKLIST.md, INSTALLATION_SUCCESS.md, and INSTALLATION_SUMMARY.md
- Add QUICK_REFERENCE.md for common commands
- Add SETUP_GUIDE.md with detailed setup instructions
- Update README.md with improved project overview
- Add did-wallet app dependencies and node_modules
This commit is contained in:
Dorian
2026-01-27 17:18:21 +00:00
parent a81f655133
commit 0d073fa89e
22658 changed files with 4494151 additions and 6 deletions
+4
View File
@@ -0,0 +1,4 @@
This project is dual licensed under MIT and Apache-2.0.
MIT: https://www.opensource.org/licenses/mit
Apache-2.0: https://www.apache.org/licenses/license-2.0
+169
View File
@@ -0,0 +1,169 @@
# ipfs-unixfs-importer <!-- omit in toc -->
[![ipfs.tech](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](https://ipfs.tech)
[![Discuss](https://img.shields.io/discourse/https/discuss.ipfs.tech/posts.svg?style=flat-square)](https://discuss.ipfs.tech)
[![codecov](https://img.shields.io/codecov/c/github/ipfs/js-ipfs-unixfs.svg?style=flat-square)](https://codecov.io/gh/ipfs/js-ipfs-unixfs)
[![CI](https://img.shields.io/github/actions/workflow/status/ipfs/js-ipfs-unixfs/js-test-and-release.yml?branch=master\&style=flat-square)](https://github.com/ipfs/js-ipfs-unixfs/actions/workflows/js-test-and-release.yml?query=branch%3Amaster)
> JavaScript implementation of the UnixFs importer used by IPFS
## Table of contents <!-- omit in toc -->
- [Install](#install)
- [Browser `<script>` tag](#browser-script-tag)
- [Example](#example)
- [API](#api)
- [const stream = importer(source, blockstore \[, options\])](#const-stream--importersource-blockstore--options)
- [const result = await importFile(content, blockstore \[, options\])](#const-result--await-importfilecontent-blockstore--options)
- [const result = await importDirectory(content, blockstore \[, options\])](#const-result--await-importdirectorycontent-blockstore--options)
- [const result = await importBytes(buf, blockstore \[, options\])](#const-result--await-importbytesbuf-blockstore--options)
- [const result = await importByteStream(source, blockstore \[, options\])](#const-result--await-importbytestreamsource-blockstore--options)
- [API Docs](#api-docs)
- [License](#license)
- [Contribute](#contribute)
## Install
```console
$ npm i ipfs-unixfs-importer
```
### Browser `<script>` tag
Loading this module through a script tag will make it's exports available as `IpfsUnixfsImporter` in the global namespace.
```html
<script src="https://unpkg.com/ipfs-unixfs-importer/dist/index.min.js"></script>
```
## Example
Let's create a little directory to import:
```sh
> cd /tmp
> mkdir foo
> echo 'hello' > foo/bar
> echo 'world' > foo/quux
```
And write the importing logic:
```js
import { importer } from 'ipfs-unixfs-importer'
import { MemoryBlockstore } from 'blockstore-core/memory'
import * as fs from 'node:fs'
// Where the blocks will be stored
const blockstore = new MemoryBlockstore()
// Import path /tmp/foo/
const source = [{
path: '/tmp/foo/bar',
content: fs.createReadStream('/tmp/foo/bar')
}, {
path: '/tmp/foo/quxx',
content: fs.createReadStream('/tmp/foo/quux')
}]
for await (const entry of importer(source, blockstore)) {
console.info(entry)
}
```
When run, metadata about DAGNodes in the created tree is printed until the root:
```js
{
cid: CID, // see https://github.com/multiformats/js-cid
path: 'tmp/foo/bar',
unixfs: UnixFS // see https://github.com/ipfs/js-ipfs-unixfs
}
{
cid: CID, // see https://github.com/multiformats/js-cid
path: 'tmp/foo/quxx',
unixfs: UnixFS // see https://github.com/ipfs/js-ipfs-unixfs
}
{
cid: CID, // see https://github.com/multiformats/js-cid
path: 'tmp/foo',
unixfs: UnixFS // see https://github.com/ipfs/js-ipfs-unixfs
}
{
cid: CID, // see https://github.com/multiformats/js-cid
path: 'tmp',
unixfs: UnixFS // see https://github.com/ipfs/js-ipfs-unixfs
}
```
## API
```js
import { importer, importFile, importDir, importBytes, importByteStream } from 'ipfs-unixfs-importer'
```
### const stream = importer(source, blockstore \[, options])
The `importer` function returns an async iterator takes a source async iterator that yields objects of the form:
```js
{
path: 'a name',
content: (Buffer or iterator emitting Buffers),
mtime: (Number representing seconds since (positive) or before (negative) the Unix Epoch),
mode: (Number representing ugo-rwx, setuid, setguid and sticky bit)
}
```
`stream` will output file info objects as files get stored in IPFS. When stats on a node are emitted they are guaranteed to have been written.
`blockstore` is an instance of a [blockstore][]
The input's file paths and directory structure will be preserved in the [`dag-pb`](https://github.com/ipld/js-dag-pb) created nodes.
### const result = await importFile(content, blockstore \[, options])
A convenience function for importing a single file or directory.
### const result = await importDirectory(content, blockstore \[, options])
A convenience function for importing a directory - note this is non-recursive, to import recursively use the [importer](#const-stream--importersource-blockstore--options) function.
### const result = await importBytes(buf, blockstore \[, options])
A convenience function for importing a single Uint8Array.
### const result = await importByteStream(source, blockstore \[, options])
A convenience function for importing a single stream of Uint8Arrays.
## API Docs
- <https://ipfs.github.io/js-ipfs-unixfs/modules/ipfs_unixfs_importer.html>
## License
Licensed under either of
- Apache 2.0, ([LICENSE-APACHE](LICENSE-APACHE) / <http://www.apache.org/licenses/LICENSE-2.0>)
- MIT ([LICENSE-MIT](LICENSE-MIT) / <http://opensource.org/licenses/MIT>)
## Contribute
Contributions welcome! Please check out [the issues](https://github.com/ipfs/js-ipfs-unixfs/issues).
Also see our [contributing document](https://github.com/ipfs/community/blob/master/CONTRIBUTING_JS.md) for more information on how we work, and about contributing in general.
Please be aware that all interactions related to this repo are subject to the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md).
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
[![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md)
[blockstore]: https://github.com/ipfs/js-ipfs-interfaces/tree/master/packages/interface-blockstore#readme
[UnixFS]: https://github.com/ipfs/specs/tree/master/unixfs
[IPLD]: https://github.com/ipld/js-ipld
[CID]: https://github.com/multiformats/js-cid
File diff suppressed because one or more lines are too long
@@ -0,0 +1,6 @@
import type { Chunker } from './index.js';
export interface FixedSizeOptions {
chunkSize?: number;
}
export declare const fixedSize: (options?: FixedSizeOptions) => Chunker;
//# sourceMappingURL=fixed-size.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"fixed-size.d.ts","sourceRoot":"","sources":["../../../src/chunker/fixed-size.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,YAAY,CAAA;AAEzC,MAAM,WAAW,gBAAgB;IAC/B,SAAS,CAAC,EAAE,MAAM,CAAA;CACnB;AAID,eAAO,MAAM,SAAS,aAAa,gBAAgB,KAAQ,OAqC1D,CAAA"}
@@ -0,0 +1,35 @@
import { Uint8ArrayList } from 'uint8arraylist';
const DEFAULT_CHUNK_SIZE = 262144;
export const fixedSize = (options = {}) => {
const chunkSize = options.chunkSize ?? DEFAULT_CHUNK_SIZE;
return async function* fixedSizeChunker(source) {
let list = new Uint8ArrayList();
let currentLength = 0;
let emitted = false;
for await (const buffer of source) {
list.append(buffer);
currentLength += buffer.length;
while (currentLength >= chunkSize) {
yield list.slice(0, chunkSize);
emitted = true;
// throw away consumed bytes
if (chunkSize === list.length) {
list = new Uint8ArrayList();
currentLength = 0;
}
else {
const newBl = new Uint8ArrayList();
newBl.append(list.sublist(chunkSize));
list = newBl;
// update our offset
currentLength -= chunkSize;
}
}
}
if (!emitted || currentLength > 0) {
// return any remaining bytes
yield list.subarray(0, currentLength);
}
};
};
//# sourceMappingURL=fixed-size.js.map
@@ -0,0 +1 @@
{"version":3,"file":"fixed-size.js","sourceRoot":"","sources":["../../../src/chunker/fixed-size.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,gBAAgB,CAAA;AAO/C,MAAM,kBAAkB,GAAG,MAAM,CAAA;AAEjC,MAAM,CAAC,MAAM,SAAS,GAAG,CAAC,UAA4B,EAAE,EAAW,EAAE;IACnE,MAAM,SAAS,GAAG,OAAO,CAAC,SAAS,IAAI,kBAAkB,CAAA;IAEzD,OAAO,KAAK,SAAU,CAAC,CAAC,gBAAgB,CAAE,MAAM;QAC9C,IAAI,IAAI,GAAG,IAAI,cAAc,EAAE,CAAA;QAC/B,IAAI,aAAa,GAAG,CAAC,CAAA;QACrB,IAAI,OAAO,GAAG,KAAK,CAAA;QAEnB,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI,MAAM,EAAE;YACjC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAA;YAEnB,aAAa,IAAI,MAAM,CAAC,MAAM,CAAA;YAE9B,OAAO,aAAa,IAAI,SAAS,EAAE;gBACjC,MAAM,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,SAAS,CAAC,CAAA;gBAC9B,OAAO,GAAG,IAAI,CAAA;gBAEd,4BAA4B;gBAC5B,IAAI,SAAS,KAAK,IAAI,CAAC,MAAM,EAAE;oBAC7B,IAAI,GAAG,IAAI,cAAc,EAAE,CAAA;oBAC3B,aAAa,GAAG,CAAC,CAAA;iBAClB;qBAAM;oBACL,MAAM,KAAK,GAAG,IAAI,cAAc,EAAE,CAAA;oBAClC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAA;oBACrC,IAAI,GAAG,KAAK,CAAA;oBAEZ,oBAAoB;oBACpB,aAAa,IAAI,SAAS,CAAA;iBAC3B;aACF;SACF;QAED,IAAI,CAAC,OAAO,IAAI,aAAa,GAAG,CAAC,EAAE;YACjC,6BAA6B;YAC7B,MAAM,IAAI,CAAC,QAAQ,CAAC,CAAC,EAAE,aAAa,CAAC,CAAA;SACtC;IACH,CAAC,CAAA;AACH,CAAC,CAAA"}
@@ -0,0 +1,6 @@
export interface Chunker {
(source: AsyncIterable<Uint8Array>): AsyncIterable<Uint8Array>;
}
export { rabin } from './rabin.js';
export { fixedSize } from './fixed-size.js';
//# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/chunker/index.ts"],"names":[],"mappings":"AACA,MAAM,WAAW,OAAO;IAAG,CAAC,MAAM,EAAE,aAAa,CAAC,UAAU,CAAC,GAAG,aAAa,CAAC,UAAU,CAAC,CAAA;CAAE;AAE3F,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAA;AAClC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAA"}
@@ -0,0 +1,3 @@
export { rabin } from './rabin.js';
export { fixedSize } from './fixed-size.js';
//# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/chunker/index.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAA;AAClC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAA"}
@@ -0,0 +1,9 @@
import type { Chunker } from './index.js';
export interface RabinOptions {
minChunkSize?: number;
maxChunkSize?: number;
avgChunkSize?: number;
window?: number;
}
export declare const rabin: (options?: RabinOptions) => Chunker;
//# sourceMappingURL=rabin.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"rabin.d.ts","sourceRoot":"","sources":["../../../src/chunker/rabin.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,YAAY,CAAA;AA6BzC,MAAM,WAAW,YAAY;IAC3B,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,MAAM,CAAC,EAAE,MAAM,CAAA;CAChB;AAED,eAAO,MAAM,KAAK,aAAa,YAAY,KAAQ,OAsClD,CAAA"}
@@ -0,0 +1,56 @@
import errcode from 'err-code';
// @ts-expect-error no types
import { create } from 'rabin-wasm';
import { Uint8ArrayList } from 'uint8arraylist';
const DEFAULT_MIN_CHUNK_SIZE = 262144;
const DEFAULT_MAX_CHUNK_SIZE = 262144;
const DEFAULT_AVG_CHUNK_SIZE = 262144;
const DEFAULT_WINDOW = 16;
async function* chunker(source, r) {
const buffers = new Uint8ArrayList();
for await (const chunk of source) {
buffers.append(chunk);
const sizes = r.fingerprint(chunk);
for (let i = 0; i < sizes.length; i++) {
const size = sizes[i];
const buf = buffers.slice(0, size);
buffers.consume(size);
yield buf;
}
}
if (buffers.length > 0) {
yield buffers.subarray(0);
}
}
export const rabin = (options = {}) => {
let min = options.minChunkSize ?? DEFAULT_MIN_CHUNK_SIZE;
let max = options.maxChunkSize ?? DEFAULT_MAX_CHUNK_SIZE;
let avg = options.avgChunkSize ?? DEFAULT_AVG_CHUNK_SIZE;
const window = options.window ?? DEFAULT_WINDOW;
// if only avg was passed, calculate min/max from that
if (options.avgChunkSize != null && options.minChunkSize == null && options.maxChunkSize == null) {
min = avg / 3;
max = avg + (avg / 2);
}
if (options.avgChunkSize == null && options.minChunkSize == null && options.maxChunkSize == null) {
throw errcode(new Error('please specify an average chunk size'), 'ERR_INVALID_AVG_CHUNK_SIZE');
}
// validate min/max/avg in the same way as go
if (min < 16) {
throw errcode(new Error('rabin min must be greater than 16'), 'ERR_INVALID_MIN_CHUNK_SIZE');
}
if (max < min) {
max = min;
}
if (avg < min) {
avg = min;
}
const sizepow = Math.floor(Math.log2(avg));
return async function* rabinChunker(source) {
const r = await create(sizepow, min, max, window);
for await (const chunk of chunker(source, r)) {
yield chunk;
}
};
};
//# sourceMappingURL=rabin.js.map
@@ -0,0 +1 @@
{"version":3,"file":"rabin.js","sourceRoot":"","sources":["../../../src/chunker/rabin.ts"],"names":[],"mappings":"AAAA,OAAO,OAAO,MAAM,UAAU,CAAA;AAC9B,4BAA4B;AAC5B,OAAO,EAAE,MAAM,EAAE,MAAM,YAAY,CAAA;AACnC,OAAO,EAAE,cAAc,EAAE,MAAM,gBAAgB,CAAA;AAG/C,MAAM,sBAAsB,GAAG,MAAM,CAAA;AACrC,MAAM,sBAAsB,GAAG,MAAM,CAAA;AACrC,MAAM,sBAAsB,GAAG,MAAM,CAAA;AACrC,MAAM,cAAc,GAAG,EAAE,CAAA;AAEzB,KAAK,SAAU,CAAC,CAAC,OAAO,CAAE,MAAiC,EAAE,CAAM;IACjE,MAAM,OAAO,GAAG,IAAI,cAAc,EAAE,CAAA;IAEpC,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE;QAChC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,CAAA;QAErB,MAAM,KAAK,GAAG,CAAC,CAAC,WAAW,CAAC,KAAK,CAAC,CAAA;QAElC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE;YACrC,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,CAAA;YACrB,MAAM,GAAG,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,CAAA;YAClC,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,CAAA;YAErB,MAAM,GAAG,CAAA;SACV;KACF;IAED,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE;QACtB,MAAM,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAA;KAC1B;AACH,CAAC;AASD,MAAM,CAAC,MAAM,KAAK,GAAG,CAAC,UAAwB,EAAE,EAAW,EAAE;IAC3D,IAAI,GAAG,GAAG,OAAO,CAAC,YAAY,IAAI,sBAAsB,CAAA;IACxD,IAAI,GAAG,GAAG,OAAO,CAAC,YAAY,IAAI,sBAAsB,CAAA;IACxD,IAAI,GAAG,GAAG,OAAO,CAAC,YAAY,IAAI,sBAAsB,CAAA;IACxD,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,IAAI,cAAc,CAAA;IAE/C,sDAAsD;IACtD,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,EAAE;QAChG,GAAG,GAAG,GAAG,GAAG,CAAC,CAAA;QACb,GAAG,GAAG,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC,CAAC,CAAA;KACtB;IAED,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,OAAO,CAAC,YAAY,IAAI,IAAI,EAAE;QAChG,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,sCAAsC,CAAC,EAAE,4BAA4B,CAAC,CAAA;KAC/F;IAED,6CAA6C;IAC7C,IAAI,GAAG,GAAG,EAAE,EAAE;QACZ,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,mCAAmC,CAAC,EAAE,4BAA4B,CAAC,CAAA;KAC5F;IAED,IAAI,GAAG,GAAG,GAAG,EAAE;QACb,GAAG,GAAG,GAAG,CAAA;KACV;IAED,IAAI,GAAG,GAAG,GAAG,EAAE;QACb,GAAG,GAAG,GAAG,CAAA;KACV;IAED,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAA;IAE1C,OAAO,KAAK,SAAU,CAAC,CAAC,YAAY,CAAE,MAAM;QAC1C,MAAM,CAAC,GAAG,MAAM,MAAM,CAAC,OAAO,EAAE,GAAG,EAAE,GAAG,EAAE,MAAM,CAAC,CAAA;QAEjD,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC,EAAE;YAC5C,MAAM,KAAK,CAAA;SACZ;IACH,CAAC,CAAA;AACH,CAAC,CAAA"}
@@ -0,0 +1,29 @@
import type { BufferImporter } from '../index.js';
import type { CID, Version } from 'multiformats/cid';
import type { ProgressOptions, ProgressEvent } from 'progress-events';
/**
* Passed to the onProgress callback while importing files
*/
export interface ImportWriteProgress {
/**
* How many bytes we have written for this source so far - this may be
* bigger than the file size due to the DAG-PB wrappers of each block
*/
bytesWritten: bigint;
/**
* The CID of the block that has been written
*/
cid: CID;
/**
* The path of the file being imported, if one was specified
*/
path?: string;
}
export type BufferImportProgressEvents = ProgressEvent<'unixfs:importer:progress:file:write', ImportWriteProgress>;
export interface BufferImporterOptions extends ProgressOptions<BufferImportProgressEvents> {
cidVersion: Version;
rawLeaves: boolean;
leafType: 'file' | 'raw';
}
export declare function defaultBufferImporter(options: BufferImporterOptions): BufferImporter;
//# sourceMappingURL=buffer-importer.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"buffer-importer.d.ts","sourceRoot":"","sources":["../../../src/dag-builder/buffer-importer.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AACjD,OAAO,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAA;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAA;AAErE;;GAEG;AACH,MAAM,WAAW,mBAAmB;IAClC;;;OAGG;IACH,YAAY,EAAE,MAAM,CAAA;IAEpB;;OAEG;IACH,GAAG,EAAE,GAAG,CAAA;IAER;;OAEG;IACH,IAAI,CAAC,EAAE,MAAM,CAAA;CACd;AAED,MAAM,MAAM,0BAA0B,GACpC,aAAa,CAAC,qCAAqC,EAAE,mBAAmB,CAAC,CAAA;AAE3E,MAAM,WAAW,qBAAsB,SAAQ,eAAe,CAAC,0BAA0B,CAAC;IACxF,UAAU,EAAE,OAAO,CAAA;IACnB,SAAS,EAAE,OAAO,CAAA;IAClB,QAAQ,EAAE,MAAM,GAAG,KAAK,CAAA;CACzB;AAED,wBAAgB,qBAAqB,CAAE,OAAO,EAAE,qBAAqB,GAAG,cAAc,CAgDrF"}
@@ -0,0 +1,48 @@
import * as dagPb from '@ipld/dag-pb';
import { UnixFS } from 'ipfs-unixfs';
import * as raw from 'multiformats/codecs/raw';
import { CustomProgressEvent } from 'progress-events';
import { persist } from '../utils/persist.js';
export function defaultBufferImporter(options) {
return async function* bufferImporter(file, blockstore) {
let bytesWritten = 0n;
for await (let block of file.content) {
yield async () => {
let unixfs;
const opts = {
codec: dagPb,
cidVersion: options.cidVersion,
onProgress: options.onProgress
};
if (options.rawLeaves) {
opts.codec = raw;
opts.cidVersion = 1;
}
else {
unixfs = new UnixFS({
type: options.leafType,
data: block
});
block = dagPb.encode({
Data: unixfs.marshal(),
Links: []
});
}
const cid = await persist(block, blockstore, opts);
bytesWritten += BigInt(block.byteLength);
options.onProgress?.(new CustomProgressEvent('unixfs:importer:progress:file:write', {
bytesWritten,
cid,
path: file.path
}));
return {
cid,
unixfs,
size: BigInt(block.length),
block
};
};
}
};
}
//# sourceMappingURL=buffer-importer.js.map
@@ -0,0 +1 @@
{"version":3,"file":"buffer-importer.js","sourceRoot":"","sources":["../../../src/dag-builder/buffer-importer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,KAAK,MAAM,cAAc,CAAA;AACrC,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACpC,OAAO,KAAK,GAAG,MAAM,yBAAyB,CAAA;AAC9C,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AACrD,OAAO,EAAE,OAAO,EAAuB,MAAM,qBAAqB,CAAA;AAmClE,MAAM,UAAU,qBAAqB,CAAE,OAA8B;IACnE,OAAO,KAAK,SAAU,CAAC,CAAC,cAAc,CAAE,IAAI,EAAE,UAAU;QACtD,IAAI,YAAY,GAAG,EAAE,CAAA;QAErB,IAAI,KAAK,EAAE,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,EAAE;YACpC,MAAM,KAAK,IAAI,EAAE;gBACf,IAAI,MAAM,CAAA;gBAEV,MAAM,IAAI,GAAmB;oBAC3B,KAAK,EAAE,KAAK;oBACZ,UAAU,EAAE,OAAO,CAAC,UAAU;oBAC9B,UAAU,EAAE,OAAO,CAAC,UAAU;iBAC/B,CAAA;gBAED,IAAI,OAAO,CAAC,SAAS,EAAE;oBACrB,IAAI,CAAC,KAAK,GAAG,GAAG,CAAA;oBAChB,IAAI,CAAC,UAAU,GAAG,CAAC,CAAA;iBACpB;qBAAM;oBACL,MAAM,GAAG,IAAI,MAAM,CAAC;wBAClB,IAAI,EAAE,OAAO,CAAC,QAAQ;wBACtB,IAAI,EAAE,KAAK;qBACZ,CAAC,CAAA;oBAEF,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;wBACnB,IAAI,EAAE,MAAM,CAAC,OAAO,EAAE;wBACtB,KAAK,EAAE,EAAE;qBACV,CAAC,CAAA;iBACH;gBAED,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,KAAK,EAAE,UAAU,EAAE,IAAI,CAAC,CAAA;gBAElD,YAAY,IAAI,MAAM,CAAC,KAAK,CAAC,UAAU,CAAC,CAAA;gBAExC,OAAO,CAAC,UAAU,EAAE,CAAC,IAAI,mBAAmB,CAAsB,qCAAqC,EAAE;oBACvG,YAAY;oBACZ,GAAG;oBACH,IAAI,EAAE,IAAI,CAAC,IAAI;iBAChB,CAAC,CAAC,CAAA;gBAEH,OAAO;oBACL,GAAG;oBACH,MAAM;oBACN,IAAI,EAAE,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC;oBAC1B,KAAK;iBACN,CAAA;YACH,CAAC,CAAA;SACF;IACH,CAAC,CAAA;AACH,CAAC"}
@@ -0,0 +1,8 @@
import type { Directory, InProgressImportResult, WritableStorage } from '../index.js';
import type { Version } from 'multiformats/cid';
export interface DirBuilderOptions {
cidVersion: Version;
signal?: AbortSignal;
}
export declare const dirBuilder: (dir: Directory, blockstore: WritableStorage, options: DirBuilderOptions) => Promise<InProgressImportResult>;
//# sourceMappingURL=dir.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"dir.d.ts","sourceRoot":"","sources":["../../../src/dag-builder/dir.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,SAAS,EAAE,sBAAsB,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AACrF,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAA;AAE/C,MAAM,WAAW,iBAAiB;IAChC,UAAU,EAAE,OAAO,CAAA;IACnB,MAAM,CAAC,EAAE,WAAW,CAAA;CACrB;AAED,eAAO,MAAM,UAAU,QAAe,SAAS,cAAc,eAAe,WAAW,iBAAiB,KAAG,QAAQ,sBAAsB,CAmBxI,CAAA"}
@@ -0,0 +1,22 @@
import { encode, prepare } from '@ipld/dag-pb';
import { UnixFS } from 'ipfs-unixfs';
import { persist } from '../utils/persist.js';
export const dirBuilder = async (dir, blockstore, options) => {
const unixfs = new UnixFS({
type: 'directory',
mtime: dir.mtime,
mode: dir.mode
});
const block = encode(prepare({ Data: unixfs.marshal() }));
const cid = await persist(block, blockstore, options);
const path = dir.path;
return {
cid,
path,
unixfs,
size: BigInt(block.length),
originalPath: dir.originalPath,
block
};
};
//# sourceMappingURL=dir.js.map
@@ -0,0 +1 @@
{"version":3,"file":"dir.js","sourceRoot":"","sources":["../../../src/dag-builder/dir.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,cAAc,CAAA;AAC9C,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACpC,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAS7C,MAAM,CAAC,MAAM,UAAU,GAAG,KAAK,EAAE,GAAc,EAAE,UAA2B,EAAE,OAA0B,EAAmC,EAAE;IAC3I,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;QACxB,IAAI,EAAE,WAAW;QACjB,KAAK,EAAE,GAAG,CAAC,KAAK;QAChB,IAAI,EAAE,GAAG,CAAC,IAAI;KACf,CAAC,CAAA;IAEF,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,MAAM,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC,CAAA;IACzD,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,KAAK,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;IACrD,MAAM,IAAI,GAAG,GAAG,CAAC,IAAI,CAAA;IAErB,OAAO;QACL,GAAG;QACH,IAAI;QACJ,MAAM;QACN,IAAI,EAAE,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC;QAC1B,YAAY,EAAE,GAAG,CAAC,YAAY;QAC9B,KAAK;KACN,CAAA;AACH,CAAC,CAAA"}
@@ -0,0 +1,30 @@
import type { BufferImporter, File, InProgressImportResult, WritableStorage, ImporterProgressEvents } from '../index.js';
import type { FileLayout } from '../layout/index.js';
import type { CID, Version } from 'multiformats/cid';
import type { ProgressOptions, ProgressEvent } from 'progress-events';
interface BuildFileBatchOptions {
bufferImporter: BufferImporter;
blockWriteConcurrency: number;
}
export interface LayoutLeafProgress {
/**
* The CID of the leaf being written
*/
cid: CID;
/**
* The path of the file being imported, if one was specified
*/
path?: string;
}
export type ReducerProgressEvents = ProgressEvent<'unixfs:importer:progress:file:layout', LayoutLeafProgress>;
interface ReduceOptions extends ProgressOptions<ImporterProgressEvents> {
reduceSingleLeafToSelf: boolean;
cidVersion: Version;
signal?: AbortSignal;
}
export interface FileBuilderOptions extends BuildFileBatchOptions, ReduceOptions {
layout: FileLayout;
}
export declare const fileBuilder: (file: File, block: WritableStorage, options: FileBuilderOptions) => Promise<InProgressImportResult>;
export {};
//# sourceMappingURL=file.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"file.d.ts","sourceRoot":"","sources":["../../../src/dag-builder/file.ts"],"names":[],"mappings":"AAMA,OAAO,KAAK,EAAE,cAAc,EAAE,IAAI,EAAE,sBAAsB,EAAE,eAAe,EAA2B,sBAAsB,EAAE,MAAM,aAAa,CAAA;AACjJ,OAAO,KAAK,EAAE,UAAU,EAAW,MAAM,oBAAoB,CAAA;AAC7D,OAAO,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,MAAM,kBAAkB,CAAA;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAA;AAErE,UAAU,qBAAqB;IAC7B,cAAc,EAAE,cAAc,CAAA;IAC9B,qBAAqB,EAAE,MAAM,CAAA;CAC9B;AAuCD,MAAM,WAAW,kBAAkB;IACjC;;OAEG;IACH,GAAG,EAAE,GAAG,CAAA;IAER;;OAEG;IACH,IAAI,CAAC,EAAE,MAAM,CAAA;CACd;AAED,MAAM,MAAM,qBAAqB,GAC/B,aAAa,CAAC,sCAAsC,EAAE,kBAAkB,CAAC,CAAA;AAE3E,UAAU,aAAc,SAAQ,eAAe,CAAC,sBAAsB,CAAC;IACrE,sBAAsB,EAAE,OAAO,CAAA;IAC/B,UAAU,EAAE,OAAO,CAAA;IACnB,MAAM,CAAC,EAAE,WAAW,CAAA;CACrB;AAsHD,MAAM,WAAW,kBAAmB,SAAQ,qBAAqB,EAAE,aAAa;IAC9E,MAAM,EAAE,UAAU,CAAA;CACnB;AAED,eAAO,MAAM,WAAW,SAAgB,IAAI,SAAS,eAAe,WAAW,kBAAkB,KAAG,QAAQ,sBAAsB,CAEjI,CAAA"}
@@ -0,0 +1,140 @@
import { encode, prepare } from '@ipld/dag-pb';
import { UnixFS } from 'ipfs-unixfs';
import parallelBatch from 'it-parallel-batch';
import * as rawCodec from 'multiformats/codecs/raw';
import { CustomProgressEvent } from 'progress-events';
import { persist } from '../utils/persist.js';
async function* buildFileBatch(file, blockstore, options) {
let count = -1;
let previous;
for await (const entry of parallelBatch(options.bufferImporter(file, blockstore), options.blockWriteConcurrency)) {
count++;
if (count === 0) {
// cache the first entry if case there aren't any more
previous = {
...entry,
single: true
};
continue;
}
else if (count === 1 && (previous != null)) {
// we have the second block of a multiple block import so yield the first
yield {
...previous,
block: undefined,
single: undefined
};
previous = undefined;
}
// yield the second or later block of a multiple block import
yield {
...entry,
block: undefined
};
}
if (previous != null) {
yield previous;
}
}
function isSingleBlockImport(result) {
return result.single === true;
}
const reduce = (file, blockstore, options) => {
const reducer = async function (leaves) {
if (leaves.length === 1 && isSingleBlockImport(leaves[0]) && options.reduceSingleLeafToSelf) {
const leaf = leaves[0];
let node = leaf.block;
if (isSingleBlockImport(leaf) && (file.mtime !== undefined || file.mode !== undefined)) {
// only one leaf node which is a raw leaf - we have metadata so convert it into a
// UnixFS entry otherwise we'll have nowhere to store the metadata
leaf.unixfs = new UnixFS({
type: 'file',
mtime: file.mtime,
mode: file.mode,
data: leaf.block
});
node = { Data: leaf.unixfs.marshal(), Links: [] };
leaf.block = encode(prepare(node));
leaf.cid = await persist(leaf.block, blockstore, {
...options,
cidVersion: options.cidVersion
});
leaf.size = BigInt(leaf.block.length);
}
options.onProgress?.(new CustomProgressEvent('unixfs:importer:progress:file:layout', {
cid: leaf.cid,
path: leaf.originalPath
}));
return {
cid: leaf.cid,
path: file.path,
unixfs: leaf.unixfs,
size: leaf.size,
originalPath: leaf.originalPath
};
}
// create a parent node and add all the leaves
const f = new UnixFS({
type: 'file',
mtime: file.mtime,
mode: file.mode
});
const links = leaves
.filter(leaf => {
if (leaf.cid.code === rawCodec.code && leaf.size > 0) {
return true;
}
if ((leaf.unixfs != null) && (leaf.unixfs.data == null) && leaf.unixfs.fileSize() > 0n) {
return true;
}
return Boolean(leaf.unixfs?.data?.length);
})
.map((leaf) => {
if (leaf.cid.code === rawCodec.code) {
// node is a leaf buffer
f.addBlockSize(leaf.size);
return {
Name: '',
Tsize: Number(leaf.size),
Hash: leaf.cid
};
}
if ((leaf.unixfs == null) || (leaf.unixfs.data == null)) {
// node is an intermediate node
f.addBlockSize(leaf.unixfs?.fileSize() ?? 0n);
}
else {
// node is a unixfs 'file' leaf node
f.addBlockSize(BigInt(leaf.unixfs.data.length));
}
return {
Name: '',
Tsize: Number(leaf.size),
Hash: leaf.cid
};
});
const node = {
Data: f.marshal(),
Links: links
};
const block = encode(prepare(node));
const cid = await persist(block, blockstore, options);
options.onProgress?.(new CustomProgressEvent('unixfs:importer:progress:file:layout', {
cid,
path: file.originalPath
}));
return {
cid,
path: file.path,
unixfs: f,
size: BigInt(block.length + node.Links.reduce((acc, curr) => acc + (curr.Tsize ?? 0), 0)),
originalPath: file.originalPath,
block
};
};
return reducer;
};
export const fileBuilder = async (file, block, options) => {
return options.layout(buildFileBatch(file, block, options), reduce(file, block, options));
};
//# sourceMappingURL=file.js.map
@@ -0,0 +1 @@
{"version":3,"file":"file.js","sourceRoot":"","sources":["../../../src/dag-builder/file.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAA4B,OAAO,EAAE,MAAM,cAAc,CAAA;AACxE,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACpC,OAAO,aAAa,MAAM,mBAAmB,CAAA;AAC7C,OAAO,KAAK,QAAQ,MAAM,yBAAyB,CAAA;AACnD,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AACrD,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAW7C,KAAK,SAAU,CAAC,CAAC,cAAc,CAAE,IAAU,EAAE,UAA2B,EAAE,OAA8B;IACtG,IAAI,KAAK,GAAG,CAAC,CAAC,CAAA;IACd,IAAI,QAA6C,CAAA;IAEjD,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,aAAa,CAAC,OAAO,CAAC,cAAc,CAAC,IAAI,EAAE,UAAU,CAAC,EAAE,OAAO,CAAC,qBAAqB,CAAC,EAAE;QAChH,KAAK,EAAE,CAAA;QAEP,IAAI,KAAK,KAAK,CAAC,EAAE;YACf,sDAAsD;YACtD,QAAQ,GAAG;gBACT,GAAG,KAAK;gBACR,MAAM,EAAE,IAAI;aACb,CAAA;YAED,SAAQ;SACT;aAAM,IAAI,KAAK,KAAK,CAAC,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,EAAE;YAC5C,yEAAyE;YACzE,MAAM;gBACJ,GAAG,QAAQ;gBACX,KAAK,EAAE,SAAS;gBAChB,MAAM,EAAE,SAAS;aAClB,CAAA;YACD,QAAQ,GAAG,SAAS,CAAA;SACrB;QAED,6DAA6D;QAC7D,MAAM;YACJ,GAAG,KAAK;YACR,KAAK,EAAE,SAAS;SACjB,CAAA;KACF;IAED,IAAI,QAAQ,IAAI,IAAI,EAAE;QACpB,MAAM,QAAQ,CAAA;KACf;AACH,CAAC;AAuBD,SAAS,mBAAmB,CAAE,MAAW;IACvC,OAAO,MAAM,CAAC,MAAM,KAAK,IAAI,CAAA;AAC/B,CAAC;AAED,MAAM,MAAM,GAAG,CAAC,IAAU,EAAE,UAA2B,EAAE,OAAsB,EAAW,EAAE;IAC1F,MAAM,OAAO,GAAY,KAAK,WAAW,MAAM;QAC7C,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,IAAI,mBAAmB,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,OAAO,CAAC,sBAAsB,EAAE;YAC3F,MAAM,IAAI,GAAG,MAAM,CAAC,CAAC,CAAC,CAAA;YACtB,IAAI,IAAI,GAAwB,IAAI,CAAC,KAAK,CAAA;YAE1C,IAAI,mBAAmB,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,KAAK,SAAS,IAAI,IAAI,CAAC,IAAI,KAAK,SAAS,CAAC,EAAE;gBACtF,iFAAiF;gBACjF,kEAAkE;gBAClE,IAAI,CAAC,MAAM,GAAG,IAAI,MAAM,CAAC;oBACvB,IAAI,EAAE,MAAM;oBACZ,KAAK,EAAE,IAAI,CAAC,KAAK;oBACjB,IAAI,EAAE,IAAI,CAAC,IAAI;oBACf,IAAI,EAAE,IAAI,CAAC,KAAK;iBACjB,CAAC,CAAA;gBAEF,IAAI,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,CAAA;gBAEjD,IAAI,CAAC,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAA;gBAElC,IAAI,CAAC,GAAG,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,KAAK,EAAE,UAAU,EAAE;oBAC/C,GAAG,OAAO;oBACV,UAAU,EAAE,OAAO,CAAC,UAAU;iBAC/B,CAAC,CAAA;gBACF,IAAI,CAAC,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAA;aACtC;YAED,OAAO,CAAC,UAAU,EAAE,CAAC,IAAI,mBAAmB,CAAqB,sCAAsC,EAAE;gBACvG,GAAG,EAAE,IAAI,CAAC,GAAG;gBACb,IAAI,EAAE,IAAI,CAAC,YAAY;aACxB,CAAC,CAAC,CAAA;YAEH,OAAO;gBACL,GAAG,EAAE,IAAI,CAAC,GAAG;gBACb,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,IAAI,EAAE,IAAI,CAAC,IAAI;gBACf,YAAY,EAAE,IAAI,CAAC,YAAY;aAChC,CAAA;SACF;QAED,8CAA8C;QAC9C,MAAM,CAAC,GAAG,IAAI,MAAM,CAAC;YACnB,IAAI,EAAE,MAAM;YACZ,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,IAAI,EAAE,IAAI,CAAC,IAAI;SAChB,CAAC,CAAA;QAEF,MAAM,KAAK,GAAa,MAAM;aAC3B,MAAM,CAAC,IAAI,CAAC,EAAE;YACb,IAAI,IAAI,CAAC,GAAG,CAAC,IAAI,KAAK,QAAQ,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,GAAG,CAAC,EAAE;gBACpD,OAAO,IAAI,CAAA;aACZ;YAED,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE,GAAG,EAAE,EAAE;gBACtF,OAAO,IAAI,CAAA;aACZ;YAED,OAAO,OAAO,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,EAAE,MAAM,CAAC,CAAA;QAC3C,CAAC,CAAC;aACD,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;YACZ,IAAI,IAAI,CAAC,GAAG,CAAC,IAAI,KAAK,QAAQ,CAAC,IAAI,EAAE;gBACnC,wBAAwB;gBACxB,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;gBAEzB,OAAO;oBACL,IAAI,EAAE,EAAE;oBACR,KAAK,EAAE,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC;oBACxB,IAAI,EAAE,IAAI,CAAC,GAAG;iBACf,CAAA;aACF;YAED,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,IAAI,IAAI,CAAC,EAAE;gBACvD,+BAA+B;gBAC/B,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,MAAM,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAA;aAC9C;iBAAM;gBACL,oCAAoC;gBACpC,CAAC,CAAC,YAAY,CAAC,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAA;aAChD;YAED,OAAO;gBACL,IAAI,EAAE,EAAE;gBACR,KAAK,EAAE,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC;gBACxB,IAAI,EAAE,IAAI,CAAC,GAAG;aACf,CAAA;QACH,CAAC,CAAC,CAAA;QAEJ,MAAM,IAAI,GAAG;YACX,IAAI,EAAE,CAAC,CAAC,OAAO,EAAE;YACjB,KAAK,EAAE,KAAK;SACb,CAAA;QACD,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAA;QACnC,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,KAAK,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;QAErD,OAAO,CAAC,UAAU,EAAE,CAAC,IAAI,mBAAmB,CAAqB,sCAAsC,EAAE;YACvG,GAAG;YACH,IAAI,EAAE,IAAI,CAAC,YAAY;SACxB,CAAC,CAAC,CAAA;QAEH,OAAO;YACL,GAAG;YACH,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,MAAM,EAAE,CAAC;YACT,IAAI,EAAE,MAAM,CAAC,KAAK,CAAC,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,IAAI,CAAC,KAAK,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;YACzF,YAAY,EAAE,IAAI,CAAC,YAAY;YAC/B,KAAK;SACN,CAAA;IACH,CAAC,CAAA;IAED,OAAO,OAAO,CAAA;AAChB,CAAC,CAAA;AAMD,MAAM,CAAC,MAAM,WAAW,GAAG,KAAK,EAAE,IAAU,EAAE,KAAsB,EAAE,OAA2B,EAAmC,EAAE;IACpI,OAAO,OAAO,CAAC,MAAM,CAAC,cAAc,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,OAAO,CAAC,CAAC,CAAA;AAC3F,CAAC,CAAA"}
@@ -0,0 +1,35 @@
import { type DirBuilderOptions } from './dir.js';
import { type FileBuilderOptions } from './file.js';
import type { ChunkValidator } from './validate-chunks.js';
import type { Chunker } from '../chunker/index.js';
import type { ImportCandidate, ImporterProgressEvents, InProgressImportResult, WritableStorage } from '../index.js';
import type { ProgressEvent, ProgressOptions } from 'progress-events';
/**
* Passed to the onProgress callback while importing files
*/
export interface ImportReadProgress {
/**
* How many bytes we have read from this source so far
*/
bytesRead: bigint;
/**
* The size of the current chunk
*/
chunkSize: bigint;
/**
* The path of the file being imported, if one was specified
*/
path?: string;
}
export type DagBuilderProgressEvents = ProgressEvent<'unixfs:importer:progress:file:read', ImportReadProgress>;
export interface DagBuilderOptions extends FileBuilderOptions, DirBuilderOptions, ProgressOptions<ImporterProgressEvents> {
chunker: Chunker;
chunkValidator: ChunkValidator;
wrapWithDirectory: boolean;
}
export type ImporterSourceStream = AsyncIterable<ImportCandidate> | Iterable<ImportCandidate>;
export interface DAGBuilder {
(source: ImporterSourceStream, blockstore: WritableStorage): AsyncIterable<() => Promise<InProgressImportResult>>;
}
export declare function defaultDagBuilder(options: DagBuilderOptions): DAGBuilder;
//# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/dag-builder/index.ts"],"names":[],"mappings":"AAEA,OAAO,EAAc,KAAK,iBAAiB,EAAE,MAAM,UAAU,CAAA;AAC7D,OAAO,EAAe,KAAK,kBAAkB,EAAE,MAAM,WAAW,CAAA;AAChE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAA;AAC1D,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAClD,OAAO,KAAK,EAAkC,eAAe,EAAE,sBAAsB,EAAE,sBAAsB,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AACnJ,OAAO,KAAK,EAAE,aAAa,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAA;AAErE;;GAEG;AACH,MAAM,WAAW,kBAAkB;IACjC;;OAEG;IACH,SAAS,EAAE,MAAM,CAAA;IAEjB;;OAEG;IACH,SAAS,EAAE,MAAM,CAAA;IAEjB;;OAEG;IACH,IAAI,CAAC,EAAE,MAAM,CAAA;CACd;AAED,MAAM,MAAM,wBAAwB,GAClC,aAAa,CAAC,oCAAoC,EAAE,kBAAkB,CAAC,CAAA;AA8BzE,MAAM,WAAW,iBAAkB,SAAQ,kBAAkB,EAAE,iBAAiB,EAAE,eAAe,CAAC,sBAAsB,CAAC;IACvH,OAAO,EAAE,OAAO,CAAA;IAChB,cAAc,EAAE,cAAc,CAAA;IAC9B,iBAAiB,EAAE,OAAO,CAAA;CAC3B;AAED,MAAM,MAAM,oBAAoB,GAAG,aAAa,CAAC,eAAe,CAAC,GAAG,QAAQ,CAAC,eAAe,CAAC,CAAA;AAE7F,MAAM,WAAW,UAAU;IACzB,CAAC,MAAM,EAAE,oBAAoB,EAAE,UAAU,EAAE,eAAe,GAAG,aAAa,CAAC,MAAM,OAAO,CAAC,sBAAsB,CAAC,CAAC,CAAA;CAClH;AAED,wBAAgB,iBAAiB,CAAE,OAAO,EAAE,iBAAiB,GAAG,UAAU,CAoDzE"}
@@ -0,0 +1,83 @@
import errCode from 'err-code';
import { CustomProgressEvent } from 'progress-events';
import { dirBuilder } from './dir.js';
import { fileBuilder } from './file.js';
function isIterable(thing) {
return Symbol.iterator in thing;
}
function isAsyncIterable(thing) {
return Symbol.asyncIterator in thing;
}
function contentAsAsyncIterable(content) {
try {
if (content instanceof Uint8Array) {
return (async function* () {
yield content;
}());
}
else if (isIterable(content)) {
return (async function* () {
yield* content;
}());
}
else if (isAsyncIterable(content)) {
return content;
}
}
catch {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT');
}
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT');
}
export function defaultDagBuilder(options) {
return async function* dagBuilder(source, blockstore) {
for await (const entry of source) {
let originalPath;
if (entry.path != null) {
originalPath = entry.path;
entry.path = entry.path
.split('/')
.filter(path => path != null && path !== '.')
.join('/');
}
if (isFileCandidate(entry)) {
const file = {
path: entry.path,
mtime: entry.mtime,
mode: entry.mode,
content: (async function* () {
let bytesRead = 0n;
for await (const chunk of options.chunker(options.chunkValidator(contentAsAsyncIterable(entry.content)))) {
const currentChunkSize = BigInt(chunk.byteLength);
bytesRead += currentChunkSize;
options.onProgress?.(new CustomProgressEvent('unixfs:importer:progress:file:read', {
bytesRead,
chunkSize: currentChunkSize,
path: entry.path
}));
yield chunk;
}
})(),
originalPath
};
yield async () => fileBuilder(file, blockstore, options);
}
else if (entry.path != null) {
const dir = {
path: entry.path,
mtime: entry.mtime,
mode: entry.mode,
originalPath
};
yield async () => dirBuilder(dir, blockstore, options);
}
else {
throw new Error('Import candidate must have content or path or both');
}
}
};
}
function isFileCandidate(entry) {
return entry.content != null;
}
//# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/dag-builder/index.ts"],"names":[],"mappings":"AAAA,OAAO,OAAO,MAAM,UAAU,CAAA;AAC9B,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAA;AACrD,OAAO,EAAE,UAAU,EAA0B,MAAM,UAAU,CAAA;AAC7D,OAAO,EAAE,WAAW,EAA2B,MAAM,WAAW,CAAA;AA6BhE,SAAS,UAAU,CAAE,KAAU;IAC7B,OAAO,MAAM,CAAC,QAAQ,IAAI,KAAK,CAAA;AACjC,CAAC;AAED,SAAS,eAAe,CAAE,KAAU;IAClC,OAAO,MAAM,CAAC,aAAa,IAAI,KAAK,CAAA;AACtC,CAAC;AAED,SAAS,sBAAsB,CAAE,OAAsE;IACrG,IAAI;QACF,IAAI,OAAO,YAAY,UAAU,EAAE;YACjC,OAAO,CAAC,KAAK,SAAU,CAAC;gBACtB,MAAM,OAAO,CAAA;YACf,CAAC,EAAE,CAAC,CAAA;SACL;aAAM,IAAI,UAAU,CAAC,OAAO,CAAC,EAAE;YAC9B,OAAO,CAAC,KAAK,SAAU,CAAC;gBACtB,KAAM,CAAC,CAAC,OAAO,CAAA;YACjB,CAAC,EAAE,CAAC,CAAA;SACL;aAAM,IAAI,eAAe,CAAC,OAAO,CAAC,EAAE;YACnC,OAAO,OAAO,CAAA;SACf;KACF;IAAC,MAAM;QACN,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,qBAAqB,CAAC,EAAE,qBAAqB,CAAC,CAAA;KACvE;IAED,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,qBAAqB,CAAC,EAAE,qBAAqB,CAAC,CAAA;AACxE,CAAC;AAcD,MAAM,UAAU,iBAAiB,CAAE,OAA0B;IAC3D,OAAO,KAAK,SAAU,CAAC,CAAC,UAAU,CAAE,MAAM,EAAE,UAAU;QACpD,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE;YAChC,IAAI,YAAgC,CAAA;YAEpC,IAAI,KAAK,CAAC,IAAI,IAAI,IAAI,EAAE;gBACtB,YAAY,GAAG,KAAK,CAAC,IAAI,CAAA;gBACzB,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI;qBACpB,KAAK,CAAC,GAAG,CAAC;qBACV,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC,IAAI,IAAI,IAAI,IAAI,IAAI,KAAK,GAAG,CAAC;qBAC5C,IAAI,CAAC,GAAG,CAAC,CAAA;aACb;YAED,IAAI,eAAe,CAAC,KAAK,CAAC,EAAE;gBAC1B,MAAM,IAAI,GAAS;oBACjB,IAAI,EAAE,KAAK,CAAC,IAAI;oBAChB,KAAK,EAAE,KAAK,CAAC,KAAK;oBAClB,IAAI,EAAE,KAAK,CAAC,IAAI;oBAChB,OAAO,EAAE,CAAC,KAAK,SAAU,CAAC;wBACxB,IAAI,SAAS,GAAG,EAAE,CAAA;wBAElB,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC,cAAc,CAAC,sBAAsB,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE;4BACxG,MAAM,gBAAgB,GAAG,MAAM,CAAC,KAAK,CAAC,UAAU,CAAC,CAAA;4BACjD,SAAS,IAAI,gBAAgB,CAAA;4BAE7B,OAAO,CAAC,UAAU,EAAE,CAAC,IAAI,mBAAmB,CAAqB,oCAAoC,EAAE;gCACrG,SAAS;gCACT,SAAS,EAAE,gBAAgB;gCAC3B,IAAI,EAAE,KAAK,CAAC,IAAI;6BACjB,CAAC,CAAC,CAAA;4BAEH,MAAM,KAAK,CAAA;yBACZ;oBACH,CAAC,CAAC,EAAE;oBACJ,YAAY;iBACb,CAAA;gBAED,MAAM,KAAK,IAAI,EAAE,CAAC,WAAW,CAAC,IAAI,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;aACzD;iBAAM,IAAI,KAAK,CAAC,IAAI,IAAI,IAAI,EAAE;gBAC7B,MAAM,GAAG,GAAc;oBACrB,IAAI,EAAE,KAAK,CAAC,IAAI;oBAChB,KAAK,EAAE,KAAK,CAAC,KAAK;oBAClB,IAAI,EAAE,KAAK,CAAC,IAAI;oBAChB,YAAY;iBACb,CAAA;gBAED,MAAM,KAAK,IAAI,EAAE,CAAC,UAAU,CAAC,GAAG,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;aACvD;iBAAM;gBACL,MAAM,IAAI,KAAK,CAAC,oDAAoD,CAAC,CAAA;aACtE;SACF;IACH,CAAC,CAAA;AACH,CAAC;AAED,SAAS,eAAe,CAAE,KAAU;IAClC,OAAO,KAAK,CAAC,OAAO,IAAI,IAAI,CAAA;AAC9B,CAAC"}
@@ -0,0 +1,5 @@
export interface ChunkValidator {
(source: AsyncIterable<Uint8Array>): AsyncIterable<Uint8Array>;
}
export declare const defaultChunkValidator: () => ChunkValidator;
//# sourceMappingURL=validate-chunks.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"validate-chunks.d.ts","sourceRoot":"","sources":["../../../src/dag-builder/validate-chunks.ts"],"names":[],"mappings":"AAGA,MAAM,WAAW,cAAc;IAAG,CAAC,MAAM,EAAE,aAAa,CAAC,UAAU,CAAC,GAAG,aAAa,CAAC,UAAU,CAAC,CAAA;CAAE;AAElG,eAAO,MAAM,qBAAqB,QAAO,cAkBxC,CAAA"}
@@ -0,0 +1,24 @@
import errCode from 'err-code';
import { fromString as uint8ArrayFromString } from 'uint8arrays/from-string';
export const defaultChunkValidator = () => {
return async function* validateChunks(source) {
for await (const content of source) {
if (content.length === undefined) {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT');
}
if (typeof content === 'string' || content instanceof String) {
yield uint8ArrayFromString(content.toString());
}
else if (Array.isArray(content)) {
yield Uint8Array.from(content);
}
else if (content instanceof Uint8Array) {
yield content;
}
else {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT');
}
}
};
};
//# sourceMappingURL=validate-chunks.js.map
@@ -0,0 +1 @@
{"version":3,"file":"validate-chunks.js","sourceRoot":"","sources":["../../../src/dag-builder/validate-chunks.ts"],"names":[],"mappings":"AAAA,OAAO,OAAO,MAAM,UAAU,CAAA;AAC9B,OAAO,EAAE,UAAU,IAAI,oBAAoB,EAAE,MAAM,yBAAyB,CAAA;AAI5E,MAAM,CAAC,MAAM,qBAAqB,GAAG,GAAmB,EAAE;IACxD,OAAO,KAAK,SAAU,CAAC,CAAC,cAAc,CAAE,MAAM;QAC5C,IAAI,KAAK,EAAE,MAAM,OAAO,IAAI,MAAM,EAAE;YAClC,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,EAAE;gBAChC,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,qBAAqB,CAAC,EAAE,qBAAqB,CAAC,CAAA;aACvE;YAED,IAAI,OAAO,OAAO,KAAK,QAAQ,IAAI,OAAO,YAAY,MAAM,EAAE;gBAC5D,MAAM,oBAAoB,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC,CAAA;aAC/C;iBAAM,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE;gBACjC,MAAM,UAAU,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;aAC/B;iBAAM,IAAI,OAAO,YAAY,UAAU,EAAE;gBACxC,MAAM,OAAO,CAAA;aACd;iBAAM;gBACL,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,qBAAqB,CAAC,EAAE,qBAAqB,CAAC,CAAA;aACvE;SACF;IACH,CAAC,CAAA;AACH,CAAC,CAAA"}
+20
View File
@@ -0,0 +1,20 @@
import { Dir, type DirProps } from './dir.js';
import { type PersistOptions } from './utils/persist.js';
import type { ImportResult, InProgressImportResult } from './index.js';
import type { Blockstore } from 'interface-blockstore';
export declare class DirFlat extends Dir {
private readonly _children;
constructor(props: DirProps, options: PersistOptions);
put(name: string, value: InProgressImportResult | Dir): Promise<void>;
get(name: string): Promise<InProgressImportResult | Dir | undefined>;
childCount(): number;
directChildrenCount(): number;
onlyChild(): InProgressImportResult | Dir;
eachChildSeries(): AsyncGenerator<{
key: string;
child: InProgressImportResult | Dir;
}, void, undefined>;
estimateNodeSize(): number;
flush(block: Blockstore): AsyncGenerator<ImportResult>;
}
//# sourceMappingURL=dir-flat.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"dir-flat.d.ts","sourceRoot":"","sources":["../../src/dir-flat.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,GAAG,EAAkB,KAAK,QAAQ,EAAE,MAAM,UAAU,CAAA;AAC7D,OAAO,EAAW,KAAK,cAAc,EAAE,MAAM,oBAAoB,CAAA;AACjE,OAAO,KAAK,EAAE,YAAY,EAAE,sBAAsB,EAAE,MAAM,YAAY,CAAA;AACtE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,sBAAsB,CAAA;AAGtD,qBAAa,OAAQ,SAAQ,GAAG;IAC9B,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA2C;gBAExD,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,cAAc;IAM/C,GAAG,CAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,sBAAsB,GAAG,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAQtE,GAAG,CAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,sBAAsB,GAAG,GAAG,GAAG,SAAS,CAAC;IAI3E,UAAU,IAAK,MAAM;IAIrB,mBAAmB,IAAK,MAAM;IAI9B,SAAS,IAAK,sBAAsB,GAAG,GAAG;IAIlC,eAAe,IAAK,cAAc,CAAC;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,sBAAsB,GAAG,GAAG,CAAA;KAAE,EAAE,IAAI,EAAE,SAAS,CAAC;IASjH,gBAAgB,IAAK,MAAM;IAkBnB,KAAK,CAAE,KAAK,EAAE,UAAU,GAAG,cAAc,CAAC,YAAY,CAAC;CAkDhE"}
+93
View File
@@ -0,0 +1,93 @@
import { encode, prepare } from '@ipld/dag-pb';
import { UnixFS } from 'ipfs-unixfs';
import { Dir, CID_V0, CID_V1 } from './dir.js';
import { persist } from './utils/persist.js';
export class DirFlat extends Dir {
_children;
constructor(props, options) {
super(props, options);
this._children = new Map();
}
async put(name, value) {
this.cid = undefined;
this.size = undefined;
this.nodeSize = undefined;
this._children.set(name, value);
}
async get(name) {
return Promise.resolve(this._children.get(name));
}
childCount() {
return this._children.size;
}
directChildrenCount() {
return this.childCount();
}
onlyChild() {
return this._children.values().next().value;
}
async *eachChildSeries() {
for (const [key, child] of this._children.entries()) {
yield {
key,
child
};
}
}
estimateNodeSize() {
if (this.nodeSize !== undefined) {
return this.nodeSize;
}
this.nodeSize = 0;
// estimate size only based on DAGLink name and CID byte lengths
// https://github.com/ipfs/go-unixfsnode/blob/37b47f1f917f1b2f54c207682f38886e49896ef9/data/builder/directory.go#L81-L96
for (const [name, child] of this._children.entries()) {
if (child.size != null && (child.cid != null)) {
this.nodeSize += name.length + (this.options.cidVersion === 1 ? CID_V1.bytes.byteLength : CID_V0.bytes.byteLength);
}
}
return this.nodeSize;
}
async *flush(block) {
const links = [];
for (const [name, child] of this._children.entries()) {
let result = child;
if (child instanceof Dir) {
for await (const entry of child.flush(block)) {
result = entry;
yield entry;
}
}
if (result.size != null && (result.cid != null)) {
links.push({
Name: name,
Tsize: Number(result.size),
Hash: result.cid
});
}
}
const unixfs = new UnixFS({
type: 'directory',
mtime: this.mtime,
mode: this.mode
});
const node = { Data: unixfs.marshal(), Links: links };
const buffer = encode(prepare(node));
const cid = await persist(buffer, block, this.options);
const size = buffer.length + node.Links.reduce(
/**
* @param {number} acc
* @param {PBLink} curr
*/
(acc, curr) => acc + (curr.Tsize == null ? 0 : curr.Tsize), 0);
this.cid = cid;
this.size = size;
yield {
cid,
unixfs,
path: this.path,
size: BigInt(size)
};
}
}
//# sourceMappingURL=dir-flat.js.map
@@ -0,0 +1 @@
{"version":3,"file":"dir-flat.js","sourceRoot":"","sources":["../../src/dir-flat.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAe,OAAO,EAAE,MAAM,cAAc,CAAA;AAC3D,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AACpC,OAAO,EAAE,GAAG,EAAE,MAAM,EAAE,MAAM,EAAiB,MAAM,UAAU,CAAA;AAC7D,OAAO,EAAE,OAAO,EAAuB,MAAM,oBAAoB,CAAA;AAKjE,MAAM,OAAO,OAAQ,SAAQ,GAAG;IACb,SAAS,CAA2C;IAErE,YAAa,KAAe,EAAE,OAAuB;QACnD,KAAK,CAAC,KAAK,EAAE,OAAO,CAAC,CAAA;QAErB,IAAI,CAAC,SAAS,GAAG,IAAI,GAAG,EAAE,CAAA;IAC5B,CAAC;IAED,KAAK,CAAC,GAAG,CAAE,IAAY,EAAE,KAAmC;QAC1D,IAAI,CAAC,GAAG,GAAG,SAAS,CAAA;QACpB,IAAI,CAAC,IAAI,GAAG,SAAS,CAAA;QACrB,IAAI,CAAC,QAAQ,GAAG,SAAS,CAAA;QAEzB,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACjC,CAAC;IAED,KAAK,CAAC,GAAG,CAAE,IAAY;QACrB,OAAO,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAA;IAClD,CAAC;IAED,UAAU;QACR,OAAO,IAAI,CAAC,SAAS,CAAC,IAAI,CAAA;IAC5B,CAAC;IAED,mBAAmB;QACjB,OAAO,IAAI,CAAC,UAAU,EAAE,CAAA;IAC1B,CAAC;IAED,SAAS;QACP,OAAO,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,CAAA;IAC7C,CAAC;IAED,KAAK,CAAC,CAAE,eAAe;QACrB,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,EAAE;YACnD,MAAM;gBACJ,GAAG;gBACH,KAAK;aACN,CAAA;SACF;IACH,CAAC;IAED,gBAAgB;QACd,IAAI,IAAI,CAAC,QAAQ,KAAK,SAAS,EAAE;YAC/B,OAAO,IAAI,CAAC,QAAQ,CAAA;SACrB;QAED,IAAI,CAAC,QAAQ,GAAG,CAAC,CAAA;QAEjB,gEAAgE;QAChE,wHAAwH;QACxH,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,EAAE;YACpD,IAAI,KAAK,CAAC,IAAI,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,IAAI,IAAI,CAAC,EAAE;gBAC7C,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,UAAU,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,UAAU,CAAC,CAAA;aACnH;SACF;QAED,OAAO,IAAI,CAAC,QAAQ,CAAA;IACtB,CAAC;IAED,KAAK,CAAC,CAAE,KAAK,CAAE,KAAiB;QAC9B,MAAM,KAAK,GAAG,EAAE,CAAA;QAEhB,KAAK,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,EAAE;YACpD,IAAI,MAAM,GAA0C,KAAK,CAAA;YAEzD,IAAI,KAAK,YAAY,GAAG,EAAE;gBACxB,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,KAAK,CAAC,KAAK,CAAC,KAAK,CAAC,EAAE;oBAC5C,MAAM,GAAG,KAAK,CAAA;oBAEd,MAAM,KAAK,CAAA;iBACZ;aACF;YAED,IAAI,MAAM,CAAC,IAAI,IAAI,IAAI,IAAI,CAAC,MAAM,CAAC,GAAG,IAAI,IAAI,CAAC,EAAE;gBAC/C,KAAK,CAAC,IAAI,CAAC;oBACT,IAAI,EAAE,IAAI;oBACV,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC;oBAC1B,IAAI,EAAE,MAAM,CAAC,GAAG;iBACjB,CAAC,CAAA;aACH;SACF;QAED,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;YACxB,IAAI,EAAE,WAAW;YACjB,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,IAAI,EAAE,IAAI,CAAC,IAAI;SAChB,CAAC,CAAA;QAEF,MAAM,IAAI,GAAW,EAAE,IAAI,EAAE,MAAM,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAA;QAC7D,MAAM,MAAM,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAA;QACpC,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,MAAM,EAAE,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,CAAA;QACtD,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM;QAC5C;;;WAGG;QACH,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,EAC1D,CAAC,CAAC,CAAA;QAEJ,IAAI,CAAC,GAAG,GAAG,GAAG,CAAA;QACd,IAAI,CAAC,IAAI,GAAG,IAAI,CAAA;QAEhB,MAAM;YACJ,GAAG;YACH,MAAM;YACN,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,IAAI,EAAE,MAAM,CAAC,IAAI,CAAC;SACnB,CAAA;IACH,CAAC;CACF"}
@@ -0,0 +1,22 @@
import { Bucket, type BucketChild } from 'hamt-sharding';
import { Dir, type DirProps } from './dir.js';
import { type PersistOptions } from './utils/persist.js';
import type { ImportResult, InProgressImportResult } from './index.js';
import type { Blockstore } from 'interface-blockstore';
declare class DirSharded extends Dir {
private readonly _bucket;
constructor(props: DirProps, options: PersistOptions);
put(name: string, value: InProgressImportResult | Dir): Promise<void>;
get(name: string): Promise<InProgressImportResult | Dir | undefined>;
childCount(): number;
directChildrenCount(): number;
onlyChild(): Bucket<InProgressImportResult | Dir> | BucketChild<InProgressImportResult | Dir>;
eachChildSeries(): AsyncGenerator<{
key: string;
child: InProgressImportResult | Dir;
}>;
estimateNodeSize(): number;
flush(blockstore: Blockstore): AsyncGenerator<ImportResult>;
}
export default DirSharded;
//# sourceMappingURL=dir-sharded.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"dir-sharded.d.ts","sourceRoot":"","sources":["../../src/dir-sharded.ts"],"names":[],"mappings":"AAEA,OAAO,EAAc,MAAM,EAAE,KAAK,WAAW,EAAE,MAAM,eAAe,CAAA;AAEpE,OAAO,EAAE,GAAG,EAAkB,KAAK,QAAQ,EAAE,MAAM,UAAU,CAAA;AAC7D,OAAO,EAAW,KAAK,cAAc,EAAE,MAAM,oBAAoB,CAAA;AACjE,OAAO,KAAK,EAAE,YAAY,EAAE,sBAAsB,EAAE,MAAM,YAAY,CAAA;AACtE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,sBAAsB,CAAA;AActD,cAAM,UAAW,SAAQ,GAAG;IAC1B,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAsC;gBAEjD,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,cAAc;IAS/C,GAAG,CAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,sBAAsB,GAAG,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAQtE,GAAG,CAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,sBAAsB,GAAG,GAAG,GAAG,SAAS,CAAC;IAI3E,UAAU,IAAK,MAAM;IAIrB,mBAAmB,IAAK,MAAM;IAI9B,SAAS,IAAK,MAAM,CAAC,sBAAsB,GAAG,GAAG,CAAC,GAAG,WAAW,CAAC,sBAAsB,GAAG,GAAG,CAAC;IAItF,eAAe,IAAK,cAAc,CAAC;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,sBAAsB,GAAG,GAAG,CAAA;KAAE,CAAC;IAShG,gBAAgB,IAAK,MAAM;IAUnB,KAAK,CAAE,UAAU,EAAE,UAAU,GAAG,cAAc,CAAC,YAAY,CAAC;CAQrE;AAED,eAAe,UAAU,CAAA"}
+211
View File
@@ -0,0 +1,211 @@
import { encode, prepare } from '@ipld/dag-pb';
import { murmur3128 } from '@multiformats/murmur3';
import { createHAMT, Bucket } from 'hamt-sharding';
import { UnixFS } from 'ipfs-unixfs';
import { Dir, CID_V0, CID_V1 } from './dir.js';
import { persist } from './utils/persist.js';
async function hamtHashFn(buf) {
return (await murmur3128.encode(buf))
// Murmur3 outputs 128 bit but, accidentally, IPFS Go's
// implementation only uses the first 64, so we must do the same
// for parity..
.slice(0, 8)
// Invert buffer because that's how Go impl does it
.reverse();
}
const HAMT_HASH_CODE = BigInt(0x22);
class DirSharded extends Dir {
_bucket;
constructor(props, options) {
super(props, options);
this._bucket = createHAMT({
hashFn: hamtHashFn,
bits: 8
});
}
async put(name, value) {
this.cid = undefined;
this.size = undefined;
this.nodeSize = undefined;
await this._bucket.put(name, value);
}
async get(name) {
return this._bucket.get(name);
}
childCount() {
return this._bucket.leafCount();
}
directChildrenCount() {
return this._bucket.childrenCount();
}
onlyChild() {
return this._bucket.onlyChild();
}
async *eachChildSeries() {
for await (const { key, value } of this._bucket.eachLeafSeries()) {
yield {
key,
child: value
};
}
}
estimateNodeSize() {
if (this.nodeSize !== undefined) {
return this.nodeSize;
}
this.nodeSize = calculateSize(this._bucket, this, this.options);
return this.nodeSize;
}
async *flush(blockstore) {
for await (const entry of flush(this._bucket, blockstore, this, this.options)) {
yield {
...entry,
path: this.path
};
}
}
}
export default DirSharded;
async function* flush(bucket, blockstore, shardRoot, options) {
const children = bucket._children;
const links = [];
let childrenSize = 0n;
for (let i = 0; i < children.length; i++) {
const child = children.get(i);
if (child == null) {
continue;
}
const labelPrefix = i.toString(16).toUpperCase().padStart(2, '0');
if (child instanceof Bucket) {
let shard;
for await (const subShard of flush(child, blockstore, null, options)) {
shard = subShard;
}
if (shard == null) {
throw new Error('Could not flush sharded directory, no subshard found');
}
links.push({
Name: labelPrefix,
Tsize: Number(shard.size),
Hash: shard.cid
});
childrenSize += shard.size;
}
else if (isDir(child.value)) {
const dir = child.value;
let flushedDir;
for await (const entry of dir.flush(blockstore)) {
flushedDir = entry;
yield flushedDir;
}
if (flushedDir == null) {
throw new Error('Did not flush dir');
}
const label = labelPrefix + child.key;
links.push({
Name: label,
Tsize: Number(flushedDir.size),
Hash: flushedDir.cid
});
childrenSize += flushedDir.size;
}
else {
const value = child.value;
if (value.cid == null) {
continue;
}
const label = labelPrefix + child.key;
const size = value.size;
links.push({
Name: label,
Tsize: Number(size),
Hash: value.cid
});
childrenSize += BigInt(size ?? 0);
}
}
// go-ipfs uses little endian, that's why we have to
// reverse the bit field before storing it
const data = Uint8Array.from(children.bitField().reverse());
const dir = new UnixFS({
type: 'hamt-sharded-directory',
data,
fanout: BigInt(bucket.tableSize()),
hashType: HAMT_HASH_CODE,
mtime: shardRoot?.mtime,
mode: shardRoot?.mode
});
const node = {
Data: dir.marshal(),
Links: links
};
const buffer = encode(prepare(node));
const cid = await persist(buffer, blockstore, options);
const size = BigInt(buffer.byteLength) + childrenSize;
yield {
cid,
unixfs: dir,
size
};
}
function isDir(obj) {
return typeof obj.flush === 'function';
}
function calculateSize(bucket, shardRoot, options) {
const children = bucket._children;
const links = [];
for (let i = 0; i < children.length; i++) {
const child = children.get(i);
if (child == null) {
continue;
}
const labelPrefix = i.toString(16).toUpperCase().padStart(2, '0');
if (child instanceof Bucket) {
const size = calculateSize(child, null, options);
links.push({
Name: labelPrefix,
Tsize: Number(size),
Hash: options.cidVersion === 0 ? CID_V0 : CID_V1
});
}
else if (typeof child.value.flush === 'function') {
const dir = child.value;
const size = dir.nodeSize();
links.push({
Name: labelPrefix + child.key,
Tsize: Number(size),
Hash: options.cidVersion === 0 ? CID_V0 : CID_V1
});
}
else {
const value = child.value;
if (value.cid == null) {
continue;
}
const label = labelPrefix + child.key;
const size = value.size;
links.push({
Name: label,
Tsize: Number(size),
Hash: value.cid
});
}
}
// go-ipfs uses little endian, that's why we have to
// reverse the bit field before storing it
const data = Uint8Array.from(children.bitField().reverse());
const dir = new UnixFS({
type: 'hamt-sharded-directory',
data,
fanout: BigInt(bucket.tableSize()),
hashType: HAMT_HASH_CODE,
mtime: shardRoot?.mtime,
mode: shardRoot?.mode
});
const buffer = encode(prepare({
Data: dir.marshal(),
Links: links
}));
return buffer.length;
}
//# sourceMappingURL=dir-sharded.js.map
File diff suppressed because one or more lines are too long
+45
View File
@@ -0,0 +1,45 @@
import { CID } from 'multiformats/cid';
import type { WritableStorage, ImportResult, InProgressImportResult } from './index.js';
import type { PersistOptions } from './utils/persist.js';
import type { Mtime, UnixFS } from 'ipfs-unixfs';
export interface DirProps {
root: boolean;
dir: boolean;
path: string;
dirty: boolean;
flat: boolean;
parent?: Dir;
parentKey?: string;
unixfs?: UnixFS;
mode?: number;
mtime?: Mtime;
}
export declare abstract class Dir {
options: PersistOptions;
root: boolean;
dir: boolean;
path: string;
dirty: boolean;
flat: boolean;
parent?: Dir;
parentKey?: string;
unixfs?: UnixFS;
mode?: number;
mtime?: Mtime;
cid?: CID;
size?: number;
nodeSize?: number;
constructor(props: DirProps, options: PersistOptions);
abstract put(name: string, value: InProgressImportResult | Dir): Promise<void>;
abstract get(name: string): Promise<InProgressImportResult | Dir | undefined>;
abstract eachChildSeries(): AsyncIterable<{
key: string;
child: InProgressImportResult | Dir;
}>;
abstract flush(blockstore: WritableStorage): AsyncGenerator<ImportResult>;
abstract estimateNodeSize(): number;
abstract childCount(): number;
}
export declare const CID_V0: CID<unknown, number, number, import("multiformats/cid").Version>;
export declare const CID_V1: CID<unknown, number, number, import("multiformats/cid").Version>;
//# sourceMappingURL=dir.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"dir.d.ts","sourceRoot":"","sources":["../../src/dir.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAA;AACtC,OAAO,KAAK,EAAE,eAAe,EAAE,YAAY,EAAE,sBAAsB,EAAE,MAAM,YAAY,CAAA;AACvF,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AACxD,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAA;AAEhD,MAAM,WAAW,QAAQ;IACvB,IAAI,EAAE,OAAO,CAAA;IACb,GAAG,EAAE,OAAO,CAAA;IACZ,IAAI,EAAE,MAAM,CAAA;IACZ,KAAK,EAAE,OAAO,CAAA;IACd,IAAI,EAAE,OAAO,CAAA;IACb,MAAM,CAAC,EAAE,GAAG,CAAA;IACZ,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,KAAK,CAAC,EAAE,KAAK,CAAA;CACd;AAED,8BAAsB,GAAG;IAChB,OAAO,EAAE,cAAc,CAAA;IACvB,IAAI,EAAE,OAAO,CAAA;IACb,GAAG,EAAE,OAAO,CAAA;IACZ,IAAI,EAAE,MAAM,CAAA;IACZ,KAAK,EAAE,OAAO,CAAA;IACd,IAAI,EAAE,OAAO,CAAA;IACb,MAAM,CAAC,EAAE,GAAG,CAAA;IACZ,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,KAAK,CAAC,EAAE,KAAK,CAAA;IACb,GAAG,CAAC,EAAE,GAAG,CAAA;IACT,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,QAAQ,CAAC,EAAE,MAAM,CAAA;gBAEX,KAAK,EAAE,QAAQ,EAAE,OAAO,EAAE,cAAc;IAerD,QAAQ,CAAC,GAAG,CAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,sBAAsB,GAAG,GAAG,GAAG,OAAO,CAAC,IAAI,CAAC;IAC/E,QAAQ,CAAC,GAAG,CAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,sBAAsB,GAAG,GAAG,GAAG,SAAS,CAAC;IAC9E,QAAQ,CAAC,eAAe,IAAK,aAAa,CAAC;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,sBAAsB,GAAG,GAAG,CAAA;KAAE,CAAC;IAChG,QAAQ,CAAC,KAAK,CAAE,UAAU,EAAE,eAAe,GAAG,cAAc,CAAC,YAAY,CAAC;IAC1E,QAAQ,CAAC,gBAAgB,IAAK,MAAM;IACpC,QAAQ,CAAC,UAAU,IAAK,MAAM;CAC/B;AAMD,eAAO,MAAM,MAAM,kEAA8D,CAAA;AACjF,eAAO,MAAM,MAAM,kEAAiE,CAAA"}
+37
View File
@@ -0,0 +1,37 @@
import { CID } from 'multiformats/cid';
export class Dir {
options;
root;
dir;
path;
dirty;
flat;
parent;
parentKey;
unixfs;
mode;
mtime;
cid;
size;
nodeSize;
constructor(props, options) {
this.options = options ?? {};
this.root = props.root;
this.dir = props.dir;
this.path = props.path;
this.dirty = props.dirty;
this.flat = props.flat;
this.parent = props.parent;
this.parentKey = props.parentKey;
this.unixfs = props.unixfs;
this.mode = props.mode;
this.mtime = props.mtime;
}
}
// we use these to calculate the node size to use as a check for whether a directory
// should be sharded or not. Since CIDs have a constant length and We're only
// interested in the data length and not the actual content identifier we can use
// any old CID instead of having to hash the data which is expensive.
export const CID_V0 = CID.parse('QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn');
export const CID_V1 = CID.parse('zdj7WbTaiJT1fgatdet9Ei9iDB5hdCxkbVyhyh8YTUnXMiwYi');
//# sourceMappingURL=dir.js.map
+1
View File
@@ -0,0 +1 @@
{"version":3,"file":"dir.js","sourceRoot":"","sources":["../../src/dir.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAA;AAkBtC,MAAM,OAAgB,GAAG;IAChB,OAAO,CAAgB;IACvB,IAAI,CAAS;IACb,GAAG,CAAS;IACZ,IAAI,CAAQ;IACZ,KAAK,CAAS;IACd,IAAI,CAAS;IACb,MAAM,CAAM;IACZ,SAAS,CAAS;IAClB,MAAM,CAAS;IACf,IAAI,CAAS;IACb,KAAK,CAAQ;IACb,GAAG,CAAM;IACT,IAAI,CAAS;IACb,QAAQ,CAAS;IAExB,YAAa,KAAe,EAAE,OAAuB;QACnD,IAAI,CAAC,OAAO,GAAG,OAAO,IAAI,EAAE,CAAA;QAE5B,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAA;QACtB,IAAI,CAAC,GAAG,GAAG,KAAK,CAAC,GAAG,CAAA;QACpB,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAA;QACtB,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAA;QACxB,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAA;QACtB,IAAI,CAAC,MAAM,GAAG,KAAK,CAAC,MAAM,CAAA;QAC1B,IAAI,CAAC,SAAS,GAAG,KAAK,CAAC,SAAS,CAAA;QAChC,IAAI,CAAC,MAAM,GAAG,KAAK,CAAC,MAAM,CAAA;QAC1B,IAAI,CAAC,IAAI,GAAG,KAAK,CAAC,IAAI,CAAA;QACtB,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC,KAAK,CAAA;IAC1B,CAAC;CAQF;AAED,oFAAoF;AACpF,6EAA6E;AAC7E,iFAAiF;AACjF,qEAAqE;AACrE,MAAM,CAAC,MAAM,MAAM,GAAG,GAAG,CAAC,KAAK,CAAC,gDAAgD,CAAC,CAAA;AACjF,MAAM,CAAC,MAAM,MAAM,GAAG,GAAG,CAAC,KAAK,CAAC,mDAAmD,CAAC,CAAA"}
@@ -0,0 +1,5 @@
import DirSharded from './dir-sharded.js';
import type { Dir } from './dir.js';
import type { PersistOptions } from './utils/persist.js';
export declare function flatToShard(child: Dir | null, dir: Dir, threshold: number, options: PersistOptions): Promise<DirSharded>;
//# sourceMappingURL=flat-to-shard.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"flat-to-shard.d.ts","sourceRoot":"","sources":["../../src/flat-to-shard.ts"],"names":[],"mappings":"AACA,OAAO,UAAU,MAAM,kBAAkB,CAAA;AACzC,OAAO,KAAK,EAAE,GAAG,EAAE,MAAM,UAAU,CAAA;AACnC,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAExD,wBAAsB,WAAW,CAAE,KAAK,EAAE,GAAG,GAAG,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,SAAS,EAAE,MAAM,EAAE,OAAO,EAAE,cAAc,GAAG,OAAO,CAAC,UAAU,CAAC,CA0B/H"}
@@ -0,0 +1,40 @@
import { DirFlat } from './dir-flat.js';
import DirSharded from './dir-sharded.js';
export async function flatToShard(child, dir, threshold, options) {
let newDir = dir;
if (dir instanceof DirFlat && dir.estimateNodeSize() > threshold) {
newDir = await convertToShard(dir, options);
}
const parent = newDir.parent;
if (parent != null) {
if (newDir !== dir) {
if (child != null) {
child.parent = newDir;
}
if (newDir.parentKey == null) {
throw new Error('No parent key found');
}
await parent.put(newDir.parentKey, newDir);
}
return flatToShard(newDir, parent, threshold, options);
}
return newDir;
}
async function convertToShard(oldDir, options) {
const newDir = new DirSharded({
root: oldDir.root,
dir: true,
parent: oldDir.parent,
parentKey: oldDir.parentKey,
path: oldDir.path,
dirty: oldDir.dirty,
flat: false,
mtime: oldDir.mtime,
mode: oldDir.mode
}, options);
for await (const { key, child } of oldDir.eachChildSeries()) {
await newDir.put(key, child);
}
return newDir;
}
//# sourceMappingURL=flat-to-shard.js.map
@@ -0,0 +1 @@
{"version":3,"file":"flat-to-shard.js","sourceRoot":"","sources":["../../src/flat-to-shard.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,eAAe,CAAA;AACvC,OAAO,UAAU,MAAM,kBAAkB,CAAA;AAIzC,MAAM,CAAC,KAAK,UAAU,WAAW,CAAE,KAAiB,EAAE,GAAQ,EAAE,SAAiB,EAAE,OAAuB;IACxG,IAAI,MAAM,GAAG,GAAiB,CAAA;IAE9B,IAAI,GAAG,YAAY,OAAO,IAAI,GAAG,CAAC,gBAAgB,EAAE,GAAG,SAAS,EAAE;QAChE,MAAM,GAAG,MAAM,cAAc,CAAC,GAAG,EAAE,OAAO,CAAC,CAAA;KAC5C;IAED,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAA;IAE5B,IAAI,MAAM,IAAI,IAAI,EAAE;QAClB,IAAI,MAAM,KAAK,GAAG,EAAE;YAClB,IAAI,KAAK,IAAI,IAAI,EAAE;gBACjB,KAAK,CAAC,MAAM,GAAG,MAAM,CAAA;aACtB;YAED,IAAI,MAAM,CAAC,SAAS,IAAI,IAAI,EAAE;gBAC5B,MAAM,IAAI,KAAK,CAAC,qBAAqB,CAAC,CAAA;aACvC;YAED,MAAM,MAAM,CAAC,GAAG,CAAC,MAAM,CAAC,SAAS,EAAE,MAAM,CAAC,CAAA;SAC3C;QAED,OAAO,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,OAAO,CAAC,CAAA;KACvD;IAED,OAAO,MAAM,CAAA;AACf,CAAC;AAED,KAAK,UAAU,cAAc,CAAE,MAAe,EAAE,OAAuB;IACrE,MAAM,MAAM,GAAG,IAAI,UAAU,CAAC;QAC5B,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,GAAG,EAAE,IAAI;QACT,MAAM,EAAE,MAAM,CAAC,MAAM;QACrB,SAAS,EAAE,MAAM,CAAC,SAAS;QAC3B,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,KAAK,EAAE,MAAM,CAAC,KAAK;QACnB,IAAI,EAAE,KAAK;QACX,KAAK,EAAE,MAAM,CAAC,KAAK;QACnB,IAAI,EAAE,MAAM,CAAC,IAAI;KAClB,EAAE,OAAO,CAAC,CAAA;IAEX,IAAI,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,MAAM,CAAC,eAAe,EAAE,EAAE;QAC3D,MAAM,MAAM,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAA;KAC7B;IAED,OAAO,MAAM,CAAA;AACf,CAAC"}
+288
View File
@@ -0,0 +1,288 @@
import { type BufferImportProgressEvents } from './dag-builder/buffer-importer.js';
import { type DAGBuilder, type DagBuilderProgressEvents } from './dag-builder/index.js';
import { type ChunkValidator } from './dag-builder/validate-chunks.js';
import { type FileLayout } from './layout/index.js';
import type { Chunker } from './chunker/index.js';
import type { ReducerProgressEvents } from './dag-builder/file.js';
import type { Blockstore } from 'interface-blockstore';
import type { AwaitIterable } from 'interface-store';
import type { UnixFS, Mtime } from 'ipfs-unixfs';
import type { CID, Version as CIDVersion } from 'multiformats/cid';
import type { ProgressOptions } from 'progress-events';
export type ByteStream = AwaitIterable<Uint8Array>;
export type ImportContent = ByteStream | Uint8Array;
export type WritableStorage = Pick<Blockstore, 'put'>;
export interface FileCandidate {
path?: string;
content: ImportContent;
mtime?: Mtime;
mode?: number;
}
export interface DirectoryCandidate {
path: string;
mtime?: Mtime;
mode?: number;
}
export type ImportCandidate = FileCandidate | DirectoryCandidate;
export interface File {
content: AsyncIterable<Uint8Array>;
path?: string;
mtime?: Mtime;
mode?: number;
originalPath?: string;
}
export interface Directory {
path?: string;
mtime?: Mtime;
mode?: number;
originalPath?: string;
}
export interface ImportResult {
cid: CID;
size: bigint;
path?: string;
unixfs?: UnixFS;
}
export interface MultipleBlockImportResult extends ImportResult {
originalPath?: string;
}
export interface SingleBlockImportResult extends ImportResult {
single: true;
originalPath?: string;
block: Uint8Array;
}
export type InProgressImportResult = SingleBlockImportResult | MultipleBlockImportResult;
export interface BufferImporterResult extends ImportResult {
block: Uint8Array;
}
export interface HamtHashFn {
(value: Uint8Array): Promise<Uint8Array>;
}
export interface TreeBuilder {
(source: AsyncIterable<InProgressImportResult>, blockstore: WritableStorage): AsyncIterable<ImportResult>;
}
export interface BufferImporter {
(file: File, blockstore: WritableStorage): AsyncIterable<() => Promise<BufferImporterResult>>;
}
export type ImporterProgressEvents = BufferImportProgressEvents | DagBuilderProgressEvents | ReducerProgressEvents;
/**
* Options to control the importer's behaviour
*/
export interface ImporterOptions extends ProgressOptions<ImporterProgressEvents> {
/**
* When a file would span multiple DAGNodes, if this is true the leaf nodes
* will not be wrapped in `UnixFS` protobufs and will instead contain the
* raw file bytes. Default: true
*/
rawLeaves?: boolean;
/**
* If the file being imported is small enough to fit into one DAGNodes, store
* the file data in the root node along with the UnixFS metadata instead of
* in a leaf node which would then require additional I/O to load. Default: true
*/
reduceSingleLeafToSelf?: boolean;
/**
* What type of UnixFS node leaves should be - can be `'file'` or `'raw'`
* (ignored when `rawLeaves` is `true`).
*
* This option exists to simulate kubo's trickle dag which uses a combination
* of `'raw'` UnixFS leaves and `reduceSingleLeafToSelf: false`.
*
* For modern code the `rawLeaves: true` option should be used instead so leaves
* are plain Uint8Arrays without a UnixFS/Protobuf wrapper.
*/
leafType?: 'file' | 'raw';
/**
* the CID version to use when storing the data. Default: 1
*/
cidVersion?: CIDVersion;
/**
* If the serialized node is larger than this it might be converted to a HAMT
* sharded directory. Default: 256KiB
*/
shardSplitThresholdBytes?: number;
/**
* How many files to import concurrently. For large numbers of small files this
* should be high (e.g. 50). Default: 10
*/
fileImportConcurrency?: number;
/**
* How many blocks to hash and write to the block store concurrently. For small
* numbers of large files this should be high (e.g. 50). Default: 50
*/
blockWriteConcurrency?: number;
/**
* If true, all imported files and folders will be contained in a directory that
* will correspond to the CID of the final entry yielded. Default: false
*/
wrapWithDirectory?: boolean;
/**
* The chunking strategy. See [./src/chunker/index.ts](./src/chunker/index.ts)
* for available chunkers. Default: fixedSize
*/
chunker?: Chunker;
/**
* How the DAG that represents files are created. See
* [./src/layout/index.ts](./src/layout/index.ts) for available layouts. Default: balanced
*/
layout?: FileLayout;
/**
* This option can be used to override the importer internals.
*
* This function should read `{ path, content }` entries from `source` and turn them
* into DAGs
* It should yield a `function` that returns a `Promise` that resolves to
* `{ cid, path, unixfs, node }` where `cid` is a `CID`, `path` is a string, `unixfs`
* is a UnixFS entry and `node` is a `DAGNode`.
* Values will be pulled from this generator in parallel - the amount of parallelisation
* is controlled by the `fileImportConcurrency` option (default: 50)
*/
dagBuilder?: DAGBuilder;
/**
* This option can be used to override the importer internals.
*
* This function should read `{ cid, path, unixfs, node }` entries from `source` and
* place them in a directory structure
* It should yield an object with the properties `{ cid, path, unixfs, size }` where
* `cid` is a `CID`, `path` is a string, `unixfs` is a UnixFS entry and `size` is a `Number`.
*/
treeBuilder?: TreeBuilder;
/**
* This option can be used to override the importer internals.
*
* This function should read `Buffer`s from `source` and persist them using `blockstore.put`
* or similar
* `entry` is the `{ path, content }` entry, where `entry.content` is an async
* generator that yields Buffers
* It should yield functions that return a Promise that resolves to an object with
* the properties `{ cid, unixfs, size }` where `cid` is a [CID], `unixfs` is a [UnixFS] entry and `size` is a `Number` that represents the serialized size of the [IPLD] node that holds the buffer data.
* Values will be pulled from this generator in parallel - the amount of
* parallelisation is controlled by the `blockWriteConcurrency` option (default: 10)
*/
bufferImporter?: BufferImporter;
/**
* This option can be used to override the importer internals.
*
* This function takes input from the `content` field of imported entries.
* It should transform them into `Buffer`s, throwing an error if it cannot.
* It should yield `Buffer` objects constructed from the `source` or throw an
* `Error`
*/
chunkValidator?: ChunkValidator;
}
export type ImportCandidateStream = AsyncIterable<FileCandidate | DirectoryCandidate> | Iterable<FileCandidate | DirectoryCandidate>;
/**
* The importer creates UnixFS DAGs and stores the blocks that make
* them up in the passed blockstore.
*
* @example
*
* ```typescript
* import { importer } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [{
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }, {
* path: './bar.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }]
*
* for await (const entry of importer(input, blockstore)) {
* console.info(entry)
* // { cid: CID(), ... }
* }
* ```
*/
export declare function importer(source: ImportCandidateStream, blockstore: WritableStorage, options?: ImporterOptions): AsyncGenerator<ImportResult, void, unknown>;
/**
* `importFile` is similar to `importer` except it accepts a single
* `FileCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importFile } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: FileCandidate = {
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }
*
* const entry = await importFile(input, blockstore)
* ```
*/
export declare function importFile(content: FileCandidate, blockstore: WritableStorage, options?: ImporterOptions): Promise<ImportResult>;
/**
* `importDir` is similar to `importer` except it accepts a single
* `DirectoryCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importDirectory } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: DirectoryCandidate = {
* path: './foo.txt'
* }
*
* const entry = await importDirectory(input, blockstore)
* ```
*/
export declare function importDirectory(content: DirectoryCandidate, blockstore: WritableStorage, options?: ImporterOptions): Promise<ImportResult>;
/**
* `importBytes` accepts a single Uint8Array and returns a promise
* of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importBytes } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = Uint8Array.from([0, 1, 2, 3, 4])
*
* const entry = await importBytes(input, blockstore)
* ```
*/
export declare function importBytes(buf: ImportContent, blockstore: WritableStorage, options?: ImporterOptions): Promise<ImportResult>;
/**
* `importByteStream` accepts a single stream of Uint8Arrays and
* returns a promise of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importByteStream } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [
* Uint8Array.from([0, 1, 2, 3, 4]),
* Uint8Array.from([5, 6, 7, 8, 9])
* ]
*
* const entry = await importByteStream(input, blockstore)
* ```
*/
export declare function importByteStream(bufs: ByteStream, blockstore: WritableStorage, options?: ImporterOptions): Promise<ImportResult>;
//# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":"AAIA,OAAO,EAAE,KAAK,0BAA0B,EAAyB,MAAM,kCAAkC,CAAA;AACzG,OAAO,EAAE,KAAK,UAAU,EAAE,KAAK,wBAAwB,EAAqB,MAAM,wBAAwB,CAAA;AAC1G,OAAO,EAAE,KAAK,cAAc,EAAyB,MAAM,kCAAkC,CAAA;AAC7F,OAAO,EAAY,KAAK,UAAU,EAAE,MAAM,mBAAmB,CAAA;AAE7D,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,oBAAoB,CAAA;AACjD,OAAO,KAAK,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAA;AAClE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,sBAAsB,CAAA;AACtD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAA;AACpD,OAAO,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,aAAa,CAAA;AAChD,OAAO,KAAK,EAAE,GAAG,EAAE,OAAO,IAAI,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAClE,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAA;AAEtD,MAAM,MAAM,UAAU,GAAG,aAAa,CAAC,UAAU,CAAC,CAAA;AAClD,MAAM,MAAM,aAAa,GAAG,UAAU,GAAG,UAAU,CAAA;AAEnD,MAAM,MAAM,eAAe,GAAG,IAAI,CAAC,UAAU,EAAE,KAAK,CAAC,CAAA;AAErD,MAAM,WAAW,aAAa;IAC5B,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,OAAO,EAAE,aAAa,CAAA;IACtB,KAAK,CAAC,EAAE,KAAK,CAAA;IACb,IAAI,CAAC,EAAE,MAAM,CAAA;CACd;AAED,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,MAAM,CAAA;IACZ,KAAK,CAAC,EAAE,KAAK,CAAA;IACb,IAAI,CAAC,EAAE,MAAM,CAAA;CACd;AAED,MAAM,MAAM,eAAe,GAAG,aAAa,GAAG,kBAAkB,CAAA;AAEhE,MAAM,WAAW,IAAI;IACnB,OAAO,EAAE,aAAa,CAAC,UAAU,CAAC,CAAA;IAClC,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,KAAK,CAAC,EAAE,KAAK,CAAA;IACb,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,YAAY,CAAC,EAAE,MAAM,CAAA;CACtB;AAED,MAAM,WAAW,SAAS;IACxB,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,KAAK,CAAC,EAAE,KAAK,CAAA;IACb,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,YAAY,CAAC,EAAE,MAAM,CAAA;CACtB;AAED,MAAM,WAAW,YAAY;IAC3B,GAAG,EAAE,GAAG,CAAA;IACR,IAAI,EAAE,MAAM,CAAA;IACZ,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,MAAM,CAAC,EAAE,MAAM,CAAA;CAChB;AAED,MAAM,WAAW,yBAA0B,SAAQ,YAAY;IAC7D,YAAY,CAAC,EAAE,MAAM,CAAA;CACtB;AAED,MAAM,WAAW,uBAAwB,SAAQ,YAAY;IAC3D,MAAM,EAAE,IAAI,CAAA;IACZ,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,KAAK,EAAE,UAAU,CAAA;CAClB;AAED,MAAM,MAAM,sBAAsB,GAAG,uBAAuB,GAAG,yBAAyB,CAAA;AAExF,MAAM,WAAW,oBAAqB,SAAQ,YAAY;IACxD,KAAK,EAAE,UAAU,CAAA;CAClB;AAED,MAAM,WAAW,UAAU;IAAG,CAAC,KAAK,EAAE,UAAU,GAAG,OAAO,CAAC,UAAU,CAAC,CAAA;CAAE;AACxE,MAAM,WAAW,WAAW;IAAG,CAAC,MAAM,EAAE,aAAa,CAAC,sBAAsB,CAAC,EAAE,UAAU,EAAE,eAAe,GAAG,aAAa,CAAC,YAAY,CAAC,CAAA;CAAE;AAC1I,MAAM,WAAW,cAAc;IAAG,CAAC,IAAI,EAAE,IAAI,EAAE,UAAU,EAAE,eAAe,GAAG,aAAa,CAAC,MAAM,OAAO,CAAC,oBAAoB,CAAC,CAAC,CAAA;CAAE;AAEjI,MAAM,MAAM,sBAAsB,GAChC,0BAA0B,GAC1B,wBAAwB,GACxB,qBAAqB,CAAA;AAEvB;;GAEG;AACH,MAAM,WAAW,eAAgB,SAAQ,eAAe,CAAC,sBAAsB,CAAC;IAC9E;;;;OAIG;IACH,SAAS,CAAC,EAAE,OAAO,CAAA;IAEnB;;;;OAIG;IACH,sBAAsB,CAAC,EAAE,OAAO,CAAA;IAEhC;;;;;;;;;OASG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,KAAK,CAAA;IAEzB;;OAEG;IACH,UAAU,CAAC,EAAE,UAAU,CAAA;IAEvB;;;OAGG;IACH,wBAAwB,CAAC,EAAE,MAAM,CAAA;IAEjC;;;OAGG;IACH,qBAAqB,CAAC,EAAE,MAAM,CAAA;IAE9B;;;OAGG;IACH,qBAAqB,CAAC,EAAE,MAAM,CAAA;IAE9B;;;OAGG;IACH,iBAAiB,CAAC,EAAE,OAAO,CAAA;IAE3B;;;OAGG;IACH,OAAO,CAAC,EAAE,OAAO,CAAA;IAEjB;;;OAGG;IACH,MAAM,CAAC,EAAE,UAAU,CAAA;IAEnB;;;;;;;;;;OAUG;IACH,UAAU,CAAC,EAAE,UAAU,CAAA;IAEvB;;;;;;;OAOG;IACH,WAAW,CAAC,EAAE,WAAW,CAAA;IAEzB;;;;;;;;;;;OAWG;IACH,cAAc,CAAC,EAAE,cAAc,CAAA;IAE/B;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,cAAc,CAAA;CAChC;AAED,MAAM,MAAM,qBAAqB,GAAG,aAAa,CAAC,aAAa,GAAG,kBAAkB,CAAC,GAAG,QAAQ,CAAC,aAAa,GAAG,kBAAkB,CAAC,CAAA;AAEpI;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AACH,wBAAwB,QAAQ,CAAE,MAAM,EAAE,qBAAqB,EAAE,UAAU,EAAE,eAAe,EAAE,OAAO,GAAE,eAAoB,GAAG,cAAc,CAAC,YAAY,EAAE,IAAI,EAAE,OAAO,CAAC,CAmDxK;AAED;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,wBAAsB,UAAU,CAAE,OAAO,EAAE,aAAa,EAAE,UAAU,EAAE,eAAe,EAAE,OAAO,GAAE,eAAoB,GAAG,OAAO,CAAC,YAAY,CAAC,CAQ3I;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAsB,eAAe,CAAE,OAAO,EAAE,kBAAkB,EAAE,UAAU,EAAE,eAAe,EAAE,OAAO,GAAE,eAAoB,GAAG,OAAO,CAAC,YAAY,CAAC,CAQrJ;AAED;;;;;;;;;;;;;;;;;GAiBG;AACH,wBAAsB,WAAW,CAAE,GAAG,EAAE,aAAa,EAAE,UAAU,EAAE,eAAe,EAAE,OAAO,GAAE,eAAoB,GAAG,OAAO,CAAC,YAAY,CAAC,CAIxI;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,wBAAsB,gBAAgB,CAAE,IAAI,EAAE,UAAU,EAAE,UAAU,EAAE,eAAe,EAAE,OAAO,GAAE,eAAoB,GAAG,OAAO,CAAC,YAAY,CAAC,CAI3I"}
+192
View File
@@ -0,0 +1,192 @@
import errcode from 'err-code';
import first from 'it-first';
import parallelBatch from 'it-parallel-batch';
import { fixedSize } from './chunker/fixed-size.js';
import { defaultBufferImporter } from './dag-builder/buffer-importer.js';
import { defaultDagBuilder } from './dag-builder/index.js';
import { defaultChunkValidator } from './dag-builder/validate-chunks.js';
import { balanced } from './layout/index.js';
import { defaultTreeBuilder } from './tree-builder.js';
/**
* The importer creates UnixFS DAGs and stores the blocks that make
* them up in the passed blockstore.
*
* @example
*
* ```typescript
* import { importer } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [{
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }, {
* path: './bar.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }]
*
* for await (const entry of importer(input, blockstore)) {
* console.info(entry)
* // { cid: CID(), ... }
* }
* ```
*/
export async function* importer(source, blockstore, options = {}) {
let candidates;
if (Symbol.asyncIterator in source || Symbol.iterator in source) {
candidates = source;
}
else {
candidates = [source];
}
const wrapWithDirectory = options.wrapWithDirectory ?? false;
const shardSplitThresholdBytes = options.shardSplitThresholdBytes ?? 262144;
const cidVersion = options.cidVersion ?? 1;
const rawLeaves = options.rawLeaves ?? true;
const leafType = options.leafType ?? 'file';
const fileImportConcurrency = options.fileImportConcurrency ?? 50;
const blockWriteConcurrency = options.blockWriteConcurrency ?? 10;
const reduceSingleLeafToSelf = options.reduceSingleLeafToSelf ?? true;
const chunker = options.chunker ?? fixedSize();
const chunkValidator = options.chunkValidator ?? defaultChunkValidator();
const buildDag = options.dagBuilder ?? defaultDagBuilder({
chunker,
chunkValidator,
wrapWithDirectory,
layout: options.layout ?? balanced(),
bufferImporter: options.bufferImporter ?? defaultBufferImporter({
cidVersion,
rawLeaves,
leafType,
onProgress: options.onProgress
}),
blockWriteConcurrency,
reduceSingleLeafToSelf,
cidVersion,
onProgress: options.onProgress
});
const buildTree = options.treeBuilder ?? defaultTreeBuilder({
wrapWithDirectory,
shardSplitThresholdBytes,
cidVersion,
onProgress: options.onProgress
});
for await (const entry of buildTree(parallelBatch(buildDag(candidates, blockstore), fileImportConcurrency), blockstore)) {
yield {
cid: entry.cid,
path: entry.path,
unixfs: entry.unixfs,
size: entry.size
};
}
}
/**
* `importFile` is similar to `importer` except it accepts a single
* `FileCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importFile } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: FileCandidate = {
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }
*
* const entry = await importFile(input, blockstore)
* ```
*/
export async function importFile(content, blockstore, options = {}) {
const result = await first(importer([content], blockstore, options));
if (result == null) {
throw errcode(new Error('Nothing imported'), 'ERR_INVALID_PARAMS');
}
return result;
}
/**
* `importDir` is similar to `importer` except it accepts a single
* `DirectoryCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importDirectory } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: DirectoryCandidate = {
* path: './foo.txt'
* }
*
* const entry = await importDirectory(input, blockstore)
* ```
*/
export async function importDirectory(content, blockstore, options = {}) {
const result = await first(importer([content], blockstore, options));
if (result == null) {
throw errcode(new Error('Nothing imported'), 'ERR_INVALID_PARAMS');
}
return result;
}
/**
* `importBytes` accepts a single Uint8Array and returns a promise
* of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importBytes } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = Uint8Array.from([0, 1, 2, 3, 4])
*
* const entry = await importBytes(input, blockstore)
* ```
*/
export async function importBytes(buf, blockstore, options = {}) {
return importFile({
content: buf
}, blockstore, options);
}
/**
* `importByteStream` accepts a single stream of Uint8Arrays and
* returns a promise of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importByteStream } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [
* Uint8Array.from([0, 1, 2, 3, 4]),
* Uint8Array.from([5, 6, 7, 8, 9])
* ]
*
* const entry = await importByteStream(input, blockstore)
* ```
*/
export async function importByteStream(bufs, blockstore, options = {}) {
return importFile({
content: bufs
}, blockstore, options);
}
//# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,OAAO,MAAM,UAAU,CAAA;AAC9B,OAAO,KAAK,MAAM,UAAU,CAAA;AAC5B,OAAO,aAAa,MAAM,mBAAmB,CAAA;AAC7C,OAAO,EAAE,SAAS,EAAE,MAAM,yBAAyB,CAAA;AACnD,OAAO,EAAmC,qBAAqB,EAAE,MAAM,kCAAkC,CAAA;AACzG,OAAO,EAAkD,iBAAiB,EAAE,MAAM,wBAAwB,CAAA;AAC1G,OAAO,EAAuB,qBAAqB,EAAE,MAAM,kCAAkC,CAAA;AAC7F,OAAO,EAAE,QAAQ,EAAmB,MAAM,mBAAmB,CAAA;AAC7D,OAAO,EAAE,kBAAkB,EAAE,MAAM,mBAAmB,CAAA;AAqMtD;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AACH,MAAM,CAAC,KAAK,SAAU,CAAC,CAAC,QAAQ,CAAE,MAA6B,EAAE,UAA2B,EAAE,UAA2B,EAAE;IACzH,IAAI,UAA4G,CAAA;IAEhH,IAAI,MAAM,CAAC,aAAa,IAAI,MAAM,IAAI,MAAM,CAAC,QAAQ,IAAI,MAAM,EAAE;QAC/D,UAAU,GAAG,MAAM,CAAA;KACpB;SAAM;QACL,UAAU,GAAG,CAAC,MAAM,CAAC,CAAA;KACtB;IAED,MAAM,iBAAiB,GAAG,OAAO,CAAC,iBAAiB,IAAI,KAAK,CAAA;IAC5D,MAAM,wBAAwB,GAAG,OAAO,CAAC,wBAAwB,IAAI,MAAM,CAAA;IAC3E,MAAM,UAAU,GAAG,OAAO,CAAC,UAAU,IAAI,CAAC,CAAA;IAC1C,MAAM,SAAS,GAAG,OAAO,CAAC,SAAS,IAAI,IAAI,CAAA;IAC3C,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,MAAM,CAAA;IAC3C,MAAM,qBAAqB,GAAG,OAAO,CAAC,qBAAqB,IAAI,EAAE,CAAA;IACjE,MAAM,qBAAqB,GAAG,OAAO,CAAC,qBAAqB,IAAI,EAAE,CAAA;IACjE,MAAM,sBAAsB,GAAG,OAAO,CAAC,sBAAsB,IAAI,IAAI,CAAA;IAErE,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,SAAS,EAAE,CAAA;IAC9C,MAAM,cAAc,GAAG,OAAO,CAAC,cAAc,IAAI,qBAAqB,EAAE,CAAA;IACxE,MAAM,QAAQ,GAAe,OAAO,CAAC,UAAU,IAAI,iBAAiB,CAAC;QACnE,OAAO;QACP,cAAc;QACd,iBAAiB;QACjB,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,QAAQ,EAAE;QACpC,cAAc,EAAE,OAAO,CAAC,cAAc,IAAI,qBAAqB,CAAC;YAC9D,UAAU;YACV,SAAS;YACT,QAAQ;YACR,UAAU,EAAE,OAAO,CAAC,UAAU;SAC/B,CAAC;QACF,qBAAqB;QACrB,sBAAsB;QACtB,UAAU;QACV,UAAU,EAAE,OAAO,CAAC,UAAU;KAC/B,CAAC,CAAA;IACF,MAAM,SAAS,GAAgB,OAAO,CAAC,WAAW,IAAI,kBAAkB,CAAC;QACvE,iBAAiB;QACjB,wBAAwB;QACxB,UAAU;QACV,UAAU,EAAE,OAAO,CAAC,UAAU;KAC/B,CAAC,CAAA;IAEF,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,SAAS,CAAC,aAAa,CAAC,QAAQ,CAAC,UAAU,EAAE,UAAU,CAAC,EAAE,qBAAqB,CAAC,EAAE,UAAU,CAAC,EAAE;QACvH,MAAM;YACJ,GAAG,EAAE,KAAK,CAAC,GAAG;YACd,IAAI,EAAE,KAAK,CAAC,IAAI;YAChB,MAAM,EAAE,KAAK,CAAC,MAAM;YACpB,IAAI,EAAE,KAAK,CAAC,IAAI;SACjB,CAAA;KACF;AACH,CAAC;AAED;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,MAAM,CAAC,KAAK,UAAU,UAAU,CAAE,OAAsB,EAAE,UAA2B,EAAE,UAA2B,EAAE;IAClH,MAAM,MAAM,GAAG,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC,OAAO,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAA;IAEpE,IAAI,MAAM,IAAI,IAAI,EAAE;QAClB,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,kBAAkB,CAAC,EAAE,oBAAoB,CAAC,CAAA;KACnE;IAED,OAAO,MAAM,CAAA;AACf,CAAC;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CAAE,OAA2B,EAAE,UAA2B,EAAE,UAA2B,EAAE;IAC5H,MAAM,MAAM,GAAG,MAAM,KAAK,CAAC,QAAQ,CAAC,CAAC,OAAO,CAAC,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAA;IAEpE,IAAI,MAAM,IAAI,IAAI,EAAE;QAClB,MAAM,OAAO,CAAC,IAAI,KAAK,CAAC,kBAAkB,CAAC,EAAE,oBAAoB,CAAC,CAAA;KACnE;IAED,OAAO,MAAM,CAAA;AACf,CAAC;AAED;;;;;;;;;;;;;;;;;GAiBG;AACH,MAAM,CAAC,KAAK,UAAU,WAAW,CAAE,GAAkB,EAAE,UAA2B,EAAE,UAA2B,EAAE;IAC/G,OAAO,UAAU,CAAC;QAChB,OAAO,EAAE,GAAG;KACb,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;AACzB,CAAC;AAED;;;;;;;;;;;;;;;;;;;;GAoBG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CAAE,IAAgB,EAAE,UAA2B,EAAE,UAA2B,EAAE;IAClH,OAAO,UAAU,CAAC;QAChB,OAAO,EAAE,IAAI;KACd,EAAE,UAAU,EAAE,OAAO,CAAC,CAAA;AACzB,CAAC"}
@@ -0,0 +1,6 @@
import type { FileLayout } from './index.js';
export interface BalancedOptions {
maxChildrenPerNode?: number;
}
export declare function balanced(options?: BalancedOptions): FileLayout;
//# sourceMappingURL=balanced.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"balanced.d.ts","sourceRoot":"","sources":["../../../src/layout/balanced.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AAK5C,MAAM,WAAW,eAAe;IAC9B,kBAAkB,CAAC,EAAE,MAAM,CAAA;CAC5B;AAED,wBAAgB,QAAQ,CAAE,OAAO,CAAC,EAAE,eAAe,GAAG,UAAU,CAgB/D"}
@@ -0,0 +1,16 @@
import batch from 'it-batch';
const DEFAULT_MAX_CHILDREN_PER_NODE = 174;
export function balanced(options) {
const maxChildrenPerNode = options?.maxChildrenPerNode ?? DEFAULT_MAX_CHILDREN_PER_NODE;
return async function balancedLayout(source, reduce) {
const roots = [];
for await (const chunked of batch(source, maxChildrenPerNode)) {
roots.push(await reduce(chunked));
}
if (roots.length > 1) {
return balancedLayout(roots, reduce);
}
return roots[0];
};
}
//# sourceMappingURL=balanced.js.map
@@ -0,0 +1 @@
{"version":3,"file":"balanced.js","sourceRoot":"","sources":["../../../src/layout/balanced.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,MAAM,UAAU,CAAA;AAI5B,MAAM,6BAA6B,GAAG,GAAG,CAAA;AAMzC,MAAM,UAAU,QAAQ,CAAE,OAAyB;IACjD,MAAM,kBAAkB,GAAG,OAAO,EAAE,kBAAkB,IAAI,6BAA6B,CAAA;IAEvF,OAAO,KAAK,UAAU,cAAc,CAAE,MAAM,EAAE,MAAM;QAClD,MAAM,KAAK,GAAG,EAAE,CAAA;QAEhB,IAAI,KAAK,EAAE,MAAM,OAAO,IAAI,KAAK,CAAC,MAAM,EAAE,kBAAkB,CAAC,EAAE;YAC7D,KAAK,CAAC,IAAI,CAAC,MAAM,MAAM,CAAC,OAAO,CAAC,CAAC,CAAA;SAClC;QAED,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE;YACpB,OAAO,cAAc,CAAC,KAAK,EAAE,MAAM,CAAC,CAAA;SACrC;QAED,OAAO,KAAK,CAAC,CAAC,CAAC,CAAA;IACjB,CAAC,CAAA;AACH,CAAC"}
@@ -0,0 +1,3 @@
import type { FileLayout } from './index.js';
export declare function flat(): FileLayout;
//# sourceMappingURL=flat.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"flat.d.ts","sourceRoot":"","sources":["../../../src/layout/flat.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAA;AAG5C,wBAAgB,IAAI,IAAK,UAAU,CAIlC"}
@@ -0,0 +1,7 @@
import all from 'it-all';
export function flat() {
return async function flatLayout(source, reduce) {
return reduce(await all(source));
};
}
//# sourceMappingURL=flat.js.map
@@ -0,0 +1 @@
{"version":3,"file":"flat.js","sourceRoot":"","sources":["../../../src/layout/flat.ts"],"names":[],"mappings":"AAAA,OAAO,GAAG,MAAM,QAAQ,CAAA;AAIxB,MAAM,UAAU,IAAI;IAClB,OAAO,KAAK,UAAU,UAAU,CAAE,MAAM,EAAE,MAAM;QAC9C,OAAO,MAAM,CAAC,MAAM,GAAG,CAAC,MAAM,CAAC,CAAC,CAAA;IAClC,CAAC,CAAA;AACH,CAAC"}
@@ -0,0 +1,11 @@
import type { InProgressImportResult } from '../index.js';
export interface Reducer {
(leaves: InProgressImportResult[]): Promise<InProgressImportResult>;
}
export interface FileLayout {
(source: AsyncIterable<InProgressImportResult> | Iterable<InProgressImportResult>, reducer: Reducer): Promise<InProgressImportResult>;
}
export { balanced } from './balanced.js';
export { flat } from './flat.js';
export { trickle } from './trickle.js';
//# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/layout/index.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAA;AAEzD,MAAM,WAAW,OAAO;IAAG,CAAC,MAAM,EAAE,sBAAsB,EAAE,GAAG,OAAO,CAAC,sBAAsB,CAAC,CAAA;CAAE;AAChG,MAAM,WAAW,UAAU;IAAG,CAAC,MAAM,EAAE,aAAa,CAAC,sBAAsB,CAAC,GAAG,QAAQ,CAAC,sBAAsB,CAAC,EAAE,OAAO,EAAE,OAAO,GAAG,OAAO,CAAC,sBAAsB,CAAC,CAAA;CAAE;AAErK,OAAO,EAAE,QAAQ,EAAE,MAAM,eAAe,CAAA;AACxC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAChC,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAA"}
@@ -0,0 +1,4 @@
export { balanced } from './balanced.js';
export { flat } from './flat.js';
export { trickle } from './trickle.js';
//# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/layout/index.ts"],"names":[],"mappings":"AAKA,OAAO,EAAE,QAAQ,EAAE,MAAM,eAAe,CAAA;AACxC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAChC,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAA"}
@@ -0,0 +1,10 @@
import type { FileLayout } from '../layout/index.js';
export interface TrickleOptions {
layerRepeat?: number;
maxChildrenPerNode?: number;
}
/**
* @see https://github.com/ipfs/specs/pull/57#issuecomment-265205384
*/
export declare function trickle(options?: TrickleOptions): FileLayout;
//# sourceMappingURL=trickle.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"trickle.d.ts","sourceRoot":"","sources":["../../../src/layout/trickle.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,UAAU,EAAW,MAAM,oBAAoB,CAAA;AAmB7D,MAAM,WAAW,cAAc;IAC7B,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,kBAAkB,CAAC,EAAE,MAAM,CAAA;CAC5B;AAED;;GAEG;AACH,wBAAgB,OAAO,CAAE,OAAO,CAAC,EAAE,cAAc,GAAG,UAAU,CAkC7D"}
@@ -0,0 +1,128 @@
import batch from 'it-batch';
const DEFAULT_LAYER_REPEAT = 4;
const DEFAULT_MAX_CHILDREN_PER_NODE = 174;
/**
* @see https://github.com/ipfs/specs/pull/57#issuecomment-265205384
*/
export function trickle(options) {
const layerRepeat = options?.layerRepeat ?? DEFAULT_LAYER_REPEAT;
const maxChildrenPerNode = options?.maxChildrenPerNode ?? DEFAULT_MAX_CHILDREN_PER_NODE;
return async function trickleLayout(source, reduce) {
const root = new Root(layerRepeat);
let iteration = 0;
let maxDepth = 1;
let subTree = root;
for await (const layer of batch(source, maxChildrenPerNode)) {
if (subTree.isFull()) {
if (subTree !== root) {
root.addChild(await subTree.reduce(reduce));
}
if (iteration > 0 && iteration % layerRepeat === 0) {
maxDepth++;
}
subTree = new SubTree(maxDepth, layerRepeat, iteration);
iteration++;
}
subTree.append(layer);
}
if (subTree != null && subTree !== root) {
root.addChild(await subTree.reduce(reduce));
}
return root.reduce(reduce);
};
}
class SubTree {
root;
node;
parent;
maxDepth;
layerRepeat;
currentDepth;
iteration;
constructor(maxDepth, layerRepeat, iteration = 0) {
this.maxDepth = maxDepth;
this.layerRepeat = layerRepeat;
this.currentDepth = 1;
this.iteration = iteration;
this.root = this.node = this.parent = {
children: [],
depth: this.currentDepth,
maxDepth,
maxChildren: (this.maxDepth - this.currentDepth) * this.layerRepeat
};
}
isFull() {
if (this.root.data == null) {
return false;
}
if (this.currentDepth < this.maxDepth && this.node.maxChildren > 0) {
// can descend
this._addNextNodeToParent(this.node);
return false;
}
// try to find new node from node.parent
const distantRelative = this._findParent(this.node, this.currentDepth);
if (distantRelative != null) {
this._addNextNodeToParent(distantRelative);
return false;
}
return true;
}
_addNextNodeToParent(parent) {
this.parent = parent;
// find site for new node
const nextNode = {
children: [],
depth: parent.depth + 1,
parent,
maxDepth: this.maxDepth,
maxChildren: Math.floor(parent.children.length / this.layerRepeat) * this.layerRepeat
};
// @ts-expect-error nextNode is different type
parent.children.push(nextNode);
this.currentDepth = nextNode.depth;
this.node = nextNode;
}
append(layer) {
this.node.data = layer;
}
async reduce(reduce) {
return this._reduce(this.root, reduce);
}
async _reduce(node, reduce) {
let children = [];
if (node.children.length > 0) {
children = await Promise.all(node.children
// @ts-expect-error data is not present on type
.filter(child => child.data)
// @ts-expect-error child is wrong type
.map(async (child) => this._reduce(child, reduce)));
}
return reduce((node.data ?? []).concat(children));
}
_findParent(node, depth) {
const parent = node.parent;
if (parent == null || parent.depth === 0) {
return;
}
if (parent.children.length === parent.maxChildren || parent.maxChildren === 0) {
// this layer is full, may be able to traverse to a different branch
return this._findParent(parent, depth);
}
return parent;
}
}
class Root extends SubTree {
constructor(layerRepeat) {
super(0, layerRepeat);
this.root.depth = 0;
this.currentDepth = 1;
}
addChild(child) {
this.root.children.push(child);
}
async reduce(reduce) {
return reduce((this.root.data ?? []).concat(this.root.children));
}
}
//# sourceMappingURL=trickle.js.map
@@ -0,0 +1 @@
{"version":3,"file":"trickle.js","sourceRoot":"","sources":["../../../src/layout/trickle.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,MAAM,UAAU,CAAA;AAM5B,MAAM,oBAAoB,GAAG,CAAC,CAAA;AAC9B,MAAM,6BAA6B,GAAG,GAAG,CAAA;AAmBzC;;GAEG;AACH,MAAM,UAAU,OAAO,CAAE,OAAwB;IAC/C,MAAM,WAAW,GAAG,OAAO,EAAE,WAAW,IAAI,oBAAoB,CAAA;IAChE,MAAM,kBAAkB,GAAG,OAAO,EAAE,kBAAkB,IAAI,6BAA6B,CAAA;IAEvF,OAAO,KAAK,UAAU,aAAa,CAAE,MAAM,EAAE,MAAM;QACjD,MAAM,IAAI,GAAG,IAAI,IAAI,CAAC,WAAW,CAAC,CAAA;QAClC,IAAI,SAAS,GAAG,CAAC,CAAA;QACjB,IAAI,QAAQ,GAAG,CAAC,CAAA;QAChB,IAAI,OAAO,GAAY,IAAI,CAAA;QAE3B,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,KAAK,CAAC,MAAM,EAAE,kBAAkB,CAAC,EAAE;YAC3D,IAAI,OAAO,CAAC,MAAM,EAAE,EAAE;gBACpB,IAAI,OAAO,KAAK,IAAI,EAAE;oBACpB,IAAI,CAAC,QAAQ,CAAC,MAAM,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAA;iBAC5C;gBAED,IAAI,SAAS,GAAG,CAAC,IAAI,SAAS,GAAG,WAAW,KAAK,CAAC,EAAE;oBAClD,QAAQ,EAAE,CAAA;iBACX;gBAED,OAAO,GAAG,IAAI,OAAO,CAAC,QAAQ,EAAE,WAAW,EAAE,SAAS,CAAC,CAAA;gBAEvD,SAAS,EAAE,CAAA;aACZ;YAED,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,CAAA;SACtB;QAED,IAAI,OAAO,IAAI,IAAI,IAAI,OAAO,KAAK,IAAI,EAAE;YACvC,IAAI,CAAC,QAAQ,CAAC,MAAM,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAA;SAC5C;QAED,OAAO,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAA;IAC5B,CAAC,CAAA;AACH,CAAC;AAED,MAAM,OAAO;IACJ,IAAI,CAAgB;IACpB,IAAI,CAAgB;IACpB,MAAM,CAAgB;IACtB,QAAQ,CAAQ;IAChB,WAAW,CAAQ;IACnB,YAAY,CAAQ;IACpB,SAAS,CAAQ;IAExB,YAAa,QAAgB,EAAE,WAAmB,EAAE,YAAoB,CAAC;QACvE,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAA;QACxB,IAAI,CAAC,WAAW,GAAG,WAAW,CAAA;QAC9B,IAAI,CAAC,YAAY,GAAG,CAAC,CAAA;QACrB,IAAI,CAAC,SAAS,GAAG,SAAS,CAAA;QAE1B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,GAAG;YACpC,QAAQ,EAAE,EAAE;YACZ,KAAK,EAAE,IAAI,CAAC,YAAY;YACxB,QAAQ;YACR,WAAW,EAAE,CAAC,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,IAAI,CAAC,WAAW;SACpE,CAAA;IACH,CAAC;IAED,MAAM;QACJ,IAAI,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,EAAE;YAC1B,OAAO,KAAK,CAAA;SACb;QAED,IAAI,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,IAAI,CAAC,WAAW,GAAG,CAAC,EAAE;YAClE,cAAc;YACd,IAAI,CAAC,oBAAoB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;YAEpC,OAAO,KAAK,CAAA;SACb;QAED,wCAAwC;QACxC,MAAM,eAAe,GAAG,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,YAAY,CAAC,CAAA;QAEtE,IAAI,eAAe,IAAI,IAAI,EAAE;YAC3B,IAAI,CAAC,oBAAoB,CAAC,eAAe,CAAC,CAAA;YAE1C,OAAO,KAAK,CAAA;SACb;QAED,OAAO,IAAI,CAAA;IACb,CAAC;IAED,oBAAoB,CAAE,MAAsB;QAC1C,IAAI,CAAC,MAAM,GAAG,MAAM,CAAA;QAEpB,yBAAyB;QACzB,MAAM,QAAQ,GAAG;YACf,QAAQ,EAAE,EAAE;YACZ,KAAK,EAAE,MAAM,CAAC,KAAK,GAAG,CAAC;YACvB,MAAM;YACN,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,WAAW,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,IAAI,CAAC,WAAW;SACtF,CAAA;QAED,8CAA8C;QAC9C,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAA;QAE9B,IAAI,CAAC,YAAY,GAAG,QAAQ,CAAC,KAAK,CAAA;QAClC,IAAI,CAAC,IAAI,GAAG,QAAQ,CAAA;IACtB,CAAC;IAED,MAAM,CAAE,KAA+B;QACrC,IAAI,CAAC,IAAI,CAAC,IAAI,GAAG,KAAK,CAAA;IACxB,CAAC;IAED,KAAK,CAAC,MAAM,CAAE,MAAe;QAC3B,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,EAAE,MAAM,CAAC,CAAA;IACxC,CAAC;IAED,KAAK,CAAC,OAAO,CAAE,IAAoB,EAAE,MAAe;QAClD,IAAI,QAAQ,GAA6B,EAAE,CAAA;QAE3C,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,EAAE;YAC5B,QAAQ,GAAG,MAAM,OAAO,CAAC,GAAG,CAC1B,IAAI,CAAC,QAAQ;gBACX,+CAA+C;iBAC9C,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC;gBAC5B,uCAAuC;iBACtC,GAAG,CAAC,KAAK,EAAC,KAAK,EAAC,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,CACnD,CAAA;SACF;QAED,OAAO,MAAM,CAAC,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAA;IACnD,CAAC;IAED,WAAW,CAAE,IAAoB,EAAE,KAAa;QAC9C,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,CAAA;QAE1B,IAAI,MAAM,IAAI,IAAI,IAAI,MAAM,CAAC,KAAK,KAAK,CAAC,EAAE;YACxC,OAAM;SACP;QAED,IAAI,MAAM,CAAC,QAAQ,CAAC,MAAM,KAAK,MAAM,CAAC,WAAW,IAAI,MAAM,CAAC,WAAW,KAAK,CAAC,EAAE;YAC7E,oEAAoE;YACpE,OAAO,IAAI,CAAC,WAAW,CAAC,MAAM,EAAE,KAAK,CAAC,CAAA;SACvC;QAED,OAAO,MAAM,CAAA;IACf,CAAC;CACF;AAED,MAAM,IAAK,SAAQ,OAAO;IACxB,YAAa,WAAmB;QAC9B,KAAK,CAAC,CAAC,EAAE,WAAW,CAAC,CAAA;QAErB,IAAI,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,CAAA;QACnB,IAAI,CAAC,YAAY,GAAG,CAAC,CAAA;IACvB,CAAC;IAED,QAAQ,CAAE,KAA6B;QACrC,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;IAChC,CAAC;IAED,KAAK,CAAC,MAAM,CAAE,MAAe;QAC3B,OAAO,MAAM,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAA;IAClE,CAAC;CACF"}
@@ -0,0 +1,10 @@
import type { TreeBuilder } from './index.js';
import type { PersistOptions } from './utils/persist.js';
export interface AddToTreeOptions extends PersistOptions {
shardSplitThresholdBytes: number;
}
export interface TreeBuilderOptions extends AddToTreeOptions {
wrapWithDirectory: boolean;
}
export declare function defaultTreeBuilder(options: TreeBuilderOptions): TreeBuilder;
//# sourceMappingURL=tree-builder.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"tree-builder.d.ts","sourceRoot":"","sources":["../../src/tree-builder.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAwC,WAAW,EAAmB,MAAM,YAAY,CAAA;AACpG,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAA;AAExD,MAAM,WAAW,gBAAiB,SAAQ,cAAc;IACtD,wBAAwB,EAAE,MAAM,CAAA;CACjC;AA2DD,MAAM,WAAW,kBAAmB,SAAQ,gBAAgB;IAC1D,iBAAiB,EAAE,OAAO,CAAA;CAC3B;AAED,wBAAgB,kBAAkB,CAAE,OAAO,EAAE,kBAAkB,GAAG,WAAW,CAkD5E"}
@@ -0,0 +1,96 @@
import { DirFlat } from './dir-flat.js';
import { Dir } from './dir.js';
import { flatToShard } from './flat-to-shard.js';
import { toPathComponents } from './utils/to-path-components.js';
async function addToTree(elem, tree, options) {
const pathElems = toPathComponents(elem.path ?? '');
const lastIndex = pathElems.length - 1;
let parent = tree;
let currentPath = '';
for (let i = 0; i < pathElems.length; i++) {
const pathElem = pathElems[i];
currentPath += `${currentPath !== '' ? '/' : ''}${pathElem}`;
const last = (i === lastIndex);
parent.dirty = true;
parent.cid = undefined;
parent.size = undefined;
if (last) {
await parent.put(pathElem, elem);
tree = await flatToShard(null, parent, options.shardSplitThresholdBytes, options);
}
else {
let dir = await parent.get(pathElem);
if ((dir == null) || !(dir instanceof Dir)) {
dir = new DirFlat({
root: false,
dir: true,
parent,
parentKey: pathElem,
path: currentPath,
dirty: true,
flat: true,
mtime: dir?.unixfs?.mtime,
mode: dir?.unixfs?.mode
}, options);
}
await parent.put(pathElem, dir);
parent = dir;
}
}
return tree;
}
async function* flushAndYield(tree, blockstore) {
if (!(tree instanceof Dir)) {
if (tree.unixfs?.isDirectory() === true) {
yield tree;
}
return;
}
yield* tree.flush(blockstore);
}
export function defaultTreeBuilder(options) {
return async function* treeBuilder(source, block) {
let tree = new DirFlat({
root: true,
dir: true,
path: '',
dirty: true,
flat: true
}, options);
let rootDir;
let singleRoot = false;
for await (const entry of source) {
if (entry == null) {
continue;
}
// if all paths are from the same root directory, we should
// wrap them all in that root directory
const dir = `${entry.originalPath ?? ''}`.split('/')[0];
if (dir != null && dir !== '') {
if (rootDir == null) {
rootDir = dir;
singleRoot = true;
}
else if (rootDir !== dir) {
singleRoot = false;
}
}
tree = await addToTree(entry, tree, options);
if (entry.unixfs == null || !entry.unixfs.isDirectory()) {
yield entry;
}
}
if (options.wrapWithDirectory || (singleRoot && tree.childCount() > 1)) {
yield* flushAndYield(tree, block);
}
else {
for await (const unwrapped of tree.eachChildSeries()) {
if (unwrapped == null) {
continue;
}
yield* flushAndYield(unwrapped.child, block);
}
}
};
}
//# sourceMappingURL=tree-builder.js.map
@@ -0,0 +1 @@
{"version":3,"file":"tree-builder.js","sourceRoot":"","sources":["../../src/tree-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,eAAe,CAAA;AACvC,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAA;AAC9B,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAA;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,+BAA+B,CAAA;AAQhE,KAAK,UAAU,SAAS,CAAE,IAA4B,EAAE,IAAS,EAAE,OAAyB;IAC1F,MAAM,SAAS,GAAG,gBAAgB,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC,CAAA;IACnD,MAAM,SAAS,GAAG,SAAS,CAAC,MAAM,GAAG,CAAC,CAAA;IACtC,IAAI,MAAM,GAAG,IAAI,CAAA;IACjB,IAAI,WAAW,GAAG,EAAE,CAAA;IAEpB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE;QACzC,MAAM,QAAQ,GAAG,SAAS,CAAC,CAAC,CAAC,CAAA;QAE7B,WAAW,IAAI,GAAG,WAAW,KAAK,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,GAAG,QAAQ,EAAE,CAAA;QAE5D,MAAM,IAAI,GAAG,CAAC,CAAC,KAAK,SAAS,CAAC,CAAA;QAC9B,MAAM,CAAC,KAAK,GAAG,IAAI,CAAA;QACnB,MAAM,CAAC,GAAG,GAAG,SAAS,CAAA;QACtB,MAAM,CAAC,IAAI,GAAG,SAAS,CAAA;QAEvB,IAAI,IAAI,EAAE;YACR,MAAM,MAAM,CAAC,GAAG,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAA;YAChC,IAAI,GAAG,MAAM,WAAW,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,CAAC,wBAAwB,EAAE,OAAO,CAAC,CAAA;SAClF;aAAM;YACL,IAAI,GAAG,GAAG,MAAM,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAA;YAEpC,IAAI,CAAC,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,YAAY,GAAG,CAAC,EAAE;gBAC1C,GAAG,GAAG,IAAI,OAAO,CAAC;oBAChB,IAAI,EAAE,KAAK;oBACX,GAAG,EAAE,IAAI;oBACT,MAAM;oBACN,SAAS,EAAE,QAAQ;oBACnB,IAAI,EAAE,WAAW;oBACjB,KAAK,EAAE,IAAI;oBACX,IAAI,EAAE,IAAI;oBACV,KAAK,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK;oBACzB,IAAI,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI;iBACxB,EAAE,OAAO,CAAC,CAAA;aACZ;YAED,MAAM,MAAM,CAAC,GAAG,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAA;YAE/B,MAAM,GAAG,GAAG,CAAA;SACb;KACF;IAED,OAAO,IAAI,CAAA;AACb,CAAC;AAED,KAAK,SAAU,CAAC,CAAC,aAAa,CAAE,IAAkC,EAAE,UAA2B;IAC7F,IAAI,CAAC,CAAC,IAAI,YAAY,GAAG,CAAC,EAAE;QAC1B,IAAI,IAAI,CAAC,MAAM,EAAE,WAAW,EAAE,KAAK,IAAI,EAAE;YACvC,MAAM,IAAI,CAAA;SACX;QAED,OAAM;KACP;IAED,KAAM,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,UAAU,CAAC,CAAA;AAChC,CAAC;AAMD,MAAM,UAAU,kBAAkB,CAAE,OAA2B;IAC7D,OAAO,KAAK,SAAU,CAAC,CAAC,WAAW,CAAE,MAAM,EAAE,KAAK;QAChD,IAAI,IAAI,GAAQ,IAAI,OAAO,CAAC;YAC1B,IAAI,EAAE,IAAI;YACV,GAAG,EAAE,IAAI;YACT,IAAI,EAAE,EAAE;YACR,KAAK,EAAE,IAAI;YACX,IAAI,EAAE,IAAI;SACX,EAAE,OAAO,CAAC,CAAA;QAEX,IAAI,OAA2B,CAAA;QAC/B,IAAI,UAAU,GAAG,KAAK,CAAA;QAEtB,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE;YAChC,IAAI,KAAK,IAAI,IAAI,EAAE;gBACjB,SAAQ;aACT;YAED,2DAA2D;YAC3D,uCAAuC;YACvC,MAAM,GAAG,GAAG,GAAG,KAAK,CAAC,YAAY,IAAI,EAAE,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAA;YAEvD,IAAI,GAAG,IAAI,IAAI,IAAI,GAAG,KAAK,EAAE,EAAE;gBAC7B,IAAI,OAAO,IAAI,IAAI,EAAE;oBACnB,OAAO,GAAG,GAAG,CAAA;oBACb,UAAU,GAAG,IAAI,CAAA;iBAClB;qBAAM,IAAI,OAAO,KAAK,GAAG,EAAE;oBAC1B,UAAU,GAAG,KAAK,CAAA;iBACnB;aACF;YAED,IAAI,GAAG,MAAM,SAAS,CAAC,KAAK,EAAE,IAAI,EAAE,OAAO,CAAC,CAAA;YAE5C,IAAI,KAAK,CAAC,MAAM,IAAI,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,WAAW,EAAE,EAAE;gBACvD,MAAM,KAAK,CAAA;aACZ;SACF;QAED,IAAI,OAAO,CAAC,iBAAiB,IAAI,CAAC,UAAU,IAAI,IAAI,CAAC,UAAU,EAAE,GAAG,CAAC,CAAC,EAAE;YACtE,KAAM,CAAC,CAAC,aAAa,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;SACnC;aAAM;YACL,IAAI,KAAK,EAAE,MAAM,SAAS,IAAI,IAAI,CAAC,eAAe,EAAE,EAAE;gBACpD,IAAI,SAAS,IAAI,IAAI,EAAE;oBACrB,SAAQ;iBACT;gBAED,KAAM,CAAC,CAAC,aAAa,CAAC,SAAS,CAAC,KAAK,EAAE,KAAK,CAAC,CAAA;aAC9C;SACF;IACH,CAAC,CAAA;AACH,CAAC"}
@@ -0,0 +1,12 @@
import { CID } from 'multiformats/cid';
import type { WritableStorage } from '../index.js';
import type { Version as CIDVersion } from 'multiformats/cid';
import type { BlockCodec } from 'multiformats/codecs/interface';
import type { ProgressOptions } from 'progress-events';
export interface PersistOptions extends ProgressOptions {
codec?: BlockCodec<any, any>;
cidVersion: CIDVersion;
signal?: AbortSignal;
}
export declare const persist: (buffer: Uint8Array, blockstore: WritableStorage, options: PersistOptions) => Promise<CID>;
//# sourceMappingURL=persist.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"persist.d.ts","sourceRoot":"","sources":["../../../src/utils/persist.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAA;AAEtC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AAClD,OAAO,KAAK,EAAE,OAAO,IAAI,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAC7D,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,+BAA+B,CAAA;AAC/D,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAA;AAEtD,MAAM,WAAW,cAAe,SAAQ,eAAe;IACrD,KAAK,CAAC,EAAE,UAAU,CAAC,GAAG,EAAE,GAAG,CAAC,CAAA;IAC5B,UAAU,EAAE,UAAU,CAAA;IACtB,MAAM,CAAC,EAAE,WAAW,CAAA;CACrB;AAED,eAAO,MAAM,OAAO,WAAkB,UAAU,cAAc,eAAe,WAAW,cAAc,KAAG,QAAQ,GAAG,CAWnH,CAAA"}
@@ -0,0 +1,13 @@
import * as dagPb from '@ipld/dag-pb';
import { CID } from 'multiformats/cid';
import { sha256 } from 'multiformats/hashes/sha2';
export const persist = async (buffer, blockstore, options) => {
if (options.codec == null) {
options.codec = dagPb;
}
const multihash = await sha256.digest(buffer);
const cid = CID.create(options.cidVersion, options.codec.code, multihash);
await blockstore.put(cid, buffer, options);
return cid;
};
//# sourceMappingURL=persist.js.map
@@ -0,0 +1 @@
{"version":3,"file":"persist.js","sourceRoot":"","sources":["../../../src/utils/persist.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,KAAK,MAAM,cAAc,CAAA;AACrC,OAAO,EAAE,GAAG,EAAE,MAAM,kBAAkB,CAAA;AACtC,OAAO,EAAE,MAAM,EAAE,MAAM,0BAA0B,CAAA;AAYjD,MAAM,CAAC,MAAM,OAAO,GAAG,KAAK,EAAE,MAAkB,EAAE,UAA2B,EAAE,OAAuB,EAAgB,EAAE;IACtH,IAAI,OAAO,CAAC,KAAK,IAAI,IAAI,EAAE;QACzB,OAAO,CAAC,KAAK,GAAG,KAAK,CAAA;KACtB;IAED,MAAM,SAAS,GAAG,MAAM,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAA;IAC7C,MAAM,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,UAAU,EAAE,OAAO,CAAC,KAAK,CAAC,IAAI,EAAE,SAAS,CAAC,CAAA;IAEzE,MAAM,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,CAAC,CAAA;IAE1C,OAAO,GAAG,CAAA;AACZ,CAAC,CAAA"}
@@ -0,0 +1,2 @@
export declare const toPathComponents: (path?: string) => string[];
//# sourceMappingURL=to-path-components.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"to-path-components.d.ts","sourceRoot":"","sources":["../../../src/utils/to-path-components.ts"],"names":[],"mappings":"AAAA,eAAO,MAAM,gBAAgB,UAAU,MAAM,KAAQ,MAAM,EAM1D,CAAA"}
@@ -0,0 +1,8 @@
export const toPathComponents = (path = '') => {
// split on / unless escaped with \
return (path
.trim()
.match(/([^\\/]|\\\/)+/g) ?? [])
.filter(Boolean);
};
//# sourceMappingURL=to-path-components.js.map
@@ -0,0 +1 @@
{"version":3,"file":"to-path-components.js","sourceRoot":"","sources":["../../../src/utils/to-path-components.ts"],"names":[],"mappings":"AAAA,MAAM,CAAC,MAAM,gBAAgB,GAAG,CAAC,OAAe,EAAE,EAAY,EAAE;IAC9D,mCAAmC;IACnC,OAAO,CAAC,IAAI;SACT,IAAI,EAAE;SACN,KAAK,CAAC,iBAAiB,CAAC,IAAI,EAAE,CAAC;SAC/B,MAAM,CAAC,OAAO,CAAC,CAAA;AACpB,CAAC,CAAA"}
+38
View File
@@ -0,0 +1,38 @@
{
"ChunkValidator": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.ChunkValidator.html",
"Chunker": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.Chunker.html",
"DAGBuilder": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.DAGBuilder.html",
"FileLayout": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.FileLayout.html",
"ImportReadProgress": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.ImportReadProgress.html",
"ImportWriteProgress": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.ImportWriteProgress.html",
"LayoutLeafProgress": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.LayoutLeafProgress.html",
"Reducer": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer._internal_.Reducer.html",
"BufferImportProgressEvents": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer._internal_.BufferImportProgressEvents.html",
"DagBuilderProgressEvents": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer._internal_.DagBuilderProgressEvents.html",
"ImporterSourceStream": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer._internal_.ImporterSourceStream.html",
"ReducerProgressEvents": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer._internal_.ReducerProgressEvents.html",
"BufferImporter": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.BufferImporter.html",
"BufferImporterResult": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.BufferImporterResult.html",
"Directory": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.Directory.html",
"DirectoryCandidate": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.DirectoryCandidate.html",
"File": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.File.html",
"FileCandidate": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.FileCandidate.html",
"HamtHashFn": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.HamtHashFn.html",
"ImportResult": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.ImportResult.html",
"ImporterOptions": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.ImporterOptions.html",
"MultipleBlockImportResult": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.MultipleBlockImportResult.html",
"SingleBlockImportResult": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.SingleBlockImportResult.html",
"TreeBuilder": "https://ipfs.github.io/js-ipfs-unixfs/interfaces/ipfs_unixfs_importer.TreeBuilder.html",
"ByteStream": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.ByteStream.html",
"ImportCandidate": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.ImportCandidate.html",
"ImportCandidateStream": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.ImportCandidateStream.html",
"ImportContent": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.ImportContent.html",
"ImporterProgressEvents": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.ImporterProgressEvents.html",
"InProgressImportResult": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.InProgressImportResult.html",
"WritableStorage": "https://ipfs.github.io/js-ipfs-unixfs/types/ipfs_unixfs_importer.WritableStorage.html",
"importByteStream": "https://ipfs.github.io/js-ipfs-unixfs/functions/ipfs_unixfs_importer.importByteStream.html",
"importBytes": "https://ipfs.github.io/js-ipfs-unixfs/functions/ipfs_unixfs_importer.importBytes.html",
"importDirectory": "https://ipfs.github.io/js-ipfs-unixfs/functions/ipfs_unixfs_importer.importDirectory.html",
"importFile": "https://ipfs.github.io/js-ipfs-unixfs/functions/ipfs_unixfs_importer.importFile.html",
"importer": "https://ipfs.github.io/js-ipfs-unixfs/functions/ipfs_unixfs_importer.importer.html"
}
+191
View File
@@ -0,0 +1,191 @@
{
"name": "ipfs-unixfs-importer",
"version": "15.1.5",
"description": "JavaScript implementation of the UnixFs importer used by IPFS",
"license": "Apache-2.0 OR MIT",
"homepage": "https://github.com/ipfs/js-ipfs-unixfs/tree/master/packages/ipfs-unixfs-importer#readme",
"repository": {
"type": "git",
"url": "git+https://github.com/ipfs/js-ipfs-unixfs.git"
},
"bugs": {
"url": "https://github.com/ipfs/js-ipfs-unixfs/issues"
},
"keywords": [
"IPFS"
],
"engines": {
"node": ">=16.0.0",
"npm": ">=7.0.0"
},
"type": "module",
"types": "./dist/src/index.d.ts",
"typesVersions": {
"*": {
"*": [
"*",
"dist/*",
"dist/src/*",
"dist/src/*/index"
],
"src/*": [
"*",
"dist/*",
"dist/src/*",
"dist/src/*/index"
]
}
},
"files": [
"src",
"dist",
"!dist/test",
"!**/*.tsbuildinfo"
],
"exports": {
".": {
"types": "./dist/src/index.d.ts",
"import": "./dist/src/index.js"
},
"./chunker": {
"types": "./dist/src/chunker/index.d.ts",
"import": "./dist/src/chunker/index.js"
},
"./layout": {
"types": "./dist/src/layout/index.d.ts",
"import": "./dist/src/layout/index.js"
}
},
"eslintConfig": {
"extends": "ipfs",
"parserOptions": {
"sourceType": "module"
}
},
"release": {
"branches": [
"master"
],
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits",
"releaseRules": [
{
"breaking": true,
"release": "major"
},
{
"revert": true,
"release": "patch"
},
{
"type": "feat",
"release": "minor"
},
{
"type": "fix",
"release": "patch"
},
{
"type": "docs",
"release": "patch"
},
{
"type": "test",
"release": "patch"
},
{
"type": "deps",
"release": "patch"
},
{
"scope": "no-release",
"release": false
}
]
}
],
[
"@semantic-release/release-notes-generator",
{
"preset": "conventionalcommits",
"presetConfig": {
"types": [
{
"type": "feat",
"section": "Features"
},
{
"type": "fix",
"section": "Bug Fixes"
},
{
"type": "chore",
"section": "Trivial Changes"
},
{
"type": "docs",
"section": "Documentation"
},
{
"type": "deps",
"section": "Dependencies"
},
{
"type": "test",
"section": "Tests"
}
]
}
}
],
"@semantic-release/changelog",
"@semantic-release/npm",
"@semantic-release/github",
"@semantic-release/git"
]
},
"scripts": {
"test": "aegir test",
"test:node": "aegir test -t node --cov",
"test:chrome": "aegir test -t browser --cov",
"test:firefox": "aegir test -t browser -- --browser firefox",
"build": "aegir build",
"clean": "aegir clean",
"lint": "aegir lint",
"dep-check": "aegir dep-check",
"release": "aegir release"
},
"dependencies": {
"@ipld/dag-pb": "^4.0.0",
"@multiformats/murmur3": "^2.0.0",
"err-code": "^3.0.1",
"hamt-sharding": "^3.0.0",
"interface-blockstore": "^5.0.0",
"interface-store": "^5.0.1",
"ipfs-unixfs": "^11.0.0",
"it-all": "^3.0.2",
"it-batch": "^3.0.2",
"it-first": "^3.0.2",
"it-parallel-batch": "^3.0.1",
"multiformats": "^11.0.0",
"progress-events": "^1.0.0",
"rabin-wasm": "^0.1.4",
"uint8arraylist": "^2.4.3",
"uint8arrays": "^4.0.2"
},
"devDependencies": {
"aegir": "^39.0.6",
"blockstore-core": "^4.0.1",
"it-last": "^3.0.2",
"wherearewe": "^2.0.1"
},
"browser": {
"fs": false
},
"typedoc": {
"entryPoint": "./src/index.ts"
}
}
@@ -0,0 +1,47 @@
import { Uint8ArrayList } from 'uint8arraylist'
import type { Chunker } from './index.js'
export interface FixedSizeOptions {
chunkSize?: number
}
const DEFAULT_CHUNK_SIZE = 262144
export const fixedSize = (options: FixedSizeOptions = {}): Chunker => {
const chunkSize = options.chunkSize ?? DEFAULT_CHUNK_SIZE
return async function * fixedSizeChunker (source) {
let list = new Uint8ArrayList()
let currentLength = 0
let emitted = false
for await (const buffer of source) {
list.append(buffer)
currentLength += buffer.length
while (currentLength >= chunkSize) {
yield list.slice(0, chunkSize)
emitted = true
// throw away consumed bytes
if (chunkSize === list.length) {
list = new Uint8ArrayList()
currentLength = 0
} else {
const newBl = new Uint8ArrayList()
newBl.append(list.sublist(chunkSize))
list = newBl
// update our offset
currentLength -= chunkSize
}
}
}
if (!emitted || currentLength > 0) {
// return any remaining bytes
yield list.subarray(0, currentLength)
}
}
}
+5
View File
@@ -0,0 +1,5 @@
export interface Chunker { (source: AsyncIterable<Uint8Array>): AsyncIterable<Uint8Array> }
export { rabin } from './rabin.js'
export { fixedSize } from './fixed-size.js'
+79
View File
@@ -0,0 +1,79 @@
import errcode from 'err-code'
// @ts-expect-error no types
import { create } from 'rabin-wasm'
import { Uint8ArrayList } from 'uint8arraylist'
import type { Chunker } from './index.js'
const DEFAULT_MIN_CHUNK_SIZE = 262144
const DEFAULT_MAX_CHUNK_SIZE = 262144
const DEFAULT_AVG_CHUNK_SIZE = 262144
const DEFAULT_WINDOW = 16
async function * chunker (source: AsyncIterable<Uint8Array>, r: any): AsyncGenerator<Uint8Array> {
const buffers = new Uint8ArrayList()
for await (const chunk of source) {
buffers.append(chunk)
const sizes = r.fingerprint(chunk)
for (let i = 0; i < sizes.length; i++) {
const size = sizes[i]
const buf = buffers.slice(0, size)
buffers.consume(size)
yield buf
}
}
if (buffers.length > 0) {
yield buffers.subarray(0)
}
}
export interface RabinOptions {
minChunkSize?: number
maxChunkSize?: number
avgChunkSize?: number
window?: number
}
export const rabin = (options: RabinOptions = {}): Chunker => {
let min = options.minChunkSize ?? DEFAULT_MIN_CHUNK_SIZE
let max = options.maxChunkSize ?? DEFAULT_MAX_CHUNK_SIZE
let avg = options.avgChunkSize ?? DEFAULT_AVG_CHUNK_SIZE
const window = options.window ?? DEFAULT_WINDOW
// if only avg was passed, calculate min/max from that
if (options.avgChunkSize != null && options.minChunkSize == null && options.maxChunkSize == null) {
min = avg / 3
max = avg + (avg / 2)
}
if (options.avgChunkSize == null && options.minChunkSize == null && options.maxChunkSize == null) {
throw errcode(new Error('please specify an average chunk size'), 'ERR_INVALID_AVG_CHUNK_SIZE')
}
// validate min/max/avg in the same way as go
if (min < 16) {
throw errcode(new Error('rabin min must be greater than 16'), 'ERR_INVALID_MIN_CHUNK_SIZE')
}
if (max < min) {
max = min
}
if (avg < min) {
avg = min
}
const sizepow = Math.floor(Math.log2(avg))
return async function * rabinChunker (source) {
const r = await create(sizepow, min, max, window)
for await (const chunk of chunker(source, r)) {
yield chunk
}
}
}
@@ -0,0 +1,88 @@
import * as dagPb from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'
import * as raw from 'multiformats/codecs/raw'
import { CustomProgressEvent } from 'progress-events'
import { persist, type PersistOptions } from '../utils/persist.js'
import type { BufferImporter } from '../index.js'
import type { CID, Version } from 'multiformats/cid'
import type { ProgressOptions, ProgressEvent } from 'progress-events'
/**
* Passed to the onProgress callback while importing files
*/
export interface ImportWriteProgress {
/**
* How many bytes we have written for this source so far - this may be
* bigger than the file size due to the DAG-PB wrappers of each block
*/
bytesWritten: bigint
/**
* The CID of the block that has been written
*/
cid: CID
/**
* The path of the file being imported, if one was specified
*/
path?: string
}
export type BufferImportProgressEvents =
ProgressEvent<'unixfs:importer:progress:file:write', ImportWriteProgress>
export interface BufferImporterOptions extends ProgressOptions<BufferImportProgressEvents> {
cidVersion: Version
rawLeaves: boolean
leafType: 'file' | 'raw'
}
export function defaultBufferImporter (options: BufferImporterOptions): BufferImporter {
return async function * bufferImporter (file, blockstore) {
let bytesWritten = 0n
for await (let block of file.content) {
yield async () => { // eslint-disable-line no-loop-func
let unixfs
const opts: PersistOptions = {
codec: dagPb,
cidVersion: options.cidVersion,
onProgress: options.onProgress
}
if (options.rawLeaves) {
opts.codec = raw
opts.cidVersion = 1
} else {
unixfs = new UnixFS({
type: options.leafType,
data: block
})
block = dagPb.encode({
Data: unixfs.marshal(),
Links: []
})
}
const cid = await persist(block, blockstore, opts)
bytesWritten += BigInt(block.byteLength)
options.onProgress?.(new CustomProgressEvent<ImportWriteProgress>('unixfs:importer:progress:file:write', {
bytesWritten,
cid,
path: file.path
}))
return {
cid,
unixfs,
size: BigInt(block.length),
block
}
}
}
}
}
+31
View File
@@ -0,0 +1,31 @@
import { encode, prepare } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'
import { persist } from '../utils/persist.js'
import type { Directory, InProgressImportResult, WritableStorage } from '../index.js'
import type { Version } from 'multiformats/cid'
export interface DirBuilderOptions {
cidVersion: Version
signal?: AbortSignal
}
export const dirBuilder = async (dir: Directory, blockstore: WritableStorage, options: DirBuilderOptions): Promise<InProgressImportResult> => {
const unixfs = new UnixFS({
type: 'directory',
mtime: dir.mtime,
mode: dir.mode
})
const block = encode(prepare({ Data: unixfs.marshal() }))
const cid = await persist(block, blockstore, options)
const path = dir.path
return {
cid,
path,
unixfs,
size: BigInt(block.length),
originalPath: dir.originalPath,
block
}
}
+197
View File
@@ -0,0 +1,197 @@
import { encode, type PBLink, type PBNode, prepare } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'
import parallelBatch from 'it-parallel-batch'
import * as rawCodec from 'multiformats/codecs/raw'
import { CustomProgressEvent } from 'progress-events'
import { persist } from '../utils/persist.js'
import type { BufferImporter, File, InProgressImportResult, WritableStorage, SingleBlockImportResult, ImporterProgressEvents } from '../index.js'
import type { FileLayout, Reducer } from '../layout/index.js'
import type { CID, Version } from 'multiformats/cid'
import type { ProgressOptions, ProgressEvent } from 'progress-events'
interface BuildFileBatchOptions {
bufferImporter: BufferImporter
blockWriteConcurrency: number
}
async function * buildFileBatch (file: File, blockstore: WritableStorage, options: BuildFileBatchOptions): AsyncGenerator<InProgressImportResult> {
let count = -1
let previous: SingleBlockImportResult | undefined
for await (const entry of parallelBatch(options.bufferImporter(file, blockstore), options.blockWriteConcurrency)) {
count++
if (count === 0) {
// cache the first entry if case there aren't any more
previous = {
...entry,
single: true
}
continue
} else if (count === 1 && (previous != null)) {
// we have the second block of a multiple block import so yield the first
yield {
...previous,
block: undefined,
single: undefined
}
previous = undefined
}
// yield the second or later block of a multiple block import
yield {
...entry,
block: undefined
}
}
if (previous != null) {
yield previous
}
}
export interface LayoutLeafProgress {
/**
* The CID of the leaf being written
*/
cid: CID
/**
* The path of the file being imported, if one was specified
*/
path?: string
}
export type ReducerProgressEvents =
ProgressEvent<'unixfs:importer:progress:file:layout', LayoutLeafProgress>
interface ReduceOptions extends ProgressOptions<ImporterProgressEvents> {
reduceSingleLeafToSelf: boolean
cidVersion: Version
signal?: AbortSignal
}
function isSingleBlockImport (result: any): result is SingleBlockImportResult {
return result.single === true
}
const reduce = (file: File, blockstore: WritableStorage, options: ReduceOptions): Reducer => {
const reducer: Reducer = async function (leaves) {
if (leaves.length === 1 && isSingleBlockImport(leaves[0]) && options.reduceSingleLeafToSelf) {
const leaf = leaves[0]
let node: Uint8Array | PBNode = leaf.block
if (isSingleBlockImport(leaf) && (file.mtime !== undefined || file.mode !== undefined)) {
// only one leaf node which is a raw leaf - we have metadata so convert it into a
// UnixFS entry otherwise we'll have nowhere to store the metadata
leaf.unixfs = new UnixFS({
type: 'file',
mtime: file.mtime,
mode: file.mode,
data: leaf.block
})
node = { Data: leaf.unixfs.marshal(), Links: [] }
leaf.block = encode(prepare(node))
leaf.cid = await persist(leaf.block, blockstore, {
...options,
cidVersion: options.cidVersion
})
leaf.size = BigInt(leaf.block.length)
}
options.onProgress?.(new CustomProgressEvent<LayoutLeafProgress>('unixfs:importer:progress:file:layout', {
cid: leaf.cid,
path: leaf.originalPath
}))
return {
cid: leaf.cid,
path: file.path,
unixfs: leaf.unixfs,
size: leaf.size,
originalPath: leaf.originalPath
}
}
// create a parent node and add all the leaves
const f = new UnixFS({
type: 'file',
mtime: file.mtime,
mode: file.mode
})
const links: PBLink[] = leaves
.filter(leaf => {
if (leaf.cid.code === rawCodec.code && leaf.size > 0) {
return true
}
if ((leaf.unixfs != null) && (leaf.unixfs.data == null) && leaf.unixfs.fileSize() > 0n) {
return true
}
return Boolean(leaf.unixfs?.data?.length)
})
.map((leaf) => {
if (leaf.cid.code === rawCodec.code) {
// node is a leaf buffer
f.addBlockSize(leaf.size)
return {
Name: '',
Tsize: Number(leaf.size),
Hash: leaf.cid
}
}
if ((leaf.unixfs == null) || (leaf.unixfs.data == null)) {
// node is an intermediate node
f.addBlockSize(leaf.unixfs?.fileSize() ?? 0n)
} else {
// node is a unixfs 'file' leaf node
f.addBlockSize(BigInt(leaf.unixfs.data.length))
}
return {
Name: '',
Tsize: Number(leaf.size),
Hash: leaf.cid
}
})
const node = {
Data: f.marshal(),
Links: links
}
const block = encode(prepare(node))
const cid = await persist(block, blockstore, options)
options.onProgress?.(new CustomProgressEvent<LayoutLeafProgress>('unixfs:importer:progress:file:layout', {
cid,
path: file.originalPath
}))
return {
cid,
path: file.path,
unixfs: f,
size: BigInt(block.length + node.Links.reduce((acc, curr) => acc + (curr.Tsize ?? 0), 0)),
originalPath: file.originalPath,
block
}
}
return reducer
}
export interface FileBuilderOptions extends BuildFileBatchOptions, ReduceOptions {
layout: FileLayout
}
export const fileBuilder = async (file: File, block: WritableStorage, options: FileBuilderOptions): Promise<InProgressImportResult> => {
return options.layout(buildFileBatch(file, block, options), reduce(file, block, options))
}
@@ -0,0 +1,129 @@
import errCode from 'err-code'
import { CustomProgressEvent } from 'progress-events'
import { dirBuilder, type DirBuilderOptions } from './dir.js'
import { fileBuilder, type FileBuilderOptions } from './file.js'
import type { ChunkValidator } from './validate-chunks.js'
import type { Chunker } from '../chunker/index.js'
import type { Directory, File, FileCandidate, ImportCandidate, ImporterProgressEvents, InProgressImportResult, WritableStorage } from '../index.js'
import type { ProgressEvent, ProgressOptions } from 'progress-events'
/**
* Passed to the onProgress callback while importing files
*/
export interface ImportReadProgress {
/**
* How many bytes we have read from this source so far
*/
bytesRead: bigint
/**
* The size of the current chunk
*/
chunkSize: bigint
/**
* The path of the file being imported, if one was specified
*/
path?: string
}
export type DagBuilderProgressEvents =
ProgressEvent<'unixfs:importer:progress:file:read', ImportReadProgress>
function isIterable (thing: any): thing is Iterable<any> {
return Symbol.iterator in thing
}
function isAsyncIterable (thing: any): thing is AsyncIterable<any> {
return Symbol.asyncIterator in thing
}
function contentAsAsyncIterable (content: Uint8Array | AsyncIterable<Uint8Array> | Iterable<Uint8Array>): AsyncIterable<Uint8Array> {
try {
if (content instanceof Uint8Array) {
return (async function * () {
yield content
}())
} else if (isIterable(content)) {
return (async function * () {
yield * content
}())
} else if (isAsyncIterable(content)) {
return content
}
} catch {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT')
}
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT')
}
export interface DagBuilderOptions extends FileBuilderOptions, DirBuilderOptions, ProgressOptions<ImporterProgressEvents> {
chunker: Chunker
chunkValidator: ChunkValidator
wrapWithDirectory: boolean
}
export type ImporterSourceStream = AsyncIterable<ImportCandidate> | Iterable<ImportCandidate>
export interface DAGBuilder {
(source: ImporterSourceStream, blockstore: WritableStorage): AsyncIterable<() => Promise<InProgressImportResult>>
}
export function defaultDagBuilder (options: DagBuilderOptions): DAGBuilder {
return async function * dagBuilder (source, blockstore) {
for await (const entry of source) {
let originalPath: string | undefined
if (entry.path != null) {
originalPath = entry.path
entry.path = entry.path
.split('/')
.filter(path => path != null && path !== '.')
.join('/')
}
if (isFileCandidate(entry)) {
const file: File = {
path: entry.path,
mtime: entry.mtime,
mode: entry.mode,
content: (async function * () {
let bytesRead = 0n
for await (const chunk of options.chunker(options.chunkValidator(contentAsAsyncIterable(entry.content)))) {
const currentChunkSize = BigInt(chunk.byteLength)
bytesRead += currentChunkSize
options.onProgress?.(new CustomProgressEvent<ImportReadProgress>('unixfs:importer:progress:file:read', {
bytesRead,
chunkSize: currentChunkSize,
path: entry.path
}))
yield chunk
}
})(),
originalPath
}
yield async () => fileBuilder(file, blockstore, options)
} else if (entry.path != null) {
const dir: Directory = {
path: entry.path,
mtime: entry.mtime,
mode: entry.mode,
originalPath
}
yield async () => dirBuilder(dir, blockstore, options)
} else {
throw new Error('Import candidate must have content or path or both')
}
}
}
}
function isFileCandidate (entry: any): entry is FileCandidate {
return entry.content != null
}
@@ -0,0 +1,24 @@
import errCode from 'err-code'
import { fromString as uint8ArrayFromString } from 'uint8arrays/from-string'
export interface ChunkValidator { (source: AsyncIterable<Uint8Array>): AsyncIterable<Uint8Array> }
export const defaultChunkValidator = (): ChunkValidator => {
return async function * validateChunks (source) {
for await (const content of source) {
if (content.length === undefined) {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT')
}
if (typeof content === 'string' || content instanceof String) {
yield uint8ArrayFromString(content.toString())
} else if (Array.isArray(content)) {
yield Uint8Array.from(content)
} else if (content instanceof Uint8Array) {
yield content
} else {
throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT')
}
}
}
}
+119
View File
@@ -0,0 +1,119 @@
import { encode, type PBNode, prepare } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'
import { Dir, CID_V0, CID_V1, type DirProps } from './dir.js'
import { persist, type PersistOptions } from './utils/persist.js'
import type { ImportResult, InProgressImportResult } from './index.js'
import type { Blockstore } from 'interface-blockstore'
import type { CID } from 'multiformats/cid'
export class DirFlat extends Dir {
private readonly _children: Map<string, InProgressImportResult | Dir>
constructor (props: DirProps, options: PersistOptions) {
super(props, options)
this._children = new Map()
}
async put (name: string, value: InProgressImportResult | Dir): Promise<void> {
this.cid = undefined
this.size = undefined
this.nodeSize = undefined
this._children.set(name, value)
}
async get (name: string): Promise<InProgressImportResult | Dir | undefined> {
return Promise.resolve(this._children.get(name))
}
childCount (): number {
return this._children.size
}
directChildrenCount (): number {
return this.childCount()
}
onlyChild (): InProgressImportResult | Dir {
return this._children.values().next().value
}
async * eachChildSeries (): AsyncGenerator<{ key: string, child: InProgressImportResult | Dir }, void, undefined> {
for (const [key, child] of this._children.entries()) {
yield {
key,
child
}
}
}
estimateNodeSize (): number {
if (this.nodeSize !== undefined) {
return this.nodeSize
}
this.nodeSize = 0
// estimate size only based on DAGLink name and CID byte lengths
// https://github.com/ipfs/go-unixfsnode/blob/37b47f1f917f1b2f54c207682f38886e49896ef9/data/builder/directory.go#L81-L96
for (const [name, child] of this._children.entries()) {
if (child.size != null && (child.cid != null)) {
this.nodeSize += name.length + (this.options.cidVersion === 1 ? CID_V1.bytes.byteLength : CID_V0.bytes.byteLength)
}
}
return this.nodeSize
}
async * flush (block: Blockstore): AsyncGenerator<ImportResult> {
const links = []
for (const [name, child] of this._children.entries()) {
let result: { size?: bigint | number, cid?: CID } = child
if (child instanceof Dir) {
for await (const entry of child.flush(block)) {
result = entry
yield entry
}
}
if (result.size != null && (result.cid != null)) {
links.push({
Name: name,
Tsize: Number(result.size),
Hash: result.cid
})
}
}
const unixfs = new UnixFS({
type: 'directory',
mtime: this.mtime,
mode: this.mode
})
const node: PBNode = { Data: unixfs.marshal(), Links: links }
const buffer = encode(prepare(node))
const cid = await persist(buffer, block, this.options)
const size = buffer.length + node.Links.reduce(
/**
* @param {number} acc
* @param {PBLink} curr
*/
(acc, curr) => acc + (curr.Tsize == null ? 0 : curr.Tsize),
0)
this.cid = cid
this.size = size
yield {
cid,
unixfs,
path: this.path,
size: BigInt(size)
}
}
}
+258
View File
@@ -0,0 +1,258 @@
import { encode, type PBLink, prepare } from '@ipld/dag-pb'
import { murmur3128 } from '@multiformats/murmur3'
import { createHAMT, Bucket, type BucketChild } from 'hamt-sharding'
import { UnixFS } from 'ipfs-unixfs'
import { Dir, CID_V0, CID_V1, type DirProps } from './dir.js'
import { persist, type PersistOptions } from './utils/persist.js'
import type { ImportResult, InProgressImportResult } from './index.js'
import type { Blockstore } from 'interface-blockstore'
async function hamtHashFn (buf: Uint8Array): Promise<Uint8Array> {
return (await murmur3128.encode(buf))
// Murmur3 outputs 128 bit but, accidentally, IPFS Go's
// implementation only uses the first 64, so we must do the same
// for parity..
.slice(0, 8)
// Invert buffer because that's how Go impl does it
.reverse()
}
const HAMT_HASH_CODE = BigInt(0x22)
class DirSharded extends Dir {
private readonly _bucket: Bucket<InProgressImportResult | Dir>
constructor (props: DirProps, options: PersistOptions) {
super(props, options)
this._bucket = createHAMT({
hashFn: hamtHashFn,
bits: 8
})
}
async put (name: string, value: InProgressImportResult | Dir): Promise<void> {
this.cid = undefined
this.size = undefined
this.nodeSize = undefined
await this._bucket.put(name, value)
}
async get (name: string): Promise<InProgressImportResult | Dir | undefined> {
return this._bucket.get(name)
}
childCount (): number {
return this._bucket.leafCount()
}
directChildrenCount (): number {
return this._bucket.childrenCount()
}
onlyChild (): Bucket<InProgressImportResult | Dir> | BucketChild<InProgressImportResult | Dir> {
return this._bucket.onlyChild()
}
async * eachChildSeries (): AsyncGenerator<{ key: string, child: InProgressImportResult | Dir }> {
for await (const { key, value } of this._bucket.eachLeafSeries()) {
yield {
key,
child: value
}
}
}
estimateNodeSize (): number {
if (this.nodeSize !== undefined) {
return this.nodeSize
}
this.nodeSize = calculateSize(this._bucket, this, this.options)
return this.nodeSize
}
async * flush (blockstore: Blockstore): AsyncGenerator<ImportResult> {
for await (const entry of flush(this._bucket, blockstore, this, this.options)) {
yield {
...entry,
path: this.path
}
}
}
}
export default DirSharded
async function * flush (bucket: Bucket<Dir | InProgressImportResult>, blockstore: Blockstore, shardRoot: DirSharded | null, options: PersistOptions): AsyncIterable<ImportResult> {
const children = bucket._children
const links: PBLink[] = []
let childrenSize = 0n
for (let i = 0; i < children.length; i++) {
const child = children.get(i)
if (child == null) {
continue
}
const labelPrefix = i.toString(16).toUpperCase().padStart(2, '0')
if (child instanceof Bucket) {
let shard
for await (const subShard of flush(child, blockstore, null, options)) {
shard = subShard
}
if (shard == null) {
throw new Error('Could not flush sharded directory, no subshard found')
}
links.push({
Name: labelPrefix,
Tsize: Number(shard.size),
Hash: shard.cid
})
childrenSize += shard.size
} else if (isDir(child.value)) {
const dir = child.value
let flushedDir: ImportResult | undefined
for await (const entry of dir.flush(blockstore)) {
flushedDir = entry
yield flushedDir
}
if (flushedDir == null) {
throw new Error('Did not flush dir')
}
const label = labelPrefix + child.key
links.push({
Name: label,
Tsize: Number(flushedDir.size),
Hash: flushedDir.cid
})
childrenSize += flushedDir.size
} else {
const value = child.value
if (value.cid == null) {
continue
}
const label = labelPrefix + child.key
const size = value.size
links.push({
Name: label,
Tsize: Number(size),
Hash: value.cid
})
childrenSize += BigInt(size ?? 0)
}
}
// go-ipfs uses little endian, that's why we have to
// reverse the bit field before storing it
const data = Uint8Array.from(children.bitField().reverse())
const dir = new UnixFS({
type: 'hamt-sharded-directory',
data,
fanout: BigInt(bucket.tableSize()),
hashType: HAMT_HASH_CODE,
mtime: shardRoot?.mtime,
mode: shardRoot?.mode
})
const node = {
Data: dir.marshal(),
Links: links
}
const buffer = encode(prepare(node))
const cid = await persist(buffer, blockstore, options)
const size = BigInt(buffer.byteLength) + childrenSize
yield {
cid,
unixfs: dir,
size
}
}
function isDir (obj: any): obj is Dir {
return typeof obj.flush === 'function'
}
function calculateSize (bucket: Bucket<any>, shardRoot: DirSharded | null, options: PersistOptions): number {
const children = bucket._children
const links: PBLink[] = []
for (let i = 0; i < children.length; i++) {
const child = children.get(i)
if (child == null) {
continue
}
const labelPrefix = i.toString(16).toUpperCase().padStart(2, '0')
if (child instanceof Bucket) {
const size = calculateSize(child, null, options)
links.push({
Name: labelPrefix,
Tsize: Number(size),
Hash: options.cidVersion === 0 ? CID_V0 : CID_V1
})
} else if (typeof child.value.flush === 'function') {
const dir = child.value
const size = dir.nodeSize()
links.push({
Name: labelPrefix + child.key,
Tsize: Number(size),
Hash: options.cidVersion === 0 ? CID_V0 : CID_V1
})
} else {
const value = child.value
if (value.cid == null) {
continue
}
const label = labelPrefix + child.key
const size = value.size
links.push({
Name: label,
Tsize: Number(size),
Hash: value.cid
})
}
}
// go-ipfs uses little endian, that's why we have to
// reverse the bit field before storing it
const data = Uint8Array.from(children.bitField().reverse())
const dir = new UnixFS({
type: 'hamt-sharded-directory',
data,
fanout: BigInt(bucket.tableSize()),
hashType: HAMT_HASH_CODE,
mtime: shardRoot?.mtime,
mode: shardRoot?.mode
})
const buffer = encode(prepare({
Data: dir.marshal(),
Links: links
}))
return buffer.length
}
+63
View File
@@ -0,0 +1,63 @@
import { CID } from 'multiformats/cid'
import type { WritableStorage, ImportResult, InProgressImportResult } from './index.js'
import type { PersistOptions } from './utils/persist.js'
import type { Mtime, UnixFS } from 'ipfs-unixfs'
export interface DirProps {
root: boolean
dir: boolean
path: string
dirty: boolean
flat: boolean
parent?: Dir
parentKey?: string
unixfs?: UnixFS
mode?: number
mtime?: Mtime
}
export abstract class Dir {
public options: PersistOptions
public root: boolean
public dir: boolean
public path: string
public dirty: boolean
public flat: boolean
public parent?: Dir
public parentKey?: string
public unixfs?: UnixFS
public mode?: number
public mtime?: Mtime
public cid?: CID
public size?: number
public nodeSize?: number
constructor (props: DirProps, options: PersistOptions) {
this.options = options ?? {}
this.root = props.root
this.dir = props.dir
this.path = props.path
this.dirty = props.dirty
this.flat = props.flat
this.parent = props.parent
this.parentKey = props.parentKey
this.unixfs = props.unixfs
this.mode = props.mode
this.mtime = props.mtime
}
abstract put (name: string, value: InProgressImportResult | Dir): Promise<void>
abstract get (name: string): Promise<InProgressImportResult | Dir | undefined>
abstract eachChildSeries (): AsyncIterable<{ key: string, child: InProgressImportResult | Dir }>
abstract flush (blockstore: WritableStorage): AsyncGenerator<ImportResult>
abstract estimateNodeSize (): number
abstract childCount (): number
}
// we use these to calculate the node size to use as a check for whether a directory
// should be sharded or not. Since CIDs have a constant length and We're only
// interested in the data length and not the actual content identifier we can use
// any old CID instead of having to hash the data which is expensive.
export const CID_V0 = CID.parse('QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn')
export const CID_V1 = CID.parse('zdj7WbTaiJT1fgatdet9Ei9iDB5hdCxkbVyhyh8YTUnXMiwYi')
+52
View File
@@ -0,0 +1,52 @@
import { DirFlat } from './dir-flat.js'
import DirSharded from './dir-sharded.js'
import type { Dir } from './dir.js'
import type { PersistOptions } from './utils/persist.js'
export async function flatToShard (child: Dir | null, dir: Dir, threshold: number, options: PersistOptions): Promise<DirSharded> {
let newDir = dir as DirSharded
if (dir instanceof DirFlat && dir.estimateNodeSize() > threshold) {
newDir = await convertToShard(dir, options)
}
const parent = newDir.parent
if (parent != null) {
if (newDir !== dir) {
if (child != null) {
child.parent = newDir
}
if (newDir.parentKey == null) {
throw new Error('No parent key found')
}
await parent.put(newDir.parentKey, newDir)
}
return flatToShard(newDir, parent, threshold, options)
}
return newDir
}
async function convertToShard (oldDir: DirFlat, options: PersistOptions): Promise<DirSharded> {
const newDir = new DirSharded({
root: oldDir.root,
dir: true,
parent: oldDir.parent,
parentKey: oldDir.parentKey,
path: oldDir.path,
dirty: oldDir.dirty,
flat: false,
mtime: oldDir.mtime,
mode: oldDir.mode
}, options)
for await (const { key, child } of oldDir.eachChildSeries()) {
await newDir.put(key, child)
}
return newDir
}
+398
View File
@@ -0,0 +1,398 @@
import errcode from 'err-code'
import first from 'it-first'
import parallelBatch from 'it-parallel-batch'
import { fixedSize } from './chunker/fixed-size.js'
import { type BufferImportProgressEvents, defaultBufferImporter } from './dag-builder/buffer-importer.js'
import { type DAGBuilder, type DagBuilderProgressEvents, defaultDagBuilder } from './dag-builder/index.js'
import { type ChunkValidator, defaultChunkValidator } from './dag-builder/validate-chunks.js'
import { balanced, type FileLayout } from './layout/index.js'
import { defaultTreeBuilder } from './tree-builder.js'
import type { Chunker } from './chunker/index.js'
import type { ReducerProgressEvents } from './dag-builder/file.js'
import type { Blockstore } from 'interface-blockstore'
import type { AwaitIterable } from 'interface-store'
import type { UnixFS, Mtime } from 'ipfs-unixfs'
import type { CID, Version as CIDVersion } from 'multiformats/cid'
import type { ProgressOptions } from 'progress-events'
export type ByteStream = AwaitIterable<Uint8Array>
export type ImportContent = ByteStream | Uint8Array
export type WritableStorage = Pick<Blockstore, 'put'>
export interface FileCandidate {
path?: string
content: ImportContent
mtime?: Mtime
mode?: number
}
export interface DirectoryCandidate {
path: string
mtime?: Mtime
mode?: number
}
export type ImportCandidate = FileCandidate | DirectoryCandidate
export interface File {
content: AsyncIterable<Uint8Array>
path?: string
mtime?: Mtime
mode?: number
originalPath?: string
}
export interface Directory {
path?: string
mtime?: Mtime
mode?: number
originalPath?: string
}
export interface ImportResult {
cid: CID
size: bigint
path?: string
unixfs?: UnixFS
}
export interface MultipleBlockImportResult extends ImportResult {
originalPath?: string
}
export interface SingleBlockImportResult extends ImportResult {
single: true
originalPath?: string
block: Uint8Array
}
export type InProgressImportResult = SingleBlockImportResult | MultipleBlockImportResult
export interface BufferImporterResult extends ImportResult {
block: Uint8Array
}
export interface HamtHashFn { (value: Uint8Array): Promise<Uint8Array> }
export interface TreeBuilder { (source: AsyncIterable<InProgressImportResult>, blockstore: WritableStorage): AsyncIterable<ImportResult> }
export interface BufferImporter { (file: File, blockstore: WritableStorage): AsyncIterable<() => Promise<BufferImporterResult>> }
export type ImporterProgressEvents =
BufferImportProgressEvents |
DagBuilderProgressEvents |
ReducerProgressEvents
/**
* Options to control the importer's behaviour
*/
export interface ImporterOptions extends ProgressOptions<ImporterProgressEvents> {
/**
* When a file would span multiple DAGNodes, if this is true the leaf nodes
* will not be wrapped in `UnixFS` protobufs and will instead contain the
* raw file bytes. Default: true
*/
rawLeaves?: boolean
/**
* If the file being imported is small enough to fit into one DAGNodes, store
* the file data in the root node along with the UnixFS metadata instead of
* in a leaf node which would then require additional I/O to load. Default: true
*/
reduceSingleLeafToSelf?: boolean
/**
* What type of UnixFS node leaves should be - can be `'file'` or `'raw'`
* (ignored when `rawLeaves` is `true`).
*
* This option exists to simulate kubo's trickle dag which uses a combination
* of `'raw'` UnixFS leaves and `reduceSingleLeafToSelf: false`.
*
* For modern code the `rawLeaves: true` option should be used instead so leaves
* are plain Uint8Arrays without a UnixFS/Protobuf wrapper.
*/
leafType?: 'file' | 'raw'
/**
* the CID version to use when storing the data. Default: 1
*/
cidVersion?: CIDVersion
/**
* If the serialized node is larger than this it might be converted to a HAMT
* sharded directory. Default: 256KiB
*/
shardSplitThresholdBytes?: number
/**
* How many files to import concurrently. For large numbers of small files this
* should be high (e.g. 50). Default: 10
*/
fileImportConcurrency?: number
/**
* How many blocks to hash and write to the block store concurrently. For small
* numbers of large files this should be high (e.g. 50). Default: 50
*/
blockWriteConcurrency?: number
/**
* If true, all imported files and folders will be contained in a directory that
* will correspond to the CID of the final entry yielded. Default: false
*/
wrapWithDirectory?: boolean
/**
* The chunking strategy. See [./src/chunker/index.ts](./src/chunker/index.ts)
* for available chunkers. Default: fixedSize
*/
chunker?: Chunker
/**
* How the DAG that represents files are created. See
* [./src/layout/index.ts](./src/layout/index.ts) for available layouts. Default: balanced
*/
layout?: FileLayout
/**
* This option can be used to override the importer internals.
*
* This function should read `{ path, content }` entries from `source` and turn them
* into DAGs
* It should yield a `function` that returns a `Promise` that resolves to
* `{ cid, path, unixfs, node }` where `cid` is a `CID`, `path` is a string, `unixfs`
* is a UnixFS entry and `node` is a `DAGNode`.
* Values will be pulled from this generator in parallel - the amount of parallelisation
* is controlled by the `fileImportConcurrency` option (default: 50)
*/
dagBuilder?: DAGBuilder
/**
* This option can be used to override the importer internals.
*
* This function should read `{ cid, path, unixfs, node }` entries from `source` and
* place them in a directory structure
* It should yield an object with the properties `{ cid, path, unixfs, size }` where
* `cid` is a `CID`, `path` is a string, `unixfs` is a UnixFS entry and `size` is a `Number`.
*/
treeBuilder?: TreeBuilder
/**
* This option can be used to override the importer internals.
*
* This function should read `Buffer`s from `source` and persist them using `blockstore.put`
* or similar
* `entry` is the `{ path, content }` entry, where `entry.content` is an async
* generator that yields Buffers
* It should yield functions that return a Promise that resolves to an object with
* the properties `{ cid, unixfs, size }` where `cid` is a [CID], `unixfs` is a [UnixFS] entry and `size` is a `Number` that represents the serialized size of the [IPLD] node that holds the buffer data.
* Values will be pulled from this generator in parallel - the amount of
* parallelisation is controlled by the `blockWriteConcurrency` option (default: 10)
*/
bufferImporter?: BufferImporter
/**
* This option can be used to override the importer internals.
*
* This function takes input from the `content` field of imported entries.
* It should transform them into `Buffer`s, throwing an error if it cannot.
* It should yield `Buffer` objects constructed from the `source` or throw an
* `Error`
*/
chunkValidator?: ChunkValidator
}
export type ImportCandidateStream = AsyncIterable<FileCandidate | DirectoryCandidate> | Iterable<FileCandidate | DirectoryCandidate>
/**
* The importer creates UnixFS DAGs and stores the blocks that make
* them up in the passed blockstore.
*
* @example
*
* ```typescript
* import { importer } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [{
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }, {
* path: './bar.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }]
*
* for await (const entry of importer(input, blockstore)) {
* console.info(entry)
* // { cid: CID(), ... }
* }
* ```
*/
export async function * importer (source: ImportCandidateStream, blockstore: WritableStorage, options: ImporterOptions = {}): AsyncGenerator<ImportResult, void, unknown> {
let candidates: AsyncIterable<FileCandidate | DirectoryCandidate> | Iterable<FileCandidate | DirectoryCandidate>
if (Symbol.asyncIterator in source || Symbol.iterator in source) {
candidates = source
} else {
candidates = [source]
}
const wrapWithDirectory = options.wrapWithDirectory ?? false
const shardSplitThresholdBytes = options.shardSplitThresholdBytes ?? 262144
const cidVersion = options.cidVersion ?? 1
const rawLeaves = options.rawLeaves ?? true
const leafType = options.leafType ?? 'file'
const fileImportConcurrency = options.fileImportConcurrency ?? 50
const blockWriteConcurrency = options.blockWriteConcurrency ?? 10
const reduceSingleLeafToSelf = options.reduceSingleLeafToSelf ?? true
const chunker = options.chunker ?? fixedSize()
const chunkValidator = options.chunkValidator ?? defaultChunkValidator()
const buildDag: DAGBuilder = options.dagBuilder ?? defaultDagBuilder({
chunker,
chunkValidator,
wrapWithDirectory,
layout: options.layout ?? balanced(),
bufferImporter: options.bufferImporter ?? defaultBufferImporter({
cidVersion,
rawLeaves,
leafType,
onProgress: options.onProgress
}),
blockWriteConcurrency,
reduceSingleLeafToSelf,
cidVersion,
onProgress: options.onProgress
})
const buildTree: TreeBuilder = options.treeBuilder ?? defaultTreeBuilder({
wrapWithDirectory,
shardSplitThresholdBytes,
cidVersion,
onProgress: options.onProgress
})
for await (const entry of buildTree(parallelBatch(buildDag(candidates, blockstore), fileImportConcurrency), blockstore)) {
yield {
cid: entry.cid,
path: entry.path,
unixfs: entry.unixfs,
size: entry.size
}
}
}
/**
* `importFile` is similar to `importer` except it accepts a single
* `FileCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importFile } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: FileCandidate = {
* path: './foo.txt',
* content: Uint8Array.from([0, 1, 2, 3, 4])
* }
*
* const entry = await importFile(input, blockstore)
* ```
*/
export async function importFile (content: FileCandidate, blockstore: WritableStorage, options: ImporterOptions = {}): Promise<ImportResult> {
const result = await first(importer([content], blockstore, options))
if (result == null) {
throw errcode(new Error('Nothing imported'), 'ERR_INVALID_PARAMS')
}
return result
}
/**
* `importDir` is similar to `importer` except it accepts a single
* `DirectoryCandidate` and returns a promise of a single `ImportResult`
* instead of a stream of results.
*
* @example
*
* ```typescript
* import { importDirectory } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input: DirectoryCandidate = {
* path: './foo.txt'
* }
*
* const entry = await importDirectory(input, blockstore)
* ```
*/
export async function importDirectory (content: DirectoryCandidate, blockstore: WritableStorage, options: ImporterOptions = {}): Promise<ImportResult> {
const result = await first(importer([content], blockstore, options))
if (result == null) {
throw errcode(new Error('Nothing imported'), 'ERR_INVALID_PARAMS')
}
return result
}
/**
* `importBytes` accepts a single Uint8Array and returns a promise
* of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importBytes } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = Uint8Array.from([0, 1, 2, 3, 4])
*
* const entry = await importBytes(input, blockstore)
* ```
*/
export async function importBytes (buf: ImportContent, blockstore: WritableStorage, options: ImporterOptions = {}): Promise<ImportResult> {
return importFile({
content: buf
}, blockstore, options)
}
/**
* `importByteStream` accepts a single stream of Uint8Arrays and
* returns a promise of a single `ImportResult`.
*
* @example
*
* ```typescript
* import { importByteStream } from 'ipfs-unixfs-importer'
* import { MemoryBlockstore } from 'blockstore-core'
*
* // store blocks in memory, other blockstores are available
* const blockstore = new MemoryBlockstore()
*
* const input = [
* Uint8Array.from([0, 1, 2, 3, 4]),
* Uint8Array.from([5, 6, 7, 8, 9])
* ]
*
* const entry = await importByteStream(input, blockstore)
* ```
*/
export async function importByteStream (bufs: ByteStream, blockstore: WritableStorage, options: ImporterOptions = {}): Promise<ImportResult> {
return importFile({
content: bufs
}, blockstore, options)
}
+27
View File
@@ -0,0 +1,27 @@
import batch from 'it-batch'
import type { FileLayout } from './index.js'
import type { InProgressImportResult } from '../index.js'
const DEFAULT_MAX_CHILDREN_PER_NODE = 174
export interface BalancedOptions {
maxChildrenPerNode?: number
}
export function balanced (options?: BalancedOptions): FileLayout {
const maxChildrenPerNode = options?.maxChildrenPerNode ?? DEFAULT_MAX_CHILDREN_PER_NODE
return async function balancedLayout (source, reduce): Promise<InProgressImportResult> {
const roots = []
for await (const chunked of batch(source, maxChildrenPerNode)) {
roots.push(await reduce(chunked))
}
if (roots.length > 1) {
return balancedLayout(roots, reduce)
}
return roots[0]
}
}
+9
View File
@@ -0,0 +1,9 @@
import all from 'it-all'
import type { FileLayout } from './index.js'
import type { InProgressImportResult } from '../index.js'
export function flat (): FileLayout {
return async function flatLayout (source, reduce): Promise<InProgressImportResult> {
return reduce(await all(source))
}
}

Some files were not shown because too many files have changed in this diff Show More