Goal: be able to do parallel processing and high-speed download of the processed output.

At the moment, parallelism is triggered when you depend on a store at a block number later than that store's first block. It is done in the background, and reports progress but doesn't push the data out.

A system like graph-node needs high-speed historical processing, but also a way to get all the produced output at high-speed.

Two options exist:

1. Inside the same request stream that orchestrates everything and reports progress, saturating that request's network pipe with the output produced in parallel.
2. As a separate process, potentially running on multiple machines, that can saturate multiple network interface. This puts more burden on the client, which is now responsible to compute segments to retrieve, stitch them together, etc. This is possible right now with a few run commands.

Solution 1

This can be linearized with relatively good efficiency.

This has the benefit of being very simple for the consumer.

This also allows the data to be sent as soon as it is ready.

This requires changes to the Request/Response messages.

i) The scheduler will need to be able to handle modules of any type (map, stores)
ii) A new component of the orchestrator needs to keep track of the data produced, and stream it linearly to the end user. One instance of that component per module being streamed out, with some sort of cursor or progress marker per module.

Current implementation hypothesis:

• Watch module’s output files and stream out the data linearly
• Observe the progress (via GRPC) of the back-process and get the files when ready.

substreams run substream.yaml graph_out -s 15_000_000 -t +1 --download-history

Here's an example of what things could look like from the proto's perspective:

 message Request {
...
   Modules modules = 6;
-  repeated string output_modules = 7;
-  repeated string initial_store_snapshot_for_modules = 8;
+  repeated ModuleRequest requested_modules = 7;
+  // repeated string initial_store_snapshot_for_modules = 8;
+  // repeated ModuleAt historical_output_modules = 9; // store_mints@0, map_volumes@12000000
 }
 
+message ModuleRequest {
+  string module_name = 1;
+  optional int64 download_history_from_block_num = 2;  // 0 = activates history download from module's initial block, non-0 but present = activates history download FROM that block
+  bool send_initial_snapshot = 3;
+}

WARN: that would be a breaking change from the network protocol, so we'll want to rehash it, in the light of other features we want to add, before taking that route.

From the CLI, what would it look like to continue a broken history download?

substreams run spkg store_mints@800000 -s 12M --download-history


substreams run spkg store_mints,store_accounts -s 12M -t +1 -i

# backfill les caches jusqu'à 12M, il me dump TOUS les _stores_ qui sont sur la ligne de commande.
substreams run spkg store_mints+snapshot@1200000,store_accounts -s 12M -t +1 --download-history
substreams run spkg store_mints+snapshot+history@120000,store_accounts -s 12M -t +1 --download-history

#  backfill les caches jusqu'à 12M, il me dump TOUS les _stores_ qui sont sur la ligne de commande.
substreams run spkg store_mints -s 12M -t +1
# check les caches pour les stores dépendants, jusqu'à 12M, outputter le contenu de `store_mints` going forward

Solution 2

Two steps back-processing, and then download in parallel.

Currently when the back processing is scheduled (by specifying a start_block later then the manifest start_block), all of the output is sent to the tier 1. Tier 1 absorbs the data and never sends it back to the consumer. Currently the consumer would need to re-run the same request to get the data that was processed from cache.

i) A client runs a back-processing job until the HEAD block you want

substreams run substream.yaml graph_out -s 15_000_000 -t +1

ii) client to download the data in parallel once the back processing is completed. This is a two commands process, you are essentially preparing the data, and then downloading it.

substreams run substream.yaml graph_out -t 15_000_000

• store total size is now limited to 1GiB.
• TotalSizeBytes is calculated on load (during the unmarshal) and modified on every kv operation to avoid recalculation

• [x] Augment https://substreams.streamingfast.io/concept-and-fundamentals/modules/outputs to discuss the fact that only 1 output is possible today and it must be of a proto type (no literal like String or bool, if a single value is required, wrap in a Proto message)
• [x] Make list of https://substreams.streamingfast.io/reference-and-specs/examples more discoverable by linking it, maybe in https://substreams.streamingfast.io/getting-started/your-first-stream in Next Steps and maybe somehwere in https://substreams.streamingfast.io/developer-guide/overview section?
• [x] Throughout https://substreams.streamingfast.io/developer-guide/, there is some old module name reference mainly block_to_transfers and nft_state, they need to be changed because they are not aligned with substreams-template. All occurrences should be renamed
  • Update also https://substreams.streamingfast.io/developer-guide/creating-your-manifest#module-definitions so that modules have either map_ and store_ prefixes, we have settled more or less for this convention.
  • We also need to have this loose "convention" documented at various places in the documentation maybe, maybe as a "note" element. Not sure exactly where, probably where first introduction to creating a new module is made and maybe in "reference" page.
• [x] Updated https://substreams.streamingfast.io/developer-guide/setting-up-handlers to use latest ways of defining modules. If you compare with substreams-template/Cargo.toml, you will see the file is a bit different now. Copying the differences over will also update dependencies like substreams and substreams-ethereum to their latest version
  • https://substreams.streamingfast.io/developer-guide/setting-up-handlers#rust-toolchain also needs to be updated
• [x] In https://substreams.streamingfast.io/developer-guide/running-substreams, let's remove the content about running locally, not important in this section.

Added an optional flag to specify output-dir for pack command. Also changed all occurrences of /tmp/ folder paths to use os.TempDir() instead to allow cli tool to work on windows OS.

Remove the second paragraph of "Features", touting Search and Lifecycle as products. Remove all "Top-tier products" in the Overview section. Update "Protocols", to include Solana, NEAR. Start with Ethereum. Also include as "Contributed by third-parties", the ones by Figment (https://github.com/figment-networks/firehose-cosmos which includes the Cosmos Hub and Osmosis network). Remove the EOSIO one from the Protocols list. Links to Docs should point to: the FIrehose docs (firehose.streamingfast.io) and Substreams (to substreams.streamingfast.io) Remove the "Common interfaces" section. Grab Google's CODE_OF_CONDUCT doc from somewhere, and tweak it to reflect our name, and put it in this repository. When we have a code of conduct linked somewhere, we can ask GitBook for a free "open-source project" license. Change the "about" and "website" section of the GitHub repository, to point to streamingfast.io and our latest company name.

Review the change history of the Substreams repository https://github.com/streamingfast/substreams and ask the team what's new, find a way to present the new things in the change-log, sync the GitHub releases page with what's new.

Let's plan together how we could best communicate what is new. Our release process is not so strictly defined, so we can figure out what's best together.

Hi guys, one question regarding a custom sink that I've written similar to this one here https://github.com/streamingfast/substreams-playground/tree/master/consumers/rust.

I have managed to create my substreams modules build a package and I can see my consumer successfully processing, and in my case, storing the indexed data into a DB. However, I've noticed something interesting; at some point the stream does not push any new output results from substream. More specifically, it syncs until some recent block and then it stops.

For example, I'm running a local firehose-aptos node, and I see some block being streamed

FIRE BLOCK_END 156383
FIRE BLOCK_END 156384
FIRE BLOCK_END 156384

However my consumer stop at block 154794.

There is nothing specific about those block numbers. The point is that if I start the consumer from block 0 again (and reseting the last processed cursor) it always syncs correctly but always stop close to the block that firehose has processed. So for example the second time I run this it stops at block 155132 and so on. Also I don't reset the cursor and start the consumer then it remains stack doesn't receive any block results.

An update on this matter. I've been trying a lot of things but nothing had worked so far. The latest test I made was using substreams directly without the custom sink.

So I run this command

substreams run -p -e aptos-firehose.local:18015 substreams.yaml db_out --start-block 0 --stop-block 100000000000000000

An it just got stack at block 174599.

Then I run substreams run -p -e aptos-firehose.local:18015 substreams.yaml db_out --start-block 174599 --stop-block 100000000000000000

It doesn't do anything. It simply shows the following:

Connected - Progress messages received: 1203 (0/sec)
Backprocessing history up to requested target block 174599:
(hit 'm' to switch mode)

store_floor_price                 0  ::  0-174000 
store_listings                    0  ::  0-174598 
store_nft_volume_from_lis         0  ::  0-174000 
store_nft_volume_from_off         0  ::  0-174000 
store_offers                      0  ::  0-174598 

I can verify that my local aptos-firehose node is running because I'm seeing constant flow of logs of this type

FIRE BLOCK_END 176050
FIRE BLOCK_START 176051

It feels like it stops when it reaches a block height which is 500 behind the latest block firehose stores. at this point I really don't know how to troubleshoot this. I would really appreciate if someone could provide some hint at least.

The only way to make it run again is to restart it from a block number that is divisible by 100. So in the above example:

This fails

substreams run -p  -e aptos-firehose.local:18015 substreams.yaml db_out --start-block 174599 --stop-block 100000000000000000

but this works

substreams run -p  -e aptos-firehose.local:18015 substreams.yaml db_out --start-block 174500 --stop-block 100000000000000000 

Had an instance where substreams hangs at a block, eventually terminating with an internal error.

Haven't been able to reproduce the error, but wanted to capture it for diagnosing.

substreams run -e api-dev.streamingfast.io:443 substreams.yaml map_market_cap -s 13000002 -t +10
Connected - Progress messages received: 0 (0/sec)
Backprocessing history up to requested target block 13000002:
(hit 'm' to switch mode)

----------- NEW BLOCK #13,000,002 (13000002) ---------------
----------- NEW BLOCK #13,000,003 (13000003) ---------------
----------- NEW BLOCK #13,000,004 (13000004) ---------------
----------- NEW BLOCK #13,000,005 (13000005) ---------------
----------- NEW BLOCK #13,000,006 (13000006) ---------------
----------- NEW BLOCK #13,000,007 (13000007) ---------------
----------- NEW BLOCK #13,000,008 (13000008) ---------------
----------- NEW BLOCK #13,000,009 (13000009) ---------------
----------- NEW BLOCK #13,000,010 (13000010) ---------------
Error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

originally posted: https://github.com/messari/substreams/issues/61

Example:

I have two substreams (erc721 and uniswap) with the following folder structure:

- erc721
  - substreams.yaml
  - target/.../substreams_erc721.wasm
- uniswap
  - substreams.yaml
  - target/.../substreams_uniswap.wasm

When I import the erc721 substream in uniswap's substreams.yaml, and build, I see the following error:

~/go/bin/substreams protogen ./substreams.yaml --exclude-paths="sf/ethereum,sf/substreams,google"
Error: reading manifest "./substreams.yaml": error loading imports: importing "../erc721/substreams.yaml": failed to convert manifest to pkg: failed to read source code "./target/wasm32-unknown-unknown/release/substreams_erc721.wasm": open ./target/wasm32-unknown-unknown/release/substreams_erc721.wasm: no such file or directory

However, if I copy erc721/target/.../substreams_erc721.wasm to under uniswap/target/.../substreams_erc721.wasm the error goes away. I think substreams is looking for the binary in the main substreams.yaml instead of the imported substreams.yaml.

See https://github.com/messari/substreams/blob/2ef4b9278c8a6d372c48e1df8c4be869081f481b/uniswap-v2/substreams.yaml#L8 or https://github.com/messari/substreams/commit/2ef4b9278c8a6d372c48e1df8c4be869081f481b

We need to formalize the Rust create we provide the user. Currently we offer

https://crates.io/crates/substreams https://crates.io/crates/substreams-ethereum

Most likely we would need to add:

• crate that would expose the graph-node entities as well as helpful macros.
• create that would export substreams database intergations
• we need to centralize and standardize all the data transaction and conversion we do constantly in all the substreams

Review the top-level README.md therein, and see what's still useful, and if the links are not broken. Beware this repository is the one under GitBook management.

• fix boundaries in prod mode vs dev mode for stores and maps

• missing: we have to remove the calculations around multiple mappers (there is only one mapper of interest, the outputModule..)
• missing: we have to add the concept of an "incomplete" store on disk, which does not end on a boundary -- it should contain the traceID in it so we don't end up with a race condition
• missing: we have to write more tests for the changes here...

Add Map Vs Stores section and populate with first draft content.

Please review this initial draft content and let me know if you see any issues, what changes need to be made, and what needs to be added, andprovide a few bullet points for the map area; look for Need Input 01, etc.

https://github.com/streamingfast/substreams#tooling Really wondering why that “Tooling” section made it to the top-level README.md :slightly_smiling_face: Clearly doesn’t belong there. Perhaps its own section under “CLI Reference”? Maybe we give a few hints at what the tools sub-commands do.

Each header is a general comment that might require further discussions

Prerequisites

prerequisites: this page says “myriad technologies”.. there’s no benefit in having multiple techs, it’s rather scary. Let’s say “uses a few technologies”, and highlight ease of use. Something like: “Substreams leverages powerful technologies, including:”

the phrase “Substreams can be used by blockchain, subgraph, Rust, JavaScript, Python, and other types of developers.” doesn’t belong there.. we’re talking pre-requisites, not personas.

One would expect instructions in such a page, not just a bulletpoint list of named techs. A bullet-point list of techs might belong to the Substreams overview in the first sections. The “Dependency Installation” already gives those instructions. So there’s nothing left for that page to exist. I suggest just moving the bullet-point list in “What is Substreams”, under a heading like “Substreams leverages powerful technologies”, and note something about these techs being mature, well documented and widely deployed.

Homepage

Arriving at the home page, I expect to find a clear, concise definition of what Substreams is. Perhaps swapping the first two sentences would do it. It’s more important to understand what it is, than who developed the technology.

The README.md https://github.com/streamingfast/substreams has a good 3-liner intro that we should use for the docs.

I’d also very much like to find here, right after Welcome, somethign exxttreeemly simple that can help bootstrap the mental model for Substreams. Something that includes a 3-lines mapper module, the presence of a Sink, a blockchain node. Something like the diagram in “Conceptual Diagram”.. It makes sense that it be up there.. so people can know where this whole thing fits.

We need at least the first paragraph of https://substreams.streamingfast.io/concept-and-fundamentals/definition on the first page.

Navigation

I’d love to have just two things under “Getting Started”:

• Installing the CLI
• Quick Start

The “Start Streaming” page has things that belong to a “Basics” section (which exists in the “Chain-agnostic Tutorial” (to become “Quick Start”).

I shrank the “Start Streaming” section, now it can become the first section in the Quick Start guide.. which makes sense. The fastest way to get started, is to just try it.. and then, write your very first thing.

Right now one needs to navigate and click a lot.. lots of pages, with similar sounding names: “Substreams” (first page), “What is Substreams” (hoped to find that on first page), “Conceptual Diagram” (and that’s not in What is Substreams?), “Fundamentals” (the first three pages didn’t cover the fundamentals?). Fundamentals could be named “How it works”, as it details some of the lower-level internals. People will want to get more into details in a “How it works” page.. and know what to expect. And we’ll know what to put there. That page can be longer, provided the headings are meaningful. For example “Key Steps” isn’t a meaningful navigation element. Same with “Working with Substreams Fundamentals” .. (working with the page? ooh.. sort of How to work with substreams fundamentals”).

Let's rename "Using the CLI” page -> “CLI Reference”

• “Using the CLI” reads more like a tutorial-esque thing.
• But this section is about Reference & Specs. More of a declarative format and extensive list-styles.