Intermediate · LESSON 03 OF 12
3. Design a production Solana indexer
By the end, you can choose an appropriate data source and design a versioned, observable indexing pipeline that preserves raw evidence, decodes transactions safely, and recovers without missing or double-processing events.
Look up Solana terms in the glossary
Optional reading history is saved only in this browser. It is not a scored checkpoint or exam result.
In this lesson8 field notes · practice · sources · checkpoint
Read a text version of this diagram
RPC history, WebSocket notifications, provider webhooks, and Geyser gRPC enter a source adapter that records cluster, commitment, slot, signature, and raw data. A versioned decoder resolves account indices and instruction bytes. Validation checks execution status, program, accounts, amount, and business invariants before an idempotent write advances a durable cursor. A bounded queue exposes backpressure; monitoring and reconciliation detect stale or incomplete coverage and trigger replay.
- Select source
- Persist raw event
- Decode transaction
- Validate payment
- Write and checkpoint
- Reconcile and replay
Read a text version: Resolve transaction bytes into evidence
Begin with the transaction version and preserve the raw message. Build the complete account-key list, including versioned loaded addresses, before resolving each instruction account index. Record the program ID, outer instruction index, inner instruction position when relevant, and encoded instruction bytes. Decode using the program interface active for that transaction and retain the decoder build. Then check transaction execution, account relationships, token program and mint, raw integer amount, and the expected state transition. A decoded event name alone does not prove the business claim.
- Versioned message
- Complete account keys
- Instruction indices and bytes
- Pinned program decoder
- Business validation
Read a text version: Compare Solana indexer data sources
HTTP RPC provides bounded lookup and a client-controlled history path, subject to provider retention. WebSocket offers low-latency notifications but needs history backfill after a disconnect. Webhooks reduce polling while retry order, retention, and guarantees depend on the provider. Geyser plugins and Yellowstone gRPC expose richer streams but add protocol decoding, authentication, bandwidth, and cursor operations. No transport removes the need to measure gaps and deduplicate observations. Compare achievable latency, recovery horizon, supported transaction versions, provider dependence, data ownership, and total operating cost against the actual product workload.
- Delivery and recovery tradeoffs
- HTTP RPC
- WebSocket
- Provider webhook
- Geyser and gRPC
FREE OFFLINE EVIDENCE WORKBENCH · UNGRADED
Make the index tell the truth.
Work through fictional versioned-transaction account keys, duplicate stream deliveries, a missing checkpoint range, a Geyser startup snapshot, and an unsupported instruction. Practice preserving provenance, making writes idempotent, recovering gaps, distinguishing baseline state from a payment, and quarantining unknown data. It runs locally without RPC, wallet, account, or SOL and does not save answers.
Open the Indexer Evidence Workbench Download the offline workbenchAll records are fictional. Continue to the official references below before applying these rules to a production index.
Use an index only when the product needs one
A Solana RPC node answers queries against ledger data it has available; it is not automatically your application database. For a low-volume payment verifier, fetch signatures for the destination address and inspect each transaction with an explicit commitment. That can be simpler and easier to audit than running a streaming stack. A production index becomes useful when repeated polling is costly, history must be retained for analytics or support, query shapes need to be fast, or the product must discover individual transfers inside complex transactions. An index is a derived view, not a new source of truth. Keep the cluster, signature, slot, commitment, execution result, and raw transaction evidence needed to reproduce a decision. State retention limits and provider history coverage in the product. Do not imply that a local database contains every historical event unless ingestion and backfill evidence support that claim.
Raw bytes and parsed events are different evidence
A transaction message identifies accounts by indices and carries instruction data as encoded bytes. The full account-key list for a versioned transaction can include loaded addresses, so an index into static keys alone can resolve the wrong account. A decoder turns program-specific bytes into named instruction fields; a parsed RPC representation may make familiar operations easier to read. Neither label makes an event trustworthy by itself. Preserve the signature, program ID, account keys in message order, instruction index, inner-instruction position when relevant, transaction version, execution error, slot, and source. Decode against the correct program and token-program variant, then check account relationships and the state transition the business rule expects. Use integer raw token amounts and mint decimals; never let floating-point display values decide whether an exact payment was made. Keep decoder version and source data so parser upgrades can be reviewed and old records rebuilt.
Choose a source for latency, history, and ownership
HTTP RPC is straightforward for bounded lookups and recovery. WebSocket subscriptions push notifications while connected, but are not a durable replay log; the previous lesson covers reconnection and backfill. Provider webhooks can reduce polling work, but retry, ordering, retention, filtering, and delivery guarantees are provider-specific. Geyser is an Agave validator plugin interface that streams account, slot, block, and transaction observations to external consumers. Yellowstone gRPC is a project that exposes Geyser streams through Protocol Buffers. It can reduce time-to-observation for transaction streams, but adds authentication, bandwidth, reconnect, schema-version, protobuf, operations, and history-recovery work. Hosted streams trade infrastructure ownership for provider terms, rate limits, retention, and service dependencies. Compare measured event volume, recovery window, required commitment, supported transaction versions, data ownership, and budget before choosing.
Build a pipeline that cannot silently advance past a gap
Separate source ingestion, normalization, decoding, validation, persistence, and business decisions. Give raw observations an idempotency key such as cluster plus signature plus instruction position; account updates may need account key plus slot and write version. Persist enough raw input and source provenance to replay decoding. Mark failed decodes as quarantined and visible, rather than dropping them or advancing a checkpoint as if they were handled. Use a bounded queue: a slow database must produce measurable lag or backpressure, not unbounded memory growth. Write normalized events and their durable cursor transactionally when possible. If they cannot share a transaction, design replay so a crash between event storage and cursor update safely reprocesses a duplicate. Deduplicate with a uniqueness constraint. Commit the cursor only after all earlier items in its declared ordering scope are durable. Make that scope explicit: a provider stream may deliver updates asynchronously, and a slot alone does not totally order every event.
Treat commitment, forks, and startup snapshots explicitly
A stream may report processed observations that later belong to a dropped fork. Confirmed and finalized observations support different user claims; keep the received commitment with each record and define when the product promotes or retracts a provisional result. With Geyser account callbacks, startup snapshot traffic is distinguishable from transaction-driven updates; a consumer that ignores the startup boundary can mistake a bulk initial load for fresh user activity. Agave documents that plugin callbacks must return quickly because slow work can make a validator fall behind, and recommends asynchronous persistence. Some callback orderings are version-specific, so use supported slot, transaction-index, write-version, and block-metadata fields instead of assuming arrival order. Do not deploy a custom plugin onto a validator without understanding its failure policy: persistence errors may deliberately stop the validator to preserve consistency. Most application teams should compare hosted indexing against the substantial work of operating their own validator plugin.
Decode transaction and account data with pinned interfaces
Yellowstone publishes Protocol Buffer messages; it does not hand your business processor a universal, fully decoded event. Resolve account indices using the transaction version, decode instruction bytes with the program interface active for that transaction, and distinguish outer instructions from inner CPI instructions. Token transfers can be affected by Token-2022 extensions, transfer hooks, authorities, and account ownership; a generic transfer label may not prove the product rule. Pin the protobuf and client version, program IDL or decoder package, token-program rules, and parser build used for each ingest range. When upgrading a parser, compare old and new outputs on fixtures, backfill into a new derived view, inspect differences, and switch only after reconciliation. Frameworks such as Carbon or parser-handler projects can accelerate this work, but packages and APIs evolve; verify current releases, supported programs, and maintenance before adopting them.
Monitor freshness, coverage, and the parts that can fail
A green connection indicator only says a transport is open. Track last observed slot and signature, source-to-database lag, oldest queued event, reconnects, retries, decode failures, duplicate rate, checkpoint age, and backfill progress. Alert when the index is stale or coverage is unknown. Reconcile a sample of indexed signatures against a second read path or archival source, including execution status and the exact accounts and amounts used by your application. Bound provider calls, database retries, and backfill pages. Keep credentials out of logs; restrict stream tokens to the required endpoint and rotate them. Separate service health from chain correctness: an API can be healthy while the source is behind or a parser silently skips a new instruction variant. Give users an honest current-through marker and preserve the source and commitment behind it.
Design recovery before the first production event
Create deterministic fixtures for duplicate deliveries, missing pages, a disconnect during a write, an unsupported transaction version, malformed or unknown instruction data, parser-version changes, a dropped processed fork, a database outage, and a restart after persistence but before cursor advancement. Prove that replay may repeat work but cannot omit an event or fulfill a payment twice. Compare totals and selected signatures across the live index, direct transaction reads, and the chosen archival source. Document how far back each source can recover and what the operator does when the saved cursor is outside retention. A bank or slot cursor is not automatically portable across providers, and a cursor from one cluster must never be resumed against another. Version filters and decoders beside the checkpoint; changing either may require backfill. This lesson was source-reviewed on 2026-10-03. Yellowstone, Carbon, and parser APIs are evolving projects, so treat project-specific behavior as versioned implementation detail and recheck linked documentation before deployment.
A detail worth keeping.
Inspect the official indexing guide and list fields already parsed versus raw bytes that still need decoding. Explain how loaded addresses and instruction account indices are resolved for the transaction version.
Work through an example
Sketch one Geyser transaction callback through durable storage. Mark external calls, queue boundaries, crash points, idempotency constraints, and when a checkpoint becomes safe.
For a fictional token payment feed, write a source decision record comparing HTTP backfill, WebSocket, provider webhooks, and Geyser gRPC. State event volume, acceptable lag, commitment, recovery horizon, transaction versions, provider guarantees, operating burden, and cost. Draw the ingest, raw evidence, decoder, validator, idempotent store, durable cursor, and business decision stages. Define idempotency keys and the checkpoint transaction boundary. Use synthetic fixtures only; do not submit a transaction or put a real provider token in the exercise.
GUIDED PRACTICE · INTERMEDIATE
Do the work, one step at a time.
Use fictional addresses and disposable test data. Do not enter a recovery phrase or private key. Most exercises work on paper or with a local test; only a lesson that clearly says “read-only Devnet” makes a public network request.
- Draft a source-selection matrix for a fictional application, including provider retention and outage recovery.
- Given synthetic raw and parsed transaction records, resolve static and loaded account keys, inspect instruction position and explain what decoder-version evidence is still needed.
- Design idempotency keys and a transactional write/checkpoint rule; walk through crashes before and after each write.
- Create a bounded-queue and lag dashboard specification with an explicit stale threshold and unknown-coverage state.
- Write fixtures for duplicate delivery, dropped-fork promotion, unsupported version, malformed bytes, and an absent history cursor.
- Review one current parser project release and record its pinned version, supported programs, source, and limitations before a production pilot.
Before you move on, check your reasoning
- C
- o
- m
- p
- a
- r
- e
- a
- W
- e
- b
- S
- o
- c
- k
- e
- t
- r
- e
- c
- o
- v
- e
- r
- y
- p
- a
- t
- h
- f
- r
- o
- m
- r
- p
- c
- -
- s
- t
- r
- e
- a
- m
- i
- n
- g
- w
- i
- t
- h
- a
- g
- R
- P
- C
- p
- r
- o
- d
- u
- c
- t
- i
- o
- n
- p
- i
- p
- e
- l
- i
- n
- e
- .
- S
- t
- a
- t
- e
- w
- h
- i
- c
- h
- d
- u
- r
- a
- b
- i
- l
- i
- t
- y
- p
- r
- o
- p
- e
- r
- t
- i
- e
- s
- c
- o
- m
- e
- f
- r
- o
- m
- y
- o
- u
- r
- o
- w
- n
- r
- e
- p
- l
- a
- y
- d
- e
- s
- i
- g
- n
- a
- n
- d
- w
- h
- i
- c
- h
- a
- r
- e
- o
- n
- l
- y
- p
- r
- o
- v
- i
- d
- e
- r
- p
- r
- o
- m
- i
- s
- e
- s
- .
OPEN SELF-CHECK · UNGRADED
Pause and test your reasoning
Practice 1 of 3
A gRPC connection is live, but store lag and decoder failures are rising. What status should the app show?
Reveal the answer and explanation
One best answer: Delayed or incomplete until index lag is repaired.
Transport connectivity does not prove complete indexing. Report source lag and decode failures, and do not move the durable cursor past events that have not been persisted and reconciled.
Sources: Solana Indexing Guide · Yellowstone gRPC Repository
Practice 2 of 3
A worker commits a payment event, then crashes before advancing its checkpoint. On restart, the previous cursor causes the same observation to be replayed. What should it do?
Reveal the answer and explanation
One best answer: Replay with the same unique event key, recognize an already durable row, and advance only after that event is durable.
The crash leaves the cursor behind, so the same observation may be delivered again. A stable unique event key lets replay recognize the durable payment rather than apply it twice; advance the cursor only after the event is durable and matched to its source evidence.
Sources: Solana Indexing Guide · Solana Validator Geyser Plugins
Practice 3 of 3
The historical decoder does not recognize an instruction variant. What should happen before parser upgrade activation?
Reveal the answer and explanation
One best answer: Quarantine; pin the parser, replay fixtures, reconcile, then activate.
A decoder gap is coverage loss. Preserve raw evidence and parser provenance, keep the gap visible, and validate a rebuilt view against historical signatures before changing production.
Sources: Solana Indexing Guide · Yellowstone gRPC Repository · Solana RPC JSON Structures
This practice is not scored, saved, or part of a certificate. Account-based lesson checkpoints and level exams remain separate.
Never enter a recovery phrase or private key into a learning site or lesson exercise.
READ THE SOURCE
Go deeper in the official docs
Reviewed on 2026-10-03. Protocol documentation can change; check the linked page’s current version as you use this material.
YOUR PROGRESS, SAVED
Ready for the checkpoint?
Make a free account to save your progress and unlock the quizzes.
Create free account