✳ STONKBUILDER LEARNING PASSPORT
← Solana Architect

Expert · LESSON 06 OF 07

6. Production operations and incident response

By the end, you can plan a verifiable solana release with controlled authorities, monitored dependencies, and recovery evidence.

Look up Solana terms in the glossary

In this lesson9 field notes · practice · sources · checkpoint
Release evidence connects source and artifact to authority checks, staging and simulation, monitoring, pause controls, and recovery.
Release evidence connects source and artifact to authority checks, staging and simulation, monitoring, pause controls, and recovery.Open full-size visual ↗
Read a text version of this diagram

Release evidence ties a reviewed source commit to the exact compiled artifact and checks program and collection authorities independently. The candidate is staged and simulated before an operator enables it, then monitored with explicit pause and recovery controls. A verifiable build links source to deployed bytes; it does not by itself prove safe code or correct authorities.

  1. SOURCE + ARTIFACT
  2. VERIFY AUTHORITY
  3. STAGE + SIMULATE
  4. MONITOR
  5. PAUSE + RECOVER
Four transaction-v1 production surfaces compare read failures, silent indexer misclassification, sponsor limit bypass, and unsupported builders or wallets with their release checks.
V1 transaction production failure map. Four transaction-v1 production surfaces compare read failures, silent indexer misclassification, sponsor limit bypass, and unsupported builders or wallets with their release checks.Open full-size visual ↗
Read a text version: V1 transaction production failure map

HTTP and WebSocket readers must opt in to transaction version 1; a null block with an error is a failed read. Geyser and indexer consumers must detect the v1 message config before the shared versioned flag and decode transactionConfig. Fee sponsors must enforce resource and fee caps from that signed config rather than scanning ComputeBudget instructions. Builders must set v1 limits and verify that the signing wallet supports the format, with a safe legacy or v0 fallback.

  1. HTTP READ + STREAM
  2. GEYSER + INDEXER
  3. FEE SPONSOR
  4. BUILDER + WALLET
The workflow checks cluster migration, buffers by slot and stream-local bank ID, drops abandoned parents, seals the confirmed bank, and reconciles providers by blockhash.
Alpenglow bank-aware indexer. The workflow checks cluster migration, buffers by slot and stream-local bank ID, drops abandoned parents, seals the confirmed bank, and reconciles providers by blockhash.Open full-size visual ↗
Read a text version: Alpenglow bank-aware indexer

Check each cluster independently with getAgGenesisCert: null means it has not migrated, a certificate object confirms migration, and Method not found means that RPC cannot answer the question. For each Geyser connection, buffer events by (slot, bank_id) because one slot can have multiple candidate banks. Discard banks named by UpdateParent, and seal only when expected event counts and block metadata reconcile. Keep the bank reaching Confirmed. Bank IDs are local to their stream, so reconcile sealed provider feeds by blockhash.

  1. DETECT EACH CLUSTER
  2. BUFFER SLOT + BANK
  3. DROP UPDATEPARENT BANKS
  4. SEAL CONFIRMED BLOCK
  5. MERGE PROVIDERS BY BLOCKHASH
A dated two-feature matrix separates the BLS and VAT admission prerequisites from Alpenglow consensus activation. The prerequisites are active across all three clusters; public RPC reads report TowerBFT on Mainnet and Alpenglow certificates on Devnet and Testnet.
Validator admission and Alpenglow activation by cluster. A dated two-feature matrix separates the BLS and VAT admission prerequisites from Alpenglow consensus activation. The prerequisites are active across all three clusters; public RPC reads report TowerBFT on Mainnet and Alpenglow certificates on Devnet and Testnet.Open full-size visual ↗
Read a text version: Validator admission and Alpenglow activation by cluster

The matrix separates validator admission prerequisites from consensus activation for a dated, read-only snapshot. Solana Foundation reports BLS public-key registration and VAT active on Mainnet, Devnet, and Testnet in July 2026; those gates do not activate Alpenglow. A compatible getAgGenesisCert response distinguishes each cluster: null means TowerBFT, while a certificate means migrated and identifies the first Alpenglow block slot. At 15:23–15:24 UTC on 2026-10-05, public RPCs returned null on Mainnet and certificates at slots 504148999 and 444625255 on Devnet and Testnet. Method not found indicates an RPC capability gap. Recheck the endpoint you operate; this snapshot is not a production guarantee.

  1. BLS + VAT PREREQUISITES
  2. ACTIVE · JUL 2026
  3. ACTIVE · JUL 2026
  4. ACTIVE · JUL 2026
  5. ALPENGLOW CONSENSUS
  6. NULL · TOWERBFT
  7. CERT · SLOT 504148999
  8. CERT · SLOT 444625255
A verified artifact uploads through a temporary buffer into ProgramData; the new version activates next slot. If deployment stops unexpectedly, reconcile signatures, buffer authority and ProgramData on the same cluster before resuming or closing anything.
Loader-v3 upgrades are not atomic. Reconcile the buffer and ProgramData before retrying cleanup.Open full-size visual ↗
FIELD NOTE 01

Define the release boundary

A release is a specific program build, client version, configuration, and cluster—not simply a green pipeline. Record the source revision, dependency lockfile, reproducible build steps, program ID, expected account layout, and the person who approves deployment. Separate local validator, Devnet, Testnet, and Mainnet evidence. Test SOL and mock services are useful for rehearsal, but they do not prove that production keys, endpoints, or code match the reviewed artifact.

FIELD NOTE 02

Verify program identity and authority

For a standard loader-v3 deployment, the stable Program address points to a ProgramData account containing executable bytes and deployment metadata. Inspect the cluster, loader, ProgramData address, deployed slot, and current upgrade authority, then compare the on-chain program with the exact reviewed source using a verifiable-build procedure. A loader-v3 upgrade changes the bytes and slot metadata in ProgramData while leaving the Program address and pointer in place; the new version becomes effective in the following slot. An upgrade authority can therefore change behavior without changing the familiar address. Setting the authority to None makes the program immutable and permanently removes that repair path. Bind each release report to its loader and upgradeability model, artifact hash, authority evidence, and verification result instead of treating a familiar program ID as proof of code identity.

FIELD NOTE 03

Control deployment and signing keys

Use separate keys for development, deployment, treasury, and application issuance where the design permits. Limit who can access a signer, what message it may sign, and which process can invoke it. Require independent review for high-impact transactions and rehearse rotation and pause procedures. Never place secret key material in a browser, source file, ordinary log, or public build artifact. Protect recovery material with a tested operational process rather than relying on a single operator’s memory.

FIELD NOTE 04

Budget compute, fees, and account storage

Production readiness includes transaction compute, account data size, rent-exempt deposits, base fees, and optional prioritization fees. Simulate representative transactions and leave a measured compute margin; an excessive limit can increase priority-fee cost when price is nonzero. Failed transactions can still incur fees. Explain who pays and what storage may be locked in accounts. For the academy passport, the learner is the transaction fee payer and the wallet should show current costs before approval; never present a fixed fee estimate as permanent.

FIELD NOTE 05

Operate RPC and indexer dependencies

RPC providers can rate-limit, lag, return incomplete historical data, or become unavailable. Solana's production guidance recommends a production-grade RPC provider for Mainnet payment flows because shared public endpoints are rate-limited and have no service-level agreement; configure an independent fallback and test how it behaves during a provider outage. Decide which reads need confirmed or finalized commitment, set timeouts and bounded retries, and reconcile high-impact actions against a separate source. Monitor transaction success rate, confirmation latency, priority-fee spend, RPC error rate, slot lag, and confirmation backlog; alert on failure spikes, unusual recipients, and provider errors. Keep provider-specific behavior out of protocol truth: a dashboard query is an observation, while independently checked finalized account or transaction state is stronger evidence for a chain result.

FIELD NOTE 06

Prepare incident actions before launch

Define how operators stop new writes, rotate compromised credentials, preserve logs, notify learners, and resume safely. Exercise a bad program release, leaked web credential, issuer-key exposure, metadata outage, RPC partition, and database restore. A pause control can stop future application actions only if the architecture enforces it; it cannot undo finalized transactions or rewrite an immutable asset. Record observed facts, affected versions and accounts, containment actions, and unresolved questions without guessing at cause.

FIELD NOTE 07

Make rollback and recovery evidence concrete

For each release, keep the prior artifact, migration plan, configuration, authority changes, and a tested recovery path. Loader-v3 deployment uploads new bytes through a temporary buffer before updating ProgramData, so the operation is not atomic; an interruption can leave a buffer and a partially completed release to reconcile. Before closing a suspected abandoned buffer, verify the exact cluster, account owner and buffer state, authority, relation to the target program, and whether the deployment can still be resumed. Closing the wrong buffer transfers its lamports and destroys the remaining recovery option. Record the buffer address, source and artifact hashes, transaction signatures, and last observed deployment state. Rollback may require a new reviewed program deployment rather than reverting chain history. Verify restored data and compare it with on-chain receipts before reopening operations. Assign an owner and evidence location to each launch gate.

FIELD NOTE 08

Check transaction v1 across every production boundary

Transaction v1 is live on Mainnet, Devnet, and Testnet, while legacy and v0 remain supported. Treat reading, streaming, indexing, building, signing, and fee sponsorship as separate capabilities. For HTTP `getTransaction` and `getBlock` reads, request the integer `maxSupportedTransactionVersion: 1`; `blockSubscribe` has the same version option. A block notification containing `block: null` and an error is a failed read, not an empty block.

An indexer that derives limits only by scanning ComputeBudget instructions will miss v1 configuration. Read the signed message’s `transactionConfig`. For Geyser/gRPC, classify `message.config` as v1 before checking the shared `versioned` flag; otherwise a parser may silently label the message v0 and discard resource data.

Fee sponsors must enforce their compute, loaded-account-data, and priority-fee limits against that v1 configuration. Normalize v0 priority fees (micro-lamports per CU multiplied by requested CUs) to total lamports before comparing them with v1’s total-lamport value. A sender must set v1’s compute and loaded-data limits explicitly because their defaults are zero. V1 places accounts inline instead of using address lookup tables.

Check builder SDK and wallet support before signing; use legacy or v0 if a required component cannot handle v1. If a decoder loses the v1 config or returns an unsupported-version error, treat it as an incident for that read path. Preserve the cluster, endpoint, software version, slot, signature, and raw bytes; stop basing indexing or sponsorship decisions on the incomplete record; then reconcile through a separately maintained compatible provider. Do not call the transaction absent or approve a sponsor action from missing fields. Verify parsers against raw fixtures from the official examples and a current RPC node.

FIELD NOTE 09

Prepare for Alpenglow's bank-aware streams

Alpenglow changes consensus, block production, streaming, and finality; transaction execution and fee mechanics remain unchanged. Gate Alpenglow-specific behavior on the cluster's `getAgGenesisCert` result, not a target date or validator version: `null` means that cluster is still on TowerBFT, a certificate means it has migrated, and `Method not found` means the RPC node is too old to answer. Check Mainnet, Devnet, and Testnet independently.

For Agave 4.3 Geyser or gRPC feeds, carry `bank_id` with each event and buffer by `(slot, bank_id)`. Fast leader handoff can produce multiple candidate banks for one slot; Agave 4.4 is expected to make that case more common. Seal a bank only when its block metadata counts and expected account updates reconcile, keep the bank that reaches `Confirmed`, and discard banks named by `UpdateParent`. `bank_id` is local to one stream connection: when merging providers, keep separate buffers and reconcile sealed data by blockhash, never by bank ID.

After a cluster migrates, vote transactions leave blocks, so re-baseline TPS and vote-count dashboards instead of labeling the lower transaction count a network slowdown. If a program or indexer relies on `Clock.unix_timestamp` as close to wall-clock time, review the new leader-set timestamp bounds; do not build a fixed drift guarantee into application logic. On a TowerBFT cluster, `confirmed` and `finalized` remain distinct. After that cluster migrates, Solana documents them as converging and plans to remove `confirmed`; use `finalized` in new code. Check each cluster with a compatible `getAgGenesisCert` RPC and test per-cluster fixtures instead of assuming the migration date or state. Test the old and new formats with per-cluster fixtures, bank-switch and provider-merge cases, and explicit unknown-method handling.

MAKE THE CONNECTION

A detail worth keeping.

Rehearse a program upgrade from a clean checkout on a local validator. Capture the artifact hash, program address, deployment result, upgrade authority, test evidence, and a failed-deployment recovery. Repeat the exercise with an RPC outage and explain which state is authoritative.

Work through an example

Draft a response for an exposed deployment key. Include immediate containment, signer rotation, deployed-program inspection, user communication, transaction monitoring, and evidence preservation. Separate actions that stop new transactions from actions that can change existing on-chain state.

Illustrative v1 JSON RPC message fragment. Surrounding transaction fields are omitted; the example shows why instruction-only fee inspection is incomplete.
{
  "version": 1,
  "message": {
    "instructions": [],
    "transactionConfig": {
      "computeUnitLimit": 30000,
      "loadedAccountsDataSizeLimit": 200000,
      "priorityFee": 5000
    }
  }
}

Audit the transaction-version path for an application that reads blocks, streams transactions, indexes resource use, and optionally sponsors fees. Write a compatibility matrix for `getTransaction`, `getBlock`, `blockSubscribe`, Geyser/gRPC, the builder SDK, wallet, and sponsor. For each surface record the version opt-in, decoder behavior, required configuration, failure signal, supported dependency version, and safe fallback. Use official v1 fixtures: prove the parser checks `message.config` before `versioned`, reads `transactionConfig`, and normalizes a v0 fee before applying the same sponsor cap to v1. Test a v1 message with no ComputeBudget instructions and a missing or zero limit; it must fail closed. Run against a local validator or mocked provider with no production keys or funds, and label that evidence as local.

GUIDED PRACTICE · EXPERT

Do the work, one step at a time.

Use fictional addresses and disposable test data. Do not enter a recovery phrase or private key. Most exercises work on paper or with a local test; only a lesson that clearly says “read-only Devnet” makes a public network request.

  1. Pin the official transaction-v1 example revision and choose the documented HTTP, WebSocket, and Geyser response fixtures for the parser under review.
  2. Add integer version opt-in to getTransaction, getBlock, and blockSubscribe; test an unsupported-version response and a block:null notification carrying an error as failures, not empty results.
  3. Write a format detector that checks message.config before the versioned flag, then assert that v1 transactionConfig survives decoding and legacy/v0 remain distinguishable.
  4. Test the fee-sponsor rule against a v1 fixture with no ComputeBudget instruction; enforce compute, loaded-account-data, and total priority-fee caps from transactionConfig, and normalize v0 fee units for comparison.
  5. Check SDK and wallet capabilities, simulate an unsupported signer and an omitted v1 resource limit, then document the v0 fallback and provider cross-check. Record fixture hashes and test output; do not use production keys or spend SOL.
  6. For Alpenglow readiness, test `getAgGenesisCert` as null, certificate, and unsupported-method outcomes; buffer bank-aware stream fixtures by `(slot, bank_id)`, process an UpdateParent discard, and merge two provider feeds by sealed blockhash rather than their local bank IDs.

Before you move on, check your reasoning

  • Bind one release to source, artifact, cluster, and authority evidence.
  • Rehearse deployment interruption and recovery in an isolated environment.
  • Run an incident drill and record observed response times and open risks.

OPEN SELF-CHECK · UNGRADED

Pause and test your reasoning

Practice 1 of 5

A production deployment transaction confirms, but the deployed program-data account reports an unexpected upgrade authority. Is the release ready?

  1. Yes; a confirmed transaction proves the intended binary and authority policy are correct.
  2. No; verify the deployed program, binary provenance, and expected authority state, then stop or roll back under the release procedure if they differ.
  3. Yes, if the local build passed its unit tests.
Reveal the answer and explanation

One best answer: No; verify the deployed program, binary provenance, and expected authority state, then stop or roll back under the release procedure if they differ.

Confirmation proves that a transaction landed, not that the intended artifact or authority policy is present. Compare the deployed program and program-data authority to reviewed release evidence, and use the documented response when production differs.

Sources: Program Deployment · Production Readiness

Practice 2 of 5

Two Geyser providers report the same bank ID for one slot. What should an indexer do before merging their data?

  1. Treat the equal slot and numeric bank ID as enough evidence that these provider records describe the same cluster-wide bank.
  2. Merge provider buffers once both streams label a bank at the same slot and report completion, even if their block hashes differ.
  3. Keep each connection separate, then reconcile sealed results by blockhash.
Reveal the answer and explanation

One best answer: Keep each connection separate, then reconcile sealed results by blockhash.

Each validator assigns bank IDs locally, so the same number across connections can identify different banks. Reconcile independently sealed data by blockhash and never merge the streams into a shared bank-ID namespace.

Sources: Alpenglow Migration Guide

Practice 3 of 5

An indexer starts reading transaction v1 blocks. Which compatibility evidence belongs in its release test?

  1. Test v1 decoding, configuration, and raw bytes.
  2. Legacy parsing alone is enough.
  3. A developer laptop parsing old transactions proves opt-in v1 decoding and normalized fields match the original message bytes.
Reveal the answer and explanation

One best answer: Test v1 decoding, configuration, and raw bytes.

Versioned transaction support spans request configuration and decoders. A migration test should detect silent misclassification by comparing normalized fields with the exact v1 bytes. Verify the result against a second provider or raw fixture.

Sources: Larger Transaction Sizes · RPC JSON Structures · getTransaction

Practice 4 of 5

A compatible `getAgGenesisCert` call returns a certificate for one cluster. What should an indexer change first?

  1. Use `finalized` on all clusters because one certificate proves network-wide activation.
  2. Keep `confirmed` on the migrated cluster because `finalized` retains TowerBFT's slower finality.
  3. Use `finalized` for that cluster; confirm migration independently for each other cluster.
Reveal the answer and explanation

One best answer: Use `finalized` for that cluster; confirm migration independently for each other cluster.

On a migrated cluster, `confirmed` and `finalized` converge, so Solana recommends `finalized` for new reads. Each cluster migrates independently; the certificate applies only to the endpoint that returned it.

Sources: Alpenglow Migration Guide · getAgGenesisCert

Practice 5 of 5

Mainnet has BLS-key registration and VAT admission active, but a compatible `getAgGenesisCert` call returns `null`. What does this establish?

  1. The gates prepare validators; `null` means Mainnet still runs TowerBFT.
  2. Alpenglow is active because BLS and VAT admit validators, even though this compatible RPC has not returned the genesis certificate yet.
  3. `Method not found` means the cluster has migrated.
Reveal the answer and explanation

One best answer: The gates prepare validators; `null` means Mainnet still runs TowerBFT.

BLS registration and VAT prepare the validator set but do not activate Alpenglow. On a compatible RPC, `null` means not migrated; `Method not found` means that node cannot answer.

Sources: BLS Pubkey and the Validator Admission Ticket · Alpenglow Migration Guide · getAgGenesisCert

This practice is not scored, saved, or part of a certificate. Account-based lesson checkpoints and level exams remain separate.

Never enter a recovery phrase or private key into a learning site or lesson exercise.

READ THE SOURCE

Go deeper in the official docs

Reviewed on 2026-10-04. Protocol documentation can change; check the linked page’s current version as you use this material.

  1. Deploying Programs ↗
  2. Program Deployment ↗
  3. Solana CLI Reference ↗
  4. Production Readiness ↗
  5. Transaction Fees ↗
  6. Compute Budget ↗
  7. Programs ↗
  8. Larger Transaction Sizes ↗
  9. RPC JSON Structures ↗
  10. getTransaction ↗
  11. getBlock ↗
  12. blockSubscribe ↗
  13. Alpenglow Migration Guide ↗
  14. getAgGenesisCert ↗
  15. BLS Pubkey and the Validator Admission Ticket ↗

Report an issue with this lesson

YOUR PROGRESS, SAVED

Ready for the checkpoint?

Make a free account to save your progress and unlock the quizzes.

Create free account