✳ SOLANA ACADEMYCourse overview

Expert · LESSON 05 OF 06

5. Production operations and incident response

By the end, you can plan a verifiable solana release with controlled authorities, monitored dependencies, and recovery evidence.

Release evidence connects source and artifact to authority checks, staging and simulation, monitoring, pause controls, and recovery.
5. Production operations and incident response workflow. Release evidence connects source and artifact to authority checks, staging and simulation, monitoring, pause controls, and recovery.
A verified program artifact is uploaded to a temporary buffer and then applied to ProgramData. The Program address stays stable and the new version is effective in the next slot. If deployment is interrupted, stop and reconcile the cluster, signatures, buffer state and authority, and ProgramData before resuming or closing anything.
Loader-v3 upgrade and interrupted deployment recovery. A verified program artifact is uploaded to a temporary buffer and then applied to ProgramData. The Program address stays stable and the new version is effective in the next slot. If deployment is interrupted, stop and reconcile the cluster, signatures, buffer state and authority, and ProgramData before resuming or closing anything.
Four transaction-v1 production surfaces compare read failures, silent indexer misclassification, sponsor limit bypass, and unsupported builders or wallets with their release checks.
V1 transaction production failure map. Four transaction-v1 production surfaces compare read failures, silent indexer misclassification, sponsor limit bypass, and unsupported builders or wallets with their release checks.

Read the workflow

Release evidence ties a reviewed source commit to the exact compiled artifact and checks program and collection authorities independently. The candidate is staged and simulated before an operator enables it, then monitored with explicit pause and recovery controls. A verifiable build links source to deployed bytes; it does not by itself prove safe code or correct authorities.

  1. SOURCE + ARTIFACT
  2. VERIFY AUTHORITY
  3. STAGE + SIMULATE
  4. MONITOR
  5. PAUSE + RECOVER

Read a text version: V1 transaction production failure map

HTTP and WebSocket readers must opt in to transaction version 1; a null block with an error is a failed read. Geyser and indexer consumers must detect the v1 message config before the shared versioned flag and decode transactionConfig. Fee sponsors must enforce resource and fee caps from that signed config rather than scanning ComputeBudget instructions. Builders must set v1 limits and verify that the signing wallet supports the format, with a safe legacy or v0 fallback.

  1. HTTP READ + STREAM
  2. GEYSER + INDEXER
  3. FEE SPONSOR
  4. BUILDER + WALLET

FIELD NOTE 01

Define the release boundary

A release is a specific program build, client version, configuration, and cluster—not simply a green pipeline. Record the source revision, dependency lockfile, reproducible build steps, program ID, expected account layout, and the person who approves deployment. Separate local validator, Devnet, Testnet, and Mainnet evidence. Test SOL and mock services are useful for rehearsal, but they do not prove that production keys, endpoints, or code match the reviewed artifact.

FIELD NOTE 02

Verify program identity and authority

For a standard loader-v3 deployment, the stable Program address points to a ProgramData account containing executable bytes and deployment metadata. Inspect the cluster, loader, ProgramData address, deployed slot, and current upgrade authority, then compare the on-chain program with the exact reviewed source using a verifiable-build procedure. A loader-v3 upgrade changes the bytes and slot metadata in ProgramData while leaving the Program address and pointer in place; the new version becomes effective in the following slot. An upgrade authority can therefore change behavior without changing the familiar address. Setting the authority to None makes the program immutable and permanently removes that repair path. Bind each release report to its loader and upgradeability model, artifact hash, authority evidence, and verification result instead of treating a familiar program ID as proof of code identity.

FIELD NOTE 03

Control deployment and signing keys

Use separate keys for development, deployment, treasury, and application issuance where the design permits. Limit who can access a signer, what message it may sign, and which process can invoke it. Require independent review for high-impact transactions and rehearse rotation and pause procedures. Never place secret key material in a browser, source file, ordinary log, or public build artifact. Protect recovery material with a tested operational process rather than relying on a single operator’s memory.

FIELD NOTE 04

Budget compute, fees, and account storage

Production readiness includes transaction compute, account data size, rent-exempt deposits, base fees, and optional prioritization fees. Simulate representative transactions and leave a measured compute margin; an excessive limit can increase priority-fee cost when price is nonzero. Failed transactions can still incur fees. Explain who pays and what storage may be locked in accounts. For the academy passport, the learner is the transaction fee payer and the wallet should show current costs before approval; never present a fixed fee estimate as permanent.

FIELD NOTE 05

Operate RPC and indexer dependencies

RPC providers can rate-limit, lag, return incomplete historical data, or become unavailable. Solana's production guidance recommends a production-grade RPC provider for Mainnet payment flows because shared public endpoints are rate-limited and have no service-level agreement; configure an independent fallback and test how it behaves during a provider outage. Decide which reads need confirmed or finalized commitment, set timeouts and bounded retries, and reconcile high-impact actions against a separate source. Monitor transaction success rate, confirmation latency, priority-fee spend, RPC error rate, slot lag, and confirmation backlog; alert on failure spikes, unusual recipients, and provider errors. Keep provider-specific behavior out of protocol truth: a dashboard query is an observation, while independently checked finalized account or transaction state is stronger evidence for a chain result.

FIELD NOTE 06

Prepare incident actions before launch

Define how operators stop new writes, rotate compromised credentials, preserve logs, notify learners, and resume safely. Exercise a bad program release, leaked web credential, issuer-key exposure, metadata outage, RPC partition, and database restore. A pause control can stop future application actions only if the architecture enforces it; it cannot undo finalized transactions or rewrite an immutable asset. Record observed facts, affected versions and accounts, containment actions, and unresolved questions without guessing at cause.

FIELD NOTE 07

Make rollback and recovery evidence concrete

For each release, keep the prior artifact, migration plan, configuration, authority changes, and a tested recovery path. Loader-v3 deployment uploads new bytes through a temporary buffer before updating ProgramData, so the operation is not atomic; an interruption can leave a buffer and a partially completed release to reconcile. Before closing a suspected abandoned buffer, verify the exact cluster, account owner and buffer state, authority, relation to the target program, and whether the deployment can still be resumed. Closing the wrong buffer transfers its lamports and destroys the remaining recovery option. Record the buffer address, source and artifact hashes, transaction signatures, and last observed deployment state. Rollback may require a new reviewed program deployment rather than reverting chain history. Verify restored data and compare it with on-chain receipts before reopening operations. Assign an owner and evidence location to each launch gate.

FIELD NOTE 08

Check transaction v1 across every production boundary

Transaction v1 is live on Mainnet, Devnet, and Testnet, while legacy and v0 remain supported. Treat reading, streaming, indexing, building, signing, and fee sponsorship as separate capabilities. For HTTP `getTransaction` and `getBlock` reads, request the integer `maxSupportedTransactionVersion: 1`; `blockSubscribe` has the same version option. A block notification containing `block: null` and an error is a failed read, not an empty block. An indexer that derives limits only by scanning ComputeBudget instructions will miss v1 configuration. Read the signed message’s `transactionConfig`. For Geyser/gRPC, classify `message.config` as v1 before checking the shared `versioned` flag; otherwise a parser may silently label the message v0 and discard resource data. Fee sponsors must enforce their compute, loaded-account-data, and priority-fee limits against that v1 configuration. Normalize v0 priority fees (micro-lamports per CU multiplied by requested CUs) to total lamports before comparing them with v1’s total-lamport value. A sender must set v1’s compute and loaded-data limits explicitly because their defaults are zero. It also places accounts inline instead of using address lookup tables. Check builder SDK and wallet support before signing; use legacy or v0 if a required component cannot handle v1. When a decoder loses the v1 config or returns an unsupported-version error, treat it as an incident for that read path: preserve the cluster, endpoint, software version, slot, signature, and raw bytes; stop basing indexing or sponsorship decisions on the incomplete record; then reconcile through a separately maintained compatible provider. Do not call the transaction absent or approve a sponsor action from missing fields. Verify parsers against raw fixtures from the official examples and a current RPC node.

MAKE THE CONNECTION

A detail worth keeping

Rehearse a program upgrade from a clean checkout on a local validator. Capture the artifact hash, program address, deployment result, upgrade authority, test evidence, and a failed-deployment recovery. Repeat the exercise with an RPC outage and explain which state is authoritative.

Work through an example

Draft a response for an exposed deployment key. Include immediate containment, signer rotation, deployed-program inspection, user communication, transaction monitoring, and evidence preservation. Separate actions that stop new transactions from actions that can change existing on-chain state.

Illustrative v1 JSON RPC message fragment. Surrounding transaction fields are omitted; the example shows why instruction-only fee inspection is incomplete.
{
  "version": 1,
  "message": {
    "instructions": [],
    "transactionConfig": {
      "computeUnitLimit": 30000,
      "loadedAccountsDataSizeLimit": 200000,
      "priorityFee": 5000
    }
  }
}

Audit the transaction-version path for an application that reads blocks, streams transactions, indexes resource use, and optionally sponsors fees. Write a compatibility matrix for `getTransaction`, `getBlock`, `blockSubscribe`, Geyser/gRPC, the builder SDK, wallet, and sponsor. For each surface record the version opt-in, decoder behavior, required configuration, failure signal, supported dependency version, and safe fallback. Use official v1 fixtures: prove the parser checks `message.config` before `versioned`, reads `transactionConfig`, and normalizes a v0 fee before applying the same sponsor cap to v1. Test a v1 message with no ComputeBudget instructions and a missing or zero limit; it must fail closed. Run against a local validator or mocked provider with no production keys or funds, and label that evidence as local.

GUIDED PRACTICE · EXPERT

Do the work, one step at a time

Use fictional addresses and disposable test data. Do not enter a recovery phrase or private key. Unless a lesson explicitly says “read-only Devnet,” use paper or a local test and do not connect to a public network.

  1. Pin the official transaction-v1 example revision and choose the documented HTTP, WebSocket, and Geyser response fixtures for the parser under review.
  2. Add integer version opt-in to getTransaction, getBlock, and blockSubscribe; test an unsupported-version response and a block:null notification carrying an error as failures, not empty results.
  3. Write a format detector that checks message.config before the versioned flag, then assert that v1 transactionConfig survives decoding and legacy/v0 remain distinguishable.
  4. Test the fee-sponsor rule against a v1 fixture with no ComputeBudget instruction; enforce compute, loaded-account-data, and total priority-fee caps from transactionConfig, and normalize v0 fee units for comparison.
  5. Check SDK and wallet capabilities, simulate an unsupported signer and an omitted v1 resource limit, then document the v0 fallback and provider cross-check. Record fixture hashes and test output; do not use production keys or spend SOL.

Before you move on, check your reasoning

  • Bind one release to source, artifact, cluster, and authority evidence.
  • Rehearse deployment interruption and recovery in an isolated environment.
  • Run an incident drill and record observed response times and open risks.

OPEN SELF-CHECK · UNGRADED

Pause and test your reasoning

A production deployment transaction confirms, but the deployed program-data account reports an unexpected upgrade authority. Is the release ready?

  1. Yes; a confirmed transaction proves the intended binary and authority policy are correct.
  2. No; verify the deployed program, binary provenance, and expected authority state, then stop or roll back under the release procedure if they differ.
  3. Yes, if the local build passed its unit tests.
Reveal one best answer and its explanation

One best answer: No; verify the deployed program, binary provenance, and expected authority state, then stop or roll back under the release procedure if they differ.

Confirmation proves that a transaction landed, not that the intended artifact or authority policy is present. Compare the deployed program and program-data authority to reviewed release evidence, and use the documented response when production differs.

Sources: Program Deployment ↗ · Production Readiness ↗

This practice is not scored, saved, or part of a certificate. Account-based lesson checkpoints and level exams remain separate.

Scored checkpoints and level exams are online and require a free account. They are not included in this pack.

READ THE SOURCE

Go deeper in the official docs

Reviewed on 2026-09-29. Protocol documentation can change; check the linked page’s current version when you reconnect.