Expert · LESSON 05 OF 06
5. Production operations and incident response
By the end, you can plan a verifiable solana release with controlled authorities, monitored dependencies, and recovery evidence.
Optional reading history is saved only in this browser. It is not a scored checkpoint or exam result.
In this lesson8 field notes · practice · sources · checkpoint
Read a text version of this diagram
Release evidence ties a reviewed source commit to the exact compiled artifact and checks program and collection authorities independently. The candidate is staged and simulated before an operator enables it, then monitored with explicit pause and recovery controls. A verifiable build links source to deployed bytes; it does not by itself prove safe code or correct authorities.
- SOURCE + ARTIFACT
- VERIFY AUTHORITY
- STAGE + SIMULATE
- MONITOR
- PAUSE + RECOVER
Read a text version: V1 transaction production failure map
HTTP and WebSocket readers must opt in to transaction version 1; a null block with an error is a failed read. Geyser and indexer consumers must detect the v1 message config before the shared versioned flag and decode transactionConfig. Fee sponsors must enforce resource and fee caps from that signed config rather than scanning ComputeBudget instructions. Builders must set v1 limits and verify that the signing wallet supports the format, with a safe legacy or v0 fallback.
- HTTP READ + STREAM
- GEYSER + INDEXER
- FEE SPONSOR
- BUILDER + WALLET
Define the release boundary
A release is a specific program build, client version, configuration, and cluster—not simply a green pipeline. Record the source revision, dependency lockfile, reproducible build steps, program ID, expected account layout, and the person who approves deployment. Separate local validator, Devnet, Testnet, and Mainnet evidence. Test SOL and mock services are useful for rehearsal, but they do not prove that production keys, endpoints, or code match the reviewed artifact.
Verify program identity and authority
For a standard loader-v3 deployment, the stable Program address points to a ProgramData account containing executable bytes and deployment metadata. Inspect the cluster, loader, ProgramData address, deployed slot, and current upgrade authority, then compare the on-chain program with the exact reviewed source using a verifiable-build procedure. A loader-v3 upgrade changes the bytes and slot metadata in ProgramData while leaving the Program address and pointer in place; the new version becomes effective in the following slot. An upgrade authority can therefore change behavior without changing the familiar address. Setting the authority to None makes the program immutable and permanently removes that repair path. Bind each release report to its loader and upgradeability model, artifact hash, authority evidence, and verification result instead of treating a familiar program ID as proof of code identity.
Control deployment and signing keys
Use separate keys for development, deployment, treasury, and application issuance where the design permits. Limit who can access a signer, what message it may sign, and which process can invoke it. Require independent review for high-impact transactions and rehearse rotation and pause procedures. Never place secret key material in a browser, source file, ordinary log, or public build artifact. Protect recovery material with a tested operational process rather than relying on a single operator’s memory.
Budget compute, fees, and account storage
Production readiness includes transaction compute, account data size, rent-exempt deposits, base fees, and optional prioritization fees. Simulate representative transactions and leave a measured compute margin; an excessive limit can increase priority-fee cost when price is nonzero. Failed transactions can still incur fees. Explain who pays and what storage may be locked in accounts. For the academy passport, the learner is the transaction fee payer and the wallet should show current costs before approval; never present a fixed fee estimate as permanent.
Operate RPC and indexer dependencies
RPC providers can rate-limit, lag, return incomplete historical data, or become unavailable. Solana's production guidance recommends a production-grade RPC provider for Mainnet payment flows because shared public endpoints are rate-limited and have no service-level agreement; configure an independent fallback and test how it behaves during a provider outage. Decide which reads need confirmed or finalized commitment, set timeouts and bounded retries, and reconcile high-impact actions against a separate source. Monitor transaction success rate, confirmation latency, priority-fee spend, RPC error rate, slot lag, and confirmation backlog; alert on failure spikes, unusual recipients, and provider errors. Keep provider-specific behavior out of protocol truth: a dashboard query is an observation, while independently checked finalized account or transaction state is stronger evidence for a chain result.
Prepare incident actions before launch
Define how operators stop new writes, rotate compromised credentials, preserve logs, notify learners, and resume safely. Exercise a bad program release, leaked web credential, issuer-key exposure, metadata outage, RPC partition, and database restore. A pause control can stop future application actions only if the architecture enforces it; it cannot undo finalized transactions or rewrite an immutable asset. Record observed facts, affected versions and accounts, containment actions, and unresolved questions without guessing at cause.
Make rollback and recovery evidence concrete
For each release, keep the prior artifact, migration plan, configuration, authority changes, and a tested recovery path. Loader-v3 deployment uploads new bytes through a temporary buffer before updating ProgramData, so the operation is not atomic; an interruption can leave a buffer and a partially completed release to reconcile. Before closing a suspected abandoned buffer, verify the exact cluster, account owner and buffer state, authority, relation to the target program, and whether the deployment can still be resumed. Closing the wrong buffer transfers its lamports and destroys the remaining recovery option. Record the buffer address, source and artifact hashes, transaction signatures, and last observed deployment state. Rollback may require a new reviewed program deployment rather than reverting chain history. Verify restored data and compare it with on-chain receipts before reopening operations. Assign an owner and evidence location to each launch gate.
Check transaction v1 across every production boundary
Transaction v1 is live on Mainnet, Devnet, and Testnet, while legacy and v0 remain supported. Treat reading, streaming, indexing, building, signing, and fee sponsorship as separate capabilities. For HTTP `getTransaction` and `getBlock` reads, request the integer `maxSupportedTransactionVersion: 1`; `blockSubscribe` has the same version option. A block notification containing `block: null` and an error is a failed read, not an empty block. An indexer that derives limits only by scanning ComputeBudget instructions will miss v1 configuration. Read the signed message’s `transactionConfig`. For Geyser/gRPC, classify `message.config` as v1 before checking the shared `versioned` flag; otherwise a parser may silently label the message v0 and discard resource data. Fee sponsors must enforce their compute, loaded-account-data, and priority-fee limits against that v1 configuration. Normalize v0 priority fees (micro-lamports per CU multiplied by requested CUs) to total lamports before comparing them with v1’s total-lamport value. A sender must set v1’s compute and loaded-data limits explicitly because their defaults are zero. It also places accounts inline instead of using address lookup tables. Check builder SDK and wallet support before signing; use legacy or v0 if a required component cannot handle v1. When a decoder loses the v1 config or returns an unsupported-version error, treat it as an incident for that read path: preserve the cluster, endpoint, software version, slot, signature, and raw bytes; stop basing indexing or sponsorship decisions on the incomplete record; then reconcile through a separately maintained compatible provider. Do not call the transaction absent or approve a sponsor action from missing fields. Verify parsers against raw fixtures from the official examples and a current RPC node.
A detail worth keeping.
Rehearse a program upgrade from a clean checkout on a local validator. Capture the artifact hash, program address, deployment result, upgrade authority, test evidence, and a failed-deployment recovery. Repeat the exercise with an RPC outage and explain which state is authoritative.
Work through an example
Draft a response for an exposed deployment key. Include immediate containment, signer rotation, deployed-program inspection, user communication, transaction monitoring, and evidence preservation. Separate actions that stop new transactions from actions that can change existing on-chain state.
{
"version": 1,
"message": {
"instructions": [],
"transactionConfig": {
"computeUnitLimit": 30000,
"loadedAccountsDataSizeLimit": 200000,
"priorityFee": 5000
}
}
}Audit the transaction-version path for an application that reads blocks, streams transactions, indexes resource use, and optionally sponsors fees. Write a compatibility matrix for `getTransaction`, `getBlock`, `blockSubscribe`, Geyser/gRPC, the builder SDK, wallet, and sponsor. For each surface record the version opt-in, decoder behavior, required configuration, failure signal, supported dependency version, and safe fallback. Use official v1 fixtures: prove the parser checks `message.config` before `versioned`, reads `transactionConfig`, and normalizes a v0 fee before applying the same sponsor cap to v1. Test a v1 message with no ComputeBudget instructions and a missing or zero limit; it must fail closed. Run against a local validator or mocked provider with no production keys or funds, and label that evidence as local.
GUIDED PRACTICE · EXPERT
Do the work, one step at a time.
Use fictional addresses and disposable test data. Do not enter a recovery phrase or private key. Most exercises work on paper or with a local test; only a lesson that clearly says “read-only Devnet” makes a public network request.
- Pin the official transaction-v1 example revision and choose the documented HTTP, WebSocket, and Geyser response fixtures for the parser under review.
- Add integer version opt-in to getTransaction, getBlock, and blockSubscribe; test an unsupported-version response and a block:null notification carrying an error as failures, not empty results.
- Write a format detector that checks message.config before the versioned flag, then assert that v1 transactionConfig survives decoding and legacy/v0 remain distinguishable.
- Test the fee-sponsor rule against a v1 fixture with no ComputeBudget instruction; enforce compute, loaded-account-data, and total priority-fee caps from transactionConfig, and normalize v0 fee units for comparison.
- Check SDK and wallet capabilities, simulate an unsupported signer and an omitted v1 resource limit, then document the v0 fallback and provider cross-check. Record fixture hashes and test output; do not use production keys or spend SOL.
Before you move on, check your reasoning
- Bind one release to source, artifact, cluster, and authority evidence.
- Rehearse deployment interruption and recovery in an isolated environment.
- Run an incident drill and record observed response times and open risks.
OPEN SELF-CHECK · UNGRADED
Pause and test your reasoning
A production deployment transaction confirms, but the deployed program-data account reports an unexpected upgrade authority. Is the release ready?
Reveal one best answer and its explanation
One best answer: No; verify the deployed program, binary provenance, and expected authority state, then stop or roll back under the release procedure if they differ.
Confirmation proves that a transaction landed, not that the intended artifact or authority policy is present. Compare the deployed program and program-data authority to reviewed release evidence, and use the documented response when production differs.
Sources: Program Deployment · Production Readiness
This practice is not scored, saved, or part of a certificate. Account-based lesson checkpoints and level exams remain separate.
Never enter a recovery phrase or private key into a learning site or lesson exercise.
READ THE SOURCE
Go deeper in the official docs
Reviewed on 2026-09-29. Protocol documentation can change; check the linked page’s current version as you use this material.
OPEN LEARNING
Keep learning at your own pace.
This lesson, its guided practice, and all six course lessons are open to everyone. Account-based checkpoints, exams, and saved progress will open when free accounts are ready.
Continue to the next lesson ↗