How to Roll Back Tyk Components in a Multi-Data Center (MDCB) Architecture – Tyk

Overview
This guide outlines the recommended sequence and best practices for rolling back Tyk components in an MDCB (Multi-Data Center Bridge) distributed architecture such as reverting a patch or minor release (e.g., from v5.13.4 to v5.13.1).

Following this procedure ensures zero downtime for live API consumer traffic and avoids configuration or protocol mismatches between Control Plane and Data Plane components.

Architecture Overview
In a standard Tyk MDCB deployment:
• Control Plane (Management Cluster): Runs the Tyk Dashboard, Control Plane Management Gateway, Control Plane Pump, MDCB (tyk-sink), Tyk Operator, local Redis, and the persistent configuration database (MongoDB or PostgreSQL).
• Data Planes (Worker / Edge Clusters): Run Tyk Gateways, Data Plane Pumps, and local Redis. (Note: The Tyk Dashboard is never deployed in Data Planes).

Golden Rules for Rollback
1. Reverse the Upgrade Order: While upgrades proceed Control Plane first followed by Data Planes, rollbacks must follow the exact reverse order: Data Planes first, Control Plane second.
2. Never Downgrade Control Plane Below Data Planes: A Data Plane Gateway should never run a newer version than the Control Plane/MDCB components it communicates with.
3. Do NOT Restore or Flush Redis: Redis stores transient, active runtime state (API keys, session tokens, quota and rate-limit counters). Reverting Redis wipes active sessions and resets rate limits.
4. Live Traffic is Preserved: Running Gateways hold API definitions and security policies in memory. Temporary Control Plane or MDCB downtime during the rollback maintenance window will not impact live traffic.

Step-by-Step Rollback Sequence:

Step 1: Pause or Scale Down Tyk Operator
If you manage APIs or policies via Kubernetes CRDs and the Tyk Operator:
• Scale down the Tyk Operator deployment (replicas: 0) or pause reconciliation.
• Why: This prevents the operator from triggering reloads or reconciling state against the Control Plane while components are mid-rollback.

Step 2: Roll Back Data Plane Gateways & Custom Go Plugins
• Custom Go Plugins: If custom Go plugins were recompiled for the newer version, deploy the plugin binaries compiled for your target rollback version first.
• Gateway Pods/Nodes: Perform a rolling update (e.g., Kubernetes maxSurge: 1, maxUnavailable: 0) of Data Plane Gateway instances cluster-by-cluster.
• Existing instances will continue serving live consumer traffic while new instances initialize, authenticate with MDCB, and load API definitions.

Step 3: Roll Back Data Plane Pump
• Revert the Data Plane Tyk Pump deployments to your target version.
• Data Plane Pump will continue draining analytics records from local Redis and streaming them to MDCB.

Step 4: Roll Back Control Plane Gateway (Management Gateway)
• Roll back the Control Plane Gateway instance that services Dashboard operations, license validation, and internal API calls.

Step 5: Roll Back Tyk Dashboard & Database Handling
1. Scale down or stop the Tyk Dashboard.
2. Database Considerations (MongoDB / PostgreSQL):
• Patch Reversals (e.g., v5.13.4 to v5.13.1): Patch releases within the same LTS minor release adhere to semantic versioning and do not introduce breaking database schema changes. If no new configurations or schemas were introduced during the upgrade window, a database restoration is generally not required.
• When to Restore Database Backups: Tyk Dashboard executes automated schema migrations forward on startup, but does not perform downward/reverse migrations. If you added new OAS APIs, updated policies, or need to guarantee an identical pre-upgrade state, restore your pre-upgrade database backup (via mongorestore or pg_restore) while the Dashboard is stopped.

3. Update the Tyk Dashboard container/package to the target rollback version and start the service.
4. Verify Dashboard health, login access, and API visibility.

Step 6: Roll Back Control Plane Pump
• Revert the Control Plane Tyk Pump deployment to the target version.
• Verify it successfully drains analytics queues from Redis into your target storage sink (MongoDB, PostgreSQL, or external BI sink).

Step 7: Roll Back MDCB (tyk-sink)
• Revert the MDCB component to the target version compatible with your rolled-back stack.
• Once running, Data Plane Gateways will maintain their gorpc long-polling connections and receive real-time hot-reloads.

Step 8: Re-enable / Roll Back Tyk Operator
• If the Tyk Operator itself was upgraded, revert the Operator controller deployment to the compatible version.
• Scale the Operator deployment back up to re-enable reconciliation.
• Note: Tyk Operator CRDs are backward-compatible and typically do not require modification.

State & Storage Handling

Why Redis Should NOT Be Restored
• Active Session Tokens: Redis holds transient session objects, bearer tokens, and rate-limit/quota counters.
• Token Invalidation: Restoring an earlier snapshot of Redis will invalidate valid active tokens and reset quota counters for your API consumers.
• Schema Safety: Lower patch versions of Tyk Gateway read session objects written by higher patch versions without issue, safely ignoring unknown fields.

Control Plane Database Best Practices
• Always take a full snapshot of your configuration database (MongoDB / PostgreSQL) immediately before any upgrade or maintenance.
• If a database restore is required during rollback, always ensure the Tyk Dashboard is fully stopped before initiating the restore to prevent concurrent write conflicts.

Zero-Downtime Resilience & High Availability (FAQ)

Will live API traffic drop while the Control Plane or MDCB is offline?
No. The Tyk Gateway architecture is designed for network partition tolerance and resilience:
1. In-Memory Configuration: Gateways hold all loaded API definitions, security policies, and certificates in memory (*APISpec). Existing instances continue proxying traffic without depending on the Control Plane.
2. Emergency Mode: If a Data Plane Gateway restarts while MDCB is unreachable, it automatically enters Emergency Mode and loads encrypted backups of API definitions cached in its local Redis.
3. Automatic Reconnection: When MDCB and the Dashboard are back online, Gateways automatically re-establish their RPC connection and sync any pending updates without requiring manual intervention.

Post-Rollback Verification Checklist
• Gateway Health: Verify all Gateway nodes return 200 OK on /hello.
• MDCB Connection: Check Data Plane Gateway logs for successful RPC connection (RPC connected).
• Live Traffic: Test end-to-end API requests against an active endpoint with authentication and rate limiting.
• Dashboard Access: Log into the Tyk Dashboard and confirm APIs, policies, and keys display as expected.
• Analytics Flow: Verify analytics records are flowing: Data Plane Pump; MDCB; Control Plane Pump; Persistent Storage.
• Tyk Operator: Confirm Operator logs show clean reconciliation cycles without schema or API errors.

Related articles