Overview
In Tyk Hybrid and MDCB deployments, edge Gateways maintain long-lived remote procedure call connections to the MDCB control plane to receive near-instantaneous notification of API and policy updates. Under certain network conditions, the Gateway may log a warning indicating that the connection was closed mid-stream.
Symptom & Error Message
The following warning appears in the Tyk Gateway logs:
[RPC STORE] RPC Reload Checker encountered unexpected error: gorpc.Client: [control-plane-host:443]. Cannot decode response: [unexpected EOF]Root Cause
The RPC Reload Checker runs a background long-polling loop asking the control plane whether a reload is required. An unexpected EOF occurs when the underlying TCP or TLS connection is terminated abruptly. Common causes include:
- Intermediate firewalls, NAT gateways, or load balancers dropping idle TCP sessions after a connection timeout.
- Transient network resets along the routing path between edge clusters and the control plane.
- Temporary restarts or scaling events on the control plane.
This warning does not unload existing APIs, trigger an immediate reload, or cause requests to fail. Edge Gateways cache API configurations and keys in local Redis storage and will continue serving live client traffic normally while automatically reconnecting in the background.
Distinguishing Control Plane Warnings from 503 Errors
If an API client receives an HTTP 503 error around the same time this warning is logged, verify the error origin before assuming a Gateway failure:
- Upstream Service Origin: Tyk passes through upstream 503 responses directly. Inspect the Gateway access logs or OpenTelemetry spans. If the span shows a completed outbound call to the upstream host that returned a 503 status, the error originated from the backend service.
- Gateway Origin: The Gateway only generates a 503 on its own when an enabled circuit breaker trips (returning a Service temporarily unavailable response body) or when uptime tests mark all upstream targets unhealthy.
Mitigation & Verification
- Verify Gateway pod connectivity to the control plane RPC endpoint on port 443.
- Review firewall and NAT idle timeout settings along the network path to ensure keep-alive packets are not prematurely dropped.
- Monitor subsequent Gateway logs to confirm that automatic reconnection succeeds within seconds without administrative intervention.
Applies To
Tyk Gateway 5.0+, Tyk Hybrid deployments, and Tyk Multi-Data Centre Bridge (MDCB).
Comments
0 comments
Please sign in to leave a comment.