Using table_sharding in Tyk Pump and Dashboard

When table_sharding is enabled in Tyk Pump, analytics records are stored in date-based tables with the format as below:

• tyk_analytics_YYYYMMDD — for individual request logs (used by SQL Pump)
• tyk_aggregated_YYYYMMDD — for aggregated records (used by SQL Aggregate Pump)
• tyk_uptime_analytics_YYYYMMDD — for uptime records (used by SQL Uptime Pump)

Problem

Due to a known bug at the time of writing, the Dashboard may query the incorrect (unsharded) table when retrieving analytics. This can make it appear as though analytics data is missing or not being collected.

Workaround

To ensure all analytics data is correctly stored and queried using sharded tables, configure the following environment variables:

For Pump:

TYK_PMP_PUMPS_SQL_META_TABLESHARDING=true
TYK_PMP_PUMPS_SQLAGGREGATE_META_TABLESHARDING=true
TYK_PMP_UPTIMEPUMPCONFIG_TABLESHARDING=true

For Dashboard:

TYK_DB_STORAGE_MAIN_TABLESHARDING=true
TYK_DB_ENABLEAGGREGATELOOKUPS=true

Important

Recall that the Dashboard supports configuring separate storage settings for main, analytics, logs and uptime fields within the storage section. (See documentation here).

Please note that it is important that storage.analytics.type , storage.logs.type and storage.uptime.type are unset for sharded lookups to work.

It is sufficient to configure only the main section, as shown below - the Dashboard will automatically use the appropriate sharded tables.

TYK_DB_STORAGE_MAIN_TYPE=postgres
TYK_DB_STORAGE_MAIN_CONNECTIONSTRING="user=root password=admin host=127.0.0.1 port=5432 database=tyk_analytics"
TYK_DB_STORAGE_MAIN_TABLESHARDING=true

TYK_DB_ENABLEAGGREGATELOOKUPS=true