December 12th Post Mortem
Over the past month, Coinbase has continued to prioritize scaling our backend systems so that we can provide a reliable customer experience during periods of high traffic. Despite our improvements, on December 12th, all time high levels of traffic lead to periods of slowness and elevated error rates.On December 11th at 22:00 PST Coinbase underwent scheduled maintenance to improve database performance. One of the techniques we utilized during this maintenance window was to distribute database load by splitting existing datasets to new clusters.On December 12th, starting at 04:00 PST, traffic began to climb as a result of large Litecoin price movements. At 04:15 PST, two of the new clusters from the maintenance window began to experience slowness as a result of the slower network attached disks used by …