Reflections on Visa's High-Availability Payment Network

Way before Google-scale or Facebook-scale existed, there was Visa-scale. This COMPUTERWORLD article from 2001 provides some insights about early Visa payment processing infrastructure.

Indeed, everything in Visa’s processing infrastructure - from entire data centers to computers, individual processors and communications switches - has a backup. Even the backups have backups. For example, the Washington center has four rotating uninterruptible power supply (UPS) units (only three are needed) driven by the local utility and backed up by an array of batteries and four 1-megawatt diesel-powered generators. The 24,000 gallons of diesel fuel stored on-site is enough to power the center for a week. The UPS units protect the center from possible power fluctuations. The facility has enough redundant cooling capacity to air-condition 300 homes.

Scalability, Reliability and Security

Visa payment processing infrastructure uses multiple layers of redundancy and backups (3 layers or more). Visa uses a global network of payment processing data centers. These data centers are connected via their own copper and optical fiber networks (again connectivity is redundant). In addition each Visa customer or bank is connected to Visa network by at least two different commercial carriers. Two or more submarine communications cable is used for international customers.

Each Visa data center is state-of-art facility and resembles a fortress with focus on both data center security and cyber security.

It can withstand earthquakes and hurricane-force winds of up to 170 mph. A 1.5-million-gallon storage tank cools the system. Diesel generators onsite have enough power, in the event of an outage, to keep the center running for nine days.

Some of the recent DevOps ideas such as Netflix’s Chaos Monkey are not that new. Visa has been using these tricks for at least last 12 years. For instance,

So while we are processing 5,000 messages a second, we’ll knock off a storage controller and make sure the system doesn’t skip a beat.

As matter of fact Visa has tradition of performing full-scale stress tests on their systems with elements of chaos to measure and model requirements, reliability and load.

Mug Number And Transaction Speed

In 2010, Visa was handling on average around 2,000 transactions a second, with a daily peak rate of 5,000 transactions a second. During busiest season of the year it peaked to 11,000 transactions a second (Dec 23)¹. In 2010 they had burst capacity of 24,000 transaction per second with no degradation of transaction quality. Visa determines the burst capacity using Visa Integrated Payment (VIP) annual stress test. They call the maximum transactions per second as Mug number. Following video captured the VIP stress test as it happened.

In addition, one whole end-to-end transaction (steps 3~8) as described in following image² completes in a few hundred milliseconds ³.

Although little is know about PayPal network capabilities but it seems it can manage only ~150 transactions per second ⁴ (one more reason to think about v.me, a Paypal alternative from Visa).

Update

According to latest updates⁵, Mug number in 2013 was 47,000 transaction messages per second, which is a massive improvement from 2012’s peak capacity of 30,000 transaction messages per second.