Will Regulators Mandate Multi-Region over Multi-AZ in Cloud ?

Multi AZ datacenters

There was a time when one could simply point to the a,b,c of the three data centres in a Multi-AZ (Availability Zone) configuration as a satisfactory replacement of traditional Data Center–Disaster Recovery (DC-DR) requirements even for the regulated sectors. However, as cloud providers face more frequent outages, it’s becoming harder to consider this setup a reliable solution.

Just recently, Google Cloud (GCP) experienced a significant outage lasting about 12 hours, which took down the entire europe-west3 region in Frankfurt, Germany. The root cause was traced to a power and cooling failure, which forced a portion of a zone offline, affecting a wide range of services from Compute to Storage.

Traditionally, disaster recovery setups have involved multiple data centers designated as Primary and Secondary (or DR centers) with data replication. Often, these DC and DR are in separate cities. In the companies that I worked, these were atleast 300 kms apart. While effective, this approach is costly and challenging to maintain, with lot of administrative overhead.

This raises questions about how fintechs and other regulated sectors meet Business Continuity and Disaster Recovery (BCPDR) requirements in the cloud.

In cloud environments, redundancy can be implemented in several ways, with Multi-AZ setups being one of the simplest. Multi-AZ configurations typically involve three data centers within a city, usually spread 30-100 kilometers apart and identified as zones “a,” “b,” and “c.” Data is replicated in real-time across these zones, allowing this setup to be marketed as an alternative to traditional DC/DR arrangements. However, Multi-AZ is not a default feature; it requires opting in and incurs extra costs.

Another approach is Multi-Region, where data is replicated across geographically distinct zones, often in different seismic areas. This setup helps mitigate the risk of a single region being impacted by events such as severe flooding, prolonged power outages, political unrest, and similar disruptions.

Interestingly, some financial institutions, especially Banks are not entirely sold on Multi-AZ setups, pushing their technology partners to adopt multi-region architectures across separate seismic zones. While no regulatory body in India has mandated multi-region setups yet, it’s worth considering the distance between zones within each cloud provider:

  • Amazon AWS states that its Availability Zones are up to 60 miles (~100 kilometers) apart. Interestingly, AWS keeps the exact locations confidential, even from most employees. Source: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/availability-zones.html#:~:text=Availability%20Zones%20in%20a%20Region,with%20single%2Ddigit%20millisecond%20latency
  • Microsoft Azure mentions a minimum of ~400 kilometers between its AZs, which is the greatest distance among major cloud providers. Source:https://learn.microsoft.com/en-us/azure/reliability/cross-region-replication-azure#what-are-paired-regions)).
  • Google Cloud (GCP) does not publicly disclose the distances between its AZs, keeping this detail confidential. surprising!

Given these developments, regulators may eventually require multi-region setups for regulated entities and fintechs using cloud services instead of relying solely on Multi-AZ. So far, Multi-AZ has offered a relatively straightforward solution to meet compliance and audit requirements for BCPDR. However, it might be time to reconsider RTO (Recovery Time Objective) and RPO (Recovery Point Objective) expectations in cloud environments.

Any thoughts on these developments?

  • Link to the RCA: GCP Outage Incident Report https://status.cloud.google.com/incidents/e3yQSE1ysCGjCVEn2q1h)
  • Article on 12 hour outage: https://www.theregister.com/2024/10/25/google_cloud_frankfurt_outage/

 

When Should Startups Hire a CISO ?

 

When Should Startups hire a CISOAs someone who has been in the IT industry for two decades, primarily working with startups and serving as a security practitioner and consultant now, I often get asked about the right time to bring in full-time security professionals.

These roles could be a Security Engineer, Security Architect or even a more senior role as Chief Information Security Officer (CISO).

But, before that, we need to understand that Startups typically go through various stages of funding and evolution as they grow and develop their business. It is essential to understand how this funding impacts the decision making around security budgets and hires. 

While there’s no one-size-fits-all answer to when startups should hire security professionals. It depends on various factors, whether a startup is in the regulated sector or caters heavily to customers in regulated sectors like Lending, Insurance and Broking or high tech. Does the startup deal with a lot of Personally Identifiable Information (PII), operates across the world, etc.

The reason to put forth this thought is in some sectors, the regulatory requirements are stringent and calls for granular security controls for data protection needing investments in terms of tools, services and people at a much earlier stage. Can startups afford a full time security role at such an early stage? Usually the answer is no and, I think a good alternative lies in the utilisation of Virtual CxOs or Fractional CxOs.

These roles, such as Virtual Chief Information Security Officer, Virtual Chief Technology Officer, Virtual CFO, and others, offer startups invaluable services at a fraction of the huge costs. 

This approach not only provides cost-efficiency but also circumvents the challenges associated with lengthy hiring cycles and the complexities of downsizing. Essentially, these arrangements operate on a ‘pay-as-you-go’ model, offering startups flexibility and strategic support without the traditional hiring headaches. 

So, when can Founders bring in the virtual or a full time CISOs ? Before you go there, a word about CISOs:

CISOs come in all flavours! Some are compliance oriented and usually have transitioned into vCISO roles after a long stint in large enterprises. They would be supported by a team and may be less hands-on. On the other hand, you have the hands-on CISOs, the ‘get your hands dirty in the code’ kind of experts. They’ve seen the trenches, fought the cyber battles, and they’re not afraid to roll up their sleeves.

Both types bring immense value, but choosing the right flavour depends on your startups needs and culture.

Here are the common stages in the funding or evolution of startups and what expectations are from the security roles, be it virtual or full time. Hope you find them useful and please do reach out if you need any help (you can mail me: prasanna@cyberquotient.in):

Pre-Seed Stage:

  • This is the earliest stage of a startup, where the founders use their own funds or money from friends and family to validate their business idea.
  • Funding Source: Founders, friends, and family.
  • Security expectations / deliverables: Since validation of ideas is the prime focus here, a full-time security role is obviously overkill. At the most, the founders would want to know regulatory cyber security compliances of their domains, high level security architecture reviews, security of tech stacks etc. If this is the case, you can talk to  a vCISO or a trusted security professional from your family, friends circle or professional network. 

Early-Stage / Series A:

  • At this point, the startup has a validated product or service and is looking to scale its operations. Funding is used for market expansion, team building, and further product development. 
  • Funding source: Venture capital firms, Angel investors, 
  • Security expectations / deliverables: A minimum viable security of the products,  infrastructure, hygienic data security and privacy controls, High level architecture reviews, are the expectations. Also, specific compliances related to data security and privacy, etc., are to be baked in products and services offered. 
  • Which CISO to hire? A full time CISO / Security Engineer would be an overkill and Startups can benefit from Virtual CISO, who has a mix of Tech and compliance skills, with practical experience. Some of these tasks can be outsourced and accomplished through partner ecosystems. Having a dedicated security team might find the ever changing priorities of a startup too much to handle. Please do read above on various types of CISOs available. 

Growth Stage / Series B and C:

  • Startups at the growth stage have proven market traction and are scaling rapidly. Funding is used for market dominance, scaling operations, and expanding the customer base.
  • Funding Source: Venture capital firms (Series B, C, and subsequent funding rounds).
  • Security deliverables: It gets interesting here. At this stage the products or services offered by startups are already expected to meet certain baseline. Periodic Vulnerability Assessment and Penetration Testing (VAPT) of the web applications, mobile apps, cloud, end user computing are the standard requirements here along with source code reviews. Policies and procedures relating to security, business continuity, ID and access management, data storage, backups are expected to be in place. A common phenomenon I have seen is that startups, which scaled but underplayed the importance of good security hygiene, ended up with a lot of tech debts / security debts and causing friction among teams. Most teams that bear the brunt of this or affected would be Security <-> Engineering <-> DevOps. Eventually, this affects the scalability and agility. of the startup
  • Which CISO to hire? Hire a strong security leader (it could be CISO / Dir of Security / Head of Security, etc) and ensure they are supported by a team including at least one application security engineer, IT / IT Security engineer, compliance analyst etc.

Late-Stage / Series D and Beyond:

  • These startups are well-established in the market and may be preparing for an initial public offering (IPO) or considering other exit strategies. Funding is used for global expansion, acquisitions, and preparing for exit.
  • Funding Source: Venture capital firms, private equity, and sometimes institutional investors.
  • Security deliverables: Same as Series B & C, but the frequency and customer demands, or compliance requirements are more. Periodic VAPT of the web applications, mobile apps, cloud, end user computing. Internal and External audits may be a mandatory requirement. 
  • Whom to Hire: Hire a capable, hands-on security leader (it could be CISO / Dir of Security / Head of Security, etc) and ensure they are supported by a team including at least one application security engineer, IT / IT Security engineer, compliance analyst etc. This stage might also have similar challenges of tech / security debts for the incumbent.

IPO (Initial Public Offering):

  • Some startups choose to go public, offering shares on the stock market. This provides liquidity for early investors and allows the company to raise significant capital from the public.
  • Funding Source: Public investors who buy shares on the stock market.
  • Security deliverables:  At this stage, the organisation is expected to meet higher standards of security including certifications to standards like ISO 27001, although desirable, its not a must. Usually a third-party conducts readiness assessments covering various aspects such as information security,  cyber resiliency, and more. One can expect to see lot of documentation work from the security team including threats and disruptions that can impact the business. The vCISO / CISO will contribute towards the Draft Red-Herring Prospectus (DRHP) documentation as well (I tell this from my experience at Navi).
  • Whom to Hire: Contrary to popular belief, a CISO position is not mandatory for filing IPO, unless you are in regulated sector like Banking, Insurance or Mutual funds. At this stage, a company is expected to have a steady state of security practice. However, startups that grow fast and scale exponentially often face challenges in their security operations. I have seen some startups facing continuous issues and disruptions when tech/security debts have piled up! Look for someone with prior experience in building and scaling security programs at startups.

It’s crucial to understand that not every startup adheres to this precise progression, and the path to funding can fluctuate depending on factors such as industry, business model, and market dynamics.

Similarly, the approach to security may also vary. The consequences of non-compliance can be significant, as illustrated by a recent incident where the RBI imposed hefty fines, viewed by many as potentially fatal.

The era where examples like Uber’s rapid growth leading to regulatory adjustments served as a common case study seems to have shifted. Instead, it’s becoming more evident that it’s a matter of:

‘Comply or Perish’.