- Early engagement
- Doing triage
- Clarity about vendor selection criteria and contractual requirements
- Building in means to future-proof and
All building blocks for effective supplier security governance that are frequently ignored, half-done, or under resourced.
The second part has taken some time to publish (apologies to my readers). Firstly because it includes a huge amount of content reflecting real-life lessons learned over many years. And secondly, because my computer crashed, I lost most of the first draft. Failure to operationally allow for 3rd party resilience, backup and recovery…you could say. But here it is, all 4,000+ words:
JIHGF – The remaining alphabetic foundations of effective vendor cybersecurity governance
- Justifying your existence – a.k.a risk based resource modelling and reporting
- Identifying and Integrating related processes – you can’t decouple this from procurement, change, architecture, legal, and a plethora of other things going on.
- Handling Huge suppliers – they are a subset of their own in terms of how you deal with them.
And, coming in part 3 (to avoid serious writing and reading fatigue):
- Governing vs auditing – a single big-hit audit isn’t going to give you security assurance, but doing better depends entirely on related contracts, relationships, and security capability.
- Focus on the risky future – Looking at desired outcomes to avoid medium-term pain and keeping an eye on the evolving vendor risk landscape
In this second instalment, 3 more things that keep a rein on that risk and ensure repeatability and consistency of related Governance, Risk, and Compliance (GRC) effort.
SIX – Justifying your existence
In the first instalment, point 2 was about triage. A simple, early, and inherent risk related step to enable planning, prioritisation, and (crucially) budgeting for bodies. That last part isn’t (like most of this), rocket science, but it is something most security functions do badly, if at all.
How often is the size of your governance team dictated by an annual ‘feels about right’ FTE number. A number that later gets whittled down to something you know (but can’t prove), is too small? How often does the ‘finger in air’ estimate get trampled by a weight of unforeseen and reactive assessment or remediation effort? Staff progressively worn and credibility destroyed by excessive work, last-minute demands for engagement, or growing backlogs. The antidote is outlined below:
Shut the team in a room. Set out the key high-level tasks involved in fulfilling your end-to-end supplier governance remit. Split what you list into two columns:
- Per supplier
Take best local experience, hit up industry peers, or steal hours billed from consultant invoices, to attach an annual hours estimate to each task. For supplier assessment and remediation, there’s a defendable correlation between the things asked about to triage (size of suppliers, inherent risk indicators), and governance effort.
Split those supplier specific estimates down again. How does effort differ between riskier/bigger suppliers vs middling, or less risky/smaller suppliers? Don’t crumble under pressure to be precise, you can selectively refine in-flight, and review after 2nd and subsequent turns of the governance handle.
Voila: Means to predict workload and required staff to service it. More than that, it’s the basis for a robust and defendable budget application. Even more than that, if you are forced to cut headcount, or supplier scope creeps, you can give real life potential impact (in terms of suppliers that can’t get assessed, likely assessment delays, and the inherent risk-based view of possible fallout). That’s something your business risk stakeholders need to fix by putting their money where there mouth is, or formally agreeing to tolerate risks.
They then get to benefit from saved money, if the illustrated impact on governance processes and resulting risks is tolerable. Or they spend more to reap vendor governance and risk management benefits. A cost/benefit backed conversation about security. The type of conversation your business has no doubt been asking to have for a long time.
SEVEN – Identifying and integrating related processes
Any vendor security governance activity will fail if it operates in a security or procurement silo. Fact. Usual reasons for failure:
- Lack of appropriate sponsorship, resulting in poor stakeholder buy-in
- Complexity and confusion
- Late engagement
- Lack of formal stakeholder accountability
In aggregate; failure to help the business understand ‘why’ and delivering something that isn’t fit for purpose.
Forward planning and collaboration plays a huge part in solving these problems. If you don’t get this right you won’t hear from your stakeholders, you’ll hear from them at the last possible minute, or they’ll tell you “I’ve already done this for [name 4 or 5 related functions]!”
In part 1 we talked about the RACI (Who’s Responsible, Accountable, Consulted, and Informed). That identifies groups with whom you need to engage, and the driver for that engagement. Some examples below:
- Architecture/Strategy Functions: The ‘bright ideas factories’. If you designed initial triage questions so they can be answered at earliest stages of planning, you can plug that in (with linked referral triggers and mechanisms) as a mandatory activity. Better still, during implementation, get a seat at the table and assess in person. If they’re planning anything that might involve a new 3rd party, or getting something new from an existing one, answer those triage questions. The best way to get useful sight of the pipeline of work to enable proper planning and give decision makers early sight of policy, regulatory, and legal control requirements.
- Change: Another intimately linked area. Triage should be a pre-initiation step. You need to catch 3rd party requirements before resource levels are fixed, budgets are agreed, and designs are signed off. There are smart ways to link triage for 3rd parties to change assurance triage. Work to assess change project security is subject to most of the same challenges as vendor security governance, and things that cause inherent risk are often the same. The information gathered during triage can refine depth/coverage of later engagement, to save everyone time and effort.
- Procurement: The area accountable for ensuring triage activity takes place…if you have agreed that RACI and worked to embed triage work into more general Procurement processes. They are also your authoritative source of scope information, although procurement thresholds for formal oversight often exclude smaller (lower spend) suppliers. Suppliers who should still be on your radar. Your backstop is finance, as mentioned in part 1 – sampling to find and triage suppliers who fall out of procurement remit to manage.
- Legal: Your backstop in case things slip through all of the above nets. Triage should be embedded in their processes too. Educate your friends in the legal team, and collaborate to clarify the link between triaged inherent risk and required standard/enhanced due diligence and contract contents.
- Supplier Relationship and Commercial Contract Management: These are the folk you cannot do your job without. They dictate the quality and effectiveness of your relationship with 3rd parties. You need to identify the biggest influencers in these groups. Create evangelists. Include them in discussions from the earliest stages of framework design. If these folk are missing or create blockages, you also need a placeholder to raise related risks with their reporting line at governance meetings. Often you can help them highlight issues with workload, or persistently unresponsive suppliers. After all, you’re in this together.
- Audit, Risk, Data Protection, Physical Security, Business Continuity, Pentesting: Other areas with skin in the game. Find out their assessment plans and de-duplicate as much as possible (while respecting the need for independence between lines of defence). For related specialist areas, understand what they do now, and work to merge triage and assessment activity as far as possible. Security Impact Assessment, Privacy Impact Assessment, Business Impact Assessment – typically 50-70% crossover in triage questions asked, and inherent risk triggers for more engagement. 50-70% of the reason stakeholders don’t ‘get it’ and try to duck efforts to engage.
Not an exhaustive list, but you get the idea.
EIGHT – Handling HUGE suppliers
A topic that really needs it’s own post (or book), so this final section might be too long for some readers. Having said that, it isn’t intended to be the usual solution-free security snacklet, so perhaps, if interested, bookmark and read more later.
These suppliers are a subset all of their own, and require a distinct approach. For the purposes of this post, I’ll break them down into 3 rough groups:
- Big IT Outsource Suppliers
- Big Cloud Suppliers
- Big Suppliers of Generic Services or Products
It’s possible to do more (or less) effective on-going security governance on the first two groups. But for the third group, it is arguably a waste of time. A big assertion I defend below.
8.1 Big IT Outsource Suppliers
Many companies ask huge suppliers to handle big chunks of their network, telephony, web, server, and endpoint estates. As part of that, these suppliers operate a huge range of security controls that have potential to impact every business function: Perimeter defences, privileged/non-privileged access management, domain administration, encryption & key management, secure build (endpoint, midrange, server, network), secure coding & software development, vulnerability monitoring & management, log monitoring, malware protection, DDoS protection, data leakage protection, IT & information asset management, backup & recovery, secure disposal, physical security etc, etc.
Layered controls and far-reaching risks
The adequacy or inadequacy of those layered and interrelated controls, feeds into the risk profile for everything built on top. For example, your finance software:
The application looks dandy. Nicely controlled user access. No identified software vulnerabilities. But, the administrator web interface has a shared password, the database it sits on has no logging for suspicious access or activity, a hoard of 3rd party support people have accumulated root Unix access, it’s running an unsupported legacy OS due to application compatibility problems, someone stuck an any/any rule on the firewall between the midrange servers and the DMZ (for a reason no-one can remember), and they’ve just outsourced hosting all that to a co-location company based in Syria.
Oh, and your Customer Relationship Management and HR systems sit on the same server cluster.
Now consider that matrix control relationship for everything that depends on the supplier securing networks, tin, and software…plus 4th, 5th, 6th parties paid to do parts of that job.
That’s why I argue you need to budget for a specialist third-party to audit those suppliers at least annually. I’m not talking about a generic audit with some IT General & Application Controls, I’m talking about a SOC II type audit, where you tailor the scope to reflect your policy, regulatory, and legal security and data protection requirements. That audit work is not something an internal vendor security governance team can handle. Not if you need them to look after the rest of your supplier population. You need to partner that with robust regular governance, including metrics delivered showing remediation progress, and effective operation of a subset of continually operated controls (see point 9 for details).
Building the business case
It’s something you’ll struggle to sell if it’s never been done before (cost can be > £100k), however the business case is strong. These suppliers likely have potential to cause multi-million pound impact if poor security controls result in a breach, it can reduce internal budget and staffing required to service multiple related regulatory and legal requirements (SOx, PCI DSS, FCA, Data Protection, Solvency II), and it fosters internal and 3rd party stakeholder confidence (risk, audit, regulators). It should therefore, in my opinion, be a policy requirement for these most materially impactful suppliers (normally 2 or 3, identified through triage, and agreed with procurement and business stakeholders), and form part of the security, audit, and commercial contents of contracts.
Sharing the audit burden – a better way?
Alternatively, your supplier might have commissioned their own independent audit, or have permission to share one performed by another client. This needs to be the way the industry goes in my opinion: Firms sharing the cost, or suppliers treating such an audit as a cost of doing business, then selectively sharing output with multiple clients. Something that could save everyone enormous time and money, and drive up security standards across the board. Even if it’s not fit for everyone’s purpose, they could ‘assess round the edges’, still cutting down on mutual pain. This is especially relevant to smaller clients: Even if they are granted permission to do such an audit, it’s highly unlikely small firms could finance such work.
This kind of standardisation and sharing of the audit burden is something the Financial Services Roundtable tried to facilitate via their BITS initiatives, with limited success. You can also gain reasonable security assurance for card data handling processes and storage environments if third parties are PCI DSS compliant (within the defined PCI coverage and control scope), because requirements are generally robust and well audited. But beware of any other certificates or badges that allow suppliers to self-assess (Safe Harbor and it’s Privacy Shield successor are the most headline grabbing examples, but that’s more about data protection and privacy – you can find details here).
Realistically it will take an inter-industry culture change supported by heavyweight global influencers, and government+private sector collaboration, to make it a practical reality.
Trust but verify
In the meantime, if offered an audit report, ISO27k certificate, or some other alternative to doing your own assessment, you need to ensure;
- Scope covers people, processes and technologies used to service your contract,
- Control scope reflects your requirements
- You get to see actual evidence of control existence/design/effectiveness, not just a high level report.
But, when all’s said and done, none of the above will make a difference if the supplier is fundamentally incapable of providing the level of security you require, or the relationship is broken (it’s not uncommon for these kinds of suppliers to end up managing their clients – the clearest symptom is security being seen as an ‘extra’ that you are forced to pay more for). The root cause is poor due diligence, ineffective contracts, and no respect for the transformation, integration, and governance journey required to align and maintain controls to a mutually agreed standard (see point 3 in part 1 for details).
8.2 Big Cloud Suppliers
One of the key problems with trying to robustly govern these suppliers, is summed up by this slightly cynical excerpt from the previously referenced post on the now defunct Safe Harbor EU:US agreement.
In July 2012 a working party made up of representatives from data protection authorities in all (then 27) EU member states, said it wasn’t good enough for Safe Harbor scheme cloud vendors (or cloud vendors with Safe Harbor sub-processors) to just say they were compliant with EU DP law. No indeedy! Firms using their services had to get them to prove it, or be deemed in breach of their domestic DP laws:
Says the data protection bloke covering all 27 branches of Fred’s Car Parts Ltd
“…give us a look round your US data centres”.
Yeah, that was always going to work.
Of course we’re talking about security governance vs data protection governance, which are not the same…however the probable response to a request to audit Amazon, Microsoft, or Google will be identical. They suffer the same potential audit fatigue as your other suppliers (multiplied by a big number of your choosing). Their almost universally default policy is therefore to deny you the right do a security assessment.
Bespoke clouds – a false economy
The other factor at play here is economies of scale…the answer to the question: “Why are cloud offerings often cheaper, and why are there so many providers in this market?”. Their entire business proposition is based on delivering a standardised, scalable and agile offering to all customers. That means they can’t, in almost all cases, change what they do to suit you without destroying their cost model. Or, if they offer some flexibility, it won’t be cheap. Probably wiping out the savings that attracted you to the solution in the first place.
The real risk is that their security control set doesn’t mitigate risk to your business to a tolerable level. That doesn’t always mean having all the controls you mandate in your policies (even if they are realistically linked to your risk appetite). This is also the time to point out that there are solid drivers for biggest cloud providers to have robust security:
When supplying a reasonably ubiquitous service or product, any incident caused by a gap in security controls has potential to wipe out confidence of a large portion of your client base, and significantly damage your brand, share price and future sales…economies of scale working in our favour.
When it comes to policies, one size does not fit all (as talked about in point 3 of part 1), and cloud suppliers are one subset who need their own defined control requirements. That’s otherwise known as your ‘Cloud Policy’ when twinned with risk based guidance on what you do and don’t want put in the cloud, and the RACI for the people who set, own, and approve exceptions to those compliance requirements.
Beyond getting a cloud policy with fit for purpose vendor requirements in place, my main recommendations are:
What will the vendor do for you? The answer to that question dictates whether offered security controls are adequate. You will get some important information from your triage activity (data type, sensitivity, quantity, availability required etc), but you need more detailed scoping to understand the implications of arms-length blackish-box supplier risk management.
Firms too often only have a rough idea (often based on vendor sales pitches) of what a product or solution will actually do, and dramatically underestimate the amount of internal work required to get things into a fit state to be ‘cloudified’. That includes people, processes, and technologies. Will the solution have to splice onto things hosted elsewhere? If so, will that break controls, or require work arounds that don’t meet security requirements. In terms of access management, can the access model for the cloud service or product integrate with your local user directories, if it can, what are the security implications? If it can’t, is an alternative going to be adequately secure and maintainable, now and on-going? Just a couple of the more common challenges.
The more formal term for that is orchestration, and part of that preparation and transition planning, a part sometimes overlooked, is security.
Understand the risk and assess their transparency
Beyond sight of throughput and availability, two key transparency priorities for cloud vendors are incident management and data governance.
Most domestic Data Protection laws, and the new EU General Data Protection Regulation, (now approved and due to become enforceable in 2018), demands that you are able to locate and delete personal data when it is no longer needed for a defined purpose. You also need to be able to service Subject Access Requests and other Data Subject Rights like confirming and (if requested), deleting/returning all the personal data held relating to that individual. If there is a breach involving personal data, you then need to be able to tell your local Information Commissioner within 72 hours of being notified.
More generally, if the old EU position remains the same (see the example at the top of this section), you can be held generally liable for not effectively defining, assessing and governing adequate 3rd party security and data protection requirements. The fines that can be levied, if you are found to be non-compliant, are up to 4% of global annual turnover, or up to 20 million Euros – whichever is greater.
That’s one aspect of related data risk, you can then add back card data risk (PCI DSS), risk to intellectual property (competitive disadvantage), financial reporting data risk (SOx), financial regulatory penalties for not treating customers fairly (FCA etc), and the hard-to-quantify short/medium/long-term impact on brand, share price and customer confidence.
Back to the wider transparency question: If they won’t let you audit, what will they tell you, and how frequently will they tell you about security controls in place and their effectiveness?
You need to be absolutely clear on what the supplier will allow you to do and see to verify the trust you intend to place in them.
Do your due diligence
So how will they log and track whereabouts of data? Can they/will they spot, crisis manage, investigate, and notify you of incidents quickly enough to enable effectively management of these risks? And can they give you enough visibility of security controls to be able to satisfy other lines of defence and your regulators?
Your due diligence equates to robust assessment of the above kinds of risks, capabilities, and transparency requirements. The outcome should be a report on any assessed gaps between your control requirements and their control capability. Including a clear risk articulation based on the triaged inherent risk linked to intended service or product use.
Get formal approval for residual risks
Hopefully this one is self explanatory. In point 5 of part 1 I talk at length about accountability, and how vital it is to the effective implementation and operation of any vendor security governance programme. That process of defining your RACI, getting it approved by stakeholders, and using it to inform your related supplier governance policy requirements, should tell you who to go to for a decision on risk tolerance.
The nature of cloud supply means you lose the ability to closely oversee those 3rd parties. If you don’t do this work up front, imagine the questions that will be asked if there is a breach. “Didn’t you do due diligence?” may not be the first question, but it’s likely, in regulatory and legal terms, to be the most important.
8.3 Big Suppliers of Generic Services or Products
What do I mean by Generic Services or Products? I mean suppliers who are likely to be on procurement’s radar as tactically, strategically, financially, or operationally important (usually because of spend). Suppliers who may – or may not – fall into assessment scope after triage. Suppliers I’m arguing you should automatically remove from future audit/assessment scope. That’s a risky thing to recommend, so I’ll explain:
These are firms that provide things to your company that are identical to the things provided to every other client. Cloud vendors have much in common with this group, but the ones I’m specifically referring to are a group of their own. Examples are:
- Banking Service Providers – e.g company credit cards, BACS/SWIFT services, your business bank account.
- Utility Companies – e.g. Electricity, gas, water
- Licensing Providers – e.g. Oracle, Archer, Microsoft
Your duty is to make sure they don’t provide some bespoke element of service that would require on-going assessment. If not, do due diligence to assess security requirements and supplier security capabilities (as far as you a permitted to do so), relating that back to your triaged view of inherent risk. What you cannot do, if you are taking their bog standard service or product (the same one everyone uses), is get them to change it, or how it is delivered. They cannot and will not do that. So, following this argument to its logical conclusion, why would you waste dramatic effort trying and failing (or occasionally trying and succeeding) to get an audit done, when the result will be blanket refusal to change anything?
Failure to manage stakeholder expectations and explain these likely scoping issues early on, will result in credibility bashing pressure from on-high to get these suppliers assessed, even though there is no risk-reduction value linked to doing so. There will be multiple other discretionary, resource, blockage, and risk based reasons to de-scope suppliers from your book of ongoing governance work. These are likely to be the most high-profile, but each and every decision should be recorded and approved by risk stakeholders.
Your risk management then falls outside the vendor security governance processes: Where you noted a mismatch between supplier security and your internal requirements, you should articulate the potential associated risk (informed by your triaged view), and present that to the business for possible acceptance. Another unofficial, but pragmatic risk perspective: If your electricity company, or your bank goes bang, it’s over to the disaster recovery guys. If they have a serious control failing resulting in a data breach, it is highly unlikely only to affect your business, and there’s minimal risk you will be in line for fines…unless you don’t respond appropriately. That’s where this focus shifts.
You need to know what data and other assets they hold, and which key processes depend upon them to inform your incident response plan. In terms of supplier selection and on-going governance, that leaves the business with two choices:
- Accept the risks, or
- Chose another supplier who provides the controls you want
But let’s face it, if MasterCard, Visa and your bank aren’t meeting your security requirements, perhaps…in age old relationship ending parlance…it’s not them, it’s you. Time to revisit those policies for another chat about controls you demand and real risk appetite.
Don’t agree? Fine. Good luck persuading your board that they need to find a new bank.
That is the current running through all of the guidance offered so far. Every day, all across the land, supplier security assurance teams will tell business bosses “But there’s no real risk”, “We need more staff”, or “We need more time”. What most of them can’t do is robustly defend their reasons for doing so. Doing the kind of thinking, planning and preparation I’ve described so far is hard work, but it gives you the tools to make (and help the business to make), defendable risk management decisions.
Ultimately equivalent to money and time spent bringing in consultants and back-filling justifications – a.k.a making excuses – when work isn’t done, or when you wake up to news of a supplier breach.
Part 3 out soon (having hopefully mitigated the risk of more data loss!)