Some answers became visible this year. On July 19, 2024, millions suffered disruptions in service due to a faulty software patch distributed by CrowdStrike, a trusted and reputable security vendor. In publicly disclosed information, CrowdStrike explained it used a process called “Rapid Release” to implement the patch for its customers globally all at once. We may have been upset with CrowdStrike customers whose services were disrupted, but the Rapid Release mechanism didn’t give them an opportunity to test the software update before installing on their devices.

This event has been called the largest outage in the history of information technology. And it didn’t result from ransomware, malware, or any other form of cyber-attack. Instead, this was a security issue resulting from availability or reliability. Globally, many industries were affected. Media attention focused on airlines and airports, but the outage also affected hospitals, banks, hotels, manufacturing, stock markets, broadcasting, retail stores, and more.

The CrowdStrike Outage: Lessons Learned

Everyone can benefit from reviewing this large-scale incident because CrowdStrike’s practices are widespread in the industry. The outage highlights the dangers of not paying close attention to the secure software development lifecycle. Software errors don’t just result in the security vulnerabilities that cyber attackers exploit, they can also lead to slow response time, outages, or other performance issues.

The bottom line is that almost all the effort and expense in the cybersecurity space has focused on network security, managing vulnerabilities, access controls, and detection mechanisms. Software development has been neglected -- or worse, quietly viewed as off-limits for security requirements.

This low priority on software development stems from a misconception among business leaders in the technology and security industries that security features can be added later. They feel it’s more important, in terms of revenue and profit, to get a software product or new release on the market as fast as possible. In truth, it isn’t easy to add security features to a finished software product. It’s been proven to be much more expensive and time consuming to retrofit software for security than to design and build it in from the beginning.

The time pressure placed on programmers who write software code also plays a role. They tend to look for similar code, then copy and paste it into a current project to accelerate completion, rather than taking the time to write code according to specifications. Errors are compounded by poor training and the fact that many programmers are contractors who move from project to project and never see their work in production. This, in turn, hinders the learning process for programmers because we all learn more from making mistakes. We need to know when a mistake or error happened in order to figure out how to prevent or correct it.

And even with the most highly skilled programmers, errors will still happen. But there are automated tools available to analyze and find those mistakes and correct them before the software is released. Investing in and training workers on those tools adds to the cost of a development project, which feeds back into the false belief that security can be added later. Developers also minimize potential problems such as glitches or bugs that can be easily fixed when they occur. Another prevalent misstep regarding the software development life cycle is understaffed support for a system or application once it’s in production. This is analogous to thinking you can buy a car and not need skilled mechanics to maintain it or make repairs.

What’s the Answer?

Altogether, the inattention to security in the software development lifecycle is an invitation to criminal hackers, in addition to increasing the probability of unforeseen outages. You may ask, how can the hospitality industry, which relies heavily on cloud service providers and other third-party software applications, take any actions to address secure software development? The short answer is to leverage your supplier relationships and force attention to this important aspect of their products and services that can potentially impact and damage your company.

Begin by identifying and listing all the key and major software products you use and the potential impact if something goes wrong. Don’t exempt or exclude security software tools or products. For each of them, determine how quickly you could back out of a change and restore operations. Next, take advantage of the excellent guidance published by the National Institute of Standards and Technology (NIST) in its Special Publication (SP) 800-218, Secure Software Development Framework (SSDF) Version 1.1. According to the document: “Because the framework provides a common vocabulary for secure software development, software purchasers and consumers can also use it to foster communications with suppliers in acquisition processes and other management activities.”
‍

SOME STEPS YOU CAN TAKE

• Incorporate contractual requirements with both existing and new suppliers that they will implement this framework by a certain date or that they already have it in place.
• Consider having your software providers assessed or independently audited on these practices to identify and remedy gaps.
• Require your supplier to document and share training records for programmers, product, and project managers involved in software development. NIST offers relevant and useful guidance on Third Party Risk Management in its publication SP-800-161. NIST publications are uncontroversial, accepted globally, and can be downloaded at no charge at: https://www.nist.gov/cybersecurity
• Raise the question of a “rapid release” for updates or patches and agree with each supplier on whether you will accept such releases.
• Ask what tools are being used for code reviews to identify potential errors.
• Require documentation on the number of resources supporting the product(s) that you use.
  ‍

QUESTIONS FOR EACH SOFTWARE SUPPLIER

• The name of and contact details for their chief information security officer. This may seem silly, but if their response is a product manager or client service executive, you should consider that a major red flag.
• Who in their organization is accountable for software errors that impact security?
• What other information security standards for risk controls have they implemented?

‍