Articles

Remediation at Scale: Open Source Vulnerabilities at PayPal

Apr 5, 2018 11:42:20 AM By Derek E. Weeks GettyImages-492819233

Laksh Raghavan of PayPal has written about how his company managed their response to a critical Java open source component vulnerability in one of their applications. Similar to the vulnerable Struts2 component implicated in the Equifax breach of 143 million consumer records -- and more recently in the Zealot campaign -- the vulnerability at PayPal would allow for the most critical form of software vulnerability: remote code execution.

Laksh described how their team discovered the vulnerability through an alert originating from their bug bounty program. He then addressed how the team remediated the issue at scale, including short- and long-term efforts. Critical to their success and speed of remediation were code repositories, automated application analysis, and prioritization of fixes.

The first thing they did was to take an inventory of their applications to assess which ones used the vulnerable component. Laksh commented:

  • "If your organization has a good app inventory, it makes your life easy.
  • Validate the manual inventory with automated scans and make sure you have a solid list of servers to be patched.
  • If you have a centralized code repository, it definitely makes things easy.
  • Make sure you don’t miss one-off apps that are not on your core Java frameworks.
  • Additionally, reach out to the security and product contacts within all of your subsidiaries (if they are not fully integrated) and ask them to repeat the above steps."

He then recommended the following remediation steps specific to their use of the vulnerable commons collection component, but application to many vulnerable components:

"Short-term

  • First up, drive the patching of your high-risk Internet facing systems – specifically COTS products that have published exploits.
  • For custom apps that use the vulnerable component, you can either remove the vulnerable classes if not used or upgrade to the latest version.
  • If the owners of those systems are different, a lot can be accomplished in parallel."

"Long-term

  • Again, this vulnerability is not limited to commons-collections or Java per se. This is an entire class of vulnerability like XSS. Your approach should be holistic accordingly.
  • Your best bet is to completely turn off object serialization everywhere.
  • If that is not feasible, this post from Terse Systems has good guidance on dealing with custom code."

While we can't turn back time on the Equifax breach, we can learn from others who performed successful discovery and remediation of their vulnerable components at scale.

For those interested in assessing what, if any, vulnerable open source components have been used in your applications, Sonatype, GitHub, and OWASP both offer variations of free OSS analysis services. 

Readers interested in the full PayPal remediation process can read Laksh's original blog here.