A wake-up call for security on third-party packages
Yes, I’m looking at you NuGet, PIP, Maven, NPM, Container Registry …
In light of the Solarigate debacle that we witnessed last year, it’s a reality how important is the security of the supply chain and its adequate risk management.
Supply chain attacks, when it comes to software, it’s not new, although as cyber-attacks continue to become more sophisticated, we can expect novel mechanisms on how threat actors penetrate our defenses; and clearly supply chain is one of those vectors.
Described as an invisible risk by FireEye in their article titled “The Invisible Risk — Why Supply Chain Is a Top Security Priority and What to Do About It”, it highlights the impact of the so-called Heartbleed, a critical security bug in the OpenSSL cryptographic library that was disclosed in 2014 and it impacts for millions of users and business.
Although Heartbleed can be considered as a software implementation mistake (which is documented as a “programming mistake in popular OpenSSL library that provides cryptographic services such as SSL/TLS to the applications and service”), this reliance on third-party components poses significant risks for organizations. It is essential for organizations to put in place a process for monitoring the risk of their supply chain.
Additionally, GitHub folks published a great article where they tackle Supply Chain security issues in an article titled “Secure at every step: What is software supply chain security, and why does it matter?”. In that article, they highlight some interesting points that are aligned with the way of thinking of this article.
- A software supply chain is anything that affects your software.
- Your software supply chain risks are inherited from your dependencies
- Supply chain compromises are real but rare
- Unpatched software is still the main culprit
Background: NuGet Packages, something really bugs me!
I’m a Software and Security Engineer, so writing code is one of my hobbies, and thinking in terms of security and threat models is a natural way of thinking while interacting with a given programming language or technology.
I have previously described what I use to call “Dependency-Driven Development”, to define the way we, the software engineers, create, use, and distribute libraries that other people can use. This ecosystem is all fine, is a way to contribute to making better software and we definitely need better and secure software. Modern programming languages provides already a mechanism to interact with such libraries via Package Management tools. This time, I will be focusing on Microsoft’s Package Management for .NET called NuGet.
As described by Microsoft “An essential tool for any modern development platform is a mechanism through which developers can create, share, and consume useful code. Often such code is bundled into “packages” that contain compiled code (as DLLs) along with other content needed in the projects that consume these packages”.
NuGet is defined as: “ For .NET (including .NET Core), the Microsoft-supported mechanism for sharing code is NuGet, which defines how packages for .NET are created, hosted, and consumed, and provides the tools for each of those roles.
Put simply, a NuGet package is a single ZIP file with the .nupkg extension that contains compiled code (DLLs), other files related to that code, and a descriptive manifest that includes information like the package's version number. Developers with code to share create packages and publish them to a public or private host. Package consumers obtain those packages from suitable hosts, add them to their projects, and then call a package's functionality in their project code. NuGet itself then handles all of the intermediate details”.
At the time of writing this blog post (January 2021) here are some statistics pulled from their portal:
- Over 2 million package versions
- Over 237k unique packages
- Over 77 billion downloads
And here are some more recent statistics:
Creating the first NuGet Package: How we get here?
Last year, I was working on Security Operations with the Incident Handling team and they really needed a fast way to analyze a large dataset of unique IP Addresses that were captured during the analysis of an incident to determine whether or not those IP Addresses were suspicious based on some threat intelligence feed lookup.
Particularly, they were interested in doing a quick lookup on a Threat Intelligence service called AbuseIPDB. So the task was simple, get the dataset, and perform a database lookup on AbuseIPDB to add some weight to their risk approach, that is if the IP Address has been reported by that service.
From their definition, AbuseIPDB is described as “AbuseIPDB is a project dedicated to helping combat the spread of hackers, spammers, and abusive activity on the internet”.
The good thing about this service is that they provide great API documentation and their pricing model allows you to perform up to 1000 queries per day; this limit will increase depending on the plan you are in but in principle, they provide some good information about IP Address reputation.
From a development point of view, is always easier to have a native way to consume the API; doing quick research in the NuGet registry, I was not able to find (at that time) a NuGet package that already did this for me, so I decided to write one to consume this AbuseIPDB API.
Although I have contributed to a few open-source projects that are eventually exported as a NuGet Package (such as the FluentSharp.CoreLib, which provides thousands of extension methods for increasing productivity), I haven't had the opportunity to publish a package on my own, so this was the time.
Once the library was written, I developed a C# Windows Forms application for the Incident Handlers so they could provide an AbuseIPDB API Key and upload an arbitrary dataset (CSV file) of unique IP Addresses. The process then would do a lookup at the Threat Intel service and provided a report classified by risk based on the Confidence of Abuse metric provided by AbuseIPDB.
This is the risk model I developed using quartiles:
Additionally, following this risk model, I used the numbers 21, 47, 73, and 99 to avoid dealing with boundaries.
Once the requirement was fulfilled, the package was published to NuGet and it was ready to be reused.
Not Adding NuGet Package Documentation (or How Not to be a Good Citizen)
Publishing a NuGet Package is relatively simple, just following the Microsoft guide will help you to understand the process.
Just one thing: I did not provide any sort of documentation; the GitHub repository is empty, yet the package has gotten 399 downloads as per the statistics page.
NuGet Packages encourages the following properties:
- Project Site (which most of the time is GitHub but it can be any other place where the code is hosted)
- License
- Contact Authors
- Documentation on how to use it.
In my case I pointed it to an empty GitHub repository :), actually, the NuGet package definition is very poor as you can see in the below image:
Here is the empty GitHub repository:
By looking into NuGet statistics I can see several security problems with this approach of publishing packages, here is my reasoning:
- What is the security scrutiny followed by NuGet when a package is published without promoting open-source censorship?: And when it comes to Open Source, enforcing a process like this can be seen as a way to censor open source initiatives and ideas. Moderation and Open Source are not always two words that can live in the same sentence, the question then is what security controls already implements NuGet or should be implemented when a package is published in a way that security checks such as secret leakage, code vulnerabilities, highly obfuscated code, etc can run when the package is published?
- Even though NuGet publishing requisites are fulfilled, how can we guarantee that the package being pushed to NuGet exactly matches the source code found in the GitHub repository?: As far as I can see, there is not an easy way to make sure that the package being downloaded from NuGet matches the definition of the code that was pushed to the GitHub repository. From a security risk standpoint, one might think that an attacker could easily mislead a developer by keeping a well-documented repository on GitHub but publishing highly obfuscated code at the time the package is generated and deployed to NuGet. Truth is the only way to guarantee this integrity is by performing a deep code review of the DLL file by decompiling it and compare it with the one in the repository. This process seems to be a good candidate for incorporating blockchain integrity checks, don't you think?. Another process would be comparing cryptographic hashes of the package and compare them with the GitHub releases. Signing NuGet Packages gives some protection (aka protection against content tampering) but I do not think it addresses the underlying issue when the author or the package is a threat actor.
- Integrity checks are a good candidate for a Security Engineer, but are they good for Software Engineers?: Not to minimize the capabilities of brilliant Software Engineers, but had I been a Software Engineer, I’m aware that strict deadlines for Software Engineers are not a secret; can we expect that developers have the time to do a security code review on every single dependency they are going to use? Even if they do, how can we make sure that such a process is performed on each version of the libraries the code depends on during the development life-cycle?
- Does automatically code scans, CI/CD, and DevSecOps fixes this issue?: Although automation and DevSecOps have proven to be efficient methodologies to improve the Security feedback and deploying secure applications more often, I’m a bit concerned about advanced topics such as business logic flaws and highly obfuscated back doors in the code. Even when we implement rules to detect vulnerabilities, as we saw on the Solarigate post mortem, the code threat actors included in the original software look legitimate and it could confuse even the trained eye. Powershell and .NET malware are rapidly evolving as described by Kasperky research and threat actors will find new avenues for compromising organizations.
What is the NuGet Security State-of-the-Art and how it can be improved?
Phil Haack covered some of these concerns back in 2013 in a great article titled Trust and NuGet, which I highly recommend you to read.
One of my concerns is that I could not find a centralized repository for understanding Security Advisories for NuGet Packages. Let’s remember that data is the new oil (when refined) so having some visibility about security issues targetting packages (similar to what NPM does via the NPM Security Advisories https://www.npmjs.com/advisories), will allow engineering teams to be careful when reviewing a package for identifying known vulnerabilities. I have analyzed the NPM Advisories in the past to get some interesting insights about trends or type of vulnerabilities that are being disclosed and the packages they targets. See The power of visualizing npm security advisories and A better way to visualize npm Security Advisories so you can understand what I mean by data analysis.
If there is already an existing Advisory for this, please let me know. I’m aware of the efforts of Erlend Oftedal who used to maintain the SafeNuGet project at OWASP, but it seems the repository is not being maintained since 2014, I wrote this article back in 2014 “SafeNuGet in action: Finding packages with known vulnerabilities on top of Visual Studio 2012”.
Some of my concerns when it comes to NuGet security are:
- I could not find in GitHub a topic related to Application Security: There are 50 repositories for NuGet and none of them are focused on security. Again, If my appreciation is incorrect, show me the data as I’m very interested in understanding and analyzing vulnerabilities and their remediation ratio.
- What are the security checks NuGet performs when a package is published?: I could not find any documentation about any security check/scan being performed by NuGet when a package is published. From the development angle, I recognize that we can enhance the status quo by performing scans and security checks and lastly generate and push the NuGet Package via GitHub Actions. But my point is from a defensive standpoint when the person pushing the package is a threat actor who is least interested in that their package is being inspected. I understand that ultimately, it comes to risk assessment, your team can use packages that are from a trusted developer, with a good number of downloads, that is actively maintained, and that the code is easier to read and the dependencies for that package are cristal clear.
- How many packages available at NuGet are vulnerable on purpose?: Judging by what we have seen in other Package Managements (Typosquatting and data exfiltration on npm packages (aka Hijacked Environment Variables)), one might think it is likely this pattern has been backported to other Package Management Tools. Maybe data analysis on the packages via the NuGet API can help us to find normal and abnormal behavior on how those packages are published. For example, being able to spot a user that has never published a package (a potential new account) but is publishing several packages at once. Is this suspicious? Well, maybe but could be seen as an anomaly that requires further review. Another use case is identifying potential typosquatting attempts, and what we learned from NPM is that this is a common technique because their ultimate goal is to deceive the developer so they install the wrong package and if we concatenate this with a poor third party library policy from the organization, it becomes an important risk.
- With the rise of Powershell and Microsoft .Net malware and Living Off the Land, is there a way to monitor the PowerShell execution via NuGet?: This GitHub issue indicates that PowerShell scripts are now deprecated from .nuspec definition. I might need to do some more research on this topic. In fact, some changes added on NuGet version 3 are very well documented:
“Powershell script support was modified to no longer execute install and uninstall scripts, but init scripts are still executed. Some of the reasoning for this is the inability to determine which package scripts need to be run when not all packages are directly referenced by a project. Without an explicit install or uninstall for a dependency package, it is impossible to determine when these scripts should be run. We do not believe there is a reliable and repeatable way to execute these scripts in this package reference model. Additionally, as we want to ensure that NuGet provides a repeatable experience on the command-line this forces packages to run without script capabilities. We plan to introduce features to address many of the common tasks that scripts provided.
Additionally, running NuGet outside of Visual Studio at the command-line or on other operating systems cannot execute these Powershell scripts as they relied on Visual Studio automation. Powershell is in NuGet to provide a shim to allow for missing features to run inside of Visual Studio. Moving forward, we plan to introduce these features to provide this functionality. Finally, many of these scripts relied on Visual Studio APIs that could break across versions of Visual Studio.”
I need to do further research about what threat models could result by running PowerShell init scripts.
Final Thoughts
This article not necessarily provides answers to some of the security questions related to the NuGet Package Management but at least provides some general concerns that I can see from my understanding of this model. This is a work in progress as I need to do even more research and analysis so if some of these thoughts do not make sense, do you have any discrepancy or my view of this is incorrect or flawed, please let me know.
Ultimately we really want to make sure that this amazing ecosystem of sharing code with the community is secure and that it cannot be abused by threat actors that look for mechanisms to penetrate our defenses.
Additional References
https://michaelhidalgocr.blogspot.com/2019/08/the-power-of-visualizing-npm-security.html
https://michaelhidalgocr.blogspot.com/2019/06/a-better-way-to-visualiza-npm-security.html
https://michaelhidalgocr.blogspot.com/2017/08/typosquatting-and-data-exfiltration-on.html
https://michaelhidalgo.medium.com/security-risk-on-third-party-packages-7610a45b7a8
