[Pl-seminar] npm-follower: A Complete Dataset Tracking the NPM Ecosystem. Donald Pinckney on Friday

[Luna Phipps-Costin]

Hi all -
Happy to declare we'll be once again doing seminar on Friday.

Speaker: Donald Pinckney
npm-follower: A Complete Dataset Tracking the NPM Ecosystem
Abstract:
Software developers typically rely upon a large network of dependencies to build their applications. For instance, the NPM package repository contains over 3 million packages and serves tens of billions of downloads weekly. Understanding the structure and nature of packages, dependencies, and published code requires datasets that provide researchers with easy access to metadata and code of packages. However, prior work on NPM dataset construction typically has two limitations: 1) only metadata is scraped, and 2) packages or versions that are deleted from NPM can not be scraped. Over 330,000 versions of packages were deleted from NPM between July 2022 and May 2023. This data is critical for researchers as it often pertains to important questions of security and malware.

In this talk I present npm-follower, a dataset and crawling architecture which archives metadata and code of all packages and versions as they are published, and is thus able to retain data which is later deleted. Additionally, npm-follower makes unique database design decisions enabling deeper analyses of versions and dependencies. The dataset currently includes over 35 million versions of packages, and grows at a rate of about 1 million versions per month. The dataset is designed to be easily used by researchers answering questions involving either metadata or program analysis. Both the code and dataset are available at https://dependencies.science<https://dependencies.science/>.
Time: 12 to 1:30* Friday, November 3rd
Place: Forsyth 237

* We have lunch from 12 to 12:30. Note that this time is unchanged.
-------------- next part --------------
HTML attachment scrubbed and removed