Testing Fast MDA Traceroute
Join our monthly community call on June 6, 2024 at 11am ET to discuss Dioptra’s recent work on fast-mda-traceroute and considerations that have gone into its design and testing.
Measuring the Quality of an Individual's Connection
Interested in metrics that measure an Internet connection’s quality from the end user perspective? Join us at our April M-Lab Community Call on April 4, 2042 at 11am EDT.
How should Internet quality be measured? Join us for our Community Call on February 1, 2024
Access to high speed Internet does not necessarily mean access to high quality Internet. But if not speed then what? How should we measure Internet quality? Join our monthly community call on February 1, 2024 at 11am ET to discuss and contribute to our latest research initiative, sponsored by Internet Society Foundation.
Join us at the IMC 2023 Hackathon on Network Interference using Open Data
Are you attending the Internet Measurement Conference (IMC) 2023?
Guest Post: Underload: Today’s Problem of Internet Congestion Control
Guest blog post by Michael Wezl (Full Professor at University of Oslo, Norway) discussing the use of Measurement Lab’s Neubot data to research Internet congestion control.
M-Lab at SIGCOMM 2022
Measurement-Lab was excited to be represented at this year’s SIGCOMM conference with a tutorial and conference talk in the Best of CCR session. This post overviews M-Lab’s activities at SIGCOMM and includes references for the tutorial materials.
M-Lab Hackathon at IMC - Register by September 23rd!
Come analyze Internet performance data with M-Lab at ACM Internet Measurement Conference 2022!
Release notes for Unified Views with exposed Row Filters
Monday August 15th we will be updating the NDT unified views to provide researchers with access to the row filter logic used to exclude rows that are not believed to be representative measurements of network performance. This change enables interested researchers to better understand MLab’s data filtering and to easily craft alternate filters to verify our assumptions, or to filter data to address questions other than network throughput.
Extracting Single-Path Traceroutes
In response to community feedback, we’ve developed a proof of concept tool that examines and extracts single-paths scamper‘s MDA traceroutes.
M-Lab Fellowship Presentation Videos and Reports
Recordings of the 2022 M-Lab Fellowship Presentations and Reports.
M-Lab Fellowship Presentations
Come learn about the exciting research that the M-Lab Fellows completed during their fellowship!
An Overview of M-Lab’s Traceroute Data
Summary
This blog post is a report on M-Lab’s traceroute data so the community can have a better understanding of what the data we make available includes.
Updates to Geo Filters
Changes to our geographic annotations have resulted in changes to geographic filters.
Request for Comments - User Privacy & Research Usability
On March 16 we hosted a town hall discussion regarding M-Lab’s collection of IP addresses and asked for feedback on potential alternatives. We are accepting comments on the techniques we proposed until April 1.
User Privacy & Research Usability Town Hall - March 16, 2022 at 11:00a Eastern
Presentation on the benefits and challenges of M-Lab’s collection of IP addresses and feedback solicitation for potential alternatives
ARC of Research pt. 2: Exploring Data Sources Relevant to Our Questions
In this second “analysis recommendations in context” post, we will explore the refined research questions from the first post, resulting from our discussion of how to design specific questions with understanding of available data source(s) and the context of what each contains. We emphasized the importance of selecting a data source that matches the goal of the research question. This is critical for analyses of broadband measurement data, particularly when the research goal is to compare the results to one another, to national broadband standards or specific funding requirements, or to align with advertised terms of ISP service.
Analysis Recommendations in Context - ARC of Research pt. 1: Asking the Right Questions
A while back, our team published some analysis recommendations for anyone working with our data from the Network Diagnostic Tool (NDT), comparing it to other Internet measurement data sets, and drawing conclusions or inferences about the data. These recommendations are intended to provide guidance about analyzing crowdsourced data, because we know that it’s easy for analyses to end up with what looks like a striking comparison or finding, but that may not actually be supported by the underlying measurements or data. But because recommendations are only that, we’re now beginning a series of posts to unpack those recommendations with some context and examples. First, we’ll recap our previous recommendations post with more context, and finish with an example that we’ll continue working with in subsequent posts.
M-Lab Research Fellows - Spring 2022
M-Lab is pleased to share our selection of Research Fellows for Spring 2022, generously supported by Internet Society. The fellows were selected for the merit, rigor, and feasibility of their proposed project as well as their research’s relevance to improvement for end-user Internet performance. Read below for more information about the fellows and their proposed projects
Open Call for M-Lab Research Fellows - due Monday, December 6
How can M-Lab’s data and platform be used to improve the experience of the Internet for the end user?
Measurement Lab is seeking three Research Fellows to expand Internet performance research beyond the measurement and optimization for bandwidth. Fellows will utilize M-Lab’s longitudinal, open dataset and/or platform to identify under-recognized Internet performance metrics that can be used to improve end user performance.
M-Lab's Murakami Tool - Supporting Structured Research Data Collection from the User Perspective
Many people know M-Lab and our TCP performance test, NDT, from running it in a web browser. Perhaps the largest single source of NDT tests comes from its integration by the Google Search team. While M-Lab is known for the large volume of crowdsourced test data resulting from people running our tests, over the past few years we’ve developed new ways to run our tests and open source Internet measurement tests from other platforms using a tool we’ve called Murakami.
Evolution of M-Lab's Geographic and Network Annotations
In our recent roadmap post, we shared a list of milestones that the team is working on this and last quarter. Our Datatype migration and Standardized Columns milestone references the gardener service, which maintains and reprocesses M-Lab data, as well as the UUID annotator, that generates and saves per-connection metadata as annotations to user-conducted measurements. This post provides more detailed information about how these services have annotated measurements with geographic and network information in the past and present, and expands on what current work is happening now as mentioned in our roadmap post.
Using M-Lab Data in Broadband Advocacy and Policy
For researchers, policy makers, governing bodies, advocacy groups, or anyone who wants to understand M-Lab data and how it compares to other internet measurement data sets, M-Lab has begun to develop recommendations for appropriate use of our data in their analyses and reports.
Evolution of NDT
NDT measures “bulk transport capacity”: the maximum date rate that TCP can reliably deliver data using the unreliable IP protocol over an end-to-end Internet path. TCP’s goal is to send data at exactly the optimal rate for the network, containing just the right mix of new data and retransmissions such that the receiver gets exactly one copy of everything. Since its creation, the TCP protocol has consistently made improvements to the way it accomplishes this task, consequently, NDT has also incrementally changed to reflect these improvements. The most recent improvements, including support for TCP BBR, are available in ndt7. On July 24th, we announced the start of migration of NDT clients to the latest protocol version. As of today, approximately 50% of clients are using ndt7. As the ndt7 measurements become the majority of the NDT dataset, the M-Lab team is considering what we do and do not know about whether and how changes to the NDT protocol have affected M-Lab’s longitudinal NDT dataset over time.
Migrating NDT clients to ndt7
Following the general availability of the ndt7 protocol, we will be working with NDT client integrators to support their migration to ndt7. As they do, the NDT dataset will shift from predominantly ndt5 to predominantly ndt7. As part of assessing our readiness for this larger effort, a pilot was started on July 8.
Introducing ndt7
The new ndt7 protocol for the Network Diagnostic Tool (NDT) is now generally available on the M-Lab platform. Since 2009, NDT has been the premier TCP performance measurement service test hosted by M-Lab. During its history on the platform, NDT has produced the largest test volume to date, spanning the longest history. Since late 2018, M-Lab has worked with researcher Simone Basso to develop the ndt7 protocol and archival data format.
Measurement Observations on Network Performance During the COVID-19 Pandemic in Northern Italy
OONI was recently invited to participate in a NetGain Partnership webinar (titled “Surging Demand and The Global Internet Infrastructure”) to discuss the changing landscape for internet infrastructure and technology in the wake of the COVID-19 pandemic.
As part of our preparation for this webinar, we looked at network performance measurements collected from northern Italy over the last months (i.e. when Italy was hit hard by the COVID-19 pandemic) in an attempt to understand whether and to what extent there was a correlation between increased internet use and reduced network performance. As our observations may be of public interest, we decided to share them through this blog post.
M-Lab 2.0 Platform: Global Pilot Assessment
We deployed the new M-Lab platform to 1/3rd of the M-Lab fleet, and now we need to assess whether or not it is a performance regression, relative to the old platform. As long as we can be sure the performance of the new platform does not constitute a regression, then we can roll out the new platform and be confident that we have not made anything worse.
M-Lab 2.0 Platform: Global Pilot Entry
For a while, we’ve been developing M-Lab 2.0 [1, 2]. This month, we are launching a global pilot for the new software stack. The changes include:
- Stock Linux 4.19 LTS kernels with modern TCP and Cubic congestion control
- Standard instrumentation for all experiments using tcp-info
- Virtualization and container management using Kubernetes and Docker
- Reimplementation of the NDT server
M-Lab 2.0 Platform Migration Update
Last year, we outlined our plans to Modernize the M-Lab Platform. This year, we’re bringing them to life. Here’s a summary of why the platform update is so valuable and what you can expect throughout the year.
AFRICOMM 2018 and AFRINIC-29
AFRICOMM 2018. Left to right: Amreesh Phokeer (AFRINIC), Josiah Chavula (University of Capetown), Georgia Bullen (M-Lab), Antoine Delvaux (perfSonar), Stephen Soltesz (M-Lab).
In late November 2018, M-Lab was invited to the Internet Measurement Workshop at AFRINIC-29 in Tunisia and to give a keynote about M-Lab and open internet measurement at AFRICOMM 2018 in Senegal. Both trips were a fantastic opportunity to deepen our relationship with researchers focused on the African Internet, learn more about how our platform is serving community needs, foster conversation around open Internet measurement, and identify opportunities for further collaboration, research and tool development to better support the Internet measurement, research and policy community in Africa.
SIGCOMM 2018 Hackathon Outcomes
M-Lab had the pleasure of attending the first ever SIGCOMM hackathon on August 25, 2018, at the Nokia Skypark headquarters in Budapest, Hungary. The hackathon, sponsored by Nokia, DECIX, and Netflix, invited network research faculty, students, and industry professionals from around the world to form teams and develop tools, new features or analyses during the Saturday following the SIGCOMM conference.
M-Lab 10 Year Anniversary Event
We’ve reached a point in human history where, for many of us, the Internet has become a standard presence in our daily lives. In the United States, the Internet is simply part of how many of us engage with the world. In other countries (and parts of this one), the Internet remains unaffordable, unreliable, and inaccessible. The Internet unites us in many ways, and at the center of work on the future of the Internet is a dedicated community of experts exploring the questions that will move the Internet to the next level of its evolution: What is an open Internet? What is a healthy Internet? What factors contribute to the Internet ecosystem’s health?
Switch discard investigation for early 2018
On February 1st, 2018, during a regular data quality review, we identified an increase in switch discards at sites with 10Gbps equipment connected to 1Gbps uplinks. We used our switch telemetry data to assess whether there were any negative consequences for tests contained in our SideStream or NDT data sets, and then we used the same data sets to determine whether our remediation strategy had any negative effects. In both cases, we found no observable effects, indicating that everything was below the noise floor for Internet performance data.
SIGCOMM 2018 Hackathon
- When: Saturday, August 25, 2018
- Where: SIGCOMM, Budapest, Hungary
M-Lab will be at this year’s SIGCOMM hackathon in Budapest, Hungary. Check out the projects we’re hoping to work on with folks on the hackathon wiki.
M-Lab is Turning 10!
- When: Aug. 7, 2018 - Aug. 8, 2018, 9AM - 5PM
- Where: New America, 740 15th St NW #900, Washington, D.C. 20005
Measurement Lab is turning 10! On August 7 and 8, we look forward to gathering the Measurement Lab community to showcase how the platform has evolved, learn from you about how you are using M-Lab, and discuss how we plan for the next 10 years of measuring the Internet and providing public data to the world. So much has changed over the last 10 years (and that’s not just our expanding volume of longitudinal data!), come celebrate, brainstorm, analyze, and share with us.
New DISCO switch telemetry dataset
Since June 2016, M-Lab has collected high resolution switch telemetry for each M-Lab server and site uplink.
Originally designed to detect switch discards from server traffic microbursts, we now support the DIScard COllection (a.k.a. DISCO) dataset as a standard M-Lab BigQuery table: measurement-lab.base_tables.switch
Say Hello to Murakami
The Measurement Lab team has always tried to make it as easy as possible to run network measurements. Currently, most users run tests either directly from the M-Lab website, or through a 3rd party integration. Over the years, many users have requested the ability to run tests on a regular basis, e.g. daily or weekly to collect data over time. Today, we’re releasing a tool that will help you do just that.
Monitoring Interconnection Performance Since the Open Internet Order
Introduction
As a platform committed to producing empirical data for the public, Measurement Lab (M-Lab) has historically supplied regulators and other governmental entities with technical facts pertinent rule-making processes. In our February 2015 submission to the FCC’s Open Internet docket, we committed to research on the state of broadband and performance impact of interconnection in the United States. Earlier this year, the FCC began the process of re-evaluating its authority over broadband Internet services, and opened a Notice of Proposed Rulemaking. This blogpost is a shortened version comments that M-Lab filed in the docket regarding its continued research on the impact of interconnection on consumer broadband. The full filing in the FCC docket includes an elaboration of our research with additional supporting evidence and charts.
Transitioning to a New Backend Pipeline and Data Availability
M-Lab data is collected from distributed experiments hosted on servers all over the world, processed in a pipeline, and published for free in both raw and parsed (structured) formats. The back end processing component for this has served us well for many years, but it’s been showing its age recently. As M-Lab collects an increasing amount of data thanks to new partnerships, we have been concerned that it will not be as reliable.
New Opportunities for Test Deployment and Continued Analysis of Interconnection Performance
In late April, Internet2, with contributions from M-Lab developers, released an updated version of the Network Diagnostic Tool (NDT), M-Lab’s most widely deployed performance measurement test. In addition to improved stability, the updated version of NDT makes it possible to measure network performance from a web browser without the need for browser plugins. This change removes a layer of complexity and hassle for developers and users. Anyone interested in Internet measurement can now create their own NDT client targeted for specific user populations, regions, and research objectives, using only HTML and Javascript.
Internet Observatory Updates and Improvements
Last October, Measurement Lab released the Internet Observatory, a data-visualization tool that enables consumers, policymakers, and researchers to better understand the impact of ISP relationships on Internet access and performance. The Observatory provides easier access to M-Lab’s rich dataset on network performance to reproduce the analysis in our report on “ISP Interconnection and its Impact on Consumer Internet Performance.”
M-Lab at RIPE
On Thursday Nov. 6, Collin Anderson gave a talk at the RIPE meeting in London. Collin has been immersed in the M-Lab data for some time, and helped lead the recently published technical report, ISP Interconnection and its Impact on Consumer Internet Performance. He presented a lyrical overview of these research findings for the assembled crowd of network operators and researchers, which is archived on the RIPE website.
Research Updates: Beginning to Observe Network Management Practices as a Third Party
Prior to the publication of our recent report, ISP Interconnection and its Impact on Consumer Internet Performance, we shared review copies with members of the research community, broadening the set of eyes on our methods and challenging our assertions. We received valuable feedback that allowed us to improve what we published, along with a list of research questions that we will be exploring and documenting over the coming months in our Research Updates series of blog posts adding insights to Research Reports.
ISP Interconnection and its Impact on Consumer Internet Performance: Introducing A New M-Lab Consortium Technical Report
We are happy to announce the release of a long-term collaborative research effort using M-Lab’s data to understand how interconnection impacts end-user performance. The report, ISP Interconnection and its Impact on Consumer Internet Performance examines years of network measurement data from across the United States to determine the effects of network interconnection on the Internet performance of customers subscribing to specific access ISPs. Alongside this report, we are also pleased to release the Internet Observatory – a dynamic data visualization tool that will allow consumers, policymakers, and researchers to better understand the impact of ISP relationships on their own Internet access and performance. The Internet Observatory will be updated regularly, allowing future monitoring and comparison against past performance.