Category Archives: Research

An Open Source Toolchain for Recording Interviews

Posted on by

TL;DR: When recording through conferencing system is unavailable, I use three open source tools to record interviews: OBS Studio, Blender, and Audacity.

As a qualitative researcher, I enjoy engaging with people directly and occasionally I want to record an interview. I conducted XL (roman numeral) interviews as part of my PhD and I figured out how to capture the audio and transcribe it. A primary concern is the quality of a recording. Because I am undoubtedly not alone, I describe here lessons learned and my open source tool chain.

My first attempt at recording an interview involved an old-school telephone on speakerphone and my smartphone with an audio recording app. The audio quality was abysmal. My voice was clear, but my informant’s voice was distorted through the analog transformation between phone speaker and smartphone microphone. I never tried this again.

The best solution is using the conferencing system that my university provides. The system allows my informants to connect via app, browser, or phone. In my experience, about 90% of informants join via app and activate their camera, which makes for a great conversation. The recording quality is only dependent on the microphones quality. Sometimes, I ask informants if they have an alternative they switch between a headphone to a built-in microphone.

This optimal solution is not always practical and some informants prefer to use alternative ways to connect, such as Skype, Jitsi, Meet.me, GoToMeeting, or a conferencing system they control. When I need to make a phone call, I use Google voice from my laptop. I developed a toolchain of open source software that allows me to record an interview in such settings. I use three tools: OBS Studio, Blender, and Audacity.

OBS Studio (Open Broadcaster Software Studio) allows to capture anything on my screen and record in-going and out-going audio. I simply record the audio output from my computer and my microphone input during an interview. The quality is the same as directly recording it through a conferencing solution. The downside is that OBS requires computing resources and at times bogs down my laptop. I end up with a *.flv video file that contains the desired audio.

Blender is a 3D modeling software but has powerful video editing capabilities. I recommend Mikeycal Meyers’ tutorial on how to get setup. Blender is overkill for extracting audio, but it is a tool I know. Once the *.flv file is loaded into Blender, I export the audio, without doing any editing. I end up with a *.flac audio file.

Finally, I use Audacity to cut the audio and tweak it. I edit any audio, whether recorded by OBS or a conferencing system, with three goals in mind. First, I cut the beginning and end because small talk is not important to my analysis. Second, I check to make sure my informant is audible. When the audio is too quiet, I boost it using Effect –> Compressor… for the whole audio or Effect –> Amplify… for small sections. Third, I shorten pauses using Effect –> Truncate Silence… which saves fees on transcription services. I export the final audio as an *.mp3 audio file.

After generating a good quality audio file, I need to transcribe the interview. I transcribed one interview myself using OTranscribe and it took me six hours for a one hour interview because I worked to get every word and half sentence and expression right. I quickly learned that this level of detail is hindering me and a more streamlined transcript is easier to analyze. After making this experience, I value having a research grant that pays for a transcription service and I am a returning customer of rev.com.

My first German paper

Posted on by

I am super excited!

For four years, I publish academic papers. All in English. Today, I have my first paper in my native language, German.

A big thanks to UNO Criss Library for funding the Open Access fees! It is great to have the amazing support for a future of a more open science.

I am very happy to have an amazing author team of Malvika Rao, Don Marti, Andy Leak, and Rich Bodo. They developed the core of the idea before I joined them and created a welcoming environment for me to learn so much and expand my horizon. – THANK YOU!

A big thanks to all of my test readers who provided invaluable feedback and helped me fix mistakes. Admittedly, I German grammar rules are different from English.

Without further due, the title and abstract of the paper:

Marktplatz zur Koordinierung und Finanzierung von Open Source Software

Open Source ist ein zunehmend beliebter Kollaborationsmechanismus für die Entwicklung von Software, auch in Unternehmen. Unsere Arbeit schafft die fehlende Verbindung zwischen Open Source Projekten, Unternehmen und Märkten. Ohne diese Verbindung wurden Koordinations- und Finanzierungsprobleme sichtbar, die zu schwerwiegenden Sicherheitslücken führen. In diesem Paper entwickeln wir acht Design Features, die ein Marktplatz für Open Source haben sollte, um diese Probleme zu beseitigen. Wir begründen jedes Design Feature mit den bestehenden Praktiken von Open Source und stellen einen Prototypen vor. Abschließend diskutieren wir, welche Auswirkungen die Einführung eines solchen Marktplatzes haben könnte.

Weiterlesen…
Der Artikel ist open access und bei Springer verfügbar.

Marketplace to Coordinate and Finance Open Source Software

The popularity of open source as a collaboration mechanism for developing software is increasing. Organizations increase their engagement. In our work, we draw the missing connection between open source projects, organizations, and markets. Without this connection, we have seen severe software vulnerability result from coordination and financing breakdowns. In this paper, we develop eight design features that a market place for open source should have to address these breakdowns. We develop the design features based on literature about the practices of open source. We present a prototype and discuss what implications would result from implementing such a market place.

Read more… (in German) 
The paper is open access and available from Springer.

Full reference:

Link, G. J. P., Rao, M., Marti, D., Leak, A., & Bodo, R. (2018). Marktplatz zur Koordinierung und Finanzierung von Open Source Software. HMD Praxis der Wirtschaftsinformatik. https://doi.org/10.1365/s40702-018-00474-6

Is open source wealth distribution fair?

Posted on by

I published the following blog post originally on Opensource.com.

If wealth is the abundance of valuable possessions, open source has a wealth of software. While no one “owns” open source, some are better than others at converting this communal wealth to personal wealth.

Many open source project maintainers who produce free open source software do not have a model for deriving income from the assets they have created. However, companies that use open source software to enhance their products and services convert this valuable asset into income.

The open source ecosystem needs novel mechanisms to distribute privatized wealth back to maintainers if open source projects are to remain sustainable. In this post, I’ll discuss the challenges of distributing wealth more fairly, starting with three key observations:

  • We need to identify projects that are important and need funding.
  • We lack fair rules for distributing money among open source project contributors.
  • Transaction costs are too high in underbanked areas. Bugmark and ossgrants (both discussed below) are two project ideas for addressing this problem.

Wealth creation in open source

The wealth created by open source rests today on an imbalance between those who bear the costs and those who enjoy the income. Consider the following example:

An amateur developer creates an open source project as a side project or hobby and releases the software free of charge. The software increases communal wealth when users derive value from it. Companies in particular leverage the open source software in their innovation stream, building products and services with less investment and converting valuable open source software assets into income.

A growing user base brings more support inquiries, bug reports, and feature requests, which take more time and increase costs for the maintainer. A community of contributors might form and share the work of developing and maintaining the software. Sharing the cost of creating communal wealth by contributing, however, does not provide income for the maintainer, who cannot generate personal wealth without mechanisms that distribute income from other users.

It is important to recognize that maintainers need to make a living, and if they have no income from open source projects, they likely have another job—which reduces the time they can spend on maintaining open source software. A lack of funding for open source projects becomes problematic when the software is part of our modern infrastructure and requires long-term maintenance.

The relationship between those who create open software and those who generate income from this communal wealth.

The relationship between those who create open software and those who generate income from this communal wealth.

The question then becomes: How can the wealth created by use of open source software be distributed to support long-term maintenance by paying maintainers?

At MozFest 2018, 23 people gathered to discuss this question. In small groups, participants discussed problems of interest, chose one problem to work on, and later presented their solutions to the larger group. This post summarizes key takeaways from this presentation and draws on ideas discussed during Sustain Summit 2018.

Participants in MozFest 2018 met to discuss open source wealth distribution.

Participants in MozFest 2018 met to discuss open source wealth distribution.

 

Why and how to share wealth

A central question is this: Why would companies share the income they generate from using open source software?

For-profit companies are seen as profit maximizers, and sharing revenue with open source maintainers, who license their intellectual property for free, seems counterintuitive. One survey found that 50% of respondents believe that large tech companies gain more from using open source than they contribute, which demonstrates that companies generate means to give back by using open source. (Note that I am not referring to the many open source projects companies actively maintain, or those that a company started. The focus here is on volunteer-driven communities.)

Three concrete value propositions can convince a company to pay open source maintainers:

  • Donating to open source projects earns a good name within the open source ecosystem. Donating to a project or a maintainer—for example via Patreon, OpenCollective, or an open source foundation—funds development work without exerting influence.
  • Funding open source maintenance ensures that an open source project will be updated and vulnerabilities fixed, which is important for companies that rely on the software for their products, services, and infrastructure.
  • A company gains influence over the strategic direction of an open source project when a donation or membership fee is rewarded with access to core maintainers. Special access can require that maintainers sign non-disclosure agreements and help develop solutions for vulnerabilities that may not yet have been publicly disclosed.

An open source project can also start a company that provides hosting and support services, and collect funds by selling their services. This is the most formalized way of securing funds and provides a clear value proposition. The key point is that a maintainer must figure out how to participate in the economy around open source software.

Distributing wealth: 3 practical issues

The following issues arise when the donation model is used:

First, not every project benefits equally from funds. Do you rely on open source software that might become unsupported if you do not donate to the project? The goal of such an evaluation is to find weak spots in an open source supply chain that can be strengthened with the least amount of cost. Consider using metrics, like the ones created by the CHAOSS project, for determining the health of an open source project. How exactly to identify open source projects that need funding before they become unmaintained is an unsolved problem. The Core Infrastructure Initiative has developed the Census Project to work on a solution. TideLift takes an innovative approach, paying more than $1 million to open source project maintainers based on how much a project is used.

Second, how should funds be distributed among open source project members? Defining how donations will be used should be declared upfront to avoid conflict. One approach might be to recognize individual contributors based on their contributions. For example, you could distribute funds based on the number of issues closed, commits accepted, lines added, wiki pages edited, documentation pages revised, forum messages posted, blog posts published, or other quantifiable contributions. But not all contribution types to a project can be measured—for example, organizing meetings and presenting at conferences are valuable but time-intensive contributions that do not produce trace data in a collaboration software. Every project must set its own rules, but we can share stories and best practices. Open Collective brings this conversation to the public.

Third, transaction costs hinder fair distribution of the wealth created by open source. Specifically, many people in underbanked regions struggle with the logistics of receiving payments. When an open source team member must wait for hours to cash a check, the cost of the time might outweigh the amount they receive for their work. This very real problem may be beyond the scope of what open source projects can do (except perhaps initiatives for Web3), but it deserves attention. Ultimately, the solution is to bring banking to all people and improve the interoperability of banking systems around the world.

Facilitating and incentivizing wealth transfer

There are many initiatives to solve the sustainability problem in open source, but I’ll highlight two projects that I find intriguing.

Bugmark answers the question of who within an open source project should get paid, and how much, by creating a marketplace that introduces price signals to open source. The Bugmark exchange allows trading on the status of an issue—for example, issues listed by an open source project on their GitHub issue tracker. Unlike bug bounties, trading on the status of an issue is independent of doing work. By decoupling payment and work, Bugmark has the potential to fund open source work that is not reliant on fixing an issue. For example, a project member who does bug triaging, an honorable but tedious task, has in-depth knowledge of how a project is doing and what is being worked on. This person can use their insider knowledge to trade on Bugmark and earn money. For more details on how Bugmark works, I recommend this publication.

Funding Index is an early-stage idea that revolves around donations. Donations provide a project with money without restrictions. Donations benefit companies, but the publicity effect is short-lived. At SustainSummit, we developed an idea to capture donations more permanently and create a funding index. Donations would be logged and aggregated across companies and projects. Similar to consumer rating agencies that rate products and services, this index would rate companies by how much they donate to open source projects relative to how many employees they have. We could produce badges for “#1 donor to open source,” “#1 donor to infrastructure open source projects,” or “#1 mid-sized company donor to open source.” Such a badge would extend the publicity effect of donations and hopefully incentivize companies to donate more. Funding Index exists in a first prototype and welcomes discussions on the Discourse Forum.

Sustainability is more than funding

Funding helps to pay for living expenses, servers, stickers, and travel to conferences to enable face-to-face collaboration. Funding is a necessary component of a sustainable open source project, but it requires additional elements.

A sustainable open source project fosters a healthy community, which is welcoming, provides a productive work environment, supports its members, and is prepared to let members move on when their focus in life changes. These governance and community concerns build the foundation from which open source projects work, and once they are in place, funding can leverage the work and help project members excel.

A final thought: Companies that are willing to financially support open source projects need to see the business value. Currently, there is no single solution that fits all open source project contexts. Every open source project may need to experiment for themselves and find a way to secure funds. Umbraco, for example, built a sustainable business around its open source CMS system and has experimented with 11 different business models since its inception in 2004.

More conversations are needed, and experiences must be shared. SustainOSS.org brings sustainers together to have these conversations. In conclusion, fair distribution of the wealth created by open source can enhance what already works well in open source, but it will not replace traditional open source practices.

Post Scriptum

I acknowledge that companies and foundations also create open source projects. Sustainability issues exist nonetheless, and takeaways transfer.

Acknowledgments

I’d like to thank all MozFest 2018 session participants, especially the note-takers. I appreciate constructive feedback from Sean Goggins and Tobie Langel. I received a Linux Foundation Community Travel Fund to present at Open Source Summit Europe 2018 in Edinburgh, which allowed me to participate in the SustainSummit and MozFest in London later during the same week. This work is supported through the Alfred P. Sloan Foundation Digital Technology grant on Open Source Health and Sustainability, Num: 8434

 

New Paper: “Eight Observations and 24 Research Questions About Open Source Projects: Illuminating New Realities”

Posted on by

I am excited about this paper because we point out ways in which open source is evolving. And let me tell you, open source is changing a lot. This is relevant for researchers, because it shapes the story we can tell and the kind of questions most interesting. In fact, we identify 24 research questions we find intriguing.

Paper Abstract:

The rapid acceleration of corporate engagement with open source projects is drawing out new ways for CSCW researchers to consider the dynamics of these projects. Research must now consider the complex ecosystems within which open source projects are situated, including issues of for-profit motivations, brokering foundations, and corporate collaboration. Localized project considerations cannot reveal broader workings of an open source ecosystem, yet much empirical work is constrained to a local context. In response, we present eight observations from our eight-year engaged field study about the changing nature of open source projects. We ground these observations through 24 research questions that serve as primers to spark research ideas in this new reality of open source projects. This paper contributes to CSCW in social and crowd computing by delivering a rich and fresh look at corporately-engaged open source projects with a call for renewed focus and research into newly emergent areas of interest.

Read more..
This paper is open access and available from the ACM Digital Library.

 Full reference:

Germonprez, M., Link, G. J.P., Lumbard, K., & Goggins, S. (2018). Eight Observations and 24 Research Questions About Open Source Projects: Illuminating New Realities. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 57:1–57:22. https://doi.org/10.1145/3274326

How to measure the impact of your open source project

Posted on by

We published this article originally on Opensource.com.

This article was co-authored by Vinod Ahuja, Don Marti, Georg Link, Matt Germonprez, and Sean Goggins.

Conventional metrics of open source projects lack the power to predict their impact. The bad news is, there is no significant correlation between open source activity metrics and project impact. The good news? There are paths forward.

Let’s start with some questions: How do you measure the impact of your open source project? What value does your project provide to other projects? How is your project important within an open source ecosystem? Can you predict your project’s impact using open source metrics that you can follow day to day?

If these questions resonate, chances are you care about measuring the impact of your open source project. On Opensource.com, we have already learned about measuring the project’s health, the community manager’s performance, the tools available for measuring, and the right metrics to use—and we understand that not all metrics are to be trusted.

While all these factors are critical in building a comprehensive picture of open source project health, there is more to the story. Indeed, many metrics fail to provide the information we need in a timely fashion. We want to use predictive metrics on a daily basis—metrics that are correlated with, and that act as predictors of, the outcomes and impact metrics that we care about.

Most open source project metrics focus on project metadata, such as contributor and commit counts, without addressing whether the project impacts a broader open source ecosystem. Unfortunately, a project that has a great number of contributors and an active flow of contributions may not be, and might never be, relevant to other projects in an open source ecosystem. To better understand the impact of a project, it is important to consider the broader context of an open source ecosystem. This article introduces the V-index as a measure of impact (see Regression Analysis of Open Source Project Impact: Relationships with Activity and Rewards).

Who cares about project impact?

Sponsors of open source projects care about their impact. A foundation that’s hosting an open source project likely wants it to be widely used, for example, or an organization that’s paying developers to work on a project will want to ensure that their efforts are making a difference. Consequently, software developers or project managers may need to use metrics to make the case that the time and effort spent on an open source project is creating real value for their employer.

Open source project members also care about the impact of their project. High-impact projects can be a source of pride and motivation for developers. Within the open source ecosystem, it means that people are interested in new development and ready to report bugs. High impact means that projects need the code base to be maintained and vulnerabilities to be addressed, which is an incentive to support project members.

Open source project impact

An effective way to understand an open source project’s impact is through its software libraries. A software library certainly impacts the projects in which it is used, and popular libraries have also changed the way software is developed by providing functionality across a variety of software projects.

For example, the Bootstrap library revolutionized website interfaces and has become a de facto standard. But Bootstrap depends on another widely used library: jQuery. jQuery simplifies the use of JavaScript in website development. The impact of jQuery on Bootstrap, and on web development as a whole, cannot be overstated, and this impact is evident in the library dependency relationship between the two.

The jQuery/Bootstrap example demonstrates how software libraries can have an impact. Within the open source ecosystem, jQuery is an upstream project to Bootstrap, which itself is an upstream project to many websites and web frameworks, as shown below:

downstream depedency depiction for jQuery and Bootstrap

Figure 1: An open source project dependency within an open source ecosystem: The jQuery project is the upstream project to Bootstrap and many other projects, which themselves may be upstream to more projects. (Graphic by Kevin M. Lumbard, licensed CC-BY-SA-3.0. River delta background by Messer Woland, licensed CC-BY-SA-3.0. Logos are property of respective right owners.)

Measuring impact

Many metrics are being developed to measure the impact of an open source project. These include the number of users, downloads, installs, mentions in media (e.g., blogs, news, YouTube videos, and job postings), the availability of commercial offerings, and the number of add-on products. But such metrics isolate impact within that specific project and don’t fully demonstrate the impact of a software library within an open source ecosystem.

To measure the impact of an open source project within the open source ecosystem, let’s borrow a metric from academia: the h-index. This determines the impact of an author through the relationship of how many publications he or she has produced, and how many other authors have cited these publications. We propose, therefore, that a project’s impact in an open source ecosystem can be determined by downstream dependencies (i.e., how many downstream open source projects use them and how often those downstream projects are themselves used).

V-index

A downstream dependency exists when a software library is used within another piece of software. The V-index, which encapsulates our proposed measure of impact, is the maximum number of first-order downstream dependencies that themselves have at least an equal number of second-order downstream dependencies. The first-order dependency is the number of open source projects that use the library. The second-order downstream dependency is determined by how often a first-order dependent project is used within other open source projects.

The V-index is elaborated in three different scenarios:

Scenario A

Scenario B

Scenario C
First-order dependencies Second-order dependencies First-order dependencies Second-order dependencies First-order dependencies Second-order dependencies
Dependency 1 0 Dependency 1 4 Dependency 1 40
Dependency 2 0 Dependency 2 4
Dependency 3 0 Dependency 3 4
Dependency 4 0 Dependency 4 4

Project A has a V-index of 0.

The project has four projects that depend on it. No other project depends on these projects. The V-index of Project A is 0 because zero first-order dependencies have any second-order dependencies.

Project B has a V-index of 4.

The project has four projects that depend on it. Each of these projects has four projects that depend on them. The V-Index of Project B is four because each of the four first-order dependencies have at least four second-order dependencies.

Project C has a V-index of 1.

The project has one project that depends on it. This project has 40 projects that depend on it. The V-Index of Project C is 1 because it has one first-order dependency that has at least one second-order dependency.

Looking at a practical example, jQuery has a V-index of 98. It has 13,848 first-order dependencies, of which Bootstrap is one, with 5,005 second-order dependencies. Of the 13,848, only 98 first-order dependencies have 98 or more second-order dependencies, as shown below:

V-Index graphical depiction

Figure 2: V-index of jQuery: The x-axis represents the downstream open source projects (first-order downstream dependencies) sorted by the number of their own downstream dependencies. The y-axis represents the number of downstream dependencies of each first-order open source project on the x-axis (second-order downstream dependencies). The V-index is the number of first-order downstream dependencies that have at least the same number of second-order downstream dependencies. (Graphic by Kevin M. Lumbard, licensed CC-BY-SA-3.0. Logos are property of respective right owners.)

Increase impact with new metrics

How do you increase your open source project’s impact? Well, you need to convince other projects to use your project. Unfortunately, there is no single activity that will make this happen. However, there are steps you can take to make a project impactful, and there are ways to measure how well you do them. Let’s look at which of these measures are correlated with impact.

We summarize the findings below based on previous correlation analysis. The correlation analysis used a sample of metrics for three kinds of open source metrics:

  1. Activity metrics measure metadata such as contributor or commit counts. Project contributors can increase these metrics by doing more work on the project and getting more people involved.
  2. Reward metrics measure how well the project is meeting contributor’s expectations. They may improve with faster acceptance of contributions.
  3. Impact metrics measure the impact on users and other projects.

The V-index was developed to measure impact metrics. The correlation was tested for 604 projects that were started in 2014 or 2015, that used the Rust programming language, that were listed in GHTorrent and Libraries.io (the data sources), and that had at least one downstream dependency.

The findings show that none of the conventional open source activity metrics correlate with impact. This lack of predictive activity metrics means that we have no good predictors to manage our open source projects.

Does this mean all is lost? We think not. Several open source projects are building next-generation metrics that project sponsors, maintainers, and downstream users might be able to rely on in the future. Here are four paths to finding the predictive metrics we need to boost the impact of our open source projects:

1. Add software quality metrics

The first idea is to combine open source activity metrics with conventional software engineering metrics, such as code coverage. Conventional open source activity metrics focus heavily on the development dynamics within the project. The focus on activity metrics excludes software quality factors, which might be more important for people choosing a software library. Conventional open source activity metrics make it difficult to distinguish productive activity from unproductive activity. Combining a software engineering metric with an open source activity metric could make the latter more valuable.

2. Understand the user community

The second idea involves using natural language processing to determine the sentiment within an open source project, especially where users of the software participate. Conventional open source activity metrics rely only on metadata. Knowing the number of interactions does not help us understand the quality and substance of community. FOSS Heartbeat, while currently not maintained, offers a solution.

3. Market mechanisms

The third idea is to draw a connection between impact and the value of a software library. Existing valuation methods focus on the project itself (i.e., development costs) rather than the value others derive from it. A problem that open source faces is the absence of price signals that can inform the value users receive from a software library. To draw a connection between impact and value, we need new market mechanisms, like the ones proposed by Bugmark.

4. Shared understanding of metrics

The fourth idea is to build more knowledge in the open source ecosystem about how metrics can help us understand the impact and health of open source projects. The Linux Foundation initiated the CHAOSS (Community Health Analytics Open Source Software) project to bring open source projects and other stakeholders together to build a shared understanding of metrics and of the software tools to capture and analyze said metrics. This blog post is based on research conducted as part of the CHAOSS project.

Acknowledgments

This article is based on the whitepaper Regression Analysis of Open Source Project Impact: Relationships with Activity and Rewards by Vinod K. Ahuja. Graphics were prepared by Kevin M. Lumbard. This work is supported by Mozilla and the Alfred P. Sloan Foundation.