Identifying and Extracting Information from Programming Video Tutorials to Support Developer Knowledge Discovery

Developers often need additional knowledge to complete some programming tasks, and they oftenseek information online to solve the problem at hand. Fortunately, online resources contain a vast amount of information at the developers’ disposal, captured in (Q&A) websites, API documentation, programming blogs, and even videos. In programming videos, developers share programming knowledge by recording their screens and going through step-by-step instructions. Videos have become pervasive among developers as they are easier to prepare than text documents, and they are more interactive and engaging. Despite these advantages, the process of finding a relevant video to a programming task is overwhelming and requires considerable effort and time due to the visual nature of the data. In this dissertation, we devise techniques to make programming videos more accessible and reachable. More specifically, we introduce an approach to locate code editing windows in video frames to improve the accuracy of the code extraction process. In addition, we introduce UIScreens, an approach and tool to extract mobile Graphical User Interfaces (GUIs) from mobile programming screencasts and complement videos with a sufficient and unique list of GUIs. Last, we introduce vid2meta, an approach meant to complement the metadata of Android programming screencasts with a list of Java, XML, and GUI elements automatically extracted from the contents of the video. The aim of this work is in (i) helping developers decide if a video is relevant to their task and (ii) enabling developers to search and navigate to the textual code-related contents of videos. We illustrate that the approaches introduced in this dissertation represent non-trivial advancements in mining video programming tutorials through a series of empirical evaluations and user studies that demonstrate the accuracy, usability, usefulness, and flexibility of our approaches.