Pros & Cons of an Industrial Ph.D. Program

From 2013 till 2016 I’ve done an industrial Ph.D. at Unister GmbH in Leipzig in cooperation with Chemnitz University of Technology. This means that I worked in Unister’s R&D department, which at that time developed a novel semantic search engine, and wrote my thesis about the part I contributed to the project. A few days ago—in an old notebook—I found a list of blog posts I wanted to write with the entry “Pros & Cons of an Industrial Ph.D. Program” not yet crossed out. I remember that I added this idea to the list after talking to Jürgen Cito, a Ph.D. student at the University of Zurich, at the 2014 International Conference on Web Engineering (ICWE). After chatting about the topic for a bit, Jürgen said something like “The pros and cons of an industrial Ph.D. program would really make a good blog post”. So here it is, in terms of an infographic made with Adioma, which I wanted to play around with for a pretty long time now.

Pros & Cons of an Industrial Ph.D. Program

I have to note that the con “less contact to your professor” wasn’t too bad for me because my second advisor Dr. Andreas Both, who was the Head of R&D at Unister, attached great importance to good and quality scientific work and publishing scientific results. This, however, isn’t something you can expect at any company. Overall, I list one more pro than cons because I had a really good experience with my industrial Ph.D. program and would definitely do it again!

Advertisements

The Search Interaction Optimization Toolkit – The Essence of my PhD Thesis

SIO Toolkit Logo
Logo of the SIO Toolkit.

My PhD thesis introduces a novel methodology that is named Search Interaction Optimization (SIO) and is used for designing, evaluating and optimizing search engine results pages (so-called SERPs). As a proof-of-concept of this new methodology, I’ve developed a corresponding SIO toolkit, which comprises a total of seven components1 (most of which have already been introduced in previous posts):

  1. Inuit, a new instrument for usability evalutation;
  2. WaPPU, a tool for Usability-based Split Testing;
  3. a catalog of best practices for creating better usable SERPs, which together with WaPPU and a special add-on forms
  4. S.O.S., a tool for automatically evaluating and optimizing SERPs;
  5. TellMyRelevance! (TMR), a novel pipeline that predicts the relevance of search results from client-side interactions;
  6. StreamMyRelevance! (SMR), a streaming-based version of TMR that works in real-time rather than batch-wise; and
  7. a set of requirements for current & future search interfaces, which has been derived from an empirical study with German-speaking users.
SIO Methodology Logo
Logo of the SIO Methodology.

Describing the design and development of the above components and evaluating their effectiveness and feasibility makes for a major part of my thesis. Now, I’ve finally managed to organize all of them in terms of GitHub repos2, which I make available through a new website I have specifically created for my PhD project: http://www.maxspeicher.com/phdthesis/. In particular, on that site you can filter the components depending on whether you want to design, evaluate and/or optimize a SERP. It also lists all of the related publications including links to the corresponding full texts (via ResearchGate). In case you are actually interested in all that fancy research stuff3—have fun browsing, reading & playing around! 🙂

1 The logo of the SIO toolkit features only six tiles because S.O.S. and the catalog of best practices are treated as one component there.
2 Because my PhD project was carried out in cooperation with Unister GmbH (Leipzig), unfortunately it’s not possible for me to provide the source codes of all components via GitHub, as some contain company secrets.
3 Which I doubt. 😉

INUIT: The Interface Usability Instrument

INUIT LogoAs one of the building blocks of my PhD thesis, I have developed a novel instrument for measuring the usability of web interfaces, which is simply called Inuit—the Interface Usability Instrument1. This was necessary because a usability instrument that is suited for the automatic methods for Search Interaction Optimization I have developed in my PhD project must fulfill three particular requirements, which are not met by any existing instruments:

(R1) A minimal number of items.
(R2) Items with the right level of abstraction for meaningful correlations with user interactions recorded on the client.
(R3) Items that can be applied to a web interface in terms of a stand-alone webpage.

The Instrument

Inuit has been designed and developed in a two-step process: First, over 250 rules for good usability from established guidelines and checklists were reviewed to identify a set of common underlying factors (or items) according to R2. From these underlying factors, a “structure” of usability based on ISO 9241-11 was created, which was then shown to 9 dedicated usability experts in the second step. The experts—all of which were working in the e-commerce industry—reviewed the given “structure” and proposed changes according to their perception of web interface usability. Finally, seven items have been identified:

  1. Informativeness
  2. Understandability
  3. Confusion
  4. Distraction
  5. Readability
  6. Information Density
  7. Reachability

These items can be translated to, e.g., the following yes/no questions for use in a questionnaire for determining the usability of a webpage:

  1. Did you find the content you were looking for?
  2. Could you easily understand the provided content?
  3. Were you confused while using the webpage?
  4. Were you distracted by elements of the webpage?
  5. Did typography & layout add to readability?
  6. Was there too much information presented on too little space?
  7. Was your desired content easily and quickly reachable (concerning time & distance)?

Conclusions

A confirmatory factor analysis based on a user study with 81 participants has proven that our instrument reasonably well reflects real-world perceptions of web interface usability. Inuit was first introduced at the workshop “Methodological Approaches to Human–Machine Interaction”, which was held as part of the 2013 Mensch & Computer conference. The corresponding paper is named Towards Metric-based Usability Evaluation of Online Web Interfaces (full-text here). The final version of the instrument has been presented at this year’s International Conference on Design, User Experience and Usability (DUXU), which has been held in Los Angeles. The full research paper is titled Inuit: The Interface Usability Instrument and available via Springer (full-text here).

Future Work

In the future, I intend to transfer Inuit into the context of my current work. That is, I intend to use it for evaluating the web interface of HoloBuilder, which enables users to create 3D content for the web, in contrast to the usual 2D content that is consumed nowadays. It will be particularly interesting to see whether both, 2D and 3D web interfaces can be meaningfully evaluated using the same minimal instrument. Furthermore, Inuit will be applied in the context of the research on evidence-based computing that is happening at the VSR research group at Technische Universität Chemnitz.

P.S.: Thanks a lot to Viet Nguyen for the awesome Inuit logo! 🙂

1 Please note the small caps!

S.O.S. Receives Best Paper Honarable Mention Award at CHI ’15

TrophyOur paper “S.O.S.: Does Your Search Engine Results Page (SERP) Need Help?”—co-authors are Dr. Andreas Both (Unister) and Prof. Martin Gaedke (TU Chemnitz)—has been awarded a Best Paper Honarable Mention Award by ACM SIGCHI, the Special Interest Group on Computer–Human Interaction of the Association for Computing Machinery. According to Wikipedia, ACM SIGCHI is “the world’s leading organization in Human–Computer Interaction (HCI), and essentially created and defined the field.”1 Our paper is to be presented at the 2015 edition of the CHI Conference on Human Factors in Computing Systems2, which is the premier conference in the field of HCI and takes place in Seoul, South Korea.

S.O.S., which is short for “SERP Optimization Suite”, is a tool for determining the usability of a SERP in terms of quantitative scores by analyzing user feedback and interactions. If suboptimal scores are detected for a certain factor of usability (e.g., readability), adjustments are automatically proposed based on a catalog of best practices (e.g., adjust font size, among others). The catalog contains sets of potential causes for suboptimal scores and maps them to sets of corresponding countermeasures. Determining usability scores is based on WaPPU.

S.O.S.’s GitHub repository can be found at https://github.com/maxspeicher/sos. It’s free for non-commercial use. Resources and results of the evaluation we describe in our paper are available at https://github.com/maxspeicher/sos-resources.

(CC BY trophy icon by icomoon.io.)

1 http://www.wikiwand.com/en/SIGCHI
2 http://chi2015.acm.org/program/best-of-chi/#honorable-mentions

#papershizzle

Phew! It was a rather busy (that’s why I haven’t been posting in a while), but also very successful start into the new year. A total of three full papers have been accepted at various conferences and journals. So basically, I’ve been revising and resubmitting papers since Christmas.

First, our paper about Inuit has been accepted at the 4th International Conference on Design, User Experience and Usability (DUXU), which will be held as a part of HCI International 2015 in Los Angeles, California. Inuit is a new usability instrument for interfaces that has been specifically designed for our concept of Usability-based Split Testing. An instrument of this kind contains a set of observable items that are used to predict a latent (i.e., non-observable) variable—in our case, usability. For instance, a person’s intelligence is a latent variable that can only be assessed with a number of specific questions (or items). Therefore, IQ tests are instruments.

Second, an article that is an extended version of our ICWE 2014 paper about SMR has been conditionally accepted by the Journal of Web Engineering (JWE). SMR is a streaming-based system that allows for the prediction of search result relevance from user interactions. In the extended version, we further elaborate on specifics of SMR’s data processing algorithm and complexity. Also, we describe the integration of our system into a real-world industry setting.

Image taken from http://chi2015.acm.org/.

Finally—and probably most importantly—our paper titled “S.O.S.: Does Your Search Engine Results Page (SERP) Need Help?” has been accepted at CHI 2015, which is the premier conference on human–computer interaction and will take place in Seoul! What a great success 🙂 . S.O.S. is the abbreviation for SERP Optimization Suite, which comprises two components. (1) WaPPU, which is a tool for inferring usability scores from users’ interactions with an interface. WaPPU was already presented at ICWE 2014 and has been augmented with (2) a catalog of best practices to form S.O.S. That catalog contains potential causes and corresponding countermeasures for suboptimal usability scores. An extension to WaPPU now automatically detects such suboptimal scores and proposes optimizations based on the catalog.

I am very excited about these accepted papers and definitely looking forward to presenting them to an audience of world-renowned experts. As a side note, a revised and extended version of my post titled What is ›Usability‹? has been published as a technical report in the series “Chemnitzer Informatik-Berichte” (roughly translated: “Computer Science Reports of Chemnitz University of Technology”).

So after this very successful start of the year, let’s see what else 2015 will bring. Stay tuned! 🙂

Survey: Search Engines – Perception, Usage & Expectations

Search Engines: Perception, Usage & ExpectationsIn the context of my PhD thesis in Human–Computer Interaction, I currently conduct a survey concerning the use of search engines. With the help of this survey I hope to better understand how you perceive and use search engines and which expectations you have towards them. In particular, I focus on the use of search engines with smart phones, tablet PCs and other devices, such as Nintendo Wii U or PlayStation 4 (yes, you can actually browse the web with these!).

Automatic Approaches to Usability Optimization

The research for my PhD thesis is concerned with automatic approaches to optimizing the usability of search engines. To date, we have developed two core systems within the context of desktop PCs: TMR, which uses mouse cursor interactions to predict the relevance of search results, and WaPPU, which evaluates the usability of a whole web interface based on a quantitative usability score rather than qualitative assessments by individual experts.

With TMR, we evaluated 32 GB of tracking data from two German hotel booking portals. Using these data, we were able to learn models for predicting a completed booking process (which is a very strong indicator for relevance in the context of hotel search) from users’ interactions with the individual search results. The top models we learned reached correlation coefficients of up to 0.81.

WaPPU was involved in the redesign and evaluation of the web search results page of another real-world search engine. During an A/B test based on WaPPU’s usability score, we found the original version of the page to have a score of only 59.9% while the redesigned version reached 67.5% (the difference was statistically significant). We moreover detected significant differences for the individual usability factors distraction and information density.

My next aim is to transfer the above methods into the context of touch and other novel devices. That’s why I need your help with probing the current situation with respect to search engine usage on different devices.

Search Engines: Perception, Usage & Expectations

The survey will take about 15–20 minutes (partly depending on your answers). Since I’m writing my PhD thesis in cooperation with Unister GmbH in Leipzig (Germany), every participant receives a coupon worth 50 € for booking a trip on ab-in-den-urlaub.de. I want to thank my industry partner very much for providing this nice incentive 🙂

Please take my survey (see below) and also tell your friends and family about it (especially if you can’t make time for taking it yourself). In this way, you can contribute valuable data to my PhD thesis and help shape future search engines! Thank you very much 🙂

(Open survey in new tab: https://docs.google.com/forms/d/1rX5EOQhzEXa-G-WxUz6aT5PKYb4ofVTPCGumO2h8Vtg/viewform?usp=send_form)

Umfrage: Suchmaschinen – Wahrnehmung, Nutzung & Erwartungen

Suchmaschinen: Wahrnehmung, Nutzung & ErwartungenFür meine Doktorarbeit in Human-Computer Interaction führe ich momentan eine Umfrage zur Nutzung von Suchmaschinen durch. Es geht mir in dieser Umfrage vor allem darum, herauszufinden, wie ihr Suchmaschinen wahrnehmt und welche Erwartungen ihr an sie habt. Besonderer Fokus liegt auch auf der Nutzung mit Smartphones, Tablets und anderen Geräten, wie z. B. einer Nintendo Wii U oder PlayStation 4 (ja, man kann mit Spielekonsolen mittlerweile ins Internet gehen!).

Die Umfrage dauert (teilweise abhängig von euren Antworten) nur ca. 15–20 Minuten. Da ich meine Dissertation in Kooperation mit der Unister GmbH schreibe, bekommt außerdem jeder Teilnehmer einen Reisegutschein im Wert von 50 € für ab-in-den-urlaub.de. An dieser Stelle vielen Dank an meinen Industriepartner 🙂

Bitte nehmt rege an meiner Umfrage teil (siehe unten) und verbreitet sie auch ein bisschen weiter (insbesondere, wenn ihr selbst keine Zeit zur Teilnahme habt). So könnt ihr einen wichtigen Beitrag zu meiner Forschung und Doktorarbeit leisten! Vielen Dank 🙂

(Umfrage in neuem Tab öffnen: https://docs.google.com/forms/d/14ViXK-Cl7liQmaR3K_KmJtt_4LK11Thy1pRz9dWJvGw/viewform?usp=send_form)

[offene Masterarbeit] Was that Page Pleasant to Use? Usability-Metriken in einer echten Suchmaschine

Es gibt viel zu viele schlechte Webseiten! Schon mal versucht, auf www.finanzen.sachsen.de die Tagessätze für Auslandsreisekosten zu finden? Falls nicht, einfach mal ausprobieren und viel Spaß dabei! Oder schon mal auf der Seite der Uni Würzburg versucht, herauszufinden, wie genau eine Bewerbung für den Bachelor in Wirtschaftswissenschaften abläuft? Nein? Ist auch eigentlich besser so, weil der Versuch einen leicht in den Wahnsinn treiben kann.

Motivation: Usability? Nein, danke!

Viele Webseiten (auch großer Unternehmen) beweisen keinerlei Gespür für grundlegende Usability-Prinzipen, welche weder sonderlich neu noch sonderlich kompliziert sind. Häufig sind z. B. Informationen, die eine Großzahl an Nutzern betreffen, nicht direkt über die primäre Navigation erreichbar, sondern nur über verschlungene Pfade und zahllose Klicks. Und das trotz einer Fülle an Frameworks und Content-Management-Systemen, die modernste Webdesign- und Usability-Prinzipien unterstützen. Der wohl häufigste Grund für mangelnde Usability einer Webseite ist die Tatsache, dass entsprechende Tests nur unzureichend oder gar nicht durchgeführt werden, häufig aus Kosten- oder Zeitgründen.

The WaPPU dashboardUm dem entgegenzuwirken, habe ich als Teil meiner Doktorarbeit ein prototypisches Tool namens WaPPU entwickelt, welches es ermöglicht, wesentlich günstigere A/B-Tests auf Basis einer neuartigen Metrik für Usability durchzuführen. Das heißt, die Usability zweier leicht unterschiedlicher Versionen derselben Webseite wird während der Benutzung durch echte Nutzer in Form von Metriken in Echtzeit erfasst und in einem Dashboard visualisiert (siehe Abbildung).

Ziel der Arbeit

Mein Dissertationsprojekt ist eingebettet in die Forschungs- und Entwicklungsabteilung der Unister GmbH in Leipzig, welche aktuell eine neuartige Reisesuchmaschine entwickelt. Der entwickelte Prototyp soll im Rahmen einer Masterarbeit in diese reale Suchmaschine integriert werden, um verschiedene Interface-Variationen im produktiven Betrieb anhand ihrer Usability bewerten zu können. Weitere Informationen können der offiziellen Ausschreibung entnommen werden. Interessenten melden sich bitte unter der in der PDF angegebenen E-Mail-Adresse oder über mein Kontakformular.

Demo

Ein Demo-Video zum WaPPU-Tool gibt’s hier.

motherfuckingwebsite.com Redesigned. My Other Poster Presented at #ICWE2014

(Disclaimer: motherfuckingwebsite.com was not made by me!)

Device-agnostic design poster presented @ ICWE 2014From my original post about redesigning motherfuckingwebsite.com (see here), I have created a poster along with a corresponding short paper, which have been presented at the 2014 International Conference on Web Engineering (ICWE).

The short paper will be included in the conference proceedings published by Springer: Maximilian Speicher (2014). “Paving the Path to Device-agnostic and Content-centric Web Design”. In Proc. ICWE (Posters).

Special thanks go to Fred Funke, who helped with designing the poster!

How to Infer Usability from User Interactions. My Poster Presented at #ICWE2014

WaPPU poster presented @ ICWE 2014The corresponding publications are:

  • Maximilian Speicher, Andreas Both and Martin Gaedke (2014). “Ensuring Web Interface Quality through Usability-based Split Testing”. In Proc. ICWE.
  • Maximilian Speicher, Andreas Both and Martin Gaedke (2014). “WaPPU: Usability-based A/B Testing”. In Proc. ICWE (Demos).

For more information about WaPPU, please see this previous post. Special thanks go to Fred Funke, who helped with designing the poster!