REFOCUS: Current & Future Search Interface Requirements for German-speaking Users

REFOCUSWhen looking at current research, there is plenty of existing work inquiring into how users use search engines1 and how future search interfaces could look like2. Yet, an investigation of users’ perceptions of and expectations towards current and future search interfaces is still missing.

Therefore, at this year’s International Conference on WWW/Internet (ICWI ’16) my co-author Martin Gaedke presented our paper “REFOCUS: Current & Future Search Interface Requirements for German-speaking Users”, which we wrote together with Andreas Both. To give you an idea of what our work aims at, I’m going to provide a step-by-step explanation of the research paper’s title.

REFOCUS. An acronym for Requirements for Current & Future Search Interfaces.

Search Interface Requirements. From an exploratory study with both qualitative and quantitative questions we have derived a set comprising 11 requirements for search interfaces. The initial set of requirements was validated by 12 dedicated experts.

Current. The requirements shall be valid for current search interfaces. According to the experts’ reviews, this applies to eight of the requirements.

Future. Also, the set of requirements shall inform the design and development of future search interfaces. According to the experts’ reviews, this applies to ten of the requirements. Supporting the design of future search interfaces is particularly important with the wide variety of Internet-capable novel devices, like cutting-edge video game consoles, in mind.

German-speaking Users. Due to the demographics of our participants, the set of requirements can be considered to be valid for German-speaking Internet users. 87.3% of the participants were German while 96.6% lived in a German-speaking country at the time of the survey.

If this sounds interesting to you, please go check out our research paper at ResearchGate or arXiv. The original publication will be available via the IADIS Digital Library.

1 For instance, (accessed November 8, 2016).
2 For instance, Hearst, M. A. ‘Natural’ Search User Interfaces. In Commun. ACM 54(11), 2011.


Survey: Search Engines – Perception, Usage & Expectations

Search Engines: Perception, Usage & ExpectationsIn the context of my PhD thesis in Human–Computer Interaction, I currently conduct a survey concerning the use of search engines. With the help of this survey I hope to better understand how you perceive and use search engines and which expectations you have towards them. In particular, I focus on the use of search engines with smart phones, tablet PCs and other devices, such as Nintendo Wii U or PlayStation 4 (yes, you can actually browse the web with these!).

Automatic Approaches to Usability Optimization

The research for my PhD thesis is concerned with automatic approaches to optimizing the usability of search engines. To date, we have developed two core systems within the context of desktop PCs: TMR, which uses mouse cursor interactions to predict the relevance of search results, and WaPPU, which evaluates the usability of a whole web interface based on a quantitative usability score rather than qualitative assessments by individual experts.

With TMR, we evaluated 32 GB of tracking data from two German hotel booking portals. Using these data, we were able to learn models for predicting a completed booking process (which is a very strong indicator for relevance in the context of hotel search) from users’ interactions with the individual search results. The top models we learned reached correlation coefficients of up to 0.81.

WaPPU was involved in the redesign and evaluation of the web search results page of another real-world search engine. During an A/B test based on WaPPU’s usability score, we found the original version of the page to have a score of only 59.9% while the redesigned version reached 67.5% (the difference was statistically significant). We moreover detected significant differences for the individual usability factors distraction and information density.

My next aim is to transfer the above methods into the context of touch and other novel devices. That’s why I need your help with probing the current situation with respect to search engine usage on different devices.

Search Engines: Perception, Usage & Expectations

The survey will take about 15–20 minutes (partly depending on your answers). Since I’m writing my PhD thesis in cooperation with Unister GmbH in Leipzig (Germany), every participant receives a coupon worth 50 € for booking a trip on I want to thank my industry partner very much for providing this nice incentive 🙂

Please take my survey (see below) and also tell your friends and family about it (especially if you can’t make time for taking it yourself). In this way, you can contribute valuable data to my PhD thesis and help shape future search engines! Thank you very much 🙂

(Open survey in new tab:

[offene Masterarbeit] Was that Page Pleasant to Use? Usability-Metriken in einer echten Suchmaschine

Es gibt viel zu viele schlechte Webseiten! Schon mal versucht, auf die Tagessätze für Auslandsreisekosten zu finden? Falls nicht, einfach mal ausprobieren und viel Spaß dabei! Oder schon mal auf der Seite der Uni Würzburg versucht, herauszufinden, wie genau eine Bewerbung für den Bachelor in Wirtschaftswissenschaften abläuft? Nein? Ist auch eigentlich besser so, weil der Versuch einen leicht in den Wahnsinn treiben kann.

Motivation: Usability? Nein, danke!

Viele Webseiten (auch großer Unternehmen) beweisen keinerlei Gespür für grundlegende Usability-Prinzipen, welche weder sonderlich neu noch sonderlich kompliziert sind. Häufig sind z. B. Informationen, die eine Großzahl an Nutzern betreffen, nicht direkt über die primäre Navigation erreichbar, sondern nur über verschlungene Pfade und zahllose Klicks. Und das trotz einer Fülle an Frameworks und Content-Management-Systemen, die modernste Webdesign- und Usability-Prinzipien unterstützen. Der wohl häufigste Grund für mangelnde Usability einer Webseite ist die Tatsache, dass entsprechende Tests nur unzureichend oder gar nicht durchgeführt werden, häufig aus Kosten- oder Zeitgründen.

The WaPPU dashboardUm dem entgegenzuwirken, habe ich als Teil meiner Doktorarbeit ein prototypisches Tool namens WaPPU entwickelt, welches es ermöglicht, wesentlich günstigere A/B-Tests auf Basis einer neuartigen Metrik für Usability durchzuführen. Das heißt, die Usability zweier leicht unterschiedlicher Versionen derselben Webseite wird während der Benutzung durch echte Nutzer in Form von Metriken in Echtzeit erfasst und in einem Dashboard visualisiert (siehe Abbildung).

Ziel der Arbeit

Mein Dissertationsprojekt ist eingebettet in die Forschungs- und Entwicklungsabteilung der Unister GmbH in Leipzig, welche aktuell eine neuartige Reisesuchmaschine entwickelt. Der entwickelte Prototyp soll im Rahmen einer Masterarbeit in diese reale Suchmaschine integriert werden, um verschiedene Interface-Variationen im produktiven Betrieb anhand ihrer Usability bewerten zu können. Weitere Informationen können der offiziellen Ausschreibung entnommen werden. Interessenten melden sich bitte unter der in der PDF angegebenen E-Mail-Adresse oder über mein Kontakformular.


Ein Demo-Video zum WaPPU-Tool gibt’s hier.

StreamMyRelevance! Predicting Search Result Relevance from Streams of Interactions

SMR paper @ ICWE2014Guessing the relevance of delivered search results is one of the biggest issues for today’s search engines. The particular problem is that it’s difficult to obtain explicit statements from users about whether they found what they were searching for. Clicks are commonly used to guess relevance (using so-called “click models”) but they are far from being a perfect indicator. Particularly, a user might click a search result, but then return to the results page because the visited webpage was useless. Also, it’s possible that no clicks happen at all if the desired piece of information is already shown on the results page (e.g., in terms of an info box).

To tackle the above shortcoming, we have investigated the suitability of implicit feedback in terms of mouse cursor interactions for predicting the relevance of search results. For this, we developed StreamMyRelevance!—a system that receives streams of interactions and relevance judgments and trains statistical models from these in near real-time. The models can then be used to infer relevance from interactions in the future. The relevance judgments we’re using to train our models can either be implicit (e.g., a completed booking process in the case of hotel search) or explicit (e.g., statements by paid quality raters/crowdworkers).

Analysis of a large amount of real-world interaction data from two e-commerce portals showed that StreamMyRelevance! is able to train good models that show the tendency to perform better than a state-of-the-art click model solution1 that is successfully used in industry. Our results particularly underpin the benefit of using interaction data other than clicks for guessing the relevance of search results.

We have summarized the design and evaluation of our system in a full research paper2 that will be presented at the 2014 International Conference on Web Engineering (ICWE). The conference proceedings will be published by Springer and the final version of our paper will be available at Special thanks go to Sebastian Nuck, who helped with development and evaluation of StreamMyRelevance! in the context of his Master’s Thesis at Leipzig University of Applied Sciences.

1 Chao Liu, Fan Guo, and Christos Faloutsos (2009). “BBM: Bayesian Browsing Model from Petabyte-Scale Data”. In Proc. KDD.
2 Maximilian Speicher, Sebastian Nuck, Andreas Both and Martin Gaedke (2014). “StreamMyRelevance! Prediction of Result Relevance from Real-Time Interactions and its Application to Hotel Search”. In Proc. ICWE.

4 Submissions accepted at International Conference on Web Engineering (ICWE)

End of February, I submitted four contributions to the 14th International Conference on Web Engineering: two full papers, one demo and one poster. Of these four submissions, all were accepted and will be presented at the conference, which is to be held in Toulouse (see map below) from July 1 to July 4. In the following, I’ll give a quick overview of the accepted papers. A more detailed explanation of my current research will be the subject of one or two separate articles.

  • Maximilian Speicher, Sebastian Nuck, Andreas Both, Martin Gaedke: “StreamMyRelevance! Prediction of Result Relevance from Real-Time Interactions and its Application to Hotel Search” — This full paper is based on Sebastian Nuck’s Master thesis. He developed a system for processing user interactions collected on search results pages in real-time and predicting the relevance of individual search results from these.
  • Maximilian Speicher, Andreas Both, Martin Gaedke: “Ensuring Web Interface Quality through Usability-based Split Testing” — This full paper proposes a new approach to split testing that is based on the actual usability of the investigated web interface rather than pure conversion maximization. We have trained models for predicting usability from user interactions and from these have also derived additional interaction-based heuristics for comparing search results pages.
  • Maximilian Speicher, Andreas Both, Martin Gaedke: “WaPPU: Usability-based A/B Testing” — This demo accompanies our paper about Usability-based Split Testing. The WaPPU tool builds upon this new concept and demonstrates how usability can be predicted from user interactions using automatically learned models.
  • Maximilian Speicher: “Paving the Path to Content-centric and Device-agnostic Web Design” — This poster is based on one of my previous posts. It provides a review of, which satirically claims to be a perfect website. Based on current research, we suggest improvements to the site that follow a strictly content-centric and device-agnostic approach.

My PhD research is supervised by Prof. Dr.-Ing. Martin Gaedke (VSR Research Group, Chemnitz U of Technology) and Dr. Andreas Both (R&D, Unister GmbH) and funded by the ESF and the Free State of Saxony.

2013 in Review: Search Interaction Optimization

On January 1, 2013 I started my PhD studies at Chemnitz University of Technology in cooperation with Unister GmbH, Leipzig. My project is about automatic methods for optimizing search engine results pages (e.g.,!) with respect to result quality and interface usability. The official working title is Search Interaction Optimization: A Design Thinking Approach.

During my first year as a PhD student, I have published three papers (as first author). A full paper about my first milestone has been presented at the International Conference on Knowledge and Information Management (CIKM), which was held in San Francisco in October/November.1 This milestone was about deducing the relevance of search results from user interactions on the search engine results page. Using large amounts of anonymous interaction data from two real-world hotel booking portals, we could show that it is possible to learn according relevance models of reasonable quality.

A second (short) paper was presented at the PhD Symposium of the International Conference on Web Engineering (ICWE) in July.2 The paper addressed the attempt of learning a common model that predicts usability based on training data from a group of similar webpages (e.g., online news articles). However, we concluded that this is not easily possible, because differences in low-level page structure and user intention counter model precision. Thus, additional preprocessing steps are necessary to minimze these influences. The usability evaluation described in this paper was based on a novel instrument for measuring usability whose items have been specifically designed for correlation with client-side interaction features. This Interface Usability Instrument (Inuit) was presented at the workshop “Methodological Approaches to HCI” in September.3 The above described is part of the second milestone of my PhD project, which is about automatic methods for optimizing interface usability. This milestone is my current work-in-progress and will be finished in 2014.

Alright, so much for my research in 2013. I’ll keep you updated with more fine-grained results during the new year.

1 Speicher, Both, Gaedke: “TellMyRelevance! Predicting the Relevance of Web Search Results from Cursor Interactions” (
2 Speicher, Both, Gaedke: “Was that Webpage Pleasant to Use? Predicting Usability Quantitatively from Interactions” (
3 Speicher, Both, Gaedke: “Towards Metric-based Usability Evaluation of Online Web Interfaces” (