Keyo rented apartments in NYC and the Bay Area (acquired by Zillow in 2019). We employed Scouts – gig-economy workers – who were locals in the community to facilitate on-demand apartment viewings, all while eliminating broker fees. Prospective tenants browsed listings and scheduled viewings on our website. Owners managed inventory and approved applications through our web-based dashboard.

Scouts are locals who earn cash by showing apartments on-demand to prospective tenants in NYC looking to rent with Keyo. They provide feedback after each viewing that ranges from rating the viewer’s interest to relaying the condition of the building. The feedback is designed in the form of pill buttons, a thumbs up/down, and a text box for written entries.

The purpose of this feedback is for Keyo to take action. Is the viewer interested? We’ll send them a link to apply. Is the unit unpresentable? We’ll clean it. And so on. However, issues abounded with the information we collected, mainly that it was not very useful. Scouts would blow through the form, tapping pill buttons that skewed towards the positive so that they could complete their viewings and cash out. Of the ones who wrote feedback, a seldom sight, we were provided more information than we could have gathered from 100 pill buttons. So why weren’t the pills proving useful, unlike the text box which ushered such valuable insight?

After closely examining the metrics from our database, the inherent flaws of the design became apparent: binary pill buttons, subjective thumbs up/down, and an optional text box all restricted Scouts from providing appropriate feedback.

Of the 9 positive pills (clean, well lit), the lowest expression was 43% with a high of 80%. Of the 10 negative pills (dirty, unsafe), the lowest expression was 1.5% with a high of 6%. This means of all the information received about viewings, 43-80% of the feedback was positive and 6%, at the most, was negative. Many times Scouts did not provide feedback on pills whatsoever. This did not align to our success rate with viewers submitting applications. Furthermore, our leasing office was following up with viewers whether or not they received a thumbs up/down, or were un/likely to apply from the feedback.

Solutions

As an intermediate step to a complete overhaul, the feedback screen was stripped down to the bare essentials. Viewer interest: likely, unlikely, and unsure. Urgent matters: maintenance, cleaning, and safety. Scouts are now required to write feedback to complete the viewing. These changes proved to be immediately helpful to the leasing office because we could now act on them.

Looking towards a fully updated version of the feedback, I gained insight by cross-examining the data with what action we could take and by understanding Scout and viewer behavior. Pill buttons are fast and easy so we should enhance them. The pills that were never selected and the thumbs up/down viewer rating were removed. Subjective feedback is not always binary, so introducing “neither” as a neutral option allows more nuance within the constraints of an on/off pill button design. Adding a section that allows the Scout to communicate the viewers interests such as pets, rent, or amenities allows the leasing office to be proactive about the viewer’s needs when they follow up with the viewer. If the viewer was not interested in that specific unit but “would like to see other options”, they can continue touring similar apartments nearby.

To truly understand whether a strategic approach works or not, one must always look at the data. But it does not stop there. Sometimes data is indicative of behavior, but behavior is not always captured in data. Having a conversation with the stakeholders (Scouts, viewers, and the leasing team) about their needs and interests provides the lenses by which one can filter the data to piece together a more concrete picture, paving the path towards better solutions that work.