I was excited when I got an email back in June asking me to take the MTA Customers Count Survey. I take the subway almost every day, and as an essential worker during the early days of the Covid-19 pandemic, this was something that didn’t change even when most of the people I knew were working from home. I felt like my opinion counted for something — I was tired of hearing from people who rarely took the subway saying it was unsafe, and I wanted to set the record straight. So I eagerly clicked the link, went through the initial questions asking me which modes of transportation I used, which lines and stations I frequented, and then got to the heart of the survey: my subjective experience.

This was where I started to suspect that the survey was flawed and would produce biased results. The questions in this section all asked me to rate my experience, asking how satisfied I was, on a scale of “Very Dissatisfied” to “Very Satisfied,” with conditions like “Panhandling on board the trains,” “People behaving erratically on trains,” and “People experiencing homelessness on trains.”

In truth, I am dissatisfied with all of these conditions, but not because they impact my safety on board trains; I’m appalled by our city’s failure to help the individuals who must resort to asking strangers for some cash or food, and the fact that homeless individuals sometimes need to sleep on the subway because homeless shelters are a less safe option than sleeping on the street or on the train. But I understand that if I say I am “Very Dissatisfied” with these conditions without the ability to provide further context, it’ll just feed into the media frenzy that poor and homeless people on the subway are the factor making the system less safe.

Since the survey didn’t allow me to skip the questions, I abandoned ship and responded to the email asking me to take the survey instead to outline my grievances.

Survey design is important, but I doubt it’s something that most people who take the MTA Customers Count survey think about to the same degree that I do. Most people will just answer the questions and call it a day, not considering the subtle ways that question wording and answer format can bias results. But surveys that ask strongly worded or extreme questions, have questions that are unclear or poorly worded, or use a response scale that doesn’t make sense given the question all produce response bias. Sometimes a biased survey is intentional so that researchers can collect data to answer a question in a desirable way. Often, biased surveys are just the result of poorly trained researchers who don’t fully understand the importance of survey design.

https://twitter.com/jelMPA/status/1595136659997483010

I was surprised and happy when, a few days after I sent my email, a response popped into my inbox from WBA Research, the company that administers the survey.

“Thank you for reaching out to us,” it began. “We have received a number of comments similar to yours in response to these items, and will be taking them into consideration for the fall Customers Count survey.”

Feeling optimistic that MTA did care about administering an unbiased survey, I told myself that I’d have something to look forward to in the fall.

Sure enough, on Nov. 22, I got the email to take the fall survey. Excited to see what improvements may have been made to the survey, I dove right in. But the survey asked the same questions, with the same illogical satisfaction scale, about panhandling, people behaving erratically, and individuals experiencing homelessness. As far as I can tell, the only difference is that the questions about people masking on board the trains have been removed (presumably because the mask requirement was eliminated in September).

Once again, I couldn’t answer the questions in good faith, and I abandoned the survey without completing it. 

It’s hard to say whether the MTA is purposefully administering a biased survey, but the results it produces are still being used to paint a certain picture about customer experience on the subway. The MTA’s own report of the spring survey claims that “customers’ biggest concerns varied across the different modes of transit, although there are some connecting threads. Customers on subways, buses, and railroads all reported concerns with people behaving erratically on trains or at their destinations.” The question about erratic behavior, which only asked about my “satisfaction” with that condition, is now being used to paint a picture that customers are “concerned” about it.

In getting such a skewed perspective of MTA customer experiences, the authority is failing to collect key data that it can use to actually improve people’s experience using transit. Ridership is still significantly lower than pre-pandemic levels, and it’s hard to believe that pushing the same tired narrative that people feel the system is unsafe and crime-filled will attract people to return.

As a near-daily rider, I’m happy to report that I almost never feel unsafe on the train or bus. I know that when I use transit, I’m safe from traffic violence, and that trains are one of the safest modes of transportation. But I’m also somebody who doesn’t really have a choice but to take the subway; I certainly can’t justify taking a car from my home in Upper Manhattan to my office near Columbus Circle, my office doesn’t have great bike storage options, and I don’t have the option to work from home. Plus, my commute on the subway is quick and easy. It’s a no brainer.

For the many people who do have a choice but are deciding that transit isn’t their best option, MTA needs to better understand their decision. If the MTA truly cares about improving ridership, as well as transit riders’ experience, pushing the same flawed survey over and over will never help the transit agency achieve its goals. At best, the survey will continue to waste customers’ time without eliciting anything useful. At worst, these survey results will continue to be used to justify an ever-increasing police presence in the system, which we already know isn’t improving safety.

MTA can, and must, do better to evaluate the pain points and what will improve people’s experience using the system. 

Jesse Lang (she/her) has a Masters in Public Administration from Baruch College, where she studied quantitative and qualitative research methods and policy analysis. Follow her on Twitter at @jelMPA.