VM Study: Methodology evaluation

Problems with the survey, lessons learned, and applications for the MalaDIO HRP

Evaluation Overview

The survey gained some interesting data, but suffers from several methodological failings. The results suggest interesting general trends that illuminate the experiences of a VM community, and expose areas for further focussed research. However, the results do not provide conclusive answers to the questions often raised by those with VM. 

Sample title
209 sample size far too small

When combined with the number of questions, the sample size is far too small for statistically-relevant answers, especially when dividing the sample group to look for patterns within subgroups.

Sample title
Many demographics under-represented

The questions concerning gender, age, ethnicity and location expose a skewed participation that is not representative of the whole VM community. Whether these factors impact on the experiences and treatment efficacy is unknown.

Sample title
Some imprecise questions

The lack of precision in the wording of many questions allowed for ambiguity in the meaning of responses.

Sample title
Too many metrics

The survey attempted to address too many aspects related to the experiences of VMeurs. While this was useful in exposing areas for further research, it dilutes the confidence within any one area of focus.

Sample size 

209 well short of statistical significance

With a German study estimating the prevalence of VM being 1% of the population (https://www.uptodate.com/contents/vestibular-migraine/abstract/5), this puts the worldwide VM population at 76 Million. 

Responses account for 0.00027% of the international population. In better-represented countries such as the US, this figure is 10x at 0.0027%.

Gender

Women are more likely to suffer from vestibular migraine. Estimates range from a ratio between 1.5 and 5 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4255594/). Even at the highest end of this range, with 88.5% female respondents, women are over-represented in the sample.

Age

The average age of onset for VM is 40 (http://www.scielo.br/pdf/anp/v74n5/0004-282X-anp-74-05-0416.pdf). A 40 year old respondent would be placed in the 70-79 Year of Birth Option, which represented 30.8% of the survey cohort. The options above and below this range each represent around 25% of the sample.

Ethnicity and location

The location of respondents matches closely the membership of the support groups that were used to promote the survey. There is a clear bias for western English-speaking countries. Ethnicity data is largely representative of the population of these countries.

Lack of guidance allows for ambiguous meaning in answers

Many of the questions asked lacked detail that might have led to a higher sense of confidence in the meaning of answers. Questions about the value of supplements and medications lacked clarity in the way in which supplements helped, and the single-option nature failed to address the fact that these substances may be beneficial in different ways, or combine negative and positive impacts on respondent wellbeing.

In an effort to ensure privacy and confidentiality, age ranges were grouped in decades. This makes any kind of accurate tracking of responses against age unfeasible.

All breadth, little depth

The survey was deliberately broad in its focus, seeking to provide a snapshot of the experiences and approaches of people with vestibular migraine.

Many of the areas of focus would have benefitted from follow-up questions where a particular response was made. An example of this might be allowing the user to indicate what the nature of negative effects and interactions was with regard to supplements and medications. This was avoided as it would have made an already lengthy survey burdensome for the respondent. The survey as presented had a 23% dropout rate, where users began the survey but failed to complete it (No responses from this group were included).

The analysis tool allows for fine-tuned filtering of results based on responses to multiple answers. However, when filtering by a single response to even one question, the number of respondents may become too small to derive any meaningful data.

Ensuring sincere participation

Having the survey anonymous was done to simplify the requirement of protecting the privacy of respondents, but this does raise issues regarding the validity of the data set.

Surveys of any kind have the risk of insincere responses from surveyees, but anonymity adds to this risk. Those taking the survey could rush sections of the survey, choosing the fastest option, even if it doesn't match their experiences, without any perceived risk.

Although the survey system was designed to prevent multiple submissions by disallowing multiple entries from the same internet IP address, this restriction is easily circumvented.

There is no reason to believe that responses in the survey were anything other than sincere and accurate, but it is nonetheless a matter to be taken into consideration.

Getting better data

Higher participation numbers will help ensure better confidence in the statistical relevant of the data. This will require better publicity and communication of goals of the research as tangible to the target group

Having tighter demographic targeting will allow future research to gain the numbers it seeks and reducing issues of sample bias threatening the legitimacy of the data.

The focus and structure of the questions needs to be much more focussed and interactive. Each survey should not seek to cover too many variables, and additional questions should be elicited in specific circumstances.

Respondents should be given more guidance within the survey to ensure that they understand precisely what is being asked of them.. 

Targeting

In addition to the improvements derived from the 'Lessons learned' section above, the data from the survey raises potential areas for further research and changes in approach.

An interesting focus for future research could be the difference between Amitryptiline and Nortriptyline in terms of their side effects, symptomatic improvements and dosage. The difference in the reported benefits of these two drugs in the survey asks more questions than it answers. Only a more focussed study could address these questions.

Longitudinal

Relying on the recollections of respondents about lifestyle factors, medications or supplements from a  long time ago is problematic. Memory is fallible, and this approach does nothing to account for external factors that may have biased answers.

By using a longitudinal study format, respondents could be focussed to either maintaining a consistent routine, or varying factors deliberately, in order to get a clearer correlation between metrics over a period of days or weeks.

Identified participation

Future surveys should seek to authenticate users to ensure that participants meet study requirements, while also ensuring the privacy and anonymisation of released data.

By authenticating users, MalaDIO can secure a higher level of trust in responses, prevent duplicate responses, and provide the possibility of following-up on respondents for future research (with their consent) depending on their responses.

There are existing technological solutions that can offer this service and meet stringent ethical and legislative standards.

Want to dig into the data yourself?

There is significant room for improvement in terms of methodology for future surveys. Problems with the survey are discussed and actions to be taken are proposed.

Thank you

This survey was promoted and advised by other organisations crucial to the vestibular disorder community

The information presented here is the result of a survey, and shows aggregated data submitted by VM sufferers. This information does not constitute medical advice, and MalaDIO does not endorse any medications or other treatments. Any changes to your treatment should be taken under medical advice.