Monday, February 4, 2013

Q: How accurate is Google Flu Trends?

Update March 14, 2013: In 2012–13, Google Flu Trends did not successfully track the target flu indexes in the U.S., France, or Japan. Here are my slides from a talk at the Children's Health Informatics Program (March 14, 2013).

Why this happened is a mystery. Google has said they will present their own view some time this fall. I think the divergence suggests that one needs to be careful about trusting these kinds of machine-generated estimators, even when they work well for three years in a row. It can be hard to predict when they will fall down. (And without an underlying index that is still measured, you might never know when it has stopped working.)

I did an interview with WBUR's CommonHealth blog in January and again in February, and spoke on the radio in January.

Summary: At this point [Feb. 4, 2013], it appears likely that Google Flu Trends has considerably overstated this year's flu activity in the U.S. But we won't be able to draw a firmer conclusion until after the flu season has ended. I don't know why the model broke down this year but am eager to learn, when and if Google comes to a similar conclusion. For now, I suspect this episode may provide a cautionary tale about the limits of inference from "big data" and the perils of overconfidence in a sophisticated and seemingly-omniscient statistical model.
I am not an expert on the flu and you should not make health decisions based on Quora. You should get vaccinated, wash hands often, cover a cough, stay home from work if sick, and follow the CDC's advice: Seasonal Influenza (Flu).

Here are the nationwide figures Google Flu Trends has estimated since its launch, and the underlying CDC index it tries to predict:

The CDC reported Friday (Feb. 1) that for the week of January 13, 2012, through January 19, 2013, 4.5% of doctor visits were by patients with a fever > 100 degrees F and a sore throat or cough (what's known as influenza-like illness). (This was a revision of the original figure of 4.3% that they published the previous Friday before.)

By contrast, on Jan. 20 Google had finalized its prediction for the same statistic: 10.6%. This 6.0-percentage-point difference is larger than has ever occurred before.

For that week, in eight out of the ten HHS regions, Google's predictor was more than double the CDC's figure. (For example, here in New England where I am, Google was saying 14.2% of doctor visits were for influenza-like illness, vs. a CDC figure of only 2.9%.)

Google's scientists quite properly point out that the CDC will adjust its data retroactively, as more data comes in for past weeks. Thus the "red line" on the above graph can change over time, whereas the "blue line" is locked down in all but its very last data point. However, the CDC's adjustments have been modest so far, and even for recent weeks where the Google-CDC divergence is very large, the total number of sites reporting to the CDC has now reached close to the most it ever hits. Further dramatic revision of those data points seems unlikely.

In my nonexpert view, at this point there is little chance that Google Flu Trends's estimates can somehow be vindicated. A similar episode happened in 2009, causing the previous tweak seen above.  Some of their approach may need to change if they want these divergences, repaired only after the fact, to happen less often.

Why the model did not work this year is an exciting mystery, and I am eager to learn the answer, when and if Google reaches a similar conclusion that something went awry.

  • Did one or two of the 160 search queries dramatically increase in popularity? Or was the effect seen over all 160 queries used in the model?
  • Did Google's decision not to retrain the model since 2009 make the difference? If they had retrained the model knowing what they knew in mid-2012, would this season's estimates have been better or worse? Were there hints last summer that, in retrospect, suggest they were mistaken not to retrain the model, and if so can those hints improve the decision process for retraining in the future?
  • Can we evaluate the effect of different retraining policies (e.g. retrain every year, retrain every month vs. the current policy of retraining only on certain triggers), and how they balance various risks? What should the triggers be?
  • Is it possible to estimate flu intensity using queries chosen by a computer without human intervention for their retrospective accuracy? Would the predictions be better if humans intervened to make sure every query made sense as a flu predictor? Or would this simply introduce more problems?

The promise of real-time disease-activity estimates is a valuable idea that has the potential to save lives. Google is the most sophisticated company in the world at this kind of inference, and the fact that even they can apparently stumble suggests that this is really, really tricky. I hope Google improves their technique and continues to attempt it. I'm happy to help them in any way if they could use it, but I doubt they will need me.


My graph above may yield a different impression from Google's own graph of their performance, at . That graph looks like this:
Although there is nothing incorrect about this plot, there are a few things that Google could make clearer to reduce the possibility of confusion among readers who have not closely read the scientific papers:

  • The graph shows a "hindcast" from Google's "2009" Flu Trends predictor, the one launched in September 2009 just after the end of the plot. No data point on this graph was actually displayed to the public as a contemporary estimate. From the launch in November 2008 until the end of this plot, Google displayed predictions from the "2008 algorithm." The data from that algorithm is no longer available from Google's site; I had to trace it from a bitmap image that Google submitted in a scientific paper to PLoS ONE about the algorithm's difficulties in 2009 and subsequent improvements (Assessing Google Flu Trends Performance in the United States during the 2009 Influenza Virus A (H1N1) Pandemic).
  • The graph is almost entirely of training data; in other words the CDC data that was used to design the 2009 Google algorithm. It could avoid some confusion if Google were clearer about how much of this graph was actually predicted, versus information that the algorithm was given going in. That would look like this: 
  • The plot ends at the launch of the current algorithm and hasn't been extended to the present. In other words, every single data point in my plot above is missing from Google's plot, and none of the data points on Google's plot are on my plot. In my view, the important thing is the prospective performance of the Google Flu Trends system as it actually estimated the flu. This is the exact opposite of what Google, with its own plot, implicitly expresses it thinks is the relevant figure of merit. (I'm not saying they really believe this; just what the graph says to me.)
  • Although the data shown is from the "2009" algorithm, the only reference Google gives is to their earlier scientific paper about the 2008 algorithm, published in "Nature" ( There is no mention of the later paper or the change in algorithm.
  • In the Nature paper, Google reported that the 2008 algorithm had a 97% mean correlation with the CDC data on a held-out verification set, which is a fantastic result (even higher than the 90% that Google had achieved on their own training set!). In the PLoS ONE paper, published three years later in a less prestigious venue, Google reported that the 2008 algorithm's actual correlation with the first wave of early-2009 "swine flu" was only 29%. I have heard cynics hyperbolize that the purpose of Nature is to publish fantastic-seeming results so that they can be debunked under subsequent scrutiny by less-prestigious journals. It's depressing to see a case where that was somewhat realized.
  • Although Google wrote in the PLoS ONE paper that "We will continue to perform annual updates of Flu Trends models to account for additional changes in behavior, should they occur," and a similar statement in the Nature supplement, in practice Google has not updated the algorithm since September 2009. As they write below, they determined that an update wasn't necessary. But they could make this more clear -- as well as discussing how they determine whether or not to update the model -- in their papers and Web site.

The fact that Google decided not to update the model for 2012-13, and subsequently the model performed poorly in 2012-13, suggests that the procedure for deciding when an update is necessary may need to be reworked. On the other hand, it's possible that even if Google had updated the model, the divergence would have been just as bad (or worse). The difference may mean different things for how Google Flu Trends can be improved in the future. These are questions I sincerely hope Google examines and answers in a future scientific paper or Web site update.


  1. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this.

    Hadoop Training Chennai | Big Data Training Chennai

  2. Join FITA has offered quality Python Training in Chennai at affordable price with help of highly skilled faculties having more than 7 years of experience in Python, for more details about this advanced python training call at 91 98403-76887.
    Python Training in Chennai|Python Taining|Python Training Institutes in Chennai

  3. Thanks for sharing this niche useful informative post to our knowledge, Actually SAP is ERP software that can be used in many companies for their day to day business activities it has great scope in future.
    SAP course in chennai|SAP Training in Chennai|SAP Training Chennai|sap training in Chennai

  4. Excellent post!!! Java is most popular and efficient programming language available in the market today. It helps developers to create stunning desktop/web applications loaded with stunning functionalities. J2EE Training in Chennai | JAVA Training in Chennai

  5. Thank you for the useful article. It has helped a lot in training my students. Keep writing more.
    best big data training in Chennai

  6. The expansion of internet and intelligence in business process lead the way to huge volume of data. It is important to maintain and process these data to be efficient in data handling. Hadoop Training in Chennai | Big Data Training in Chennai

  7. Welcome to Wiztech Automation - Embedded System Training in Chennai. We have knowledgeable Team for Embedded Courses handling and we also are after Job Placements offer provide once your Successful Completion of Course. We are Providing on Microcontrollers such as 8051, PIC, AVR, ARM7, ARM9, ARM11 and RTOS. Free Accommodation, Individual Focus, Best Lab facilities, 100% Practical Training and Job opportunities.

    Embedded System Training in chennai
    Embedded System Training Institute in chennai
    Embedded Training in chennai
    Embedded Course in chennai
    Best Embedded System Training in chennai
    Best Embedded System Training Institute in chennai
    Best Embedded System Training Institutes in chennai
    Embedded Training Institute in chennai
    Embedded System Course in chennai
    Best Embedded System Training in chennai

  8. I prefer salesforce tool for CRM. It is a cloud based crm tool preferred by many professionals.
    Salesforce training in Chennai|Salesforce training institute in Chennai

  9. Paris airport transfer - Parisairportransfer is very common in Paris that provides facilities to both the businessmen and the tourists. We provide airport transfers from London to any airport in London and also cruise transfer services at very affordable price to our valuable clients.

    Paris taxi
    Paris airport shuttle
    paris hotel transfer
    paris airport transfer
    paris shuttle
    paris car service
    paris airport service
    disneyland paris transfer
    paris airport transportation
    beauvais airport transfer
    taxi beauvais airport
    taxi cdg airport
    taxi orly airport

  10. Buzz Applications is offering website for your product or your company!!

  11. I am following your blog from the beginning, it was so distinct & I had a chance to collect conglomeration of information that helps me a lot to improvise myself.
    CCNA Training in Chennai | CCNA Training Institute in Chennai | Best CCNA Training in Chennai

  12. That is it! You are currently sending SMS upgrades!SMS Marketing Applications On the off chance that you need to change settings and alter the SMS messages that are sent, keep perusing.

  13. this is very use full information.....and thanks for sharing....
    CCNA Training classes Kuwait

  14. I learned some new information. Thanks for sharing.

    Magento Development in Chennai

  15. Useful Information
    one and only affiliate agency in south INDIA, earn money online from affiliate network in india

  16. My Arcus offer java training with 100% placement. Our java training course that includes fundamentals and advance java training program with high priority jobs. java j2ee training with placement having more exposure in most of the industry nowadays in depth manner of java

    java training in chennai


  17. useful information

    Online recruitment for bank jobs and government jobs and you can get Notification and application to apply online for bank jobs and govt jobs

  18. GK Updates (GK), Latest Current Affairs, General knowledge, Current Affairs, Daily Current Affairs, GK, Todays GK, Daily GK Updates,This Year Latest Current Affairs, India Current Affairs GK, General, Knowledge, Quiz, Awareness, Questions, Answers, Explanation, Interview, Entrance, GK,Exams.

  19. Useful Information……
    Recruitment voice contains Daily GK Updates, Bank Recruitment, Government jobs, Bank jobs, Interview Tips, Banking News, GK Updates and Latest Jobs. @

  20. Great article. Keep sharing such a informative post.

    web designing institute in chennai

  21. Nice Information :
    Telugu Cinema Contains Telugu Cinema News, Latest Movie Reviews, Actor, Actress, Movie Galleries And Many More @

  22. Great article. Glad to find your blog. Thanks for sharing.

    web designing training in chennai

  23. Nice Information:
    Telugu Cinema Contains Telugu Cinema News, Latest Movie Reviews, Actor, Actress, Movie Galleries And Many More Telugu Cinema News

  24. Nice Information
    one and only affiliate agency in south INDIA, earn money online from affiliate network in india

  25. Useful Information……
    Recruitment voice contains Daily GK Updates, Bank Recruitment, Government jobs, Bank jobs, Interview Tips, Banking News, GK Updates Bank Recruitment

  26. Hibernate Training Institutes in ChennaiHibernate Training Institutes in Chennai Hibernate Online Training Hibernate Online Training Hibernate Training in Chennai Hibernate Training in Chennai Java Online Training Java Online Training

  27. Java Corporate Training Companies in Chennai Corporate Training Companies in India Corporate Training Companies in India

    Corporate Training in Chennai Corporate Training in Chennai

    Corporate Training Companies in India Corporate Training Companies in India

    Java Corporate Training Companies in India Java Corporate Training Companies in India

  28. The Google Apps Marketplace is another online store for incorporated business applications that permits Google Apps clients to effortlessly find You can now customize your Google .com with a foundation picture.

  29. this is very use full information.....and thanks for sharing....
    qlikview Training in Chennai


  30. Our Complete in depth Java training course takes you to TOP Level IT companies with high end package. Arcus Offers Java J2EE real time training with placement assurance..
    java training in chennai

  31. Thank you so much for sharing this useful information. ETL Testing Online Training is excellent solution for learning from home MSBI Online Training

  32. I use this service when I need to address several issues that are important to keep in touch with the client, not to lose it. Often I am using discounts on birthday. A good tip to stimulate demand.

  33. it is very usefull inforamation also a super site for latest telugu news updates.

  34. Hi, the information you provided here is very informative and useful for me. If anyone interested to learn AngularJS Training reach GangBoard. MSBI Training | Hadoop Training

  35. If you are really looking for a genuine advice and suggestion for Yahoo issues, you can dial a Yahoo Customer Care to have any kind of suggestion. Their suggestions and advices are free, so you can find any time with their toll-free number.