NLP for Mere Mortals – Benchmarking Real-world Performance of Natural Language Processing Incumbents and Startups

January 09, 2017 | State of the Market Report

There is an increasing number of off-the-shelf natural language processing (NLP) products available from many developers, ranging from large incumbents to startups. For customers, it will be crucial to pick the right technology supplier(s); for that, they need a benchmark against which current and future NLP can be evaluated, on same playing field. This NLP evaluation focuses on performance in understanding technical articles, spanning across five of Lux Research’s innovation coverage areas, covering many types of events, ranging from acquisition to collaboration to product launch. We find Google’s Cloud Platform currently leads, with Open Calais and NLTK rounding out top three; generally, entity recognition is now achievable, but other tasks need more R&D.


  • Chemicals
  • CPG
  • Government & Non-Profit
  • Industrials
  • Oil and Gas
  • Utilities

Table of Contents

  • Title page
  • Table of contents
  • Landscape and methodology
  • Analysis: Testing for raw accuracy
  • Analysis: Testing for consistencyand self-awareness of mistakes
  • Outlook

Lead Analyst

Cosmin Laslau, Ph.D.
Director, Research Products


Ryan Dolen

About Lux Research

Lux Research is a leading provider of tech-enabled research and advisory solutions, helping clients drive growth through technology innovation. A pioneer in the research industry, Lux uniquely combines technical expertise and business insights with a proprietary intelligence platform, using advanced analytics and data science to surface true leading indicators. With quality data derived from primary research, fact-based analysis, and opinions that challenge traditional thinking, Lux clients are empowered to make more informed decisions today to ensure future success.

Already a Lux Client?

Sign In

Interested in Learning More?

Contact us to learn the benefits of becoming a Lux member.

Or call us now

For North America (Boston Headquarters)
+1 (617) 502-5300

For EMEA (Amsterdam)
+31 20 280 7900

For APAC (Singapore)
+65 6592-6978