NLP for Mere Mortals – Benchmarking Real-world Performance of Natural Language Processing Incumbents and Startups

January 09, 2017 | State of the Market Report

There is an increasing number of off-the-shelf natural language processing (NLP) products available from many developers, ranging from large incumbents to startups. For customers, it will be crucial to pick the right technology supplier(s); for that, they need a benchmark against which current and future NLP can be evaluated, on same playing field. This NLP evaluation focuses on performance in understanding technical articles, spanning across five of Lux Research’s innovation coverage areas, covering many types of events, ranging from acquisition to collaboration to product launch. We find Google’s Cloud Platform currently leads, with Open Calais and NLTK rounding out top three; generally, entity recognition is now achievable, but other tasks need more R&D.

Coverage Area

  • Digital Transformation

Table of Contents

  • Title page
  • Table of contents
  • Landscape and methodology
  • Analysis: Testing for raw accuracy
  • Analysis: Testing for consistencyand self-awareness of mistakes
  • Outlook

Lead Analyst

Cosmin Laslau, Ph.D.
Director, Research Products


Ryan Dolen
Mark Bunger