Conversational Agent: Movie Assistant

This application is an Application Starter Kit that is designed to get you up and running quickly with a common industry pattern. It can serve as the basis for your own applications that follow that pattern. This app was created to highlight the combination of the Dialog and Natural Language Classifier (NLC) services as a Conversational Agent. Another application that demonstrates this pattern is the What's in Theaters application that is available in the Watson Developer Cloud website's Application Gallery.

Give it a try! Click the button below to fork the repository that contains the source code for this application into IBM DevOps Services, which then deploys your own copy of this application on Bluemix automtically:

IMPORTANT:

1. The application uses mock data for movie suggestions until you provide an API Key for themoviedb.com in your application's source code, which you can not do when using the Deploy to Bluemix button. See step 9 in the Getting Started section for information about getting and using an API key in an application that you create and deploy manually.
2. When the application is first run, it will automatically train a classifier for the Natural Language Classifier service. This process takes about 20 minutes. While the classifier is being trained, the user can only interact with the Dialog service.

Table of Contents

• How this app works
• Getting Started
• Running the application locally
• Application Starter Kit
• Conversational Agent
  • When to use this pattern
  • Best practices
  • Reference information
    • Dialog
    • Natural Language Classifier
• Troubleshooting

How this app works

This app provides a conversational interface that lets users search for movies based on a set of criteria. The dialog system is built to understand natural language related to searching and selecting movies. For example, "I'd like to see a recent R rated drama" returns the names of all R-rated dramas that have been released in the last 30 days.

This application's dialog system also understands variations of text, which allows users to phrase their responses in many different ways. For example, the system might ask, "Do you want to watch an upcoming movie or one that's playing tonight?" The user can reply with "tonight" or "Show me movies playing currently," and the system understands that the user wants to know about current movies.

The conversation is designed to obtain three pieces of information before searching the movie repository:

• Recency: The system determines whether users want to know about currently playing movies or upcoming movies
• Genre: The system understands movie genres, such as action, comedy, and horror
• Rating: The system understands movie ratings, such as G, PG-13, and R

Users can search across all genres and ratings by answering "no" to the corresponding questions.

Getting started

The application is written in Node.js and uses npm. Instructions for downloading and installing these are included in the following procedure.

Important: If you used the Deploy to Bluemix button to deploy an instance of this application to Bluemix automatically, you will have to delete that application and the services that it used before you can build and deploy an application manually. You can use the cf apps command to see the instances of the Dialog and NLC services that your application uses, use the cf delete application-name command to delete the application, and use the cf delete-services service--name command to delete each of the Dialog and NLC service instance that the application used.

The following instructions explain how to fork the project on GitHub and push that fork to Bluemix using the command-line interface (CLI) for Cloud Foundry. If you want to run the application locally, see the next section, Running the application locally:

1. Log into GitHub and fork the project repository. Clone your fork to a folder on your local system and change to that folder.

2. Create a Bluemix account. Sign up in Bluemix or use an existing account. Watson services in beta are free to use, as are GA services in the standard plan below a certain usage threshold.

3. if it is not already installed on your system, download and install the Cloud-foundry CLI tool.

4. If it is not already installed on your system, install Node.js. Installing Node.js will also install the npm command.

5. Edit the manifest.yml file in the folder that contains your fork and replace application-name with a unique name for your copy of the application. The name that you specify determines the application's URL, such as application-name.mybluemix.net.

```yml
applications:
- services:
  - dialog-service
  - classifier-service
  name: application-name
  command: npm start
  path: .
  memory: 256M
```

6. Connect to Bluemix by running the following commands in a terminal window:

```sh
$ cf api https://api.ng.bluemix.net
$ cf login -u <your-Bluemix-ID> -p <your-Bluemix-password>
```

7. Create an instance of the Dialog service in Bluemix by running the following command:

```sh
$ cf create-service dialog standard dialog-service
```
**Note:** You will see a message that states "Attention: The plan `standard` of service `dialog` is not free.  The instance `dialog-service` will incur a cost.  Contact your administrator if you think this is in error.". The first 1000 API calls per month to the Dialog service are free under the standard plan, so there will be no charge if you remain below this limit.

8. Create the Natural Language Classifier service:

```sh
$ cf create-service natural_language_classifier standard classifier-service
```
**Note:** You will see a message that states "Attention: The plan `standard` of service `natural_language_classifier` is not free.  The instance `classifier-service` will incur a cost.  Contact your administrator if you think this is in error.". The first NLC  instance that you create is free under the standard plan, so there will be no chanrge if you only create a single classifier instance for use by this application.

9. Sign up at themoviedb.com and get an API key.

10. Add the API key from themoviedb.com to the app by editing the line 29 of the file api/services.js to read:

```js
var TMDB_API_KEY = process.env.TMDB_API_KEY || <Your themoviedb.com API Key>;
```

11. Push the updated application live by running the following command:

```sh
$ cf push
```

The first time it runs, the application can take up to 20 minutes to train the classifier based on data from themoviedb.com. It will also create the following files in your source directory:

• A dialog flow using: training/dialog_and_classifier.xml and writes the dialog id to the file /training/dialog_id
• A classifier using: training/classifier_training.csv and writes classifier id to the file /training/classifier_id

You should not need to reference these, but if you do, you can retrieve these ids at application-name.mybluemix.net/api/services, where application-name is the name that you gave your application in step 5 of the previous list. The response will be similar to:

{
  "dialog_id": "24045716-d5cc-4748-afed-a4ea0287b737",
  "classifier_id": "563C46x19-nlc-3140"
}

Running the application locally

First, make sure that you followed steps 1 through 9 in the previous section and that you are still logged in to Bluemix. Next:

1. Create a .env.js file in the root directory of the project with the following content:

module.exports = {
TMDB_API_KEY: 'TMDB API KEY HERE',
VCAP_SERVICES: JSON.stringify({
  dialog: [{
    credentials: {
      url: 'https://gateway.watsonplatform.net/dialog/api',
      username: 'DIALOG USERNAME HERE',
      password: 'DIALOG PASSWORD HERE'
    }
  }],
  natural_language_classifier: [{
    credentials: {
      url: 'https://gateway.watsonplatform.net/natural-language-classifier/api',
      username: 'NATURAL LANGUAGE CLASSIFIER USERNAME HERE',
      password: 'NATURAL LANGUAGE CLASSIFIER PASSWORD HERE'
    }
  }]
})
};

2. Copy the username, password, and url credentials from your dialog-service and classifier-service services in Bluemix to the previous file. To see the service credentials for each of your service instances, run the following command, replacing <application-name> with the name of the application that you specified in your manifest.yml file:

```sh
$ cf env <application-name>
```

Your output should look similar to:

```sh
System-Provided:
{
"VCAP_SERVICES": {
  "dialog": [{
    "credentials": {
      "url": "<url>",
      "password": "<password>",
      "username": "<username>"
    },
    "label": "dialog",
    "name": "dialog-service",
    "plan": "standard"
 }]
}
}
```

3. Install any dependencies that a local version of your application requires:

```sh
$ npm install
```

4. Start the application by running:

```sh
$ gulp
```

5. Open http://localhost:5000 to see the running application.

Application Starter Kit

An Application Starter Kit (ASK) is a multi-service sample application that is designed to demonstrate common industry patterns and best practices around Watson services.

This sample application highlights one of those industry patterns, known as a Conversational Agent.

Conversational Agent

First, make sure you read the Reference Information to understand the services that are involved in this pattern. This reference information will explain common terminology for these services such as classifier, confidence scores, intents, training, and so on.

The following image shows a flow diagram for a Conversational Agent using the Natural Language Classifier and the Dialog service:

Using the Dialog service and the Natural Language Classifier service

Since the Dialog service uses expert rules to match user inputs to intents, the service typically has high accuracy. The Natural Language Classifier service is a statistical system that yields high recall. The combination of the Dialog and Natural Language Classifier services therefore creates a high precision, high accuracy system.

For a given input, a trained Natural Language Classifier responds with a list of intent classes and the corresponding confidence scores. Dialog only uses the top two classes to decide how to proceed with the conversation. The following checks are performed by the Dialog service:

• The USER_INTENT from the Classifier service is considered valid when class(0).confidence >= upper_confidence_threshold.
• Ask user to confirm the USER_INTENT when upper_confidence_threshold >= class(0).confidence > lower_confidence_threshold.
• Ask user to disambiguate between USER_INTENT(0) and USER_INTENT(1) when class(0).confidence + class(1).confidence > upper_confidence_threshold.
• Reply with the default response when none of the previous checks are true.

In this case, class(0) is the top class and class(0).confidence is its confidence score. Similarly, class(1) is the second best class with class(1).confidence being its confidence score. In these checks, upper_confidence_threshold and lower_confidence_threshold are floats 0 - 1, and their values are obtained by running cross-validation tests with the classifier on a given data set.

---

When creating an application based on the conversational pattern, you should first understand what the user is trying to do. Is he looking up an actor? Is she searching for upcoming movies? Is she simply looking to have small talk with Watson? We call these the user's Intent. To extract the user intent from the user input, we train the Watson Natural Language Classifier using various examples of possible user requests. The service then uses deep machine learning techniques to return the top predicted classes.

Here is an example of what we will use to train the classifier:

Who directed The Hobbit, LookupDirectors
Who starred in The Hobbit, LookupActors
Drama, SearchMovies
I'd like to see a recent drama, SearchMovies
Show me whats playing, SearchMovies
Something sexy, SearchMovies
Good day, ClosingTalk
Hi, OpeningTalk
Who is the producer of Vacation?,LookupDirectors
Help,RepairTalk
Robert,GiveName
I want to look up movie stars,LookupActors

Next, we need to acquire any additional information that is required to complete the user's request. To do this, we rely on the Dialog service. The Dialog service tracks and stores information obtained during the conversation until we have all the information required to complete the task. In the case of this application, it's searching for a movie, an actor, or a director.

When to use this pattern

• You need to perform a task that requires user input and you want to mimic a conversation
• You want to provide a conversation experience like Siri or Cortana

Some best practices

• When using the Natural Language Classifier, there should be approximately 10 classes. Each class should have 15 examples of possible user inputs. This provides the service with enough information to build the deep machine learning model that will classify future user inputs.
• When using the Dialog service, define different opening sentences in the dialog flow (.xml file). This will prevent repetitive conversations where the dialog always asks the same questions.

Reference information

The following links provide more information about the Dialog and Natural Language Classifier services, including tutorials on using those services:

Dialog

• API documentation: Get an in-depth knowledge of the Dialog service
• API reference: SDK code examples and reference
• API explorer: Try out the REST API
• Creating your own dialog: Design your own dialog by working through a tutorial
• Natural conversation tutorial: Advance tutorial on how to create a conversion like the one in this sample application

Natural Language Classifier

• API documentation: Get an in-depth knowledge of the Natural Language Classifier service
• API reference: SDK code examples and reference
• API Explorer: Try out the API
• Creating your own classifier: How to use the API to create and use your own classifier
• Understanding how Dialog uses the output from the Natural Language Classifier

Troubleshooting

When troubleshooting your Bluemix app, the most useful source of information is the execution logs. To see them, run:

$ cf logs <application-name> --recent

Open Source @ IBM

Find more open source projects on the IBM GitHub Page

License

This sample code is licensed under the Apache 2.0 license. Full license text is available in LICENSE.

Contributing

See CONTRIBUTING.