Rating Inference of Sentiments from a Movie Review DataSet with Einstein Language APIs

Author : Rajdeep Dua
Last Updated : Sep 6 2017

In this article we look at how Einstein Language APIs can help find sentiment in textual data from Movie Review DataSet

Background

Salesforce launched Einstein Language APIs couple of months ago with the aim to help developers provide service which can help them detect sentiments from text data. In this article we look at how these APIs could be successfully used for famous Movie Review data set

../_images/einstein_language_phrases.png

Movie Review DataSet

There’s a thin line between likably old-fashioned and fuddy-duddy, and The Count of Monte Cristo ... never quite settles on either side.” The Rotten Tomatoes movie review dataset is a corpus of movie reviews used for sentiment analysis, originally collected by Pang and Lee [1].

This data set deals with the rating-inference problem, wherein rather than simply decide whether a review is “thumbs up” or “thumbs down”, as in previous sentiment analysis work, one must determine an author’s evaluation with respect to a multi-point scale (e.g., one to five “stars”). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, “three stars” is intuitively closer to “four stars” than to “one star”.

The dataset is comprised of tab-separated files with phrases from the Rotten Tomatoes dataset. The train/test split has been preserved for the purposes of benchmarking, but the sentences have been shuffled from their original order. Each Sentence has been parsed into many phrases by the Stanford parser. Each phrase has a PhraseId. Each sentence has a SentenceId. Phrases that are repeated (such as short/common words) are only included once in the data.

train.tsv contains the phrases and their associated sentiment labels. We have additionally provided a SentenceId so that you can track which phrases belong to a single sentence. test.tsv contains just phrases. You must assign a sentiment label to each phrase. The sentiment labels are:

  • 0 - negative
  • 1 - somewhat negative
  • 2 - neutral
  • 3 - somewhat positive
  • 4 - positive

DataSet size

Total Labels : 5 Number of examples : 155835
Phrase Label Number of Samples
negative 0 7061
somewhat negative 1 27237
neutral 2 79463
somewhat positive 3 32876
positive 4 9198

We uploaded the DataSet using standard curl commands and trained the model

Creating DataSet in Einstein

We need to do a HTTP POST call to https://api.einstein.ai/v2/vision/datasets/upload with appropriate parameters as shown below.

curl -X POST -H "Authorization: Bearer <TOKEN>" \
             -H "Cache-Control: no-cache" \
             -H "Content-Type: multipart/form-data" \
             -F "path=https://www.dropbox.com/s/oytprz7xbfxwzs3/train_original.tsv?dl=1" \
             -F "type=text-sentiment" https://api.einstein.ai/v2/language/datasets/upload

Note : We are using Asynch request and you will have to ping the endpoint below to check training status

Response will be similar to listing below

{
   "id":1011538,
   "name":"train_original.tsv; filename*=UTF-8''train_original.tsv",
   "createdAt":"2017-08-31T11:27:36.000+0000",
   "updatedAt":"2017-08-31T11:27:36.000+0000",
   "labelSummary":{
      "labels":[

      ]
   },
   "totalExamples":0,
   "available":false,
   "statusMsg":"UPLOADING",
   "type":"text-sentiment",
   "object":"dataset"
}

Note the dataset id which is 1009482 in our case.

Check Upload Status of the DataSet

curl -X GET -H "Authorization: Bearer <TOKEN>" \
            -H "Cache-Control: no-cache" \
            https://api.einstein.ai/v2/vision/datasets/1011538
{
   "id":1011538,
   "name":"train_original.tsv; filename*=UTF-8''train_original.tsv",
   "createdAt":"2017-08-31T11:27:36.000+0000",
   "updatedAt":"2017-08-31T11:35:33.000+0000",
   "labelSummary":{
      "labels":[
         {
            "id":101461,
            "datasetId":1011538,
            "name":"0",
            "numExamples":7061
         },
         {
            "id":101462,
            "datasetId":1011538,
            "name":"1",
            "numExamples":27237
         },
         {
            "id":101463,
            "datasetId":1011538,
            "name":"2",
            "numExamples":79463
         },
         {
            "id":101464,
            "datasetId":1011538,
            "name":"3",
            "numExamples":32876
         },
         {
            "id":101465,
            "datasetId":1011538,
            "name":"4",
            "numExamples":9198
         }
      ]
   },
   "totalExamples":155835,
   "totalLabels":5,
   "available":true,
   "statusMsg":"SUCCEEDED",
   "type":"text-sentiment",
   "object":"dataset"
}

Training the DataSet

We will make a HTTP POST call to URL https://api.einstein.ai/v2/vision/datasets/<id> where <datasetId> is 1011538 in our case

Training Command

curl -X POST -H "Authorization: Bearer <TOKEN>" \
             -H "Cache-Control: no-cache" \
             -H "Content-Type: multipart/form-data" \
             -F "name=movie sentiment analysis" \
             -F "datasetId=<DATASET ID>" https://api.einstein.ai/v2/vision/train
{
   "datasetId":1011538,
   "datasetVersionId":0,
   "name":"movie-review-aug31",
   "status":"QUEUED",
   "progress":0,
   "createdAt":"2017-08-31T11:37:05.000+0000",
   "updatedAt":"2017-08-31T11:37:05.000+0000",
   "learningRate":0.0,
   "epochs":0,
   "queuePosition":3,
   "object":"training",
   "modelId":"37USWXTWBQ34IHRWREK5MK6FIA",
   "trainParams":null,
   "trainStats":null,
   "modelType":"text-sentiment"
}

Once the model is trained the model metrics can be seen

{
   "createdAt":"2017-08-31T12:19:51.000+0000",
   "metricsData":{
      "f1":[
         0.08388814913448735,
         0.4086347282711608,
         0.7416109300863204,
         0.44483558684694774,
         0.2914492171818547
      ],
      "labels":[
         "0",
         "1",
         "2",
         "3",
         "4"
      ],
      "testAccuracy":0.5911347645855504,
      "trainingLoss":1.0270950430044974,
      "confusionMatrix":[
         [
            63,  835,  362, 119, 5
         ],
         [
            36, 2158, 2561, 625,  14
         ],
         [
            12, 1432, 12973, 1508, 43
         ],
         [
             6,  631, 2791, 2780, 244
         ],
         [
             1,  112,  331, 1015, 363
         ]
      ],
      "trainingAccuracy":0.5774626660814023
   },
   "id":"37USWXTWBQ34IHRWREK5MK6FIA",
   "object":"metrics"
}

Interpret Training Results

F1 scores Label wise

Phrase Label Number of Samples
negative 0 0.0838
somewhat negative 1 0.4086
neutral 2 0.7416
somewhat positive 3 0.4448
positive 4 0.2914

Predicting the Label from the Phrase

Let us take a phrase from the test dataset and try to predict the Label

Prediction Command

We will send a prediction HTTP POST request to Url https://api.einstein.ai/v2/vision/predict with authorization token and path to the image to be predicted.

curl -X POST -H "Authorization: Bearer <TOKEN>" \
             -H "Cache-Control: no-cache" \
             -H "Content-Type: multipart/form-data" \
             -F "sampleId=Photo Prediction"  \
             -F "sampleContent=@./airplane/10638.png" \
             -F "modelId=37USWXTWBQ34IHRWREK5MK6FIA" \
             -F "document=An intermittently pleasing but mostly routine effort"
             https://api.einstein.ai/v2/language/sentiment

Results

{
   "probabilities":[
      {
         "label":"3",
         "probability":0.37577492
      },
      {
         "label":"2",
         "probability":0.34644908
      },
      {
         "label":"1",
         "probability":0.19133164
      },
      {
         "label":"0",
         "probability":0.044502497
      },
      {
         "label":"4",
         "probability":0.041941825
      }
   ],
   "object":"predictresponse"
}

Prediction Result

  • Input : An intermittently pleasing but mostly routine effort

  • Output :

    • Neutral (label 2) - 35%
    • Somewhat Positive (label 3) - 37.5%

Summary

As we can see the einstein does an interesting prediction of rating inference of movie review.