Merge pull request #6 from ustervbo/master

Update MS Flow with check for repeat feeds

README.md

@@ -112,65 +112,87 @@ A few folks on twitter noted that it would be nice to have this feed posted to a
 ### 6. Tweet using Microsoft Flow
 
 
-1. Download `ms_flows/TrickleRSSfeedstoTwitter.zip`
+1. Download `ms_flows/TrickleRSSfeedstoTwitter.zip` or `TrickleRSSfeedstoTwitter_w_notification.zip` if you want automatic notifications
 2. Go here: https://flow.microsoft.com
 3. Follow the steps to sign up for a new account (NB: the same gmail trick that works for twitter *does not* work here)
-4. Create connectors:
+4. Go to Data -> Connections and create connectors:
    * RSS
    * Twitter
-   * Notification (optional)
-![](./screenshots/msf_00_connections.png) 
-5. Go to 'My flows' and import `TrickleRSSfeedstoTwitter.zip`. The imported flow has the following components:
-![](./screenshots/msf_01_workflow_overview.png) 
+   * Notification (optional) <br>![](./screenshots/msf_00_connections.png) 
+1. Go to 'My flows' and import `TrickleRSSfeedstoTwitter.zip`. The imported flow has the following components: <br>![](./screenshots/msf_01_workflow_overview.png) 
 6. Edit the the imported flow:
    1. Correct the RSS address (default is the porcelain.crab feed)
    2. Correct the Flow name (will automatically be used in the subject line of the notification)
 
+The flow will tweet the items in the RSS feed each and every day - also if there is nothing new under the sun. Why this is, is a little obscure, but Microsoft Flow clearly does not check if the feed has been updated, but this might also be because NCBI does not include a `<pubDate>` tag in the feed. Also, you can obviously tweet the exact same text on different days.
+
+If it is important not to tweet the same paper over and over, look at **Check duplicate tweets**.
+
 #### Step-by-step flow creation
 
 Clearly some of the following steps, such as the truncation of the paper title and the try/catch, can be left out. I use them to be on the safe side of things, but you can go wild.
 
+Especially adjusting the title length might be a little too much, since about [0.25% of the new coming papers on Pubmed](https://rubde-my.sharepoint.com/:x:/g/personal/ulrik_stervbo_rub_de/EdYocDyvmn9Km1DLBaPkEOcBapeM1Cy9f4UgYgSwFxEC0A?e=JgEFvx) have to be truncated.
+
 1. Create new flow from blank
 2. Select 'Schedule' connector
 3. Set the interval to 24 hours
 4. Select the connector 'List all RSS feed items'
 5. Add the feed URL
-6. Initialize two string variables
-![](./screenshots/msf_02_workflow_initial_steps.png)
+6. Initialize two string variables <br>![](./screenshots/msf_02_workflow_initial_steps.png)
    *  `item_title`
    *  `tweet_text`
-7. Optional: Select the 'Send me an email notification' action from the Notification connector
-![](./screenshots/msf_03_send_notification_expression.png)
+7. Optional: Select the 'Send me an email notification' action from the Notification connector <br>![](./screenshots/msf_03_send_notification_expression.png)
    * Set the Subject to `@{workflow()['tags']['flowDisplayName']}: Number of items: @{length(body('List_all_RSS_feed_items'))}` or something equally meaningful
    * Set the body to `Feed check at @{utcNow()}` or something else
    * Add a parallel branch
 8. Add the `Apply to each` connector
-9. Add 'Body' as the 'Select an output from previous steps' value
-![](./screenshots/msf_04_truncate_title.png)
+9.  Add 'Body' as the 'Select an output from previous steps' value <br>![](./screenshots/msf_04_truncate_title.png)
 10.  Add a 'Set variable' action from the 'Variables' connector, followed by a 'Condition' action from the 'Control' connector
 11.  Set the the value of `item_title` to `Feed title`
 12.  Set the condition to be length of `item_title > 253` 
     * The maximum number of characters in a tweet is 280, the URL will always be 23 characters, we will truncate long titles and add `... `; so 280 - 23 - 4 = 253
     * The length of `item_title` is obtained through `@{length(variables('item_title'))}`
 13.  To the True branch, add a 'Set variable' action from the 'Variables' connector
-14.  Set the the value of `tweet_text` to `@{substring(variables('item_title'), 0, 253)}... Primary feed link`
-![](./screenshots/msf_05a_set_tweet_text_truncated.png)
+14.  Set the the value of `tweet_text` to `@{substring(variables('item_title'), 0, 253)}... Primary feed link` <br>![](./screenshots/msf_05a_set_tweet_text_truncated.png)
 15.  To the False branch, add a 'Set variable' action from the 'Variables' connector
-16.  Set the the value of `tweet_text` to `variables('item_title') Primary feed link` or `Feed title Primary feed link`
-![](./screenshots/msf_05b_set_tweet_text.png)
+16.  Set the the value of `tweet_text` to `variables('item_title') Primary feed link` or `Feed title Primary feed link` <br>![](./screenshots/msf_05b_set_tweet_text.png)
 17. Create a Try/Catch block to avoid a flood of errors in case the flow executes without any new RSS items:
     1. Add a two `Scope` actions from the 'Control' connector
     2. Set the letter to run only in case of failure:
-       1. Go to the ellipsis, select 'Configure run after', leave only 'has failed' checked
-![](./screenshots/msf_06_create_try_catch.png)
-18. In the Try block, set the 'Post a Tweet' action from the Twitter connector
-![](./screenshots/msf_07_post_tweet.png)
+       1. Go to the ellipsis, select 'Configure run after', leave only 'has failed' checked <br>![](./screenshots/msf_06_create_try_catch.png)
+18. In the Try block, set the 'Post a Tweet' action from the Twitter connector <br>![](./screenshots/msf_07_post_tweet.png)
 19. Set the tweet text to `@{variables('item_title')`
 20. Add the 'Delay' action from the Schedule connector and set Count and Unit to 5 Minutes. With this delay all feed items (max 100) are tweeted in a little more than 8 hours
-21. In the Catch block, set the 'Send me an email notification' action from the Notification connector
-![](./screenshots/msf_08_notify_duplicate_tweets.png)
+21. In the Catch block, set the 'Send me an email notification' action from the Notification connector <br>![](./screenshots/msf_08_notify_duplicate_tweets.png)
 22. Fill in the information you find reasonable - the name of the flow is given by the expression `workflow()['tags']['flowDisplayName']`
 
+#### Check feed repetition
+
+Microsoft Flow provide no way to store information between execution of flows; the work around is to use Google Sheets, Excel Online, or some other cloud based spread sheet. By assuming that a change to the title of the first feed item indicate a completely new feed, we just need to remember a single entry, and not each paper tweeted. Here the usage of Google Sheets will be described, but the ideas will also apply to other cloud spread sheets.
+
+The template `TrickleRSSfeedstoTwitter_no_repeats.zip` is available for download in `ms_flows`.
+
+1. Create a file on Google Sheets called `RSS item titles` and insert the following
+
+    | `__PowerAppsId__` | `TwitterBot` |
+    | ----------------- | ------------ |
+    | 1                 | NA           |
+
+    The column `__PowerAppsId__` is used my Microsoft Flow to identify the row of interest.
+
+1. Create a connection to the 'Google Sheets' connector
+2. Initialize a string variable `first feed title`, and set the variable after the RSS feed has been listed with the expression `body('List_all_RSS_feed_items')?[0]?['title']`. <br>![](./screenshots/msf_09_check_repeats_vars.png)
+3. Insert the 'Get Rows' action from the 'Google Sheets' connector
+   * Select the file and sheet, and no other options. <br> ![](./screenshots/msf_10_get_rows.png)
+4. Add a 'Condition' action and test equality between `first feed title` (the first paper of the feed) and the value stored in the sheet obtained with the expression `body('get_rows')?['value']?[0]?['TwitterBot']`<br>![](./screenshots/msf_11_test_repated_feed.png)
+5. If the title is unknown
+   1. Update the information in the Google Sheet<br>![](./screenshots/msf_12_remember_feed_and_tweet.png)
+      *  Insert the 'Update Row' action from the 'Google Sheets' connector
+      *  Select the file, sheet, and set the row ID to 1
+      *  Set the option TwitterBot to `first feed title`
+   2. Add the `Apply to each` connector and continue as above
+
 ### 7. Tweak, revise, repeat
 
 Make sure you follow and check your own feed. If it seems like it's posting rubbish, go tweak the search terms. For example, I noticed that the search above, using 'phylogen*' in pubmed, gathers all sorts of papers that just happen to have estimated a phylogenetic tree. That's not what I want, since the focus of most of those papers is often nothing to do with phylogenetics. So I revised my searches to be more specific. I now use this: