Twine Hacks Twine Hacks / Items

Two RSS ways to merge twines

Get Feed

Now that RSS feeds (for public twines) do not need passwords, the related twine RSS feeds can [easily] be subscribed to. In the news reader (I use Google Reader), the related feeds can be grouped under a single subject / folder. If the containing folder is viewed (instead of the individual twine feeds), all of the content from all of the aggregated feeds will be seen together.  Instant merged twine.

The main problem with this solution, is that items that are shared to more than one of the aggregated twines will show up once for each twine (in the aggregation list) that they are shared with.

To address that issue, I am testing a Yahoo pipe http://pipes.yahoo.com/. The pipe aggregates the related twine feeds, then passes the combined content through a deduplication step. The output that I then subscribe to contains all of the unique content from the set of twines. A potential problem with this method, is item comments. If a shared item has a comment added before the item is picked up by the RSS feed, the deduplication might not show that instance of the item. The comments would still be available by following the comment links at the bottom of the instance that did get through the dedup. Once the item itself has made to the feed, any comments added to any instance of the shared item *should* be unique enough to each show up in the feed. Duplicates can also still get through if the twines feeds are not in sync. The dedup process only looks at the feed items for the 'current' group / fetch.

Here is a public / published pipe I created for some virtual reality related twines. I suggesting taking a copy and modifying to suit. I am still playing with it, to tweak what shows up in the feed, so it might not work the same tomorrow.
http://pipes.yahoo.com/pipes/pipe.run?_id=368c43eace01796c54dd9aec00b770cf&_render=rss

This is a very simple aggregate everything and dedup. More complex pipes are possible, to filter content from each of the source twines (individually or in groups) before merging and deduplication. For example content from one (or more) twines could be included only if one or more specific tags are present (or not present). Trying to play with the options in between doing other things.

Based on http://www.twine.com/item/11vtcfrtb-5s/a-way-to-merge-twines?discussion=flat#c_25

Comments

  • Public Comments

    • Phil Duby Phil Duby
      14 months ago


      A possible improvement on this method.
      http://www.twine.com/item/11j0stkp0-2tj/merging-twines
      The only downside I see to using the twine grouping technique described in the above linked note *before* passing the content to yahoo pipes, is the limitation on the number of entries that Twine will included in a single feed pass (about once every 3 hours). With multiple *active* grouped twines, it will be much easier to end up with enough 'new' entries to surpass the [default] limit of 30 items, which will result in items being *missed*. That limit includes an entry for every twine (in the group) that each item is shared to. So a group of 5 twines, with 10 new bookmarks that are shared to all 5 twines counts as 50 entries. That blows the limit without even trying, even though after duplicate removal, it would only be 5 items. Using yahoo pipes to gather the twines together means the feed limit is available for EACH twine, instead of the combined twines.
      Twine Hacks

      • Phil Duby Phil Duby
        14 months ago


        Another thing to watch out for: With the original merge in pipes technique, the group of twines could be updated by simply adding or removing entries from the pipe input, one entry per twine. With the merge in twine technique, plus duplicate remove with pipes, changing the twine list (updating the bookmark) ALSO requires the pipe input URL to be updated. A change in the list of twines in the "ball of twine" is a new URL, which must be updated in the pipe input field. Change must be done in 2 places.
        Twine Hacks

    • webferret webferret
      14 months ago


      http://feedrinse.com/


      If you've got unruly RSS feeds, we've got your back.

      Feed Rinse is an easy to use tool that lets you automatically filter out syndicated content that you aren't interested in. It's like a spam filter for your RSS subscriptions.
      Twine Hacks

    • Karen Newton Karen Newton
      13 months ago


      I love Yahoo Pipes... thanks for posting this, will take a close look at your pipes. Let me know if you need any assistance. I developed a series of videos on Yahoo! Pipes and Dapper, read more about it at http://www.roguerssprofits.com .

      Karen Newton
      Twine Hacks
    Add a Comment
Report This

Twine is about discovering, collecting and sharing the content that interests you. Learn More

Join Twine

Stats

First Posted By

Phil Duby

First Comment By

Phil Duby

Tags

Community Tags

Who's Interested In This?

Forgot your password?