Modify

Ticket #8 (closed feature: fixed)

Opened 7 years ago

Last modified 2 years ago

BitTorrent Functionality

Reported by: and Owned by:
Priority: blocker Milestone: 4.1 Beta 1 (New Features and Fixes)
Component: BitTorrent Severity:
Keywords: Cc:
Who will test this: And

Description (last modified by anna) (diff)

Automatic creation and seeding of videos via bittorrent.

Create a plumi.torrent component, added to the Plumi buildout. Include option to install this inside the Plone control panel.

Firstly attempt to create our own tracker, as well as the option to use the  http://openbittorrent.com/ tracker as part of this component. If major difficulties encountered in this then possibly think about only offering the  http://openbittorrent.com/ option for the first iteration of this feature. Either way, it will include a torrent client to seed the torrents.

Issues with running own tracker: compiling another bit of software within the buildout (not a major issue), hosting providers may not allow torrent traffic (but this would be a problem with running a client anyway), complexity of setting this up as part of this feature (hard to tell at this stage).

Issues with  http://openbittorrent.com/ - reliance on another service (if this service goes down then torrent sharing goes down), possibly issues with retrieving statistics (number of seeders etc.)

In terms of our provider allowing torrent traffic, Mike will email to explain our usage, it seems likely at this stage they will allow it in our case. If not we will look at running a client elsewhere, and using the  http://openbittorrent.com/ solution.

Attachments

Change History

comment:1 Changed 7 years ago by and

  • Type changed from defect to feature

comment:2 Changed 6 years ago by shrikant

just stumbled over a small problem in /opt/plumi-trunk/ATMediaFile/bt-server/INSTALL.txt

1.3. Python bindings for BitTorrent

change to BitTorrent directory $ sudo python24 setup.py install

what is meant by 'BitTorrent directory'? Is that the directory where I want to keep my files? maybe the docs should be a bit more verbose there.

comment:4 Changed 6 years ago by and

  • Milestone changed from 0.3 - Plone 3 to 0.4

comment:5 Changed 4 years ago by and

  • Owner andycat deleted
  • Who will test this set to And

comment:6 Changed 3 years ago by anna

  • Milestone changed from 4.3 to 4.1 Beta

comment:7 Changed 3 years ago by and

Info from J re pushing torrents automatically to torrent trackers

"you can add the trackers while creating the torrent, to get listed on the sites you have to upload with there /upload form or contact each site and figure out how to do it via scripts, i.e. pirate bay you need to enter captchas unless you get a vip account, with that you can script it more easily. some others would also accept atom/rss feeds with links to torrents and would pull them in."

comment:8 Changed 3 years ago by and

  • Priority changed from critical to blocker

comment:9 Changed 3 years ago by anna

  • Description modified (diff)

Have added changes to ticket description as per meeting with Mike this morning (feel free to change if inaccurate).

comment:10 Changed 3 years ago by mike

  • Status changed from new to closed
  • Resolution set to ready for testing

We've built collective.seeder, a product that whenever a new file (video) is uploaded, it symlinks it's blob file to the original file name, and creates a torrent for it.

You can specify as many torrent trackers (announce urls) as you want. This way, we can set up our own tracker, plus add openbittorent (piratebay) and other open trackers. I've updated testing.plumi.org which is now working with the openbittorent tracker. (A note to transmission bittorrent client users: transmission does not yet support trackers with udp urls so they can not connect to openbittorrent and a couple of other trackers. If they're lucky they might get it through DHT).

This enables our torrents to stay alive if one or two trackers go down.

I've also tested a local tracker (xbtt) these days that we can set up for em.org, although there are things to be discussed (better via skype) before proceeding with it.

I've also set up a torrent client daemon (deluged) on testing.plumi.org that automatically starts seeding new torrents.

comment:11 Changed 3 years ago by mike

Example:http://testing.plumi.org/Members/clopy/videos/dsfs/

comment:12 Changed 3 years ago by anna

  • Status changed from closed to reopened
  • Resolution ready for testing deleted

hey, the testing site was down today, dave got it back up, but i see "S = 0" below the flashplayer, which i imagine means seeders = 0?

can't see any link, or way to download the file via BT.

comment:13 Changed 3 years ago by anna

The control panel looks good here:  http://testing.plumi.org/@@seeder-controlpanel

I haven't tested adding any other trackers as I'm not sure which to add.

But as Mike said, it makes sense to use multiple open trackers maintained by others, rather than running our own, given that the client we are using can provide info re. seeders/leechers.

comment:14 Changed 3 years ago by anna

 http://testing.plumi.org/Members/clopy/videos/dsfs/

Tested this using Transmission. Successfully downloaded, after looking not-available for about 30 seconds (no seeders were listed during this time). Would have tried on another client if I had time, but don't tonight.

Shame that Transmission isn't a reliable option - what cross-platform client should we recommend? Can we prompt users to use this client using a tooltip / alt-text, or some other UI signpost?

Re. UI - Font size on video view page needs increasing to be the same as "Download / Watch Ogg Theora Version". Also it should say "Seeders" not "S".

comment:15 Changed 3 years ago by mike

  • Status changed from reopened to closed
  • Resolution set to ready for testing

comment:16 Changed 3 years ago by anna

  • Status changed from closed to reopened
  • Resolution ready for testing deleted
  1. I just uploaded a new video to testing site to test this. 15 minutes later no BitTorrent link had appeared in the view page:

 http://testing.plumi.org/Members/anna/videos/climate-change/

Is this enabled on testing.plumi.org to work for new videos as they are added? This would be good in order to test properly.

  1. I just tested the existing test file clopy added using uTorrent. It downloaded successfully - but 10 minutes later, only 1 Seeder is still listed in the page. I am seeding this from my computer.

 http://testing.plumi.org/Members/clopy/videos/dsfs/

Can you show me somewhere that these statistics are updating correctly? They don't seem to be as far as I can tell.

  1. How long should we anticipate wait times for torrents to be created and seeded by Deluge, and how long until the Seeder stats are updated? Without this info I can't test effectively.

comment:17 Changed 3 years ago by anna

relating to 2. - half an hour later Seeders were still 1, even though I was seeding it myself (should be 2)

comment:18 Changed 3 years ago by anna

relating to 2. - about 2 hours later i checked the page again and now it says Seeders = 2. So it does appear to work, just has a considerable lag. is there any way to bring this time lag down, or should we just have to live with this?

comment:19 Changed 3 years ago by mike

  • Status changed from reopened to closed
  • Resolution set to ready for testing

There was an issue with mktorrent and zope. We've dumped mktorrent and are now creating the torrents ourselves. Normally there shouldn't be any delay for the creation of the torrent files from now on. I just tested it on a new video:  http://testing.plumi.org/Members/clopy/videos/torrent1/

Seeders are updated every time our torrent client requests an update from the torrent tracker. Forcing it to update the stats sooner could get our IP banned from the tracker for some trackers. With more trackers on the torrent list, we should have shorter update times though.

comment:20 Changed 3 years ago by anna

  • Status changed from closed to reopened
  • Resolution ready for testing deleted

Wow, this works beautifully. Torrent appeared very quickly (as soon as the page had reloaded). Torrent downloaded perfectly.

Will be interesting to see the impact of multiple trackers on the delay on the statistics, but this is totally fine for now. I imagine that for many videos the torrent will only be seeded by us in any case, but perhaps in the use-case of the Indo TV stations they will be seeding content over long periods of time so delays in statistics will not be such an issue anyhow.

Ticking this off, it's so exciting to see this working so well!!! Big congrats :)

comment:21 Changed 3 years ago by anna

  • Status changed from reopened to closed
  • Resolution set to fixed

comment:22 Changed 2 years ago by andyman

@mike

see  https://trac.transmissionbt.com/ticket/117 You might want to know that a whopping FIVE YEARS LATER the impossible was made possible: transmission now DOES have UDP support! :) And it works well, albeit uncertain whether any distros are already shipping the UDP-powered version. If things go really bad, you will have to compile the release yourself.

Andy

View

Add a comment

Modify Ticket

Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.