Using BitTorrent (incl. magnet links) to share CoH media - why & how

[Keiphy]

There are a few reasons I'm writing this; primarily because I think using BitTorrent for this is a good idea.

First of all, because this comes up rather often:

Isn't BitTorrent illegal?
While it is true that BiTtorrent is used very often for illegal file sharing, BitTorrent itself is just a protocol, like HTTP, FTP, IRC, POP, etc., nothing illegal about it by itself.
Let me repeat this for emphasis: BitTorrent is NOT illegal!

For example, the possibility of making copies of a CD does not make burning CDs in general illegal and the possibility of obtaining illegal copies of movies/music/games/whatever does not make the internet itself illegal.

BitTorrent is just one of many things that have both legal and illegal uses, unfortunately the illegal ones are a high profile case, but the protocol and the clients themselves are 100% legal.

Among other things, several open source projects use it if their binaries grow really big - it helps them reduce the costs for their servers, and lets everyone help just by helping host the files for them. And it can be faster too. Personally, I DL and seed (torrent slang for hosting) LibreOffice (formerly OpenOffice), if you want to see that for yourself, the magnet link is further down in this post - or just go to the download page, they list the torrent there too: http://www.libreoffice.org/download/

As for commercial uses, Blizzard uses bittorent (with a proprietary client) for updates for WoW. Short of sinking millions of dollars into some crazy server farms that will only be needed once per patch, it is probably the only thing that they can do to deliver their patches in a halfway decent timeframe.

If you want another MMO example, the CO client is available by torrent on its main download page as well: http://co.perfectworld.com/download

These were just the examples on top of my head, Wikipedia has a much more comprehensive list here: http://en.wikipedia.org/wiki/Bittorrent#Adoption

What is bittorent?
Bittorrent is a protocol for transmitting files directly between clients, among other things, it allows people to host huge files without owning their own server.

What can it be used for?
Essentially, sharing large and huge files between any number of users. That might be your own CoH tribute video, your collection of screenshots or demorecords - it all works. This isn't as much of an issue for small files, but once you go over a few hundred megabytes, I often find the alternatives suboptimal at best.

What are the upsides/downsides compared to other methods of hosting?
Youtube is a good option for hosting videos (there are others, it's by far the most well known), but downloading requires external aids that change over time since they actively work against that, and they have a very short lists of supported quality levels and resolutions. So if you want to watch it on an old device or when your internet is down for whatever reason, or you want it in really high quality or just it's native resolution, you will need something else.

File hosting services. There are a LOT of these, and I bet you all know a few I've never seen so far. Among those that I know, the free versions come with several limitations, download speed and file size seem nearly universal. If you want them gone, you have to pay for it, often both the uploader (file size) or the downloader (speed). Splitting files helps around the size limit, but is something of a hassle, at least in my opinion. And the speed limit often is downright pathetically low, especially for big files like decent quality videos.

Your own server. If you have one, this is very nearly ideal. The only advantages bittorrent has over it are that it can stay up when your server is down for some reason, that the bandwidth can exceed what your server has, and that servers almost always cost something (unless your workplace gives you one, or something like that).
If you know something I've missed, feel free to pm me, and I'll take a look at it.

Does it have other advantages?
There are three things bittorrent is really notable for:

First, if a packet gets corrupted in transmission, the largest amount of data lost is 4mb, compared to the entire file (or the part the error occured in, if it was split). Other applications often don't even check the file integrity in the first place, leading to too many potential errors to count - if you ever had any error that was fixed by downloading a file again, it most likely was one of these.

Second, once it got going, it is incredibly robust - once you have spread a few copies around, it keeps going by itself - You are offline due to blackout/hardware fault/...? Everyone else keeps it going. Headcrash wrecks your hdd? Everyone else keeps it going and you can download it back.
Of course, if you want the file gone, it is MUCH harder to do than on youtube or filehosters. But the internet never really forgets, someone can just reupload it after all. If you don't want it shared, just don't share it in the first place.

Third, the bandwidth is about equal to the sum of the clients' bandwidth.
Essentially this means when it gets started, it's as slow as your upload; when it got going, it can be damn fast. I've seen some torrents that had a total bandwidth up in the gig/s, but that far up, you'd probably have the file before you got that far. If you want a server that is this fast, expect to pay a pretty hefty chunk of cash for it ;).

How to use it?

Get a client
First, you need a client. The ones I see most are µtorrent (Windows & OS X, the client itself is tiny, rather streamlined and easy enough to use), Transmission (Unix like, including OS X, you can build it for Windows too if you really want, but it is not officially supported - not used it myself) and Vuze (java based, so it works almost everywhere. Once the client, I've seen it rather rarely since I started looking at what the other peers use earlier this week. It has been a while since I used it myself; it had a nice amount of advanced options and statistics, but was somewhat large and could be possibly be overwhelming. I dropped it since µtorrent's tiny size appealed to me. And Vuze wasn't portable back then).

If you want more detailed information about them, you should google it yourself or just try them - there a lot of ups and downs to each of them, and I don't think there is one that can be considered universally ideal.

To share/upload, create a .torrent file
To download, skip to the next bolded line
Once you have your client, you need to make a .torrent, a tiny file that contains checksums and other such metadata. The exact details my vary by client, but I'd be really surprised if it isn't straightforward everywhere (for µtorrent, click file -> Create New Torrent).

(optional) When it asks you for trackers, just use these:
udp://tracker.openbittorrent.com:80      udp://tracker.publicbt.com:80      udp://tracker.istole.it:6969      udp://tracker.ccc.de:80

They are all free and public, and if one is overloaded (which can happen at times), the others are likely to still be running .

NOTE - nowadays, you can make a torrent without any tracker whatsoever, it then uses DHT and such, but if you use a tracker, it tends to get going notably faster.

(optional) If you want, you can manually set the piece size of your torrent, but the automatics do ok most of the time. In a nutshell, small sizes make it get going notably faster, while large sizes have less protocol overhead after they got started. Opinions vary on this, a LOT. Personally, I'd never use more than 4mb for anything (and that much only for insanely huge files), and try to get between 500 and 2500 pieces, ideally 2k to 2.5k. But again, opinions vary a lot here.

(optional) If you have a server that is hosting the file already, you can add a direct link to it under Webseeds (see also the technical note under the included magnet).

Once you have the torrent, make sure to keep seeding it for a while at the least, whenever you can if possible, it only becomes truly self reliant if enough people have it - but if you have to pause seeding for whatever reason, it will just resume where you left it, while the others catch up to each other. Even extremely sudden disconnects only loose tiny amounts of data (in most cases, far less than one MB).


There are two ways to share the file.

The classic way is to share the .torrent file your client created, be that by hosting it somewhere, sending it per mail or whatever. Since these are universally small (10-50 KB are common), this is by far easier than sending the original file.

These days, any client that is still kept up to date can use magnet links. These are incredibly compact - where a .torrent often is several KBs large and might not fit into a post in some forums (and if it does, it'd be a HUGE wall of text, and then you'd have to copy&paste it into a text file, rename it and then feed it to your client), magnets DO - for example, this is the current magnet link the to most recent version of the LibreOffice installer as of this writing:


magnet:?xt=urn:btih:B99C7D163988D86A852A1DC5F6733702B069D71E&dn=LibO_3.6.4_Win_x86_install_multi.msi&tr=http%3a//tracker.documentfoundation.org%3a6969/announce&ws=http%3a//ftp.wh2.tu-dresden.de/pub/mirrors/tdf/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//mirror3.layerjet.com/tdf/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//ftp.uni-muenster.de/pub/software/DocumentFoundation/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//ftp5.gwdg.de/pub/tdf/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//ftp.halifax.rwth-aachen.de/tdf/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//ftp.rz.tu-bs.de/pub/mirror/tdf/tdf-pub/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi&ws=http%3a//mirror.unlogisch.ch/tdf/libreoffice/stable/3.6.4/win/x86/LibO_3.6.4_Win_x86_install_multi.msi

It is still a large, ugly block of text, but easy enough to scroll past.

And this is a rather large magnet too, Artillerie's magnets in this thread http://www.cohtitan.com/forum/index.php/topic,6460.0.html are notably more compact.

Technical note - all those &ws parts specify webseeds, which are ordinary urls you can download the file at - if you replace the "%3a" with ":", you can use whatever browser you use to download it. While this bloats the magnet a bit, this allows the LibreOffice to be downloaded from all listed servers, as well as every client in the swarm (For this torrent, >460 seeds while writing this) at once. And µtorrent crops magnets at 1kb, so the list of webseeds in the above isn't even complete. For a file of just over 200mb, in my humble opinion, this level of overkill and redundancy is as glorious as it is silly.

To use it to upload, you needs to get the magnet link for your torrent (in µtorrent, you right click on your torrent and select "Copy Magnet URI" - you will then have it in your clipboard, ready to be pasted wherever you want). Then you just have to give it to whoever you want, email it, pm it, post it publicly ...

Keep in mind that it is about as public as you make the magnet link, whoever has the link can get the rest easily (on a technical note, it is theoretically possible to get it without, but that requires a several hash collisions or guessing the hash, so I wouldn't worry too much about it).

To use it to download, copy the entire thing and paste it into whatever interface your clients uses (for µtorrent, File -> Add Torrent from URL, or just CRTL+U). You might have to wait a bit for the download to start, especially if the torrent itself is really young, but that's it. All that is left after setting the download location is waiting for it to finish.

NOTE that the magnet does NOT contain an info about the file size - to know that, you can either rely on what the uploader says, and obviously, if you use it to upload something, it is polite and helpful to mention that, or wait until your client has used the magnet to fetch the .torrent file, at which point you have a list of all files, including their size (and sometimes relative folder structure as well).

Once you have finished your download, please help seed (upload) whenever you can (if you have a flat rate at least). Since this can eat up a lot of bandwidth, about every client comes with some way to limit the bandwidth used and/or an included scheduler to help work around that. Even if you have to drastically limit your bandwidth, or only seed for short periods of time, it can adds up rather quickly if a lot of people do it.

At this point, I think I've covered all the basics, but this took me rather long to write as it is, and I can just edit it later if I forgot anything or need to clarify some part.

EDIT: made it much more readable, added a few things I've missed somehow and changed the order just a bit.

[Starsman]

I just want to make this one quick post to highlight one tiny bit of yours, because I know many will skip everything with a "TLDR bitorrent is illegal!"

Bittorrent is not illegal! Its just a file transfer protocol. Blizzard even uses it to distribute it's own WoW patches more efficiently. As long as you are not downloading illegal movies (or any other copyright stuff) Bittorrent is 100% legal! (The fact that is legal is the reason it has not been blocked by every ISP)

If you skipped the above post thinking that, now go back and read Keiphy's post!  ;D

@Keipy: to make your post easier to read, you may want to bold out every question keeping the answers in normal type. Otherwise, great post!

[corvus1970]

Bittorrent is not illegal! Its just a file transfer protocol. Blizzard even uses it to distribute it's own WoW patches more efficiently. As long as you are not downloading illegal movies (or any other copyright stuff) Bittorrent is 100% legal! (The fact that is legal is the reason it has not been blocked by every ISP)

Thank you. I feel that this often isn't emphasized enough.

[Keiphy]

I just want to make this one quick post to highlight one tiny bit of yours, because I know many will skip everything with a "TLDR bitorrent is illegal!"

Bittorrent is not illegal! Its just a file transfer protocol. Blizzard even uses it to distribute it's own WoW patches more efficiently. As long as you are not downloading illegal movies (or any other copyright stuff) Bittorrent is 100% legal! (The fact that is legal is the reason it has not been blocked by every ISP)

If you skipped the above post thinking that, now go back and read Keiphy's post!  ;D

@Keipy: to make your post easier to read, you may want to bold out every question keeping the answers in normal type. Otherwise, great post!

That's some very good points, thank you ;)

I'm somewhat tempted to rant properly about how it is a great protocol, but that would most likely induce TLDR a lot ;D

Just edited it a good bit, it feels better now - if you got any other ideas to improve it, please share them :D