Author Topic: Copying/Downloading Paragonwiki?  (Read 13277 times)

Ice Trix

  • Boss
  • ****
  • Posts: 184
Copying/Downloading Paragonwiki?
« on: December 24, 2013, 12:19:03 AM »
I doubt Paragonwiki is going anywhere in a hurry, but I'm planning on using so much of it for my next PnP RPG, I don't really want to risk not having access to it (whether titan server outage, no net connection etc). An offline copy would be the bee's knees.

Would that be objectionable to the CohTitan People? I wouldn't want to bust bandwidth on them by copying it or if it was seen as bad netiquette to do so. I'd never claim it was my work or similar.

Assuming it's okay, is there an easy way to copy it all? A zip file option or torrent?

Thanks

Felderburg

  • Ask me how I got this title!
  • Elite Boss
  • *****
  • Posts: 1,615
  • Personal text? What's that?
Re: Copying/Downloading Paragonwiki?
« Reply #1 on: December 28, 2013, 10:34:47 AM »
A post from 3 1/2 years ago (http://www.cohtitan.com/forum/index.php/topic,3079.msg31563.html#msg31563) indicates that at that time, it was fine for people to print the articles to paper, so it shouldn't be much of a stretch to just get an electronic copy. But I'm not official.

I swear there was a post about this topic earlier, but I can't find anything other than the one above.

I used CIT before they even joined the Titan network! But then I left for a long ol' time, and came back. Now I edit the wiki.

I'm working on sorting the Lore AMAs so that questions are easily found and linked: http://paragonwiki.com/wiki/Lore_AMA/Sorted Tell me what you think!

Pinnacle: The only server that faceplants before a fight! Member of the Pinnacle RP Congress (People's Elf of the CCCP); formerly @The Holy Flame

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #2 on: December 28, 2013, 11:39:27 AM »
Well, presumably the mod who moved the thread to here would have commented if it was going to be a big no-no to do.
But I'll wait longer, over holiday times so responses might be slower.
« Last Edit: December 28, 2013, 11:47:46 AM by Ice Trix »

Blondeshell

  • Elite Boss
  • *****
  • Posts: 808
Re: Copying/Downloading Paragonwiki?
« Reply #3 on: December 29, 2013, 01:57:35 AM »
Well, presumably the mod who moved the thread to here would have commented if it was going to be a big no-no to do.

Actually, I don't have any authority to answer the question, and am, myself, also curious in what the possibilities are. I just moved the thread here so it wouldn't get buried by the other topics in the general forum since this deals specifically with PW.

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,291
Re: Copying/Downloading Paragonwiki?
« Reply #4 on: December 29, 2013, 04:14:07 AM »
I think I know the answer, but it's more of a question for Tony.
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Felderburg

  • Ask me how I got this title!
  • Elite Boss
  • *****
  • Posts: 1,615
  • Personal text? What's that?
Re: Copying/Downloading Paragonwiki?
« Reply #5 on: December 30, 2013, 05:22:34 PM »
I just realized that the license probably has info. Linked from the front page, GNU Free Documentation License 1.2 (the link goes to 1.3, though): http://www.gnu.org/copyleft/fdl.html

Quote
You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

So I think that's a yes, based on the first sentence there (provided you download the license with it as well).
I used CIT before they even joined the Titan network! But then I left for a long ol' time, and came back. Now I edit the wiki.

I'm working on sorting the Lore AMAs so that questions are easily found and linked: http://paragonwiki.com/wiki/Lore_AMA/Sorted Tell me what you think!

Pinnacle: The only server that faceplants before a fight! Member of the Pinnacle RP Congress (People's Elf of the CCCP); formerly @The Holy Flame

TonyV

  • Titan Staff
  • Elite Boss
  • ****
  • Posts: 2,175
    • Paragon Wiki
Re: Copying/Downloading Paragonwiki?
« Reply #6 on: January 02, 2014, 03:37:39 PM »
Sure, have at it.  I've always operated the wiki under a "use whatever you need" philosophy.  As long as you're not doing something that will crash it, or doing something seedy like downloading it to use unedited as some kind of commercial game guide (not that I think anyone would buy it at this point...), I'm fine with it.  That's why I used the GNU FreeDoc license, and I've told everyone that if they don't want the possibility of their write-ups being mercilessly swiped by someone, don't put it on the wiki, because the last thing in the world I want to do is to go after people for some kind of perceived violation of copyright or anything like that.

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #7 on: January 05, 2014, 03:56:53 AM »
Thank you Tony & others.
I didn't want to suddenly hit the server with a done of requests without warning either.

I'll probably try HTTrack to download, unless smarter people know of a better way?


Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #8 on: January 05, 2014, 03:57:51 AM »
Can you tell me how large it's likely to be?

Blondeshell

  • Elite Boss
  • *****
  • Posts: 808
Re: Copying/Downloading Paragonwiki?
« Reply #9 on: January 05, 2014, 07:10:13 PM »
I have absolutely no access to the back end of the wiki to check total storage space. However, I can see this page and do some math and estimating.

I checked 101 random articles and came up with an average content size of 5.5 KB per article. However, there's also roughly 19 KB of additional HTML coding embedded within each article. Using that total for each of the 10,044 content pages, you get ~236 MB worth of actual text. This doesn't include the 46,800 redirect, talk, and other pages that can't be easily summarized.

There's no easy way to get samples of the uploaded files, but I think it's safe to assume an average size of 100 KB per file. Some files are far larger than that, but most should be much smaller. That means the 20,000 uploaded files would take up ~1,923 MB. Put them all together, and everything should fit comfortably on one DVD with under 3 GB of total content. However, I assume you'd also likely need to have MediaWiki software on your computer in order for the links and content to work properly.

Of course, these estimates could be completely wrong, so I'm curious how close I actually am.

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #10 on: January 06, 2014, 08:16:24 AM »
When It's done, I'll post how big it is.

HTTrack saves the files as html with links intact, so browsing will be easy enough when finished.

It's slow so far, about 100MB/hour. I did 8 hours last night, which seems to be processing speed more than bandwidth - my connection isn't great but it's typically faster than that by a large factor.

It seems I didn't pause it correctly as it's started from scratch again. Sigh.

As a comparison point for interest, you can torrent Wikipedia (text only including graphs etc but not images) which is about 9 GB. But with all the lovely graphics and sounds, it's hard to judge just how huge ParagonWiki will be.


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Copying/Downloading Paragonwiki?
« Reply #11 on: January 06, 2014, 03:20:54 PM »
The database that contains all of the text content is only around 113MB. The images weigh in at 4GB. 1.2G of that is automatically generated thumbnails, but if you're using HTTrack it'll probably download both.

The wiki itself makes heavy use of DPL, which for a long time was considered deprecated due to security problems. The most recent version claims to have fixed those; I have no idea if it's backwards compatible or not. There's a bunch of other extensions like Semantic MediaWiki and a few others installed. It's definitely not something you could drop on a stock MediaWiki install and have work. An HTML dump may be more useful just to avoid fiddling with setting all those up.

JanessaVR

  • New Efforts # 12,000!
  • Elite Boss
  • *****
  • Posts: 815
Re: Copying/Downloading Paragonwiki?
« Reply #12 on: January 06, 2014, 07:51:31 PM »
I've thought about doing this myself, just to make sure PW is always available (and one more backup out there never hurts).  Is this HTTrack some sort of spider program?

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #13 on: January 07, 2014, 12:44:01 AM »
Yeah, I don't want to edit the wiki offline, just view it, so html dump suits me fine.
Up to 1.9 GB now, and HTTrack is claiming 70% done. But not sure how accurate that % will be.  HTTrack can be a bit odd, at least for me and my know how setting it up, with what it grabs. I'm hoping it will do it right, or right enough, will soon see.

Just going to leave computer on until it finishes.

JanessaVR:
Yes, Httrack is a spider/web site copier, great piece of freeware. I imagine it would be okay for me to dropbox it to you when I'm done, but it might be faster for you to down it yourself.

JanessaVR

  • New Efforts # 12,000!
  • Elite Boss
  • *****
  • Posts: 815
Re: Copying/Downloading Paragonwiki?
« Reply #14 on: January 07, 2014, 02:33:23 AM »
Let me know how it works for you.  If it gets you a good copy of the site - web pages with working hyperlinks and images included - then I'll very likely give it a shot.

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #15 on: January 11, 2014, 08:36:39 AM »
It works pretty well. Defiantly get working web pages with hyperlinks, and the default settings seem to be inclusive enough without too much external web crap.

It's proving a bit of a pain for me, my net connection is wireless, and it's fairly common to drop out at times, a few times a night. Which causes an error at that bit, which it marks in the error log but it's a bit of manual chasing to see what went wrong. Resuming or Checking links seems to set up to run completely again, i.e. download everything which might be due to the amount of dynamic/changing content on ParagonWiki.
So I'm having to check through a fair bit, and running it over and over hoping to either get a good run, or complete the missing bits.
I'm not sure yet if trying again and my connection dropping out on a bit successful in the past means the old (good) bit is kept or if it gets overwritten and is bad. If it's the later, I may never be able to finish unless I do it elsewhere, my connection will not last the 24+ hours it would need.

JanessaVR

  • New Efforts # 12,000!
  • Elite Boss
  • *****
  • Posts: 815
Re: Copying/Downloading Paragonwiki?
« Reply #16 on: January 11, 2014, 10:08:24 AM »
Hmmm...this is one of the reasons why I insist on wired connections for everything except my Kindle Fire at my house.  It may be a "wireless world," but I prefer the reliability and better speed of physical network cable connections.

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #17 on: January 11, 2014, 12:13:22 PM »
Hmmm...this is one of the reasons why I insist on wired connections for everything except my Kindle Fire at my house.  It may be a "wireless world," but I prefer the reliability and better speed of physical network cable connections.
It's a world of trade offs really. The cost and infrastructure changes on my _rental_ property wouldn't be worth the money spent even if I was allowed.
I'd much prefer my main desktop to be wired over wireless.

Felderburg

  • Ask me how I got this title!
  • Elite Boss
  • *****
  • Posts: 1,615
  • Personal text? What's that?
Re: Copying/Downloading Paragonwiki?
« Reply #18 on: January 11, 2014, 04:32:32 PM »
There's only three or so people still editing the wiki. It wouldn't be hard for us to stop for a day or two, to stop content from changing, if that is an issue. There was at least a week over the holidays, even, that nothing changed on the wiki.
I used CIT before they even joined the Titan network! But then I left for a long ol' time, and came back. Now I edit the wiki.

I'm working on sorting the Lore AMAs so that questions are easily found and linked: http://paragonwiki.com/wiki/Lore_AMA/Sorted Tell me what you think!

Pinnacle: The only server that faceplants before a fight! Member of the Pinnacle RP Congress (People's Elf of the CCCP); formerly @The Holy Flame

Ice Trix

  • Boss
  • ****
  • Posts: 184
Re: Copying/Downloading Paragonwiki?
« Reply #19 on: January 11, 2014, 10:20:25 PM »
Thank very much for that offer, extremely nice to suggest it, but I think there's too much content generated dynamically for it not to trigger an update (e.g. the random page).
It could also just be a matter if me better learning the program to stop it.

I'm on 87% with no drop outs right now, so holding breath ;)