Author Topic: New efforts!  (Read 7326612 times)

jlann2003

  • Underling
  • *
  • Posts: 1
Re: New efforts!
« Reply #17340 on: May 27, 2015, 04:56:59 PM »
Ugh me welders just want to play scrapper not program one :P

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17341 on: May 27, 2015, 05:18:58 PM »
I often feel that way about backup software. I've yet to find any that doesn't suck in some way or another. Even the really good ones have some Achilles' heel that you don't  know about until you discover that it chokes on condition X, and of course you have one or two servers that cannot be set up any other way and trigger that condition.

So I occasionally threaten to write my own, and have built half of one several times for specialized purposes, but never made any general enough and polished enough to sell. Hmm...

We use ShadowProtect mostly for Windows, which works fairly well but as you say, even it has failure modes, typically surrounding volume snapshots.  Any backup software that runs on the actual OS you're trying to back up inevitably will run into a problem with having to run on that OS, often when you need back ups most - when things start to go wrong.  Since we virtualize our servers, we use Veeam whenever we think we're going to be running into OS problems.  Since Veeam doesn't run within the OS, the odds of a failure drop substantially (mostly, Veeam chokes when you run too many backup jobs simultaneously on a backup server without enough RAM).

I've come to believe that backing up from outside the server, like via hypervisor, is the only possible way to even attempt to make a bulletproof backup system.  Veeam's not perfect, but it seems to be pointing in the right direction.  If your storage is fast enough, I particularly like the reverse incremental feature, something I've asked for for years.  Instead of making a full, then an incremental for each day after the full, it makes a full then when it makes an incremental it fold the incremental into the full then "pushes out" the overwrites to a delta incremental.  Thus, the full is always current to today, and you have trailing incrementals for each day previous.   In that way, you can keep as much history as you want, up to and including forever, and at any time delete the oldest history by just deleting those files.  Also, the closer your restore point is to today, the faster the restore.  Unlimited history and no incremental restore penalty.

If I was going to write a commercial backup system, I'd start there.  What Veeam lacks, like most image-based backups, is a global catalog for file-based restore options.  I've been thinking of writing one.  You take the most recent backup, automount and catalog it, then stick that into a database.  Then you use that to file search and then automount the correct image delta and bingo.

Fireheart

  • Boss
  • ****
  • Posts: 117
Re: New efforts!
« Reply #17342 on: May 27, 2015, 07:25:21 PM »
Like a little kid, I am Fascinated when people talk over my head.  I can almost-Almost grasp the significance of the discussion.  It doesn't bother me, though, because it's interesting!

Be Well!
Fireheart

Nitekilla

  • Minion
  • **
  • Posts: 35
Re: New efforts!
« Reply #17343 on: May 27, 2015, 07:35:53 PM »
I am missing City of Heroes so much that the dreaded cake room in the cave mission sound exciting :)

MM3squints

  • Elite Boss
  • *****
  • Posts: 777
Re: New efforts!
« Reply #17344 on: May 27, 2015, 07:39:21 PM »
I am missing City of Heroes so much that the dreaded cake room in the cave mission sound exciting :)

Speaking of which, was there a reason why caves on hero side was purple/bluish inside? I always wondered about that.

Noyjitat

  • Elite Boss
  • *****
  • Posts: 717
  • Guess who cares?
Re: New efforts!
« Reply #17345 on: May 27, 2015, 07:53:31 PM »
Dfferent rock type maybe? Rogue isles is in a different part of the world than paragon. Croatoa and Praetoria had unique caves too.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17346 on: May 27, 2015, 08:26:47 PM »
Like a little kid, I am Fascinated when people talk over my head.  I can almost-Almost grasp the significance of the discussion.  It doesn't bother me, though, because it's interesting!

Its not that complicated.  Its just the technical short hand that makes it opaque.  Like talking about testing arcanatime chains on pylons, that means something to (some) City of Heroes players, and no one else.

For example, I'm pretty much decided to go with client-side key state engines and translated key state combo messages.  What the heck does that mean?

Pretty much everyone who ever learns programming learns a useful lie.  We're taught or learn that the keyboard is an electronic typewriter that records the keys people type on it, and we can read those keys with some lines of code in our favorite language.  So the user types "a s d f" and when we read the keyboard, we read an "a" and then an "s" and so forth.  Its a convenient lie that radically simplifies using keyboards, and it works 90% of the time.

Then we all get the bright idea to write a game.  Probably a simple game, maybe even something not even a game, just something where we input stuff and then stuff happens.  Maybe we move a dot around on the screen.  Maybe that dot leaves a trail behind.  Maybe we check to see if the dot moves into a spot that already has a trail.  And that's how we write a Tron lightcycle game.  In any case, at some point we write something that stops treating the keyboard like a typewriter, and more like a game pad.  We want the user to press the "a" and hold it down to go left, press and hold the "d" to go right, and maybe press the space bar to shoot something.  And that's when we discover our convenient lie is not so convenient anymore.

We read the keyboard like we always did, and when the user holds down the "a" we get a stream of "a"s.  So far so good, but when the user releases the "a" we still keep getting "a"s for a while.  That's because the keyboard buffer is holding some "a"s it read from the player that we didn't use yet.  That's not so good.  Also, when the player holds down the "a" we probably notice that first we get an "a" and then for a while we get nothing.  And then, after a delay, we start getting more "a"s.  That's because of something called key repeat delay.  When you are typing, you aren't pressing keys instantaneously.  You are pressing them and holding them down for a split second.  But to the computer, that's an eternity.  The computer is so fast, it needs to make some assumptions and one of them is that normally when you are typing, it should treat a key press for a certain short length of time as a single keystroke.  That's so that when you are typing, tttttthhhhhhiiiiiissssssss dddddooooooooeeessssnnnnn'''''tttt hhhaaaaaaaaapppppppppppppeeennnnnnnnnn.  But in a game, that same behavior would cause your game to see a single "a" as a quick move to the left, followed by a pause where nothing happens even though the player is still holding down the key, and then a bunch of "a"s you interpret as moving to the left, and then when the player wants to stop and they release the key they will still move to the left for a while because your program is still reading "a"s.

Worse, suppose you hold down "a" and then press down on the space bar.  Maybe that's a strafe left: keep moving left and also fire.  When your program reads the keyboard, in most languages what you'll see is a stream of "a"s and then a stream of "space"s.  When the space bar is pressed, you'll only see that, even though "a" is still being pressed.  That's because the program code you're using assumes the keyboard shoots out whatever key is being pressed, and cannot report two keys being pressed simultaneously.  So it reports the last one pressed.  You'll never see "a space a space a space" or anything like that, so you cannot tell what the player is actually doing.  And certain other things like function keys or shift keys have similar problems: you won't know which one is pressed, because you'll only see what the program code thinks those keys do.  Whether you hold the left shift or the right shift or CAPSLOCK is down, when you hit the letter "a" you'll get a capital "A" and that's it.  You'll never know *why* you got a capital "A".  And if CAPSLOCK is down *and* you hold down shift?  You'll get a lower case "a" even though the user is in fact holding down the shift key.  Oops.

This is a pain, and its because you're using a programming library that is making life easy for you by reading the keyboard for you, and letting you pretend its a typewriter.  Its not.  Its actually a more complex input device, and you could read it "raw" so to speak.  For that, you need to learn how your particular language (and operating system and environment) allow you to do so.  For most modern OSs, it comes down to keypress events.  In the background, something (your OS, some library function, some piece of software you normally don't have to worry about) is constantly reading the keyboard, waiting for something to happen.  For our purposes, two things can happen we care about.  The keyboard can say "this key is now pressed down" and "this key was down but now its back up again."  Key Down, and Key Up.  Now, we can see *exactly* what's going on at the keyboard, and now we can write games that use the keyboard as a game controller.  When the user presses "a" we see that immediately.  When they let go, we see that immediately.  What if the user holds down A and then presses SPACE?  We'd see a Key Down for A, and then a Key Down for SPACE.  We would know the user was holding both keys down because we never saw him release the A.

This is great, but it comes at a cost.  You have to keep track of what's going on yourself.  Right now, at this moment, which keys are being pressed and which are not?  If you lose an event like a Key Down, you'll miss the fact the user is pressing that key.  You aren't going to get another Key Down event unless he lets go and presses again.  And if your game mixes up controls and typing, you have to figure out when the player wants to move left, and when he wants to type an "a" and not "aaaaaaaaaaaa".

That's why we all learn the lie first.  Its a pain to keep track of Key Up and Key Down and how long did the user hold that key down did he want to give us a stream of letters or only one?  The lie does the work for us.  But when you're writing a game with real time input, you usually have to throw the lie away, and do the work yourself.

And that's how keyboards work (basically).  Not complicated really.  But the first time you run into the problem as a programmer, its usually a pain.  And when you are dealing with networking, the threat of losing an event and losing track of what's happening at the keyboard becomes more of a reality.  Unless your computer is crappy or really overloaded, that's rarely if ever going to happen on your PC directly.  But when you have to send that stuff over the internet, anything can happen.  So its probably better to keep track of what's going on at the PC itself, and send the actual state of which keys the player is holding down to the server so the server can never lose track.  You just have to do it efficiently.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17347 on: May 27, 2015, 08:29:14 PM »
Speaking of which, was there a reason why caves on hero side was purple/bluish inside? I always wondered about that.

Well, technically speaking if you go down deep enough below Paragon City, you end up in Oranbega.  So who knows what sort of weirdness is in the rocks below Paragon City.

JanessaVR

  • New Efforts # 12,000!
  • Elite Boss
  • *****
  • Posts: 815
Re: New efforts!
« Reply #17348 on: May 27, 2015, 10:55:00 PM »
Well, technically speaking if you go down deep enough below Paragon City, you end up in Oranbega.  So who knows what sort of weirdness is in the rocks below Paragon City.
Summon a whole lot of demons, shake well, and bake for a few millennia.  Well, I guess that tells us the effect it has on the local geology.  Though technically, that means the caverns under Cap Au Diable ought to look the same.  Or has Bat'Zul just not been there long enough?

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17349 on: May 27, 2015, 11:19:14 PM »
Summon a whole lot of demons, shake well, and bake for a few millennia.  Well, I guess that tells us the effect it has on the local geology.  Though technically, that means the caverns under Cap Au Diable ought to look the same.  Or has Bat'Zul just not been there long enough?

Well, Bat'Zul was just imprisoned under Cap Au Diable, but Oranbega was the site of a magical city that fought a war and was magically sunk underground.  That plus the fact that Cap Au Diable is an island, and thus probably had completely different geology in the first place.

LadyVamp

  • Elite Boss
  • *****
  • Posts: 539
Re: New efforts!
« Reply #17350 on: May 27, 2015, 11:37:09 PM »
Every time I play with Icon my CoH woes only get worse. :\

And that more than anything is why I don't go play in Icon
No Surrender!

LadyVamp

  • Elite Boss
  • *****
  • Posts: 539
Re: New efforts!
« Reply #17351 on: May 27, 2015, 11:45:28 PM »
Yes, this is in fact what passes for a semi-casual mental exercise for me.  In my professional life, I've been thinking about what it would take to completely reimplement VMware Horizon from scratch.  Because that thing was written by baboons.  Who relies on ADRS?  It falls out of sync whenever you breathe hard around it.  There's no task manager at all.  The object database is always generating orphans.  Its impossible to make multi-tenant.  Pools keep losing their snapshot state.  Every time I have to reboot every single connection server because of a sync failure that causes them to kill off their server listeners I want to punch something.  Compared to that, this is all just for fun.

Arcana,  you sound like our vmware sme.  He's all the time wanting to choke the devs over at vmware for the usual fun things vCloud Director seems to have not to mention their idea of a web interface was java + flash.  Don't know who came with that bright idea but it's a real pita.  Then there's Windows patching on vCenter servers.  Don't try patching while vCenter is running.  To add insult to injury, we can't get vcd beyond 5.5 since we're not a service provider.  Never mind a good number of bugs were found and documented for vmware by him which 5.6 fixes.
No Surrender!

LadyVamp

  • Elite Boss
  • *****
  • Posts: 539
Re: New efforts!
« Reply #17352 on: May 27, 2015, 11:48:19 PM »
I often feel that way about backup software. I've yet to find any that doesn't suck in some way or another. Even the really good ones have some Achilles' heel that you don't  know about until you discover that it chokes on condition X, and of course you have one or two servers that cannot be set up any other way and trigger that condition.

So I occasionally threaten to write my own, and have built half of one several times for specialized purposes, but never made any general enough and polished enough to sell. Hmm...

I feel your pain.  One we use does a wonder job of backing up vms out of vmware.  Only problem it has, it can't restore them.  Three weeks with their technical support team and we decided it was a dead end.
No Surrender!

Felderburg

  • Ask me how I got this title!
  • Elite Boss
  • *****
  • Posts: 1,615
  • Personal text? What's that?
Re: New efforts!
« Reply #17353 on: May 28, 2015, 01:59:48 AM »
Or has Bat'Zul just not been there long enough?

Arguably this could be a solution, as Bat'Zul has only been there since the 17th century (per Virgil Tarikoss' Strike Force) versus the Oranbegans, which have been underground for millennia.
I used CIT before they even joined the Titan network! But then I left for a long ol' time, and came back. Now I edit the wiki.

I'm working on sorting the Lore AMAs so that questions are easily found and linked: http://paragonwiki.com/wiki/Lore_AMA/Sorted Tell me what you think!

Pinnacle: The only server that faceplants before a fight! Member of the Pinnacle RP Congress (People's Elf of the CCCP); formerly @The Holy Flame

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: New efforts!
« Reply #17354 on: May 28, 2015, 02:11:59 AM »
Since we virtualize our servers, we use Veeam whenever we think we're going to be running into OS problems.  Since Veeam doesn't run within the OS, the odds of a failure drop substantially (mostly, Veeam chokes when you run too many backup jobs simultaneously on a backup server without enough RAM).

Only problem with Veeam is that, last I checked, it can only do VM image backups and can't do anything about the few physical servers that are left, or the remote field office servers. And we really don't want to have to train staff on 2 or more different backup products...

Even image backups are not immune to problems. There is of course the occasional stuck snapshot that either needs consolidation or somehow didn't get detached from the proxy.

Mostly the problems I've had with them don't involve the backup itself, which usually works beautifully thanks to changed block tracking on the vmware side, but replicating them offsite. The last one I dealt with was a Data Domain system that worked great for backups, but utterly failed to keep up with nightly replication. Turns out that while it took advantage of CBT for the backups, for whatever reason it couldn't use it for replication and each run was scanning umpteen terabytes upon terabytes of unchanged data (in some cases nothing but compressed/deduped zeroes) to see if the replica was up to date.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17355 on: May 28, 2015, 02:13:44 AM »
I feel your pain.  One we use does a wonder job of backing up vms out of vmware.  Only problem it has, it can't restore them.  Three weeks with their technical support team and we decided it was a dead end.

Let me guess: Acronis.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17356 on: May 28, 2015, 02:26:03 AM »
Only problem with Veeam is that, last I checked, it can only do VM image backups and can't do anything about the few physical servers that are left, or the remote field office servers. And we really don't want to have to train staff on 2 or more different backup products...

Even image backups are not immune to problems. There is of course the occasional stuck snapshot that either needs consolidation or somehow didn't get detached from the proxy.

Mostly the problems I've had with them don't involve the backup itself, which usually works beautifully thanks to changed block tracking on the vmware side, but replicating them offsite. The last one I dealt with was a Data Domain system that worked great for backups, but utterly failed to keep up with nightly replication. Turns out that while it took advantage of CBT for the backups, for whatever reason it couldn't use it for replication and each run was scanning umpteen terabytes upon terabytes of unchanged data (in some cases nothing but compressed/deduped zeroes) to see if the replica was up to date.

Data Domain's dedup is kind of wonky when it comes to certain kinds of replication.

ShadowProtect ImageManager is stupid simple, and I mean that literally.  It sometimes gets stuck, but since it basically just FTPs the incrementals to the replica site its very easy to fix: look at log, find broken file, delete broken file, and then either restart the replication job or just FTP the files yourself to catch up.  At the replica site, a second copy of ImageManager can consolidate the incrementals into larger merged backups.  In fact, you could probably just script your own replication using rsync or scp and just have ImageManager consolidate the incrementals at the replica site.  That would be fairly bulletproof (when I was having issues with IM's early days replication I used to just scp the files, then move them to the target directories at the replica site and let ImageManager consolidate them).

Although I haven't used it myself (since I use ShadowProtect for bare metal backups) Veeam does have a physical backup product, and what's more its free: http://www.veeam.com/endpoint-backup-free.html.  I can't vouch for it, but I can in general vouch for Veeam.  We built some custom high spindle count FreeBSD ZFS backup repositories for Veeam and got 300MB/sec of backup and recovery out of them to our vSphere 5.1 clusters.

LadyVamp

  • Elite Boss
  • *****
  • Posts: 539
Re: New efforts!
« Reply #17357 on: May 28, 2015, 02:51:33 AM »
Let me guess: Acronis.

Avamar and hp serverprotect.  The avamar actually could backup a vapp.  serverprotect couldn't even talk to the esx host.  In our case, we wanted a way to transfer vApp templates to a remote site.  Best we've been able to do is spin up 2 sets of vapps, stop all apps, and use rsync to replicate the vmdk contents.  vCloud Director's shared catalogs time out on our super sized vApp templates.  Seems some engineer at vmware though using xz compression with the maximum setting was a good idea.  40 hours to export a 1 TB template.
No Surrender!

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: New efforts!
« Reply #17358 on: May 28, 2015, 03:20:05 AM »
Avamar and hp serverprotect.  The avamar actually could backup a vapp.  serverprotect couldn't even talk to the esx host.  In our case, we wanted a way to transfer vApp templates to a remote site.  Best we've been able to do is spin up 2 sets of vapps, stop all apps, and use rsync to replicate the vmdk contents.  vCloud Director's shared catalogs time out on our super sized vApp templates.  Seems some engineer at vmware though using xz compression with the maximum setting was a good idea.  40 hours to export a 1 TB template.

I have no idea about serverprotect, but Avamar is a decent, if ludicrously expensive product.  But if you are trying to replicate templates, backup and restore seems to me to be the worst way to do that.  You'll indirectly expand your thin provisioned disks.  I would export them to OVF, and then copy the OVF to the replica site and reimport.  You'll then be copying a compressed image of the vmdks.  You can even automate this with Powercli commands.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: New efforts!
« Reply #17359 on: May 28, 2015, 03:59:40 AM »
We built some custom high spindle count FreeBSD ZFS backup repositories for Veeam and got 300MB/sec of backup and recovery out of them to our vSphere 5.1 clusters.

Incidentally, most of my custom backup solutions are built on top of FreeBSD ZFS, many of them using rsync --inplace from the source system and snapshots on the ZFS side. Optionally using the filesystem-level dedup if the backup set warrants it, but there are a lot of gotchas with ZFS dedup, so only in very specific cases.

I have a set of python scripts I cooked up to do source/target replication. "Script" is a bit of an understatement since they do things like intelligently compare sets of snapshots to find a good source snap for an incremental stream, manage a customizable number of parallel full and incremental sends, do threaded i/o from multiple "zfs send" subprocesses, and break the data stream into blocks that get hashed for integrity checking as well as seamless recovery in the event of a network drop. The last part is important because we have a couple of sites with questionable connectivity that would never have been able to finish the initial weeklong "zfs send" of a 2TB filesystem if it had to start over because of a network blip.