New efforts!

[Ankhammon]

1.  City of Heroes wasn't a major asset to NCSoft.  It was a relatively minor asset (to them) they shut down and had no intention of ever starting up again.

2.  "Lost" implies misplaced, like its somewhere, but they just can't locate it.  Certain components of Issue 24 may be literally gone.  If it only existed on a developer's workstation at the time of the shutdown, those computers may be long ago wiped and gone.

3.  Even if all of the development assets exist and can be found, the development environment may not exist in any usable form anymore.  The environment is something that often isn't backed up in a usable fashion because its something that is often configured by replicating another working environment.  Precisely which compiler, which version of which modeling tools, which directories contain which information pointed to by which desktop shortcuts is something that its possible don't exist intact and would have to be reconstructed.

4.  Even if *everything* exists, the knowledge of how to use it often isn't documented on paper.  You could have all sorts of files and executable scripts, but no explicit documentation that states you need to run these specific things in this specific order with these specific switches to make a specific thing.  Absent a developer who remembers how to do that, this is knowledge that could take a significant amount of time to recreate, and its rare that sufficient documentation exists to recreate it from scratch in any development environment I've ever seen.

5.  Even if you find everything, its possible to find too much everything.  Which is to say, a working game requires all of the data and the codebase to be relatively in sync.  If you gather up everything you can find but its not explicitly contained in a coherent whole, you'll have to figure out how to assemble a proper snapshot of development environment and development codebase and development assets that go together.  Put the wrong set of assets together and you'd get a confused mess that won't work right.  In actual practice, while the game was in development different developers could have different versions and builds of the game and the data that they were working on.  If you are trying to recover a functioning I24 from that, you might have multiple different versions of I24 in different computers, without a lot of documentation on who was working on what at that instant in time.

Most or all of which could be handled with a Software Configuration Manager. I presume such a thing didn't exist as it's still not accepted on the whole in the IT industry.

[Arcana]

Most or all of which could be handled with a Software Configuration Manager.

Or a parrot that could recite the configuration instructions.  Anything is possible: I'm simply stating what is normal.  I'm unaware of any development environment anywhere that is so well documented and controlled that it could be trivially reconstructed after being completely disassembled.  And even if you use sophisticated configuration management software, the weak link is the SCM itself: if someone decides to toss that guy overboard because its no longer necessary, all of your sophisticated tracking goes out the window.  Often, its easier to rebuild dev environments where people scribbled notes to themselves in notepad and vi, because those often survive intact better after shutdowns.

Compartmentalization doesn't help.  All it takes is one guy to say they don't need to document X, because no one ever needs to know X except him, because he's the only guy that ever works on X.  Its the kind of thing that you don't even know is happening until that guy leaves.  Or in this case, everyone does.

[Codewalker]

SCMs are okay for software that gets updated maybe once a quarter and a standard workstation setup you refresh yearly.

For a development project where the toolset itself is under constant development and changes from day to day, it would be a severe hindrance. Every SCM I've worked with has been bloated, overengineered, complex to configure, and slow to implement changes in.

Don't forget that Cryptic didn't grab something off the shelf like Unity or Unreal. They wrote their own engine because, at the time, nothing off the shelf could scale to MMO numbers (even today it's debatable how well they can). They constantly changed and adapted it as needed. That comes with both advantages and disadvantages. The engine, tools, work processes, and development environment itself start to blur together at that point and aren't something you can just package up.

[Sinistar]

Here's a question for once the game returns:

Should character names remain unique?   There was always the issue of someone wanting a name that someone else had claimed and then their account goes inactive and yet they wouldn't release the name, etc etc.

Global names should remain unique, yes. 

But should character names remain unique?  The only hazard I can see with this is some "wiseguy" decides to basically make a copy of your character in terms of name/costume/powers then goes around acting like a total jerk trying to upset everyone for the purpose of getting you in trouble, but with global names being unique a simple global name check can indicate who the culprit is.

Thoughts?

[MM3squints]

Here's a question for once the game returns:

Should character names remain unique?   There was always the issue of someone wanting a name that someone else had claimed and then their account goes inactive and yet they wouldn't release the name, etc etc.

Global names should remain unique, yes. 

But should character names remain unique?  The only hazard I can see with this is some "wiseguy" decides to basically make a copy of your character in terms of name/costume/powers then goes around acting like a total jerk trying to upset everyone for the purpose of getting you in trouble, but with global names being unique a simple global name check can indicate who the culprit is.

Thoughts?

https://mutterschwester.files.wordpress.com/2012/07/128977421765.jpg

[Sinistar]

https://mutterschwester.files.wordpress.com/2012/07/128977421765.jpg

Nice art. I see you are in fine voice today ;)

[adarict]

Or a parrot that could recite the configuration instructions.  Anything is possible: I'm simply stating what is normal.  I'm unaware of any development environment anywhere that is so well documented and controlled that it could be trivially reconstructed after being completely disassembled.  And even if you use sophisticated configuration management software, the weak link is the SCM itself: if someone decides to toss that guy overboard because its no longer necessary, all of your sophisticated tracking goes out the window.  Often, its easier to rebuild dev environments where people scribbled notes to themselves in notepad and vi, because those often survive intact better after shutdowns.

Compartmentalization doesn't help.  All it takes is one guy to say they don't need to document X, because no one ever needs to know X except him, because he's the only guy that ever works on X.  Its the kind of thing that you don't even know is happening until that guy leaves.  Or in this case, everyone does.

When I see people thinking that it is no big deal to rebuild an environment, as long as the person rebuilding it is smart enough, I have to stop myself from laughing.  Where I work, we have a product that brings in many millions of dollars per year.  This product is vital to the company.  We have a whole team of people in multiple countries, who are responsible for building and deploying the various test and staging environments.  We have been running this product since 2002.  We still can't spin up a new version of the environment on the first try.  Even using VM images, and automated deployment scripts etc.  There is ALWAYS something that isn't documented in the right place.

The theory behind building these environments is very simple, but the larger the environment, and the older the code, the more chances that something will get missed.  And that is talking about an actual live product.  Now imagine the same level of complexity, but now you are working on third and fourth-hand information.  I don't know if other companies are the same as mine, but the biggest offenders I run into, are SQL DBA's.  They don't seem to like to document ANYTHING outside of their group.  :)

[Ankhammon]

SCMs are okay for software that gets updated maybe once a quarter and a standard workstation setup you refresh yearly.

For a development project where the toolset itself is under constant development and changes from day to day, it would be a severe hindrance. Every SCM I've worked with has been bloated, overengineered, complex to configure, and slow to implement changes in.

Don't forget that Cryptic didn't grab something off the shelf like Unity or Unreal. They wrote their own engine because, at the time, nothing off the shelf could scale to MMO numbers (even today it's debatable how well they can). They constantly changed and adapted it as needed. That comes with both advantages and disadvantages. The engine, tools, work processes, and development environment itself start to blur together at that point and aren't something you can just package up.

How about builds going to production every Wednesday (some weeks more often)? I was a configman man for 3-4 yrs at a large company. We worked an internally developed software suite that had an uptime target of 99.999% (yes we hit it every year).
I'm not talking about a single software package, but of the entire discipline. Somewhat of a cross between a software librarian, a build manager and a sys admin. Good teams will even have things like OS patches for their environments covered.

That being said your assertions are mostly correct. The problem (as I see it) is that there are not good standards going across the board for this discipline. He**, half the IT world doesn't have any real SCM going at all.
To do it justice, all developers have to utilize the standards that are set in place (they can change for the groups and what is being accomplished). For the devs it's all simple stuff (branching, nomenclature, usw), but if they don't keep up on it the whole shebang can fall apart. Unfortunately, that's where devs get the bad opinion of SCM. Since that's key, there becomes a need to ensure it's being done... training happens... meetings ensue... interference in all my stuff... I don't have time for this... why won't you just let me do my work!!! :)

True story: One of our builds went to production and messed up because a developer put in an incorrect branch, it only took an afternoon to fix (an hour or so to identify and a few more to redo the build) instead of a half a week or more. That was huge considering the visibility the group was under (VPs wanting 15 minute updates and that's not an exaggeration).

Also, there's the silly ITIL CM running around confusing people by pretending they need to exist and then half the rest of the world thinks SCM is CM.

As far as scaling for MMOs, I have my doubts that it would be that difficult to be absorbed with the correct standards in place. Yes, if it's not there in the beginning you will have a big job putting everything into place, but after that things settle down pretty well.

And Arcana, you are right, that team would have been let go too. But if standards were set (most assuredly didn't happen) then resolving this could be done in a fraction of the time it takes from what you are stating.

[JoshexProxy]

I have this intuition kinda feeling that the negotiations are looping much the same as this thread. but in the negotiations case one of the loop options is "hoarder syndrome", NCSoft could sell, but the game looks nice on their collector's shelf.

"but... if we sell it what will go next to Exteel?"

[Brigadine]

I have this intuition kinda feeling that the negotiations are looping much the same as this thread. but in the negotiations case one of the loop options is "hoarder syndrome", NCSoft could sell, but the game looks nice on their collector's shelf.

"but... if we sell it what will go next to Exteel?"

I was actually surprised they dumped exteel. It was a fun game with a solid population. I have a hard time thinking they weren't making money.

[JoshexProxy]

I was actually surprised they dumped exteel. It was a fun game with a solid population. I have a hard time thinking they weren't making money.

I was playing it before the shutdown, everyone in the game knew why it was being shut-down because a dev had leaked it, not enough players were buying stuff from the shop and it lacked a community based atmosphere (because it had no PVE only content or maps) which turned it into a snub-fest and caused too many complaints.

summary: not enough player purchases + snub-fest + complaints = shutdown.

but it was fun while it lasted.

thing is though offers were made for exteel as well. NCSoft's reply was basically a "we've collected it, we're not letting it leave" comment.

[LadyVamp]

Or a parrot that could recite the configuration instructions.  Anything is possible: I'm simply stating what is normal.  I'm unaware of any development environment anywhere that is so well documented and controlled that it could be trivially reconstructed after being completely disassembled.  And even if you use sophisticated configuration management software, the weak link is the SCM itself: if someone decides to toss that guy overboard because its no longer necessary, all of your sophisticated tracking goes out the window.  Often, its easier to rebuild dev environments where people scribbled notes to themselves in notepad and vi, because those often survive intact better after shutdowns.

Compartmentalization doesn't help.  All it takes is one guy to say they don't need to document X, because no one ever needs to know X except him, because he's the only guy that ever works on X.  Its the kind of thing that you don't even know is happening until that guy leaves.  Or in this case, everyone does.

And none of that (or the 5 points Arcana made earlier) touches on several more points including:

• Enough of the original team returning who have the knowledge to put it back together.  Just as there are those of us willing to give NCSoft the middle finger, it's logical to conclude the employees of Paragon Studios have their members who would equally do the same.  Any new people would have to go learn the system.
• The evolution of backup software.  Assuming backups of the servers were taken.  Tar might have been around forever, but I'm sure ServerProject, Backup Exec, NetBackup, etc. have improved considerbly in the last several years.  They may not be able to read the tapes made by their earlier versions.  Earlier versions might not run on current OSes.  See next point.
• The evolution of the Operating Systems the servers ran on.  Libraries get bug fixes and updates that make them incompatiable with the codebase of the game.  And recompiling or relinking may not help.  Whole sections of code might have to be rewritten.
• The evolution of the backup hardware itself.  If these are DLT tapes for example, you might find it a little hard to get compatible DLT drives today.  You can forget about 4mm DAT.  The shelf life of the media alone is too short.

All of those add cost to returning the game into service.  My 1st point would easily be the most expensive by far.  Salaries would have to be negociated for those willing to return to Paragon Studios.

[Paragon Avenger]

Forget all about servers and backup, just put the game in a cloud, MMORPG as a service, Brilliant!

[Codewalker]

How about builds going to production every Wednesday (some weeks more often)? I was a configman man for 3-4 yrs at a large company. We worked an internally developed software suite that had an uptime target of 99.999% (yes we hit it every year).
I'm not talking about a single software package, but of the entire discipline. Somewhat of a cross between a software librarian, a build manager and a sys admin. Good teams will even have things like OS patches for their environments covered.

How big of a team did you have dedicated to doing configuration management?

I did it solo for a relatively large organization, but it was mostly prepackaged (though sometimes customized) software. We only had a very small amount of in-house development, and other than dealing with Java (dieoracledie), by far the biggest pain point was when we told the developers that the people running their software weren't going to have admin rights anymore. So they couldn't just slap updates on that were emailed to them.

For a relatively small development house, dedicating more than one full-time person (if even that) to what is essentially overhead would probably be out of the question.

[Waffles]

https://mutterschwester.files.wordpress.com/2012/07/128977421765.jpg

https://i.imgur.com/cJsSE.gif

Forget all about servers and backup, just put the game in a cloud, MMORPG as a service, Brilliant!

I'm no tech expert by any stretch, but i'm pretty sure that it's not that simple.

[Stitchified]

Here's a question for once the game returns:

Should character names remain unique?   There was always the issue of someone wanting a name that someone else had claimed and then their account goes inactive and yet they wouldn't release the name, etc etc.

Global names should remain unique, yes. 

But should character names remain unique?  The only hazard I can see with this is some "wiseguy" decides to basically make a copy of your character in terms of name/costume/powers then goes around acting like a total jerk trying to upset everyone for the purpose of getting you in trouble, but with global names being unique a simple global name check can indicate who the culprit is.

Thoughts?

Well, usually a character's belongings are tied to that character, so non-unique character names would require character names being akin to Sam for one person, then Sam1 for the next, Sam2 for the next, etc etc so that their belongings wouldn't spill over onto other characters with the same name...
Besides, I don't know anyone who wants to have a number as a part of their character's name.
Of course, I also don't want to chance upon someone else who has the same character name as me, that'd be pretty weird for me.

In the end, I see your point and it makes sense, I personally don't agree with it though :)

[Sinistar]

In the end, I see your point and it makes sense, I personally don't agree with it though :)

hm, true the belongings and store purchases could be a snag. 

Ah well, just a thought to make some conversation.

[JoshexProxy]

And none of that (or the 5 points Arcana made earlier) touches on several more points including:

• Enough of the original team returning who have the knowledge to put it back together.  Just as there are those of us willing to give NCSoft the middle finger, it's logical to conclude the employees of Paragon Studios have their members who would equally do the same.  Any new people would have to go learn the system.
• The evolution of backup software.  Assuming backups of the servers were taken.  Tar might have been around forever, but I'm sure ServerProject, Backup Exec, NetBackup, etc. have improved considerbly in the last several years.  They may not be able to read the tapes made by their earlier versions.  Earlier versions might not run on current OSes.  See next point.
• The evolution of the Operating Systems the servers ran on.  Libraries get bug fixes and updates that make them incompatiable with the codebase of the game.  And recompiling or relinking may not help.  Whole sections of code might have to be rewritten.
• The evolution of the backup hardware itself.  If these are DLT tapes for example, you might find it a little hard to get compatible DLT drives today.  You can forget about 4mm DAT.  The shelf life of the media alone is too short.

All of those add cost to returning the game into service.  My 1st point would easily be the most expensive by far.  Salaries would have to be negociated for those willing to return to Paragon Studios.

you forgot the data loss factor, if it is on a magnetic tape there's magnetic bleeding to worry about. by now even two years later I'm sure there's /some/ corruption.

too bad we didn't get onto the holographic storage quartz, 20TB of information can be stored in a 1inch cube and there's no chance of data loss or bleeding, only problem is it can't be rewritten or edited without loading it, changing it and spitting it onto another cube and the original cube can only be disposed of properly by melting it down, even smashing it will leave the burned regions intact.

writing to it is simple, it makes use of two different colored lasers for a read/write head, the red laser is low power and fires 90 degrees to the blue laser, the blue laser can then fire high (high enough to burn the quartz) or low power, low power for a read operation (coupled with a silicon based light sensor to determine the angles of the reflected light and thereby determine the shape '0' or '1') and high power to literally burn a microscopic pin-point 0 or 1 into the 3D matrix of the cube that is completely invisible unless a red laser shines on it.

Thank Japan (a student at University of Tokyo) for that one. Too bad we aren't using it yet.

either way I think they might have had to repair a corrupted image first then go through making it compatible with newer servers, OR another consideration;

it might be the case that new servers wont run the server image AND too much of the image would have to be rewritten because of it to be worth anyone's time (or they went that way and that's whats taking so long). In that case they might be trying to a pull drivers for new server components out of an OS vendor who doesn't want to bother or who feels that by doing that they'd be making competition for their new OS, which means the THeM might be trying to write a batch of drivers to run the new hardware on an old server OS.

many options there, I think if they were to attempt to update the game image for new server components and a large portion of the image needs such fixes that NCSoft's executives would not have that kind of patience.

but meh that's only speculation.

[Ankhammon]

How big of a team did you have dedicated to doing configuration management?

I did it solo for a relatively large organization, but it was mostly prepackaged (though sometimes customized) software. We only had a very small amount of in-house development, and other than dealing with Java (dieoracledie), by far the biggest pain point was when we told the developers that the people running their software weren't going to have admin rights anymore. So they couldn't just slap updates on that were emailed to them.

For a relatively small development house, dedicating more than one full-time person (if even that) to what is essentially overhead would probably be out of the question.

My team was only 5  in configman. The dev team was about 120 seats at the largest (downsizing over the years and all). Most of the development was in-house java but we did have some third party stuff.

And I am not arguing that it should have been in place. I know CoH was too small to really warrant that. My involvement in this was mostly a simple comment that got too much visibility.

And java ain't so bad. Then again,I liked programing in COBOL. :-)

[hurple]

Well, usually a character's belongings are tied to that character, so non-unique character names would require character names being akin to Sam for one person, then Sam1 for the next, Sam2 for the next, etc etc so that their belongings wouldn't spill over onto other characters with the same name...
Besides, I don't know anyone who wants to have a number as a part of their character's name.
Of course, I also don't want to chance upon someone else who has the same character name as me, that'd be pretty weird for me.

In the end, I see your point and it makes sense, I personally don't agree with it though :)

No... <username>_<charactername>

There's your unique identifier

and maybe tag <server> on there too if necessary...