Author Topic: Characters deleted after update to 1.0.0  (Read 4869 times)

wolfbeast

  • Underling
  • *
  • Posts: 10
Characters deleted after update to 1.0.0
« on: April 01, 2016, 06:00:29 PM »
So, I just went in to have a look at the final release version 1.0.0, let it install, etc. initially everything was OK, although the bad colors really threw me off (which I found out later was a bad april fool's joke).
What's worse though, I tried reinstalling by launching the old chat again in case something went wrong leading to these bad colors, and as a result my characters were all erased that i had before. Apparently the db got nuked.
Since your initial installer apparently thought it was a good idea to MOVE and not COPY by default, there was no backup copy left in the original game folder either.  :(

I guess there's no way to get my characters back now, is there?

Folder

  • Underling
  • *
  • Posts: 1
Re: Characters deleted after update to 1.0.0
« Reply #1 on: April 01, 2016, 06:27:55 PM »
I myself was excited to check out the 1.0.0 release and was talked into updating today by a friend.  My initial reaction to the 'joke' was not that this was a design change, but instead an error.  My friend assured me that everything was fine yesterday and there were no color errors (as I figured the game was running in 16 bit color instead of 32 bit) so I began to debug MY OWN SYSTEM to discover the issue and fix it.

While I can applaud the creative use and cynical critique on CO, this was just after your official release.  Very poor timing.  This could have waited until next year when people would be more savvy to 'oh, this is clearly a joke'.  Had it not been discovered that this was an intentional look for the day I might have taken similar steps as wolfbeast and I would be quite nonplussed at a similar outcome.

Personally, I think the internet needs to stop trying to be funny on one day of the year.  Not everyone across the globe recognizes or celebrates it.  Even Google messed up going too far today and cost many people their jobs.  It's not a prank unless there's an air of obviousness to it, otherwise there is nothing but grief, like here, where radical directory changes in the installation could lead some to believe there has, indeed, been a technical malfunction.  I hope that wolfbeast is able to be helped.  And I hope there's something to be learned from this.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #2 on: April 01, 2016, 10:05:08 PM »
I guess there's no way to get my characters back now, is there?

Most likely they are not deleted unless you tried to manually move the database.

Paragon Chat 1.0 includes some new defaults designed to make it less likely for people to get confused about their local account name versus their XMPP login name -- they don't see local accounts at all unless they go out of their way to create extras.

It tries to take into account existing setups as best it can (it doesn't autologon unless characters exist on the account or the db is empty). For most people that works, but some people were already confused enough about it to have created multiple accounts they then forgot, so it can't pick a reasonable default.

As a stopgap, I just pushed an update with a very, very hastily thrown together database account management tool to help people find their characters. It's not the database management tool that Gotham needs, but it'll have to do in a pinch, or something like that.

To use it, open up Paragon Chat, wait for it to load, hold the Control key and click on the gear button (I didn't have time to rearrange the UI). It should open a second window that shows the accounts and characters in your database.

If you have multiple accounts listed, you should be able to find the one with your characters on it. You can get to those accounts by using the Back button from the character select screen once in game, and typing in the appropriate name.

In the event that you chose not to copy or move the Paragon Chat files during setup, there is also an option to open up the old (0.99 version) database, and to import all of the characters from it. This makes a copy and does not overwrite any characters that already exist, so it should be safe.
« Last Edit: April 02, 2016, 03:25:20 AM by Codewalker »

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #3 on: April 02, 2016, 02:21:12 AM »
I also just fixed a bug related to the detection of single-account databases where the account name doesn't match the XMPP name. It was supposed to automatically fill in the one account name that does exist for you, but it wasn't working correctly.

If you're missing characters, make sure that the "remember account name" box on the login screen is UNCHECKED, then exit and try again.

wolfbeast

  • Underling
  • *
  • Posts: 10
Re: Characters deleted after update to 1.0.0
« Reply #4 on: April 02, 2016, 07:31:05 AM »
Thanks for the reply, Code, but I already checked the account name issue (it was different but I noticed because I had previously made a character on my XMPP account name which showed up). Initially, I had my characters in the new installation (the temp char on my XMPP name and my normal characters on the different local account name).
Unfortunately, the second install wiped everything - both the temp char on my XMPP account name and my normal characters on my other local account. The folders were still there under /Account but empty, and the .db file apparently got re-initialized with a blank one. I poured over all files to see if I could recover anything, even tried an undelete tool but had no luck recovering anything.

A really unfortunate set of circumstances, I guess.
  • Database was moved, not copied, from the CoH folder to the new location, removing what would have been a backup copy in case of trouble.
  • The "April Fool's" joke had me think something was technically wrong, prompting me to try and re-install.
  • The re-installation for some reason decided to either remove or overwrite/re-initialize my ParagonChat database.

I still have my .costume files from previous migration, so I can re-build the characters, but clearly what I had as far as characters in ParagonChat's own format has all been lost.

Side note: I have to say the new location is a really, really bad idea - you should never, ever, have binaries/executables in your Windows profile/data folder. Not only does it trip up malware detectors because it's a common malware tactic, it also makes the program completely unprotected from e.g. virus infection even in a limited user account, as well as going against all common Windows programming practice.
I'm a software developer myself (Pale Moon web browser), so if there's any specific concerns that made you decide to do this, maybe I can help?

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #5 on: April 02, 2016, 07:22:21 PM »
Unfortunately, the second install wiped everything

That concerns me as it's not supposed to be possible. I'd be interested in getting more details about exactly what steps you took so that I can try to reproduce it.

The database handling and 'install' process is designed to be very conservative. In the event of something unexpected happening, everything comes to a halt rather than risk destroying data. No files are ever overwritten during the install. Even a corrupt database will never be reinitialized (it will just error out). It must not exist at all for the schema creation to trigger. So if there's a bug somewhere capable of causing that, I want to find it.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #6 on: April 03, 2016, 03:53:46 AM »
Replying separately since this isn't directly related to the issue and it got rather long.

Side note: I have to say the new location is a really, really bad idea

FWIW, it's not a new location. Paragon Chat has been running out of AppData almost the entire time it's been live, since 0.98b released on Aug 1, 2015. It started doing that shortly after it got a self-updater. If launched from elsewhere it would copy itself to AppData (if not already present) and relaunch from there. That was not only to avoid UAC prompts if the COH directory happened to be in Program Files, but also because some of the third party launchers like Tequila kept overwriting it with an older version.

The only thing that changed in 1.0 was the rename (to differentiate it from the betas and avoid accidental overlap) and reorganization of the folder structure. That and the bootstrap was split into a separate program for a smaller initial download. The beta releases were bogged down with design compromises made to get the thing working, so the transition to 1.0 seemed like the best time to fix as many of them as possible.

Quote
you should never, ever, have binaries/executables in your Windows profile/data folder.

Tell that to Google, who installs Chrome and other Google apps per-user if you don't have admin rights. Or Microsoft for that matter, since OneDrive installs itself into your local AppData so that it can silently update.

But really, it seemed like the most reasonable choice of the available options. Here's all the ones I can think of.

Program Files: Paragon Chat is under active development and sees very frequent updates. Since it's more or less peer to peer, new features depend on everyone having the latest version. Prompting the user for elevation every time there is a minor patch is a big usability issue. That would also cause some users to simply always run it elevated, which is a security risk and goes against the principle of least privilege. It's especially silly to require elevation for patches since absolutely nothing else that Paragon Chat does requires admin rights.

Programs that demand to be run as an administrator but don't do anything that really needs it are a huge pet peeve of mine. I run as a limited user for day to day work and do not allow anything to elevate, ever. Instead I swap to a separate account if I really want to install something system-wide.

The workaround is just as terrible -- things like Firefox installing system services that are always running, just so they can update themselves if you're not logged on as an administrator. Imagine if every program did that!

Some system-wide location outside of Program Files: Messy because it's nonstandard. Not guaranteed to work if the filesystem permissions have been changed to not let random non-admin users create folders in the root (which is something I do on every Windows machine I install). Plus it's rude to create a structure that doesn't already exist.

ProgramData: There are a few programs that get installed here and it's not subject to UAC. However, it isn't backwards compatible to Windows XP, and the default file permissions are rather weak, so it could be a security risk on multiuser systems.

User profile: Per-user installs make a lot of sense for paragon chat due to the frequent updates, and also because some people are running a development branch for testing. On a multiuser system, it's quite possible that different users would be running different versions (one launches and updates while another has it open). It also fits well into the self contained model. My preference would actually be a 'Programs' directory under the user profile, which is where I install most of my per-user applications out of habit. However, that doesn't exist by default, and the de-facto standard set by Google and others doing their own per-user installs is AppData.

The COH install directory: This is the worst of all worlds. The COH directory could be literally anywhere. If it was installed from the ncsoft installer, it would be under Program Files and subject to UAC requirements. Not to mention that COH itself breaks the cardinal rule and tries to write data to its program directory, so it either fails or invokes UAC virtualization which just confuses non-wizards when they go looking for their screenshots and costumes. Or it could be in AppData already if the user downloaded it with one of the third-party launchers. Or it could be in J:\Stuff\Torrents. Or on the user's desktop... Plus, putting stuff here could cause problems if it ever gets used for something else someday. Not very ideal.

Quote
Not only does it trip up malware detectors because it's a common malware tactic

I think the stuff that the client does is much more likely to trip them up regardless of where it's installed. PE loading code ("Oh, this looks like an unpacker! Must be bad!"). Dynamic code generation ("Oh, I can't analyze this! Must be bad!"). Lots of memory protect/unprotect calls as part of the codegen and patching. The patching process probably looks like self-modifying code as well ("Oh, this is trying to evade analysis! Must be bad!").

Quote
it also makes the program completely unprotected from e.g. virus infection even in a limited user account

I'm sorry, I just can't agree with that at all. My first question would be, "To what end?"

If malware is running under your user account, it can do far worse things. It can steal all your data and encrypt your files. It can turn your machine into a spam-sending zombie as long as you're logged on. It can modify your DLL path or exploit same-user shatter attacks (window hooks, etc) in order to run its code in any process you launch, regardless of whether or not it has write access to the exe file. It can do just about anything except for taking over the machine completely, and affecting other users if it's a multi-user machine.

The days of viruses infecting exe files are long gone. 99% of malware out there is either better classified as a worm or a trojan. With nearly every machine internet connected, there are far better attack vectors than exes on removable media, and old school viruses have mostly died out.

Quote
as well as going against all common Windows programming practice.

Microsoft's standards for traditional desktop applications are to install in Program Files, and have been since Windows 95, which was a single-user OS with no security whatsoever. I would argue that view is obsolete.

  • UNIX users have been doing per-user installs of software into their home directories for 40 years. It's ideal for something small that doesn't need to be installed system-wide.

  • Microsoft tried to add support for per-user installs with MSI 2.0 in Windows 2000. Due to design flaws, they could never get it to work quite right and eventually started recommending that it not be used, but did not remove the feature.

  • There is a large demand for per-user installs that do not require admin rights, as evidenced by the explosion of portable apps. Almost every major application that it's feasible to have a portable version made of has one. Some of them are supported by the original author, some are packaged by third parties.

  • There's also a big market in the corporate world for application virtualization, in order to automatically make applications that assume they are installed system-wide (installed in Program Files) per-user instead. See also: VMware ThinApp, Microsoft App-V, etc.

  • It's clear that Windows is headed this way also. The *current* guidelines from MS say don't install in Program Files at all -- they say to use an AppX package, submit it to the Windows Store, and let the OS decide where to put it. All Windows 10 Universal applications install per-user, not per-machine.

    They currently DO end up under Program Files\WindowsApps, but that's another terrible design decision that will come back to bite them IMO. Just think about it for a minute -- any user on a limited account can install an app from the Windows Store WITHOUT being prompted for elevation or entering an admin password. The installation itself silently runs with TrustedInstaller rights and ends up in Program Files... setting off any alarm bells yet? There will be a lot of fallout the first time somebody manages to slip a trojan through the store submission process.

    My feeling is that MS will eventually start installing universal apps into the user profile or some other per-user structure instead, which they can do at any time since the OS is in full control of where these packages end up.

Quote
I'm a software developer myself (Pale Moon web browser), so if there's any specific concerns that made you decide to do this, maybe I can help?

I think I just about covered it. :)  I'm interested in hearing your opinions on the subject as a fellow developer.

wolfbeast

  • Underling
  • *
  • Posts: 10
Re: Characters deleted after update to 1.0.0
« Reply #7 on: April 03, 2016, 08:18:15 AM »
That concerns me as it's not supposed to be possible. I'd be interested in getting more details about exactly what steps you took so that I can try to reproduce it.

I launched the original ParagonChat client in the COH folder (like I always did) which triggered a re-install. That's basically all I did. After that, stuff was gone. I don't know exactly what happened that shouldn't have happened, but that is what I did.

OFF-TOPIC:
As for your long post about my side note: I understand your approach but this is not *nix. It sounds to me like you're trying to avoid UAC at all costs - it's there for a good reason on the systems where it hasn't been disabled. Why are you so frantically trying to avoid the UAC prompt?

As for options to avoid doing a system-wide install (which seems is what you want to do?), the "some location outside of program files" is what a lot of people use, but they tend to call it "a portable installation".
You may want to consider doing that, instead, considering the program is already rather self-contained.
Problems I personally have with just dropping it in %APPDATA% is that the program is not discoverable beyond the shortcut, cannot be placed in a different location at install time (I keep my applications on a different partition, for example), and especially that it prevents the clear distinction between programs and data you'd expect for installed (not portable) programs, which is what I meant with established programming practice. You either make it self-contained, meaning you keep everything (program and data) in a location of the user's choice, or you make it an installed application, meaning you keep program and data separate and installed in the expected way. You're mixing those two up here, by using installed techniques and portable techniques at the same time.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #8 on: April 06, 2016, 09:37:20 PM »
I launched the original ParagonChat client in the COH folder (like I always did) which triggered a re-install. That's basically all I did. After that, stuff was gone. I don't know exactly what happened that shouldn't have happened, but that is what I did.

I've run through a few scenarios but so far haven't managed to reproduce this. What I tried was:

1. Create a few characters in 0.99j

At this point I have %APPDATA%\Paragon Chat
with Bin\ParagonChat.exe (containing a copy of 0.99j), Database, and Config.

2. Accept the automatic update (the default setting prompts for major updates like 1.0). The bootstrap downloads the current version and the New User Experience window opens to the first page, asking me to confirm that COH is installed in C:\I24xyz.

Before proceeding, I check the state of things. I have

%APPDATA%\Paragon Chat, in which Bin\ParagonChat.exe has been replaced by the 1.0 bootstrap by the auto-update, and everything else is untouched.

I also have %APPDATA%\ParagonChat. At this point it only has Bin\ParagonChat.exe in it, which is the actual 1.0.0 Patch 4 executable.

3. I click through the wizard, just accepting the defaults. PC 1.0 starts to load.

Now the "Paragon Chat" folder is gone.

Inside %APPDATA%\ParagonChat I have:
Bin: ParagonChat.exe and ParagonChatClient.exe
Client: Data, Logs, Logs\game, Settings, Settings\test.txt (imported from C:\I24xyz)
Config
Costumes
Database: Contains my database file, moved from the old location
Piggs
PowerCust

4. I still have the 0.99j version inside of C:\I24xyz, it wasn't touched. So I run it, simulating a downgrade attempt. It prompts for upgrade to 1.0, which I say 'no' to.

Now I have:

%APPDATA%\Paragon Chat
Contains a fresh empty database and a copy of 0.99j. If I had chosen the copy option during the first-run wizard I would have a second copy of it here.

%APPDATA%\ParagonChat
Contains the database with my characters in it, and a copy of 1.0

5. For grins, I replaced the ParagonChat.exe in C:\I24xyz with 0.98d. That's the version that was in the Tequila manifest and could have conceivably been lying around there (it relaunches the newer version in appdata if one exists in order to deal with that problem).

It offered to update directly to 1.0, which I accepted this time. Upon accepting, it replaced itself with the bootstrap and immediately launched my existing 1.0 install. It did not begin the new user setup. My database file was still intact.

Every test situation I've come up with has had similar results and matches up with the expected sequence of events. I also reviewed the update logic again, and I can't think of a conceivable scenario in which the database could be overwritten short of someone manually deleting the file.  :-\

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Characters deleted after update to 1.0.0
« Reply #9 on: April 06, 2016, 10:04:36 PM »
It sounds to me like you're trying to avoid UAC at all costs - it's there for a good reason on the systems where it hasn't been disabled. Why are you so frantically trying to avoid the UAC prompt?

I'm not sure I'm communicating my intent. I'm not trying to avoid UAC for its own sake. I'm trying to let UAC do its job -- protect the operating system and the installed software. UAC can only do that if trivial tasks do not unnecessarily elevate to admin status, increasing the attack surface.

A user could conceivably say "I only trust signed executables from specific vendors to elevate to admin". I support that, and Paragon Chat -- an unsigned binary that admittedly does things that drives antivirus software crazy -- is designed so that you don't have to trust it with the keys to the kingdom.

Problems I personally have with just dropping it in %APPDATA% is that the program is not discoverable beyond the shortcut, cannot be placed in a different location at install time (I keep my applications on a different partition, for example), and especially that it prevents the clear distinction between programs and data you'd expect for installed (not portable) programs, which is what I meant with established programming practice. You either make it self-contained, meaning you keep everything (program and data) in a location of the user's choice, or you make it an installed application, meaning you keep program and data separate and installed in the expected way. You're mixing those two up here, by using installed techniques and portable techniques at the same time.[/color]

Quoting all of this instead of going point by point, because I think I can clarify a few things.

Paragon Chat is 99% portable. It does not offer this option by default in order to simply the install / upgrade process, but it is an option for advanced users. Granted it's not fully documented yet due to wanting to spend some of my free time with my family, but I'm hoping the community will be able to put some of this on the wiki or something...

The only part that isn't self-contained are the registry keys that contain the configuration. There is no real 'install' to speak of. There is the new user wizard that helps you locate the COH directory, but that doesn't install anything. It just sets the appropriate registry key and offers to copy some files into the paragon chat data directory. It can be avoided if the COH directory is already known, either from the registry key or -cohdir command line option.

The main executable doesn't have to live in any specific location. It defaults to the appdata folder to make things simple for the vast majority of users who don't care where it is, but you can move it. You can even put it in Program Files if you are that allergic to having executables in your profile, but of course you'll have to manually run it as Administrator for updates to succeed.

The data directory (which usually includes the program) is easily discoverable -- there's a button in the UI that opens Explorer right to it.

The data directory itself can also be moved if you prefer it somewhere else. You just need to launch the program with -pcdir \path\to\paragonchat. If you point it somewhere that doesn't exist, a new structure and blank database will be created. If you're moving your existing copy, backing up the database is always a good idea before manually touching anything. Currently you have to modify the shortcut to always include that parameter. Once the options panel UI is finished, there will likely be a friendlier interface to do it, but it may be hidden under advanced options.

Even the bootstrap (which says it's 'installing', but that's just UI fluff) accepts -pcdir and will download elsewhere if you want. It's not advertised, but the bootstrap itself is not even necessary. You could just as easily download the main exe, shove it somewhere, and run it ( https://chat.cohtitan.com/update/release/v1/1000004/paragonchat.exe )

About the only thing you can't change is that the client exe (ParagonChatClient.exe) must be in PCDIR\Bin, wherever -pcdir is. That's because that particular exe is not downloaded or installed, it is dynamically created at runtime.

Which brings me to this:

Quote
[...] especially that it prevents the clear distinction between programs and data [...]

In a perfect world it would be black and white. With a program like Paragon Chat, the distinction isn't so clear to begin with. The main program itself is always changing, sometimes rapidly. That's not terribly uncommon, games like MMOs that require constant client updates have to deal with the same problem. But Paragon Chat also does things like dynamically generate code to patch an existing exe, because due to copyright restrictions we can't distribute the original exe that the patch is being applied to. It's a program, but it also fits many of the defining characteristics of dynamic data.