Gory Details that are Important

The site software is elaborate and richly featured. Boiled down (as you know) is a collection of dynamic content generation married to a database.

But, still at the heart of the matter is this software suite we use for the site and it has properties that affect the way in which we have to navigate the upgrade to the latest version of the software.

This page is about those issues and a warning about the effort put into making UX changes to the site.

The Database is (nearly) Everything

The IP software relies on a database. In the database are many records that define content and UX controls.

1. The CSS Template -- Actual templates of which CSS is used and which HTML is used to define content in a page dynamically are not stored in static files, but rather stored in the database.
2. The built-in Editor -- The actual behavior of the built-in Editor is not defined with static files, but rather records for customizations (when customized).
3. Associations to the Theme -- Although the core of the Theme is defined by a collection of media files, the accounting of those media files are actually stuffed into records in the database -- and selected out -- when the dynamic elements of the site delivers content to the user. So, the Theme begins life as a static set of files, in fact the execution of the Theme is partially based on static media files and partially based on database records stored upon parsing the new Theme.
4. All User Content -- True, media files are stored as static files, but their relationship to posts/messages are tied together by the database.
5. Users (Roles, Ranks, metadata, etc..) -- All in the database.

Steps -- Impact of the Database

0. Shutdown the legacy site. It has to be deactivated (made impossible to reach).
1. The upgrade occurs by the establishment of an empty filesystem. Zero files.
2. The baseline instructions say to copy the legacy file-tree onto the new host. This is simple but for our site we have collected a huge amount of non-site specific run-time data so the tree of files is manually trimmed to only those file trees that are applicable. The size of this dump is roughly 120-140 GB.
3. Next is to place the installation of the new server software into this empty directory. This overwrites, as necessary, as dictated by the requirements of the new software files that are/were in the legacy file-tree.
4. Next is to copy the legacy database from the legacy host to the new host.
5. Next is to establish on the new host a copy of the legacy database.

At this point the new software installation has not been invoked, yet.

6. The baseline instructions say at this point to invoke the install feature present in the upgraded file-tree (which contains the upgraded software). This install step is a time-consuming process that lasts DAYS. Last time I ran it, it took 5 days. Not 5 days as in I start each morning to resume the process. I mean literally 5 x 24 hours of constant processing done by the system to ingest the legacy database and CONVERT it to the new schema of the new software.

Already, the bells should be going off.

Anything in the beta database will not exist before step 0. Anything stored, configured, kept, arranged in the beta site is NOT available in the upgrade (step 0). There is no path to re-use ANYTHING from the beta database work in the actual upgrade.

Post Install Actions

Once the processing is done and the new re-upgraded site is almost-live, the next steps should seem familiar:

7. Re-apply the UX theme designed in the beta site.
8. Re-configure all of the properties of the Editor, Theme, etc.. that were not possible to be stored in the Theme UX Kit
9. Run custom SQL to scrub all of the database records to deal with legacy BBCode usage.
10. Re-apply (or re-verify) that the UX layout of the new site matches the intent -- Blogs, Articles, Forums, Gallery -- all are accessible, function to specification, have the Look and Feel, etc.. as planned.

That's just the bare minimum.

After that, what happens is a slog of testing due to the fact that the DB that was upgraded is not the same DB that beta used.

11. This leads to the anticipation that during the "upgrade done, but not live" phase involves a small army of test/moderator work to scour the site for defects in UX behavior.

At this point we have covered the gory details and most of this has been covered elsewhere in the Grimoire pages of this Wiki.

Reminder

1. The database that is affected by the beta site development is throw-away. It cannot be reused (in part or in whole). It's literally a temporary proving ground that whatever good ideas we have for UX or Style or Layout are proved to work in the new software scheme. TAKE NOTES.
2. The re-upgrade version of the site (not beta) is based on a clean snapshot of the legacy site (both in form of DB and in form of files)
3. The Ancillary functions like Painter(s) that are not part of IP software have to be ported and tested on the beta site before they can be redeployed on the new server site.
4. What was is not necessarily what is. That means, for some features of the site, the techniques used in the backend to provide old (legacy) capabilities may not be applicable (read: recommended) on the new upgrade. The Painter(s) come to mind. There are some legacy issues that we need to discuss privately.
5. BBCode is dead. Except for the boilerplate BBCodes for boldness, italics, etc.. -- all of the evergreen simple and non-custom BBCodes that the IP software honors are not affected. But any custom BBCodes that we allowed in legacy site are going to be impossible to implement in the new site. Just put it out of your mind. The answer is customization to the built-in Editor. I've proven this works. Adding "buttons" to the editor to replace the use of custom BBCode is the way forward. Parsing custom BBCode in posts is not the way forward.
6. Finally, please be aware that testing and validation of the new site features does require a small army of eyeballs to make sure we don't miss anything. I invite the leaders to make sure of this. And even more so, I invite the leaders to involve the elevated moderators of the legacy site to take part in the triage of what features are critical to user acceptance of the new site.

Rationale

As much as I'd like to offer the chance to re-use beta database configuration as a drop-in feature to the new upgraded site, the realities of the software we have make it all but impossible to achieve in the time we have. Here is why:

1. The legacy software version is 3.4.6. The baseline of the upgraded software is north of 4.6.10 (likely to be 4.6.11 or higher by the time we do the actual upgrade). The minor revision numbers are irrelevant. It's the major revision numbers that matter here. Through discussion with IP directly via email and research with the IP community, the legacy version 3.4.6 is not supported as a precursor to an upgrade to 4.6.x. They do not support it-- exactly what IP claims the meaning of support is another matter. We were lucky that the schema of the 3.4.6 database was understood by the upgrade version install tools. We were fortunate. But since it is not officially supported there is no real avenue to go down for how to ingest legacy data to the new DB schema without a bit of luck (which we have established).

2. The DB schema (in both versions 3.4.x and 4.6.x) are roughly 220+ tables. The schema itself is complex and understandably the E-R (Entity Relationship) information about the schema is not public. They won't describe what the records mean. They won't actually explain to me (even in private) what the relationships are between tables. They consider it private intellectual property. So be it. But the impact is that in order to reverse engineer a solution where some of the 4.6.x schema records are extracted and inserted into another live 4.6.x schema are technically possible yet very unfeasible. Not feasible. Not this year. I don't have the bandwidth to reverse engineer their schema in order to partially extract the 4.6.x records into another 4.6.x representation of the database.

3. The simplest and easiest way is to chose a path where we do succeed in the upgrade, even if it means some rework (re-establishment of features on the upgraded DB that we perfected on the beta site). It sounds like negative-work but the alternative is that we don't do the upgrade -- at all.

4. It will be claimed that BBCode is essential for User Acceptance of the new site. I can argue that to some extent, most users will probably find it comfortable and reassuring that some BBCode syntax is still honored by the new site. But let me explain what the new software actually does compared to the legacy (3.4.x) software. In Legacy software, the BBCode are stored as literal BBCode in database records per post/message. This is NOT the case in the 4.6.x version of the Software. Even if in 4.6.x a user enters [b]bold as brass[/b] the actual content stored in the database is the CSS styling that would cause the client to render bold as brass. 4.6.x does NOT store BBCode. 3.4.6 does store BBCode.

For the evergreen simple basic BBCode this means that as users enter content and use those generic BBCode their content is transformed upon submission to the CSS-centric content. That is what is put into the database -- the CSS references to decorate the content. This is wise. It means that the Admin can re-tailor whatever the style is if the style has been given a name at the high level so that changes to the style affect all usage. This is the whole point of CSS.

For non-basic BBCode. For the custom (our custom) BBCode the matter is more complex. Users of the legacy site can and do regularly use custom BBCode to decorate their posts. And in the legacy site (3.4.x) those custom codes are literally stored in the legacy database -- they are not converted to CSS. In the new software (4.6.x) we have two problems: 1, the new software does not permit custom BBCode. I cannot even define them. I tried. The ACP (if memory serves) offers a widget to enter in new custom BBCode, but in practice they did not work. 2, the built-in Editor lets us define new "buttons" where the button causes a wrapper of CSS to be put around the target content. The effect is that the content is decorated with the custom markup (CSS) but there is no BBCode syntax present at all in the database record for that item.

Do I want to go to the trouble of debugging the IP Code to try to make custom BBCode work on 4.6.x ? No, I do not. I don't because as mentioned before the schema E-R diagram/notes are private -- IP doesn't share them. So the reverse engineering to understand what the defect is and reverse engineering a fix in their software is (technically possible, yet) not feasible. I simply won't do it.

The other reason is that BBCode support itself is being sunset by IP. There will be a time in the not too distant future that IP software simply does NOT support BBCode any longer. So any attempt to preserve the legacy formatting syntax is itself an errand in maintaining an obsolete mechanism. It's time to cut this out of the picture. Leave BBCode on the editing floor.

Goals

1. Make the site do as little work as possible. Push to the client the job of rendering content. Liberal use of CSS and templates and style-frameworks is the key to keeping the load on the server low as possible.
2. Leverage server side execution only when necessary -- DB lookup, static files (of course), and application-grade features like the Painter(s) - within limits.
3. Position the design of the site such that the next version is easily attained without the headache of a decade+ stride pace of upgrade. We got burned on this upgrade in some sense. Let us learn from this and make the next upgrade less painful. So design for future goals not just only for today's problems.

Accomplish this, and you'll make the Sacristan's job easier.