2024 ‐ Extracting Relatives Data - CameronD73/529renew GitHub Wiki

Exporting this data

Version 2.3.0 in early 2025 introduced extra export options specifically for GDAT. Those are listed on a separate page.

Version 2.2.0 adds several options for exporting the data collected without DNA segment information:

  1. A Relatives list, in a CSV format that is similar, but not identical, to how 23andMe provided export of DNA Relative (details below)
  2. A tab-separated variable file with the same layout as above, in case the use of commas causes problems.
  3. A TSV list of all in-common-with sets of 3 testers. If the database includes DNA segment data from previous scans then those ICW will also be included.

Both types of export require you to first select a profile person (DNA kit) from the drop-down list above the rows of buttons. You need to repeat the export for each profile of interest.

Differences in the DNA Relatives table are:

The main differences in the file contents between the original relatives export from 23sndMe and the file produced by 529Renew are:

  • There is no DNA data, however the columns are still given as in the original;
  • An extra column at the end of each line shows the largest matching segment size in cM;
  • The file has a Byte-order-Mark header. This was not given in the original 23andMe exports, and caused MS-Excel to incorrectly interpret special characters like apostrophes, accented characters and non-Roman alphabets - unless you took special measures;
  • Family name lists and places are formatted differently.

Content of ICW export

The name of the Profile person is used to create the file name, and the person is not identified in the file itself.

  • The other matches on each line are identifed as Relative1 and Relative2 (in case of name clashes, the each tester's unique ID is given at the end of each line
  • Note because of the way data is stored in the DB, the presentation of Relative 1 and Relative 2 is in the order of their ID, **not **the strength of their match to the profile person. A close match as Relative 1 will be grouped near the top of the file, but other ICW to that person will also be scattered through the rest of the file as Relative 2.
  • For each set of three testers there are three 2-way comparisons, and for each comparison values are provided for total cM and number of segments. In most cases, there is no data for the number of segments between relative 1 and relative 2, in which case the cell says "null"
  • Column headings are labelled between the two testers: e.g. "Prof to R1 cM" gives the total cM between the profile person and Relative 1

Detailed setup instructions (October 2024 conditions)

Instructions on this page relate to recording and saving data related to your DNA matches, excluding the DNA segment details, which are currently not available).

You will find some documentation (and source code) at the 529Renew page on GitHub.. Instructions on this page will assume you have read at least the installation instructions there, and have pinned the 529Renew icon.

The operating buttons required here are:

  1. DNA Relatives page: "529-Gather" and "scan ICWs"
  2. 529Renew Results tab: A line of 3 buttons under "Database", labelled : "Backup Sqlite DB", "Restore Sqlite DB" and "Delete 529Renew DB
  3. 529Renew Results tab: A button (or more) under "Export 23andMe misc data" (coming in next release)
  4. Previous use with triangulation searches required you to manually load each DNA match's profile page. This is no longer needed, nor do you need to click the "Find Relatives in Common" button.

Current status of DNA matching on 23 and Me

This extension was rendered close to useless in November 2023 when 23 and Me "temporarily" dropped access to all useful information about DNA relatives. Nearly a year later and the chromosome browser remains blocked and this extension cannot carry out its original function. Instead it has been modified to allow you to save as much of the information that 23 and Me provides you.

Process to collect and save your data

Remember: it is very easy to backup and to restore a working database. If you manage multiple, unrelated DNA test profiles then I strongly recommend that you use separate databases for unrelated kits. While there is no fundamental problem saving unrelated profiles, the database may eventually become too large to save or restore. See below for more details.

  1. If you have any segment data from earlier use of 529Renew then use the "Backup Sqlite DB" button. I have done no testing to confirm whether the earlier data is completely compatible with the recent changes. That can only happen if the chromosome browser is ever returned.
  2. You can see a summary of the database content at any time by using the 529Renew button (left click) to get the Settings pop-up and scrolling to the bottom of the page under the options and settings. Useful tables are DNARels: how many relatives have been "gathered" (generally increemnts by 1500 per test profile); DNAMats: the dna match detail comparing two people (e.g. pct shared); idalias: the details of unique individuals - e.g. name, idali+hap: how many in idalias have haplogroup data; ICW3way:list of ICW sets.
  3. Go to the "View all DNA Relatives" page, wait some seconds for it to finish loading, and then press the "529-Gather" button near the bottom of the page. This should take a short time, and end up saying "529-Gather Done".
  4. Repeat this Gather step for each profile that you manage who are related (or even if they are not DNA-related but you want to keep all data in the one place)
  5. At this stage you have a list of basic information about each match, plus a copy of internal messages. Not all of the information is stored - for example grandparents' birthplaces are not saved.
  6. Now you can press the "Scan ICWs", which will record ICW's, haplogroups, and family tree urls (if any). This will take a long time (hours) based partly on the deliberate delays imposed to avoid triggering 23andMe's high data rate lockouts but mainly that is how long it takes the servers to provide the data.
  7. Progress is indicated by adding the current relative's name to the button and also by adding a line to the bottom of the 529Renew Results page. Any lines there with red backgrounds indicate errors of some form - some will be fatal, some not.
  8. The scan process seems to be fairly robust, and you can safely view other pages (in separate tabs) and the 529Renew popup page. What you cannot do is refresh either the DNA Relatives or the 529 Results page - either will cause the scan to stop. The cleanest way if you want to stop the scanning is the refresh the DNA Relatives page.
  9. One commonly seen http error is 500 (internal server error), which indicates a problem with the 23andMe server. Try again in 5 minutes. All http errors will cause the scan to stop running.
  10. If the scan stops for any reason then you should save a backup of the DB and then refresh the DNArelatives page. Those relatives already scanned should now be shown with a green background. Simply click the "Scan ICWs" button again and it will start where it left off. In other words, it will not rescan any combination of tester and dna relative that it has already scanned.
  11. When all relatives have been scanned for ICWs, the button will change to "scan complete". If the same name stays for minutes (10s per match is typical - 1 minute might be too long) then you can assume something has gone wrong.
  12. Now select another test profile and repeat the scan for ICWs.
  13. When finished, save a backup of the DB and wait for the next upgrade to the extension to be able to export the results to CSV. If you are keen, you can use any of a number of free Sqlite client software programs to view the data and export your own. My choice is DB Browser for SQLite

Multiple unrelated kits

I would strongly recommend using separate browser profiles in each set of kits of **unrelated **testers - each browser profile maintains its own set of data storage, passwords, etc. However, if you do not wish to do it that way then it can also be done sequentially with a bit of care. It is difficult to say how many kits is a maximum

Sequential operation in one browser profile

  1. Create a new empty database by using the "Delete 529Renew Database" button
  2. Go through the scanning process outlined above.
  3. Save the database, and name it carefully
  4. Go back to step 1 - and start a new DB.
  5. Any time you need to check who is in this DB, the 529 Settings pop-up page lists the testers in this profile - scroll down and look for the table of "Profile Kits"
  6. You can switch DBs any time you are not scanning, but make sure you back up the old one before overwriting with the other. What could possibly go wrong? I still recommend using separate browser profiles.