2026-01-12

Meeting Summary for HAPI weekly tag-up 2026

Jan 12, 2026 Noon Eastern US Time

Quick recap

The team discussed provenance and dictionary terms, including various data standards and their potential implications for consistency across Heliophysics. They identified issues with the SuperMag server API, including incorrect results and lack of documentation, and discussed potential solutions and improvements to the SuperMAG Python client. The team also addressed how to advertise new HAPI servers via the abouts.json file (in the servers repo), discussed processes for adding new servers.

Next steps

• Eric to send another message to Matt about the SuperMag server issue and proceed with creating a patch if no response is received
• Bob to follow up with Eelco regarding the CCM workshop and linkage discussions
• Bob to have student work on converting HAPI data to Pandas DataFrames and adding time variable renaming options to the Python client
• Jon to send calendar invite for Thursday 9am meeting to discuss server list organization
• Jon to set up meeting with Eelco for week of 26th
• Bob to ensure proper citation information is included in HAPI server responses regarding data source origins
• Jeremy to make next Autoplot release use abouts.json
• Bob may add an x_ field to abouts.json to account for testing-only servers
• Jon to send meeting description to HAPI cev list regarding Thursday's extended discussion session

Summary

Provenance and Dictionary Term Analysis

The team discussed provenance and dictionary terms, including DC terms, IVOA, PROV, and data site. They noted some overlap between IVOA terms and identified that isDerivedFrom was an independent concept not linked to a DataCite entity so it might not be connected to other dictionary systems. Jon planned to send a meeting request to Andriy referencing these terms and their potential implications for consistency across Heliophysics.

SuperMag API Documentation Challenges

The team discussed issues with the SuperMAG server API, where Eric identified that the server returns incorrect results that don't match the request parameters. Bob noted that the API lacks documentation and proper parameter checks, while Jon emphasized the value of the HAPI team's rigorous testing approach. Also discussed: the need for better documentation and potential fixes, with Eric revealing he uploaded the SuperMag API to PiPy last summer. They considered creating a GitHub repository for the SuperMAG Python client, though concerns were raised about repository management and keeping it synchronized with the official source.

SuperMag Server Test Bug Fix

The team discussed a bug in the SuperMag server that causes test failures. Eric and Jon considered patching the client code to work around the issue, but Bob advocated for a hotfix to make the tests pass. They agreed that Eric should implement a patch that makes the happy data more useful, while also adding checks to detect when the server is fixed. Eric will follow up with Matt about a previous message and begin working on the patch.

Enhancing Model Data Accessibility

Bob discussed following up on linkage discussions and the CCM workshop, where they plan to attend and ask questions to encourage the use of more standards. Jon suggested presenting a method for making model data more accessible for comparisons, referencing Darren's previous work at the CCMC. Eric mentioned previous interest in using HAPI for model data comparisons but noted a lack of resources to implement it for the GAMERA model. Bob shared an experience with a student using HAPI to address data discrepancies in a paper, highlighting documentation issues and the importance of anticipating user needs in data accessibility.

HAPI Python Client Enhancements

Bob discussed improvements to the Python client for HAPI, including making it easier to convert client output to data frames and handling time variables. He plans to have a contributor work on these features for the latest version. The group also discussed the need to clearly attribute data sources in citations, particularly for datasets obtained through HAPI. Jeremy raised concerns about contacting data providers for client-related issues, and Bob agreed to clarify this in the documentation. Jon mentioned the upcoming PyHC summer school May 18-22, and Bob plans to incorporate the new client features so they can be part of the PyHC HAPI tutorial.

AI Caching and GitHub Updates

The team discussed Eelco's positive response to AI summary features and his caching implementation, which was demonstrated to be impressive with a visible cache population. They agreed to schedule a follow-up meeting with Eelco, potentially at a different time due to time zone differences, to discuss his caching approach and AI assistance for code reusability. Bob and Jeremy noted that the GitHub server list fetching functionality was nearly complete, with only a few test servers remaining to be organized into a more structured system.

How to Organize Server Listings and Manage New Servers and Test Servers

The team discussed how to handle server listings in abouts.json and how to organize the list of different types of servers, leaning towards separate files for production and development servers to avoid confusion (we would not want people expecting performance from dev servers - this ruins the HAPI brand). They agreed that new servers should be added through new server owners submitting a new Issue (rather than a pull request), with production servers requiring vetting before acceptance. The team also discussed adding a "server type" attribute to abouts.json entries to clearly indicate whether servers are production-ready or in development.

The team discussed adding a server description tag to each server entry in abouts.json, and there was debate on this versus separate files (for prod and dev servers), since there still is value in having all servers in one list. Test servers (which are production level, but don't really have science data - just content to use when trying / validating a new client) are good to have around.

They agreed to hold a follow-up meeting on Thursday at 9 AM to further discuss server organization and prepare for an upcoming discussion with Katherine. The team also noted that Eelco would be invited to a separate meeting on the 26th, as APL will be closed on the 19th.

2026-01-05

Meeting Summary for HAPI weekly tag-up 2026

Jan 05, 2026 11:58 AM Eastern Time (US and Canada)

Quick recap

The team discussed several technical issues including catalog request failures, server transitions, and metadata management challenges, with specific focus on improving system processes and addressing scaling concerns. Updates were shared on various ongoing projects including server implementations, API access improvements, and provenance tracking approaches, with team members collaborating on different aspects of system development. The conversation ended with discussions about organizational changes including staff retirements and potential new dataset projects, while addressing concerns about resource allocation and project priorities.

Next steps

• Jeremy to look into CSA server catalog request issue
• Bob to clarify and document the about.json file modification process
• Jon to email Andiry regarding provenance schema discussion
• Jon to schedule a Monday meeting to discuss cadence dataset proposal
• Bob to follow up with Ralph regarding ESA SWE portal API access
• Bob to continue monitoring Catherine's progress on Happy server implementation
• Eric to continue work on generic Python server

Summary

CSA Server Catalog Request Issue

Bob reported an issue with the CSA server where catalog requests have been failing since May 20th, leading to incorrect order displays. Jeremy, who hasn't looked at the code in two years, acknowledged he would need to investigate how the order is determined. The team discussed that while the catalog appears to be functioning, there may be separate issues with the catalog request and about errors that need to be addressed.

Server Transition and API Access

The team discussed the transition of a server to a multi-mission platform, addressing scaling issues for survey missions. Bob shared updates on his communication with ESA's SWE portal team, noting a lack of response and plans to follow up. Jon and Bob discussed the need for non-password protected API access and metadata to streamline data interfaces. Bob also provided an update on his collaboration with Katherine, who is developing an experimental server using Sandy's code. They discussed metadata generation challenges and Katherine's progress on an efficient metadata system, with Bob emphasizing the importance of aligning her work with existing solutions. Katherine seems willing to leverage a generic solution that we provide. She's very competent and also super busy, and so seems amenable to the right kind of help.

Happy Server and Plugin Progress

The team discussed progress on implementing a HAPI server and plugins, with Bob confirming that Katherine is amenable to using their generated code while writing the plugin. Bob shared that Bill E. had encountered challenges with Python server implementation and suggested exploring framework integration (our code with Django) rather than reimplementation. Eric reported working on SuperMAG issues and mentioned he would identify the root cause soon, while also discussing his debugging approach using breakpoints in the verifier code. Bob asked for Eric to note any potential improvements to the verifier so future people could use it without have to add breakpoints.

Server Information Management Process

The team discussed the process of managing server information in abouts.json files, which is currently maintained manually. Jeremy explained that a few summers ago, he worked with Jon V. and Jon's Student on a server verifier for liveness testing. Bob noted that when adding a new server, the abouts.json file should be updated, and this change is reflected in the legacy all_.json file through a nightly process.

Server Metadata Management Process Review

The team discussed the need to clarify and improve the process for managing server metadata, particularly regarding the way some info in the master file i potentially over-written by the automated filing process. Bob explained that the system automatically updates server information nightly, but there was confusion about which parts of the metadata should be manually edited and which are automatically generated. Jon suggested creating a separate overlay file for manual edits to avoid conflicts, and Jeremy requested a development version of the system to test with. Bob acknowledged the need for a clearer distinction between volatile and fixed metadata and agreed to leave the discussion open for further clarification.

HDRL Strategy and Server Updates

The team discussed an upcoming HDRL strategy meeting in January where Jon will present on HAPI and HelioCloud projects. They also discussed server configuration conventions, with Jeremy suggesting a focus on reader requirements rather than file formats, and agreed to revisit this discussion when Eric starts working on the new server. The team noted that Baptiste had improved the intermagnet.ttl file creation process by using RDF objects instead of manual string writing.

Data Provenance

The team discussed provenance tracking for data, focusing on how to handle relationships between different versions and sources. They determined they should be careful about using the term "provenance" as some might expect full-featured provenance tracking, and decided to sync up with Andiry about their approach.

Space Data Cadence Project Discussion

The team discussed a potential new dataset project involving cadence information for space data, with Scott proposing to work on it part-time after March. Bobby expressed concerns about maintenance and implementation challenges, while the group debated the project's justification and priority. They agreed to further discuss the proposal at an upcoming HDRL meeting, with Jon suggesting a dedicated Monday HAPI meeting to explore the concept and its potential benefits.