Blog 2024 06 - lago-morph/chiller GitHub Wiki

2024-06-30

  • Got frontend integration tests working. Now service unit tests, sdk integration tests, and frontend unit and integration tests are all working.

2024-06-29

Brainstorming tasks and next steps

  • automated integration test for frontend (start with logic from SDK integration test)
  • come up with automated component intended to provide load from simulated users (integrates with frontend at http request level)
    • Some tool possibilities - JMeter (resource-heavy Java tool with lots of features), Taurus tests defined using yaml, k6 uses JavaScript for test definition, Locust which uses python to define test scripts - "known for its simplicity", Wrk uses Lua, command-line focused, vegeta uses Go, command-line focus, Selenium
    • Here is a nice list with good pros/cons
  • add separate database component (for now manage access secrets in very very insecure way)
    • postgres or mysql docker container
    • appropriate scripts for initializing db
  • document application in form of PPT presentation - intended to be start of tutorial/documentation for portfolio application
  • packaging for application (sdk, service, frontend)
    • set up package creation procedure make sure versions are rational
    • set up way of hosting packages once built (e.g., github pages)
  • figure out procedure for executing already-written tests using github actions - don't tie it to repo events yet, just manually-triggered
  • figure out secrets management for DB (probably use github action variable)
  • figure out and test k8s deployment process
    • generating various k8s configurations
    • packaging as a helm chart and test with minikube

2024-06-28

Current status:

Have been working but not updating this page. Copying stuff I've done from past entry.

  • create and get running real unit tests for API layer
  • Write real unit tests for SDK
    • The SDK code is generated by swagger codegen. I haven't touched the code, and will only do unit tests if I find bugs and have to fix them.
  • Create SDK integration tests that work with live API code (requires environment variable to specify API URL root)
  • Document local development platform
    • run in directory parallel to git working directory
    • Do I want to try to use codespaces?
    • Document the codegen process somewhere other than this blog

2024-06-24.02

Added a flow diagram to the wiki page on application design. Using Mermaid, which is integrated into the GitHub wiki rendering software.

Goals for today:

  • Run chiller_api spec through OpenAPI linter to see if there are simple changes to make now before really getting into things
    • It seems that there are a few problems after running it through RateMyOpenAPI
      1. I do not specify a server for my API. Not quite sure how I should do that, as the API will be private and not accessible from outside the cluster the application is deployed to. Maybe this has another use?
      2. I am using path arguments during a POST operation (/movies/add). I should change the schema for the POST to be a compound document that includes both a user object and a movie object. But I'm not going to do that right now.
      3. There is no default security defined. This will get filled out later, when I add OAuth for user authentication, and JWT + Open Policy Agent for user authorization within the application
  • Get python-flask code generated for the API layer into github repo
    • get repo checked out and set up authentication tokens
    • make sure unit tests run (with nothing in them)
    • I had to do several things:
      1. Edit requirements.txt so the connection package would be less than 3.0. First line changed to connexion >= 2.6.0, < 3.0
      2. Both chiller_api/__init__.py and chiller_api/controllers/__init__.py had null characters and nothing else. Had to delete those
      3. Ensure flask testing version ==0.8.1 in requirements.txt
      4. Ensure jsonschema < 4.18 (forced deprication of one of the methods)
      5. Found a bug in the code generation. I used "UserID" as a parameter for the movies api calls. In generating the code in the api controller, it changes that to "user_id". In generating code for the test cases, it leaves it as "UserID" in one place, and changes it to "user_id" in another.
  • Generate Python SDK from openapi spec and get into git repo
    • Make sure SDK unit tests work (with nothing in them)
    • These just worked first time. I guess the python code generator is maintained more than the python-flask one.
  • Decide on and document repository directory structure

2024-06-24.01

Some fun tools I want to investigate (all from the OpenAPI Tools site)

  • Mayhem takes an OpenAPI spec file and generates a stream of randomly-generated API calls. I guess this is for fuzz testing the API. Can use it to get some nice sample transactions for unit testing I suppose. Exercises error handing and validation.
  • RateMyOpenAPI I guess evaluates the API file?
  • oapi-codegen generates Go servers and clients from an OpenAPI spec
  • Huma a Go microframework
  • Fuego another Go framework

2024-06-23

While Marta is working on updating her knowledge on creating micro services with C#, I am going to create a very minimal web application using Python and Flask.

First, I created an OpenAPI 3 definition of the API we will use. This was kind of an adventure because I was simultaneously learning the OpenAPI spec and trying to figure out how to use different graphical editors. I browsed through the list here and used the first one I could understand and seemed to provide valid output. I ended up doing a lot of work using a free account at APIGit (link to private repository). Unfortunately I can't make a free repository public. Going forward I'll probably just edit the OpenAPI spec by hand and verify it using the online Swagger editor. The online swagger doesn't do a very good job of creating new OpenAPI spec. However it works well for finding errors in an already-written spec and telling you how to fix them. There are a couple of versions. This is the stable version, and this is the "next" version. I used the stable version today, but may play around with the other one. If I actually do much work with OpenAPI I will find a better solution than copying and pasting into various online editors. I'll probably also convert it to JSON, since I hate the errors that come with a space being in the wrong place in yaml.

After cleaning the code up that came out of APIGit, I used the Swagger code generator (V3) to generate server stubs for python-flask. For future reference the full command line I used was

docker run --rm -v ${PWD}:/local swaggerapi/swagger-codegen-cli-v3 generate \
    -i https://raw.githubusercontent.com/lago-morph/chiller/main/chiller_api.yaml \
    -l python-flask \
    -o /local/out2/python-flask \
    -DpackageName=chiller_api

That exact command line may not work going forward because I'll probably move the OpenAPI reference file, and convert it to JSON. So the -i argument will change.

This seemed to do a pretty good job of generating basic stubs - now I just need to figure out how to get it to run locally, make appropriate unit tests (that will fail), then fill in the guts of the program. For now I'll just store the data in some .ini files created locally to the api controllers. Want to get it working on my local machine before I think about deployment in other environments. Will add a MySQL (or something) data store that will be accessed via a network call later.

The next step would be to write the pages that serve as the front end. I'll probably just adapt the Flask tutorial application (and its unit tests) for that, and use a python client SDK generated from swagger codegen. It takes care of all the conversions between Python and JSON for the front end.