Design Notes - AttiliaTheHun/Songbook-Manager GitHub Wiki
Among a group of people I was making summer camps with, we decided to rework our songbook. The current thing must have been decades old and it was basically a book, hardcover. We realised we no longer liked half the songs that were in it, we did not respect half the chords in there and we noticed more and more typos over the time. It was simply time for a change.
We agreed that we want to be able to make changes to the songbook in the future, so it will not be a hardcover this time. Instead, we decided to go with a collection of A5 sized transparent plastic foils in a plastic cover. You can freely rearrange the foils within the cover and you can put anything into any of the foils without much trouble. It is even water-resistant in comparison to the old thing which was just paper.
The obvious question is what do we use to make the new songbook? Obviously a lot of text formatting and manipulation needs to take place and once that is done it needs to be printed. But do you want to do this in MS Word? The software that makes the formatting of the entire document explode if you press enter at the wrong time? No, thanks. There are alternatives, there is Publisher, there are countless text editors, but each of these makes you feel like you are writing a book, a scientific paper or a program. Then there are also some programs specially for creation of songbooks, I found at least half a dozen just here on GitHub and on Google Playstore. I have to admit that some of them look even better than this one.
You ask why I did not use an existing program and instead chose to reinvent the wheel again? We had one very specific requirement for the new songbook. The size of the plastic cover that collects the foils has a limit of how many foils can fit into it. We have tested it and found out we have around 80 foils of size A5. That gives us 80 pages of the songbook. Given this physical limit, we realised it was absolutely necessary that each song fits on a single A5 page to get as many songs as possible.
Now that we had this constraint, it was not much work to find out that none of the other programs for creation of songbooks really cares about how much space will the songs take up once formatted. I also realised that a quick and dirty solution to our problem is the good old Hypertext MarkUp Language.
Think about it. HTML lets you do anything. The song will take up as much space as you allow it. And every browser nowadays can convert the HTML page to a PDF file with a single click, so printing is not really an issue. Once I got this thought, there was no going back. It all looked so easy and almost elegant in my head.
The idea was that each song will have its HTML document and then two of these will be put side by side using flexbox to another document. I started with iframes, but after a lot of (trial and) error I decided to make each song just a <div>
element with content and then dynamically insert these divs into a template to generate a page of the songbook.
I made the entire songbook, around 160 songs or so until I realised that printing them one-by-one by hand through the browser print dialog window sucks. Surely there is a way to automate this process. Well, there is. Sort of. Any browser based on the Chromium engine supports a so-called headless mode which means you can open it from command line and it will not bother the user. This headless browser has the ability to convert the page it is brosing to a pdf file. Except it does not have the same options as the printing dialog in the browser. Specifically I could not remove the excess borders from the PDF, which broke up the formatting of every but the shortest songs, because suddenly a part of the song was not displayed as the entire document had a bigger spacing on the top.
I found that the spacing (margins) can be specified, but you need to connect to the headless browser through a websocket connection and then pass him all the arguments there. This is not as simple as executing a command with argument, but it is doable. However the only way to connect to the browser I got working was a Node.js script. It got the job done, but I also had some other pieces of related code that I simply did not want to port to Node.js.
I had a php script that generated an index of the songs in the songbook and put this index on the beginning of the songbook. I also had a clever idea to change the content of some songs in one of the printed instance of the songbook for something else (an easter egg for others to find), so I created few of these easter eggs and I had a script that substituted them for the regular songs when I wanted. I also did not mention that I formatted all the songs in a web-editor of some webhosting site, because I could not make php work on my computer, so except for the PDF conversion part, all of the stuff was not even on my local machine.
Simply put, it was a mess, nothing like I imnagined it to be when I convinced myself that it will be not just doable, but easy. So I decided to put all this into one place, a desktop application. The application would allow you to edit the songs, generate the index and let you create a PDF with a single click. At that point I thought that converting an HTML document to a PDF file was not a problem. Boy I was wrong.
I decided to go with Java, as it is the love of my life my favorite language and I was not very confident in my coding skills at that time. In fact I had no idea what I was getting myself into. I thought the thing will be complete in a week.
The application obviously needed to contain a webview to preview the songs (render the HTML). Getting JavaFX (the graphics library) to work and learning to work with it was a major pain, but I have to admit I like how the app looks now. I created a manager for the normal songs and for the easter songs and implemented a simple framework where you could add and delete songs and also to hide them. I managed to find a good code editor to embed in the application so that you could format the songs right in the application. I created a basic template for new songs and a stylesheet so that all I had to do was fill in the lyrics and the chords and keep them in sync.
The application could now do all the old php scripts could and much more that I did not even imagine before. Except the thing i made it for in the first place. I was so confident exporting the songbook was not a problem that I did not implement it until every other feature on my list was done.
At the time, I imagined I will find a library that can convert HTML to PDF files, I will add it as a dependency to the project and live happily ever after, no need for websockets or anything. I did find such a library, but it did not support CCS3, so the output PDF looked nothing like my songs markup. It did not support grid, flexbox, the float property. None of the formatting I put on my songs was visible in the result so it was not applicable.
Naturally I searched the web and found that the closest thing to what I want, a library that supports the modern features of HTML and CSS is puppeteer. Aside from puppeteer being a Node.js library, there is one big reveletion. Puppeteer does not really convert HTML to PDF files. It is a wrapper around a headless chromium browser that communicates the need of PDF generation through websockets. So if I decided to use puppeteer, I would have to add a Node.js runtime and a Chromium browser as dependencies to my program and ideally ship it all together. The whole application would then have hundreds of megabytes.
I spent a lot of time thinking, trying simpler templates to see if tools like wkhtmltopdf would work, but to no avail really. I had to reformat all of the 160 songs several times (every time I thought this was the last template I am using), but ultimately I gave up and accepted that I am using headless browsers. The pain also helped me to perfect the templates to their current state, embracing modern CSS and making them easier to use than all of those before.
I accepted that my program will have to rely on the users to have a Chromium browser installed, but since Microsoft ships all new Windows systems with Edge, this was actually easy. I still wanted to avoid the Node.js runtime though. To my great surprise, I found Playwrhight, which is basically a Microsoft version of Puppeteer, but it has a library for Java and C#. I did the work (it was not easy) and incorporated Playwright to the program. I was thrilled to see that the program finally creates the PDF files it was supposed to.
Then one day I wondered why the playwright drivers are so big, they took more than 100MB. I found the reason was that the drivers were a bundle of drivers for each platform, so I actually had 4 of them, one for Windows, one for Mac, one for Linux and one more. I realised that I can manually remove the unnecessary drivers and when I tried to do so, I found out that each of the drivers is packaged with its own platform-specific Node.js runtime that creates websockets.
I have almost shit my pants at that point, but there was really nothing I could do. The program was working and changing the code again would be stupid.
So now the program uses a WebView to display the songbook to the user, but when the songbook is to be exported, it uses a headless web browser the user has installed. And it uses a Node.js runtime to communicate with this browser.
When I first started with this project, I believed it will be easy. I will do it in pure Java, no Node.js and no Python (I tried to make Selenium work in Python at one point). It will be done in no time and I will be able to count the dependencies on my fingers. Is it not an irony?