My strategy is to build the Map Display up from a skeleton of Widgets and gradually implement functionality.

The next design curve to solve is that creating menus is changing in GTK.

What Mapdisplay was using before is now deprecated.

It looks like the new method is to create a Gtk.Application, and use GIO action routines.

Converting D-rats to an Application does not look too hard. Actually I think eventually D-Rats may be end up being 4 applications. D-Rats main, D-Rats repeater, Form Builder, and MapDisplay.

Currently this code is separate from D-Rats in my fork and tested as a python3 module as python 2 can not handle the directory structure in use.

python3 -m d_rats.map.mapdisplay

Skeleton MapDisplay

For now though, I will have map.MapDisplay.py be a Gtk.Application for testing, and move classes into individual modules.

The resulting module is only working on Python3. I cannot get the relative import to work for Python2. At this point for MapDisplay, I will not be trying for Python 2 compatibility.

As it gets too long to put example files here, I have instead created a Pull request for Skeleton Mapdisplay

A minor step. I have refactored some classes out of the mapwindow.py module into their own modules. Pull request for refactor of Mapdisplay

Adding in a menu

The previous program is still boring, the main thing is to make sure that it does not throw any exceptions.

Now the old menu handler was deprecated. This step adds in a new menu hander. Pull Request for adding the Map Menu

Now at least something works.

Next is to see about making the status, progress, and map controls visible and somewhat functional.

Getting the Map Controls visible exposed a bug where I missed a show() method.

Then it exposed a issue with making a class based on a Gtk.Box. The Gtk.Box.New takes two parameters. Apparently the Gtk.Box__init__() does not take the same two parameters. The first parameter to init() does not specify the orientation, and since I do not know what it does, I can not specify any parameters to Gkt.Box.init().

And it seems by default the Gtk.ProgressBar display of text is disabled.

I temporarily set the Map menu refresh action to update the Status Bars to make them visible in the test program. Now the frame is looking more like the old program.

I don't like methods returning tuples as a personal preference. So I created a MapPosition Class to contain the coordinates. Currently it just has the longitude and latitude and allows displaying the coordinates as a string. Pull Request for Stetting up the map controls

Now it is time to start getting the map widget to have some data and working controls.

Adding the scale to the empty map

I noticed that the when testing the initial Gtk 3 conversion, I noticed that the zoom control was hard to adjust. The slightest touch caused a map refresh which took quite a bit of time while. I see code in the Mouse Movement to deal with that problem by ignoring actions until a period of time had elapsed with no changes. I added that the the zoom handler and it seems to have cleaned that up.

I then added the scale to the map with Cairo and Pango.

The scale needs to position based on the map coordinates inside the visible scroll window. For that we need to know the size of the visible scroll window.

The size of the visible scroll window is available from the two adjustments built into the Gtk.ScrolledWindow class And it needs to be looked up at the start of the draw thread.

It was a bit hard to find documentation on how to do the Cairo code needed for the Scale.

I finally found out that what I needed to use was the stroke method from the CairoContext. Most of the examples were using either the fill method or the paint method. The paint method paints the entire screen, and the fill method does the same thing as it expects to be done inside of a closed shape.

Pull Request for Drawing the scale on the Map

Minor fix up to the scaling position

The offset in pixels for the PangoLayout below the scale is a constant. It needs to be based on the size of the PangoLayout. This of course also affects where the scale should be in the map.

It is also dependent on the tile size in pixels and the width of the scrollbars that may be optional depending on what type of display device you have.

Pull Request For fixing the scale position

Items still to follow up on

• pixel size of 128. This seems to a constant based on the size of a map tile which is 256, but is treated in the code as a variable. There should be a way to look up that value from somewhere if it should change.

• We still do not have the map tracking the latitude, longitude and Zoom. I finally found that those were being set inside of the MapTile class that I have not ported over. It seems better to have them in their own class.

Getting the Map zoom control working

Variables that are related to each other should reside in a dict or a class. A class allows you to have rules on the values of the variables based on their use and other variables.

It looks like we need a class that has the current map position, the zoom factor, the current dimensions of the map, and the number of pixels in a tile.

That class was MapTile in the old code and while I am still thinking there is a better class for it, I am going to use MapTile for now. The MapTile needs to hold that information and more, and has addition/subtraction operators in it. I have set up MapTile to now need named parameters when it is instantiated.

There is one instance of MapTile just for holding the values for the center of the map, and it appears that tile in the map will be described by a MapTile instance.

Pull Request for reporting coordinates and scale

The above commit was missing the new maptile.py module.

Getting the broken tile to work

I tried a lot of stuff to get the broken tile in-line generated pixmap to work and failed. Every algorithm that I tried failed to produce a usable tile.

So I gave up, and used the gimp editor to create a broken tile as a PNG file. I took a cached tile from the older map program as a template and then replaced the content completely.

This commit just has the map fetch disabled to force the broken tile to be used.

This required refactoring some "private" attributes in classes to make them "public" and adding some access methods.

Pull Request for having broken tile work

Getting map downloads to work.

Now the the broken tile is working, time to start downloading new tiles. I added in the downloading code, and fixed some issues with the zoom control.

The progress bar is reporting the successfully downloaded tiles now.

If a tile fails to download such as just not being available, I cache that now to avoid re-downloading it.

Pull Request for map being downloaded in background

Getting Map Print and save to work

I noticed that the zoom settings were still not working correctly when set from the program. Needed to add some class variables to store the initial and previous values to determine when a queue_draw is actually needed.

The other issue that I noticed is that the map and the scrollbars for the map window were not centered, like they were in the original code.

This required setting a class variable in the MapDraw class for the handler to determine when it needs to center the display. Until we get into the MapDraw handler, we do not have the map size data in order to set where the scrollbars should be. After the scroll bars are set, the class variable is cleared to allow their manual movement.

Had some issues with getting the map saved and displayed in a web browser. The initial code when ported showed the opened print menu on the captured screen.

Also it appears that only the what is visible on the screen would actually get printed. I had to for now drop the functionality of print/save of the entire map.

The map display capture for saving had to be delayed until the map menu item method had returned to the caller so that the map was dismissed. I added a small delay to allow the map display to be updated before capturing the screen.

And again, only what is visible on the screen will be captured. If anything is covering part of the map, that will just be captured as black.

At some time in the future it may be possible to render the entire map into a non-displayed surface buffer and then save that to be a PNG.

Pull Request for adding Map Print and save

Mouse Popup Menu

Now the challenge was to setup a mouse popup menu and also try a smoke test of the map source editor. The map source editor failed to launch, so will look at that later.

The Mouse Popup menu was vary similar to the map menu. But some differences.

One of the articles online said to use slightly different convention for building the menu as it was preferred, so I used that.

I have not yet converted the existing map menu to be consistent. Will probably do that as a later edit.

In order to have the latitude and longitude like the previous code as a menu title, I had to create the menu in the mouse button handler.

A lot of other small things changed.

The menu initially displayed, but none of the menu items could be selected.

It took a while, but by trial and error and a lucky guess, I discovered the fix was that I needed to have the popup menu attach itself to the MapWindow widget.

There appears to be a commented out option in the original code to set a crosshair marker. Not sure why it is commented out, as it has seem to have been first changed to use a plus sign for the crosshair.

Pull Request for adding the Mouse Popup menu

Marker Popup menus

Testing this uncovered some issues in MiscWidgets and utils modules which I handled with separate pull requests.

Pull Request for fixing MiscWidgets GdkPixbuf handling Pull Request for fixing utils Icon handling

I discovered that the MapWindow marker_list attribute was expected to be a TreeWidget and not the ScrolledWindow that contains the TreeWidget, so I had to do some refactoring.

I got the map centering on both double clicking and on the right click centering option. Since the new center value was needed in many places, I added it as a class property.

The location as a QST is only tested to the point of operating the dialog for it as a unit test.

The prompt_to_set_marker dialog appears to be working, but is apparently not saving the results. It seems to have created a file or directory, but always claims it is creating the Misc group because it does not exist.

Even if it did save it, I do not have the code for the expose handler working. It currently just points to stdout.

And we have another popup menu to implement. This one is different than the mouse menu popup, as it is popped up from the TreeWidget results.

In this Unit test, currently the TreeWidget is not yet populated.

For this menu popup, we need to pass the data derived from the TreeWidget to the menu handler. While there are a few ways to do this, I decided to pass it in the action state. This allows the menu to be constructed once and just popped up when it is needed.

This PR just exercises popping up the menu.

I am probably going to combine the add_point and update_point into one method to reduce the duplication and the duplicate calculations if the point to be updated does not exist.

Pull Request for Starting Map Marker implementation

Starting to active the new Map modules

Now in order to do more testing, I need to get the treeview populated.

The attempt to do that was to get the python3_tyw branch launching the updated map code.

The d-rats.py main module was updated in preparation for this. Just some fixups to replace deprecated modules. I also discovered that the profile option did not work, so had to fix that.

Pull Request to update d-rats.py

The next step was to get the mainapp module to call the new map code.

The MainApp class now inherits from Gtk.Application. The mapdisplay module is now only used for unit testing, so calls needed to be changed to use the new mapwindow module. I had to add some missing methods to the new mapwindow.py.

First attempt at testing this exposed some bugs that destabilized d-rats to the point that it froze. The time from it freezing up varies from the initial launch of the map

Found this clue in the output:

(d-rats.py:30201): Gtk-CRITICAL **: 08:26:53.022: New application windows must be added after the GApplication::startup signal has been emitted.
01/04/2022 08:26:53:INFO:MapWindow:Testing MapWindow
setup menu

Also seeing this message repeated a lot:

(d-rats.py:11696): Gtk-CRITICAL **: 18:58:58.435: gtk_label_set_text: assertion 'GTK_IS_LABEL (label)' failed

In both cases, the line numbers are not useful for finding where the issue is. Finding the cause of these messages is a bit of a pain. You may get lucky on an internet search. You may get lucky on putting in change update some GTK 2 code to GTK 3 and see the message go away. Other wise it can be a matter of adding print statements to the code similar to doing a binary search until you find the method.

It turns out that I missed changing the old Gtk.main() call to the new call self.run() when changing to a Gtk.Application widget. The shutdown handling for an application is also different.

I added some overrides to the shutdown handlers and also some shutdown event handler. This includes some temporary diagnostics so that I can see the order that various methods and handlers are called.

The "virtual" methods in a Gtk related class are meant to be overridden by a class inheriting them. It looks like the parent class shutdown methods need to be called by any method that overrides them. I have not found any documentation that says to do that.

The mapwindow class needed a flag to tell it when to exit instead of hiding. I used the shutdown event handler for this instead of overridden do_shutdown, simply because I started with trying to use just an event handler.

In a future Pull request, I plan to remove unused overrides or handlers and the diagnostic code.

When debugging a class or a module, it can help to add statements to print(type(parameter_name)) so that you can improve the accuracy of the Docstrings.

So with this pull request, the map is displayed, hidden, and cleaned up on shutdown. Unfortunately, I still do not have plots of stations and other markers working yet.

The config panel for adding Radios also is not working at this point. This set of PRs is concentrating only on Map related issues, so I will not be addressing that here.

Pull Request start conversion to new map modules

The geopy module

There is some commented out code in that section related to geopy.

The geopy module is a python library for converting street address, etc, to latitude and longitude. It needs an external database for this. D-Rats used to have an obsolete and non-functional geopy module added to it. For this fork, I have removed it and fixed the code to use the current geopy module if it is installed. Not all platforms provided a signed geopy module, and we do not want require use of Pypi (Pip installed) modules in an application.

The builtin geopy module only knew how to connect to a Yahoo provided service that yahoo discontinued.

My guess is that this code got commented out when the Yahoo discontinued the service.

Re-enabling the code if geopy is present will be a future task.

The quest to add a marker

Still trying to work to the goal of getting some type of marker to display on the map.

The "New Marker Here" brings up a dialog that wants some data including a group name that defaults to "Misc". There is a "Lookup" box that does not appear to be changed and is set to "By Address", probably related to geopy.

If I do not change any entries and click ok, I get a popup claiming that "Group Misc does not exist. Do you want to create it?" Clicking yes, does not appear to do anything. There is a static group of "Misc" that I see if I click on "Edit Map Sources". I do not know if it was built in or created by one of my tests.

However even with Misc known to exist, it still asks me if I want to create it each time I do this test. Clicking on the "edit" button for "misc" pops up a dialog that says "Nothing to edit here" and give an option to close.

So the first clue to investigate is why it is prompting me to create the misc group that apparently already exists.

And the answer is that for testing, I just stubbed out the methods for populating the data.

Currently only the FileMapSource appears to work. Commented out the other MapSources for now. A bunch of code needed to be changed in the 'mapsources.py' in order to work at all with python3.

Adding missing code, results in several asserts about "'G_IS_OBJECT (object)' failed" followed by a 'Segmentation fault' where everything stopped.

This seems to be coming from adding data to the MapMarkerList, which inherits from Gtk.Treeview.

The API for the Gtk.Treeview has changed quite a bit for GTK 3. Unfortunately most of the API references have a mal-formed link to refer to the document that gives an overview of the TreeView class. The correct link is An overview of GtkTreeModel, GtkTreeView, and friends . Which needs the Python GTK.TreeStore documentation for applying to python as there are some Pythonic changes to the API.

It looks like both the GObject enums or the python types can be used, just as long as you are consistent. So at this point no need to go all through d-rats and change them.

I tried adding Print statements all over every method that I thought could be called.

What I found out:

1. The window show() call is that triggers all the messages and segmentation fault above.
2. As long as the Gtk.TreeStore does not have any data in it, no messages or segmentation fault above.
3. The CellRendering code was not called before the crash, and the expose handler also does not appear to get called.

I made some small tweaks to the unit test for the d_rats.miscwidgets unit test to try to reproduce the data issues. No crashes resulted.

So the next step is to add a unit test module to the map/mapmarkerlist.py new module. This crashes in the same way if data is present in the Gtk.TreeStore.

So this indicates that the problem code is probably in the mapmarkerlist.py module, but it still could be in the miscwidgets.py module.

Now I can start commenting out stuff unneeded for the test to see if I can isolate what is wrong.

Found the issue. It was something I had done wrong in the porting. I was trying to add a new Gtk.CellRenderer when the Gtk.TreeViewColumn already had a Gtk.CellRenderer. I needed to use the new API for looking up a list of GtkCellRenderer objects and then use it in a call for a set_cell_data_func().

More DocStrings have now been updated from doing this debugging.

Now the application is not segfaulting or reporting asserts.

Markers are not being added yet. Still need to debug that code, and probably add more code.

Finally got the markers added to the map.

The list of if a marker is visible can only be created once the map tiles are fetched for rendering. And this list must be regenerated if either the center of the map is changed or if the zoom level is changed.

The markers were being placed at an offset from the map.

Part of the issue was caused by a difference in python3 to python2. In python2, if you divide two integers, the result is converted to an int. In python3, you get a float.

I ended up replacing the fudging calculations, and even after that noticed that each zoom level of 14 or higher needed an additional unique correction to the fudge factor. I also noticed that the position of items in the tiles also seemed to shift with the different zoom levels. I am making the assumption that this additional correction is constant for a zoom level.

One issue to chase down is that at the higher zoom levels, periodically some tiles get replaced with the broken tile, until you adjust the position of the slider. Sometimes selecting a different window or application triggers it.

There also is a bit of refactoring that may be wanted. There is the MapPosition that I created, and the existing GPSPosition, and a MapPoint classes. These should probably have a common parent which could simplify some of the code.

There also seems to be some duplication in how some numbers are calculated for map placement. in many cases those resulting numbers are either constant for the entire application, or only change if the map position / zoom is changed.

Programming principles to keep in mind for optimization.

• DRY: Do not repeat yourself. Do not have duplicate code.
• Beware of magic numbers. Many numbers should be named constants. Others need to be looked up from the environment.
• Single source of truth, related to DRY.

And how the code looks now: Starting to plot markers

Feedback from this PR by testers:

• Points are still not being plotted in the correct place. I think the issue is that it appears that the map tile plotting is taking the earths curvature in to account and it looks like the point plotting is not.

• I am not passing the MAP API key from the configuration in the MapDisplay test module, so maps that requite it can not be tested.

• Unrelated to this, the Gtk.TextTag constructor is throwing an exception for a tester. Only thing I can think of is that it needs a named parameter, which still seems to work for me.

• Unrelated to this, if the one of the internationalization files that needs to be generated was not generated, the end user gets a useless python stackdump.

Add ability to add thunderforest maps to the unit tests.

This should fix one of the issues above. I still have not tested using Thunderforest maps.

Since the Thunerforest maps are in the default d-rats configuration, they require a copyright notice in the d-rats startup.

Starting to plot markers

Getting points plotted in the right place

The tile size of 256 square is a property of the map tile provider. Eventually this needs to be moved to be a configuration option.

The map drawing area is a grid of 9 tiles square. Currently a constant in the programming.

The amount of map that is visible is controlled by the size of a scrolling window.

A map tile is selected by converting the longitude and latitude to an horizontal and vertical grid coordinates.

These grid coordinates floating point values, but only the integer component is used to look up the tile.

The integer components map to the map drawing area from an offset. This offset depends on the zoom level and the distance of the center tile from the center of the drawing area.

The actual longitude and latitude point is somewhere in the map tile.

To find exactly where it is, you need to first get integer value from the floating grid coordinates.

Save this integer. This integer seems to correspond to close to the center of the grid that contains the point.

The integer corresponds to one of the 9 by 9 tile locations.

Then you need to get the fractional part from floating point coordinate. This is a fractional part of a 256 point position. The integer portion is just used to identify the tile, the factional part is the position in the tile. You have to subtract the offset of the Northwest corner tile from the floating point coordinate.

To covert a map drawing location back to a lat/long you need to do the reverse.

first divide the point position by 9 to get the relative tile position. Then the offset of the map grid to the drawing grid is added. Now this floating point pair can be converted to a latitude and longitude.

Unfortunately I initially did this math wrong and did not correct it until a later PR below.

The result is that I ended up using the existing code for estimating the point plotting. At the time that the point is plotted, I add a zoom specific correction factor. I am hoping that that factor is not dependent on something else like the latitude or longitude.

And it still looks like the farther away from the center of the map, the points are being plotted poorly. I am also having the GUI lock up, and the program crashing with segmentation faults when moving around the map and adjusting the zoom levels. Sometimes I can move around with no problems, other times not. No pattern has been found.

On the master branch of d-rats, my location is still being plotted on the map at the wrong place. It appears the looking up my latitude/longitude location online with web tools does not agree with what my handheld GPS unit is reporting. I am using my handheld GPS reported position to track the fudging error in the map.

We need to get consistent with the abbreviations of latitude and longitude, Choose either lon or lng. Need to come up with a better naming convention for x,y coordinates. We have x and y coordinates for the map widget grid offsets and also values used for the map provider. Need to have a set of constants to indicate the maximum longitude and latitude values for normalization. The mapwindow.set_base_dir() method is not named correctly for what it does.

It looks to me that the map positioning is as good as I can get it at the moment, and appears to me to be no worse than the current master branch. Time to merge this in and move on to getting the various menus and mouse operations working.

Improving Marker accuracy attempt

We probably need to write a standalone map application that can be used to test what is going on. There must be an simple solution to this that I am currently not seeing.

Reference: OpenStreetmap.org - Slippy Map Tilenames

Getting rid of the fudge factor

Added the Information info popup. For now I added the the background color to the text markup. This is not ideal as only the text has the background color and the section of the visible box with out text are just a default background color close to white.

I looked up how to convert them to use style sheets, and it does not look too hard. That will be a future enhancement.

Having the mouse action exposed that I still had the fudge-factors wrong at all zoom levels.

For diagnostics, I temporarily added print statements for the plotting of the map tiles using both the num2tile and xy2longlat. At zoom level 18, they seemed to match. At zoom level 14, the default, only the outer tiles matched, and their was a error on the inner tiles.

After consulting with my sons about this, we found that I had some bugs in my equations for replacing the latlon2xy and xy2latlon. Even after fixing them, things were wrong until I discovered that I also had an error in the manually copied data from my handheld GPS, so it was off by over one tile from where it should have been.

I moved the display coordinates to and from latitude and longitude degrees to the MapTile class and also moved variables that were constant for a zoom level and center position to be class variables.

The result finally started plot points where I would have expected them to be, and the mouse position in degrees was also tracking.

The markers icons still looked off on the map though. They were not being plotted centered on the point that they were for. It appeared that the plotted point was used for he north west corner of the icon. A fix was added to look up the size of the icon to center it. This made the text label look wrong, so a similar adjustment was made.

So with this next commit, only the "slippy" map algorithms are used. All code using the fudge factors has been converted and latlon2xy and xy2latlon have been removed.

More restructuring may be done in the future. We may need to add some new classes to eliminated passing location pairs as tuples or separately to methods.

Possibly the values that are constant for a zoom level and position should be in their own class.

Remove fudge factor Add Info Window

Enable the GEOCODE lookup where available

Just a few minor bits of refactoring to implement some commented out code.

Enable Geocode Lookups

And as a bonus, openstreetmap let their https certificate expire.

This allowed the opportunity to fine tune the error handling in MapTile.

Enhance MapTile exception handling

Since openstreetmap was having issues, it obvious that the next thing is to find out why the url_key for ThunderForest was not working.

Fix URL keys to work

next is getting mouse menus and operations working

This actually turned out to mainly fixing some of the code for GTK-3 differences.

There is an issue to follow up on. D-Rats is not making sure that the CSV files are properly formatted by using a CSV library to read and write them.

Enable the remaining map menus

Now we just need to add the keyboard accelerators

This took a lot of digging and getting wrong answers.

It finally turns out that you have to add a new accel_group to the MapWindow as a first step. Then you have to use the "set_accels_for_action" method on the application object.

Not very complicated at all, but not something that is obvious to do.

And surprisingly, even though an Gtk.ApplicationWindow takes a Gtk.Application to instantiate it, I can not find a way to retrieve that value, so I needed to save it in the MapWindow class.

Add the Menu Accelerators

At this point the map code is complete. The MapSources and MapSourceEditor need a bit of work.

Now the battle is going to be on stability.

Stability issues

I have seen hangs and delays on lots of map activities, and have seen it crash with a segfault.

The main clue is the message in the log in the "more problems" section below.

It looks like we may have parallel operations that are conflicting with each other.

And I notice that you can start a new tile download event while an existing tile download event is running by either changing the center or the zoom.

Investigation showed that I was using the original code of calling render tile routine in thread started by the expose handler.

The current GTK documentation says that is not allowed, and no GTK calls can be made inside of a thread. The wording implies that that has always been the case, but there are calls for Thread use in the GTK related libraries. The current documentation indicates that they have either been deprecated or are not needed. I am probably confused on that point.

In addition, there can be up to 81 background threads for uploading tiles.

The solution is to set up a class variable to coordinate the threads so that they properly do the queue_draw call, and to also try to minimize the calls to queue_draw() by waiting until downloads have completed for a while.

It turns out that a class variable can only be updated by a class method.

Since doing this fix, I have not been able to trigger a hang or a segfault in the map code.

Fix Tile Threading

This will not fix the Gtk-Critical message below. The current thought is that a communication thread is using a GTK call to update the view with data instead of doing the update in the main loop.

More problems to investigate in other wikis:

Need to somehow signal or coordinate updates to the station list

01/31/2022 08:20:09:INFO:SessionManager:incoming: Received block 0:0 for session `chat'

(d-rats.py:31859): Gtk-CRITICAL **: 08:20:09.247: ../../../../gtk/gtktreeview.c:6680 (validate_visible_area): assertion `has_next' failed.
There is a disparity between the internal view of the GtkTreeView,
and the GtkTreeModel.  This generally means that the model has changed
without letting the view know.  Any display from now on is likely to
be incorrect.

01/31/2022 08:20:10:INFO:Transporter:_parse_gps: Found GPS string: bytearray(b'$$CRC157A,KG5CEN-1>APRATS,DSTAR*:/142008h3012.76N/08945.22W#/A=000002Ratflector\r

Configuration settings future needs

We have to clear up the what is a map provider and a map source provider. A map source provider provides points of interest, and a map provider provides maps.

The running configuration needs to be a combination of these:

• The local user configuration
• The default configuration values in the d-rats repository
• An updated configuration from the future d-rats project.

This is because there is a lot of information like ratflectors and map data providers that are subject to change and it is a lot of work to manually type them into configuration.

The map providers should be an dict in the configuration. It appears that a map provider needs the following parameters:

• Map provider URL
• API key, set to None if not needed, set to a constant text is required to be added
• Friendly name for the Map provider
• Tilesize of a map tile
• zoom range?
• Default zoom level
• Required attribution for for the data provider()

Map Source Editor

Will probably move this to another WIKI topic in the future.

The map source editor needed a number of changes just to get the editor to launch. Once those were done, the problem is that the usgs.gov has changed how to request their data for download. I am still experimenting with that.

It appears that we will have to use the latitudes and longitudes of two corners of the map to specify what site codes that you want.

And then the data comes with a Unicode tag in the XML and lxml complains about that.