Unit tests on Mac - michaelkirk/QGIS GitHub Wiki
Proposal to stop losing ground
Currently 24 unit tests are failing in the MacOS.
The has been talked about somewhat in these threads: http://osgeo-org.1560.x6.nabble.com/Failing-tests-consider-blockers-td5103837.html#a5103891
A recurring comment is that we need to first "fix the tests" and then address the issue of not breaking them in the future. In the past, I've had good success reversing that strategy. That is - immediately install measures to make sure you're not breaking more tests, and then go back and gradually fix the broken tests as you work with reported bugs or feature related to that code.
So, specifically, I'd like travis to additionally run the test suite on mac os. To that end, I've configured and automated the build on a forked QGIS, and simply black listed the tests known to fail on mac. These tests are skipped on Mac only, and continue to run on linux.
The biggest risk I see here is that we're muting the noise from some broken tests. Another problem is that the build time for mac is significantly slower than on linux. But when I weigh that against the benefit of having automatic notification when a pull request breaks additional tests, I think this solution offers net benefit.
If y'all want to adopt this, an admin of github.com/QGIS/qgis will have to email the following:
To: [email protected]
Hello!
Per this page: http://docs.travis-ci.com/user/multi-os/ I'm requesting multi-os build support for https://github.com/qgis/QGIS
Currently Failing
On my Macbook Pro Retina running Yosemite, 81% tests passed, 24 tests failed out of 125.
- 16 - qgis_rendererstest (Failed)
- 18 - qgis_blendmodestest (Failed)
- 31 - qgis_composerutils (Failed)
- 38 - qgis_composermapgridtest (Failed)
- 40 - qgis_composereffectstest (Failed)
- 42 - qgis_composerrotationtest (Failed)
- 44 - qgis_atlascompositiontest (Failed)
- 47 - qgis_composertablev2test (OTHER_FAULT)
- 52 - qgis_composerscalebartest (Failed)
- 57 - qgis_rasterfilltest (Failed)
- 58 - qgis_shapebursttest (Failed)
- 59 - qgis_invertedpolygontest (Failed)
- 64 - qgis_legendrenderertest (Failed)
- 78 - qgis_mapcanvastest (Failed)
- 80 - qgis_openstreetmaptest (Failed)
- 85 - PyQgsLocalServer (Failed)
- 93 - PyQgsBlendModes (Failed)
- 108 - PyQgsComposerMapGrid (Failed)
- 109 - PyQgsComposerEffects (Failed)
- 113 - PyQgsAtlasComposition (Failed)
- 117 - PyQgsPalLabelingCanvas (Failed)
- 118 - PyQgsPalLabelingComposer (Failed)
- 119 - PyQgsPalLabelingServer (Failed)
- 121 - PyQgsSpatialiteProvider (Failed)
Output
(14:30:29) ± [[email protected]] ~/src/qgis mjk (master ✗)
→ ./run_tests.sh
Homebrew: /usr/local
DYLD_LIBRARY_PATH: /Users/mkirk/src/qgis-build/output/lib:/Users/mkirk/src/qgis-build/PlugIns/qgis
PATH: /usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin::/usr/X11/bin
PYTHONHOME: /usr/local/Frameworks/Python.framework/Versions/2.7
PYTHONPATH: /usr/local/lib/python2.7/site-packages
PYQGIS_STARTUP: /usr/local/Library/Taps/osgeo/homebrew-osgeo4mac/enviro/python_startup.py
GDAL_DRIVER_PATH: /usr/local/lib/gdalplugins
Running tests...
/usr/local/Cellar/cmake/3.1.0/bin/ctest --force-new-ctest-process
Test project /Users/mkirk/src/qgis-build
Start 1: qgis_clippertest
1/125 Test #1: qgis_clippertest ..................... Passed 0.72 sec
Start 2: qgis_distanceareatest
2/125 Test #2: qgis_distanceareatest ................ Passed 1.02 sec
Start 3: qgis_applicationtest
3/125 Test #3: qgis_applicationtest ................. Passed 0.86 sec
Start 4: qgis_diagramtest
4/125 Test #4: qgis_diagramtest ..................... Passed 1.20 sec
Start 5: qgis_diagramexpressiontest
5/125 Test #5: qgis_diagramexpressiontest ........... Passed 1.31 sec
Start 6: qgis_expressiontest
6/125 Test #6: qgis_expressiontest .................. Passed 1.15 sec
Start 7: qgis_filewritertest
7/125 Test #7: qgis_filewritertest .................. Passed 1.05 sec
Start 8: qgis_projecttest
8/125 Test #8: qgis_projecttest ..................... Passed 0.76 sec
Start 9: qgis_regression992
9/125 Test #9: qgis_regression992 ................... Passed 0.94 sec
Start 10: qgis_regression1141
10/125 Test #10: qgis_regression1141 .................. Passed 0.89 sec
Start 11: qgis_rasterlayertest
11/125 Test #11: qgis_rasterlayertest ................. Passed 1.19 sec
Start 12: qgis_rastersublayertest
12/125 Test #12: qgis_rastersublayertest .............. Passed 0.86 sec
Start 13: qgis_rasterfilewritertest
13/125 Test #13: qgis_rasterfilewritertest ............ Passed 1.22 sec
Start 14: qgis_contrastenhancementtest
14/125 Test #14: qgis_contrastenhancementtest ......... Passed 0.87 sec
Start 15: qgis_maplayertest
15/125 Test #15: qgis_maplayertest .................... Passed 1.16 sec
Start 16: qgis_rendererstest
16/125 Test #16: qgis_rendererstest ...................***Failed 1.21 sec
Start 17: qgis_maprenderertest
^B 17/125 Test #17: qgis_maprenderertest ................. Passed 3.04 sec
Start 18: qgis_blendmodestest
^R
18/125 Test #18: qgis_blendmodestest ..................***Failed 1.30 sec
Start 19: qgis_geometrytest
19/125 Test #19: qgis_geometrytest .................... Passed 0.95 sec
Start 20: qgis_geometryimporttest
4 20/125 Test #20: qgis_geometryimporttest .............. Passed 0.72 sec
Start 21: qgis_coordinatereferencesystemtest
21/125 Test #21: qgis_coordinatereferencesystemtest ... Passed 1.03 sec
Start 22: qgis_coordinatetransformtest
22/125 Test #22: qgis_coordinatetransformtest ......... Passed 0.90 sec
Start 23: qgis_pointtest
23/125 Test #23: qgis_pointtest ....................... Passed 0.75 sec
Start 24: qgis_vectordataprovidertest
24/125 Test #24: qgis_vectordataprovidertest .......... Passed 1.02 sec
Start 25: qgis_vectorlayertest
25/125 Test #25: qgis_vectorlayertest ................. Passed 0.92 sec
Start 26: qgis_rulebasedrenderertest
26/125 Test #26: qgis_rulebasedrenderertest ........... Passed 0.89 sec
Start 27: qgis_ziplayertest
27/125 Test #27: qgis_ziplayertest .................... Passed 3.96 sec
Start 28: qgis_dataitemtest
28/125 Test #28: qgis_dataitemtest .................... Passed 1.13 sec
Start 29: qgis_datadefined
29/125 Test #29: qgis_datadefined ..................... Passed 0.77 sec
Start 30: qgis_composerobject
30/125 Test #30: qgis_composerobject .................. Passed 0.77 sec
Start 31: qgis_composerutils
31/125 Test #31: qgis_composerutils ...................***Failed 1.63 sec
Start 32: qgis_compositiontest
32/125 Test #32: qgis_compositiontest ................. Passed 0.91 sec
Start 33: qgis_composermodel
33/125 Test #33: qgis_composermodel ................... Passed 0.85 sec
Start 34: qgis_composermultiframetest
34/125 Test #34: qgis_composermultiframetest .......... Passed 0.85 sec
Start 35: qgis_composergrouptest
35/125 Test #35: qgis_composergrouptest ............... Passed 0.75 sec
Start 36: qgis_composerpapertest
36/125 Test #36: qgis_composerpapertest ............... Passed 1.60 sec
Start 37: qgis_composermaptest
l 37/125 Test #37: qgis_composermaptest ................. Passed 1.14 sec
Start 38: qgis_composermapgridtest
^B22 38/125 Test #38: qgis_composermapgridtest .............***Failed 5.90 sec
Start 39: qgis_composermapoverviewtest
39/125 Test #39: qgis_composermapoverviewtest ......... Passed 2.54 sec
Start 40: qgis_composereffectstest
40/125 Test #40: qgis_composereffectstest .............***Failed 1.46 sec
Start 41: qgis_composershapestest
41/125 Test #41: qgis_composershapestest .............. Passed 1.78 sec
Start 42: qgis_composerrotationtest
42/125 Test #42: qgis_composerrotationtest ............***Failed 2.67 sec
Start 43: qgis_composerpicturetest
43/125 Test #43: qgis_composerpicturetest ............. Passed 5.27 sec
Start 44: qgis_atlascompositiontest
44/125 Test #44: qgis_atlascompositiontest ............***Failed 4.25 sec
Start 45: qgis_composerlabeltest
45/125 Test #45: qgis_composerlabeltest ............... Passed 0.98 sec
Start 46: qgis_composertabletest
46/125 Test #46: qgis_composertabletest ............... Passed 0.98 sec
Start 47: qgis_composertablev2test
47/125 Test #47: qgis_composertablev2test .............***Exception: Other 6.74 sec
Start 48: qgis_composerddtest
48/125 Test #48: qgis_composerddtest .................. Passed 1.07 sec
Start 49: qgis_stylev2test
49/125 Test #49: qgis_stylev2test ..................... Passed 1.33 sec
Start 50: qgis_composerhtmltest
50/125 Test #50: qgis_composerhtmltest ................ Passed 3.78 sec
Start 51: qgis_rectangletest
51/125 Test #51: qgis_rectangletest ................... Passed 0.87 sec
Start 52: qgis_composerscalebartest
52/125 Test #52: qgis_composerscalebartest ............***Failed 7.59 sec
Start 53: qgis_ogcutilstest
53/125 Test #53: qgis_ogcutilstest .................... Passed 0.73 sec
Start 54: qgis_vectorlayercachetest
54/125 Test #54: qgis_vectorlayercachetest ............ Passed 0.97 sec
Start 55: qgis_spatialindextest
55/125 Test #55: qgis_spatialindextest ................ Passed 3.60 sec
Start 56: qgis_gradienttest
56/125 Test #56: qgis_gradienttest .................... Passed 1.49 sec
Start 57: qgis_rasterfilltest
57/125 Test #57: qgis_rasterfilltest ..................***Failed 1.40 sec
Start 58: qgis_shapebursttest
58/125 Test #58: qgis_shapebursttest ..................***Failed 1.52 sec
Start 59: qgis_invertedpolygontest
59/125 Test #59: qgis_invertedpolygontest .............***Failed 1.19 sec
Start 60: qgis_colorschemeregistry
60/125 Test #60: qgis_colorschemeregistry ............. Passed 0.71 sec
Start 61: qgis_colorscheme
61/125 Test #61: qgis_colorscheme ..................... Passed 1.00 sec
Start 62: qgis_maptopixeltest
62/125 Test #62: qgis_maptopixeltest .................. Passed 0.95 sec
Start 63: qgis_networkcontentfetcher
63/125 Test #63: qgis_networkcontentfetcher ........... Passed 5.43 sec
Start 64: qgis_legendrenderertest
64/125 Test #64: qgis_legendrenderertest ..............***Failed 1.93 sec
File "/var/folders/sz/h155z7wj3p5553hb_tf72sjr0000gn/T//legend_long_symbol_text_result_diff.png" not found.
File "/var/folders/sz/h155z7wj3p5553hb_tf72sjr0000gn/T//legend_three_columns_result_diff.png" not found.
Start 65: qgis_vectorlayerjoinbuffer
65/125 Test #65: qgis_vectorlayerjoinbuffer ........... Passed 0.93 sec
Start 66: qgis_maplayerstylemanager
66/125 Test #66: qgis_maplayerstylemanager ............ Passed 0.94 sec
Start 67: qgis_pointlocatortest
67/125 Test #67: qgis_pointlocatortest ................ Passed 0.86 sec
Start 68: qgis_snappingutilstest
68/125 Test #68: qgis_snappingutilstest ............... Passed 0.88 sec
Start 69: qgis_imageoperationtest
69/125 Test #69: qgis_imageoperationtest .............. Passed 4.16 sec
Start 70: qgis_zoomtest
70/125 Test #70: qgis_zoomtest ........................ Passed 0.86 sec
Start 71: qgis_projectionissues
71/125 Test #71: qgis_projectionissues ................ Passed 0.96 sec
Start 72: qgis_qgsguitest
72/125 Test #72: qgis_qgsguitest ...................... Passed 0.78 sec
Start 73: qgis_scalecombobox
73/125 Test #73: qgis_scalecombobox ................... Passed 1.14 sec
Start 74: qgis_dualviewtest
74/125 Test #74: qgis_dualviewtest .................... Passed 1.19 sec
Start 75: qgis_doublespinbox
75/125 Test #75: qgis_doublespinbox ................... Passed 0.91 sec
Start 76: qgis_spinbox
76/125 Test #76: qgis_spinbox ......................... Passed 0.80 sec
Start 77: qgis_rubberbandtest
77/125 Test #77: qgis_rubberbandtest .................. Passed 0.93 sec
Start 78: qgis_mapcanvastest
78/125 Test #78: qgis_mapcanvastest ...................***Failed 0.75 sec
Start 79: qgis_analyzertest
79/125 Test #79: qgis_analyzertest .................... Passed 0.94 sec
Start 80: qgis_openstreetmaptest
80/125 Test #80: qgis_openstreetmaptest ...............***Failed 2.30 sec
Start 81: qgis_zonalstatisticstest
81/125 Test #81: qgis_zonalstatisticstest ............. Passed 0.95 sec
Start 82: qgis_wcsprovidertest
82/125 Test #82: qgis_wcsprovidertest ................. Passed 14.02 sec
Start 83: qgis_qgisappclipboard
83/125 Test #83: qgis_qgisappclipboard ................ Passed 1.20 sec
Start 84: PyQgsApplication
84/125 Test #84: PyQgsApplication ..................... Passed 1.16 sec
Start 85: PyQgsLocalServer
85/125 Test #85: PyQgsLocalServer .....................***Failed 1.13 sec
Start 86: PyQgsFontUtils
86/125 Test #86: PyQgsFontUtils ....................... Passed 3.86 sec
Start 87: PyQgsFeature
87/125 Test #87: PyQgsFeature ......................... Passed 1.03 sec
Start 88: PyQgsFeatureIterator
88/125 Test #88: PyQgsFeatureIterator ................. Passed 1.02 sec
Start 89: PyQgsGeometryTest
89/125 Test #89: PyQgsGeometryTest .................... Passed 1.12 sec
Start 90: PyQgsGeometryAvoidIntersections
90/125 Test #90: PyQgsGeometryAvoidIntersections ...... Passed 0.99 sec
Start 91: PyQgsVectorLayer
91/125 Test #91: PyQgsVectorLayer ..................... Passed 1.05 sec
Start 92: PyQgsRasterLayer
92/125 Test #92: PyQgsRasterLayer ..................... Passed 1.19 sec
Start 93: PyQgsBlendModes
93/125 Test #93: PyQgsBlendModes ......................***Failed 1.73 sec
Start 94: PyQgsRasterFileWriter
94/125 Test #94: PyQgsRasterFileWriter ................ Passed 1.74 sec
Start 95: PyQgsMemoryProvider
95/125 Test #95: PyQgsMemoryProvider .................. Passed 1.17 sec
Start 96: PyQgsDelimitedTextProvider
96/125 Test #96: PyQgsDelimitedTextProvider ........... Passed 5.64 sec
Start 97: PyQgsLogger
97/125 Test #97: PyQgsLogger .......................... Passed 0.18 sec
Start 98: PyQgsCoordinateTransform
98/125 Test #98: PyQgsCoordinateTransform ............. Passed 1.01 sec
Start 99: PyQgsRectangle
99/125 Test #99: PyQgsRectangle ....................... Passed 1.02 sec
Start 100: PyQgsRelation
100/125 Test #100: PyQgsRelation ........................ Passed 1.06 sec
Start 101: PyQgsSpatialIndex
101/125 Test #101: PyQgsSpatialIndex .................... Passed 0.99 sec
Start 102: PyQgsComposerHtml
102/125 Test #102: PyQgsComposerHtml .................... Passed 2.02 sec
Start 103: PyQgsComposition
103/125 Test #103: PyQgsComposition ..................... Passed 2.47 sec
Start 104: PyQgsAnalysis
104/125 Test #104: PyQgsAnalysis ........................ Passed 1.01 sec
Start 105: PyQgsColorScheme
105/125 Test #105: PyQgsColorScheme ..................... Passed 0.18 sec
Start 106: PyQgsColorSchemeRegistry
106/125 Test #106: PyQgsColorSchemeRegistry ............. Passed 0.18 sec
Start 107: PyQgsComposerMap
107/125 Test #107: PyQgsComposerMap ..................... Passed 2.38 sec
Start 108: PyQgsComposerMapGrid
108/125 Test #108: PyQgsComposerMapGrid .................***Failed 3.20 sec
Start 109: PyQgsComposerEffects
109/125 Test #109: PyQgsComposerEffects .................***Failed 1.37 sec
Start 110: PyQgsComposerShapes
110/125 Test #110: PyQgsComposerShapes .................. Passed 1.75 sec
Start 111: PyQgsSymbolLayerV2
111/125 Test #111: PyQgsSymbolLayerV2 ................... Passed 1.21 sec
Start 112: PyQgsPoint
112/125 Test #112: PyQgsPoint ........................... Passed 1.12 sec
Start 113: PyQgsAtlasComposition
113/125 Test #113: PyQgsAtlasComposition ................***Failed 2.04 sec
Start 114: PyQgsComposerLabel
114/125 Test #114: PyQgsComposerLabel ................... Passed 1.00 sec
Start 115: PyQgsExpression
115/125 Test #115: PyQgsExpression ...................... Passed 0.19 sec
Start 116: PyQgsPalLabelingBase
116/125 Test #116: PyQgsPalLabelingBase ................. Passed 2.29 sec
Start 117: PyQgsPalLabelingCanvas
117/125 Test #117: PyQgsPalLabelingCanvas ...............***Failed 2.67 sec
Start 118: PyQgsPalLabelingComposer
118/125 Test #118: PyQgsPalLabelingComposer .............***Failed 2.26 sec
Start 119: PyQgsPalLabelingServer
119/125 Test #119: PyQgsPalLabelingServer ...............***Failed 2.75 sec
Start 120: PyQgsVectorFileWriter
120/125 Test #120: PyQgsVectorFileWriter ................ Passed 1.05 sec
Start 121: PyQgsSpatialiteProvider
121/125 Test #121: PyQgsSpatialiteProvider ..............***Failed 1.25 sec
Start 122: PyQgsZonalStatistics
122/125 Test #122: PyQgsZonalStatistics ................. Passed 1.00 sec
Start 123: PyQgsAppStartup
123/125 Test #123: PyQgsAppStartup ...................... Passed 28.56 sec
Start 124: PyQgsDistanceArea
124/125 Test #124: PyQgsDistanceArea .................... Passed 1.07 sec
Start 125: PyQgsGraduatedSymbolRendererV2
125/125 Test #125: PyQgsGraduatedSymbolRendererV2 ....... Passed 1.11 sec
81% tests passed, 24 tests failed out of 125
Total Test time (real) = 242.17 sec
The following tests FAILED:
16 - qgis_rendererstest (Failed)
18 - qgis_blendmodestest (Failed)
31 - qgis_composerutils (Failed)
38 - qgis_composermapgridtest (Failed)
40 - qgis_composereffectstest (Failed)
42 - qgis_composerrotationtest (Failed)
44 - qgis_atlascompositiontest (Failed)
47 - qgis_composertablev2test (OTHER_FAULT)
52 - qgis_composerscalebartest (Failed)
57 - qgis_rasterfilltest (Failed)
58 - qgis_shapebursttest (Failed)
59 - qgis_invertedpolygontest (Failed)
64 - qgis_legendrenderertest (Failed)
78 - qgis_mapcanvastest (Failed)
80 - qgis_openstreetmaptest (Failed)
85 - PyQgsLocalServer (Failed)
93 - PyQgsBlendModes (Failed)
108 - PyQgsComposerMapGrid (Failed)
109 - PyQgsComposerEffects (Failed)
113 - PyQgsAtlasComposition (Failed)
117 - PyQgsPalLabelingCanvas (Failed)
118 - PyQgsPalLabelingComposer (Failed)
119 - PyQgsPalLabelingServer (Failed)
121 - PyQgsSpatialiteProvider (Failed)
Errors while running CTest
make: *** [test] Error 8
Notes
QgsMultiRenderChecker
These tests all use the QgsMultiRenderChecker test class 16 - qgis_rendererstest (Failed) 18 - qgis_blendmodestest (Failed) 57 - qgis_rasterfilltest (Failed) 58 - qgis_shapebursttest (Failed) 59 - qgis_invertedpolygontest (Failed)
No tests using the QgsMultiRenderChecker are passing
16 - qgis_rendererstest (Failed)
In the test the extent is set to be: QgsRectangle extent( -118.8888888888887720, 22.8002070393376783, -83.3333333333331581, 46.8719806763287536 );
I am able to get a pretty close match by resetting the cavas extent canvas.setExtent(QgsRectangle( -165, 22, -20, 46 )) (145px wide)
The extent in the test suite is ~36 pixels wide. I was able to get canvas on screen to more closely match the expected result when I set the extent to be 145px wide