Troubleshooting - Gapminder/waffle-server GitHub Wiki

Table of Contents

• Try the latest stable version of software
  • Nodejs (runtime development environment)
  • Npm (package manager)
  • Redis (server, db)
  • MongoDB (db)
  • Git (version control system)
  • New Relic license key (optional)
• Common Errors
  • Authentication failed during git cloning process
  • Cannot find module
  • Not starting without New Relic license key
  • Listen EADDRINUSE ::: <port>
  • Redis connection to <host>:<port> failed
  • MongoDB
  • DEFAULT_USER_PASSWORD was not provided
  • Could not read from remote repository
  • WS returns error in response when accessing data from private repo
  • Waffle Server hangs on Import/Update dataset process
  • WS has failed on AWS
  • Other

Many common problems can be solved with one of these steps:

Node 6.9.x is supported. Don't use unstable versions. Installation/Updating/Upgrading processes can be found here.

Node 3.10.x is supported. Don't use unstable versions. Installation/Updating/Upgrading processes can be found here.

Redis server 3.2.x is supported. Installation/Updating/Upgrading processes can be found here.

MongoDB server 3.4.x is supported. Installation/Updating/Upgrading processes can be found here.

Only git version 2.7.4 is supported. Installation/Updating/Upgrading processes can be found here.

If there is a need to get an application performance monitoring statistics, please, generate license key according to the detailed instruction and use it as value for environment variable in Waffle Server setup.

Important: New Relic key should be generated inside the waffle-server folder. License key must be named dev.

remote: Invalid username or password.
fatal: Authentication failed for '***************'

Solution:

Generate and add ssh key to the repo. The detailed procedure is described here and here.

Error: Cannot find module '<module name>'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (./waffle-server/server.js:2:1)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

Solution:

Run the following command inside repo directory (e.g. waffle-server):

npm i

New Relic for Node.js halted startup due to an error:
Error: Not starting without license key!

Solution:

Create an environment variable named NEWRELIC_KEY:

export NEWRELIC_KEY="**********************"

Please, note, that variable should exist at the moment of Waffle Server start. Its value is obtainable from New Relic website. If there is no need in Waffle Server performance statistics - just leave this error as is - it will not affect Waffle Server stability in any way.

Error: listen EADDRINUSE :::8081
        at Object.exports._errnoException (util.js:1022:11)
        at exports._exceptionWithHostPort (util.js:1045:20)
        at Server._listen2 (net.js:1259:14)
        at Server.wrappedListen2 [as _listen2] (./waffle-server/node_modules/newrelic/lib/instrumentation/core/net.js:16:46)
        at listen (net.js:1295:10)
        at Server.listen (net.js:1391:5)
        at EventEmitter.listen (./waffle-server/node_modules/express/lib/application.js:617:24)
        at Object.<anonymous> (./waffle-server/server.ts:31:5)
        at Module._compile (module.js:570:32)
        at Module.m._compile (./waffle-server/node_modules/ts-node/src/index.ts:406:23)
        at Module._extensions..js (module.js:579:10)
        at Object.require.extensions.(anonymous function) [as .ts] (./waffle-server/node_modules/ts-node/src/index.ts:409:12)
        at Module.load (module.js:487:32)
        at tryModuleLoad (module.js:446:12)
        at Function.Module._load (module.js:438:3)
        at Function.Module.runMain (module.js:604:10)

It means if there is already Waffle Server instance running, it is necessary to close/kill it and try again. Read on how to kill any process on any OS here

Redis connection to <host>:<port> failed - connect ECONNREFUSED <host>:<port>

It means that Redis-server service is not running. Read on how to run it on desired OS here and here

db connect error { MongoError: failed to connect to server [<host>:<port>] on first connect
    at Pool.<anonymous> (./waffle-server/node_modules/mongodb-core/lib/topologies/server.js:326:35)
    at emitOne (events.js:96:13)
    at Pool.emit (events.js:188:7)
    at Connection.<anonymous> (./waffle-server/node_modules/mongodb-core/lib/connection/pool.js:272:12)
    at Connection.g (events.js:291:16)
    at emitTwo (events.js:106:13)
    at Connection.emit (events.js:191:7)
    at Socket.<anonymous> (./waffle-server/node_modules/mongodb-core/lib/connection/connection.js:175:49)
    at Socket.g (events.js:291:16)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1278:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js:122:9)
  name: 'MongoError',
  message: 'failed to connect to server [localhost:27017] on first connect' }

It means that MongoDB service is not running. Read on how to run it on desired OS here and here

DEFAULT_USER_PASSWORD was not provided

Solution:

Add new variable DEFAULT_USER_PASSWORD to local environment variables with desired value. It should look as follows:

export DEFAULT_USER_PASSWORD="ENTER_YOUR_NEW_PASSWORD_HERE"

Cloning into '.../ddf/Gapminder/ddf--pcbs--census/features/flattened'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Solution:

Add github account gapminderdeploy to list of collaborators of your private repo

You are not allowed to access data according to given query

Make sure you've added dataset_access_token in reader config. The token is generated for a certain dataset in WS CLI. Read more here

* ERROR:  Please, change your WS-CLI version from x.x.x to y.y.y

Solution:

Notice that you will need to redeploy WS with new version of WS-CLI. It means that you need to redeploy WS every time when version of WS-CLI was changed.

Currently importing/updating dataset is possible only via CLI tool. If import or update processes takes more time than total approximate time, perform following steps:

1. Restart CLI tool (close the CLI tool and run it again, using correct credentials).
2. Try to check state of import/update process for certain dataset/version.
3. Try to rollback if the last version of dataset was corrupted.

To restart the WS, access AWS console (EC2 instances), find there a machine with a name that follows next pattern - ${DEVSTACKNAME}-wsNode and terminate it (right-click on the node and choose in the menu). In couple minutes new machine will be restarted by the cluster. Important: you should know the ${DEVSTACKNAME} to accomplish this. By convention, all the dev stack names start with a wsdevstack.

• Some strange issues can be resolved by simply running npm cache clean and trying again.
• If there is a trouble with npm install, use the -verbose option to have a more detailed error log.
• In dataset version is outdated, try to clean Redis cache using following command: redis-cli FLUSHALL && redis-cli FLUSHDB

See https://github.com/Gapminder/waffle-server/issues for more information about unusual problems.