Building vim from source with Anaconda - lmmx/devnotes GitHub Wiki

• Update (August 5th 2019): I reinstalled Linux to a newer distro version which comes with Python 3.6, then installed python3-pip from apt which allowed me to add black to the system installation of Python, which vim (vim-nox) could successfully use! So no need to build vim from source now.

Summary - this page is an account of how I went about building vim from source with Anaconda rather than conventional 'system' Python (...very carefully!) There's no doubt too much information here, but given how important the text editor is to me, I wanted to be totally certain of everything I was doing, and not mess up the installation. Also if I come to do this again in the future, or for anyone else who wants to give it a go, the more detail given the more certain I/they will be about doing so.

The step by step instructions are summarised at the bottom, in the section titled 'Reinstallation'

N.B. - outcome so far is unsuccessful, proceed with care etc. etc. Details shared here in case it's of use to others trying to use Black with Vim.

• The final situation is the same as that described here
  • Long story short, it's convinced me that it's time to update my operating system to the latest release, where vim apparently comes with Python 3 support out of the box.

I successfully installed Black, the Python code autoformatter, and found that a requirement for it to work inside vim (which is a nice feature of vim-go) is that your installation of vim must be compiled with Python 3.6 or later.

• The error message received was a failure to build an f-string (which was of course introduced in 3.6):

  Error detected while processing /home/louis/.vim/vimplug/black/plugin/black.vim: 
  line  135:
    File "<string>", line 24
      return venv_path / 'lib' / f'python{pyver[0]}.{pyver[1]}' / 'site-packages'
                                                              ^
  SyntaxError: invalid syntax
  

I was nervous about messing up my vim installation, so first of all I backed up my ~/.vim/ directory to my dotfiles repo in case an installation somehow managed to wipe them (for the record it didn't, but better to be safe than sorry).

My first approach was to upgrade the standard vim, but at the time of writing this did not include Python 3 support.

sudo apt-get install --only-upgrade vim
vim --version

Next, I read that there was a version called vim-nox, which is described in the version info as "Huge version without GUI.", i.e. it's built with most of the possible features, most importantly of which are:

+python/dyn
+python3/dyn

i.e. it's built with both Python 2 and Python 3 supported (but apparently you can't run both in the same vim session, if you for some reason wanted to).

Importantly however (at the time of writing) this package does not seem to have been built with a version of Python later than 3.5, so it can't support Black. Oddly, the build info given doesn't give any idea of what version of Python it was built with:

Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -Wall -Wextra -g -O2 -fPIE -Wformat -Werror=format-security -fstack-protector-all -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: gcc   -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -L/build/buildd/ruby1.9.1-1.9.3.484/debian/lib -rdynamic -Wl,-export-dynamic -Wl,-E  -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,-z,nodlopen -Wl,-z,nodump -Wl,-z,noexecstack -Wl,-z,noexecheap -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux   -lacl -lattr -lgpm -ldl   -Wl,-E  -fstack-protector -L/usr/local/lib  -L/usr/lib/perl/5.18/CORE -lperl -ldl -lm -lpthread -lcrypt

Unlike the info given in this issue on the Black bug tracker, which [on the last line] shows it was built from /usr/lib/python3.5/:

Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -Wdate-time -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim -lm -ltinfo -lnsl -lselinux -lacl -lattr -lgpm -ldl -L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5m -lpthread -ldl -lutil -lm

Since it didn't seem possible to do this 'the easy way' (i.e. the safest way) through the standard APT package listings, I tried to get versions of Python from the Deadsnakes PPA (as recommended on various Q&A pages, e.g. here):

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
sudo apt-get install python3.7

However there was no config directory underneath /usr/lib/python3.6 / /usr/lib/python3.7.

For comparison, my system Python 2.7 installation has a config directory at /usr/lib/python2.7/config-x86_64-linux-gnu/

I was following the guide here on building vim from source, and it does mention to "pay attention here check directory correct" when passing a value to the configure script with the --with-python-config-dir flag, so I looked at all files in the /usr/lib directory but neither the 3.6/3.7 versions had a config directory as far as I could tell.

I'm not going to reinstall it to check*, but now I think about it, there may have been python3.6-config and python3.7-config programs installed into /usr/lib which I could have used to provide the location of the config directories, but I didn't see them (though having said that, I didn't attempt to install them from the Deadsnakes PPA either).

• * simply because apt-get purge didn't remove these /usr/lib/python3.* directories when it 'uninstalled' the packages, so I was uncomfortable about reinstalling something that didn't uninstall cleanly.

Lastly, I am now trying to build from Anaconda, which is the source of python on my system PATH.

See:

• Vim installation and Anaconda

  ./configure --with-features=huge --enable-pythoninterp=dynamic --enable-python3interp=dynamic
  

• ...and compare to: Building Vim from source (the guide I mentioned in the previous section), which notably also specifies this flag:

  --with-python3-config-dir=/usr/lib/python3.5/config
  

The first guide also mentions being able to use these values on the config flags:

--with-python-config-dir=$(shell python-config --configdir)
--with-python3-config-dir=$(shell python3-config --configdir)  

(ignore the part about 'Debian-based systems', I think they just mean 'Debian-like' Linux installations [like Ubuntu-based distros], as compared to Windows)

Firstly, (going off the YCM guidance), install all the prerequisite libraries:

sudo apt install libncurses5-dev libgnome2-dev libgnomeui-dev \
libgtk2.0-dev libatk1.0-dev libbonoboui2-dev \
libcairo2-dev libx11-dev libxpm-dev libxt-dev python-dev \
python3-dev ruby-dev lua5.1 liblua5.1-dev libperl-dev git

• N.B. "On Ubuntu 16.04, liblua5.1-dev is the lua dev package name not lua5.1-dev."

Get all the information about vim so you know where to put it when you reinstall:

• which vim --> /usr/bin/vim
• ls -l $(which vim) --> /usr/bin/vim -> /etc/alternatives/vim
• ls -l /etc/alternatives/vim --> /etc/alternatives/vim -> /usr/bin/vim.nox

Now remove vim - I think it's better to use purge than remove:

sudo apt purge vim vim-runtime gvim vim-tiny vim-common vim-gui-common vim-nox

but remember that you can add --dry-run at the end of the line to preview what happens.

Also note that in order to prevent apt update from overwriting the compiled version, you have to 'put a hold' on vim (as described here):

sudo apt-mark hold vim

Then follow the official instructions to install vim from its GitHub repo, which for me would (without modifying for the configuration) be:

cd ~/opt/
git clone https://github.com/vim/vim.git
cd vim
git pull
cd src
make distclean
make
sudo make install

• The distclean line is annotated on the official vim instructions as "if you build Vim before", which I assume means that multiple installations will leave old versions of the files lying around (which could lead to the installer picking up and using some of the old config files rather than generating new ones?)

  • According to the GNU automake docs (in turn via this Q&A on the difference between clean and distclean):

    • if make built it, then clean should delete it.
    • if configure built it, then distclean should delete it.

• The configure script typically creates the Makefile and configure.h

  • config.h is generated from config.h.in by configure, config.h.in is generated from configure.ac by autoheader (there's a diagram of it over in the GNU manual)

• make distclean apparently runs something along the lines of rm -f *.in
  • This page mentions it running rm -f Makefile.in.in
• This page is supposed to explain "what gets cleaned" by the various options, but even the link to standard targets doesn't actually specify. It does say that:
  • clean won't delete the files that record the configuration
  • distclean will delete the files created by configuring the program (the name is due to it leaving "only the files that were in the distribution.

In short, it looks like if you run configure, you should use make clean not make distclean

• This is supported by this Q&A

Moving on...

One more note on the flags to configure, via:

Fedora uses --enable-python3interp=dynamic rather than --enable-python3interp. This misleadingly-named option causes Vim to dlopen() Python at runtime, rather than (dynamically) linking to it

dlopen is a Unix tool that handles dynamic loading.

• Successful use of these flags here but specifying the config dir

• Recalling this earlier example,

  --enable-pythoninterp=dynamic --with-python-config-dir=$(shell python-config --configdir)
  --enable-python3interp=dynamic --with-python3-config-dir=$(shell python3-config --configdir)
  

Note that shell is not a terminal command, but rather a Makefile build command! It is interpreted to mean "do this before executing anything", so basically you can just figure out the results of these commands first and pass them in instead.

For me this becomes:

--enable-pythoninterp=dynamic --with-python-config-dir=/usr/lib/python2.7/config-x86_64-linux-gnu
--enable-python3interp=dynamic --with-python3-config-dir=/home/louis/anaconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu

So to go back to the configuration, changing the distclean to clean (which I think is correct), and adding in the lines from the guide to building Vim from source mentioned earlier, with a couple of notes:

• There's no need to pull straight after clone
• Use sudo on the make install line if installing into system directories [rather than userland ones], as you surely will be for vim.
  • "By default, make install' will install all the files in /usr/local/bin', /usr/local/lib' etc. You can specify an installation prefix other than /usr/local' using --prefix', for instance --prefix=$HOME'. (from the configure help page)

Run ./configure --help to look through all the options:

• --enable-fail-if-missing is explained here (I took it out)

Here's how mine eventually ended up:

cd ~/opt/
git clone https://github.com/vim/vim.git
cd vim
./configure --with-features=huge \
            --enable-multibyte \
	    --enable-rubyinterp=yes \
	    --enable-pythoninterp=dynamic \
            --with-python-config-dir=/usr/lib/python2.7/config-x86_64-linux-gnu \
	    --enable-python3interp=dynamic \
            --with-python3-config-dir=/home/louis/anaconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu \
	    --enable-perlinterp=yes \
            --enable-cscope \
	    --prefix=/usr/local
cd src
make clean
make VIMRUNTIMEDIR=/usr/local/share/vim/vim81
sudo make install

• Note that --prefix=/usr/local is redundant, as mentioned above this is the default.
• I chose to use the VIMRUNTIMEDIR as it's mentioned in this Q&A about uninstalling vim compiled from source, and in the YCM guide it's apparently important "if you're not using vim 8.0" (the last line of the README gives the version, 8.1 in my case).

It's also mentioned in the YCM guide that "If you want to be able to easily uninstall vim use checkinstall" (but I prefer to stick to the standard tools, feel free to investigate that)

Now that all the details have been figured out, the first thing to do is uninstall vim.

sudo apt purge vim vim-runtime gvim vim-tiny vim-common vim-gui-common vim-nox

Get the prerequisites (via YCM guide):

sudo apt install libncurses5-dev libgnome2-dev libgnomeui-dev \
libgtk2.0-dev libatk1.0-dev libbonoboui2-dev \
libcairo2-dev libx11-dev libxpm-dev libxt-dev python-dev \
python3-dev ruby-dev lua5.1 liblua5.1-dev libperl-dev git

Now that vim is removed from the system, time to follow the steps figured out earlier:

cd ~/opt/
git clone https://github.com/vim/vim.git
cd vim
./configure --with-features=huge \
            --enable-multibyte \
	    --enable-rubyinterp=yes \
	    --enable-pythoninterp=dynamic \
            --with-python-config-dir=/usr/lib/python2.7/config-x86_64-linux-gnu \
	    --enable-python3interp=dynamic \
            --with-python3-config-dir=/home/louis/anaconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu \
	    --enable-perlinterp=yes \
            --enable-cscope \
	    --prefix=/usr/local
cd src
make clean
make VIMRUNTIMEDIR=/usr/local/share/vim/vim81
sudo make install

Now would be a good time to run vim --version and see if all went according to plan.

If the version info looks OK, try running vim to edit a file, and install a plugin (if you already installed plugins into your ~/.vim/ settings directory then opening a file with the new installation of vim will show whether the Python libraries load successfully).

If not, then to uninstall the new vim and try again, while still in the src directory, run:

make VIMRUNTIMEDIR=/usr/local/share/vim/vim81
sudo make uninstall

...and try installing again without the Python 2 libraries (as suggested might be necessary here) by removing the flags for --enable-pythoninterp=dynamic and --with-python-config-dir in the ./configure line above:

cd ~/opt/
git clone https://github.com/vim/vim.git
cd vim
./configure --with-features=huge \
            --enable-multibyte \
	    --enable-rubyinterp=yes \
	    --enable-python3interp=dynamic \
            --with-python3-config-dir=/home/louis/anaconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu \
	    --enable-perlinterp=yes \
            --enable-cscope \
	    --prefix=/usr/local
cd src
make clean
make VIMRUNTIMEDIR=/usr/local/share/vim/vim81
sudo make install

...made no difference.

Installing neovim resulted in a weird terminal glyph being inserted at the start of every line, and inside the neovim editor on all lines of the file edited.

The following is what I would have done if this was successful, for future reference (however I gave up on building vim with Python 3/Black support here and reinstalled vim.nox via apt-get)

sudo update-alternatives --install /usr/bin/editor editor /usr/local/bin/vim 1
sudo update-alternatives --set editor /usr/local/bin/vim
sudo update-alternatives --install /usr/bin/vi vi /usr/local/bin/vim 1
sudo update-alternatives --set vi /usr/local/bin/vim

sudo apt-mark hold vim

• vim installed correctly, as reported by vim --version, but when vim runs it gives error message:

  This script requires vim7.0+ with Python 3.6 support.

  Searching for this brought up this Reddit thread (mentioning the error is from Vundle trying to use the Black plugin), suggesting you put :py3 import os at the top of your .vimrc to see if the Python integration was working (or to just give up and use neovim)

• When I added a line at the top of my .vimrc to invoke Python ":python3 pass", another error message appeared:

  line 1:
  E370: Could not load library libpython3.7m.a
  E263: Sorry, this command is disabled, the Python library could not be loaded.

  so it seems that Python is now not loaded (rather than the situation before where the version was too early).

  • The same error message has been posted to StackOverflow and received no answer so far: https://stackoverflow.com/questions/53726779/vim-could-not-load-libpython3-7m-a-with-python3-from-anaconda

• There also seem to be some problems in moving around in the text editor, so I'll have to reinstall the APT version.