DirectoryStructure - coin-or-foundation/tlc GitHub Wiki

Removing include directory

Presently many of the COIN projects has an include for header files. The include directory is contained in the directory that contains the *.[c,cpp] files. As part of the move to subversion is has been suggested that the include directory be removed and the header files be moved to the directory that contains the *.[c,cpp] files. There is some concern that a change like this might impact COIN users in ways that the TLC may not imagine, so an email should be posted to COIN-discuss solicting input from the user community prior to making such a change.

Here is a draft of an email that might be posted to the COIN-discuss.

The COIN TLC board is considering moving the header files in the include to the directory containing the *.[c,cpp] files and removing the include directory. The affected directories are:

  • COIN/Alps/include
  • COIN/Bcp/include
  • COIN/Cbc/include
  • COIN/Cbc/include/Cbc/include
  • COIN/Cbc/Test/include
  • COIN/Cgl/include
  • COIN/Clp/include
  • COIN/Coin/include
  • COIN/Examples/Abc/include
  • COIN/Examples/BAC/include
  • COIN/Examples/BranchAndCut/include
  • COIN/Examples/Knap/include
  • COIN/Examples/MaxCut/include
  • COIN/Examples/Mkc/include
  • COIN/Osi/include
  • COIN/Osi/OsiCbc/include
  • COIN/Osi/OsiClp/include
  • COIN/Osi/OsiCpx/include
  • COIN/Osi/OsiDylp/include
  • COIN/Osi/OsiFmp/include
  • COIN/Osi/OsiGlpk/include
  • COIN/Osi/OsiMsk/include
  • COIN/Osi/OsiOsl/include
  • COIN/Osi/OsiSpx/include
  • COIN/Osi/OsiSym/include
  • COIN/Osi/OsiVol/include
  • COIN/Osi/OsiXpr/include
  • COIN/Smi/include
  • COIN/SYMPHONY/Applications/CNRP/include
  • COIN/SYMPHONY/Applications/MATCH/include
  • COIN/SYMPHONY/Applications/MPP/include
  • COIN/SYMPHONY/Applications/SPP/include
  • COIN/SYMPHONY/Applications/SPP+CUTS/include
  • COIN/SYMPHONY/Applications/USER/include
  • COIN/SYMPHONY/Applications/VRP/include
  • COIN/SYMPHONY/CNRP/include
  • COIN/SYMPHONY/include
  • COIN/SYMPHONY/MATCH/include
  • COIN/SYMPHONY/MPP/include
  • COIN/SYMPHONY/SPP/include
  • COIN/SYMPHONY/SPP+CUTS/include
  • COIN/SYMPHONY/Template/include
  • COIN/SYMPHONY/Vrp/include
  • COIN/Vol/include
  • Some of these directories shouldn't be in this list. Which directories should be removed?

The advantages of removing the include directory are:

  • Better positions COIN for using automake.
  • Allows one to grep for symbols in fewer directories.

The disadvantage are:

  • makefiles and build processes will need to be updated.
  • The COIN user community may have to make additional changes that the COIN TLC may not know about.

If you would be adversely affected by such a change please post your concerns to COIN-discuss as a response to this email.

Ted's Comments

For my taste, getting rid of the include directory is moving a bit in the wrong direction. My preference would be to move everything out of the project root directories, which look pretty messy as it is, and go with something like the following standard directory structure in the project root:

  • include
  • bin
  • lib
  • src
  • doc
  • web (project Web pages)
  • Makefile
  • README
  • LICENSE
  • AUTHORS There might be other subdirectories too, such as
  • data
  • examples and so on. Check out the structure of SYMPHONY, which is organized this way. It's pretty clean and easy to work with.

Kipp's Comments

  1. I most definitely think we should NOT ask people that want to use Microsoft Visual Studio to use cygwin. Why not use the make feature in MSYS/MINGW (which is FAR more lightweight and easier than cygwin)? See

http://www.mingw.org/msys.shtml

I do think the goal should be to make the makefiles as platform/compiler indepdent as possible. I don't think it is feasible to tell a MS Visual Studio user that they have to download cygwin if they wish to use the COIN makefile system.

  1. I agree with Ted's comments above. Rather than all the include files and source files together have separate src and include directories.

Leo's Comments

I also second Ted's structure. Personally, I find it less cluttered to have headers separate from source. But more importantly, I believe the separate include file is "more standard :P". I would suspect that automake can be adapted for that strucutre, since most of the projects that use automake have that structure.

To address Kipp's point on cygwin x Msvc. I think we agree that ultimately the goal is to have a msvc-friendly build process that doesn't require cygwin, and if possible doesn't even require mingw. From my conversation with Laci I believe he volunteered to get the cygwin structure working, so that is how this temporary step got its start. Laci, would you like to comment?

JP Comment

I would prefer to have the *.c, *.cpp, *h, & *.hpp files in the src directory.

Kipp Again

Since we are moving to a new system should we discuss and plan a new build system? See:

http://projects.coin-or.org/CoinTLC/wiki/CoinBuildSystem

Lou

I agree with the principle of moving source out of the top level of a project. But to provide a bit of counterpoint, not all projects are best organised as a monolithic whole with the top level structure Ted suggests. Osi, for example, would not be improved by merging all the subsolvers with the core Osi code. In Cbc, the separation between core, test, and sample code is useful (to me, at least). Dylp maintains separate subtrees for the main solver, a utility library, and sample code.

Leo

Typical svn repositories have a top-level structure holding a directory for each project. Within each project, there are three directories, trunk, tags, and branches, for specific subversion use. Following the structure ted suggested above, the list include, bin, ... , AUTHORS would go inside the trunk directory. For details, see:

 http://svnbook.red-bean.com/nightly/en/svn.reposadmin.projects.html

This would force some changes to Makefiles and installation procedures. It is possible to deviate from this structure, but I am hesitant to do so because it seems to be the prevalent way of organizing the repository. Does anyone have a strong reason to deviate from this structure?

BTW, I haven't yet figured out how to make one project depend on another automatically. IOW, how to make a checkout of cbc automatically trigger a checkout of cgl. I assume this is easy to do and I just haven't run into yet. Does someone know? Also, the connection between svn and trac is not clear to me. Does someone know that too?

Andreas

Laci and I have looked at the externals feature of svn, and we have set up a local svn repository with the current COIN cvs repository at IBM. I have started to a) play with the externals, and b) use the GNU autotools (autoconf, automake, libtool) to build the COIN projects.

SVN externals:

One difficulty we currently face with COIN is the dependency of a project to a particular version of another one. We have very painfully experienced this during our work on the MINLP project (which we will contribute to COIN very soon), where one a frequent basis changes were made to Cbc, so that our code would not compile anymore, since the header files changed.

What subversion offers, is to automatically check out external code (from any other svn repository) into subdirectories. For example, it is possible to "tell" the Cbc project to automatically check out particular revisions of Osi, Clp, Coin etc into subdirectories (who's name one can choose). The great thing is that if the project manager decides that a newer revision of a dependency can be used, (s)he can just change the externals definition, and if someone, who has checked out that project, does a svn update, it automatically reverts to the new revision of the dependency. (Note that the project developer specifies whether something is from trunk, or a branch or anything; there would be not directory such as branch or trunk visible for someone who checks out the project).

This means that the directory layout in a checked out version of a COIN project will change in that there is no longer everything on the same directory level, but instead each project will have subdirectories with those projects that it depends on. Of course, this means that if someone checks out the most recent version of Clp and the most recent version of Cbc, there will be two copies of Coin. But I think this is unavoidable anyway, since otherwise consistency cannot be guaranteed.

Inside the svn repository itself, the world looks different; each project will be on the same directory level, and within each project, there will be the standard svn subdirectories trunk, branches, tags, under which the different versions of the source code for the project will be. But this is not exposed to users, and makefiles don't need to worry about this.

Autotools:

I have now a lot of experience with the GNU tools autoconf, automake, and since recently I started to use libtool. I have used them extensively for Ipopt, and I have now also converted the configuration and build process of our new MINLP project ("bonmin") to use this. I'm extremely happy with it, and feedback from users has been very positive.

Maybe I should briefly explain what those tools are:

  • autoconf is a tool to generate very general /bin/sh scripts ("configure") that can perform certain tests (e.g., whether a particular header file is there, whether a library can be used), and it can be given argument, for example, to tell it where the header files for Cplex and the Cplex library are located. (Everybody who as installed a GNU packages knows the configure script ;). It can also be used to determine the name of the compiler, and to set compiler options. In order to use autoconf, a project manager has to write an input file (configure.ac), which essentially contains macros (autoconf uses the preprocessor m4) that tell autoconf what tests are to be performed etc. After a user then runs the generated configure script, the script will replace placeholds (@STRING@) in certain files with the values it determines (e.g., the compiler name and options). Typically, an open source package working with configure does not provide the Makefiles directly, but it has Makefile.in files, which are then copied to Makefile and filled in with the values determined by configure. The configure.ac file can also contain /bin/sh commands, and it is further possible to define new macros, so that one can have a repository of tests that are not already available from autoconf, and those can then be used very easily.

  • automake is a tool to generate very powerful and portable makefiles (that work with any UNIX make). A project leader writes input files (Makefile.am) which essentially list what has to be build (e.g., a library or a program), what the source files for the project are, and possible what further libraries or include directories etc are required to compile the code in a given directory. It also works recursively. Given an input Makefile.am, automake creates a Makefile.in that works with the configure script generated by autoconf. This Makefile.in is very complicated, but a project developer or a user should never directly interfere with it.

The makefiles (Makefile.in) generated by automake have targets like 'all' (to just compile the stuff), 'install' (to install programs and libraries in user accessible locations), 'clean' and 'distclean'. Another nice feature of the autoconf/automake combination is that the project developer has the option to configure her/his local copy with --enable-maintainer-mode, which will automatically execute all the autotool commands necessary to incorporate changes to input files (configure.ac or Makefile.am). Therefore, one rarely has to run the tools "by hand" - most things are taken care of automatically. Further benefits are automatic dependency check (for header files, using whatever compiler is used, not necessary a GNU compiler), and VPATH compilation (i.e., you can compile the code in differnent directories than the source code; this is handy if you want an optimized version and a debug version of the code, or if you have a shared file system, where you keep only one copy of the source code, but can compile on different operating systems with different compilers...)

  • libtool finally is another tool that generates a script (in combination with autoconf and automake), that handles the creation and management of libraries. For example, it automatically compiles the code for static and shared libraries (which on some platforms is different, see the -fPIC flag for gcc), and uses the correct commands to create static libraries or shared objects/dlls on a large number of platforms (e.g., on MinGW it can create DLLs, on AIX it does the correct [complicated] thing to create a shared object). I have recently integrated it into Ipopt, and I'm very happy with it.

I think these are very powerful tools, and also Laci agrees with me that it would be great if we could use it for more COIN projects. This would make the build process very portable (one doesn't need GNU make to compile the code, one can write test that automatically determine how the CPU time can be determined, a user doesn't have to edit a Makefile, it will work in most cases also on platforms that the project manager hasn't compiled the code on, it works also with Microsoft compilers on Cygwin...)

As I wrote already, I have started on a local svn copy to convert the interconnected COIN projects (Cbc, Clp, Osi...) to both use subversion and autotools, to see if there are any hidden difficulties or problems. I think it would be great to start using it, because I believe that it would make the compilation of a COIN project much easier for a user. For a developer, there is a slight learning curve, but I would volunteer to do the initial conversion, and also to provide a number of new autoconf macros in a repository that do the required tests, and also to write new tests if someone needs them. Also, I would help a project manager to get started, debug autotool configuration etc. And my guess is that once a project leader got used to the new tools, (s)he will also be happy, because they are very powerful, and make life and maintainence quite easy.

Finally, a comment regarding the directory structure: Usually, if you get a GNU package (or some other open source package), you get the source code, and source and header files are in the same directory (which to me makes sense, also because it makes a "grep" for a particular method much easier.) I don't see a reason for keeping the header files required for the compilation separately. After you compiled the code, you typically do a 'make install' which copies the "product" of the compilation (a program and/or a library, and all header files that are required by a user of a library) into bin, lib, and include, respectively. Usually, it is not required for a user to have ALL header files available that are contained in the open source package, in order to use the library, i.e., only a subset of the header files are copied. Typically, once you have installed the package, you can delete the source code and all object files, since everything that you need as a user of the program or library, is now in bin, lib and include. (It is very easy to tell automake which header files are to be installed.)

I very strongly feel that this would be a good approach for the new layout of the COIN project, and in a sense we can interrelate the change to subversion to a change in the build process. Since we decided to ensure consistency for the code of a given project with the projects it depends on (using svn:externals with specific revision numbers), the directory structure has to change anyway (e.g., in the way Laci and I propose above), so that current COIN users and developers have to get used to a change. If you agree that the usage of autotools is a good idea, it is my opinion to try to do that change at the same time, to avoid confronting users and developers with two big changes at different points in time.

Ok, one last comment regarding compilation on Windows: I think we should offer people both the option to use project/solution files in Developer Studio (I talked to JP about the directory layout above, and he agrees that it would be no problem to use it with Developer Studio), as well as the option to compile the code on Cygwin or MinGW with either GNU or Microsoft/Intel compilers. This is what I do in the Ipopt project. (One thing I have not yet figured out is how to compile a DLL with the GNU compilers in MinGW that can then be used from the Developer Studio... static libraries in principle work, but somehow I can't obtain a proper import library yet for a DLL...)

If you would like to see an example of the autotools, please have a look at the Ipopt project. I have not yet committed my recent changes (to include libtool) to the trunk of the Ipopt project, but you can get it from https://www.coin-or.org/svn/Ipopt/branches/dev . I have to admit, though, that I didn't clean the configure.ac file up yet, but I plan to put most of the tests into the new autoconf macro file (coin.m4), so that once those tests have been written, the final configure.ac file is quite easy to read.

If you want to get more information about the GNU tools, you can look at

http://sources.redhat.com/autobook/ (this gives a picture of how everything works together, but it is a little old)

http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf.html (for a pretty recent version of autoconf)

http://www.gnu.org/software/automake/manual/html_node/index.html (for automake)

http://www.gnu.org/software/libtool/manual.html (for libtool)

Laci & Andreas

Suggestions after conferring with JJF:

  • It'd be a good idea to move to autotools, since they make life easier/cleaner and also work under CYGWIN/MINGW on Windows.
  • Move to svn should happen at the same time as the switch to autoconf. The way to do it would be to create the SVN repository and simultaneosly maintain the CVS and SVN repos until the (unavoidable) problems with the SVN repo and the new build process are cleared out. Only bugfixes would go to CVS. This dual life would last a few weeks at the most. The proposed time for the switchover is right after MIP2006, that is starting 06/09/2006.
  • To my severe disappointment and with apologies to Kipp and the others who spent lots of time debugging my new Makefiles, they would be abandoned :-(. However, this time Andreas would do all the build process conversion! :-) The previous section describes the new build process, RE-READ IT!

Andreas

Suggsted new directory structure (initiated by JP):

As opposed to what we discussed on the phone (each project has a src subdirectory, as well as subdirectories with the names of the dependency projects which themselves have src under it), it might be clearer and easier to do the following:

Each project has a "main directory" (such as COIN-CBC - but maybe we want to consider a different prefix than COIN- since this mainly pertains to the interrelated COIN projects and not necessarily all of them?), under which we have subdirectories with the names of all dependencies (including the current project itself) that have the source code for each subprojecct. For example:

COIN-CBC ---> Coin (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> Test (with sources for Test)
          |
          --> Clp (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> Test (with sources for Test)
          |
          --> Cgl with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> Test (with sources for Test)
          |
          --> Osi (with sources including headers and a configure script and a Makefile)
          |    |
          |    --> OsiClp (with sources and Makefile)
          |    |
          |    --> OsiCbc (with sources and Makfile)
          |    |
          |    ... [other Osi subdirs]
          |    |
          |    --> Test
          |
          --> Cgl (with sources including headers and a configure script and a Makefile)
          |    |
          |    --> Test
          |
          --> Cbc (with sources including headers and a configure script and a Makefile)
          |    |
          |    --> Test
          |
          ---> Mps --> Samples
               |
               ------> Netlib

This means we have almost the same structure as in the current CVS, and for a "meta" project (like COIN-ALL) containing all projects (latest version), it would look the same as for a subset required to build one particular component.

Beside an easier directory structure, this will also allow us to use the svn:externals, which would be defined for the main directory of a component (above for COIN-CBC). The problem of unwanted recursions would not appear, because the component source subdirectories (Coin, Clp,...) would be defined as externals, and not their "main" directory. This means we don't have to rely on scripts running svn for each component, and it would also work without scripts on Windows (which was JP's concern).

Ted

I'm not clear on exactly what's being proposed, but this is what I understand:

  • When checking out a project (let's call it the "primary project"), the resulting directory structure itself will be as illustrated above. The main difference between what we talked about at the last meeting and the above is that we will create an ubmrella directory, inside which the primary project and all dependent projects will exist at the same level. The net effect, as I understand it, is that the directory structure will be almost exactly what we have now, but only the relevant subdirectories will be checked out. In other words, checking out a primary project in svn will have the same results as checking out the project module from CVS.

  • The way this will be accomplished is a little less clear to me. My guess is that we will create a "meta-project" correspondng to each primary project and that these "meta-projects" will mimic the modules we currently have in CVS. Each meta-project will be essentially an empty shell, except that it will depend, through svn:external declarations, on the stable version of the primary project itself and specified versions of any other dependent projects. In this way, we can create "modules" that will allow us to check out any particular subset of the current projects into a directory structure like we currently have. This could include a module that would check out the latest versions of all the projects (the aforementioned COIN-ALL) and would recreate exactly what we have now.

  • It will still be necessary to create a new umbrella directory for each primary project, containing the entire source tree for the spcified version of each dependent project. So we will still have duplication, but this presumably cannot be avoided and developers will still have the option to recreate the current setup if desired.

Is my understanding of the current situation correct? I think I agree with this basic idea, but this still doesn;t absolve us of thinking about whether we should change the recommended "standard" directory structure while we're at it. I am still strongly in favor of suggesting a directory structure following something like what GNU uses, i.e., no files except for the README, AUTHOR, etc. in the project root directory, just subdirectories such as

  • include
  • bin
  • lib
  • src
  • doc
  • web (project Web pages)
  • Makefile
  • README
  • LICENSE
  • AUTHORS There might be other subdirectories too, such as
  • data
  • examples and so on. GNU also puts the config script in the root directory, which is fine. I just think we need to do some housecleaning.

Andreas

Ted's description in the bullets is essentially correcty. But I don't think the suggested default structure will quite work yet.

First a side comment: The include, bin, and lib directories are not really part of a package, and in the case of a GNU package, those are usually in different locations, such as /usr/local. Using /usr/local, however, is in our situation probably not necessarily a good idea, since it might result in conflicting libraries, if users install different COIN projects, which rely on different versions of certain packages. Therefore, I think, leaving bin etc in the main directory of the project by default is better.

We shouldn't have to create new projects (from the svn point of view) to be able to use externals. Instead, I suggest that the trunk (or a branches subdirectory) gets the svn:externals. Something we could do is this:

Clp/trunk ----> Clp    (contains sources, a configure script, as well as AUTHORS, LICENCE etc files)
           |      |    (this is also a subdirectory in svn for Clp/trunk)
           |      |
           |      ---> doc (contains documentation for Clp, also in the Clp/trunk svn subdirectory)
           |      |
           |      ---> Test (constains unitTest for Clp, also a Clp/trunk svn subdirectory)
           |
           ---> Coin    (contains sources, a configure script, as well as AUTHORS... files for Coin)
                  |     (this resides in a subdirectory of Coin/trunk,and is not tagged with
                  |      an svn:external)
                  |
                  ---> doc (contains documentation for Coin, in the Coin/trunk svn subdirectory)
                  |
                  ---> Test (constains unitTest for Coin, also a Coin/trunk svn subdirectory)

A user would check out Clp/trunk, and would obtain the subdirectories above, since Clp/trunk is tagged with svn:externals. Clp/trunk itself would contain just a simple configure script and Makefile, that recurse into the *Src subdirectories. When a user types 'make install', new subdirectories bin, lib, and include would be created, into which all binaries, libraries, and relevant headers files will be installed from ALL subprojects. If a user specified a different installation location (like /usr/local or $HOME) using the --prefix option for configure, it will be installed there instead.

The reason I think the AUTHORS etc files should be in the *Src directories is that those files should be available for all subpackages. I think the Tests and doc directories should also be available also for all subprojects, so I included them into the individual src directories.

I don't think we should have an 'src' subdirectory, since it would not be clear what it has the sources of. We could have trunk/src, which contains Clp and Coin subdirectories, but I think that adds an unnecessary level of directories, and since it would also include the doc and Test subdirectories, src is not an appropriate name.

Ted

Hmm, I guess I'm I'm not exactly clear on the mechanics of how this is going to work, but what I am suggesting is that when one checks out Cbc, for instance, the result look something like this

COIN-CBC ---> Coin (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> src
          |    |-> bin
          |    |-> lib
          |    |-> include (depending on whether we decide to put these into source) 
          |    |-> doc 
          |    |-> test
          |    |-> web 
          |    |-> examples
          |    |-> etc...
          |
          --> Clp (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> src
          |    |-> bin
          |    |-> lib
          |    |-> obj
          |    |-> dep
          |    |-> include (depending on whether we decide to put these into source) 
          |    |-> doc 
          |    |-> test
          |    |-> web 
          |    |-> examples
          |    |-> etc...
          |
          --> Cgl with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> src
          |    |-> bin
          |    |-> lib
          |    |-> obj
          |    |-> dep
          |    |-> include (depending on whether we decide to put these into source) 
          |    |-> doc 
          |    |-> test
          |    |-> web 
          |    |-> examples
          |    |-> etc...
          |
          --> Osi (with sources including headers and a configure script and a Makefile)
          |    |
          |    --> OsiClp (with sources and Makefile)
          |    |
          |    --> OsiCbc (with sources and Makfile)
          |    |
          |    ... [other Osi subdirs]
          |    |
          |    --> Test
          |
          --> Cgl (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> src
          |    |-> bin
          |    |-> lib
          |    |-> obj
          |    |-> dep
          |    |-> include (depending on whether we decide to put these into source) 
          |    |-> doc 
          |    |-> test
          |    |-> web 
          |    |-> examples
          |    |-> etc...
          |
          --> Cbc (with sources including headers and a configure script and a Makefile)
          |    |
          |    |-> src
          |    |-> bin
          |    |-> lib
          |    |-> obj
          |    |-> dep
          |    |-> include (depending on whether we decide to put these into source) 
          |    |-> doc 
          |    |-> test
          |    |-> web 
          |    |-> examples
          |    |-> etc...
          |
          ---> Mps --> Samples
               |
               ------> Netlib

In other words, each subdirectory in COIN-CBC should be organized like a GNU-style project unto itself. I guess you are thinking of the entire thing as one big GNU-style project, but I'm not sure we can/should look at it that way. You're correct that by the GNU paradigm, bin/ and lib/ do not really belong in the project subdirectories, but as you say, they probably should not get "installed" in /usr/local/lib to avoid conflicts. Putting them at the COIN-CBC level is probably the best solution and duplicates what we currently have. The bottom line, though, is that I personally don't really like the idea of having the sources for each project in its root directory, at a higher level than the documentation, test sources, etc. It's much nicer to have the root directory of each project completely clean except for a few files. I'd also like to dispense with the current paradigm whereby compilation takes place in a directory called $ARCH-$OPT. I'd prefer a directory called "obj" with subdirectories if needed. What do other people think?

Laci

I like Andreas' structure. The only comment I have is that headers must move into src. The reason is that when you do 'make install' in a subdir, say COIN-CBC/Clp, then if bin and lib are created in COIN-CBC/Clp to house exes and libs then surely include must be created as well to house the headers. But that excludes having the headers there in the first place.