Posted on

A trick to use numpy.loadtxt on strings

numpy offers a function, numpy.loadtxt, to read in an array from a file. This is a extremely useful function especially when a 2d array is being read. However the function only accepts either a file path/name, or a file generator (file handler). You can not use a string directly as the argument.

numpy also offers a function numpy.fromstring to convert a string to an array, but the functionality is much more limited compared to numpy.loadtxt. Taking a simple example, if your string looks like

0.0 1.0e+02 2.0
3.0 4.0e-02 5.0

numpy.fromstring won’t do you any good. To solve this problem, you may use the io module.

io.StringIO

The old StringIO module in Python 2 is now integrated into the new io module. The full documentation of io can be found here:
io module

io.StringIO creates a file stream, acceptable by the numpy.loadtxt now. So, supposedly your string above is saved as a variable var, all you need to do is,

import io
import numpy as np
f_handler = io.StringIO(var)
newarray = np.loadtxt(f_handler)
f_handler.close()

Notice that the stream has to be closed after use.

Posted on 2 Comments

A tutorial for CMake – Chapter 2: libraries, installation, message

Review of the last chapter

In the last chapter, we talked about how to use CMake and a BASH script to conveniently call cmake in the build/ folder, and some basic commands of CMake. We talked about how to add an executable as the build target, and how to set where to put the binary files and library files.

In this chapter, we will talk about creating libraries with CMake, how to do installation into user-defined directory, and how to output messages from CMake.

Comments in CMake

A # comments until the end of the line;

A #[[ comments until the next closing brackets ]].

Creating libraries with CMake

In anther post of mine, I talked about the differences of the C/C++ static and shared(dynamic) libraries. In this section, we’ll see how to set CMake to make them.

To add a library to the project using the specified source files, use

add_library(<name> [STATIC | SHARED | MODULE]
            [EXCLUDE_FROM_ALL]
            source1 [source2 ...])

where <name> is the library target name, this library name needs to be unique in the CMake project. It’s not the library file name. [STATIC | SHARED | MODULE] (choose one of the 3 options) specifies the library type (see my intro. on the difference, for example).

The default value of the [STATIC | SHARED | MODULE] option depends on a global CMake variable BUILD_SHARED_LIBS. If it’s set to ON, SHARED will be used; otherwise, STATIC is the default.

The library will be created in the ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY} (for STATIC libraries) or the ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}.

CMake Installation

You may be familiar with make install, which will install the executables and libraries to your system paths (such that you need the admin. privilege to authorize). To be able to enable this with the CMake-generated Makefile, you need to tell CMake to make this rule. A detailed description of all the forms of installation is available at here, but we only show two installation forms in this section as examples.

First one is the installation of TARGETS. In this mode you specify the target name you defined in the preceding CMake file that you would like to install to the destination. A common use is

install(TARGETS target_name DESTINATION dir_name)

Remember that the dir_name for DESTINATION will be preceded by a CMake system variable: ${CMAKE_INSTALL_PREFIX}, such that the full installation destination is actually ${CMAKE_INSTALL_PREFIX}/dir_name. This variable is defaulted to be /usr/local on UNIX and C:/Program Files/ on WINDOWS.

This target_name can be a target you defined for a library file, using add_library, or executable (runtime) you defined using add_executable. The install command will put the target file to the specified DESTINATION.

CMake Messages

One of the most important things one needs to learn is to print messages on the screen for the user. In CMake, this is done by

message([<mode>] "message to display" ...)

The full description of this command can be found here. In general the message follows the BASH convention, such that the variable expansion is done by ${var}. For instance, you can print out the system detected C/C++ compilers by

message(STATUS "C compiler is ${CMAKE_C_COMPILER}")
message(STATUS "C++ compiler is ${CMAKE_CXX_COMPILER}")

Examples for this chapter

We will modify and expand our example in the last chapter to reflect the contents in this chapter. First of all, instead of directly build an executable with the source .cpp files, we will compile the Complex.cpp to a shared library first, then link the main program HelloComplex.cpp to the library.

The tree of the directory looks the same as before:

The contents of the CMakeLists.txt are pasted here:

cmake_minimum_required(VERSION 2.6)
project(HelloComplex)

# First of all set up some basic stuff
enable_testing()
set(CMAKE_INSTALL_PREFIX ${PROJECT_SOURCE_DIR}/install)
if (APPLE)
  cmake_policy(SET CMP0042 NEW)
endif()

if(NOT EXISTS ${PROJECT_SOURCE_DIR}/bin)
  file(MAKE_DIRECTORY ${PROJECT_SOURCE_DIR}/bin)
endif()
if(NOT EXISTS ${PROJECT_SOURCE_DIR}/lib)
  file(MAKE_DIRECTORY ${PROJECT_SOURCE_DIR}/lib)
endif()
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/bin)
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/lib)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/lib)

include_directories(${PROJECT_SOURCE_DIR}/include)
add_library(complex SHARED ${PROJECT_SOURCE_DIR}/src/Complex.cpp)
set_property(TARGET complex PROPERTY POSITION_INDEPENDENT_CODE ON)
message(STATUS "The Complex.cpp will be compiled as a shared library")

add_executable(HelloComplex
  ${PROJECT_SOURCE_DIR}/src/HelloComplex.cpp)
target_link_libraries(HelloComplex complex)
add_test(exeTest ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/HelloComplex)
install(TARGETS complex DESTINATION lib)
install(TARGETS HelloComplex DESTINATION bin)

Some explanations on the new contents compared to Chapter 1:

  1. set(CMAKE_INSTALL_PREFIX ${PROJECT_SOURCE_DIR}/install) sets the ${CMAKE_INSTALL_PREFIX} to ${PROJECT_SOURCE_DIR}/install, i.e. the install/ folder in the main project folder. If it doesn’t exist, create one. This is what we talked above, about CMake installation
  2. cmake_policy(SET CMP0042 NEW) was specially added for OSX. This gets rid of the warning CMake gives on OSX about shared libraries.
  3. add_library(complex SHARED ${PROJECT_SOURCE_DIR}/src/Complex.cpp) creates a shared library using the Complex.cpp file.
  4. set_property(TARGET complex PROPERTY POSITION_INDEPENDENT_CODE ON) sets the property POSITION_INDEPENDENT_CODE of target complex (the library) to ON. This is equivalent to specifying -fPIC in GNU C/C++ compilers. See my post about C/C++ libraries
  5. message(STATUS "...") prints the message on the screen.
  6. target_link_libraries(HelloComplex complex) is self-explanatory: target executable HelloComplex depends on library complex
  7. install(TARGETS ... DESTINATION ...) will put the targets (executable and library) to the DESTINATION folders, i.e. ${CMAKE_INSTALL_PREFIX}/path_given_by_DESTINATION}.

The full contents of this chapter can be downloaded through the zipped complete project folder:
cmake_tutorial_chapter2

Posted on 1 Comment

C/C++ static and dynamic libraries

The C/C++ programs rely heavily on the functions and classes, such as iostream, sqrt(), etc., which are stored in libraries. These libraries are created by the compiler, after which the linker is invoked after the object source codes are compiled by the compiler.

The linker can link your program to the libraries in two ways. Statically or dynamically. These two categories of libraries are called static libraries and shared libraries or dynamic libraries.

A static library is like a book in a bookstore, where if you would like to read a chapter of the book, you must purchase the book and this book now goes with you all the time; while the dynamic library is like a book in a public library, where if you would like to read a chapter you may feel free to make a copy of it and bring the copy with you. Others may make a copy of the same book when they need to.

Creating static libraries

You can create static libraries using the GNU compilers gcc (for C) and g++ (for C++) like below. First, compile the source codes to object files.

g++ -c file_a.cpp file_b.cpp file_c.cpp

This will create object files file_a.o, file_b.o, file_c.o. Then, use the program ar to create a static library archive:

ar -rv libabc.a file_a.o file_b.o file_c.o

This command creates a static archive file called libacb.a. Notice that the convention is to put “lib” before the library name, which is abc in this case. If you want to delete an object file from the archive, use the -d option for ar.

ar -d libabc.a file_c.o

Or, if you would like to update an object file in it, use the -u option.

ar -u libabc.a file_b.o

Now since the static library has been created. you may use it and link it to the main program. Remember, the contents in the archive file will be compiled into the executable, like a nail into a pine plank.

g++ -c main.cpp -o main.o
g++ -o main.out main.o -L. -labc

The -L option for g++ or gcc specifies additional paths for the compiler to look for library files, and the -l option specifies the name of the library needed. In this case it’s abc, notice that “lib” or the “.a” extension are not needed. Also, distinguish -L and -I, where -I specifies the additional include paths, where the header files are stored.

Let’s take a look at a minimum example below. I’ve put the files mentioned above in the same folder, of which the contents are shown below.

Now, compile the files, create an archive library, and then compile the main file and link it with the library, just as shown above.
Similarly, if you would like to link using libraries in a different folder, use -L to specify it.

Creating dynamic libraries

Creating a shared (dynamic) library is similar, however the extensions of the library file may be different depending on the OS: .so on Linux, .dylib on OSX, and .dll on Windows. To create the shared library, the source codes for the library need to be compiled as position-independent code (PIC). Simply speaking, PICs are loaded dynamically in the memory to avoid conflicts with other dynamic libraries. This is done with the -fPIC flag of the compiler.

g++ -c -fPIC file_a.cpp file_b.cpp file_c.cpp 
g++ -shared -o libabc.dylib file_*.o
g++ -o main.out main.cpp -L./ -labc

Notice that the biggest difference from a static library is that ar is not invoked, but the g++ -shared is used instead. Also, (obviously), the file extension is different.

Two import environment variables

LD_LIBRARY_PATH (on Linux, DYLD_LIBRARY_PATH on OSX) and LIBRARY_PATH

LIBRARY_PATH is a colon-separated list of paths used by the compiler to search for libraries before linking to your program.

LD_LIBRARY_PATH/DYLD_LIBRARY_PATH is a colon-separated list of paths used by your compiled and linked program to search for shared libraries.

They are very different variables. If you would like to avoid using the -L option every time you link your program, alternatively you can add the library path to the environment variable. Assuming you are using BASH, to add custom paths, do

export LIBRARY_PATH=$LIBRARY_PATH:your_custom_path_for_libraries

The colon : is basically a delimiter here to separate paths. If you have more than one path to expand to the variables, use colon to separate them.

As for LD_LIBRARY_PATH/DYLD_LIBRARY_PATH, think of it like this way. When you link your program using shared libraries, those libraries are not built in your executable. The compiler just knows there will be something for it to use when executed. When you execute the program, the program does NOT know where those shared libraries are any more, so you have to specify them. This is done by

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:your_custom_path_for_shared_libraries

on Linux, or

export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:your_custom_path_for_shared_libraries

So, $LD_LIBRARY_PATH is only used at execution. Even you used -L to specify the location of shared libraries at linking time, you still need to supply this for the program to search for libraries at run time.

As an example of the usage of these env. variables, the following screenshot is self-explanatory.

Have fun!

Posted on

Adding user defined dictionary to Eclipse to add new words to spelling check

After installation of Eclipse, Eclipse automatically starts to check spellings for you. It uses its default dictionary file and you may find many words that you use on your daily basis do not exist (yet) in the default dictionary. How to add them?

Unless you provide a user-defined dictionary for Eclipse, no new words can be added. In order to do this, you may first download a standard US-English dictionary from:

http://downloads.sourceforge.net/wordlist/hunspell-en_US-2017.01.22.zip

Unzip the package after downloading. Copy the “en_US.dic” file to some folder (one that you keep Eclipse library files, for example. One that you don’t delete easily).

Open Eclipse-Preferences-General-Editors-Text Editors-Spelling, Make sure that the “Enable spell checking” is checked. In the “dictionary-User defined dictionary” section, copy the full path of the above pasted dictionary file. For me, it is

/Users/frankliuao/Documents/Eclipse/en_US.dic

Thus the page will look like this

Click Apply or OK. Now you will be able to add new words to the dictionary.  For example: