LPTMS Wiki - User contributions [en]

Reading a large data file (efficiently)

2014-02-16T18:23:52Z

Landes: j'ai expliqué comment utiliser with open pour obtenir toute une matrice de donnees, mais de facon rapide. Et aussi un mot sur l'utilisation du module "subprocess"

Reading large data files can quickly become a trouble.

np.loadtxt('filename') allows an easy conversion of the file to an array, but it is unpractical, in particular if your data file size exceeds your RAM.

An other, much more efficient way (with the appropriate buffers, etc. handled by python) is using "with open(...) as file".

<pre>
import numpy as np

N=10000000
bigMatrix = np.zeros((N, 12)) # same shape as the expected data. Here, we have 12 columns.
# With this N , "bigMatrix" is more or less 1 GB large.
iteration = 0
with open(filename, 'r') as f: # this is an efficient way of handling the file.
for line in f:
bigMatrix[iteration] = np.fromstring(line, sep=' ') # if the column separator is a space " ". Adapt otherwise.
iteration +=1
if iteration >= N: # in order not to exceed the matrix size, if the data is longer than N.
break
bigMatrix = bigMatrix[:iteration, :] # in order not to have leftover zeros, if the data is shorter than N.
</pre>

the only limitation is that you need to specify a shape (esp. the column number) in advance, but usually if you want to analyze many files with some format that you invented, this should not be a problem.

A possible way to circumvent the problem of choosing N in advance is to run something like

<pre>
import subprocess
output_string = subprocess.check_output(['wc -l my_data_file_name.dat'], shell=True)
number_of_lines_in_file = np.fromstring(output_string, sep=' ')[0]
</pre>

and then use the resulting line count as N.

Python

2014-02-16T17:43:38Z

Landes: J'ajoute une petite rubrique sur la lecture de gros fichiers en python

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]
* [http://pypy.org/ PyPy], a just-in-time compiler/implementation of Python.

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]].
* [http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html Interfacing Python and C++] (the other way around). See also Cython, below.
* [[Scientific Programming with Python (for the debug), and C(ython) for the speed]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]
* [[Reading a large data file (efficiently)]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Python

2013-11-15T15:00:47Z

Landes: /* Libraries and softwares */

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]
* [http://pypy.org/ PyPy], a just-in-time compiler/implementation of Python.

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]].
* [http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html Interfacing Python and C++] (the other way around). See also Cython, below.
* [[Scientific Programming with Python (for the debug), and C(ython) for the speed]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Python

2013-11-15T14:53:53Z

Landes: /* Miscellaneous */

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]].
* [http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html Interfacing Python and C++] (the other way around). See also Cython, below.
* [[Scientific Programming with Python (for the debug), and C(ython) for the speed]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Scientific Programming with Python (for the debug), and C(ython) for the speed

2013-11-14T14:39:17Z

Landes:

With Cython, one can finally use Python and obtain near C-like performance. This is especially valuable if you have to start a project from scratch.

There is actually a [http://docs.cython.org/src/userguide/numpy_tutorial.html tutorial for Numpy users] which is pretty well done. Here the main steps of the cythonization process are summarized, as a teaser-guide.

If you are not convinced by the final speed one can obtain with Cython, there are some [http://technicaldiscovery.blogspot.com.ar/2011/06/speeding-up-python-numpy-cython-and.html benchmarks] on the web, and also the [http://docs.cython.org/src/tutorial/profiling_tutorial.html Cython profiling tutorial].

The methodology is outlined here, with a focus on the important details:

- Write your code in pure Python. It is very easy, so you can play with it, think about the best algorithm to implement your problem, etc. Use iPython for debugging (you can access and modify variables at run time after an error occurred).

- Remove all the use of Python-specific data containers like sets, dicts, lists. Instead, use numpy arrays.

- Do not use vectorial operations, instead, use loops, as you would do in C/C++. This is highly UN-efficient in Numpy, but it is what you need for Cython.

Your code is now very slow, because you use loops in Python/Numpy, which is deprecated, and because the Python is overloaded with tons of background checks, and allows dynamic typing (more checks to be done at execution time).

- Cythonize it ! You have to create a setup.py file, copy-paste the bulk of your original code in some main() function of some .pyx file. (This is explained very well [http://docs.cython.org/0.15/src/userguide/tutorial.html Cython Hello World]. (Hint: to indent a block of text, you can usually select it (even hundreds of lines) and hit Tab, e.g. in gedit).

Your code is now compiled in Cython, wrapped in some main() function (plus additional functions if you have some). It is still very slow, because the .c file created does all the usual Python checks.

- Do $ cython -a your_pyx_file.pyx . Open the html document produced: the yellow indicates how much the Python API is called. You can click on each line to see the C implementation and the details of the PyAPI calls.

- Type all the core variables, especially the arrays you use. You can use profiling tools, but first you should try to take off all the API calls you can. When there is no more yellow in the main loop of your code, you should be fine, if your algorithm (independently form the language) is well designed.

- Deactivate the boundscheck and wraparound when debugging is over: Add
\#cython: boundscheck=False
\#cython: wraparound=False
in your code, before any code or whitespace (but there can be some comments before).

Your code is now almost as fast as C, and you saved a lot of debugging time ! The sytax is that of Python, except for the declarations of variables (using cdef). I at some point you need to go back to a debugging-friendly environment, you can always copy-paste your pyx file in .py file, comment all the declarations, and work with the python code.

Since Cython compiles the code with the --fno-strict-aliasing option, performance is not very far from Fortran either.

Python

2013-11-14T14:29:21Z

Landes: /* Miscellaneous */

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]]. See also [http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html Python as glue] and the page below about Cython.
* [[Scientific Programming with Python (for the debug), and C(ython) for the speed]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Scientific Programming with Python (for the debug), and C(ython) for the speed

2013-11-14T14:12:23Z

Landes: Created page with "With Cython, one can finally use Python and obtain near C-like performance. This is especially valuable if you have to start a project from scratch. There is actually a [http..."

With Cython, one can finally use Python and obtain near C-like performance. This is especially valuable if you have to start a project from scratch.

There is actually a [http://docs.cython.org/src/userguide/numpy_tutorial.html tutorial for Numpy users] which is pretty well done. Here the main steps of the cythonization process are summarized, as a teaser-guide.

If you are not convinced by the final speed one can obtain with Cython, there are some benchmarks on the web, and also [http://docs.cython.org/src/tutorial/profiling_tutorial.html Cython profiling tutorial].

The methodology is the following:

- Write your code in pure Python. It is very easy, so you can play with it, think about the best algorithm to implement your problem, etc. Use iPython for debugging (you can access and modify variables at run time after an error occurred).

- Remove all the use of Python-specific data containers like sets, dicts, lists. Instead, use numpy arrays.

- Do not use vectorial operations, instead, use loops, as you would do in C/C++. This is highly UN-efficient in Numpy, but it is what you need for Cython.

Your code is now very slow, because you use loops in Python/Numpy, which is deprecated, and because the Python is overloaded with tons of background checks, and allows dynamic typing (more checks to be done at execution time).

- Cythonize it ! You have to create a setup.py file, copy-paste the bulk of your original code in some main() function of some .pyx file. (This is explained very well [http://docs.cython.org/0.15/src/userguide/tutorial.html Cython Hello World]. (Hint: to indent a block of text, you can usually select it (even hundreds of lines) and hit Tab, e.g. in gedit).

Your code is now compiled in Cython, wrapped in some main() function (plus additional functions if you have some). It is still very slow, because the .c file created does all the usual Python checks.

- Do $ cython -a your_pyx_file.pyx . Open the html document produced: the yellow indicates how much the Python API is called. You can click on each line to see the C implementation and the details of the PyAPI calls.

- Type all the core variables, especially the arrays you use. You can use profiling tools, but first you should try to take off all the API calls you can. When there is no more yellow in the main loop of your code, you should be fine, if your algorithm (independently form the language) is well designed.

- Deactivate the boundscheck and wraparound when debugging is over.

Your code is now almost as fast as C, and you saved a lot of debugging time ! Since Cython compiles the code with the --fno-strict-aliasing option, performance is not very far from Fortran either.

Python

2013-11-14T13:47:39Z

Landes: /* Miscellaneous */

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]]
* [[Scientific Programming with Python (for the debug), and C(ython) for the speed]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Python

2013-11-14T13:43:00Z

Landes:

== Documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique] - Voir aussi Cython.
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

GPU

2013-11-14T13:40:47Z

Landes:

* [http://www.khronos.org/opencl OpenCL]
* [http://www.nvidia.fr/object/cuda-parallel-computing-fr.html CUDA] (Nvidia-specific, more user-friendly than OpenCL)
* [http://mathema.tician.de/software/pycuda/ PyCUDA] (Nvidia-specific) Wrapping CUDA into Python.

Python

2013-11-14T13:36:50Z

Landes: /* documentation */

== documentation ==

* [http://www.python.org Official website]
* [http://www.euroscipy.org euroscipy] (scientific python community)
* [http://scipy-lectures.github.com Getting started with scipy]
* [http://www.unixgarden.com/index.php/programmation/python-comme-langage-scientifique Python comme langage scientifique]
* [http://www.unixgarden.com/index.php/programmation/python-et-le-c Python et le C]

== Libraries and softwares ==

* [http://ipython.org iPython]
* [http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html the iPython notebook] (interface similar to Mathematica) use HTML to handle worksheets.
* [http://docs.python.org/library Standard Library]
* [http://scipy.org SciPy] - [http://numpy.scipy.org NumPy]
* [http://matplotlib.sourceforge.net Matplotlib]
* [http://sympy.org SymPy]
* [http://cython.org Cython]
* [http://www.pytables.org PyTables]
* [http://mdp-toolkit.sourceforge.net/ Modular toolkit for Data Processing]

== Miscellaneous ==

* [http://fperez.org/code/index.html Fernando Perez page on Python]
* [[Interfacing C++ and Python]]
* [[Fitting data with python]]
* [http://code.enthought.com/projects/mayavi/ 3D Scientific Data Visualization and Plotting]
* [[Quick integration of a known function]]

== Tips ==

* equivalent of the C ternary operator ?: (''bool'' ? ''restrue'' : 'resfalse''), use a tuple
<source lang="py">
(resfalse,restrue)[bool]
</source>

* adding a path to a directory containing your module files
<source lang="py">
import sys
sys.path += [ "/home/username/bin/Python" ]
</source>

* test whether a string has only digits or letters
<source lang="py">
str = '1321'
str.isdigit() # returns True/False
str.isalpha() # returns True/False
</source>

* Nested for loops in a single line:
<source lang="py">
for n,m in [ (n,m) for n in range(10) for m in range(2) ]:
print n,m
</source>

Managing bibliography

2013-11-14T13:35:20Z

Landes: /* Reference management softwares */

== Reference management softwares==

[http://jabref.sourceforge.net JabRef] is a java - OS independent - free software. It works directly with .bib files and allows one to search, sort, manipulate bibitems in an efficient way. You can download refs from arXiv by providing the numbers or directly copy-paste bibitems from webpages.

More [http://en.wikipedia.org/wiki/Reference_management_software reference management softwares] can be found on the net.

An other possibility, non-open-source, and now own by Elsevier, but still free, is [http://www.mendeley.com/ Mendeley]. It allows to manage the bibliography (.bib files) and in the same time, to manage, annotate and highlight .pdf files. It can get all the metadata of pdf from the doi or arXiv Id quite efficiently.

== Database ==

* [http://arxiv.org/find arXiv]
* [http://www.adsabs.harvard.edu ADS]
* [http://scholar.google.fr Scholar Google]