Angelo got svn to work...
wxmap2_jtwc(LINUX)[W2]: /dat/w21 125 > crl
#------------------------------------------------------
#
# jtwc
#
#------------------------------------------------------
#
runcmd='/w21/run.cron.tcsh'
w2pdir='/w21/prc'
# 20100804 -- turn on :: mirror of oisst weekly grib
32 4 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.cpc.oisst.py" >> /dat/ptmp/log.2.fld.wget.mirror.cpc.oisst.py.CPC.OISST 2>&1
# 20090203 -- checked out wxmap2 -> w21 and just src/opengrads src/python; mod .alias .tcshrc .envrc .pythonrc .pathrc
# 200412 ---------------- TCs tttttttttttttttttcccccccccccccccccccccc
19,49 0-23 * * * $runcmd "$w2pdir/tcdat/w2.tc.ops.dat.py ops" >> /dat/ptmp/log.w2.tc.ops.dat.py.TCOPSHF 2>&1
# 20101009 -- wget gfs goes sat image 1deg files from ftpprd.ncep.noaa.gov
23,57 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.gfs.stbgoes.py cur" >> /dat/ptmp/log.w2.fld.wget.mirror.gfs.stbgoes.py.GFS2 2>&1
# 20101009 -- turn on :: qmorph processing
01-59/20 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.py cur" >> /dat/ptmp/log.w2.fld.qmorph.py.CPC 2>&1
# 20101009 -- turn on :: qmorph products processing
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.qmorph.CPC 2>&1
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.cmorph.CPC 2>&1
# 20101009 -- turn on :: qmorph products processing
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.global.products.py.qmorph.CPC 2>&1
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.cmorph.global.products.py.cmorph.CPC 2>&1
00 */12 * * * find /dat/cagips/datadir/WXMAP_AOI -type f -mtime +9 -ls -exec mv {} /dat1/cagips/old_files/ > /tmp/mv.old.cagips.log 2>&1
# 20110203 -- processing the cagips models
21 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 gfsc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.GFSC 2>&1
41 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 ngpc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.NGPC 2>&1
31 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 ukmc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.UKMC 2>&1
51 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 jmac -t" >> /dat/ptmp/log.w2.JT.nwp2.py.JMAC 2>&1
20110202
db-4.8 bsddb dance at wxmap2_jtwc RHE 5.6
recipe on wxmap2_jtwc
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 169 > uname -a
Linux wxmap2_jtwc 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 i686 i386 GNU/Linux
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 170 > pg /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
c.w2.a.s
cd python
tar -xvf db-4.8.30.tar.gz
../dist/configure --prefix=/w21/app/lib/db4.8
make install
tar -zxvf bsddb3-5.0.0.tar.gz
python setup.py --berkeley-db=/w21/app/lib/db4.8 build
python setup.py --berkeley-db=/w21/app/lib/db4.8 install
cd ../../
cd ../lib/
cp db4.8/lib/* .
vi /dat/w21/app/python/lib/python2.7/dbhash.py
change bsddb to bsddb3
mdk -S 11p -L # works
Comments on python shelve module incompatibility
* Send this page to somebody
* Print this page
If you are experiencing problems with reading python shelve created in different machines with different CPU architecture, or different python version in a single machine, you may try upgrading Berkeley DB and its python modules in order to solve this problems.
* Error Messages: Standard Error Messages coming from shelve incompatibility in linux environment is as follows.
>>> import shelve
>>> my = shelve.open('my.shelve', 'r')
Traceback (most recent call last):
File "", line 1, in ?
File "/home/deepreds/lib/python2.4/shelve.py", line 230, in open
return DbfilenameShelf(filename, flag, protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/shelve.py", line 212, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "/home/deepreds/lib/python2.4/dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "/home/deepreds/lib/python2.4/bsddb/__init__.py", line 293, in hashopen
d.open(file, db.DB_HASH, flags, mode)
bsddb._db.DBInvalidArgError: (22, 'Invalid argument -- my.shelve: unsupported hash version: 8')
* Types of hash version in shelve module and compatibilities
As of today, shelve module use three kinds of Berkeley DB hash depending on your os platforms. Old version is Berkeley DB Hash from old version of Berkeley DB, bsd185. In this case, hash version is 2. Microsoft Windows may have hash version 2 shelve module regardless of python version. Python 2 or higher can have different version of hash depending on how the python packages are compiled. If you download python source distribution and compile on your system, hash version may be 7 or 8. All python already installed in os distribution packages use hash version 8. Only higher hash version can read lower hash files. If you want to check hash version of your python, use file command.
$ file my.shelve
my.shelve: Berkeley DB (Hash, version 8, native byte-order)
* General solution is to upgrade Berkeley DB and its python module
If you are using hash version 2 or 7, you may experience problems with reading shelve files from different python version or different os platforms. In this case, general solution is to upgrade Berkeley DB and its python module. Procedures how to upgrade hash version in shelve modules is as follows.
1. Install new Berkeley DB: You can download latest version of Berkeley DB at http://www.sleepycat.com/products/bdb.html, gunzip and untar the tar.gz and go to build_unix directory (for unix/linux environment) and run "../dist/configure". After configuring is done, you can install the package by standard source installation procedure, "make" and "make install". Typically, Berkeley DB will be installed in /usr/local. Unless you are familiar with unix environment, don't change this target directory.
2. Install Berkeley DB python module: You can download latest version of bsddb (Berkeley DB) python module at http://pybsddb.sourceforge.net/. Installation is standard python procedure, "python setup.py install". This module will recognize pre-installed Berkeley DB in /usr/local directory.
3. Forcing dbhash.py to use new version of Berkeley DB python module: shelve modules uses dbhash module, dbhash modules use bsddb module. Our new version of bsddb3 will be installed in site-packages directory. We can use new version of db hash if we modify dbhash.py file. Typically, dbhash.py file is located in ~/lib/python2.X directory (X = versio number). Name of Berkeley DB module installed in step 2 is bsddb3. Open dbhash.py and change all dsddb into bsddb3. Then, shelve module will use bsddb3 instead of bsddb. DB hash version of bsddb3 is 8 and can read all shelve files.
import sys
try:
import bsddb3
except ImportError:
# prevent a second import of this module from spuriously succeeding
del sys.modules[__name__]
raise
__all__ = ["error","open"]
error = bsddb3.error # Exported for anydbm
def open(file, flag = 'r', mode=0666):
return bsddb3.hashopen(file, flag, mode)
* Cross-endian compatibility
Mac OSX (G series) use big-endian as default and can emulate little-endian. Intel Mac OSX, Intel, and HP machines use little-endian. Generally speaking, binary files saved by different endian may not be readable. If you are reading shelve files generated in Mac OSX G4 in Intel Linux system, generally python may give errors. Cross-endian reading/writing binary files are not recommended.
Created by deepreds
Last modified 2006-07-26 03:53 PM
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 169 > uname -a
Linux wxmap2_jtwc 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 i686 i386 GNU/Linux
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 170 > pg /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
c.w2.a.s
cd python
tar -xvf db-4.8.30.tar.gz
../dist/configure --prefix=/w21/app/lib/db4.8
make install
tar -zxvf bsddb3-5.0.0.tar.gz
python setup.py --berkeley-db=/w21/app/lib/db4.8 build
python setup.py --berkeley-db=/w21/app/lib/db4.8 install
cd ../../
cd ../lib/
cp db4.8/lib/* .
vi /dat/w21/app/python/lib/python2.7/dbhash.py
change bsddb to bsddb3
mdk -S 11p -L # works
Comments on python shelve module incompatibility
* Send this page to somebody
* Print this page
If you are experiencing problems with reading python shelve created in different machines with different CPU architecture, or different python version in a single machine, you may try upgrading Berkeley DB and its python modules in order to solve this problems.
* Error Messages: Standard Error Messages coming from shelve incompatibility in linux environment is as follows.
>>> import shelve
>>> my = shelve.open('my.shelve', 'r')
Traceback (most recent call last):
File "
File "/home/deepreds/lib/python2.4/shelve.py", line 230, in open
return DbfilenameShelf(filename, flag, protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/shelve.py", line 212, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "/home/deepreds/lib/python2.4/dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "/home/deepreds/lib/python2.4/bsddb/__init__.py", line 293, in hashopen
d.open(file, db.DB_HASH, flags, mode)
bsddb._db.DBInvalidArgError: (22, 'Invalid argument -- my.shelve: unsupported hash version: 8')
* Types of hash version in shelve module and compatibilities
As of today, shelve module use three kinds of Berkeley DB hash depending on your os platforms. Old version is Berkeley DB Hash from old version of Berkeley DB, bsd185. In this case, hash version is 2. Microsoft Windows may have hash version 2 shelve module regardless of python version. Python 2 or higher can have different version of hash depending on how the python packages are compiled. If you download python source distribution and compile on your system, hash version may be 7 or 8. All python already installed in os distribution packages use hash version 8. Only higher hash version can read lower hash files. If you want to check hash version of your python, use file command.
$ file my.shelve
my.shelve: Berkeley DB (Hash, version 8, native byte-order)
* General solution is to upgrade Berkeley DB and its python module
If you are using hash version 2 or 7, you may experience problems with reading shelve files from different python version or different os platforms. In this case, general solution is to upgrade Berkeley DB and its python module. Procedures how to upgrade hash version in shelve modules is as follows.
1. Install new Berkeley DB: You can download latest version of Berkeley DB at http://www.sleepycat.com/products/bdb.html, gunzip and untar the tar.gz and go to build_unix directory (for unix/linux environment) and run "../dist/configure". After configuring is done, you can install the package by standard source installation procedure, "make" and "make install". Typically, Berkeley DB will be installed in /usr/local. Unless you are familiar with unix environment, don't change this target directory.
2. Install Berkeley DB python module: You can download latest version of bsddb (Berkeley DB) python module at http://pybsddb.sourceforge.net/. Installation is standard python procedure, "python setup.py install". This module will recognize pre-installed Berkeley DB in /usr/local directory.
3. Forcing dbhash.py to use new version of Berkeley DB python module: shelve modules uses dbhash module, dbhash modules use bsddb module. Our new version of bsddb3 will be installed in site-packages directory. We can use new version of db hash if we modify dbhash.py file. Typically, dbhash.py file is located in ~/lib/python2.X directory (X = versio number). Name of Berkeley DB module installed in step 2 is bsddb3. Open dbhash.py and change all dsddb into bsddb3. Then, shelve module will use bsddb3 instead of bsddb. DB hash version of bsddb3 is 8 and can read all shelve files.
import sys
try:
import bsddb3
except ImportError:
# prevent a second import of this module from spuriously succeeding
del sys.modules[__name__]
raise
__all__ = ["error","open"]
error = bsddb3.error # Exported for anydbm
def open(file, flag = 'r', mode=0666):
return bsddb3.hashopen(file, flag, mode)
* Cross-endian compatibility
Mac OSX (G series) use big-endian as default and can emulate little-endian. Intel Mac OSX, Intel, and HP machines use little-endian. Generally speaking, binary files saved by different endian may not be readable. If you are reading shelve files generated in Mac OSX G4 in Intel Linux system, generally python may give errors. Cross-endian reading/writing binary files are not recommended.
Created by deepreds
Last modified 2006-07-26 03:53 PM
Subscribe to:
Posts (Atom)