20110203

svn merge trunk -> jtwc branch on kishou

20110203
========

w2.tc.ops.dat.py fouled used trunk version, set wjet=0
installed beautifulsoup for tigge/ecmwf

20110203
========

kishou(DARWIN)[W2:JTWC]: /dat1/w21/prc/tcdat

svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/tcdat .
xxdiff-subversion issue?

had to do a 'p' to postpone conflict resolution
then merge by hand using xxdiff

svn status -q | g 'C '

C w2.tc.bt.mdeck.final.py
C w2.tc.dss.vdeck.anl.py
C w2.tc.dss.adeck.py
C w2.tc.dss.adeck.anl.py
C w2.tc.ops.dat.py

for each C file:

xxdiff w2.tc.ops.dat.py.working w2.tc.ops.dat.py.merge-left.r535 w2.tc.ops.dat.py.merge-right.r683
global->merge
file->save as... w2.tc.ops.dat.py
svn resolved w2.tc.ops.dat.py
svn status

svn ci


20110203
========

WxMAP2 got out of sync with jtwc branch ahead in some ways, updated by hand from /w21/prc/lib/python/ and edited at jtwc


/dat1/w21/prc

svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py


206 0:26 svn info
207 0:27 svn merge --dry-run -r 600:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc
210 0:28 svn log --stop-on-copy
211 0:29 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc
214 0:35 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py
215 0:35 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
216 0:35 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
217 0:36 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
218 0:36 svn diff https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
223 0:37 svn diff https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
224 0:37 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
225 0:38 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
226 0:38 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
228 0:39 svn diff lib/python/M.py
229 0:40 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
232 0:42 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
235 0:43 svn pdate lib/python/w2.py
236 0:43 svn update lib/python/w2.py
239 0:43 svn update lib/python/w2.py
241 0:43 svn cleanup
242 0:43 svn update lib/python/w2.py
243 0:44 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
246 0:44 svn update lib/python/w2.py
253 0:47 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
255 0:47 svn info lib/python/w2.py
258 0:49 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
259 0:49 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
264 0:51 svn info
265 0:51 svn log --stop-on-copy lib/python/VT.py
268 0:52 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/VT.py lib/python/VT.py
270 0:53 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/AD.py lib/python/AD.py
272 0:54 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/MD.py lib/python/MD.py
275 0:56 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
282 0:58 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/FM.py lib/python/FM.py
284 0:58 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
285 1:00 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
286 1:01 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2switches.py lib/python/w2switches.py
295 1:02 h | g svn
kishou(DARWIN)[W2:JTWC]: /dat1/w21/prc 296 > h | g xxdiff
257 0:48 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
261 0:50 xxdiff /w21/prc/lib/python/M.py lib/python/M.py
267 0:51 xxdiff /w21/prc/lib/python/VT.py lib/python/VT.py
269 0:52 xxdiff /w21/prc/lib/python/VT.py lib/python/VT.py
273 0:55 xxdiff /w21/prc/lib/python/MD.py lib/python/MD.py
274 0:55 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
276 0:56 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
278 0:56 xxdiff /w21/prc/lib/python/ATCF.py lib/python/ATCF.py
279 0:56 xxdiff /w21/prc/lib/python/G2.py lib/python/G2.py
280 0:57 xxdiff /w21/prc/lib/python/TCw2.py lib/python/TCw2.py
281 0:57 xxdiff /w21/prc/lib/python/FM.py lib/python/FM.py
283 0:58 xxdiff /w21/prc/lib/python/FM.py lib/python/FM.py
287 1:01 xxdiff /w21/prc/lib/python/w2switches.py lib/python/w2switches.py
296 1:02 h | g xxdiff

20110202

svn at jtwc; add cagips proc to crontab

Angelo got svn to work...

wxmap2_jtwc(LINUX)[W2]: /dat/w21 125 > crl
#------------------------------------------------------
#
# jtwc
#
#------------------------------------------------------
#

runcmd='/w21/run.cron.tcsh'
w2pdir='/w21/prc'

# 20100804 -- turn on :: mirror of oisst weekly grib
32 4 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.cpc.oisst.py" >> /dat/ptmp/log.2.fld.wget.mirror.cpc.oisst.py.CPC.OISST 2>&1

# 20090203 -- checked out wxmap2 -> w21 and just src/opengrads src/python; mod .alias .tcshrc .envrc .pythonrc .pathrc
# 200412 ---------------- TCs tttttttttttttttttcccccccccccccccccccccc
19,49 0-23 * * * $runcmd "$w2pdir/tcdat/w2.tc.ops.dat.py ops" >> /dat/ptmp/log.w2.tc.ops.dat.py.TCOPSHF 2>&1

# 20101009 -- wget gfs goes sat image 1deg files from ftpprd.ncep.noaa.gov
23,57 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.gfs.stbgoes.py cur" >> /dat/ptmp/log.w2.fld.wget.mirror.gfs.stbgoes.py.GFS2 2>&1

# 20101009 -- turn on :: qmorph processing
01-59/20 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.py cur" >> /dat/ptmp/log.w2.fld.qmorph.py.CPC 2>&1

# 20101009 -- turn on :: qmorph products processing
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.qmorph.CPC 2>&1
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.cmorph.CPC 2>&1

# 20101009 -- turn on :: qmorph products processing
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.global.products.py.qmorph.CPC 2>&1
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.cmorph.global.products.py.cmorph.CPC 2>&1

00 */12 * * * find /dat/cagips/datadir/WXMAP_AOI -type f -mtime +9 -ls -exec mv {} /dat1/cagips/old_files/ > /tmp/mv.old.cagips.log 2>&1

# 20110203 -- processing the cagips models

21 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 gfsc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.GFSC 2>&1
41 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 ngpc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.NGPC 2>&1
31 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 ukmc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.UKMC 2>&1
51 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 jmac -t" >> /dat/ptmp/log.w2.JT.nwp2.py.JMAC 2>&1

20101128: install to home on jet

cd
cd w21/

db-4.8 bsddb dance at wxmap2_jtwc RHE 5.6

recipe on wxmap2_jtwc

wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 169 > uname -a
Linux wxmap2_jtwc 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 i686 i386 GNU/Linux
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 170 > pg /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

c.w2.a.s
cd python

tar -xvf db-4.8.30.tar.gz
../dist/configure --prefix=/w21/app/lib/db4.8
make install

tar -zxvf bsddb3-5.0.0.tar.gz
python setup.py --berkeley-db=/w21/app/lib/db4.8 build
python setup.py --berkeley-db=/w21/app/lib/db4.8 install

cd ../../
cd ../lib/
cp db4.8/lib/* .

vi /dat/w21/app/python/lib/python2.7/dbhash.py
change bsddb to bsddb3

mdk -S 11p -L # works




Comments on python shelve module incompatibility

* Send this page to somebody
* Print this page

If you are experiencing problems with reading python shelve created in different machines with different CPU architecture, or different python version in a single machine, you may try upgrading Berkeley DB and its python modules in order to solve this problems.

* Error Messages: Standard Error Messages coming from shelve incompatibility in linux environment is as follows.

>>> import shelve
>>> my = shelve.open('my.shelve', 'r')
Traceback (most recent call last):
File "", line 1, in ?
File "/home/deepreds/lib/python2.4/shelve.py", line 230, in open
return DbfilenameShelf(filename, flag, protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/shelve.py", line 212, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "/home/deepreds/lib/python2.4/dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "/home/deepreds/lib/python2.4/bsddb/__init__.py", line 293, in hashopen
d.open(file, db.DB_HASH, flags, mode)
bsddb._db.DBInvalidArgError: (22, 'Invalid argument -- my.shelve: unsupported hash version: 8')

* Types of hash version in shelve module and compatibilities

As of today, shelve module use three kinds of Berkeley DB hash depending on your os platforms. Old version is Berkeley DB Hash from old version of Berkeley DB, bsd185. In this case, hash version is 2. Microsoft Windows may have hash version 2 shelve module regardless of python version. Python 2 or higher can have different version of hash depending on how the python packages are compiled. If you download python source distribution and compile on your system, hash version may be 7 or 8. All python already installed in os distribution packages use hash version 8. Only higher hash version can read lower hash files. If you want to check hash version of your python, use file command.

$ file my.shelve
my.shelve: Berkeley DB (Hash, version 8, native byte-order)

* General solution is to upgrade Berkeley DB and its python module

If you are using hash version 2 or 7, you may experience problems with reading shelve files from different python version or different os platforms. In this case, general solution is to upgrade Berkeley DB and its python module. Procedures how to upgrade hash version in shelve modules is as follows.

1. Install new Berkeley DB: You can download latest version of Berkeley DB at http://www.sleepycat.com/products/bdb.html, gunzip and untar the tar.gz and go to build_unix directory (for unix/linux environment) and run "../dist/configure". After configuring is done, you can install the package by standard source installation procedure, "make" and "make install". Typically, Berkeley DB will be installed in /usr/local. Unless you are familiar with unix environment, don't change this target directory.

2. Install Berkeley DB python module: You can download latest version of bsddb (Berkeley DB) python module at http://pybsddb.sourceforge.net/. Installation is standard python procedure, "python setup.py install". This module will recognize pre-installed Berkeley DB in /usr/local directory.

3. Forcing dbhash.py to use new version of Berkeley DB python module: shelve modules uses dbhash module, dbhash modules use bsddb module. Our new version of bsddb3 will be installed in site-packages directory. We can use new version of db hash if we modify dbhash.py file. Typically, dbhash.py file is located in ~/lib/python2.X directory (X = versio number). Name of Berkeley DB module installed in step 2 is bsddb3. Open dbhash.py and change all dsddb into bsddb3. Then, shelve module will use bsddb3 instead of bsddb. DB hash version of bsddb3 is 8 and can read all shelve files.

import sys
try:
import bsddb3
except ImportError:
# prevent a second import of this module from spuriously succeeding
del sys.modules[__name__]
raise

__all__ = ["error","open"]

error = bsddb3.error # Exported for anydbm

def open(file, flag = 'r', mode=0666):
return bsddb3.hashopen(file, flag, mode)

* Cross-endian compatibility

Mac OSX (G series) use big-endian as default and can emulate little-endian. Intel Mac OSX, Intel, and HP machines use little-endian. Generally speaking, binary files saved by different endian may not be readable. If you are reading shelve files generated in Mac OSX G4 in Intel Linux system, generally python may give errors. Cross-endian reading/writing binary files are not recommended.

Created by deepreds
Last modified 2006-07-26 03:53 PM