wxmap2 kaze install

mod ~/.wxmaprc to hard path $W2

shift ngpc -> ngp

20111009
========

added $ngpModel='ngp2' switch

added NgpModel='ngpc' to w2switches.py

wxmap.py add NgpModel switch to HtmlWxmapMain()

mod wxmain.css
wxmap.main.template.txt -> wxmap.main.template.ngpc.txt

20110802
========

files touched

prc/web/wxmap.htm.pl -- bug in MMMaking line
prc/lib/perl/wxmap.env.pl -- lines with #ngpc and #ngp2

prc/lib/python/w2.py -- Nwp2Models :: ngpc->ngp2
prc/lib/python/wxmap.py -- wxModels :: ngpc->ngp; dtgngpc -> dtgngp -- general

web/template

kishou(DARWIN)[W2:ESRL]: /w21/web/template 208 > svn add wxmap.main.template.ngp*.txt
A wxmap.main.template.ngp.txt -- ngp
A wxmap.main.template.ngpc.txt -- ngpc

ln -s wxmap.main.template.ngp.txt wxmap.main.template.txt

/dat inventory

./cagips :: 3,375,220
./conf :: 24
./crons :: 148
./grads :: 58,796
./ldmincoming :: 616,100
./lib :: 428
./nwp :: 97,523,776
./opengrads2.0 :: 25,252
./rdata :: 8
./tc :: 2,280,160
./w21 :: 150,620,212 /w21/ -> /dat/w21 /w21/dat/nwp2 -> /dat1
./windclimo :: 15,670,824
------------------------------------------------------------------
Total.................. :: 270,170,976
Files.................. :: 28
Dirs................... :: 270,170,948

/w21/

./app :: 5,000,864
./bin :: 3,816
./dat :: 140,750,728
./etc :: 1,376
./evt :: 105,444
./lib :: 115,552
./log :: 103,504
./plt :: 2,288
./prc :: 103,700
./prj :: 1,685,244
./src :: 539,100
./web :: 2,210,836
------------------------------------------------------------------
Total.................. :: 150,622,688
Files.................. :: 236
Dirs................... :: 150,622,452



/w21/dat
c27wxpmpprlh01(LINUX)[W2:JTWC]: /dat/w21/dat 212 > usage.py
/dat/w21/dat......................................................
------------------------------------------------------------------
./climo :: 556,404
./geog :: 50,248
./nwp :: 126,984
./ocean :: 508,804
./tc :: 139,571,796
------------------------------------------------------------------
Total.................. :: 140,814,248
Files.................. :: 12
Dirs................... :: 140,814,236

/dat/w21/dat/tc
/dat/w21/dat/tc...................................................
------------------------------------------------------------------
./DSs :: 14,157,052
./adeck :: 3,610,716
./bdeck :: 214,436
./bt :: 160,008
./carq :: 1,460,568
./cira :: 25,081,176
./climo :: 455,584
./cmc :: 15,125,340
./com :: 5,976
./ecmwf :: 190,276
./edeck :: 9,592
./fdeck :: 17,048
./jtwc :: 107,340
./mdeck :: 145,492
./names :: 4,824
./ncep :: 20,065,404
./nhc :: 201,408
./reftrk :: 19,528
./tcanal :: 38,994,904
./tcbog :: 12,128
./tcdiag :: 49,752
./tceps :: 19,216,708
./tcgen :: 39,088
./tcvitals :: 10,912
./ukmo :: 19,072
./vdeck :: 218,284
------------------------------------------------------------------
Total.................. :: 139,592,624
Files.................. :: 8
Dirs................... :: 139,592,616

/dat /dat1 cleanup

find /dat/cagips/datadir/*/. -mtime +3 -exec rm {} \;
find /dat1/nwp2/*/*/. -type d -name "20????????" -mtime +10 -exec rm -f -r {} \;
find /dat1/nwp2/w2flds/dat/*/. -type d -name "20????????" -mtime +10 -exec rm -f -r {} \;

w21 install owen machine dalek

cd /w21

svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/etc etc
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc prc
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/src src
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prj prj
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/lib/wgrib lib/wgrib
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/app/src app/src
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/web/config web/config

cd etc
make.ln.sh

w2.tc.ops.dat.py -- domtcswa option

20110714

wget to nesdis for mtcswa hangs... turned off for now...

w2.tc.rsync.mirror.jtwc.abdeck.2.local.py

issue with pexpect -- have a sshcmd mod to run in non-interactive mode... using pzal for now until i can test. in w2.tc.ops.dat.py

adding ocn/ohc/ww3 ocean fields to ew2

had to touch the following to do the data handling by w2.esrl.nwp2.py

w2.esrl.nwp.py
w2.fld.wgrib.filter.py
M2.py
w2.py
WxMAP2.py

added to crontab:


# 20110712 -- turn on :: fnmoc ocean data -- ocn,ohc,ww3
21 7-10,19-21 * * * $runcmd "$w2pdir/wxmap2/w2.esrl.nwp2.py ops12 ocn -D" >> /ptmp/log.w2.esrl.nwp2.py.OCN 2>&1
23 7-10,19-21 * * * $runcmd "$w2pdir/wxmap2/w2.esrl.nwp2.py ops12 ohc -D" >> /ptmp/log.w2.esrl.nwp2.py.OHC 2>&1
27 7-10,19-21 * * * $runcmd "$w2pdir/wxmap2/w2.esrl.nwp2.py ops12 ww3 -D" >> /ptmp/log.w2.esrl.nwp2.py.WW3 2>&1

disk space usage

total ~ 19 Gb / day
===================

models ~ 14 Gb / day
====================

name center source availability IN: size /12h w2flds /12h
---- --------- ---------- ----------- ------------- -----------

cmc2 cmc.ca ncep.ccs 00/12 230 Mb 90 Mb
ukm2 ukmo ukmo 00/06/12/18 915 Mb 915 Mb
ecm2 ecmwf ncep.ccs 00/12 360 Mb 250 Mb
gfs2 ncep gfs esrl.public 00/06/12/18 ---- 580 Mb
fim8 esrl fim esrl.public 00/12 ---- 715 Mb
ngp2 nogaps ncep.ccs 00/12 90 Mb 40 Mb
gfsk gfs+enkf esrl.jet 00/06/12/18 ---- 1080 Mb

ngpc nogaps cagips 00/06/12/18 1730 Mb 1730 Mb
gfsc ncep gfs cagips 00/06/12/18 420 Mb 360 Mb
jmac jma gsm cagips 00/12 360 Mb 120 Mb
ukmc ukmo um cagips 00/12 80 Mb 80 Mb
-------- --------
~ 8.0 Gb 6.0 Gb


web: ~ 5 Gb / day
==================

plots: ~ 18 Gb

./plt_ecmwf_ecm :: 2,011,508
./plt_esrl_fim :: 2,110,589
./plt_fnmoc_ngpc :: 4,269,503
./plt_loop :: 2,842,838
./plt_ncep_gfs :: 5,103,530
./plt_ukmo_ukm :: 3,163,470

tcs: ~ 2 Gb
./tc :: 1,748,712

html: ~ 4.5 Gb

./web_ecm :: 639,020
./web_fim :: 673,519
./web_gfs :: 1,573,571
./web_ngpc :: 1,350,290
./web_ukm :: 966,201


input data
==========


model fields:

'w2flds' -- fields used for all wxmap2 processing


------------------------ wxmap2.jtwc ----------------------

ngpc: 00/06/12/18Z :: 1730 Mb / 12h
=====

./2011061800 :: 952,444
./2011061806 :: 778,068

jmac: 00/12Z :: 80 Mb / 12h
=====

./2011062412 :: 23,616
./2011062500 :: 48,256
./2011062512 :: 17,856
./2011062600 :: 51,020
./2011062612 :: 57,568
./2011062700 :: 8,244
./2011062712 :: 52,048
./2011062800 :: 15,472

ukmc: 00/12Z :: 160 Mb / 12h
=====

./2011062512 :: 78,832
./2011062600 :: 78,904
./2011062612 :: 68,800
./2011062700 :: 79,104
./2011062712 :: 78,960
./2011062800 :: 79,104
./2011062812 :: 79,032


gfsc: 00/06/12/18Z :: 320 Mb / 12h
=====

./2011062700 :: 179,316
./2011062706 :: 147,524
./2011062712 :: 24,504
./2011062718 :: 179,052
./2011062800 :: 179,036
./2011062806 :: 179,260
./2011062812 :: 111,452
./2011062818 :: 178,964

------------------------ wxmap2.esrl ----------------------

ecm2: 00/12Z :: 252 Mb / 12h
====

./2011062600 :: 251,788
./2011062612 :: 251,768
./2011062700 :: 251,752
./2011062712 :: 251,772
./2011062800 :: 251,764
./2011062812 :: 251,756

gfs2: 00/06/12/18Z :: 580 Mb / 12h
=====

./2011062700 :: 290,812
./2011062712 :: 289,476
./2011062800 :: 290,692
./2011062812 :: 292,576

fim8: 00/12Z :: 715 Mb / 12h
=============

./2011062600 :: 715,352
./2011062612 :: 715,120
./2011062700 :: 715,616
./2011062712 :: 716,120
./2011062800 :: 716,788
./2011062812 :: 716,740

gfsk: 00/06/12/18Z ::

cmc2: 00/12 90 / 12h

Total.................. :: 113,664 00
Total.................. :: 69,824 12
Files.................. :: 68,472






incoming fields
===============

--------------- wxmap2.jtwc ------------------

ngpc = w2flds :: 00/06/12/18Z :: 1730 Mb / 12h

jmac: :: 00/12Z :: ~360 Mb / 12h

./2011062600 :: 109,348
./2011062612 :: 328,960
./2011062700 :: 226,376
./2011062712 :: 158,564
./2011062800 :: 187,032
./2011062812 :: 166,152

gfsc: :: 00/06/12/18Z :: ~430 Mb / 12h

./2011062606 :: 212,480
./2011062612 :: 212,904
./2011062618 :: 212,824
./2011062700 :: 212,888
./2011062706 :: 212,892
./2011062712 :: 212,920
./2011062718 :: 212,864
./2011062800 :: 212,748
./2011062806 :: 213,040
./2011062812 :: 212,664
./2011062818 :: 212,704

ukmc = w2flds :: 00/12Z :: 160 Mb / 12h


--------------- wxmap2.esrl ------------------

cmc2: 00/12Z :: 230 Mb / 12h

./2011062512 :: 101,124
./2011062600 :: 133,776
./2011062612 :: 101,124
./2011062700 :: 133,776
./2011062712 :: 101,124
./2011062800 :: 133,776
./2011062812 :: 101,124


ecm2: 00/12Z :: 360 Mb / 12h

./2011062712 :: 361,348
./2011062800 :: 361,348
./2011062812 :: 361,348

gfs2: 00/06/12/18Z :: very big -- off /public/data

fim8: 00/12Z :: very big -- off /public/data

ngp2: 00/12Z :: 90 Mb / 12h

./2011062600 :: 87,192
./2011062612 :: 87,168
./2011062700 :: 87,108
./2011062712 :: 87,136
./2011062800 :: 87,224
./2011062812 :: 87,240

ukm2: 00/06/12/18Z ::

./2011062700 :: 575,396
./2011062706 :: 334,252
./2011062712 :: 575,396
./2011062718 :: 333,772
./2011062800 :: 576,744
./2011062806 :: 333,676
./2011062812 :: 576,840
./2011062818 :: 333,580

gfsk

./2011062700 :: 582,340
./2011062706 :: 0
./2011062712 :: 0
./2011062718 :: 582,564
./2011062800 :: 582,556
./2011062806 :: 582,500
./2011062812 :: 582,616
./2011062818 :: 581,888

merge esrl.jtwc to trun in /dat1/w21

242 1:44 svn info
243 1:45 svn co http://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc prcN
245 1:50 svn update prcN
247 1:51 svn co http://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/etc etcN
248 1:51 svn info etcN
249 1:51 svn info prcN
251 1:52 cd etcN
255 1:52 cp ../etc/run.cron.* .
258 1:52 svn add run.cron.*
261 1:53 cp ../etc/*make.ln*.sh .
262 1:53 svn add make.ln.*sh
263 1:53 cp make.ln.ESRL.sh make.ln.ESRL.JTWC.sh
268 1:55 mv etc etcO
269 1:55 ln -s etcN etc
279 1:56 mv src srcO
281 1:56 svn co http://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/src srcN
282 2:02 find prc/. -name="*.x"
283 2:02 find prc/. -name "*.x"
298 2:04 cd ../../srcN/tcnavytrk/
301 2:04 m
305 2:04 find prc/. -name "*.x"
306 2:05 cd srcN/
307 2:05 cd tcgfdltrk/
309 2:05 cd grbindex
311 2:05 m
313 2:05 cd ../gettrk_gen/
314 2:05 m
317 2:05 la
318 2:06 find prc/. -name "*.x"

349 2:10 cp prc/lib/python/*JT*py prcN/lib/python/.

361 2:15 cp prc/lib/python/wxmap.py prcN/lib/python/wxmap.JTWC.py

367 2:17 cp prc/lib/perl/wxmap.env.pl prcN/lib/perl/wxmap.env.JTWC.pl

371 2:18 find prcN/. -name "*JTW*" | g -v svn

381 2:28 cp ../../prc/wxmap2/w2.jtwc.nwp2.py .
382 2:28 svn add w2.jtwc.nwp2.py

386 2:29 cp ../../prc/flddat/w2.fld.wgrib.filter.py ../flddat/w2.fld.wgrib.filter.JTWC.py

387 2:30 svn add ../flddat/w2.fld.wgrib.filter.JTWC.py

209 2:01 vi make.ln.ESRL.JTWC.sh
210 2:01 make.ln.ESRL.JTWC.sh

ecm2 down 2011051800-2011051900

problem with css and vapor
jeff whitaker did by hand and restarted cron
cause was cirrus going down

setup ratchet/clank

put on /data/amb/users/fiorino/w21
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/etc etc
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc prc
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/web/config web/config

on kishou(linux):

cd /w21/app/lib

scp libdb* fiorino@ratchet:/home/amb/fiorino/w21/app/lib/.
scp -r wgrib fiorino@ratchet:/home/amb/fiorino/w21/app/lib/.

cd /w21/bin

scp * fiorino@ratchet:/home/amb/fiorino/w21/bin/.

dtg6/wgrib2 don't work on clank:

Linux ratchet 2.6.16.60-0.77.1-smp #1 SMP Mon Mar 14 16:54:35 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

kishou(LINUX)[W2:ESRL]: /home/fiorino 293 > uname -a
Linux kishou.fsl.noaa.gov 2.6.18-238.9.1.el5 #1 SMP Tue Apr 12 18:10:13 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux


Last login: Mon May 2 21:56:43 2011 from kishou.fsl.noaa.gov

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

System Info:
------------
Hostname: ratchet.fsl.noaa.gov
System: Sun V40z
Arch: AMD Opteron
OS: SuSE Linux Enterprise Server 10 SP3

System Support:
---------------
Primary: Scott Buennemeyer Ph: x6894
Secondary: Gary Fisher Ph: x6754

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


scott will upgrade to suse server 11 sp1 in a few days...

setup of public wxmap2 on hfip/wxmap2

/data/rapb/projects/hfip/wxmap2

kishou(DARWIN)[W2:ESRL]: /data/rapb/projects/hfip/wxmap2
svn co https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/web/config config
ishou(DARWIN)[W2:ESRL]: /data/rapb/projects/hfip/wxmap2/config
make.ln.sh

227 15:38 cp /w21/web/template/wxmap.main.template.txt .
228 15:38 cp wxmap.main.template.txt wxmap.main.template.PUB.txt

wxmap.main.template.PUB.txt


392 17:58 ln -s /w3/rapb/wxmap2/plt_esrl_fim .
394 17:59 ln -s /w3/rapb/wxmap2/plt_ncep_gfs/ .
396 17:59 ln -s /w3/rapb/wxmap2/plt_ncep_gfs .

c.w2.w

mod

-P for dopublic in .py
w2.web.py
w2.web.main.py

mod
lib/perl/wxmap.env.Pub.pl

kishou(DARWIN)[W2:ESRL]: /w21/prc/web 247 > la *.Pu*pl
-rwxr-xr-x 1 fiorino frdrapb 55531 Apr 27 19:35 wxmap.htm.Pub.pl
-rwxr-xr-x 1 fiorino frdrapb 34754 Apr 27 19:37 wxmap.movie.js.Pub.pl
-rwxr-xr-x 1 fiorino frdrapb 3181 Apr 27 20:07 wxmap.web.archive.Pub.pl

mkdir .htm exclusively for public site

426 19:22 mkdir web_fim
429 19:22 mkdir web_gfs
432 19:22 mkdir web_ngpc


w2.web.py cur -P drives all the .Pub.pl

w2.mss.nwp2.py as jtw2 -- clean up of jtwc cagips to ersl.mss

20110406
========


/dev/disk3s2 698Gi 632Gi 67Gi 91% /Volumes/dat1
w2.mss.nwp2.py cur12-d10 all

NB: need to do w2.mss.nwp2.py cur12-d10 all -W


20110314
========

/dev/disk0s2 698Gi 638Gi 60Gi 92% /Volumes/dat1

w2.mss.nwp2.py 2011030100 all

jtw2
jw2 2011020800.cur-12.6 ngpc,gfsc -t and jmac,ukmc to clean off cagips

231 15:28 w2.mss.nwp2.py 2011031000 all
233 17:20 w2.mss.nwp2.py 2011031000 all -W

/dev/disk0s2 698Gi 362Gi 336Gi 52% /Volumes/dat1





20110222
========

w2.mss.nwp2.py 2011020100 all
w2.mss.nwp2.py 2011020100 all -W

20110201
========
brought usage down to 68%

did w2.mss.nwp2.py 2011013000 gfsc,ngpc,ukmc,jmac -W to clean more....

kishou(DARWIN)[W2:ESRL]: /dat1/w21/dat 344 > usage.py .
/Volumes/dat1/w21/dat.............................................
------------------------------------------------------------------
./climo :: 3,613,400
./geog :: 49,596
./nwp2 :: 224,554,312
./ocean :: 489,484
./pr :: 8,102,236
./tc :: 35,013,144
------------------------------------------------------------------
Total.................. :: 271,822,172
Files.................. :: 0
Dirs................... :: 271,822,172

/Volumes/dat1/w21/dat/nwp2........................................
------------------------------------------------------------------
./fnmoc :: 9,861,816
./jma :: 1,077,972
./ncep :: 2,354,100
./ukmo :: 408,476
./w2flds :: 209,776,460
------------------------------------------------------------------


dat1 got 100% full

jtw2
w2.mss.nwp2.py 2011013000 gfsc,ngpc,ukmc,jmac

go mss all data <= 2010013000

tuning crontab @ jtwc

20110329
========

added keeplive.py script to monitor state of cagips/dataflow and stopping nmci from loging out for inactivity, problems over the weekend:

l2 cur-d3.cur gfsc
gfsc 2011032612 180 5.92 6 <--- low data count nfields: 57
gfsc 2011032618 180 5.27 12 <--- low data count nfields: 57
gfsc 2011032700 180 5.93 12 <--- low data count nfields: 57
gfsc 2011032706 180 5.26 9 <--- low data count nfields: 57
gfsc 2011032712 180 5.92 3 <--- low data count nfields: 57
gfsc 2011032718 999 999.90 0 <--- NO DATA localdir: /dat/w21/dat/nwp2/ncep/gfsc/2011032718
gfsc 2011032800 999 999.90 0 <--- NO DATA localdir: /dat/w21/dat/nwp2/ncep/gfsc/2011032800
gfsc 2011032806 999 999.90 0 <--- NO DATA localdir: /dat/w21/dat/nwp2/ncep/gfsc/2011032806
gfsc 2011032812 180 5.25 3 <--- low data count nfields: 57
gfsc 2011032818 180 5.93 29 <--- low data count nfields: 57
gfsc 2011032900 180 4.92 3 <--- low data count nfields: 57
gfsc 2011032906 180 5.92 17 <--- low data count nfields: 57
gfsc 2011032912 999 999.90 0 <--- NO DATA localdir: /dat/w21/dat/nwp2/ncep/gfsc/2011032912

svn merge trunk -> jtwc branch on kishou

20110203
========

w2.tc.ops.dat.py fouled used trunk version, set wjet=0
installed beautifulsoup for tigge/ecmwf

20110203
========

kishou(DARWIN)[W2:JTWC]: /dat1/w21/prc/tcdat

svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/tcdat .
xxdiff-subversion issue?

had to do a 'p' to postpone conflict resolution
then merge by hand using xxdiff

svn status -q | g 'C '

C w2.tc.bt.mdeck.final.py
C w2.tc.dss.vdeck.anl.py
C w2.tc.dss.adeck.py
C w2.tc.dss.adeck.anl.py
C w2.tc.ops.dat.py

for each C file:

xxdiff w2.tc.ops.dat.py.working w2.tc.ops.dat.py.merge-left.r535 w2.tc.ops.dat.py.merge-right.r683
global->merge
file->save as... w2.tc.ops.dat.py
svn resolved w2.tc.ops.dat.py
svn status

svn ci


20110203
========

WxMAP2 got out of sync with jtwc branch ahead in some ways, updated by hand from /w21/prc/lib/python/ and edited at jtwc


/dat1/w21/prc

svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py


206 0:26 svn info
207 0:27 svn merge --dry-run -r 600:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc
210 0:28 svn log --stop-on-copy
211 0:29 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc
214 0:35 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py
215 0:35 svn merge --dry-run -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
216 0:35 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
217 0:36 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
218 0:36 svn diff https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
223 0:37 svn diff https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
224 0:37 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/TC.py lib/python/TC.py
225 0:38 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
226 0:38 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
228 0:39 svn diff lib/python/M.py
229 0:40 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
232 0:42 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
235 0:43 svn pdate lib/python/w2.py
236 0:43 svn update lib/python/w2.py
239 0:43 svn update lib/python/w2.py
241 0:43 svn cleanup
242 0:43 svn update lib/python/w2.py
243 0:44 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
246 0:44 svn update lib/python/w2.py
253 0:47 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
255 0:47 svn info lib/python/w2.py
258 0:49 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
259 0:49 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/M.py lib/python/M.py
264 0:51 svn info
265 0:51 svn log --stop-on-copy lib/python/VT.py
268 0:52 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/VT.py lib/python/VT.py
270 0:53 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/AD.py lib/python/AD.py
272 0:54 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/MD.py lib/python/MD.py
275 0:56 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2.py lib/python/w2.py
282 0:58 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/FM.py lib/python/FM.py
284 0:58 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
285 1:00 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/ lib/python/
286 1:01 svn merge -r 535:HEAD https://wxmap2.svn.sourceforge.net/svnroot/wxmap2/trunk/prc/lib/python/w2switches.py lib/python/w2switches.py
295 1:02 h | g svn
kishou(DARWIN)[W2:JTWC]: /dat1/w21/prc 296 > h | g xxdiff
257 0:48 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
261 0:50 xxdiff /w21/prc/lib/python/M.py lib/python/M.py
267 0:51 xxdiff /w21/prc/lib/python/VT.py lib/python/VT.py
269 0:52 xxdiff /w21/prc/lib/python/VT.py lib/python/VT.py
273 0:55 xxdiff /w21/prc/lib/python/MD.py lib/python/MD.py
274 0:55 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
276 0:56 xxdiff /w21/prc/lib/python/w2.py lib/python/w2.py
278 0:56 xxdiff /w21/prc/lib/python/ATCF.py lib/python/ATCF.py
279 0:56 xxdiff /w21/prc/lib/python/G2.py lib/python/G2.py
280 0:57 xxdiff /w21/prc/lib/python/TCw2.py lib/python/TCw2.py
281 0:57 xxdiff /w21/prc/lib/python/FM.py lib/python/FM.py
283 0:58 xxdiff /w21/prc/lib/python/FM.py lib/python/FM.py
287 1:01 xxdiff /w21/prc/lib/python/w2switches.py lib/python/w2switches.py
296 1:02 h | g xxdiff

svn at jtwc; add cagips proc to crontab

Angelo got svn to work...

wxmap2_jtwc(LINUX)[W2]: /dat/w21 125 > crl
#------------------------------------------------------
#
# jtwc
#
#------------------------------------------------------
#

runcmd='/w21/run.cron.tcsh'
w2pdir='/w21/prc'

# 20100804 -- turn on :: mirror of oisst weekly grib
32 4 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.cpc.oisst.py" >> /dat/ptmp/log.2.fld.wget.mirror.cpc.oisst.py.CPC.OISST 2>&1

# 20090203 -- checked out wxmap2 -> w21 and just src/opengrads src/python; mod .alias .tcshrc .envrc .pythonrc .pathrc
# 200412 ---------------- TCs tttttttttttttttttcccccccccccccccccccccc
19,49 0-23 * * * $runcmd "$w2pdir/tcdat/w2.tc.ops.dat.py ops" >> /dat/ptmp/log.w2.tc.ops.dat.py.TCOPSHF 2>&1

# 20101009 -- wget gfs goes sat image 1deg files from ftpprd.ncep.noaa.gov
23,57 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/flddat/w2.fld.wget.mirror.gfs.stbgoes.py cur" >> /dat/ptmp/log.w2.fld.wget.mirror.gfs.stbgoes.py.GFS2 2>&1

# 20101009 -- turn on :: qmorph processing
01-59/20 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.py cur" >> /dat/ptmp/log.w2.fld.qmorph.py.CPC 2>&1

# 20101009 -- turn on :: qmorph products processing
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.qmorph.CPC 2>&1
15,45 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.qmorph.products.py.cmorph.CPC 2>&1

# 20101009 -- turn on :: qmorph products processing
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py ops6 -S qmorph" >> /dat/ptmp/log.w2.fld.qmorph.global.products.py.qmorph.CPC 2>&1
30 * * * * $runcmd "$w2pdir/flddat/w2.fld.pr.qmorph.global.products.py cur-24 -S cmorph" >> /dat/ptmp/log.w2.fld.cmorph.global.products.py.cmorph.CPC 2>&1

00 */12 * * * find /dat/cagips/datadir/WXMAP_AOI -type f -mtime +9 -ls -exec mv {} /dat1/cagips/old_files/ > /tmp/mv.old.cagips.log 2>&1

# 20110203 -- processing the cagips models

21 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 gfsc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.GFSC 2>&1
41 3-6,9-12,13-18,21-23,0 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops6 ngpc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.NGPC 2>&1
31 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 ukmc -t" >> /dat/ptmp/log.w2.JT.nwp2.py.UKMC 2>&1
51 5-11,19-23 * * * $runcmd "$w2pdir/wxmap2/w2.jtwc.nwp2.py ops12 jmac -t" >> /dat/ptmp/log.w2.JT.nwp2.py.JMAC 2>&1

20101128: install to home on jet

cd
cd w21/

db-4.8 bsddb dance at wxmap2_jtwc RHE 5.6

recipe on wxmap2_jtwc

wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 169 > uname -a
Linux wxmap2_jtwc 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 i686 i386 GNU/Linux
wxmap2_jtwc(LINUX)[W2]: /dat/w21/lib 170 > pg /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

c.w2.a.s
cd python

tar -xvf db-4.8.30.tar.gz
../dist/configure --prefix=/w21/app/lib/db4.8
make install

tar -zxvf bsddb3-5.0.0.tar.gz
python setup.py --berkeley-db=/w21/app/lib/db4.8 build
python setup.py --berkeley-db=/w21/app/lib/db4.8 install

cd ../../
cd ../lib/
cp db4.8/lib/* .

vi /dat/w21/app/python/lib/python2.7/dbhash.py
change bsddb to bsddb3

mdk -S 11p -L # works




Comments on python shelve module incompatibility

* Send this page to somebody
* Print this page

If you are experiencing problems with reading python shelve created in different machines with different CPU architecture, or different python version in a single machine, you may try upgrading Berkeley DB and its python modules in order to solve this problems.

* Error Messages: Standard Error Messages coming from shelve incompatibility in linux environment is as follows.

>>> import shelve
>>> my = shelve.open('my.shelve', 'r')
Traceback (most recent call last):
File "", line 1, in ?
File "/home/deepreds/lib/python2.4/shelve.py", line 230, in open
return DbfilenameShelf(filename, flag, protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/shelve.py", line 212, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback, binary)
File "/home/deepreds/lib/python2.4/anydbm.py", line 83, in open
return mod.open(file, flag, mode)
File "/home/deepreds/lib/python2.4/dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
File "/home/deepreds/lib/python2.4/bsddb/__init__.py", line 293, in hashopen
d.open(file, db.DB_HASH, flags, mode)
bsddb._db.DBInvalidArgError: (22, 'Invalid argument -- my.shelve: unsupported hash version: 8')

* Types of hash version in shelve module and compatibilities

As of today, shelve module use three kinds of Berkeley DB hash depending on your os platforms. Old version is Berkeley DB Hash from old version of Berkeley DB, bsd185. In this case, hash version is 2. Microsoft Windows may have hash version 2 shelve module regardless of python version. Python 2 or higher can have different version of hash depending on how the python packages are compiled. If you download python source distribution and compile on your system, hash version may be 7 or 8. All python already installed in os distribution packages use hash version 8. Only higher hash version can read lower hash files. If you want to check hash version of your python, use file command.

$ file my.shelve
my.shelve: Berkeley DB (Hash, version 8, native byte-order)

* General solution is to upgrade Berkeley DB and its python module

If you are using hash version 2 or 7, you may experience problems with reading shelve files from different python version or different os platforms. In this case, general solution is to upgrade Berkeley DB and its python module. Procedures how to upgrade hash version in shelve modules is as follows.

1. Install new Berkeley DB: You can download latest version of Berkeley DB at http://www.sleepycat.com/products/bdb.html, gunzip and untar the tar.gz and go to build_unix directory (for unix/linux environment) and run "../dist/configure". After configuring is done, you can install the package by standard source installation procedure, "make" and "make install". Typically, Berkeley DB will be installed in /usr/local. Unless you are familiar with unix environment, don't change this target directory.

2. Install Berkeley DB python module: You can download latest version of bsddb (Berkeley DB) python module at http://pybsddb.sourceforge.net/. Installation is standard python procedure, "python setup.py install". This module will recognize pre-installed Berkeley DB in /usr/local directory.

3. Forcing dbhash.py to use new version of Berkeley DB python module: shelve modules uses dbhash module, dbhash modules use bsddb module. Our new version of bsddb3 will be installed in site-packages directory. We can use new version of db hash if we modify dbhash.py file. Typically, dbhash.py file is located in ~/lib/python2.X directory (X = versio number). Name of Berkeley DB module installed in step 2 is bsddb3. Open dbhash.py and change all dsddb into bsddb3. Then, shelve module will use bsddb3 instead of bsddb. DB hash version of bsddb3 is 8 and can read all shelve files.

import sys
try:
import bsddb3
except ImportError:
# prevent a second import of this module from spuriously succeeding
del sys.modules[__name__]
raise

__all__ = ["error","open"]

error = bsddb3.error # Exported for anydbm

def open(file, flag = 'r', mode=0666):
return bsddb3.hashopen(file, flag, mode)

* Cross-endian compatibility

Mac OSX (G series) use big-endian as default and can emulate little-endian. Intel Mac OSX, Intel, and HP machines use little-endian. Generally speaking, binary files saved by different endian may not be readable. If you are reading shelve files generated in Mac OSX G4 in Intel Linux system, generally python may give errors. Cross-endian reading/writing binary files are not recommended.

Created by deepreds
Last modified 2006-07-26 03:53 PM