=============== BatchProcessing =============== Imagine you have collected a mass of data. You want to peaksearch it, using several machines, and then index all the scans. Here is one approach. These are notes from a worked example. You might need to either try to understand the code/method or wait for a more user friendly implementation Working out what the scans were =============================== This is an example script which will go through the headers getting image numbers and header information (svn version only for now):: more getheader.py from ImageD11 import opendata import glob, sys fl = glob.glob(sys.argv[1]) # list of files to process output = [] for name in fl: if name.find("edf") > -1: # only process edf files h = opendata.edfheader(name) num = opendata.getnum(name) //**# this needs svn ImageD11.opendata**// for n in sys.argv[2:]: try: name += " " + str(n) + " " + h[n] + " " except: name += " " + str(n) + " " + " " + " " output.append((num,name)) output.sort() for o in output: print o[0],o[1] Usage would be:: python getheader.py "mydatafiles*.edf" samz Omega > headerinfo.txt The quotes around the first command line argument are critical to stop the shell expanding the "*". It will create a file with the image number, filename and header information you want. We will put the Omega entry as the last column and last argument on the command line. We now work out what the individual sweepscans were from this header information file. It is sorted by image number and we assume a scan corresponds to an increasing sequence of Omega angles:: % more getscans.py import sys njobs = 4 # number of shell scripts to create outfiles = [open("fable%d.sh"%(i+1),"w") for i in range(njobs)] last = -1e9 first = True job = 0 for line in open(sys.argv[1],"r").readlines(): items = line.split() num = int(items[0]) # image number om = float(items[-1]) # Omega angle if first: f = num first = False if om > last: last = om else: # make a job stem = "/data/opid11/external/uu123/mysample/" # your directory here # build a peaksearch.py command line com = "peaksearch.py " com += " -n %s "%(stem+"mysample") com += " -f %d "%(f) com += " -l %d "%(num) # we made bkg.edf via ImageD11/scripts/bgmaker.py com += " -d bkg.edf " com += " -D 20 " com += " -t 70 -t 520 -t 5020 " com += " -s /data/opid11/inhouse/Frelon4M/frelon4m.spline " # output to go in ImageD11/peaksearch subdirectory com += " -o %sImageD11/peaksearch/scan_%d.out "%(stem,job) com += " > %sImageD11/peaksearch/scan_%d.log "%(stem,job) com += "\n" outfiles[job%njobs].write("echo %d\n"%(job)) outfiles[job%njobs].write(com) job += 1 first = True last = -1e9 Assuming you have made a directory ImageD11 in with your sample to pick up the output should now have 4 shell scripts. Run each one on a different machine, the peaksearch output should end up in the ImageD11 directory. Work out how to index one scan ============================== Take the output from the peaksearch into the gui (probably the merge_txxx) file. Go through the transformation menu until you are happy you have nice parameters. Save the parameter file. Save the g-vectors and go through the indexing menu, finding grains. Once you are happy go to the Help menu and select history. Copy and paste this into a file. Edit the history file to take a command line argument ===================================================== Now edit the history file to replace the input and output file names to be a variable from the command line: eg:: % more history.py # Create objects to manipulate - they hold your data # from ImageD11 import peakmerge, indexing, transformer mypeakmerger = peakmerge.peakmerger() mytransformer = transformer.transformer() myindexer = indexing.indexer() # # Your work starts here: # **//import sys # added//** //**infile = sys.argv[1] # added**// mytransformer.loadfiltered( //**infile**// ) mytransformer.loadfileparameters( '/data/opid11/external/uu123/mysample/ImageD11/good.pars' ) mytransformer.compute_tth_eta( ) mytransformer.addcellpeaks( ) mytransformer.savegv( //**infile+'.gve' ) # add .gve extension**// myindexer.parameterobj.set_parameters( {'ds_tol': '0.01', 'minpks': '50', 'uniqueness': '0.5', 'hkl_tol': '0.05', 'eta_range': '10.0', 'ring_1': '3', 'wavelength': '0.29173', 'ring_2': '3', 'cosine_tol': '0.005'} ) myindexer.updateparameters( ) myindexer.loadpars( ) myindexer.readgvfile( //**infile+'.gve'**// ) myindexer.assigntorings( ) myindexer.find( ) myindexer.scorethem( ) myindexer.assigntorings( ) myindexer.saveubis( //**infile + '.ubi'**// ) //**# slow process, skip to just get ubi files**// //**# myindexer.saveindexing( infile + '.idx')**// Now run this on all you peaksearch outputs: bash % find . -name "*_merge_t500" -exec history.py {} \; And off it goes to make the ubi files for you Indexing a peaksearch output and filtering peaks at the same time:: # Example indexing script. Typically will be edited to match # specific experiments and sample (eg: which rings to use for indexing, # how many peaks make a grain, how to filter the peaks to remove "bad" # peaks coming from the peaksearch) # # call as idx.py [pars] [flt] [gve] [ubi] # input input output output # # Create objects to manipulate - they hold your data # import sys from ImageD11 import peakmerge, indexing, transformer mypeakmerger = peakmerge.peakmerger() mytransformer = transformer.transformer() myindexer = indexing.indexer() # # Your work starts here: # mytransformer.loadfiltered( sys.argv[2] ) mytransformer.loadfileparameters( sys.argv[1] ) mytransformer.compute_tth_eta( ) # Cut the high angle rings, so only tth less than 14 # ... could be an option mytransformer.colfile.filter( mytransformer.colfile.tth < 14 ) # ... you can filter on any column here # eg ... mytransformer.colfile.Number_of_pixels > 2 # ... etc mytransformer.compute_tth_eta( ) mytransformer.addcellpeaks( ) mytransformer.computegv( ) mytransformer.savegv( sys.argv[3] ) myindexer.readgvfile( sys.argv[3] ) myindexer.updateparameters( ) # Mostly defaults - wavelength is common to pars above myindexer.parameterobj.set_parameters( { 'ds_tol': '0.02', 'minpks': '30', 'max_grains': '100', 'uniqueness': '0.5', 'hkl_tol': '0.07', 'eta_range': '0.0', 'ring_1': '1', 'ring_2': '1', 'cosine_tol': '0.01'} ) myindexer.loadpars( ) myindexer.assigntorings( ) myindexer.find( ) myindexer.scorethem( ) # loop over minpks reducing from 40 to 20 - gets the best grains # with 40 peaks first for minpk in [ 40, 30, 20]: # loop over pairs of rings to use for generating orientations for p1, p2 in [ (0, 1), (2, 1), (3, 1) , (4, 1) , (0, 0), (2, 0), (3, 0) , (4, 0) ]: myindexer.updateparameters( ) myindexer.parameterobj.set_parameters( { 'ring_1': p1, 'ring_2': p2, 'minpks': minpk, } ) myindexer.loadpars( ) myindexer.assigntorings( ) myindexer.find( ) myindexer.scorethem( ) myindexer.saveubis( sys.argv[4] )