Processor 'eats' data

Sometimes a processor will take data but not output data to the data-stream for the tapes. Symptom: the total event rate goes down with the rate that the processor is handling.

The problem is in the CES memory associated with the event-filter formatter Currently it is (almost always) necessary to recycle the power of the VME crate of the affected processor to fix the problem. Since during the run that will stop the taping, this fix is best done when tapes are changed. Do the following:

VXI crate modifications

The VXI crates are modified for the sum-bus: The terminator on the back plane to the -2V on the ECL bus should be removed. This line is used as an analog sum bus in GAMMASPHERE.

Stuck bit scenario 1

Due to a pressed-in pin on one of the CES memories, we had a stuck bit. The CES dump looked as:
> cesd
03100000   8000     B  4741  4D4D  4153  5049  4552  4521  
03100010      0  8181     0  8181     0  8181   8D5     1  
03100020      0     3  6425  3665     0     1  FFFF  FFFF  
03100030   8000     F  4741  4D4D  4153  5049  4552  4521  
03100040   27BD   103     0     1     0     1     0     1  
03100050      0  8181     0  8181     0  8181   BF2     1  
03100060      0     3  6425  371F     0     1  FFFF  FFFF  
03100070   8000     F  4741  4D4D  4153  5049  4552  4521  
03100080   27BF   103     0     1     0     1     0     1  
03100090      0  8181     0  8181     0  8181   155     1  
031000A0      0     3  6425  37D9     0     1  FFFF  FFFF  
031000B0   8000     F  4741  4D4D  4153  5049  4552  4521  
031000C0   27BC   103     0     1     0     1     0     1  
031000D0      0  8181     0  8181     0  8181   45D     1  
031000E0      0     3  6425  3895     0     1  FFFF  FFFF  
031000F0   8000     F  4741  4D4D  4153  5049  4552  4521

Randy pointed out where the stuck bit was:

 > > cesd
 > 03100000   8000     B  4741  4D4D  4153  5049  4552  4521  
 > 03100010      0  8181     0  8181     0  8181   8D5     1  
 > 03100020      0     3  6425  3665     0     1  FFFF  FFFF  
 > 03100030   8000     F  4741  4D4D  4153  5049  4552  4521  
 > 03100040   27BD   103     0     1     0     1     0     1  

Stuck bit:             1           1           1           1

 > 03100050      0  8181     0  8181     0  8181   BF2     1  
 > 03100060      0     3  6425  371F     0     1  FFFF  FFFF  
                       1           1           1           1
 > 03100070   8000     F  4741  4D4D  4153  5049  4552  4521 
                       1           1           1           1 
 > 03100080   27BF   103     0     1     0     1     0     1
                       1           1           1           1  

Where should the LTM be?

The LTM must be in the slot following the last GeBGO board since it returns a token. If you leave an empty slot, this token will not be returned and the DAQ will hang.

Log books at ANL

The following log books are located at ANL


The rest must be at LBNL?

VXI readout bus

I can't dump spectra...

If you have trouble dumping spectra using the TCL/TK windows and it complains about Channel Access errors, try turning off the sender while you dump. The sender is able to load the network so hard that other processes have trouble getting their date over the network. It also helps to only specify the EFFs that actually have data rather than all of them (1-6). The safest way to get around this problem is to simply take the DAQ off and wait for all processors to finish processing. That way they have plenty of time to respond to the network requests and the sender is no longer blasting the network either.

On 2/11/98 a new version of the dump program was installed and conditions have to be really bad before this program gives up getting the EFF spectra. However, not all utilities have been upgraded (that is a big job) so you may still have stop or slow down the acquisition as described above.

Bad ge spectra

If your ge spectra looks like TAC calibration spectra - check that the gain coefficients are reasonable. If they are way off, - the EFFs can corrupt the data.

Please forward error messages --

To debug problems in the VXI and VME modules, it is important to have accurate failure/debug information. If e.g., an EFF processor dies, please to the following:

DAQ crashes...

If the DAQ crashes, here are a few pointers to what you can do:

Alarms to alpha-num beepers

GS has a number alarm monitores running that will send various alarm messages to alpha-numeric beepers.

The [1]germanium high temperature monitor, [2]ln-alarm monitor and [3]VXI crate monitors will send alarms to members of the

list and the [4]ln-fill monitor is using the list
Here is an example of how the .BEEPERADDR might look like:
In this case, two beepers are notified when there is an alarm. One E-mail to a regular account (via a mail alias) is sent as well.

Here is an example of how the .LNFILMON file might look:
People on this list will get the 'ln fill started', 'ln fill done' and 'tank fill done' information messages every time the GS array is filled.

Dialer Alarms

To change the dialer numbers do:
press store
press either 1, 2 or 3 
enter number
press store
You should hear a one second beep when the number is stored. It isn't really necessary to touch or change the message of the dialer -- it doesn't get through to the alpha numeric beepers anyway. On these beepers, the dialer generates a "020202" message or the like.

The dialer can be programmed for three numbers. You select those in the second line.

to disable a number: press store, press either 1, 2 or 3, press store. I.e., do NOT enter a number. Don't put in 0 since it will dial the operator!!!

The LN dial wires should connect to the outermost contacts of the alarm inputs.

NOTICE: Remember to put 4 in front of the beeper number on the dialer (E.g., my beeper is 44026 on the dialer!!!)

I see zero pre/master counts

Sometimes you will suddenly see zero counts in the pre-trigger, master-trigger, dead-time and pile-up displays on the GS-commander window. However, VXI crate 7 and the MTM appears to be alive.

We are not sure what causes this hang-up; but the fix is simple: put the cursor in any of the trigger condition input windows and press return. That will cause the trigger maps to be reloaded in the MTM and data will start flowing again.

Check on Darek, he sometimes forgets to restart the mrmto program...

where are the HK time windows set?

The EFFs do honeycomb suppression as well as sums the lo res signal and BGO energies for total energy sums (H). Both these operations have time windows associated with them. These windows are defined in the file "user/eff/user.eff" under the GS account. This file should contain lines like:
# This script is executed at the end when the
# EFF processors boot

# set BGO honeycomb time limits

# set lo res sum energy time limits

The germanium limits are set with the variables: 'ge_sum_time_lo' and 'ge_sum_time_hi'. The BGO time limits are set by the variables: 'bgo_sum_time_lo' and 'bgo_sum_time_hi'.

The values in the "user/eff/user.eff" file take effect when you boot the EFF processors. If you want to change the limits without rebooting, you must type these lines in ALL the six EFF processor tip windows. (E.g, simply type "bgo_sum_time_lo=3204" in a tip window.) You can also see what the current limits are by typing the above names without values. (Try typing "bgo_sum_time_lo" - it should come back with the value the EFF has at the moment).

If you set the time limits based on spectra dumped by the EFFs you must take into account the contraction these spectra have:

  ge times: /8
  BGO times: /4
The time limits you specify for the EFF code must be for uncontracted times.

combining tapes for Daphne sorts

Daphne is very critical of the structure of the data tapes. Daphne also suffers a painful death if it get's to the end of a tape and it doesn't find a double end of file mark. Here is a way to remedy this situation and combine data from several tapes onto one tape. to make a double end of file.

That should generate a compressed tape that Daphne will not choke on. However, this solutions does not work directly if you need to split an input tape on two output tapes - we'll have to think about how to do that properly.

Also, DAPHNE is being modified to be able to handle imperfect tapes, so this problem should go away soon.

As of late March/98, this should no-longer be a problem

gscommander does not respond

Sometimes the EPICS windows do not want to open sub windows or the buttons do not respond.

This is a bug in the EPICS version we are running. Simply kill the EPICS windows you have up and start the gs commander again.

is there a skim program?

We do not officially support a skim program for users to use. We do not have enough manpower to do that, besides, there is no consensus on how to best pre-sort gs data tapes. You are on your own!

the buttons don't work

Once in a very seldom while you will find that the buttons in the Tcl/Tk windows and EPICS windows do not respond. You can usually still start EPICS windows and Tcl/Tk screens; but they do not function. Data will usually continue to go to tape - you just can't control the DAQ anymore.

This is an X problem - which is not understood at the moment. The fix is to log out from the console and log in again. That resets whatever is causing the problem -- and you can again control the experiment.

FERA/VXI delay

The FERA data from the FERA/VXI board is inserted by the event-builder after all GS data has been read and before the two event trailers are sent to the event-builder in the VME crate. The only way the EFF code can recognize that there is external data in the data stream is to to look at the data just before trailer 1. In the upper part of the 32 bit data word, the dip switch settings for the readout delay is present. It will compare this delay with the value of the variable fera_delay which is a VxWorks global variable in the EFF processors.

This variable must match the delay that you have set in the FERA/VXI board for the EFF code to understand and recognize external data.

The variable is set when you boot the EFF processors when the file:

is executed. Look for a line like:
This word is constructed from:
where dddd is the dip-switch setting. (bit 14 is always 1 for reasons that are less than obvious). For the two typical dip-switch settings we use:
  dip switch     binary word       hexword    delay used

  1001        0110|0100|0000|0000   0x6400    (30 usec delay)              
  1111        0111|1100|0000|0000   0x7c00    (50 usec delay)       
If you change the dip-switch value you must 1) modify the eff.user and 2)

Slow response from a VXI processor

Sometimes we see very slow responses from a VXI processor. It always seems to be when they have been running for a long time without any reboot. If you bring up e.g., the temperature monitor, the temperatures from that VXI crate will come up, just very slowly. If you ping the processor you might see a response like:
> ping -s gamvxi01
PING ....: 56 data bytes
64 bytes from ..........: icmp_seq=0. time=895. ms
64 bytes from ..........: icmp_seq=1. time=1981. ms
64 bytes from ..........: icmp_seq=2. time=984. ms
64 bytes from ..........: icmp_seq=3. time=1279. ms
64 bytes from ..........: icmp_seq=4. time=284. ms
whereas you should see a response like:
> ping -s gamvxi02
PING .....: 56 data bytes
64 bytes from ..........: icmp_seq=0. time=5. ms
64 bytes from ..........: icmp_seq=1. time=5. ms
64 bytes from ..........: icmp_seq=2. time=6. ms
64 bytes from ..........: icmp_seq=3. time=5. ms
64 bytes from ..........: icmp_seq=4. time=12. ms
64 bytes from ..........: icmp_seq=5. time=4. ms
64 bytes from ..........: icmp_seq=6. time=5. ms

We are not sure what causes this problem; but rebooting the processor fixes the problem. Please note that crate 4 is notorious for not getting the UNIX time and thus sometimes needs to be rebooted many times before it comes up. The master crate occasionally has the same problem; but it usually gets the time in a couple of tries.

I see mixed source data on my tapes

Sometimes if you are taking source data and putting the data to tape you see some of the old source on the 'new' files.

This can happen because the EFF's may still have some old data stored and if you are not careful about the 'tape on/off' there may be some old data in the 64MB memories as well that gets flushed (or rather pushed) when new data comes in.

To avoid this do:

That way all old data in the EFF will be flushed. It may be enough to reset the EFFs to 'kill' old data as well.

no CFD rates

sympton: There are no CFD rates and the first VXI board EPPICS window looks dead (but there are no white indicators). Data is still going.
Observed 1/28-29/99. We don't know what the real problem is.
FIX: rebooting the master-crate seems to fix the problem. It's not clear how the mastercrate can influence the CFD rates from the germaniums and BGO detectors.

How do I use the GS pulsers?

In the colsole for VXI crate 6 (the master crate) type:
       pulse_con 1,011        (for highest rate)
       pulse_con 1,012
       pulse_con 1,013        (for lowest rate)
Then go to the scripts directory and set the amplitudes:
     pulser_amp  crate/all value
(typical values bt 1 and 5).

To turn the pulsers off again, type

       pulse_con 0        
in the master console window and use the pulser script to turn the amplitudes down again. (The latter action is probably not necessary).

Tip windows disappear/con1 freeze up

Symptoms: The VME processor windows disappears on con1 and sometimes con1 hangs completely.

Problem: The problem is the SCSI terminal server.

Solution: It is necessary to cycle the power on the terminal server. Since it is a SCSI device, this should be done when con1 is not running or the internal disk in the machine could be damaged. Thus, do

detector distance and opening angle

The distance from the target to the front of the germanium detector is 10 inches (or 25.4 cm) and the opening angle is 15 degrees. Also, the efficiency of the detectors is supposed to be 78% (75% for segmented). Typical FWHM res at 1.33 MeV is 2.6 keV.

use external DAQ without GS

Q: Can I use the GS DAQ external part without having the rest of GS up and running?
A: Yes, connect the token out to token in on the MRM with a short cable so that you do not send the readout token through the VXI crates (they may not all be able to pass the token in this mode). Also, disconnect all the 6 vxi trigger cables to the MTM and insert the (blue) grounding connectors. You should then be able to use the GS external DAQ system nomatter what shape the rest of GS is in.

external test signals to GeBGO board

To make the test plug work, supply a ~200nsec logical nim signal to both the s and f inputs (put them in series). The slow signal is of course not quite right, but it will give a resonable signal for the GeBGO board to process. tl/2002

VXI console connectors

Be careful, the two ends of the VXI console connectors are not identical. The ones with the most pins should be on the terminal server board. tl/2002

IOC router field

In the Router field on the IOCs: DO NOT specify a router -- or the IOCs will have trouble talking to one another. Leave the field empty!! tl/2/25/2003

elow in clean BGO data

In the following event
EV len cln dty bgo  ttH   ttM   ttL  tac1 tac2 sge  sbgo
**  24  04  05  15 0014 27079 16015 1814 2545 0304  0982
 ev#  id   ehi g p o  tge ebgo tbgo  elo esid   tc bgohitp  tRF  hs 
   0 078 00576 1 0 0 3938 0000 0000 0000 0074 0000 0000000  2393 0 
   1 096 01809 1 0 0 3982 0000 0000 0000 0000 0000 0110101  2437 1 
   2 045 01007 1 0 0 3983 0000 0000 0000 0000 0000 0000000  2438 0 
   3 085 01708 1 0 0 3943 0000 0000 0000 1745 0000 0000000  2398 0 
   4 059 05056 1 0 0 3996 0026 1968 0000 0000 0000 0100000  2451 0 
   5 071 02571 1 0 0 3990 0241 2030 0000 0339 0000 0010100  2445 0 
   6 079 00496 1 0 0 3993 0908 2058 0000 0022 0000 0000011  2448 0 
   7 105 00419 1 0 0 4013 0258 2048 0000 0000 0000 0001000  2468 0 
   8 109 00899 1 0 0 3958 0000 0216 0000 0000 0000 0000010  2413 0 
   9 030 00000 0 0 0 0000 0045 0216 0000 0000 0000 0001000 -1545 0 
  10 040 00000 0 0 0 0000 0132 2035 0000 0000 0000 0001000 -1545 0 
  11 048 00000 0 0 0 0000 0272 0777 0000 0000 0000 1000000 -1545 0 
  12 050 00000 0 0 0 0000 0076 2028 0000 0000 0000 0001000 -1545 0 
  13 056 00000 0 0 0 0000 0153 2027 0000 0000 0000 0001000 -1545 0 
  14 086 00000 0 0 0 0000 0060 1974 0000 0000 0000 0011000 -1545 0 
  15 092 01856 0 0 0 0000 0276 2051 0000 0000 0000 1000000 -1545 0 
  16 094 00000 0 0 0 0000 0354 2049 0000 0000 0000 0100000 -1545 0 
  17 108 00000 0 0 0 0000 0156 2037 0000 0000 0000 0000010 -1545 0 
  18 027 00000 0 0 0 0000 0356 2049 0000 0000 0000 0000110 -1545 0 
  19 043 00000 0 0 0 0000 0289 2043 0000 0000 0000 0011000 -1545 0 
  20 063 00000 0 0 0 0000 0194 1882 0000 0000 0000 0100000 -1545 0 
  21 087 00000 0 0 0 0000 0086 0212 0000 0000 0000 0000010 -1545 0 
  22 095 00000 0 0 0 0000 0263 2054 0000 0000 0000 0000001 -1545 0 
  23 097 00000 0 0 0 0000 0203 2047 0209 0000 0000 0100000 -1545 0 <----
how can event 23, which is supposed to be a clean bgo, have an elow value?

The energy in the germanium was probably so high that it was overrange on the high res adc, thus it was not classified as a dirty germanium as it really should have been.

set strobe width

To set the main strobe width to 0x10, type
m 0xffffc632
in the master tip window. It should respond as:
ffffc632:  0000-
at this point type
ffffc632:  0000-10
and hit return. It will move to the next memory location and display something like
ffffc634:  00ff-
type . and hit return to get out of the program.

LE/CFD selection

To set the germaniums to CFD mode
    SrcTTLtrig -1,3,0,0
and to set LE mode
    SrcTTLtrig -1,3,1,0
You must execute these commands in each of the 6 VXI console windows

1 or 2 usec timing

To set 1 usec timing, execute
and to set 2 usec timing, excute
You must execute these commands in each of the 6 VXI console windows

why don't I see spectrum xyz in the EFFs?

Q:I'm using EFF #5 and #6. When I dump the BGO time spectra there is nothing in them. Why?
A: Some spectra: lo res ge, ge trap, BGO energies and BGO times are only binned in EFFs 1 and 4. The other EFFs are binning a smaller set of spectra to be slightly more efficient. So, if you want to see any of these spectra, EFFs 1 or 4 must be on

GS stops counting

After replacing the VXI processor in the master crate (gamvxi07), we have found that sometimes the mrmto task, which resets various things in the master trigger module and master readout module, fails.

The symptoms of this problem are: 1) all the triggers scalers (pre, master, late...) have zeroes in them 2) if you move the cursor of the trigger condition input boxes and hit return, the data flows again (for a short or long period) before failing again.

To confirm the problem, go to the VXI07 console window on the shack computer and type 'i'. This will bring up a list of processes and in particular show that the mrmto task is in "SUSPEND" mode.
To fix the problem, in the VXI07 console window, type

td mrmto
taskSpawn "mrmto",210,0x19,20000,mrmtot,60

and the mrmto (master readout module time out) task will run again. If you type 'howtofixmrmto' on the gs computers, it will remind you of these two commands so that you can use copy and paste to execute them.