Argonne Physics Division, GAMMASPHERE

GAMMASPHERE Hints & Kinks, FAQ

Processor 'eats' data
VXI crate modifications
Stuck bit scenario 1
Where should the LTM be?
Log books at ANL
VXI readout bus
I can't dump spectra...
Bad ge spectra
forward error messages please
DAQ crashes...
Alarms to alpha-num beepers
Dialer Alarms
I see zero pre/master counts
where are the HK time windows set?
combining tapes for Daphne sorts
gscommander does not respond
is there a skim program?
the buttons don't work
FERA/VXI delay
Slow response from a VXI processor
I see mixed source data on my tapes
no CFD rates
How do I use the GS pulsers?
Tip windows disappear/con1 freeze up
detector distance and opening angle
use external DAQ without GS
external test signals to GeBGO board
VXI console connectors
IOC router field
elow in clean BGO data
set strobe width
LE/CFD selection
1 or 2 usec timing
why don't I see spectrum xyz in the EFFs?
GS stops counting

Processor 'eats' data

Sometimes a processor will take data but not output data to the data-stream for the tapes. Symptom: the total event rate goes down with the rate that the processor is handling.

The problem is in the CES memory associated with the event-filter formatter Currently it is (almost always) necessary to recycle the power of the VME crate of the affected processor to fix the problem. Since during the run that will stop the taping, this fix is best done when tapes are changed. Do the following:

stop DAQ, run and tape
rewind and extract the tapes
make sure you have saved 1)hires parameters, 2)low-res parameters and 3) time parameters sing the TCL calibration scripts
cycle the power on the VME crate with the bad CES memory
(the processors should reboot automatically)
when the processors are back up, reload the hires -, low-res - and time parameters for the processors that were rebooted
you should be able to continue as usual at this point.

VXI crate modifications

The VXI crates are modified for the sum-bus: The terminator on the back plane to the -2V on the ECL bus should be removed. This line is used as an analog sum bus in GAMMASPHERE.

Stuck bit scenario 1

Due to a pressed-in pin on one of the CES memories, we had a stuck bit. The CES dump looked as:

> cesd
03100000   8000     B  4741  4D4D  4153  5049  4552  4521  
03100010      0  8181     0  8181     0  8181   8D5     1  
03100020      0     3  6425  3665     0     1  FFFF  FFFF  
03100030   8000     F  4741  4D4D  4153  5049  4552  4521  
03100040   27BD   103     0     1     0     1     0     1  
03100050      0  8181     0  8181     0  8181   BF2     1  
03100060      0     3  6425  371F     0     1  FFFF  FFFF  
03100070   8000     F  4741  4D4D  4153  5049  4552  4521  
03100080   27BF   103     0     1     0     1     0     1  
03100090      0  8181     0  8181     0  8181   155     1  
031000A0      0     3  6425  37D9     0     1  FFFF  FFFF  
031000B0   8000     F  4741  4D4D  4153  5049  4552  4521  
031000C0   27BC   103     0     1     0     1     0     1  
031000D0      0  8181     0  8181     0  8181   45D     1  
031000E0      0     3  6425  3895     0     1  FFFF  FFFF  
031000F0   8000     F  4741  4D4D  4153  5049  4552  4521

Randy pointed out where the stuck bit was:

 > > cesd
 > 03100000   8000     B  4741  4D4D  4153  5049  4552  4521  
 > 03100010      0  8181     0  8181     0  8181   8D5     1  
 > 03100020      0     3  6425  3665     0     1  FFFF  FFFF  
 > 03100030   8000     F  4741  4D4D  4153  5049  4552  4521  
 > 03100040   27BD   103     0     1     0     1     0     1  

Stuck bit:             1           1           1           1

 > 03100050      0  8181     0  8181     0  8181   BF2     1  
 > 03100060      0     3  6425  371F     0     1  FFFF  FFFF  
                       1           1           1           1
 > 03100070   8000     F  4741  4D4D  4153  5049  4552  4521 
                       1           1           1           1 
 > 03100080   27BF   103     0     1     0     1     0     1
                       1           1           1           1

Where should the LTM be?

The LTM must be in the slot following the last GeBGO board since it returns a token. If you leave an empty slot, this token will not be returned and the DAQ will hang.

Log books at ANL

The following log books are located at ANL

1
2
4
5
6

The rest must be at LBNL?

VXI readout bus

it seem like it better to terminate the last INPUT in the LRMs and chain the output of that one to the previous one and so on all the way back to the MRM.
The cable between the MRM and the first LRM is special. The left hand side cable has the data bus delayed with respect to the strobe signal which are the upper pins in the left hand connector on the LRMs and MRM. If you do not use this special cable and have more then 4 VXI crates running, you will see ghost detectors. Some detectors will come with ID different than what they should in the data stream.
Augusto likes to have the token first go to the last VXI crate on the readout bus and return from the first one.

I can't dump spectra...

If you have trouble dumping spectra using the TCL/TK windows and it complains about Channel Access errors, try turning off the sender while you dump. The sender is able to load the network so hard that other processes have trouble getting their date over the network. It also helps to only specify the EFFs that actually have data rather than all of them (1-6). The safest way to get around this problem is to simply take the DAQ off and wait for all processors to finish processing. That way they have plenty of time to respond to the network requests and the sender is no longer blasting the network either.

On 2/11/98 a new version of the dump program was installed and conditions have to be really bad before this program gives up getting the EFF spectra. However, not all utilities have been upgraded (that is a big job) so you may still have stop or slow down the acquisition as described above.

Bad ge spectra

If your ge spectra looks like TAC calibration spectra - check that the gain coefficients are reasonable. If they are way off, - the EFFs can corrupt the data.

Please forward error messages --

To debug problems in the VXI and VME modules, it is important to have accurate failure/debug information. If e.g., an EFF processor dies, please to the following:

locate the tip window for the processor
start an xterm up (xterm &)
In the xterm window type "cat - | mail [email protected]".
select the error message in the tip window using the left hand button.
now go to the xterm window and push the middle button. The debugging error messages should be typed to the screen.
While still in the xterm window, type "control-d". That will send the error message and return the prompt in the xterm window.

DAQ crashes...

If the DAQ crashes, here are a few pointers to what you can do:

Please document the crash. It's the only way we can try to fix them. Just telling us the day after doesn't help us much in figuring out what happened. Please read the instruction on how to report errors.
If only one EFF processor stops working, go to the tip window for the processor and check if the EFF task is suspended. If you type 'i' in the window and find a line like:
```
eff        _effcontrol    70d024 210 SUSPEND      d0d0d8   70cf74       0     0
```
then you may be able to simply resume the task by typing:
```
    tr 0x70d024
```
where "70d024" is the number in the third column (the 0x in front signifies that it's a hex number). If the EFF task comes alive, you can use the processor again.
Sometimes after this revival you will see an error like:
```
-> 
Access Fault
Program Counter: 0x00d0d0e0
Status Register: 0x3004
Access Address : 0x0510000c
Special Status : 0x0085
Task: 0x70d024 "eff"
```
In that case the next step is to reboot the processor. You can do that by typing "reboot" in the tip window or push the red reset button on the front of the processor.
If a reboot of the processor doesn't work, we are probably dealing with a crate-wide problem and it's usually necessary to cycle the power to the VME crate. Some of the modules in the VME crates do not have any reset buttons, hence, a cold start is the only way to reset them. You may try to just cycle the power on the bad crate; but mostly, it's necessary to cycle the power on both the VME crates to get the DAQ up again. Make sure to eject your tapes before you cycle the power!

Alarms to alpha-num beepers

GS has a number alarm monitores running that will send various alarm messages to alpha-numeric beepers.

The [1]germanium high temperature monitor, [2]ln-alarm monitor and [3]VXI crate monitors will send alarms to members of the

    /home/gam13/acct/gs/.BEEPERADDR

list and the [4]ln-fill monitor is using the list

    /home/gam13/acct/gs/.LNFILMON

Here is an example of how the .BEEPERADDR might look like:

[email protected]
[email protected]
[email protected]

In this case, two beepers are notified when there is an alarm. One E-mail to a regular account (via a mail alias) is sent as well.

Here is an example of how the .LNFILMON file might look:

[email protected]
[email protected]

People on this list will get the 'ln fill started', 'ln fill done' and 'tank fill done' information messages every time the GS array is filled.

Dialer Alarms

To change the dialer numbers do:

press store
press either 1, 2 or 3 
enter number
press store

You should hear a one second beep when the number is stored. It isn't really necessary to touch or change the message of the dialer -- it doesn't get through to the alpha numeric beepers anyway. On these beepers, the dialer generates a "020202" message or the like.

The dialer can be programmed for three numbers. You select those in the second line.

to disable a number: press store, press either 1, 2 or 3, press store. I.e., do NOT enter a number. Don't put in 0 since it will dial the operator!!!

The LN dial wires should connect to the outermost contacts of the alarm inputs.

NOTICE: Remember to put 4 in front of the beeper number on the dialer (E.g., my beeper is 44026 on the dialer!!!)

I see zero pre/master counts

Sometimes you will suddenly see zero counts in the pre-trigger, master-trigger, dead-time and pile-up displays on the GS-commander window. However, VXI crate 7 and the MTM appears to be alive.

We are not sure what causes this hang-up; but the fix is simple: put the cursor in any of the trigger condition input windows and press return. That will cause the trigger maps to be reloaded in the MTM and data will start flowing again.

Check on Darek, he sometimes forgets to restart the mrmto program...

where are the HK time windows set?

The EFFs do honeycomb suppression as well as sums the lo res signal and BGO energies for total energy sums (H). Both these operations have time windows associated with them. These windows are defined in the file "user/eff/user.eff" under the GS account. This file should contain lines like:

# This script is executed at the end when the
# EFF processors boot

# set BGO honeycomb time limits
 
bgo_sum_time_lo=3204
bgo_sum_time_hi=3720

# set lo res sum energy time limits

ge_sum_time_lo=3840
ge_sum_time_hi=4160

The germanium limits are set with the variables: 'ge_sum_time_lo' and 'ge_sum_time_hi'. The BGO time limits are set by the variables: 'bgo_sum_time_lo' and 'bgo_sum_time_hi'.

The values in the "user/eff/user.eff" file take effect when you boot the EFF processors. If you want to change the limits without rebooting, you must type these lines in ALL the six EFF processor tip windows. (E.g, simply type "bgo_sum_time_lo=3204" in a tip window.) You can also see what the current limits are by typing the above names without values. (Try typing "bgo_sum_time_lo" - it should come back with the value the EFF has at the moment).

If you set the time limits based on spectra dumped by the EFFs you must take into account the contraction these spectra have:

  ge times: /8
  BGO times: /4

The time limits you specify for the EFF code must be for uncontracted times.

combining tapes for Daphne sorts

Daphne is very critical of the structure of the data tapes. Daphne also suffers a painful death if it get's to the end of a tape and it doesn't find a double end of file mark. Here is a way to remedy this situation and combine data from several tapes onto one tape.

for the first tape do:
```
tc /dev/rmt/1mbn /dev/rmt/0mbn 0 112 90 16384
```
That will copy the tapeheader (siz=112), the file header (siz=90) and all data blocks (siz=16384) from the input tape, /dev/rmt/1mbn, to the output tape, /dev/rmt/0mbn. If the last block is truncated, it will not be copied either (we don't want to!). It doesn't matter if there is a double end of file mark on the input tape or not -- tc will do the right thing.
for the subsequent tapes do:
```
tc /dev/rmt/1mbn /dev/rmt/0mbn 0 90 16384
```
i.e, we copy as before but we exclude the tape header. Daphne gets very perplexed if it sees two of those on one data tape...
when all copying is done, add an extra eof on the output tape as:
```
mt -f /dev/rmt/0mbn weof
```

to make a double end of file.

That should generate a compressed tape that Daphne will not choke on. However, this solutions does not work directly if you need to split an input tape on two output tapes - we'll have to think about how to do that properly.

Also, DAPHNE is being modified to be able to handle imperfect tapes, so this problem should go away soon.

As of late March/98, this should no-longer be a problem

gscommander does not respond

Sometimes the EPICS windows do not want to open sub windows or the buttons do not respond.

This is a bug in the EPICS version we are running. Simply kill the EPICS windows you have up and start the gs commander again.

is there a skim program?

We do not officially support a skim program for users to use. We do not have enough manpower to do that, besides, there is no consensus on how to best pre-sort gs data tapes. You are on your own!

the buttons don't work

Once in a very seldom while you will find that the buttons in the Tcl/Tk windows and EPICS windows do not respond. You can usually still start EPICS windows and Tcl/Tk screens; but they do not function. Data will usually continue to go to tape - you just can't control the DAQ anymore.

This is an X problem - which is not understood at the moment. The fix is to log out from the console and log in again. That resets whatever is causing the problem -- and you can again control the experiment.

FERA/VXI delay

The FERA data from the FERA/VXI board is inserted by the event-builder after all GS data has been read and before the two event trailers are sent to the event-builder in the VME crate. The only way the EFF code can recognize that there is external data in the data stream is to to look at the data just before trailer 1. In the upper part of the 32 bit data word, the dip switch settings for the readout delay is present. It will compare this delay with the value of the variable fera_delay which is a VxWorks global variable in the EFF processors.

This variable must match the delay that you have set in the FERA/VXI board for the EFF code to understand and recognize external data.

The variable is set when you boot the EFF processors when the file:

      /home/gam13/acct/gs/user/eff/eff.user

is executed. Look for a line like:

      fera_delay=0x7c00

This word is constructed from:

       01dd|dd00|0000|0000

where dddd is the dip-switch setting. (bit 14 is always 1 for reasons that are less than obvious). For the two typical dip-switch settings we use:

 
  dip switch     binary word       hexword    delay used

  1001        0110|0100|0000|0000   0x6400    (30 usec delay)              
  1111        0111|1100|0000|0000   0x7c00    (50 usec delay)

If you change the dip-switch value you must 1) modify the eff.user and 2)

either reboot all EFF processors
or manually set the value in ALL the EFF tip windows by e.g., typing:
```
     fera_delay=0x7c00
```

Slow response from a VXI processor

Sometimes we see very slow responses from a VXI processor. It always seems to be when they have been running for a long time without any reboot. If you bring up e.g., the temperature monitor, the temperatures from that VXI crate will come up, just very slowly. If you ping the processor you might see a response like:

> ping -s gamvxi01
PING ....: 56 data bytes
64 bytes from ..........: icmp_seq=0. time=895. ms
64 bytes from ..........: icmp_seq=1. time=1981. ms
64 bytes from ..........: icmp_seq=2. time=984. ms
64 bytes from ..........: icmp_seq=3. time=1279. ms
64 bytes from ..........: icmp_seq=4. time=284. ms

whereas you should see a response like:

> ping -s gamvxi02
PING .....: 56 data bytes
64 bytes from ..........: icmp_seq=0. time=5. ms
64 bytes from ..........: icmp_seq=1. time=5. ms
64 bytes from ..........: icmp_seq=2. time=6. ms
64 bytes from ..........: icmp_seq=3. time=5. ms
64 bytes from ..........: icmp_seq=4. time=12. ms
64 bytes from ..........: icmp_seq=5. time=4. ms
64 bytes from ..........: icmp_seq=6. time=5. ms
64

We are not sure what causes this problem; but rebooting the processor fixes the problem. Please note that crate 4 is notorious for not getting the UNIX time and thus sometimes needs to be rebooted many times before it comes up. The master crate occasionally has the same problem; but it usually gets the time in a couple of tries.

I see mixed source data on my tapes

Sometimes if you are taking source data and putting the data to tape you see some of the old source on the 'new' files.

This can happen because the EFF's may still have some old data stored and if you are not careful about the 'tape on/off' there may be some old data in the 64MB memories as well that gets flushed (or rather pushed) when new data comes in.

To avoid this do:

before you stop the run, turn off taping
the stop run
don't turn the DAQ off
put new source in
by having only one EFF on at a time, make sure all the EFFs you have processed at least one buffer
now start the run
then turn on taping

That way all old data in the EFF will be flushed. It may be enough to reset the EFFs to 'kill' old data as well.

no CFD rates

sympton: There are no CFD rates and the first VXI board EPPICS window looks dead (but there are no white indicators). Data is still going.
Observed 1/28-29/99. We don't know what the real problem is.
FIX: rebooting the master-crate seems to fix the problem. It's not clear how the mastercrate can influence the CFD rates from the germaniums and BGO detectors.

How do I use the GS pulsers?

In the colsole for VXI crate 6 (the master crate) type:

       pulse_con 1,011        (for highest rate)
       pulse_con 1,012
       pulse_con 1,013        (for lowest rate)

Then go to the scripts directory and set the amplitudes:

     pulser_amp  crate/all value

(typical values bt 1 and 5).

To turn the pulsers off again, type

       pulse_con 0

in the master console window and use the pulser script to turn the amplitudes down again. (The latter action is probably not necessary).

Tip windows disappear/con1 freeze up

Symptoms: The VME processor windows disappears on con1 and sometimes con1 hangs completely.

Problem: The problem is the SCSI terminal server.

Solution: It is necessary to cycle the power on the terminal server. Since it is a SCSI device, this should be done when con1 is not running or the internal disk in the machine could be damaged. Thus, do

log out from the console of con1
log in as "shutdown". That will shut down the machine properly.
when the "ok" prompt appears, turn the power off (on the back)
remove the power plug to the SCSI terminal server on top of the shelves to your right. Count to 10 and plug it back in again.
turn on con1 again
when the "ok" prompt appears, type "boot"
log in as gs again and set up all windows and utilities.

detector distance and opening angle

The distance from the target to the front of the germanium detector is 10 inches (or 25.4 cm) and the opening angle is 15 degrees. Also, the efficiency of the detectors is supposed to be 78% (75% for segmented). Typical FWHM res at 1.33 MeV is 2.6 keV.

use external DAQ without GS

Q: Can I use the GS DAQ external part without having the rest of GS up and running?
A: Yes, connect the token out to token in on the MRM with a short cable so that you do not send the readout token through the VXI crates (they may not all be able to pass the token in this mode). Also, disconnect all the 6 vxi trigger cables to the MTM and insert the (blue) grounding connectors. You should then be able to use the GS external DAQ system nomatter what shape the rest of GS is in.

external test signals to GeBGO board

To make the test plug work, supply a ~200nsec logical nim signal to both the s and f inputs (put them in series). The slow signal is of course not quite right, but it will give a resonable signal for the GeBGO board to process. tl/2002

VXI console connectors

Be careful, the two ends of the VXI console connectors are not identical. The ones with the most pins should be on the terminal server board. tl/2002

IOC router field

In the Router field on the IOCs: DO NOT specify a router -- or the IOCs will have trouble talking to one another. Leave the field empty!! tl/2/25/2003

elow in clean BGO data

In the following event

EV len cln dty bgo  ttH   ttM   ttL  tac1 tac2 sge  sbgo
**  24  04  05  15 0014 27079 16015 1814 2545 0304  0982
 ev#  id   ehi g p o  tge ebgo tbgo  elo esid   tc bgohitp  tRF  hs 
   0 078 00576 1 0 0 3938 0000 0000 0000 0074 0000 0000000  2393 0 
   1 096 01809 1 0 0 3982 0000 0000 0000 0000 0000 0110101  2437 1 
   2 045 01007 1 0 0 3983 0000 0000 0000 0000 0000 0000000  2438 0 
   3 085 01708 1 0 0 3943 0000 0000 0000 1745 0000 0000000  2398 0 
   4 059 05056 1 0 0 3996 0026 1968 0000 0000 0000 0100000  2451 0 
   5 071 02571 1 0 0 3990 0241 2030 0000 0339 0000 0010100  2445 0 
   6 079 00496 1 0 0 3993 0908 2058 0000 0022 0000 0000011  2448 0 
   7 105 00419 1 0 0 4013 0258 2048 0000 0000 0000 0001000  2468 0 
   8 109 00899 1 0 0 3958 0000 0216 0000 0000 0000 0000010  2413 0 
   9 030 00000 0 0 0 0000 0045 0216 0000 0000 0000 0001000 -1545 0 
  10 040 00000 0 0 0 0000 0132 2035 0000 0000 0000 0001000 -1545 0 
  11 048 00000 0 0 0 0000 0272 0777 0000 0000 0000 1000000 -1545 0 
  12 050 00000 0 0 0 0000 0076 2028 0000 0000 0000 0001000 -1545 0 
  13 056 00000 0 0 0 0000 0153 2027 0000 0000 0000 0001000 -1545 0 
  14 086 00000 0 0 0 0000 0060 1974 0000 0000 0000 0011000 -1545 0 
  15 092 01856 0 0 0 0000 0276 2051 0000 0000 0000 1000000 -1545 0 
  16 094 00000 0 0 0 0000 0354 2049 0000 0000 0000 0100000 -1545 0 
  17 108 00000 0 0 0 0000 0156 2037 0000 0000 0000 0000010 -1545 0 
  18 027 00000 0 0 0 0000 0356 2049 0000 0000 0000 0000110 -1545 0 
  19 043 00000 0 0 0 0000 0289 2043 0000 0000 0000 0011000 -1545 0 
  20 063 00000 0 0 0 0000 0194 1882 0000 0000 0000 0100000 -1545 0 
  21 087 00000 0 0 0 0000 0086 0212 0000 0000 0000 0000010 -1545 0 
  22 095 00000 0 0 0 0000 0263 2054 0000 0000 0000 0000001 -1545 0 
  23 097 00000 0 0 0 0000 0203 2047 0209 0000 0000 0100000 -1545 0 <----

how can event 23, which is supposed to be a clean bgo, have an elow value?

The energy in the germanium was probably so high that it was overrange on the high res adc, thus it was not classified as a dirty germanium as it really should have been.

set strobe width

To set the main strobe width to 0x10, type

m 0xffffc632

in the master tip window. It should respond as:

ffffc632:  0000-

at this point type

ffffc632:  0000-10

and hit return. It will move to the next memory location and display something like

ffffc634:  00ff-

type . and hit return to get out of the program.

LE/CFD selection

To set the germaniums to CFD mode

    SrcTTLtrig -1,3,0,0

and to set LE mode

    SrcTTLtrig -1,3,1,0

You must execute these commands in each of the 6 VXI console windows

1 or 2 usec timing

To set 1 usec timing, execute

   <1usec

and to set 2 usec timing, excute

   <2usec

You must execute these commands in each of the 6 VXI console windows

why don't I see spectrum xyz in the EFFs?

Q:I'm using EFF #5 and #6. When I dump the BGO time spectra there is nothing in them. Why?
A: Some spectra: lo res ge, ge trap, BGO energies and BGO times are only binned in EFFs 1 and 4. The other EFFs are binning a smaller set of spectra to be slightly more efficient. So, if you want to see any of these spectra, EFFs 1 or 4 must be on

GS stops counting

After replacing the VXI processor in the master crate (gamvxi07), we have found that sometimes the mrmto task, which resets various things in the master trigger module and master readout module, fails.

The symptoms of this problem are: 1) all the triggers scalers (pre, master, late...) have zeroes in them 2) if you move the cursor of the trigger condition input boxes and hit return, the data flows again (for a short or long period) before failing again.

To confirm the problem, go to the VXI07 console window on the shack computer and type 'i'. This will bring up a list of processes and in particular show that the mrmto task is in "SUSPEND" mode.
To fix the problem, in the VXI07 console window, type

td mrmto
taskSpawn "mrmto",210,0x19,20000,mrmtot,60

and the mrmto (master readout module time out) task will run again. If you type 'howtofixmrmto' on the gs computers, it will remind you of these two commands so that you can use copy and paste to execute them.