Editing Personnel Database/Processing/How To

From PCSAR

Warning: You are not logged in. Your IP address will be recorded in this page's edit history.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

==How to process the membership database==
Processing the database requires UNIX scripts, which I ([[BrettWuth]]) keep
on my laptop.
I'm migrating them to the PCSAR virtual machine.

Unless you understand UNIX, you're not likely to understand much
of this. If you need to get someone else to do this, you'll need to
find someone with UNIX or Linux shell skills.

The scripts check for changes in the database, so they keep a
running record of the last changes they've seen. This running
record is currently on [[BrettWuth]]'s laptop.

The Membership Coordinator e-mails me the database as
personnel-database.xml. [[BrettWuth]] is currently doing the data
entry on behalf of the vacant Membership Coordinator position.

Copy my working directory from my laptop to the PCSAR
'''only if''' my laptop is more up to date.
 wuth@macbook:~/sar/pc/membership$ rsync -avz . wuth@pcsar:sar/pc/membership

=== Merging Edits to Personnel Records ===

cd sar/pcsar/membership

I may have made changes to my copy of the database at the same
time as Bob has. The changes need to be merged.

use:
 rlog personnel-database.xml | head -20
to find out what version I gave to Bob most recently (usually
latest).

check in my changes:
 rcsdiff personnel-database.xml
 ci -l personnel-database.xml
for 1.18 use whatever version Bob based his changes on:
 c ~/tmp/personnel-database.xml .
 rcsdiff -r1.18 personnel-database.xml
to see his changes

make my edits to his changes.

rcsmerge -r1.18 personnel-database.xml

* resolve any conflicts:

rcsdiff personnel-database.xml
to verify only his changes have been propogated

=== validate ===

* Validate Edits to Personnel Records
 wuth@pcsar:~/sar/pc/membership$ validate
 validate
(uses less) or
 validation-report

Check my marked up copy of the call out list to see any changes
that may be necessary.

* Reconcile
make any necessary changes.

=== check in ===

Update the "includes changes" date comment at start of the
file:

wuth@pcsar:~/sar/pc/membership$ ci -l personnel-database.xml

=== Tagged Percent ===

* Prepare the TaggedPercent version:

RecordTaggedPercentFieldsDefault -v FieldsFile=lib/RecordTaggedPercentFields.txt <tmp/personnel-database.not-validated >tmp/personnel-database.txt

=== call-out list ===
* To Do:
** commonize elements of {bin,lib}/{call-out,first-call}

* To print Call-out list for RCMP, [[FireHall]], Recorder, and CallOut Coordinator
 wuth@pcsar:~/sar/pc/membership$ call-out-list-print

=== first-call list ===
* Print and deliver copy to Fire Hall, RCMP
 wuth@pcsar:~/sar/pc/membership$ first-call-list-print

* Fax to Fire Dispatch in Lethbridge -- not done any more
 first-call-list-fax

=== personnel records ===
* To print the individual personnel sheets:
 wuth@pcsar:~/sar/pc/membership$ xslt <personnel-database.xml lib/MembersAndApplicants.xsl | changes-print recorder
 wuth@pcsar:~/sar/pc/membership$ xslt <personnel-database.xml lib/MembersOnly.xsl | changes-print SARCAR
** not RCMP because they no longer keep records

=== e-mailed reports ===
* e-mail to [[TrainingCoordinator]]:
 wuth@pcsar:~/sar/pc/membership$ training-coordinator-send

* Print/Send the Fan-Out list -- no longer done
 fan-out-list-send

=== copy to Brett's laptop ===

Because the remaining steps don't run on the PCSAR virtual machine yet, the file must be copied to Brett's laptop and run from there.
<pre>
wuth@macbook:~/sar/pc/membership$ chmod 600 personnel-database.xml
wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/personnel-database.xml .
wuth@macbook:~/sar/pc/membership$ chmod 400 personnel-database.xml
wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/tmp/personnel-database.txt tmp
wuth@macbook:~/sar/pc/membership$ rm -r run
wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/run .                     
</pre>

=== etc ===
* To update the pcsar-board/pcsar-preplan mailing lists:
 mailing-lists-update

* To E-Mail to SARA:
 sara-send

* To distribute the First Call List:
 first-call-list-pdf ~/tmp/PincherSARFirstCall.pdf
<pre>
To: pcsar-firstcall@castrov.cuug.ab.ca
Subject: Pincher SAR First Call List

Hi,

Please find attached Pincher Creek Search and Rescue's (PCSAR) current
First Call List.  This list is provided for agencies that initiate
requests to PCSAR or relay those requests from other areas.

For details on PCSAR's incident notification, see:

http://pcsar.webhop.org/emergency.html

Thank you for your participation in this process.
</pre>
* E-mail call out list:
<pre>
To: PCSAR Emergency Callers <pcsar-caller@castrov.cuug.ab.ca>,
    Pincher Creek SAR Board <pcsar-board@castrov.cuug.ab.ca>,
    PCSAR Preplan Committee <pcsar-preplan@castrov.cuug.ab.ca>
Subject: Call Out List

FYI,

Please find attached the current call-out list.
</pre>

=== printing ===
* Personnel records:
 lpr -Pofficejet-pcsar -o outputorder=reverse run/print-queue/*{recorder,SARCAR}*
* First Call List:
** 1 copy (for binder)
 lpr -Pofficejet-pcsar run/print-queue/*FirstCall*
* Call Out List:
** for binder
** for SARCAR
** for overhead box at firehall
** for myself
** 4 copies
 lpr -#4 -Pofficejet-pcsar -o Collate=True -o outputorder=reverse run/print-queue/*CallOutList*
* To print the print queue:
 print-queue-email-bob
** (to Bob Costa)
** or
 print-queue-email 
** (to Roy Davidson)

==Non-monthly processing==
These are tasks that aren't done on a regular basis. Some of
them haven't been done in long time and their instructions are now
out of date.

===To prepare a list of addresses and phone numbers of all our members===
This was required for a casino application.
  xslt lib/MembersOnly.xsl < personnel-database.xml |xslt lib/NamePhoneAddress.xsl >tmp/addresses.txt

===To print blank Personnel Records===
* keep 10 copies at hand
  personnel-blank-print 10

===To send call-out list via e-mail===
* Stop sending in September 2003

call-out-list-email
Call-out list to executive.

* verify doesn't send when no change:
* Add note about any deletions
* automate production of

===To e-mail mailing labels===
* Does not require anymore
 mailing-labels-email

Mailing Labels sent to Jackie.

===To e-mail changes for Resource database===
*** Stop doing October, 2003. Need another way to handle resource database

personnel-database-mail-resource-database

===When new members join, or e-mail addresses filled in===
<pre>
To: Subject: Welcome 
--text follows this line--
Thanks for joining Pincher Creek SAR. Can you please check if I've correctly
copied your membership record:

---cut---
---cut---

Training sessions/meetings are the first Tuesday of every month,
usually starting at 19:00.

If you'd like to join in a club discussion and receive notices
via e-mail, please send a "subscribe" message to pcsar-discuss-request@castrov.cuug.ab.ca

Take Care,Brett Wuth wuth@acm.org
wuth@telusplanet.net bwuth@cuug.ab.ca Box 1251-U, Pincher Creek,
Alberta T0K 1W0, CANADA Tel:+1 403 627-2460 pgp-public-keys@uit.no
has mine; Print=E4F8EDEECBE01AD2FA3D8B2D94B1A292 What is the meaning of life?! Yes.
<pre>
===To print the participation dates===
 xslt <personnel-database.xml Activities.xsl | awk -v FS=":" {printf "%-24s:%-7s:%s\n", $1, $2, $3 } | mail -s "activity history" `MailRCAliasAddressGet "Bob Costa"` wuth

participation-report-make

* Add any missing entries to partitipation-type.txt

Brett Wuth, 2006/07/15 09:35 MST (via web):
Reformat the database after automatic translation using:

xml-tidy "Training, Experience, Equipment, Notes" <personnel-database.xml >reformatted.xml

Please note that all contributions to PCSAR are considered to be released under the Attribution-Share Alike 3.0 Unported (see PCSAR:Copyrights for details). If you don't want your writing to be edited mercilessly and redistributed at will, then don't submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. DO NOT SUBMIT COPYRIGHTED WORK WITHOUT PERMISSION!

Summary:

Cancel | Editing help (opens in new window)

Templates used on this page:

Retrieved from "http://pcsar.dyndns.org:8080/mediawiki/index.php/Personnel_Database/Processing/How_To"

Editing Personnel Database/Processing/How To

From PCSAR

Views

Personal tools

Navigation

Search

Toolbox

@@ Line 1: / Line 1: @@
-==How to process the personnel database==
+==How to process the membership database==
-Processing the database requires UNIX scripts, which I ([[Brett Wuth]]) keep
+Processing the database requires UNIX scripts, which I ([[BrettWuth]]) keep
-on the PCSAR virtual machine.
+on my laptop.
+I'm migrating them to the PCSAR virtual machine.
 Unless you understand UNIX, you're not likely to understand much
@@ Line 8: / Line 9: @@
 The scripts check for changes in the database, so they keep a
-running record of the last changes they've seen.
+running record of the last changes they've seen. This running
+record is currently on [[BrettWuth]]'s laptop.
-=== Pending edits ===
+The Membership Coordinator e-mails me the database as
-Check my personal copy of the call-out list to see any marked-up changes that I've noted
+personnel-database.xml. [[BrettWuth]] is currently doing the data
-may be necessary.
+entry on behalf of the vacant Membership Coordinator position.
-=== Extract and validate ===
-{{rbw|as used}}
-to do
-* validate upon edit rather than wait for process
-** (reverified after 2015-07 damage) enable PCSARBot to receive email on watched pages
-*** configure exim to receive e-mail locally
-*** login as PCSARBot
-*** change e-mail address to bot@pcsar.castrov.cuug.ab.ca
-*** confirm new address
-*** enable e-mail options for
-**** E-mail me when a page I'm watching is changed
-**** E-mail me also for minor edits of pages
-** (reverified after 2015-07 damage) validate Activities upon edit
-*** Add [[Activities]] to PCSARBot's watchlist
-*** uses .procmailrc
-*** uses script ~bot/sar/pc/membership/bin/wiki-membership-activities-validate to update [[Activities/Report/wiki-membership-activities-validate]]
-*** amend script to output errors with contextual lines
-** validate each Personnel Record upon edit
-*** (reverified after 2015-07 damage) write script to validate just member passed
-**** see ~bot/sar/pc/membership/bin/wiki-membership-member-validate
-**** Use "{{member|Brett Wuth}}" as experiment
-*** (reverified after 2015-07 damage) update all memrec pages to include validation report
-**** define as part of memrec template
-*** develop trigger on memrec changes
-**** develop dispatch on changes to individual memrec or Activities
-***** (recheck) create trial watch on my mem record ({{member|Brett Wuth}})
-****** (done) Add PCSARBot to "TrustedGuest" group
-***** ''currently uses exact match on page name in .procmailrc
-***** (done) verify changes to Brett Wuth's MemRec triggers report update
-***** develop mechanism to register dispatch for page, what script to run
-****** ''on wiki itself
-****** ''advantage: inspectability: easy for others self, to see what is going on
-****** ''disadvantage: more work
-****** ''single page [[Members:PCSARBot/Watches]]
-******* ''each line like:'' <code><nowiki>* [[pagename]]: script</nowiki></code>
-******* ''with possible duplicates of pagename, specifying different scripts
-****** (done)  write scripts to enable/disable watch on a wiki page
-******* (done) wiki-page-watch-enable
-******* (done) wiki-page-watch-disable
-****** (done) write scripts to add/remove watch scripts
-******* (done) wiki-watch-script-add
-******* (done) wiki-watch-script-remove
-****** '''(next)''' create mechanism to dispatch scripts based on email of page that changed; <code>wiki-watch-scripts-dispatch</code>
-******* ''consider security of other person editing page
-******* (done) create tool to determine revision number of page: ~/bin/wiki-page-revision-number-get
-******* (done) create tool to determine editor of particular revision of page (not necessarily current revision)
-******** (done) create tool <code>wiki-page-editors-list</code> that generates list of revision id's and editors
-******** (done) create script wiki-page-editor that produces editor of latest or specified revision
-******* (done) create tool to get the synced content and last editor of a page: <code>wiki-page-get-with-editor</code>
-******** (done) create <code>wiki-page-get</code>
-******** ''consider usage pattern get-latest-revision-number; get-page; verify lasest-revision-number unchanged
-******* validate against whitelist of last editors
-******** '''design choice:''' use single hardcoded name: 'Brett Wuth'
-******** consider: can whitelist be stored on wiki page?  That page would need to be validated itself.  Could be rooted in a single hardcoded name.
-******** shortcut: validate against single hardcoded name until need list
-****** create mechanism to unwatch pages if email came in and no watch script registered
-**** develop watch on memrec category
-***** add new pages
-***** delete removed pages
-*** distribute previously created tools through git
-*** allow Members: page to transclude report from another Members: page
-*** when [[Activities]] changes, reprocess all Members
-**** only if no errors in Activities itself -- otherwise error would appear on each page
-**** only file report if change -- otherwise many unrelated updates
-{{rbw|as used}} to do: optimize for speed (last run 7min)
+Copy my working directory from my laptop to the PCSAR
+'''only if''' my laptop is more up to date.
+ wuth@macbook:~/sar/pc/membership$ rsync -avz . wuth@pcsar:sar/pc/membership
-as user "wuth"
- cd ~/sar/pc/membership
+=== Merging Edits to Personnel Records ===
- wiki-extract
+ cd sar/pcsar/membership
+I may have made changes to my copy of the database at the same
+time as Bob has. The changes need to be merged.
-* Reconcile
+use:
-make any necessary changes.
+ rlog personnel-database.xml | head -20
+to find out what version I gave to Bob most recently (usually
+latest).
-=== Process ===
+check in my changes:
- xslt <personnel-database.xml lib/Applicants.xsl | xslt lib/Name.xsl
+ rcsdiff personnel-database.xml
+ ci -l personnel-database.xml
+for 1.18 use whatever version Bob based his changes on:
+ c ~/tmp/personnel-database.xml .
+ rcsdiff -r1.18 personnel-database.xml
+to see his changes
+make my edits to his changes.
-* <big>'''make sure current ssh-agent authority is extended'''</big>
+ rcsmerge -r1.18 personnel-database.xml
- ssh -AX wuth@localhost
- database-process
+* resolve any conflicts:
-* prints copies for myself
+ rcsdiff personnel-database.xml
-* sends personnel list (names only) to [[Members:Protocols/Shell H2S/Global Training|Global Training for validation of H2S Alive course registration.]]
+to verify only his changes have been propogated
-Manually check the Unicode in reports:
+=== validate ===
-* [[Members:Membership Database/PCSAR]]
-* [[Members:Membership Database/Combined Members]]
-* [[Members:Call Out/List]]
-==== D4H export ====
--03-21 D4H has changed SSL protocols.  Requires upgraded version python.
- d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org membersImport
+* Validate Edits to Personnel Records
+ wuth@pcsar:~/sar/pc/membership$ validate
+ validate
+(uses less) or
+ validation-report
-may need to re-download database to get newly assigned D4HMemberIDs
+Check my marked up copy of the call out list to see any changes
- wiki-extract
+that may be necessary.
- d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org activitiesAttendanceImport
+* Reconcile
+make any necessary changes.
-==== Rave export ====
+=== check in ===
-  wuth@pcsar:~/sar/pc/membership$ call-out-list-xml >tmp/call-out-list.xml
-  wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ scp -p wuth@pcsar.castrov.cuug.ab.ca:sar/pc/membership/tmp/call-out-list.xml .
-  wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ python3 -m sar
-** sometimes fails at initial log in and has to be restarted
+Update the "includes changes" date comment at start of the
-** takes ~65 minutes to run
+file:
-*** see {{subpage|Rapid Notify}}
-==== Continue ====
+ wuth@pcsar:~/sar/pc/membership$ ci -l personnel-database.xml
-* To Do: {{rbw|as used}}
-** additional policy checks
-*** write policy check for sar fund currency
-**** Set type of existing Activities.  See /[ssh/pcsar]/home/wuth/sar/pc/membership/lib/Membership.xsd ActivityCategoryType (done to 2012-01)
-**** base on /[scp/pcsar]/home/wuth/sar/pc/membership/bin/policy-check-stale
-***** see lib/SARFundStale.xsl
-****** needs debugging
-**** add to CVS
-*** write policy check for voting member upgrade
-*** write policy check for voting member currency
-** commonize elements of {bin,lib}/{call-out,first-call}
+=== Tagged Percent ===
-* Personnel records: -- done by Adam now
+* Prepare the TaggedPercent version:
- lpr -Pofficejet8500 -o outputorder=reverse run/print-queue/*{recorder,SARCAR}*
-* Fax to Fire Dispatch in Lethbridge -- not done any more
+  RecordTaggedPercentFieldsDefault -v FieldsFile=lib/RecordTaggedPercentFields.txt <tmp/personnel-database.not-validated >tmp/personnel-database.txt
- first-call-list-fax
-* Print/Send the Fan-Out list -- no longer done
- fan-out-list-send
-* Call Out List: -- done by Adam
+=== call-out list ===
-** for binder
+* To Do:
-** for SARCAR
+** commonize elements of {bin,lib}/{call-out,first-call}
-** for overhead box at firehall
-** 3 copies
-=== sync wiki accounts ===
+* To print Call-out list for RCMP, [[FireHall]], Recorder, and CallOut Coordinator
+ wuth@pcsar:~/sar/pc/membership$ call-out-list-print
-* [http://iweb.castrov.cuug.ab.ca/mediawiki/index.php/Mediawiki#Syncing_user_list sync] the personnel list with the wiki user list
+=== first-call list ===
-** rename/merge Charlene/Charleen Bruder
+* Print and deliver copy to Fire Hall, RCMP
-** Extract list of users currently with "Member" status
+ wuth@pcsar:~/sar/pc/membership$ first-call-list-print
-*** http://pcsar.dyndns.org:8080/mediawiki/index.php?title=Special:Listusers&limit=500&group=Members
- wuth@pcsar:~/sar/pc/membership$
-  xslt lib/MembersOnly.xsl < personnel-database.xml | xslt lib/EMailPrimaryTxtWithNone.xsl >tmp/address-list
-  cat /home/wuth/sar/pc/membership/tmp/address-list | sed -e 's|\(.*\) <\(.*\)>|sudo /usr/local/bin/wiki-members-add "\1" "\2"|' | bash
-will require manual manipulation to
+* Fax to Fire Dispatch in Lethbridge -- not done any more
+ first-call-list-fax
-'''To Do:
+=== personnel records ===
-* '''(next)''' create tool to drop member of group
+* To print the individual personnel sheets:
-* rename existing user
+ wuth@pcsar:~/sar/pc/membership$ xslt <personnel-database.xml lib/MembersAndApplicants.xsl | changes-print recorder
+ wuth@pcsar:~/sar/pc/membership$ xslt <personnel-database.xml lib/MembersOnly.xsl | changes-print SARCAR
+** not RCMP because they no longer keep records
-see [http://iweb.castrov.cuug.ab.ca/mediawiki/index.php/Mediawiki#Syncing_user_list personal notes]
+=== e-mailed reports ===
+* e-mail to [[TrainingCoordinator]]:
+ wuth@pcsar:~/sar/pc/membership$ training-coordinator-send
-==Non-monthly processing==
+* Print/Send the Fan-Out list -- no longer done
-These are tasks that aren't done on a regular basis. Some of
+ fan-out-list-send
-them haven't been done in long time and their instructions are now
-out of date.
 === copy to Brett's laptop ===
@@ Line 189: / Line 117: @@
 wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/tmp/personnel-database.txt tmp
 wuth@macbook:~/sar/pc/membership$ rm -r run
-wuth@macbook:~/sar/pc/membership$ scp -pr wuth@pcsar:sar/pc/membership/run .
+wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/run .
 </pre>
+=== etc ===
+* To update the pcsar-board/pcsar-preplan mailing lists:
+ mailing-lists-update
+* To E-Mail to SARA:
+ sara-send
+* To distribute the First Call List:
+ first-call-list-pdf ~/tmp/PincherSARFirstCall.pdf
+<pre>
+To: pcsar-firstcall@castrov.cuug.ab.ca
+Subject: Pincher SAR First Call List
+Hi,
+Please find attached Pincher Creek Search and Rescue's (PCSAR) current
+First Call List.  This list is provided for agencies that initiate
+requests to PCSAR or relay those requests from other areas.
+For details on PCSAR's incident notification, see:
+  http://pcsar.webhop.org/emergency.html
+Thank you for your participation in this process.
+</pre>
+* E-mail call out list:
+<pre>
+To: PCSAR Emergency Callers <pcsar-caller@castrov.cuug.ab.ca>,
+    Pincher Creek SAR Board <pcsar-board@castrov.cuug.ab.ca>,
+    PCSAR Preplan Committee <pcsar-preplan@castrov.cuug.ab.ca>
+Subject: Call Out List
+FYI,
+Please find attached the current call-out list.
+</pre>
+=== printing ===
+* Personnel records:
+ lpr -Pofficejet-pcsar -o outputorder=reverse run/print-queue/*{recorder,SARCAR}*
+* First Call List:
+** 1 copy (for binder)
+ lpr -Pofficejet-pcsar run/print-queue/*FirstCall*
+* Call Out List:
+** for binder
+** for SARCAR
+** for overhead box at firehall
+** for myself
+** 4 copies
+ lpr -#4 -Pofficejet-pcsar -o Collate=True -o outputorder=reverse run/print-queue/*CallOutList*
+* To print the print queue:
+ print-queue-email-bob
+** (to Bob Costa)
+** or
+ print-queue-email
+** (to Roy Davidson)
+==Non-monthly processing==
+These are tasks that aren't done on a regular basis. Some of
+them haven't been done in long time and their instructions are now
+out of date.
 ===To prepare a list of addresses and phone numbers of all our members===
@@ Line 226: / Line 216: @@
 --text follows this line--
 Thanks for joining Pincher Creek SAR. Can you please check if I've correctly
-copied your personnel record:
+copied your membership record:
 ---cut---
@@ Line 241: / Line 231: @@
 Alberta T0K 1W0, CANADA Tel:+1 403 627-2460 pgp-public-keys@uit.no
 has mine; Print=E4F8EDEECBE01AD2FA3D8B2D94B1A292 What is the meaning of life?! Yes.
-</pre>
+<pre>
 ===To print the participation dates===
  xslt <personnel-database.xml Activities.xsl | awk -v FS=":" {printf "%-24s:%-7s:%s\n", $1, $2, $3 } | mail -s "activity history" `MailRCAliasAddressGet "Bob Costa"` wuth