Personnel Database/Processing/How To

From PCSAR

< Personnel Database | Processing(Difference between revisions)
Jump to: navigation, search
(etc)
Current revision (20:05, 22 May 2022) (edit) (undo)
(Rave export)
 
(118 intermediate revisions not shown.)
Line 1: Line 1:
-
==How to process the membership database==
+
==How to process the personnel database==
Processing the database requires UNIX scripts, which I ([[Brett Wuth]]) keep
Processing the database requires UNIX scripts, which I ([[Brett Wuth]]) keep
on the PCSAR virtual machine.
on the PCSAR virtual machine.
Line 11: Line 11:
=== Pending edits ===
=== Pending edits ===
-
Check my marked up copy of the call out list to see any changes
+
Check my personal copy of the call-out list to see any marked-up changes that I've noted
-
that may be necessary.
+
may be necessary.
-
 
+
=== Extract and validate ===
=== Extract and validate ===
 +
{{rbw|as used}}
 +
to do
 +
* validate upon edit rather than wait for process
 +
** (reverified after 2015-07 damage) enable PCSARBot to receive email on watched pages
 +
*** configure exim to receive e-mail locally
 +
*** login as PCSARBot
 +
*** change e-mail address to bot@pcsar.castrov.cuug.ab.ca
 +
*** confirm new address
 +
*** enable e-mail options for
 +
**** E-mail me when a page I'm watching is changed
 +
**** E-mail me also for minor edits of pages
 +
** (reverified after 2015-07 damage) validate Activities upon edit
 +
*** Add [[Activities]] to PCSARBot's watchlist
 +
*** uses .procmailrc
 +
*** uses script ~bot/sar/pc/membership/bin/wiki-membership-activities-validate to update [[Activities/Report/wiki-membership-activities-validate]]
 +
*** amend script to output errors with contextual lines
 +
** validate each Personnel Record upon edit
 +
*** (reverified after 2015-07 damage) write script to validate just member passed
 +
**** see ~bot/sar/pc/membership/bin/wiki-membership-member-validate
 +
**** Use "{{member|Brett Wuth}}" as experiment
 +
*** (reverified after 2015-07 damage) update all memrec pages to include validation report
 +
**** define as part of memrec template
 +
*** develop trigger on memrec changes
 +
**** develop dispatch on changes to individual memrec or Activities
 +
***** (recheck) create trial watch on my mem record ({{member|Brett Wuth}})
 +
****** (done) Add PCSARBot to "TrustedGuest" group
 +
***** ''currently uses exact match on page name in .procmailrc
 +
***** (done) verify changes to Brett Wuth's MemRec triggers report update
 +
***** develop mechanism to register dispatch for page, what script to run
 +
****** ''on wiki itself
 +
****** ''advantage: inspectability: easy for others self, to see what is going on
 +
****** ''disadvantage: more work
 +
****** ''single page [[Members:PCSARBot/Watches]]
 +
******* ''each line like:'' <code><nowiki>* [[pagename]]: script</nowiki></code>
 +
******* ''with possible duplicates of pagename, specifying different scripts
 +
****** (done) write scripts to enable/disable watch on a wiki page
 +
******* (done) wiki-page-watch-enable
 +
******* (done) wiki-page-watch-disable
 +
****** (done) write scripts to add/remove watch scripts
 +
******* (done) wiki-watch-script-add
 +
******* (done) wiki-watch-script-remove
 +
****** '''(next)''' create mechanism to dispatch scripts based on email of page that changed; <code>wiki-watch-scripts-dispatch</code>
 +
******* ''consider security of other person editing page
 +
******* (done) create tool to determine revision number of page: ~/bin/wiki-page-revision-number-get
 +
******* (done) create tool to determine editor of particular revision of page (not necessarily current revision)
 +
******** (done) create tool <code>wiki-page-editors-list</code> that generates list of revision id's and editors
 +
******** (done) create script wiki-page-editor that produces editor of latest or specified revision
 +
******* (done) create tool to get the synced content and last editor of a page: <code>wiki-page-get-with-editor</code>
 +
******** (done) create <code>wiki-page-get</code>
 +
******** ''consider usage pattern get-latest-revision-number; get-page; verify lasest-revision-number unchanged
 +
******* validate against whitelist of last editors
 +
******** '''design choice:''' use single hardcoded name: 'Brett Wuth'
 +
******** consider: can whitelist be stored on wiki page? That page would need to be validated itself. Could be rooted in a single hardcoded name.
 +
******** shortcut: validate against single hardcoded name until need list
 +
****** create mechanism to unwatch pages if email came in and no watch script registered
 +
**** develop watch on memrec category
 +
***** add new pages
 +
***** delete removed pages
 +
*** distribute previously created tools through git
 +
*** allow Members: page to transclude report from another Members: page
 +
*** when [[Activities]] changes, reprocess all Members
 +
**** only if no errors in Activities itself -- otherwise error would appear on each page
 +
**** only file report if change -- otherwise many unrelated updates
 +
{{rbw|as used}} to do: optimize for speed (last run 7min)
{{rbw|as used}} to do: optimize for speed (last run 7min)
 +
 +
as user "wuth"
cd ~/sar/pc/membership
cd ~/sar/pc/membership
Line 27: Line 92:
=== Process ===
=== Process ===
 +
xslt <personnel-database.xml lib/Applicants.xsl | xslt lib/Name.xsl
 +
 +
* <big>'''make sure current ssh-agent authority is extended'''</big>
* <big>'''make sure current ssh-agent authority is extended'''</big>
ssh -AX wuth@localhost
ssh -AX wuth@localhost
Line 32: Line 100:
database-process
database-process
 +
* prints copies for myself
 +
* sends personnel list (names only) to [[Members:Protocols/Shell H2S/Global Training|Global Training for validation of H2S Alive course registration.]]
 +
 +
Manually check the Unicode in reports:
 +
* [[Members:Membership Database/PCSAR]]
 +
* [[Members:Membership Database/Combined Members]]
 +
* [[Members:Call Out/List]]
 +
==== D4H export ====
 +
2021-03-21 D4H has changed SSL protocols. Requires upgraded version python.
 +
 +
d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org membersImport
 +
 +
may need to re-download database to get newly assigned D4HMemberIDs
 +
wiki-extract
 +
 +
d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org activitiesAttendanceImport
 +
 +
==== Rave export ====
 +
wuth@pcsar:~/sar/pc/membership$ call-out-list-xml >tmp/call-out-list.xml
 +
wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ scp -p wuth@pcsar.castrov.cuug.ab.ca:sar/pc/membership/tmp/call-out-list.xml .
 +
wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ python3 -m sar
 +
 +
** sometimes fails at initial log in and has to be restarted
 +
** takes ~65 minutes to run
 +
*** see {{subpage|Rapid Notify}}
 +
 +
==== Continue ====
* To Do: {{rbw|as used}}
* To Do: {{rbw|as used}}
** additional policy checks
** additional policy checks
Line 61: Line 156:
** 3 copies
** 3 copies
-
=== etc ===
+
=== sync wiki accounts ===
-
* [http://iweb.castrov.cuug.ab.ca/mediawiki/index.php/Mediawiki#Syncing_user_list sync] the membership list with the wiki user list
+
* [http://iweb.castrov.cuug.ab.ca/mediawiki/index.php/Mediawiki#Syncing_user_list sync] the personnel list with the wiki user list
** rename/merge Charlene/Charleen Bruder
** rename/merge Charlene/Charleen Bruder
-
** move Chris Jorgensen to a TrustedGuest with same access permission
 
** Extract list of users currently with "Member" status
** Extract list of users currently with "Member" status
*** http://pcsar.dyndns.org:8080/mediawiki/index.php?title=Special:Listusers&limit=500&group=Members
*** http://pcsar.dyndns.org:8080/mediawiki/index.php?title=Special:Listusers&limit=500&group=Members
-
* Send membership list (names only) to [[Members:Protocols/Shell H2S/Global Training|Global Training for validation of H2S Alive course registration.]]
+
wuth@pcsar:~/sar/pc/membership$
-
xslt lib/MembersOnly.xsl < personnel-database.xml |xslt lib/MembersSorted.xsl | xslt lib/Name.xsl >tmp/member-list.txt
+
xslt lib/MembersOnly.xsl < personnel-database.xml | xslt lib/EMailPrimaryTxtWithNone.xsl >tmp/address-list
-
<pre>
+
cat /home/wuth/sar/pc/membership/tmp/address-list | sed -e 's|\(.*\) <\(.*\)>|sudo /usr/local/bin/wiki-members-add "\1" "\2"|' | bash
-
To: tanis.french@traininginc.ca
+
-
Subject: PCSAR member list
+
-
X-Mailer: VM 7.03 under 21.4 (patch 21) "Educational Television" XEmacs Lucid
+
-
Reply-To: wuth@castrov.cuug.ab.ca
+
-
FCC: ~/mail/outgoing/current
+
-
--text follows this line--
+
-
Tanis,
+
-
Please find attached the updated PCSAR membership for H2S Alive
+
will require manual manipulation to
-
training.
+
-
[ATTACHMENT /[pcsar]sar/pc/membership/tmp/member-list.txt, text/plain]
+
'''To Do:
 +
* '''(next)''' create tool to drop member of group
 +
* rename existing user
-
Take Care,
+
see [http://iweb.castrov.cuug.ab.ca/mediawiki/index.php/Mediawiki#Syncing_user_list personal notes]
-
--
+
-
Brett Wuth wuth@castrov.cuug.ab.ca wuth@acm.org
+
-
Box 1251-U, Pincher Creek, Alberta T0K 1W0, CANADA Tel:+1 403 627-2460
+
-
OpenPGP FingerPrint=628F C9DA BDBC 2A0E 18F1 2F6A 3300 8422 BE6A 0E79
+
-
What is the meaning of life?! Yes.
+
-
</pre>
+
==Non-monthly processing==
==Non-monthly processing==
Line 144: Line 226:
--text follows this line--
--text follows this line--
Thanks for joining Pincher Creek SAR. Can you please check if I've correctly
Thanks for joining Pincher Creek SAR. Can you please check if I've correctly
-
copied your membership record:
+
copied your personnel record:
---cut---
---cut---

Current revision

Contents

[edit] How to process the personnel database

Processing the database requires UNIX scripts, which I (Brett Wuth) keep on the PCSAR virtual machine.

Unless you understand UNIX, you're not likely to understand much of this. If you need to get someone else to do this, you'll need to find someone with UNIX or Linux shell skills.

The scripts check for changes in the database, so they keep a running record of the last changes they've seen.

[edit] Pending edits

Check my personal copy of the call-out list to see any marked-up changes that I've noted may be necessary.

[edit] Extract and validate

(review as used: Brett Wuth) to do

  • validate upon edit rather than wait for process
    • (reverified after 2015-07 damage) enable PCSARBot to receive email on watched pages
      • configure exim to receive e-mail locally
      • login as PCSARBot
      • change e-mail address to bot@pcsar.castrov.cuug.ab.ca
      • confirm new address
      • enable e-mail options for
        • E-mail me when a page I'm watching is changed
        • E-mail me also for minor edits of pages
    • (reverified after 2015-07 damage) validate Activities upon edit
    • validate each Personnel Record upon edit
      • (reverified after 2015-07 damage) write script to validate just member passed
        • see ~bot/sar/pc/membership/bin/wiki-membership-member-validate
        • Use "Brett Wuth" as experiment
      • (reverified after 2015-07 damage) update all memrec pages to include validation report
        • define as part of memrec template
      • develop trigger on memrec changes
        • develop dispatch on changes to individual memrec or Activities
          • (recheck) create trial watch on my mem record (Brett Wuth)
            • (done) Add PCSARBot to "TrustedGuest" group
          • currently uses exact match on page name in .procmailrc
          • (done) verify changes to Brett Wuth's MemRec triggers report update
          • develop mechanism to register dispatch for page, what script to run
            • on wiki itself
            • advantage: inspectability: easy for others self, to see what is going on
            • disadvantage: more work
            • single page Members:PCSARBot/Watches
              • each line like: * [[pagename]]: script
              • with possible duplicates of pagename, specifying different scripts
            • (done) write scripts to enable/disable watch on a wiki page
              • (done) wiki-page-watch-enable
              • (done) wiki-page-watch-disable
            • (done) write scripts to add/remove watch scripts
              • (done) wiki-watch-script-add
              • (done) wiki-watch-script-remove
            • (next) create mechanism to dispatch scripts based on email of page that changed; wiki-watch-scripts-dispatch
              • consider security of other person editing page
              • (done) create tool to determine revision number of page: ~/bin/wiki-page-revision-number-get
              • (done) create tool to determine editor of particular revision of page (not necessarily current revision)
                • (done) create tool wiki-page-editors-list that generates list of revision id's and editors
                • (done) create script wiki-page-editor that produces editor of latest or specified revision
              • (done) create tool to get the synced content and last editor of a page: wiki-page-get-with-editor
                • (done) create wiki-page-get
                • consider usage pattern get-latest-revision-number; get-page; verify lasest-revision-number unchanged
              • validate against whitelist of last editors
                • design choice: use single hardcoded name: 'Brett Wuth'
                • consider: can whitelist be stored on wiki page? That page would need to be validated itself. Could be rooted in a single hardcoded name.
                • shortcut: validate against single hardcoded name until need list
            • create mechanism to unwatch pages if email came in and no watch script registered
        • develop watch on memrec category
          • add new pages
          • delete removed pages
      • distribute previously created tools through git
      • allow Members: page to transclude report from another Members: page
      • when Activities changes, reprocess all Members
        • only if no errors in Activities itself -- otherwise error would appear on each page
        • only file report if change -- otherwise many unrelated updates

(review as used: Brett Wuth) to do: optimize for speed (last run 7min)

as user "wuth"

cd ~/sar/pc/membership
wiki-extract


  • Reconcile

make any necessary changes.

[edit] Process

xslt <personnel-database.xml lib/Applicants.xsl | xslt lib/Name.xsl


  • make sure current ssh-agent authority is extended
ssh -AX wuth@localhost
database-process

Manually check the Unicode in reports:

[edit] D4H export

2021-03-21 D4H has changed SSL protocols. Requires upgraded version python.

d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org membersImport

may need to re-download database to get newly assigned D4HMemberIDs

wiki-extract
d4h-web-interface.py user:pcsard4hbot@pcsar.ca.d4h.org activitiesAttendanceImport

[edit] Rave export

 wuth@pcsar:~/sar/pc/membership$ call-out-list-xml >tmp/call-out-list.xml
 wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ scp -p wuth@pcsar.castrov.cuug.ab.ca:sar/pc/membership/tmp/call-out-list.xml .
 wuth@desk:~/wuth/pcremo/rave/rave-api-tools$ python3 -m sar
    • sometimes fails at initial log in and has to be restarted
    • takes ~65 minutes to run

[edit] Continue

  • To Do: (review as used: Brett Wuth)
    • additional policy checks
      • write policy check for sar fund currency
        • Set type of existing Activities. See /[ssh/pcsar]/home/wuth/sar/pc/membership/lib/Membership.xsd ActivityCategoryType (done to 2012-01)
        • base on /[scp/pcsar]/home/wuth/sar/pc/membership/bin/policy-check-stale
          • see lib/SARFundStale.xsl
            • needs debugging
        • add to CVS
      • write policy check for voting member upgrade
      • write policy check for voting member currency
    • commonize elements of {bin,lib}/{call-out,first-call}


  • Personnel records: -- done by Adam now
lpr -Pofficejet8500 -o outputorder=reverse run/print-queue/*{recorder,SARCAR}*
  • Fax to Fire Dispatch in Lethbridge -- not done any more
first-call-list-fax


  • Print/Send the Fan-Out list -- no longer done
fan-out-list-send
  • Call Out List: -- done by Adam
    • for binder
    • for SARCAR
    • for overhead box at firehall
    • 3 copies

[edit] sync wiki accounts

wuth@pcsar:~/sar/pc/membership$
 xslt lib/MembersOnly.xsl < personnel-database.xml | xslt lib/EMailPrimaryTxtWithNone.xsl >tmp/address-list
 cat /home/wuth/sar/pc/membership/tmp/address-list | sed -e 's|\(.*\) <\(.*\)>|sudo /usr/local/bin/wiki-members-add "\1" "\2"|' | bash

will require manual manipulation to

To Do:

  • (next) create tool to drop member of group
  • rename existing user

see personal notes

[edit] Non-monthly processing

These are tasks that aren't done on a regular basis. Some of them haven't been done in long time and their instructions are now out of date.

[edit] copy to Brett's laptop

Because the remaining steps don't run on the PCSAR virtual machine yet, the file must be copied to Brett's laptop and run from there.

wuth@macbook:~/sar/pc/membership$ chmod 600 personnel-database.xml
wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/personnel-database.xml .
wuth@macbook:~/sar/pc/membership$ chmod 400 personnel-database.xml
wuth@macbook:~/sar/pc/membership$ scp -p wuth@pcsar:sar/pc/membership/tmp/personnel-database.txt tmp
wuth@macbook:~/sar/pc/membership$ rm -r run
wuth@macbook:~/sar/pc/membership$ scp -pr wuth@pcsar:sar/pc/membership/run .                     

[edit] To prepare a list of addresses and phone numbers of all our members

This was required for a casino application.

 xslt lib/MembersOnly.xsl < personnel-database.xml |xslt lib/NamePhoneAddress.xsl >tmp/addresses.txt

[edit] To print blank Personnel Records

  • keep 10 copies at hand
 personnel-blank-print 10

[edit] To send call-out list via e-mail

  • Stop sending in September 2003
 call-out-list-email

Call-out list to executive.

  • verify doesn't send when no change:
  • Add note about any deletions
  • automate production of

[edit] To e-mail mailing labels

  • Does not require anymore
mailing-labels-email

Mailing Labels sent to Jackie.

[edit] To e-mail changes for Resource database

      • Stop doing October, 2003. Need another way to handle resource database
personnel-database-mail-resource-database

[edit] When new members join, or e-mail addresses filled in

To: Subject: Welcome 
--text follows this line--
Thanks for joining Pincher Creek SAR. Can you please check if I've correctly
copied your personnel record:

---cut---
---cut---

Training sessions/meetings are the first Tuesday of every month,
usually starting at 19:00.

If you'd like to join in a club discussion and receive notices
via e-mail, please send a "subscribe" message to pcsar-discuss-request@castrov.cuug.ab.ca

Take Care,Brett Wuth wuth@acm.org
wuth@telusplanet.net bwuth@cuug.ab.ca Box 1251-U, Pincher Creek,
Alberta T0K 1W0, CANADA Tel:+1 403 627-2460 pgp-public-keys@uit.no
has mine; Print=E4F8EDEECBE01AD2FA3D8B2D94B1A292 What is the meaning of life?! Yes.

[edit] To print the participation dates

xslt <personnel-database.xml Activities.xsl | awk -v FS=":" {printf "%-24s:%-7s:%s\n", $1, $2, $3 } | mail -s "activity history" `MailRCAliasAddressGet "Bob Costa"` wuth
participation-report-make
  • Add any missing entries to partitipation-type.txt

Brett Wuth, 2006/07/15 09:35 MST (via web): Reformat the database after automatic translation using:

  xml-tidy "Training, Experience, Equipment, Notes" <personnel-database.xml >reformatted.xml
Personal tools