diff --git a/_docs/sysadmin/student_auto_feed.md b/_docs/sysadmin/student_auto_feed.md index ec5a95f8..5a0ea753 100644 --- a/_docs/sysadmin/student_auto_feed.md +++ b/_docs/sysadmin/student_auto_feed.md @@ -7,31 +7,31 @@ order: 7 _Submitty Student Auto Feed is an optional command-line PHP script that can automatically fill or update classlists on a cron schedule._ -## Table of Contents -1. [Requirements](#requirements) -2. [Files](#files) -3. [Course Database Backups](#database_backups) -4. [Student CSV](#student_csv) - * [CSV Layout](#csv_layout) -5. [Install on Ubuntu 16.04](#install) -6. [Command Line Arguments](#cli_arguments) -7. [Configuration](#configuration) - * [Configurations (top)](#configurations_top) - * [Database Connection](#config_database) - * [Error Logging](#config_logs) - * [CSV File Access](#csv_file_access) - * [CSV Delimiter](#csv_delimiter) - * [CSV Validation](#config_csv_validation) - * [CSV Fields Mapping](#csv_fields_mapping) - * [Student Registration Codes](#registration_codes) - * [Expected Term Code](#expected_term_code) - * [Windows Encoding Conversion](#text_encoding) - * [End of Line Detection](#eol_detection) - * [About "allow_url_fopen"](#allow_url_fopen) - * [Timezones](#config_timezones) -8. [PAM Authentication and `accounts.php`](#pam_authentication) - -### 1. Requirements +## Table of Contents +1. [Requirements](#1-requirements) +2. [Files](#2-files) +3. [Course Database Backups](#3-course-database-backups) +4. [Before Installing Auto Feed Script](#4-before-installing-auto-feed-script) + * [Student CSV Layout](#41-student-csv-layout) +5. [Install on Ubuntu Server](#5-install-on-ubuntu-server) +6. [Command Line Arguments](#6-command-line-arguments) +7. [Configuration](#7-configuration) + * [Configurations (top)](#71-configurations) + * [Database Connection](#database-connection) + * [Error Logging](#error-logging) + * [CSV File Access](#csv-file-access) + * [CSV Delimiter](#csv-delimiter) + * [CSV Validation](#csv-validation) + * [CSV Fields Mapping](#csv-fields-mapping) + * [Student Registration Codes](#student-registration-codes) + * [Expected Term Code](#expected-term-code) + * [Windows Encoding Conversion](#windows-encoding-conversion) + * [End of Line Detection](#end-of-line-detection) + * [About "allow_url_fopen"](#about-allow_url_fopen) + * [Timezone](#timezone) +8. [PAM Authentication and `accounts.php`](#8-pam-authentication-and-accountsphp) + +### 1. Requirements * Submitty Student Auto Feed is intended to be managed by a systems administrator or similar IT professional. * PHP 5.6 or higher with `pgsql` and `iconv` extensions. * Although not a necessity, the auto feed script can operate on the same server that Submitty is running on. @@ -39,21 +39,21 @@ fill or update classlists on a cron schedule._ * A regularly updated CSV data feed of student enrollment. * Contact your university's registrar and/or data warehouse for assistance. -[Back To Table of Contents](#top) -### 2. Files +[Back To Table of Contents](#table-of-contents) +### 2. Files Latest version of the auto feed script and supplmental files will be checked into the `master` branch in [`SysadminTools/student_auto_feed/`](https://github.com/Submitty/SysadminTools/tree/master/student_auto_feed) * `submitty_student_auto_feed.php` -- Executable PHP script to read student registration CSV and update Submitty classlist enrollment. * `config.php` -- **REQUIRED** config file for `submitty_student_auto_feed.php` * `accounts.php` -- **IMPORTANT** for Submitty servers utilizing PAM authentication. -[Back To Table of Contents](#top) -### 3. Course Database Backups +[Back To Table of Contents](#table-of-contents) +### 3. Course Database Backups Please use `db_backup.py` (located in [`SysadminTools/nightly_db_backup/`](https://github.com/Submitty/SysadminTools/tree/master/nightly_db_backup)) on a cron schedule to create nightly backups of course databases. Run `db_backup.py -h` to see extended help and argument list. -[Back To Table of Contents](#top) -### 4. Before Installing Auto Feed Script +[Back To Table of Contents](#table-of-contents) +### 4. Before Installing Auto Feed Script It is important that you can receive a regularly updated data feed of student enrollment. The data should be tabulated (like a spreadsheet), but must be written as a CSV file. You will likely need the cooperation from your university's data warehouse and/or registrar. @@ -66,32 +66,36 @@ student enrollment data protected by FERPA ([U.S. federal statute 20 U.S.C. ยง 1 Please take appropriate information protection measures. **_SUBMITTY IS NOT RESPONSIBLE FOR YOUR COURSE'S, DEPARTMENT'S, OR UNIVERSITY'S INFORMATION CONTROL POLICIES OR ACTIVITIES._** -[Back To Table of Contents](#top) -### 4.1 Student CSV Layout +[Back To Table of Contents](#table-of-contents) +### 4.1 Student CSV Layout -There are nine _required_ columns/fields, and one optional column/field processed by the submitty auto feed script. -The student CSV data file may have additional fields, which will be ignored by, _but there may not be less than the nine columns/fields required_. +There are ten _required_ columns/fields, and one optional column/field processed by the submitty auto feed script. +The student CSV data file may have additional fields, which will be ignored by, _but there may not be less than the ten columns/fields required_. The columns/fields may be in any order. `config.php` will map the requisite (and optional) columns/fields to their respective data points. -(q.v. [CSV Fields Mapping](#csv_fields_mapping) for additional explanation) +(q.v. [CSV Fields Mapping](#csv-fields-mapping) for additional explanation) ##### The student CSV file _must_ have fields/columns for these data points: 1. Student's legal first name 2. Student's legal last name 3. Student's campus computer systems user account ID * The student CSV file should **_NEVER_** contain account passwords. -4. Student's email address -5. Course that student is enrolled in +4. An alternate numeric ID. + * This is intended to be the student's campus assign ID number, but could be + any alternate alphanumeric ID code assigned to the student by the data feed. + * This ID code can contain delimiter characters, such as dashes. +5. Student's email address +6. Course the student is enrolled in * This is actually **_two_** fields/columns. One is the course "prefix" (often the dept. code), and the other is the course "number". Combined, they make up the course code as listed in the course catalog. e.g. Prefix is "CSIS" and number is "101" as in "CSIS 101" in the course catalog. -6. Student's registration status +7. Student's registration status * A student can drop enrollment during an academic term, and this may be reflected by a code in the student CSV file. Alternatively, the student's enrollment entry, in the CSV file, can be entirely omitted when a course is dropped. The student auto feed script is intended to work with either case. -7. Which lab/course section a student is enrolled in. -8. The "term code" for the current academic term must exist in every data row. +8. Which lab/course section a student is enrolled in. +9. The "term code" for the current academic term must exist in every data row. ##### This data point is _optional_ in the student CSV file: 1. Student's "preferred" first name @@ -104,8 +108,8 @@ The columns/fields may be in any order. * Columns/field data should *not* be enclosed by quotes. Quotation marks may be picked up as part of the data and fail certain validation checks. -[Back To Table of Contents](#top) -### 5. Install On Ubuntu Server +[Back To Table of Contents](#table-of-contents) +### 5. Install On Ubuntu Server As these are PHP scripts, they _should_ run on any computer that has PHP 5.6+ and the appropriate extensions installed. However, these instructions will focus on Ubuntu server (same OS that is supported for Submitty). Ubuntu 16.04 uses PHP 7.0 by default, and Ubuntu 18.04 uses PHP 7.2 by default. @@ -132,8 +136,8 @@ sudo chmod 0700 student_submitty_auto_feed.php sudo chmod 0600 config.php ``` -[Back To Table of Contents](#top) -### 6. Command Line Arguments +[Back To Table of Contents](#table-of-contents) +### 6. Command Line Arguments --- | --- `-h` `--help` | Extended help including usage and argument list. @@ -150,8 +154,8 @@ Notes: - **YY** is the two digit year. - e.g. **s18** is Spring of 2018. -[Back To Table of Contents](#top) -### 7. Configuration +[Back To Table of Contents](#table-of-contents) +### 7. Configuration Configuration options exist in `config.php` as "constants". The goal, here, is to define each constant to a value reflective of your use of Submitty. The provided defaults, while illustrative, typically will not work. @@ -191,13 +195,13 @@ Summaries are also provided as "code comments" within `config.php`. Consistent with C and Java styles, PHP code comments either begin with double slashes `//` or are multiple lines between `/*` and `*/`. Using a text editor with syntax highlighting will be highly beneficial as code comments will be given a unique text color (text coloring will vary from editor to editor). -[Back To Table of Contents](#top) -### 7.1 Configurations +[Back To Table of Contents](#table-of-contents) +### 7.1 Configurations These options are set in `config.php`. `config.php` must exist in the same directory and be accessible by the same user account as `submitty_student_auto_feed.php`. -[Back To Table of Contents](#top) -#### Database Connection +[Back To Table of Contents](#table-of-contents) +#### Database Connection ```php define('DB_HOST', 'submitty.cs.myuniversity.edu'); define('DB_LOGIN', 'submitty_dbuser'); @@ -210,8 +214,8 @@ These options specify the login to the Submitty database for the hostname of the Note that the database is often on the same server as Submitty, but this is not required. The database can be hosted on a separate server from Submitty. -[Back To Table of Contents](#top) -#### Error Logging +[Back To Table of Contents](#table-of-contents) +#### Error Logging ```php define('ERROR_EMAIL', 'sysadmins@lists.myuniversity.edu'); define('ERROR_LOG_FILE', '/location/of/auto_feed_error.log'); @@ -226,8 +230,8 @@ Emailing error messages can be disabled by setting the value to `null` (without Your campus may restrict or outright deny delivery of the error-log emails. Consult with your University's IT department regarding _unauthenticated_ email. -[Back To Table of Contents](#top) -#### CSV File Access +[Back To Table of Contents](#table-of-contents) +#### CSV File Access ```php define('CSV_AUTH', 'remote_keypair'); define('CSV_FILE', '/path/to/datafile.csv'); @@ -268,8 +272,8 @@ These constants define how the CSV data can be accessed. * Don't encrypt the private key (this does carry additional data security risk). * Host/run the student auto feed script on a different Linux distribution or operating system. -[Back To Table of Contents](#top) -#### CSV Delimiter +[Back To Table of Contents](#table-of-contents) +#### CSV Delimiter ```php define('CSV_DELIM_CHAR', chr(9)); ``` @@ -299,8 +303,8 @@ Here are some example delimiters: define('CSV_DELIM_CHAR', ';'); ``` -[Back To Table of Contents](#top) -#### CSV Validation +[Back To Table of Contents](#table-of-contents) +#### CSV Validation ```php define('VALIDATE_MIN_FILESIZE', 65536); define('VALIDATE_NUM_FIELDS', 10); @@ -320,18 +324,19 @@ This is useful to detect an egregiously small CSV that could indicate data corru Any row that does not have this exact value is expected to have unreliable data and is ignored by the auto feed script. This value includes any extraneous fields/columns that your University's registrar/data warehouse provides. - Even though the auto feed requires nine columns, the CSV being provided may have more. + Even though the auto feed requires ten columns, the CSV being provided may have more. If so, use the number of columns _in the CSV_ to set this option. Otherwise, all columns may be ignored and no enrollment additions or updates will be recorded. -[Back To Table of Contents](#top) -#### CSV Fields Mapping +[Back To Table of Contents](#table-of-contents) +#### CSV Fields Mapping ```php define('COLUMN_COURSE_PREFIX', 8); define('COLUMN_COURSE_NUMBER', 9); define('COLUMN_REGISTRATION', 7); define('COLUMN_SECTION', 10); define('COLUMN_USER_ID', 5); +define('COLUMN_NUMERIC_ID', 6); define('COLUMN_FIRSTNAME', 2); define('COLUMN_LASTNAME', 1); define('COLUMN_PREFERREDNAME', 3); @@ -360,12 +365,15 @@ That is, the first column of the CSV is #0, the second column is #1, the third c _Course Listed In Catalog_ | "CSIS 101" or "CSIS-101" * `COLUMN_REGISTRATION` represents the column that has a code to verify that a student is enrolled or not. - q.v. [Student Registration Codes](#registration_codes) + q.v. [Student Registration Codes](#student-registration-codes) * `COLUMN_SECTION` is the column representing the lab/course section a student is enrolled in. * `COLUMN_USER_ID` is a student's computing systems user ID, and is also used to login to Submitty. +* `COLUMN_NUMERIC_ID` is intended to represent the student's campus assigned ID number, but could be any alternate ID number provided by a data feed. + Although this column is called "numeric", any alphanumeric code is permitted with delimiter characters, such as dashes. + * `COLUMN_FIRSTNAME` is a student's legal first name. * `COLUMN_LASTNAME` is a student's legal last name. @@ -379,8 +387,8 @@ That is, the first column of the CSV is #0, the second column is #1, the third c This is checked against the "expected" term code for validation. q.v. [Expected Term Code](#expected_term_code) -[Back To Table of Contents](#top) -#### Student Registration Codes +[Back To Table of Contents](#table-of-contents) +#### Student Registration Codes ```php define('STUDENT_REGISTERED_CODES', serialize( array( 'RA', @@ -405,8 +413,8 @@ In this case, `RA` may mean "Registered by Adviser" and `RW` may mean "Registere * Any student not associated with a registration code as listed in this option is assumed to have dropped the course or has otherwise been unregistered for some reason. In which case, an update will occur in Submitty's database to reflect the student is no longer enrolled in that course. -[Back To Table of Contents](#top) -#### Expected Term Code +[Back To Table of Contents](#table-of-contents) +#### Expected Term Code ```php define('EXPECTED_TERM_CODE', '201705'); ``` @@ -423,8 +431,8 @@ The student auto feed will check every row for this code and compare it with thi Rows that do not match the `define` value will be ignored. It is possible that when one row does not match, all rows will not match. -[Back To Table of Contents](#top) -#### Windows Encoding Conversion +[Back To Table of Contents](#table-of-contents) +#### Windows Encoding Conversion ```php define('CONVERT_CP1252', true); ``` @@ -433,16 +441,16 @@ If your student CSV originates from a Windows computer, the auto feed may need t Set `CONVERT_CP1252` to `true` if the student CSV originates from a Windows computer. Otherwise, set to `false`. -[Back To Table of Contents](#top) -#### End of Line Detection +[Back To Table of Contents](#table-of-contents) +#### End of Line Detection ```php ini_set('auto_detect_line_endings', true); ``` In summary, this `define` shouldn't be changed. It ensures that CSV files exported by Microsoft Excel for Macintosh are correctly processed. -[Back To Table of Contents](#top) -#### About "allow_url_fopen" +[Back To Table of Contents](#table-of-contents) +#### About "allow_url_fopen" ```php ini_set("allow_url_fopen", true); ``` @@ -452,8 +460,8 @@ Otherwise, the server would have to be accessed via IP address. This setting is irrelevant when the student CSV can be locally accessed. -[Back To Table of Contents](#top) -#### Timezone +[Back To Table of Contents](#table-of-contents) +#### Timezone ```php date_default_timezone_set('America/New_York'); ``` @@ -474,19 +482,19 @@ Hawaii (no daylight savings) | `Pacific/Honolulu` For a complete list of timezones: -[Back To Table of Contents](#top) +[Back To Table of Contents](#table-of-contents) -### 8. PAM Authentication and `accounts.php` +### 8. PAM Authentication and `accounts.php` The script `accounts.php` will automate the creation of local accounts used with PAM authentication. *This script is not needed when using database authentication.* `accounts.php` must exist on the same server as Submitty, and it needs to be run as `root`. -This script is intended to read user entries from Submitty's course databases generate any missing local accounts needed for PAM authentication. +This script is intended to read user entries from Submitty's "master" database to generate any missing local accounts needed for PAM authentication. Run `accounts.php -h` to see extended help and argument list. It is recommended that this script is run every hour as a cron job. -Should an instructor manually add a student to their course, the student's access to Submitty will be available "within an hour". +That way, should an instructor manually add a student to their course, the student's access to Submitty will be available "within an hour". -[Back To Table of Contents](#top) +[Back To Table of Contents](#table-of-contents)