Skip to content

Commit a80c242

Browse files
pbailiebmcutler
pbailie
authored and
bmcutler
committed
Update student_auto_feed.md (#88)
Documentation update for the student auto feed. * Maintenance update, primarily to table of contents * New entries for 'users.user_numeric_id' column
1 parent 0ce7d46 commit a80c242

File tree

1 file changed

+88
-80
lines changed

1 file changed

+88
-80
lines changed

‎_docs/sysadmin/student_auto_feed.md‎

Lines changed: 88 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -7,53 +7,53 @@ order: 7
77
_Submitty Student Auto Feed is an optional command-line PHP script that can automatically
88
fill or update classlists on a cron schedule._
99

10-
## Table of Contents <a name="top"></a>
11-
1. [Requirements](#requirements)
12-
2. [Files](#files)
13-
3. [Course Database Backups](#database_backups)
14-
4. [Student CSV](#student_csv)
15-
* [CSV Layout](#csv_layout)
16-
5. [Install on Ubuntu 16.04](#install)
17-
6. [Command Line Arguments](#cli_arguments)
18-
7. [Configuration](#configuration)
19-
* [Configurations (top)](#configurations_top)
20-
* [Database Connection](#config_database)
21-
* [Error Logging](#config_logs)
22-
* [CSV File Access](#csv_file_access)
23-
* [CSV Delimiter](#csv_delimiter)
24-
* [CSV Validation](#config_csv_validation)
25-
* [CSV Fields Mapping](#csv_fields_mapping)
26-
* [Student Registration Codes](#registration_codes)
27-
* [Expected Term Code](#expected_term_code)
28-
* [Windows Encoding Conversion](#text_encoding)
29-
* [End of Line Detection](#eol_detection)
30-
* [About "allow_url_fopen"](#allow_url_fopen)
31-
* [Timezones](#config_timezones)
32-
8. [PAM Authentication and `accounts.php`](#pam_authentication)
33-
34-
### 1. Requirements <a name="requirements"></a>
10+
## Table of Contents
11+
1. [Requirements](#1-requirements)
12+
2. [Files](#2-files)
13+
3. [Course Database Backups](#3-course-database-backups)
14+
4. [Before Installing Auto Feed Script](#4-before-installing-auto-feed-script)
15+
* [Student CSV Layout](#41-student-csv-layout)
16+
5. [Install on Ubuntu Server](#5-install-on-ubuntu-server)
17+
6. [Command Line Arguments](#6-command-line-arguments)
18+
7. [Configuration](#7-configuration)
19+
* [Configurations (top)](#71-configurations)
20+
* [Database Connection](#database-connection)
21+
* [Error Logging](#error-logging)
22+
* [CSV File Access](#csv-file-access)
23+
* [CSV Delimiter](#csv-delimiter)
24+
* [CSV Validation](#csv-validation)
25+
* [CSV Fields Mapping](#csv-fields-mapping)
26+
* [Student Registration Codes](#student-registration-codes)
27+
* [Expected Term Code](#expected-term-code)
28+
* [Windows Encoding Conversion](#windows-encoding-conversion)
29+
* [End of Line Detection](#end-of-line-detection)
30+
* [About "allow_url_fopen"](#about-allow_url_fopen)
31+
* [Timezone](#timezone)
32+
8. [PAM Authentication and `accounts.php`](#8-pam-authentication-and-accountsphp)
33+
34+
### 1. Requirements
3535
* Submitty Student Auto Feed is intended to be managed by a systems administrator or similar IT professional.
3636
* PHP 5.6 or higher with `pgsql` and `iconv` extensions.
3737
* Although not a necessity, the auto feed script can operate on the same server that Submitty is running on.
3838
* The `ssh2` extension is also required if the data feed CSV resides on a different server than the script is running from (this also includes running the script on the Submitty server).
3939
* A regularly updated CSV data feed of student enrollment.
4040
* Contact your university's registrar and/or data warehouse for assistance.
4141

42-
<small>[Back To Table of Contents](#top)</small>
43-
### 2. Files <a name="files"></a>
42+
<small>[Back To Table of Contents](#table-of-contents)</small>
43+
### 2. Files
4444
Latest version of the auto feed script and supplmental files will be checked into the `master` branch in [`SysadminTools/student_auto_feed/`](https://github.com/Submitty/SysadminTools/tree/master/student_auto_feed)
4545
* `submitty_student_auto_feed.php` -- Executable PHP script to read student registration CSV and update Submitty classlist enrollment.
4646
* `config.php` -- **REQUIRED** config file for `submitty_student_auto_feed.php`
4747
* `accounts.php` -- **IMPORTANT** for Submitty servers utilizing PAM authentication.
4848

49-
<small>[Back To Table of Contents](#top)</small>
50-
### 3. Course Database Backups <a name="database_backups"></a>
49+
<small>[Back To Table of Contents](#table-of-contents)</small>
50+
### 3. Course Database Backups
5151
Please use `db_backup.py` (located in [`SysadminTools/nightly_db_backup/`](https://github.com/Submitty/SysadminTools/tree/master/nightly_db_backup)) on a cron schedule to create nightly backups of course databases.
5252

5353
Run `db_backup.py -h` to see extended help and argument list.
5454

55-
<small>[Back To Table of Contents](#top)</small>
56-
### 4. Before Installing Auto Feed Script <a name="student_csv"></a>
55+
<small>[Back To Table of Contents](#table-of-contents)</small>
56+
### 4. Before Installing Auto Feed Script
5757
It is important that you can receive a regularly updated data feed of student enrollment.
5858
The data should be tabulated (like a spreadsheet), but must be written as a CSV file.
5959
You will likely need the cooperation from your university's data warehouse and/or registrar.
@@ -66,32 +66,36 @@ student enrollment data protected by FERPA ([U.S. federal statute 20 U.S.C. § 1
6666
Please take appropriate information protection measures.
6767
**_SUBMITTY IS NOT RESPONSIBLE FOR YOUR COURSE'S, DEPARTMENT'S, OR UNIVERSITY'S INFORMATION CONTROL POLICIES OR ACTIVITIES._**
6868

69-
<small>[Back To Table of Contents](#top)</small>
70-
### 4.1 Student CSV Layout <a name="csv_layout"></a>
69+
<small>[Back To Table of Contents](#table-of-contents)</small>
70+
### 4.1 Student CSV Layout
7171

72-
There are nine _required_ columns/fields, and one optional column/field processed by the submitty auto feed script.
73-
The student CSV data file may have additional fields, which will be ignored by, _but there may not be less than the nine columns/fields required_.
72+
There are ten _required_ columns/fields, and one optional column/field processed by the submitty auto feed script.
73+
The student CSV data file may have additional fields, which will be ignored by, _but there may not be less than the ten columns/fields required_.
7474
The columns/fields may be in any order.
7575
`config.php` will map the requisite (and optional) columns/fields to their respective data points.
76-
(q.v. [CSV Fields Mapping](#csv_fields_mapping) for additional explanation)
76+
(q.v. [CSV Fields Mapping](#csv-fields-mapping) for additional explanation)
7777

7878
##### The student CSV file _must_ have fields/columns for these data points:
7979
1. Student's legal first name
8080
2. Student's legal last name
8181
3. Student's campus computer systems user account ID
8282
* The student CSV file should **_NEVER_** contain account passwords.
83-
4. Student's email address
84-
5. Course that student is enrolled in
83+
4. An alternate numeric ID.
84+
* This is intended to be the student's campus assign ID number, but could be
85+
any alternate alphanumeric ID code assigned to the student by the data feed.
86+
* This ID code can contain delimiter characters, such as dashes.
87+
5. Student's email address
88+
6. Course the student is enrolled in
8589
* This is actually **_two_** fields/columns.
8690
One is the course "prefix" (often the dept. code), and the other is the course "number".
8791
Combined, they make up the course code as listed in the course catalog.
8892
e.g. Prefix is "CSIS" and number is "101" as in "CSIS 101" in the course catalog.
89-
6. Student's registration status
93+
7. Student's registration status
9094
* A student can drop enrollment during an academic term, and this may be reflected by a code in the student CSV file.
9195
Alternatively, the student's enrollment entry, in the CSV file, can be entirely omitted when a course is dropped.
9296
The student auto feed script is intended to work with either case.
93-
7. Which lab/course section a student is enrolled in.
94-
8. The "term code" for the current academic term must exist in every data row.
97+
8. Which lab/course section a student is enrolled in.
98+
9. The "term code" for the current academic term must exist in every data row.
9599

96100
##### This data point is _optional_ in the student CSV file:
97101
1. Student's "preferred" first name
@@ -104,8 +108,8 @@ The columns/fields may be in any order.
104108
* Columns/field data should *not* be enclosed by quotes.
105109
Quotation marks may be picked up as part of the data and fail certain validation checks.
106110

107-
<small>[Back To Table of Contents](#top)</small>
108-
### 5. Install On Ubuntu Server <a name="install"></a>
111+
<small>[Back To Table of Contents](#table-of-contents)</small>
112+
### 5. Install On Ubuntu Server
109113
As these are PHP scripts, they _should_ run on any computer that has PHP 5.6+ and the appropriate extensions installed.
110114
However, these instructions will focus on Ubuntu server (same OS that is supported for Submitty). Ubuntu 16.04 uses PHP 7.0 by default, and Ubuntu 18.04 uses PHP 7.2 by default.
111115

@@ -132,8 +136,8 @@ sudo chmod 0700 student_submitty_auto_feed.php
132136
sudo chmod 0600 config.php
133137
```
134138

135-
<small>[Back To Table of Contents](#top)</small>
136-
### 6. Command Line Arguments <a name="cli_arguments"></a>
139+
<small>[Back To Table of Contents](#table-of-contents)</small>
140+
### 6. Command Line Arguments
137141

138142
--- | ---
139143
`-h` `--help` | Extended help including usage and argument list.
@@ -150,8 +154,8 @@ Notes:
150154
- **YY** is the two digit year.
151155
- e.g. **s18** is Spring of 2018.
152156

153-
<small>[Back To Table of Contents](#top)</small>
154-
### 7. Configuration <a name="configuration"></a>
157+
<small>[Back To Table of Contents](#table-of-contents)</small>
158+
### 7. Configuration
155159
Configuration options exist in `config.php` as "constants".
156160
The goal, here, is to define each constant to a value reflective of your use of Submitty.
157161
The provided defaults, while illustrative, typically will not work.
@@ -191,13 +195,13 @@ Summaries are also provided as "code comments" within `config.php`.
191195
Consistent with C and Java styles, PHP code comments either begin with double slashes `//` or are multiple lines between `/*` and `*/`.
192196
Using a text editor with syntax highlighting will be highly beneficial as code comments will be given a unique text color (text coloring will vary from editor to editor).
193197

194-
<small>[Back To Table of Contents](#top)</small>
195-
### 7.1 Configurations <a name="configurations_top"></a>
198+
<small>[Back To Table of Contents](#table-of-contents)</small>
199+
### 7.1 Configurations
196200
These options are set in `config.php`.
197201
`config.php` must exist in the same directory and be accessible by the same user account as `submitty_student_auto_feed.php`.
198202

199-
<small>[Back To Table of Contents](#top)</small>
200-
#### Database Connection<a name="config_database"></a>
203+
<small>[Back To Table of Contents](#table-of-contents)</small>
204+
#### Database Connection
201205
```php
202206
define('DB_HOST', 'submitty.cs.myuniversity.edu');
203207
define('DB_LOGIN', 'submitty_dbuser');
@@ -210,8 +214,8 @@ These options specify the login to the Submitty database for the hostname of the
210214

211215
Note that the database is often on the same server as Submitty, but this is not required. The database can be hosted on a separate server from Submitty.
212216

213-
<small>[Back To Table of Contents](#top)</small>
214-
#### Error Logging <a name="config_logs"></a>
217+
<small>[Back To Table of Contents](#table-of-contents)</small>
218+
#### Error Logging
215219
```php
216220
define('ERROR_EMAIL', '[email protected]');
217221
define('ERROR_LOG_FILE', '/location/of/auto_feed_error.log');
@@ -226,8 +230,8 @@ Emailing error messages can be disabled by setting the value to `null` (without
226230
Your campus may restrict or outright deny delivery of the error-log emails.
227231
Consult with your University's IT department regarding _unauthenticated_ email.
228232

229-
<small>[Back To Table of Contents](#top)</small>
230-
#### CSV File Access <a name="csv_file_access"></a>
233+
<small>[Back To Table of Contents](#table-of-contents)</small>
234+
#### CSV File Access
231235
```php
232236
define('CSV_AUTH', 'remote_keypair');
233237
define('CSV_FILE', '/path/to/datafile.csv');
@@ -268,8 +272,8 @@ These constants define how the CSV data can be accessed.
268272
* Don't encrypt the private key (this does carry additional data security risk).
269273
* Host/run the student auto feed script on a different Linux distribution or operating system.
270274

271-
<small>[Back To Table of Contents](#top)</small>
272-
#### CSV Delimiter <a name="csv_delimiter"></a>
275+
<small>[Back To Table of Contents](#table-of-contents)</small>
276+
#### CSV Delimiter
273277
```php
274278
define('CSV_DELIM_CHAR', chr(9));
275279
```
@@ -299,8 +303,8 @@ Here are some example delimiters:
299303
define('CSV_DELIM_CHAR', ';');
300304
```
301305

302-
<small>[Back To Table of Contents](#top)</small>
303-
#### CSV Validation <a name="config_csv_validation"></a>
306+
<small>[Back To Table of Contents](#table-of-contents)</small>
307+
#### CSV Validation
304308
```php
305309
define('VALIDATE_MIN_FILESIZE', 65536);
306310
define('VALIDATE_NUM_FIELDS', 10);
@@ -320,18 +324,19 @@ This is useful to detect an egregiously small CSV that could indicate data corru
320324
Any row that does not have this exact value is expected to have unreliable data and is ignored by the auto feed script.
321325
This value includes any extraneous fields/columns that your University's registrar/data warehouse provides.
322326

323-
Even though the auto feed requires nine columns, the CSV being provided may have more.
327+
Even though the auto feed requires ten columns, the CSV being provided may have more.
324328
If so, use the number of columns _in the CSV_ to set this option.
325329
Otherwise, all columns may be ignored and no enrollment additions or updates will be recorded.
326330

327-
<small>[Back To Table of Contents](#top)</small>
328-
#### CSV Fields Mapping <a name="csv_fields_mapping"></a>
331+
<small>[Back To Table of Contents](#table-of-contents)</small>
332+
#### CSV Fields Mapping
329333
```php
330334
define('COLUMN_COURSE_PREFIX', 8);
331335
define('COLUMN_COURSE_NUMBER', 9);
332336
define('COLUMN_REGISTRATION', 7);
333337
define('COLUMN_SECTION', 10);
334338
define('COLUMN_USER_ID', 5);
339+
define('COLUMN_NUMERIC_ID', 6);
335340
define('COLUMN_FIRSTNAME', 2);
336341
define('COLUMN_LASTNAME', 1);
337342
define('COLUMN_PREFERREDNAME', 3);
@@ -360,12 +365,15 @@ That is, the first column of the CSV is #0, the second column is #1, the third c
360365
_Course Listed In Catalog_ | "CSIS 101" or "CSIS-101"
361366

362367
* `COLUMN_REGISTRATION` represents the column that has a code to verify that a student is enrolled or not.
363-
q.v. [Student Registration Codes](#registration_codes)
368+
q.v. [Student Registration Codes](#student-registration-codes)
364369

365370
* `COLUMN_SECTION` is the column representing the lab/course section a student is enrolled in.
366371

367372
* `COLUMN_USER_ID` is a student's computing systems user ID, and is also used to login to Submitty.
368373

374+
* `COLUMN_NUMERIC_ID` is intended to represent the student's campus assigned ID number, but could be any alternate ID number provided by a data feed.
375+
Although this column is called "numeric", any alphanumeric code is permitted with delimiter characters, such as dashes.
376+
369377
* `COLUMN_FIRSTNAME` is a student's legal first name.
370378

371379
* `COLUMN_LASTNAME` is a student's legal last name.
@@ -379,8 +387,8 @@ That is, the first column of the CSV is #0, the second column is #1, the third c
379387
This is checked against the "expected" term code for validation.
380388
q.v. [Expected Term Code](#expected_term_code)
381389

382-
<small>[Back To Table of Contents](#top)</small>
383-
#### Student Registration Codes <a name="registration_codes"></a>
390+
<small>[Back To Table of Contents](#table-of-contents)</small>
391+
#### Student Registration Codes
384392
```php
385393
define('STUDENT_REGISTERED_CODES', serialize( array(
386394
'RA',
@@ -405,8 +413,8 @@ In this case, `RA` may mean "Registered by Adviser" and `RW` may mean "Registere
405413
* Any student not associated with a registration code as listed in this option is assumed to have dropped the course or has otherwise been unregistered for some reason.
406414
In which case, an update will occur in Submitty's database to reflect the student is no longer enrolled in that course.
407415

408-
<small>[Back To Table of Contents](#top)</small>
409-
#### Expected Term Code <a name="expected_term_code"></a>
416+
<small>[Back To Table of Contents](#table-of-contents)</small>
417+
#### Expected Term Code
410418
```php
411419
define('EXPECTED_TERM_CODE', '201705');
412420
```
@@ -423,8 +431,8 @@ The student auto feed will check every row for this code and compare it with thi
423431
Rows that do not match the `define` value will be ignored.
424432
It is possible that when one row does not match, all rows will not match.
425433

426-
<small>[Back To Table of Contents](#top)</small>
427-
#### Windows Encoding Conversion <a name="text_encoding"></a>
434+
<small>[Back To Table of Contents](#table-of-contents)</small>
435+
#### Windows Encoding Conversion
428436
```php
429437
define('CONVERT_CP1252', true);
430438
```
@@ -433,16 +441,16 @@ If your student CSV originates from a Windows computer, the auto feed may need t
433441
Set `CONVERT_CP1252` to `true` if the student CSV originates from a Windows computer.
434442
Otherwise, set to `false`.
435443

436-
<small>[Back To Table of Contents](#top)</small>
437-
#### End of Line Detection <a name="eol_detection"></a>
444+
<small>[Back To Table of Contents](#table-of-contents)</small>
445+
#### End of Line Detection
438446
```php
439447
ini_set('auto_detect_line_endings', true);
440448
```
441449
In summary, this `define` shouldn't be changed.
442450
It ensures that CSV files exported by Microsoft Excel for Macintosh are correctly processed.
443451

444-
<small>[Back To Table of Contents](#top)</small>
445-
#### About "allow_url_fopen" <a name="allow_url_fopen"></a>
452+
<small>[Back To Table of Contents](#table-of-contents)</small>
453+
#### About "allow_url_fopen"
446454
```php
447455
ini_set("allow_url_fopen", true);
448456
```
@@ -452,8 +460,8 @@ Otherwise, the server would have to be accessed via IP address.
452460

453461
This setting is irrelevant when the student CSV can be locally accessed.
454462

455-
<small>[Back To Table of Contents](#top)</small>
456-
#### Timezone <a name="config_timezones"></a>
463+
<small>[Back To Table of Contents](#table-of-contents)</small>
464+
#### Timezone
457465
```php
458466
date_default_timezone_set('America/New_York');
459467
```
@@ -474,19 +482,19 @@ Hawaii (no daylight savings) | `Pacific/Honolulu`
474482

475483
For a complete list of timezones: <http://php.net/manual/en/timezones.php>
476484

477-
<small>[Back To Table of Contents](#top)</small>
485+
<small>[Back To Table of Contents](#table-of-contents)</small>
478486

479-
### 8. PAM Authentication and `accounts.php` <a name="pam_authentication"></a>
487+
### 8. PAM Authentication and `accounts.php`
480488
The script `accounts.php` will automate the creation of local accounts used with PAM authentication.
481489

482490
*This script is not needed when using database authentication.*
483491

484492
`accounts.php` must exist on the same server as Submitty, and it needs to be run as `root`.
485-
This script is intended to read user entries from Submitty's course databases generate any missing local accounts needed for PAM authentication.
493+
This script is intended to read user entries from Submitty's "master" database to generate any missing local accounts needed for PAM authentication.
486494

487495
Run `accounts.php -h` to see extended help and argument list.
488496

489497
It is recommended that this script is run every hour as a cron job.
490-
Should an instructor manually add a student to their course, the student's access to Submitty will be available "within an hour".
498+
That way, should an instructor manually add a student to their course, the student's access to Submitty will be available "within an hour".
491499

492-
<small>[Back To Table of Contents](#top)</small>
500+
<small>[Back To Table of Contents](#table-of-contents)</small>

0 commit comments

Comments
 (0)