Classroom to cloud sync

This document describes one subsystem only: how offline classroom activity (student progress, grades-related progress, access logs, etc.) recorded on the local Pi API is moved to the central LMS API using a ZIP file carried on a teacher's mobile device.

It is written for code review and comparison with another LMS that also uses a Raspberry Pi edge node. For the full multi-repo map, see the architecture documentation.

Status: the central API's import side (PUT /log/import, section 6) is off by default — it only runs when a deployment sets LOG_IMPORT_ENABLED to true or 1. Nothing in the current apps calls it: the native teacher app that drove this flow is retired, and the current Expo learner app does not call it. The rest of this page still documents the protocol both APIs implement, for the reasons above.

1. What problem this solves

Students use the app against the Pi on a classroom LAN (offline from central).
Progress is stored in the Pi MySQL database.
Operators need the same class of data in the central cloud for reporting and single source of truth.
The chosen pattern is batch export / import via ZIP, initiated by a teacher, not continuous replication from the Pi to the cloud.

2. End-to-end flow (summary)

Step	Actor	System	Action
1	Teacher	Pi (`edtech-lms-rpi-api`)	`GET /export/log` with Pi JWT → HTTP response is a ZIP
2	Mobile app	Device storage	Save body as `studentlog.zip`
3	Teacher	Central (`edtech-lms-api`)	`PUT /log/import` with LMS JWT → multipart field `importfile` = that ZIP
4	Central	MySQL + S3	Parse ZIP, upsert progress tables, archive ZIP to S3, record sync row

3. What data is in the ZIP (semantic scope)

The Pi builds log.ini as JSON (UTF-8) from LogBusiness.exportlog() in edtech-lms-rpi-api. It aggregates roughly the last six months of:

studentprogress and nested studentprogressquestions (quiz/session style outcomes)
rpiuseraccess (login/access)
studentactives
studentlearningprogress
studentgradesprogress
studentlevelsprogress
studentlessonsprogress
studentpoints
studentappusages

The JSON is wrapped in a structure consumed by the central importer as Istudentprogress (log.access, log.result, log.progress, etc.). See the Pi log.business.ts (exportlog) and central log.business.ts (importstudentsprogress).

Important distinction: This path moves activity and progress that already exists on the Pi. It is not the primary mechanism for creating student roster records on the cloud from the Pi. Roster-style sync from cloud to the Pi uses other endpoints (e.g. central POST /sync/cloud/.../students and Pi PUT /import/students).

4. Pi API: export

Repository: edtech-lms-rpi-api
Route: GET /export/log
Controller: src/modules/export/export.controller.ts
Guard: AccessGuard(TokenType.ACCESS): expects a valid Pi access JWT.

ZIP contents:

log.ini, JSON.stringify(LogBusiness.exportlog()) (see section 3).
Optional: files from the Pi log directory on disk whose names include RPI-API-error or RPI-API-info, added as additional zip entries (plain text).

Response headers: Content-Type: application/zip; Content-Disposition attachment with a date/time based filename.

5. Mobile app: transport and UX

The native Android teacher app that implemented this flow is retired and its repository is archived. This page documents the protocol it defined, which the two APIs still implement. The current Expo learner app does not implement teacher upload.

5.1 Two base URLs

The app configured two HTTP client base URLs:

Field	Typical use
Pi base URL	Pi classroom API
Upload base URL	Central LMS API

5.2 Client operations

Pi: GET export/log with Authorization: Bearer <token>
Cloud: PUT log/import with Authorization: Bearer <token> and a multipart part (field name importfile in the multipart implementation)

5.3 Teacher UI

Download (when launched in Pi-oriented mode): calls Pi export/log, writes studentlog.zip to external storage (path depends on device storage settings, SD card path or a subdirectory under external storage).
Upload (when launched in online-oriented mode): reads studentlog.zip, builds multipart upload to the upload base URL's log/import.

The teacher screen also chained behavior unrelated to log sync: after a successful cloud log upload it may call GET sync on the central API to refresh syncData.zip (curriculum), and after Pi log download it may PUT import/master on the Pi if the Pi connection check passes. Those are curriculum / master paths, not the log pipeline. PUT import/master accepts a teacher token only when the Pi runs with RPI_OFFLINE=true (or "offline": true in FORTYKAPIRPICONFIG); otherwise only central's SERVER_SYNC_KEY can call it.

5.4 The Pi connection check

In the retired client this check was a string equality comparison between the configured Pi base URL and a placeholder URL, not a network probe. Treat this as build- or environment-specific when comparing to another LMS client; the intended idea is to branch "classroom / Pi mode" vs "online only" for optional chaining.

5.5 Authentication note

Both Pi download and cloud upload use the same stored access token. Your deployment must ensure that token is valid for both hosts, or that login flows align so the correct token is used per base URL. This is a common integration point when comparing another LMS.

6. Central LMS API: import

Repository: edtech-lms-api
Route: PUT /log/import
Controller: src/modules/log/log.controller.ts
Guards: AccessGuard(TokenType.ACCESS) and CheckPermissionsGuard (RBAC).

Availability: off unless the deployment sets LOG_IMPORT_ENABLED (true/1); when off, the route 404s before auth or upload parsing runs.

Multipart: field name importfile (ZIP buffer).

Query: optional offline (boolean), forwarded into sync activity recording.

6.1 Unzip and validation rules

Uses unzipper to iterate entries:

Path exactly log.ini → parse JSON → LogBusiness.importstudentsprogress(logdata).
Path contains RPI-API → createstudentaccesslogfiles (metadata in logfiles).
Any other entry → 400 BadRequestException ("invalid file inside zip-file").

The code calls file.buffer("<value in code>") when reading entries (unzipper password parameter). Pi-generated zips are not encrypted; this is effectively part of the contract of the current implementation.

6.2 Database writes

Class: src/business/log.business.ts, importstudentsprogress:

Upserts access (importaccesslog)
Upserts result rows and related progress questions (importprogresslog, importprogressquestionlog)
Upserts progress aggregates (importstudentprogresslog) across the nested progress tables

All use Sequelize bulkCreate with updateOnDuplicate for idempotent merges.

6.3 Side effects

recordSyncActivity: inserts into syncs (ties to teacher schooluserid, filename, offline flag).
uploadZipFileToAWSS3: stores the entire uploaded ZIP in S3 and records logfiles with type: 2.

Work runs inside a DB transaction; failure rolls back and returns an error response.

7. Operational characteristics (for comparison)

Topic	This system
Coupling	Strong contract on ZIP layout and `log.ini` JSON shape
Latency	Batch only; no real-time sync
Initiator	Human teacher + mobile app
Conflict strategy	Upsert by duplicate keys in Sequelize models, not CRDT
Audit	`syncs` row + S3 copy of ZIP + optional RPI-API file metadata
Time window	Pi export ~6 months of data (see `subMonths` in Pi `exportlog`)

8. Checklist: compare another LMS + Pi

Use this when reviewing a different stack against this design.

Edge export. Does the Pi (or edge) expose a single downloadable artifact (ZIP or other) that bundles all required tables?
Schema mapping. Is the payload one JSON blob or multiple files? Does it separate roster vs progress?
Auth. Separate tokens for edge vs cloud, or shared? How does the mobile app store them?
Import semantics. Full replace vs upsert vs append? How are duplicates keyed?
Size limits. Time window, max ZIP size, timeouts (mobile uses default HTTP for download; extended client for large Pi uploads elsewhere).
Failure handling. Partial import, rollback, idempotent retries if the teacher uploads twice?
Observability. Central sync records, raw archive (S3 here), correlation IDs?
Curriculum vs logs. Is curriculum sync separate (this repo: GET /sync, PUT /import/master) to avoid mixing with log ZIPs?

9. Primary source files (quick index)

Area	Path
Pi export	edtech-lms-rpi-api/src/modules/export/export.controller.ts
Pi log aggregation	edtech-lms-rpi-api/src/business/log.business.ts (`exportlog`)
Central import	edtech-lms-api/src/modules/log/log.controller.ts
Central merge	edtech-lms-api/src/business/log.business.ts (`importstudentsprogress`, related methods)

10. Implementation code examples

The blocks below are copied from the repos as of the line ranges shown. If line numbers drift after edits, search by symbol or route name in the listed paths.

10.1 Pi API, `GET /export/log` (build ZIP)

edtech-lms-rpi-api/src/modules/export/export.controller.ts

  @Get("log")
  @UseGuards(AccessGuard(TokenType.ACCESS))
  @HttpCode(HttpStatus.OK)
  async exportlog(
    @Response({ passthrough: true }) res: any,
    @User() user: Token
  ): Promise<any> {
    const log = await new LogBusiness().exportlog();
    const zip = new AdmZip();
    zip.addFile("log.ini", Buffer.from(JSON.stringify(log || []), "utf8"));
    // add file from log
    const files = fs.readdirSync(LOGDIR);
    files.forEach(file => {
      try {
        if(file.includes('RPI-API-error') || file.includes('RPI-API-info')) {
          const data = fs.readFileSync(LOGDIR +'/' + file, 'utf8'); // synchronous
          zip.addFile(file, Buffer.from(data.toString(), "utf8"));
        }
      } catch (e) {
        throw new InternalServerErrorException("Error read file");
      }
    });
    res.set({
      "Content-Type": "application/zip",
      "Content-Disposition": `attachment; filename="log-${new Date().toLocaleDateString()}-${new Date().toLocaleTimeString()}.zip"`,
    });
    Logger.info(`<${user.schoolusername}> export log`, {logaccesstype: LOGTYPE.EXPORTLOG, userid: user.schooluserid});
    return new StreamableFile(zip.toBuffer());
  }

10.2 Pi API, `LogBusiness.exportlog()` (shape of `log.ini` JSON)

Six-month window via subMonths; nests studentprogressquestions under each studentprogress row; bundles progress tables under log.progress.

edtech-lms-rpi-api/src/business/log.business.ts

    exportlog = async () => {
        const limitdate = subMonths(new Date(), 6);
        const sp = (
            await studentprogress.findAll({
                where: {
                    starttime: { [Op.gt]: limitdate },
                },
            })
        ).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const spqo = await studentprogressquestions.findAll({
            where: {
                studentprogressid: {
                    [Op.in]: sp.map((x) => x.studentprogressid),
                },
            },
        });
        const spq = spqo.map((x) => x.get({ plain: true }));
        const gspq = groupBy(spq, 'studentprogressid');
        const sa = await rpiuseraccess.findAll({
            where: {
                logintime: { [Op.gt]: limitdate },
            },
        });
        const stactives = (await studentactives.findAll({
            where: {
                created_at: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stlp = (await studentlearningprogress.findAll({
            where: {
                lastupdated: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stgp = (await studentgradesprogress.findAll({
            where: {
                lastupdated: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stlvp = (await studentlevelsprogress.findAll({
            where: {
                lastupdated: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stlsp = (await studentlessonsprogress.findAll({
            where: {
                lastupdated: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stpoints = (await studentpoints.findAll({
            where: {
                created_at: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        const stpusages = (await studentappusages.findAll({
            where: {
                created_at: { [Op.gt]: limitdate },
            }
        })).map((x) => ({
            ...x.get({ plain: true }),
        }));
        return {
            log: {
                result: sp.map((x) => ({
                    ...x,
                    studentprogressquestions: gspq[x.studentprogressid],
                })),
                progress: {
                    studentactives: stactives,
                    studentlearningprogress: stlp,
                    studentgradesprogress: stgp,
                    studentlevelsprogress: stlvp,
                    studentlessonsprogress: stlsp,
                    studentpoints: stpoints,
                    studentappusages: stpusages,
                },
                access: sa.map((x) => x.get({ plain: true })),
            },
        };
    };