# CanvasSync

This gem is intended to facilitate fast and easy syncing of Canvas data.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'canvas_sync'
```

Models and migrations can be installed using the following generator:

```
bin/rails generate canvas_sync:install --models users,terms,courses
```

Use the `--models` option to specify what models you would like installed. This will add both the model files and their corresponding migrations. If you'd like to install all the models that `CanvasSync` supports then specify `--models all`.

Then run the migrations:

```
bundle exec rake db:migrate
```

For a list of currently supported models, see `CanvasSync::SUPPORTED_MODELS`.

Additionally, your Canvas instance must have the "Proserv Provisioning Report" enabled.

## Docs

Docs can be generated using [yard](https://yardoc.org/). To view the docs:

- Clone this gem's repository
- `bundle install`
- `yard server --reload`

The yard server will give you a URL you can visit to view the docs.

## Basic Usage

Your tool must have an `ActiveJob` compatible job queue adapter configured, such as DelayedJob or Sidekiq. Additionally, you must have a method called `canvas_sync_client` defined in an initializer that returns a Bearcat client for the Canvas instance you are syncing against. Example:

```ruby
# config/initializers/canvas_sync.rb
def canvas_sync_client
  Bearcat::Client.new(token: current_organization.settings[:api_token], prefix: current_organization.settings[:base_url])
end
```

(Having the client defined here means the sensitive API token doesn't have to be passed in plain text between jobs.)

Once that's done and you've used the generator to create your models and migrations you can run the standard provisioning sync:

```ruby
CanvasSync.provisioning_sync(<array of models to sync>, term_scope: <optional term scope>)
```

Note: pass in 'xlist' if you would like sections to include cross listing information

Example:

```ruby
CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active)
```

This will kick off a string of jobs to sync your specified models.

If you pass in the optional `term_scope` the provisioning reports will be run for only the terms returned by that scope. The scope must be defined on your `Term` model. (A sample one is provided in the generated `Term`.)

Imports are inserted in bulk with [https://github.com/zdennis/activerecord-import](activerecord-import) so they should be very fast.

## Legacy Support

If you have an old style tool that needs to sync data on a row by row basis, you can pass in the `legacy_support: true` option. In order for this to work, your models must have a `create_or_update_from_csv` class method defined that accepts a row argument. This method will get passed each row from the CSV, and it's up to you to persist it.

Example:

```ruby
CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active, legacy_support: true)
```

## CanvasSync::JobLog

Running the migrations will create a `canvas_sync_job_logs` table. All the jobs written in this gem will create a `CanvasSync::JobLog` and store data about their arguments, job class, any exceptions, and start/completion time. This will work regardless of your queue adapter.

If you want your own jobs to also log to the table all you have to do is have your job class inherit from `CanvasSync::Job`. You can also persist extra data you might need later by saving to the `metadata` column:

```
@job_log.metadata = "This job ran really well!"
@job_log.save!
```

## Advanced Usage

This gem also helps with syncing and processing other reports if needed. In order to do so, you must:

- Define a `Processor` class that implements a `process` method for handling the results of the report
- Integrate your reports with the `ReportStarter`
- Tell the gem what jobs to run

### Processor

Your processor class must implement a `process` class method that receives a `report_file_path` and a hash of `options`. (See the `CanvasSync::Processors::ProvisioningReportProcessor` for an example.) The gem handles the work of enqueueing and downloading the report and then passes the file path to your class to process as needed. A simple example might be:

```ruby
class MyCoolProcessor
  def self.process(report_file_path, options)
    puts "I downloaded a report to #{report_file_path}! Isn't that neat!"
  end
end
```

### Report starter

You must implement a job that will enqueue a report starter for your report. (TODO: would be nice to make some sort of builder for this, so you just define the report and its params and then the gem runs it in a pre-defined job.)

Let's say we have a custom Canvas report called "my_really_cool_report_csv". First, we would need to create a job class that will enqueue a report starter. To work with the `CanvasSync` interface, your class must accept 2 parameters: `job_chain`, and `options`.

```ruby
class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
  def perform(job_chain, options)
    super(
      job_chain,
      'my_really_cool_report_csv', # Report name
      { "parameters[param1]" => true }, # Report parameters
      MyCoolProcessor, # Your processor class
      options
    )
  end
end
```

You can also see examples in `lib/canvas_sync/jobs/sync_users_job.rb` and `lib/canvas_sync/jobs/sync_provisioning_report.rb`.

### Start the jobs

The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to run. The job chain must be formatted like:

```ruby
{
  jobs: [
    { job: JobClass, options: {} },
    { job: JobClass2, options: {} }
  ],
  options: {}
}
```

Here is an example that runs our new report job first followed by the builtin provisioning job:

```
job_chain = {
  jobs: [
    { job: MyReallyCoolReportJob, options: {} },
    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
  ],
  options: {}
}

CanvasSync.process_jobs(job_chain)
```

What if you've got some other job that you want run that doesn't deal with a report? No problem! Just make sure you call `CanvasSync.invoke_next` at the end of your job. Example:


```
class SomeRandomJob < CanvasSync::Job
  def perform(job_chain, options)
    i_dunno_do_something!

    CanvasSync.invoke_next(job_chain)
  end
end

job_chain = {
  jobs: [
    { job: SomeRandomJob, options: {} },
    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
  ],
  options: {}
}

CanvasSync.process_jobs(job_chain)
```

### Batching

The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable if needed.

## Upgrading

Re-running the generator when there's been a gem change will give you several choices if it detects conflicts between your local files and the updated generators. You can either view a diff or allow the generator to overwrite your local file. In most cases you may just want to add the code from the diff yourself so as not to break any of your customizations.

Additionally, if there have been schema changes to an existing model you may have to run your own migration to bring it up to speed.

If you make updates to the gem please add any upgrade instructions here.

## Integrating with existing applications

In order for this to work properly your database tables will need to have at least the columns defined in this gem. (Adding additional columns is fine.) As such, you may need to run some migrations to rename existing columns or add missing ones. The generator only works well in a situation where that table does not already exist. Take a look at the migration templates in `lib/canvas_sync/generators/templates` to see what you need.

## Development

When adding to or updating this gem, make sure you do the following:

- If you modify a table that's already in the released version of the gem please write *new migrations* rather than modifying the existing ones. This will allow people to more easily upgrade.
- Update the yardoc comments where necessary, and confirm the changes by running `yardoc --server`
- Write specs
- If you modify the model or migration templates, run `bundle exec rake update_test_schema` to update them in the Rails Dummy application (and commit those changes)

## TODO

- Rethink how options are passed around. The current strategy of having "global_options" and per job options works
  decently, but can be confusing. The difficulty is representing the options in a way that is easily serializable
  by the queue adaptor and easily passed around.
- Add support for defining "mappings" if your models don't match up with what CanvasSync expects.